QuickStart: what to do about duplicate resource errors?

1

I’m trying to follow the QuickStart for AWS. My iglu stuff deployed successfully. When it comes to the pipeline, I’m getting a ton of duplicate resource errors. It appears that resources are being created in AWS, but on a single, fresh run, I’m getting these errors and I never see a success message (or the collector_dns_name) from the terraform script.

Here’s how I’m running the script:

AWS_PROFILE=my_aws_profile terraform plan  -var-file=snowflake.terraform.tfvars

Here’s the last of the output showing the resources which have been created and the cavalcade of errors:

module.transformer_enriched[0].module.telemetry[0].random_id.auto_generated_id: Creating...
module.s3_loader_bad.module.telemetry[0].random_id.auto_generated_id: Creating...
module.enrich_kinesis.module.telemetry[0].random_id.auto_generated_id: Creating...
module.collector_kinesis.module.telemetry[0].random_id.auto_generated_id: Creating...
module.s3_loader_raw.module.telemetry[0].random_id.auto_generated_id: Creating...
module.s3_loader_enriched.module.telemetry[0].random_id.auto_generated_id: Creating...
module.snowflake_loader[0].module.telemetry[0].random_id.auto_generated_id: Creating...
module.s3_loader_raw.module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=3W1a82cG1dtYISm-5OxDLQ]
module.s3_loader_enriched.module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=vZN7znK3QbmFVr_FE9NDcw]
module.collector_kinesis.module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=YQUhz-BB6oAXWdE0Ac0UbA]
module.s3_loader_bad.module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=tfvI66QVx8nRK8pagHD1OQ]
module.enrich_kinesis.module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=gUga1iIAI5SO3xVycPlNtA]
module.snowflake_loader[0].module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=fYxalU-5pEKnPqeO2_Bb2g]
module.transformer_enriched[0].module.telemetry[0].random_id.auto_generated_id: Creation complete after 0s [id=ERwZYVaqsduFn100Hwfalg]
module.transformer_enriched[0].aws_cloudwatch_log_group.log_group[0]: Creating...
module.snowflake_loader[0].aws_iam_role.iam_role: Creating...
module.snowflake_loader[0].aws_security_group.sg: Creating...
module.transformer_enriched[0].aws_security_group.sg: Creating...
module.transformer_enriched[0].aws_iam_role.iam_role: Creating...
module.transformer_enriched[0].aws_dynamodb_table.kcl: Creating...
module.snowflake_loader[0].aws_cloudwatch_log_group.log_group[0]: Creating...
module.s3_loader_bad.aws_security_group.sg: Creating...
module.bad_2_stream.aws_kinesis_stream.stream: Creating...
module.s3_pipeline_bucket[0].aws_s3_bucket.default: Creating...
module.s3_loader_enriched.aws_security_group.sg: Creating...
module.raw_stream.aws_kinesis_stream.stream: Creating...
aws_sqs_queue.message_queue[0]: Creating...
module.enrich_kinesis.aws_cloudwatch_log_group.log_group[0]: Creating...
module.s3_loader_raw.aws_cloudwatch_log_group.log_group[0]: Creating...
module.collector_kinesis.aws_iam_role.iam_role: Creating...
module.collector_lb.aws_lb_target_group.lb_tg_http: Creating...
module.s3_loader_enriched.aws_dynamodb_table.kcl: Creating...
module.transformer_enriched[0].aws_cloudwatch_log_group.log_group[0]: Creation complete after 1s [id=/aws/ec2/sp-transformer-kinesis-enriched-server]
aws_key_pair.pipeline: Creating...
module.snowflake_loader[0].aws_cloudwatch_log_group.log_group[0]: Creation complete after 1s [id=/aws/ec2/sp-snowflake-loader-server]
module.s3_loader_raw.aws_security_group.sg: Creating...
aws_sqs_queue.message_queue[0]: Creation complete after 2s [id=https://sqs.us-east-1.amazonaws.com/478156707454/sp-sf-loader.fifo]
module.enrich_kinesis.aws_security_group.sg: Creating...
module.s3_loader_raw.aws_dynamodb_table.kcl: Creating...
module.collector_kinesis.aws_cloudwatch_log_group.log_group[0]: Creating...
module.enrich_kinesis.aws_cloudwatch_log_group.log_group[0]: Creation complete after 2s [id=/aws/ec2/sp-enrich-server]
module.collector_lb.aws_security_group.lb_sg: Creating...
module.s3_loader_raw.aws_cloudwatch_log_group.log_group[0]: Creation complete after 2s [id=/aws/ec2/sp-s3-loader-raw-server]
module.s3_loader_raw.aws_iam_role.iam_role: Creating...
module.s3_loader_enriched.aws_cloudwatch_log_group.log_group[0]: Creating...
module.collector_kinesis.aws_security_group.sg: Creating...
module.enrich_kinesis.aws_dynamodb_table.kcl: Creating...
module.enriched_stream.aws_kinesis_stream.stream: Creating...
module.collector_kinesis.aws_cloudwatch_log_group.log_group[0]: Creation complete after 1s [id=/aws/ec2/sp-collector-server]
module.enrich_kinesis.aws_iam_role.iam_role: Creating...
module.collector_lb.aws_lb_target_group.lb_tg_http: Creation complete after 2s [id=arn:aws:elasticloadbalancing:us-east-1:478156707454:targetgroup/http-20230609154306509600000001/75ab786891cd921a]
module.s3_loader_bad.aws_dynamodb_table.kcl: Creating...
module.bad_1_stream.aws_kinesis_stream.stream: Creating...
module.s3_loader_enriched.aws_cloudwatch_log_group.log_group[0]: Creation complete after 2s [id=/aws/ec2/sp-s3-loader-enriched-server]
module.enrich_kinesis.aws_dynamodb_table.config: Creating...
module.s3_loader_enriched.aws_dynamodb_table.kcl: Creation complete after 6s [id=sp-s3-loader-enriched-server]
module.s3_loader_bad.aws_iam_role.iam_role: Creating...
module.s3_loader_bad.aws_cloudwatch_log_group.log_group[0]: Creating...
module.s3_loader_bad.aws_cloudwatch_log_group.log_group[0]: Creation complete after 1s [id=/aws/ec2/sp-s3-loader-bad-server]
module.s3_loader_enriched.aws_iam_role.iam_role: Creating...
module.snowflake_loader[0].aws_iam_policy.iam_policy: Creating...
module.s3_loader_bad.aws_dynamodb_table.kcl: Creation complete after 7s [id=sp-s3-loader-bad-server]
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creating...
module.transformer_enriched[0].aws_dynamodb_table.kcl: Still creating... [10s elapsed]
module.bad_2_stream.aws_kinesis_stream.stream: Still creating... [10s elapsed]
module.raw_stream.aws_kinesis_stream.stream: Still creating... [10s elapsed]
module.snowflake_loader[0].aws_iam_policy.iam_policy: Creation complete after 1s [id=arn:aws:iam::478156707454:policy/sp-snowflake-loader-server]
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creating...
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creation complete after 1s [id=table/sp-s3-loader-enriched-server]
module.collector_kinesis.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creation complete after 1s [id=table/sp-s3-loader-enriched-server]
module.s3_loader_bad.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.collector_kinesis.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 0s [id=7f77237b-7653-47b4-8a6c-2e637333749a]
module.enrich_kinesis.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.s3_loader_raw.aws_dynamodb_table.kcl: Still creating... [10s elapsed]
module.s3_loader_bad.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 1s [id=d2752859-a731-4a02-8be4-bac61f1baadf]
module.s3_loader_raw.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.enrich_kinesis.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 1s [id=2f9057aa-77a2-44b8-8f5b-71e8f870c8da]
module.transformer_enriched[0].module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.s3_loader_raw.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 0s [id=553af341-af70-4fb0-bf23-c75845454ce9]
module.s3_loader_enriched.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.enrich_kinesis.aws_dynamodb_table.kcl: Still creating... [10s elapsed]
module.transformer_enriched[0].module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 0s [id=831e2d40-0de1-433a-a360-1f8cca6cd53d]
module.snowflake_loader[0].module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creating...
module.enriched_stream.aws_kinesis_stream.stream: Still creating... [10s elapsed]
module.s3_loader_enriched.module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 1s [id=f7ea4d69-9e68-414b-95ce-c432fd91ca62]
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creating...
module.snowflake_loader[0].module.telemetry[0].snowplow_track_self_describing_event.telemetry: Creation complete after 1s [id=fca13481-8a93-43a6-9654-c54d5210f166]
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creating...
module.bad_1_stream.aws_kinesis_stream.stream: Still creating... [10s elapsed]
module.enrich_kinesis.aws_dynamodb_table.config: Still creating... [10s elapsed]
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creation complete after 1s [id=DynamoDBWriteCapacityUtilization:table/sp-s3-loader-enriched-server]
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creating...
module.s3_loader_enriched.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creation complete after 1s [id=DynamoDBReadCapacityUtilization:table/sp-s3-loader-enriched-server]
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creating...
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creation complete after 0s [id=table/sp-s3-loader-bad-server]
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creating...
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creation complete after 1s [id=table/sp-s3-loader-bad-server]
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creating...
module.transformer_enriched[0].aws_dynamodb_table.kcl: Creation complete after 15s [id=sp-transformer-kinesis-enriched-server]
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creating...
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creation complete after 1s [id=table/sp-transformer-kinesis-enriched-server]
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creation complete after 1s [id=DynamoDBWriteCapacityUtilization:table/sp-s3-loader-bad-server]
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creating...
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creating...
module.s3_loader_bad.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creation complete after 1s [id=DynamoDBReadCapacityUtilization:table/sp-s3-loader-bad-server]
module.s3_loader_raw.aws_dynamodb_table.kcl: Creation complete after 14s [id=sp-s3-loader-raw-server]
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creating...
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creating...
module.enrich_kinesis.aws_dynamodb_table.kcl: Creation complete after 14s [id=sp-enrich-server]
module.enrich_kinesis.aws_dynamodb_table.config: Creation complete after 12s [id=sp-enrich-server-config]
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creating...
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creating...
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creation complete after 0s [id=table/sp-transformer-kinesis-enriched-server]
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_event_fingerprint_config: Creating...
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creation complete after 1s [id=DynamoDBReadCapacityUtilization:table/sp-transformer-kinesis-enriched-server]
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_yauaa_enrichment_config: Creating...
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creation complete after 1s [id=table/sp-s3-loader-raw-server]
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creation complete after 1s [id=table/sp-s3-loader-raw-server]
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_referer_parser: Creating...
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_campaign_attribution: Creating...
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_target.read_target: Creation complete after 1s [id=table/sp-enrich-server]
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_ua_parser_config: Creating...
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_target.write_target: Creation complete after 1s [id=table/sp-enrich-server]
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_event_fingerprint_config: Creation complete after 1s [id=sp-enrich-server-config|id||snowplow_enrichment_event_fingerprint_config|]
module.enrich_kinesis.aws_dynamodb_table_item.iglu_resolver: Creating...
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creating...
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_yauaa_enrichment_config: Creation complete after 0s [id=sp-enrich-server-config|id||snowplow_enrichment_yauaa_enrichment_config|]
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creating...
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_campaign_attribution: Creation complete after 1s [id=sp-enrich-server-config|id||snowplow_enrichment_campaign_attribution|]
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_referer_parser: Creation complete after 1s [id=sp-enrich-server-config|id||snowplow_enrichment_referer_parser|]
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_target.write_target: Creating...
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creating...
module.enrich_kinesis.aws_dynamodb_table_item.enrichment_ua_parser_config: Creation complete after 1s [id=sp-enrich-server-config|id||snowplow_enrichment_ua_parser_config|]
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_target.read_target: Creating...
module.enrich_kinesis.aws_dynamodb_table_item.iglu_resolver: Creation complete after 1s [id=sp-enrich-server-config|id||snowplow_resolver|]
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creating...
module.transformer_enriched[0].module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creation complete after 1s [id=DynamoDBWriteCapacityUtilization:table/sp-transformer-kinesis-enriched-server]
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creating...
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_target.write_target: Creation complete after 0s [id=table/sp-enrich-server-config]
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_target.read_target: Creation complete after 0s [id=table/sp-enrich-server-config]
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creation complete after 1s [id=DynamoDBReadCapacityUtilization:table/sp-s3-loader-raw-server]
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_policy.write_policy: Creating...
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_policy.read_policy: Creating...
module.s3_loader_raw.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creation complete after 1s [id=DynamoDBWriteCapacityUtilization:table/sp-s3-loader-raw-server]
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_policy.read_policy: Creation complete after 1s [id=DynamoDBReadCapacityUtilization:table/sp-enrich-server]
module.enrich_kinesis.module.kcl_autoscaling.aws_appautoscaling_policy.write_policy: Creation complete after 1s [id=DynamoDBWriteCapacityUtilization:table/sp-enrich-server]
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_policy.read_policy: Creation complete after 1s [id=DynamoDBReadCapacityUtilization:table/sp-enrich-server-config]
module.enrich_kinesis.module.config_autoscaling.aws_appautoscaling_policy.write_policy: Creation complete after 1s [id=DynamoDBWriteCapacityUtilization:table/sp-enrich-server-config]
module.bad_2_stream.aws_kinesis_stream.stream: Still creating... [20s elapsed]
module.raw_stream.aws_kinesis_stream.stream: Still creating... [20s elapsed]
module.enriched_stream.aws_kinesis_stream.stream: Still creating... [20s elapsed]
module.bad_1_stream.aws_kinesis_stream.stream: Still creating... [20s elapsed]
module.bad_2_stream.aws_kinesis_stream.stream: Still creating... [30s elapsed]
module.raw_stream.aws_kinesis_stream.stream: Still creating... [30s elapsed]
module.enriched_stream.aws_kinesis_stream.stream: Still creating... [30s elapsed]
module.bad_1_stream.aws_kinesis_stream.stream: Still creating... [30s elapsed]
module.bad_2_stream.aws_kinesis_stream.stream: Still creating... [40s elapsed]
module.raw_stream.aws_kinesis_stream.stream: Still creating... [40s elapsed]
module.enriched_stream.aws_kinesis_stream.stream: Still creating... [40s elapsed]
module.bad_1_stream.aws_kinesis_stream.stream: Still creating... [40s elapsed]
module.bad_2_stream.aws_kinesis_stream.stream: Creation complete after 44s [id=arn:aws:kinesis:us-east-1:478156707454:stream/sp-bad-2-stream]
module.raw_stream.aws_kinesis_stream.stream: Creation complete after 44s [id=arn:aws:kinesis:us-east-1:478156707454:stream/sp-raw-stream]
module.enriched_stream.aws_kinesis_stream.stream: Creation complete after 44s [id=arn:aws:kinesis:us-east-1:478156707454:stream/sp-enriched-stream]
module.transformer_enriched[0].aws_iam_policy.iam_policy: Creating...
module.bad_1_stream.aws_kinesis_stream.stream: Creation complete after 44s [id=arn:aws:kinesis:us-east-1:478156707454:stream/sp-bad-1-stream]
module.collector_kinesis.aws_iam_policy.iam_policy: Creating...
module.enrich_kinesis.aws_iam_policy.iam_policy: Creating...
╷
│ Error: Error import KeyPair: InvalidKeyPair.Duplicate: The keypair already exists
│ 	status code: 400, request id: c97d4040-3fb4-4c9d-9186-c6b8ae6b1529
│
│   with aws_key_pair.pipeline,
│   on main.tf line 45, in resource "aws_key_pair" "pipeline":
│   45: resource "aws_key_pair" "pipeline" {
│
╵
╷
│ Error: Error creating IAM Role sp-collector-server: EntityAlreadyExists: Role with name sp-collector-server already exists.
│ 	status code: 409, request id: a25bfee8-49aa-4558-a634-344c6c114aa3
│
│   with module.collector_kinesis.aws_iam_role.iam_role,
│   on .terraform/modules/collector_kinesis/main.tf line 76, in resource "aws_iam_role" "iam_role":
│   76: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: error creating IAM policy sp-collector-server: EntityAlreadyExists: A policy called sp-collector-server already exists. Duplicate names are not allowed.
│ 	status code: 409, request id: aa7979c5-4ec8-47a4-87b1-679c457148c1
│
│   with module.collector_kinesis.aws_iam_policy.iam_policy,
│   on .terraform/modules/collector_kinesis/main.tf line 97, in resource "aws_iam_policy" "iam_policy":
│   97: resource "aws_iam_policy" "iam_policy" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-collector-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: f97a583f-2cdf-4ae7-bbba-31dd3278237c
│
│   with module.collector_kinesis.aws_security_group.sg,
│   on .terraform/modules/collector_kinesis/main.tf line 145, in resource "aws_security_group" "sg":
│  145: resource "aws_security_group" "sg" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-collector-lb' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: cce450ef-8683-49fa-856b-3d50fd924fe0
│
│   with module.collector_lb.aws_security_group.lb_sg,
│   on .terraform/modules/collector_lb/main.tf line 12, in resource "aws_security_group" "lb_sg":
│   12: resource "aws_security_group" "lb_sg" {
│
╵
╷
│ Error: Error creating IAM Role sp-enrich-server: EntityAlreadyExists: Role with name sp-enrich-server already exists.
│ 	status code: 409, request id: 1bbc6a57-301a-464e-9da1-bff760939bc5
│
│   with module.enrich_kinesis.aws_iam_role.iam_role,
│   on .terraform/modules/enrich_kinesis/main.tf line 154, in resource "aws_iam_role" "iam_role":
│  154: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: error creating IAM policy sp-enrich-server: EntityAlreadyExists: A policy called sp-enrich-server already exists. Duplicate names are not allowed.
│ 	status code: 409, request id: 7ce2381b-e1e3-489c-9a8b-0308ed3712c2
│
│   with module.enrich_kinesis.aws_iam_policy.iam_policy,
│   on .terraform/modules/enrich_kinesis/main.tf line 175, in resource "aws_iam_policy" "iam_policy":
│  175: resource "aws_iam_policy" "iam_policy" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-enrich-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: 46d3dabf-13dc-4fcd-bc1a-968d30bf4118
│
│   with module.enrich_kinesis.aws_security_group.sg,
│   on .terraform/modules/enrich_kinesis/main.tf line 306, in resource "aws_security_group" "sg":
│  306: resource "aws_security_group" "sg" {
│
╵
╷
│ Error: Error creating IAM Role sp-s3-loader-bad-server: EntityAlreadyExists: Role with name sp-s3-loader-bad-server already exists.
│ 	status code: 409, request id: a8170ae1-619e-4062-8903-d1e482e99190
│
│   with module.s3_loader_bad.aws_iam_role.iam_role,
│   on .terraform/modules/s3_loader_bad/main.tf line 110, in resource "aws_iam_role" "iam_role":
│  110: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-s3-loader-bad-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: 3c6d1afd-b345-4bbe-b0bf-351a64ef071c
│
│   with module.s3_loader_bad.aws_security_group.sg,
│   on .terraform/modules/s3_loader_bad/main.tf line 220, in resource "aws_security_group" "sg":
│  220: resource "aws_security_group" "sg" {
│
╵
╷
│ Error: Error creating IAM Role sp-s3-loader-enriched-server: EntityAlreadyExists: Role with name sp-s3-loader-enriched-server already exists.
│ 	status code: 409, request id: 08570a58-163b-4480-af75-ef3ffe5a65d2
│
│   with module.s3_loader_enriched.aws_iam_role.iam_role,
│   on .terraform/modules/s3_loader_enriched/main.tf line 110, in resource "aws_iam_role" "iam_role":
│  110: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-s3-loader-enriched-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: b4eb6992-6085-4240-9ae2-956fbc930782
│
│   with module.s3_loader_enriched.aws_security_group.sg,
│   on .terraform/modules/s3_loader_enriched/main.tf line 220, in resource "aws_security_group" "sg":
│  220: resource "aws_security_group" "sg" {
│
╵
╷
│ Error: Error creating IAM Role sp-s3-loader-raw-server: EntityAlreadyExists: Role with name sp-s3-loader-raw-server already exists.
│ 	status code: 409, request id: bc94070d-9b48-488b-84d9-eb949ed34670
│
│   with module.s3_loader_raw.aws_iam_role.iam_role,
│   on .terraform/modules/s3_loader_raw/main.tf line 110, in resource "aws_iam_role" "iam_role":
│  110: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-s3-loader-raw-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: 9e072beb-4fd7-4c80-8c95-33c0b08e8534
│
│   with module.s3_loader_raw.aws_security_group.sg,
│   on .terraform/modules/s3_loader_raw/main.tf line 220, in resource "aws_security_group" "sg":
│  220: resource "aws_security_group" "sg" {
│
╵
╷
│ Error: Error creating S3 bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
│ 	status code: 400, request id: W9APC2NQTC45BDJV, host id: xaC8vvoeeLPO292M7QmLv0rlwuTnuQn0CGYN8XBtdJK+vJjPf2iDgOfbwNU43wstaYwAmNwx5Is=
│
│   with module.s3_pipeline_bucket[0].aws_s3_bucket.default,
│   on .terraform/modules/s3_pipeline_bucket/main.tf line 1, in resource "aws_s3_bucket" "default":
│    1: resource "aws_s3_bucket" "default" {
│
╵
╷
│ Error: Error creating IAM Role sp-snowflake-loader-server: EntityAlreadyExists: Role with name sp-snowflake-loader-server already exists.
│ 	status code: 409, request id: daf02dcc-dfeb-4433-89cc-6df9c7f8909b
│
│   with module.snowflake_loader[0].aws_iam_role.iam_role,
│   on .terraform/modules/snowflake_loader/main.tf line 76, in resource "aws_iam_role" "iam_role":
│   76: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-snowflake-loader-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: b49dc039-e453-4b5e-950e-9058d5d0e782
│
│   with module.snowflake_loader[0].aws_security_group.sg,
│   on .terraform/modules/snowflake_loader/main.tf line 163, in resource "aws_security_group" "sg":
│  163: resource "aws_security_group" "sg" {
│
╵
╷
│ Error: Error creating IAM Role sp-transformer-kinesis-enriched-server: EntityAlreadyExists: Role with name sp-transformer-kinesis-enriched-server already exists.
│ 	status code: 409, request id: ff6cd411-a289-4de3-96a4-e462546c68fc
│
│   with module.transformer_enriched[0].aws_iam_role.iam_role,
│   on .terraform/modules/transformer_enriched/main.tf line 142, in resource "aws_iam_role" "iam_role":
│  142: resource "aws_iam_role" "iam_role" {
│
╵
╷
│ Error: error creating IAM policy sp-transformer-kinesis-enriched-server: EntityAlreadyExists: A policy called sp-transformer-kinesis-enriched-server already exists. Duplicate names are not allowed.
│ 	status code: 409, request id: 60e45ac4-4807-41ca-908a-24691d6371b0
│
│   with module.transformer_enriched[0].aws_iam_policy.iam_policy,
│   on .terraform/modules/transformer_enriched/main.tf line 163, in resource "aws_iam_policy" "iam_policy":
│  163: resource "aws_iam_policy" "iam_policy" {
│
╵
╷
│ Error: Error creating Security Group: InvalidGroup.Duplicate: The security group 'sp-transformer-kinesis-enriched-server' already exists for VPC 'vpc-0926b368b8db659cb'
│ 	status code: 400, request id: 2e0e615a-9447-4aca-bf43-aa04773651b7
│
│   with module.transformer_enriched[0].aws_security_group.sg,
│   on .terraform/modules/transformer_enriched/main.tf line 266, in resource "aws_security_group" "sg":
│  266: resource "aws_security_group" "sg" {
│
╵
2

Hi @scelerat it sounds like your environment is a little bit broken or potentially someone else is deploying into it as well.

An easy option if you are going for a second deployment here is to change the “prefix” to not be “sp” which will remove the conflicts you see here and deploy resources that are not going to conflict. However it seems like you have some dangling resources here with the default prefix - these will need to be deleted manually in the AWS UI as it seems Terraform did not record them into your local statefile.

3

Hi Josh, so changing the prefix indeed got rid of a lot of the errors. Now it’s down to this one,

╷
│ Error: Error creating S3 bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
│ 	status code: 400, request id: HNZS4CSHGDA9PF0B, host id: YGiOd2qKEGSl+OxxzlGzmXtLdRlmn3dJ+O/e7Hz7BFM7WPk/u3eFtCnNpiCywRZYzpySitwpJ+o=
│
│   with module.s3_pipeline_bucket[0].aws_s3_bucket.default,
│   on .terraform/modules/s3_pipeline_bucket/main.tf line 1, in resource "aws_s3_bucket" "default":
│    1: resource "aws_s3_bucket" "default" {

I searched through all the config files and the only instance of us-west-2 I can find is in the secure directory, and I’ve been editing and deploying files from the default directory. I can’t figure out where this might be coming from. Everything I’ve deployed so far (iglu) has been into us-east-1

4

How have you configured your Terraform provider? Are you reading from an AWS Profile locally?

5

yes, reading from an AWS profile locally. What is terraform provider?

I did terraform init, terraform plan, terraform apply, all prefixed with AWS_PROFILE=profilename, and that profile is for a user in us-east-1. All the resources get created in us-east-1, I just get this last message.

6

So in this case a Terraform provider is a plugin used with Terraform - you are using the “AWS” Terraform provider as part of this which can be configured as well: Terraform Registry

I was wondering if you had accidentally got an AWS provider configuration block somewhere in your Terraform which is using the wrong region somehow → the value has to be coming from somewhere!

7

So here’s what I did. Cloned the quickstart repo into a new, clean directory. Changed the prefix variable to sp5 to not interfere with any other resources created with previous terraform apply runs. Set all the required variables, VPC, ssh keys, etc. Everything refers to resources in us-east-1. I still get this error:

╷
│ Error: Error creating S3 bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
│ 	status code: 400, request id: WW6WQKY1T61QKCSA, host id: b+Lq9yU1BdKBDZLuy4xVYWZwDGq1JxO6VV4iPjBRSPSoPAnSO8LfVowcgBchpb0tBejMhl9CGdc=
│
│   with module.s3_pipeline_bucket[0].aws_s3_bucket.default,
│   on .terraform/modules/s3_pipeline_bucket/main.tf line 1, in resource "aws_s3_bucket" "default":
│    1: resource "aws_s3_bucket" "default" {
│
8

@scelerat I think I know what this is! Someone else in the world has already used sp5 which means the bucket already exists in someone else’s account. Given that s3-buckets have to be universally unique I think this is likely what has happened.

Can you try setting s3_bucket_name to something more unique than the default that is there?

9

Hm that seemed like a good guess, but not I get the same result as before, even with a very random prefix.

10

@scelerat have you set the bucket_name to something else? quickstart-examples/terraform/aws/pipeline/default/postgres.terraform.tfvars at main · snowplow/quickstart-examples · GitHub

This is a separate variable to the prefix.

11

Aha, so yes I changed the bucket name and now we’re onto a different set of errors. Changing the bucket name got me past the region error I was seeing above. Now I see this:

╷
│ Error: Got -1 status code when sending event - need 2xx or 3xx
│
│   with module.s3_loader_raw.module.telemetry[0].snowplow_track_self_describing_event.telemetry,
│   on .terraform/modules/s3_loader_raw.telemetry/main.tf line 35, in resource "snowplow_track_self_describing_event" "telemetry":
│   35: resource "snowplow_track_self_describing_event" "telemetry" {
│
╵
╷
│ Error: Got -1 status code when sending event - need 2xx or 3xx
│
│   with module.snowflake_loader[0].module.telemetry[0].snowplow_track_self_describing_event.telemetry,
│   on .terraform/modules/snowflake_loader.telemetry/main.tf line 35, in resource "snowplow_track_self_describing_event" "telemetry":
│   35: resource "snowplow_track_self_describing_event" "telemetry" {
│
╵
╷
│ Error: Got -1 status code when sending event - need 2xx or 3xx
│
│   with module.transformer_enriched[0].module.telemetry[0].snowplow_track_self_describing_event.telemetry,
│   on .terraform/modules/transformer_enriched.telemetry/main.tf line 35, in
12

Right so these seem to be issues with tracking telemetry - are you working in a corporate environment potentially that is blocking sending requests out? Easiest option here is to just disable telemetry: quickstart-examples/terraform/aws/pipeline/default/postgres.terraform.tfvars at main · snowplow/quickstart-examples · GitHub

If you change that boolean to “false” these errors should clean themselves up.

13

I am running these commands from my home computer/network and not from, e.g. an EC2 instance

14

Aand that did it! All green, got the collector_dns_name.

So to review, I think the first time I ran the quickstart, somehow terraform and/or my AWS was in a state where some resources were not being deleted. When I started fresh and using a different resource prefix, everything worked up until the s3 bucket name, which as you noticed is global, so changing that to something unique led me to hitting the telemetry issue, which was solved by changing the telemetry_enabled value to false.

Thanks for all your patient help

2 Likes
15

Great to hear you are up and running! Terraform can be a bit of a beast to debug sometimes.

16

Ok next question about sending test events:

I tried sending a test event like this:

curl 'https://{{COLLECTOR_URL}}/com.snowplowanalytics.snowplow/tp2' \
   -H 'Content-Type: application/json; charset=UTF-8' \
   -H 'Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c' \
   --data-raw '{"schema":"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4","data":[{"e":"pv","url":"/docs/open-source-quick-start/quick-start-installation-guide-on-aws/send-test-events-to-your-pipeline/","page":"Send test events to your pipeline - Snowplow Docs","refr":"https://docs.snowplow.io/","tv":"js-2.17.2","tna":"spExample","aid":"docs-example","p":"web","tz":"Europe/London","lang":"en-GB","cs":"UTF-8","res":"3440x1440","cd":"24","cookie":"1","eid":"4e35e8c6-03c4-4c17-8202-80de5bd9d953","dtm":"1626182778191","cx":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiI0YTU2ZjQyNy05MTk2LTQyZDEtOWE0YS03ZjRlNzk2OTM3ZmEifX1dfQ","vp":"863x1299","ds":"848x5315","vid":"3","sid":"87c18fc8-2055-4ec4-8ad6-fff64081c2f3","duid":"5f06dbb0-a893-472b-b61a-7844032ab3d6","stm":"1626182778194"},{"e":"ue","ue_px":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3Byb2R1Y3Rfdmlldy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJpZCI6IjVOMFctUEwwVyIsImN1cnJlbnRfcHJpY2UiOjQ0Ljk5LCJkZXNjcmlwdGlvbiI6IlB1cnBsZSBTbm93cGxvdyBIb29kaWUifX19","tv":"js-2.17.2","tna":"spExample","aid":"docs-example","p":"web","tz":"Europe/London","lang":"en-GB","cs":"UTF-8","res":"3440x1440","cd":"24","cookie":"1","eid":"542a79d3-a3b8-421c-99d6-543ff140a56a","dtm":"1626182778193","cx":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiI0YTU2ZjQyNy05MTk2LTQyZDEtOWE0YS03ZjRlNzk2OTM3ZmEifX1dfQ","vp":"863x1299","ds":"848x5315","vid":"3","sid":"87c18fc8-2055-4ec4-8ad6-fff64081c2f3","duid":"5f06dbb0-a893-472b-b61a-7844032ab3d6","refr":"https://docs.snowplow.io/","url":"/docs/open-source-quick-start/quick-start-installation-guide-on-aws/send-test-events-to-your-pipeline/","stm":"1626182778194"}]}'

where COLLECTOR_URL is the value that I got from the terraform script after it succeeded and gave me ‘collector_dns_name’

The result I get is

curl: (7) Failed to connect to {{collector_dns_name}} port 443 after 303 ms: Couldn't connect to server

Is that the correct URL? Is there more configuration to do on the AWS side?

17

That should be correct, yes.

I’m not entirely sure why you’d have gotten that response, but I can suggest first pinging the health endpoint with:

curl 'http://{{COLLECTOR_URL}}/health'

If that fails, something is wrong on the infrastructure side - the collector is either not alive at all or not reachable.

Then, I’d use either one of the trackers, or the tracking cli to send some data. The protocol the collector expects is more complicated than what you’ve sent - it still should’ve reached the collector mind you, using the CLI is just a way to avoid having to construct tracker protocol requests.

I would start with sending over http, to see if that works - then test with https. Just to try to narrow down where any issue might lie. (Ie if http works but not https, then any tls related configuration is the first place to look). :slight_smile:

18

Ok, HTTP worked, not HTTPS. I may need to fix that at some point, but now that I can send an event, how do I see where it went? Is it in a dynamoDB table or s3 bucket somewhere?

I see the streams and dynamodb tables set up in AWS, and I might even be looking at evidence in one of the streams that an event came through, but is there any way I can inspect it directly? I’m looking at the raw stream right now, and see on the GetRecords chart counts coming through at the time I did the curl requests. But if I go to the Data Viewer tab, I don’t see anything

19

So this is where the loaders come in. If you have deployed the “s3 loaders” then you should have data landing in the bucket you setup on S3 partitioned into “raw”, “enriched” and “bad”.

The raw data is notoriously hard to do much with as it is thrift encoded so best bet is to look at the split of good and bad data - the former is TSV and the later is JSON.

However to truly get something out of it you want the data loading into a warehouse - I believe you were looking at getting Snowflake configured?

21

Yes, looking at getting it into snowflake. We have a snowflake DB hosted in MS Azure