Snowflake Loader Setup Error

For engineers
1

Problem:

We are trying to set up our from S3 to Snowflake loader and have an error we can’t seem to overcome. https://github.com/snowplow-incubator/snowplow-snowflake-loader/wiki/Setup-Guide#setup is the guide we are following.

When running the java -jar snowplow-snowflake-loader-0.4.0.jar \ setup \ --config /path/to/self-describing-config.json \ --resolver /path/to/resolver.json

The following is produced:

User:snowflake-loader $ java -jar snowplow-snowflake-loader-0.4.0.jar  setup   --config ./config.json     --resolver ./resolver.json 
Setting up...
Jun 03, 2019 4:04:18 PM shadeaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Jun 03, 2019 4:04:18 PM shadeaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Exception in thread "main" java.lang.RuntimeException: Cannot retrieve JDBC password from EC2 Parameter Store. Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
	at com.snowplowanalytics.snowflake.loader.connection.Jdbc$.getConnection(Jdbc.scala:49)
	at com.snowplowanalytics.snowflake.loader.Initializer$.run(Initializer.scala:24)
	at com.snowplowanalytics.snowflake.loader.Main$.main(Main.scala:25)
	at com.snowplowanalytics.snowflake.loader.Main.main(Main.scala)

How we got here:

We followed the guide to a T. There are too many steps to outline here, if they seem important feel free to ask and I will share anything necessary. However, instead of downloading the loader, and then setting up the transformer as the guide suggests, I instead downloaded both the transformer and loader and then used the set up command on the Loader.
The guide says to use the setup command when running the Transformer jar, but that produced some errors. I induced that it meant to run the Loader, and it seems to be the right thing except we get the above error.

Contents of our config.json

Please note that there are placeholders below:

{
  "schema": "iglu:com.snowplowanalytics.snowplow.storage/snowflake_config/jsonschema/1-0-1",
  "data": {
    "name": "Snowflake Storage Target",
    "awsRegion": "us-east-1",
    "auth": {
        "roleArn": "arn:aws:iam::01234567891234:role/SnowflakeLoadRole",
        "sessionDuration": 900
    },
    "manifest": "snowplow-snowflake-manifest",
    "snowflakeRegion": "us-east-1",
    "database": "snowplowdb",
    "input": "s3://stg-datalake-snowplow-enriched-stream",
    "stage": "snowplow_stage",
    "stageUrl": "s3:stg-datalake-snowplow-snowflake/transformed",
    "warehouse": "snowplow_wh",
    "schema": "atomic",
    "account": "ourAccountName",
    "username": "snowplow_loader",
    "password": {
        "ec2ParameterStore": {
            "parameterName": "snowplow.snowflake.password"
        }
    },
    "maxError": 1,
    "purpose": "ENRICHED_EVENTS"
  }
}

Contents of our resolver.json

{  
   "schema":"iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1",
   "data":{  
      "cacheSize":500,
      "repositories":[  
         {  
            "name":"Iglu Central",
            "priority":0,
            "vendorPrefixes":[  
               "com.snowplowanalytics"
            ],
            "connection":{  
               "http":{  
                  "uri":"http://iglucentral.com"
               }
            }
         },
         {  
            "name":"Iglu Central - GCP Mirror",
            "priority":1,
            "vendorPrefixes":[  
               "com.snowplowanalytics"
            ],
            "connection":{  
               "http":{  
                  "uri":"http://mirror01.iglucentral.com"
               }
            }
         }
      ]
   }
}

My thoughts

I am honestly at a loss on where to go. I have tried quite a few things regarding the region code that i am supplying to the loader and nothing seems to change it at all. I have downloaded the source code and looked at the Scala to get an idea as to what could be causing the error, https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/master/loader/src/main/scala/com.snowplowanalytics.snowflake.loader/connection/Jdbc.scala. Nothing seems to give. Does anyone have any ideas as to why we are getting this error?

2

Hi @abrhim,

There are two things that are wrong fore sure:

  1. The exception tells you that it cannot retrieve a JDBC password. Make sure that snowplow.snowflake.password EC2 parameter exists user with which you run Loader has access to it. E.g. you can try to run aws ssm get-parameters --names "snowplow.snowflake.password"
  2. stageUrl scheme is invalid. Should be s3://stg-datalake-snowplow-snowflake/transformed instead of s3:stg-datalake-snowplow-snowflake/transformed. If Loader already managed to create a stage - you need to fix it manually.
1 Like