EMR ETL in other region than us-east-1?

For engineers AWS batch pipeline (Legacy)
1

Hi,

I’m trying to get the EMR ETL to work in my region (frankfurt or ireland). But I’m having problems getting it to work in any region outside us since I can’t bootstrap the EMR. The error is the following:

2016-06-16T14:55:37.494Z INFO Fetching file 's3://snowplow-hosted-assets/common/emr/snowplow-ami4-bootstrap-0.1.0.sh'
2016-06-16T14:55:38.123Z ERROR File s3://snowplow-hosted-assets/common/emr/snowplow-ami4-bootstrap-0.1.0.sh fetch failed
java.lang.RuntimeException: Error whilst fetching 's3://snowplow-hosted-assets/common/emr/snowplow-ami4-bootstrap-0.1.0.sh'
	at aws157.instancecontroller.util.S3Wrapper.fetchS3HadoopFileToLocal(S3Wrapper.java:386)
	at aws157.instancecontroller.util.S3Wrapper.fetchHadoopFileToLocal(S3Wrapper.java:325)
	at aws157.instancecontroller.util.S3Wrapper.fetchHadoopFileToLocal(S3Wrapper.java:336)
	at aws157.instancecontroller.common.InstanceConfigurator.runScript(InstanceConfigurator.java:484)
	at aws157.instancecontroller.common.InstanceConfigurator.runBootstrapAction(InstanceConfigurator.java:457)
	at aws157.instancecontroller.common.InstanceConfigurator.runBootstrapActions(InstanceConfigurator.java:292)
	at aws157.instancecontroller.common.InstanceConfigurator.doDistributionConfigure(InstanceConfigurator.java:193)
	at aws157.instancecontroller.common.InstanceConfigurator.access$200(InstanceConfigurator.java:67)
	at aws157.instancecontroller.common.InstanceConfigurator$1.run(InstanceConfigurator.java:244)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: DFE35E19EBBB3BE1), S3 Extended Request ID: lpjnkGfCci+EEP8XhqiabE4Q6Ig/8C1luRz8UGjo3EE8XqJIYs7B71i+CyH59uw1iw2UeEC+wWA=
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1383)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
	at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
	at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3676)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1179)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1054)
	at aws157.instancecontroller.util.S3Wrapper.copyS3ObjectToFile(S3Wrapper.java:277)
	at aws157.instancecontroller.util.S3Wrapper.getFile(S3Wrapper.java:261)
	at aws157.instancecontroller.util.S3Wrapper.fetchS3HadoopFileToLocal(S3Wrapper.java:373)
	... 8 more

and the problem is probably coming from here in the source of emr_job.rb::

        bootstrap_jar_location = if @legacy
          "s3://snowplow-hosted-assets/common/emr/snowplow-ami3-bootstrap-0.1.0.sh"
        else
          "s3://snowplow-hosted-assets/common/emr/snowplow-ami4-bootstrap-0.1.0.sh"
        end
        @jobflow.add_bootstrap_action(Elasticity::BootstrapAction.new(bootstrap_jar_location))

It seems like this location only works if the region where the EMR is running is set to us? (or, it is at least not working when i set it to either eu-central-1 or eu-west-1). When I set the region to us-east-1 it is working as expected.

Have anyone successfully ran the EMR in an EU region?

Thanks!

2

Hi @Niklas,

It should work in eu-west-1 as we have clients set up in that region.

Could you, please, share your configuration file (with sensitive info removed)?

Regards,
Ihor

3

Hi @ihor

Thanks for your response, I’m writing back for my colleague Niklas who is on vacation.

There was indeed an error in our configuration. The region properties (the one in s3 and the one in the emr section of the config) were not matching. Setting both to eu-west-1 solved the problem.

Sorry for the circumstances :slight_smile: