Eu-central-1-specific hosted assets, AccessDenied

For engineers Storage targets
1

Hello Snowplow Team,

We’re currently migrating our stack to eu-central-1. To be able to load data into Redshift, I try to use your yet unreleased r82 branch which introduces support for other regions.

According to your changes here https://github.com/snowplow/snowplow/blob/release/r82/4-storage/storage-loader/lib/snowplow-storage-loader/shredded_type.rb#L132, the Redshift storage loader tries to access the bucket snowplow-hosted-assets-eu-central-1.s3-eu-central-1.amazonaws.com. But the API user (which has s3.* access) doesn’t get access to that bucket:

Unexpected error: Expected(200) <=> Actual(403 Forbidden)
excon.error.response
  :body          => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>***</RequestId><HostId>***</HostId></Error>"
  :cookies       => [
  ]
  :headers       => {
    "Content-Type"        => "application/xml"
    "Date"                => "Thu, 28 Jul 2016 15:32:28 GMT"
    "Server"              => "AmazonS3"
    "x-amz-bucket-region" => "eu-central-1"
    "x-amz-id-2"          => "***"
    "x-amz-request-id"    => "***"
  }
  :host          => "snowplow-hosted-assets-eu-central-1.s3-eu-central-1.amazonaws.com"
  :local_address => "***"
  :local_port    => 47522
  :path          => "/"
  :port          => 443
  :reason_phrase => "Forbidden"
  :remote_ip     => "***"
  :status        => 403
  :status_line   => "HTTP/1.1 403 Forbidden\r\n"
  • Does this bucket exist?
  • Has the config file changed somehow for that new release?
2

Hi @phil,

The eu-central-1 region is troublesome at the moment. We are still in process of overcoming the obstacles on AWS side to get it working with the EMR Runner. I would stay clear of that region for now.

Hopefully, we will resolve it within the next week or so.

–Ihor

3

@Ihor is right - this support is still a work-in-progress. However I quickly checked the bucket in question and it was indeed missing the Everyone: Read permission. Now fixed. Please let us know if it works now?

4

@alex The bucket is now accessible, thank you.