Running r89 build.sh failed

For engineers Enrichment
1

Since we have to use aws federated credentails, we need to slightly modify your source code then re-build the jar. But after downloading your source code snowplow-r89-plain-of-jars.zip and building it (without any changes yet) under snowplow-r89-plain-of-jars/3-enrich/emr-etl-runner/build.sh, we got the following err:

[Coveralls] Outside the CI environment, not sending data.
Runing Rake
Unable to detect bundler spec under ‘/Users/bjff/.rvm/gems/jruby-9.1.6.0/gems/bundler-1.16.0’’ and its sub-dirs
cat: deploy/snowplow-emr-etl-runner.jar: No such file or directory

Any help would be appreciated.

2

Hi @RichardJ,

There is a weird incompatibility between warbler and bundler at the moment.

The problem is that the latest rvm installations comes with bundler 1.16, version which contains the problem.

To circumvent this issue, we use rvm 1.27.0 and force the installation of bundler 1.15.4: https://github.com/snowplow/snowplow/blob/3ce332e37596b964831ca4dd32bb1054cfce5c2e/.travis.yml#L45-L50

2 Likes
3

Thank you for this answer @BenFradet

Also rvm 1.29.3 with bundler 1.15.4 does the job. Built with on Ubuntu 17.10 Docker image.

4

Hi Ben,

I was drawn away by other projects for the last two months, and now back to this issue again. As said, we cannot use “access_key_id/secret_access_key” in config.yml, so I changed the emr_jobs.rb:

Elasticity.configure do |c|
if config[:aws][:access_key_id] == ‘iam’ and config[:aws][:secret_access_key] == ‘iam’
#federated Identity Management
credentials_from_role = Aws::InstanceProfileCredentials.new.credentials
c.access_key = credentials_from_role.access_key_id
c.secret_key = credentials_from_role.secret_access_key
c.session_token = credentials_from_role.aws_session_token
else
c.access_key = config[:aws][:access_key_id]
c.secret_key = config[:aws][:secret_access_key]
end

plus a few small changes in Gemfile & Gemfile.lock. But when running “./snowplow-emr-etl-runner --config ${RUNNER_CONFIG} …”, we got this error:

Excon::Error::Forbidden (Expected(200) <=> Actual(403 Forbidden)
excon.error.response
:body => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\nInvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.iam … ".

Any ideas? It looks like more changes may be needed?
Thanks for help,
Richard