Unable to pass credentials to containerised snowplow collector

For engineers Collectors
1

Hi All,
Currently I am using collector and enricher services on GCP standalone VM instances and working perfectly fine. But, now I would like to containerise these services and for that I have pulled collector container image from docker hub but when I am running a container using this image I am getting credential error. I am not finding a way how could I use gcp credentials in this container to make it working. Can some one help here ?

Command I am using to run docker container is given below :

docker run -itd  -v /Users/abhishek.jain/collector/collector.conf:/snowplow/collector.conf -p 8081:5000 snowplow/scala-stream-collector-pubsub --config /snowplow/collector.conf

and the error which I am receiving is :

Exception in thread "main" java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
	at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134)
2

Hi @jainahik, the trick is to mount your gcp credentials into the container as a file, and then use the GOOGLE_APPLICATION_CREDENTIALS environment variable to point at the file you mounted.

For example, if your credentials file is at /path/to/creds.json then you could modify your docker command like:

docker run \
  -v /path/to/creds.json:/snowplow/creds.json \
  -v /path/to/collector.conf:/snowplow/collector.conf \
  -e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/creds.json \
  -p 8081:5000 \
  snowplow/scala-stream-collector-pubsub --config /snowplow/collector.conf

We have some documentation for how to do this on the Enrich docker image – OK it’s not the collector, but the theory is the same.

1 Like
3

Thanks @istreeter, This solution worked for me :slight_smile:

4

@istreeter Bit curious to know little more. Can’t we assign node permissions to the container so that we don’t need to download json credentials ?

5

Hi @jainahik yes it is also possible to do this using a service account assigned to the compute instance. In this case, the collector will authenticate automatically by calling the VM’s metadata endpoint to get a token.

You can see an example of this in our quick-start terraform module for the collector. On this line a service account is attached to the instance. And this line has the docker run command, with no need to mount credentials.

I recommend our docs on the GCP terraform quickstart. If you are trying to set up collector and enrich then you might find a lot of the hard work has already been done for you!