Currently I am using collector and enricher services on GCP standalone VM instances and working perfectly fine. But, now I would like to containerise these services and for that I have pulled collector container image from docker hub but when I am running a container using this image I am getting credential error. I am not finding a way how could I use gcp credentials in this container to make it working. Can some one help here ?
Command I am using to run docker container is given below :
docker run -itd -v /Users/abhishek.jain/collector/collector.conf:/snowplow/collector.conf -p 8081:5000 snowplow/scala-stream-collector-pubsub --config /snowplow/collector.conf
and the error which I am receiving is :
Exception in thread "main" java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Hi @jainahik, the trick is to mount your gcp credentials into the container as a file, and then use the
GOOGLE_APPLICATION_CREDENTIALS environment variable to point at the file you mounted.
For example, if your credentials file is at
/path/to/creds.json then you could modify your docker command like:
docker run \
-v /path/to/creds.json:/snowplow/creds.json \
-v /path/to/collector.conf:/snowplow/collector.conf \
-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/creds.json \
-p 8081:5000 \
snowplow/scala-stream-collector-pubsub --config /snowplow/collector.conf
We have some documentation for how to do this on the Enrich docker image – OK it’s not the collector, but the theory is the same.
Thanks @istreeter, This solution worked for me
@istreeter Bit curious to know little more. Can’t we assign node permissions to the container so that we don’t need to download json credentials ?
Hi @jainahik yes it is also possible to do this using a service account assigned to the compute instance. In this case, the collector will authenticate automatically by calling the VM’s metadata endpoint to get a token.
You can see an example of this in our quick-start terraform module for the collector. On this line a service account is attached to the instance. And this line has the docker run command, with no need to mount credentials.
I recommend our docs on the GCP terraform quickstart. If you are trying to set up collector and enrich then you might find a lot of the hard work has already been done for you!