This is the part I am struggling with within the installation process.
I see that Snowplow provides a docker container here but I am not sure if I should use Compute Engine for that or Cloud Run or maybe Appengine Flexible.
And how I could get the config files to the instance if I used Compute Engine?
I followed this tutorial and was able to get it done after facing some errors.
But still think the container would have been better.
Hi @Marwan_abdel_moneim ,
Glad to hear that you managed to get it to work.
The purpose of this container is to start the Dataflow job. Internally we use Kubernetes Engine to run it, but it can also be done from a Compute Engine instance. I’m adding this precision to the documentation, thank you for noticing!
If it requires uploading config files, please add to the docs how to do that.
Config files need to be accessible from where docker run is run, for -v $PWD/config:/snowplow/config. So if you run Docker on a Compute Engine instance then you need to copy the config files on it
You don’t create the instance then run docker, right?
You configure it while creating the instance.
Anyway, so how you do it in Kubernetes Engine?
Hi @Marwan_abdel_moneim ,
Based on this thread I take it that you managed to run Beam Enrich, great !