Bad gateway error upon updating collector from 2.3.1 to 2.4.5

For engineers Collectors
22

Hey @mgloel ,

I see that only the first suggestion is applied. Your Dockerfile still uses exec form of CMD, which doesn’t invoke a shell, hence no string interpolation to inject CERT_PW. (Whether you provide the executable as first CMD argument or not, it is still exec form)

If you could replace your CMD with the following

CMD /opt/snowplow/bin/snowplow-stream-collector --config oneapp_collector.conf -Dcom.amazonaws.sdk.disableCertChecking -Dcom.amazonaws.sdk.disableCbor -Djavax.net.ssl.keyStore=/opt/snowplow/ssl/collector.p12 -Djavax.net.ssl.keyStorePassword=${CERT_PW} -Djavax.net.ssl.keyStoreType=PKCS12

which is shell form of CMD, cert password will be injected into this command as expected.

Regarding the official image, Snowplow Stream Collector has official docker images. You can pull the latest one by executing

docker pull snowplow/scala-stream-collector-kinesis:2.4.5

Before you run it, you can generate an ssl cert using your script (don’t forget to provide a non-empty password), prepare your collector config, then the rest is a matter of attaching the cert and config as volumes, along with an env var JAVA_OPTS to define all JVM options.

Please let me know if there are further questions.

Kind regards

23

Ok, thanks I totally overlooked that part about the exec form.

I changed it as you suggested to:

CMD /opt/snowplow/bin/snowplow-stream-collector --config oneapp_collector.conf -Dcom.amazonaws.sdk.disableCertChecking -Dcom.amazonaws.sdk.disableCbor -Djavax.net.ssl.keyStore=/opt/snowplow/ssl/collector.p12 -Djavax.net.ssl.keyStorePassword=${CERT_PW} -Djavax.net.ssl.keyStoreType=PKCS12

The password error disappeared. This is our cloudwatch log:

Bildschirmfoto 2022-01-19 um 15.55.12
Bildschirmfoto 2022-01-19 um 15.55.121466×393 104 KB

Unfortunately our endpoint remains unreachable and returns 502 when we send data to it. We are kind of in the situation that we had in the beginning (see above).

24

Did the collector config.hocon change in version 2.4.5?

25

A new telemetry block has been added to it.

You can see an example of the bump in our collector terraform module: Bump stream-collector to 2.4.5 (close #16) · snowplow-devops/terraform-aws-collector-kinesis-ec2@43f570d · GitHub

1 Like
26

One other nice change was that we made many of the parameters optional. This means you can configure it with a completely minimal config like this one.

27

How would that look like in the dockerfile? Unfortunately we are still running into 502s

28

Hey @mgloel ,

That reference was about using official docker image where you don’t have your own Dockerfile.

1 Like
29

We are using port 9543 instead of 443 could that be an issue in the 2.4.5 version?

30

Is there another way to make it run? Can the options be added somewhere else. We need to update the collector in order to reduce the vulnerabilities on ECR. Our IT security is complaining about it. :confused:

31

Hi @mgloel ,

I just wrote down an example deployment in our documentation. I hope it comes useful.

1 Like
32

awesome, thanks