Hello
We see a huge increase in CloudWatch costs recently.
What we see is a lot of PutMetrics requests into a CloudWatch namespace which are similar to the names of DynamoDB tables used by snowplow.
Also,I can verify that the Role is used by out snowplow pipeline.
Is there a configuration that I can set to on/off reporting the metrics?
Any ideas how to approach this ?
I found this topic Turn off EMR monitoring but there is not comment there
Thanks
Hi @eko - in the configuration files for the KCL based apps you should search for this setting:
Stream Enrich: stream-enrich/config.hocon.sample at master · snowplow/stream-enrich · GitHub
S3 Loader: snowplow-s3-loader/config.hocon.sample at master · snowplow/snowplow-s3-loader · GitHub
Elasticsearch Loader: snowplow-elasticsearch-loader/config.hocon.sample at master · snowplow/snowplow-elasticsearch-loader · GitHub
This will prevent the KCL from publishing custom metrics to CloudWatch.
Cheers,
Josh
Thanks @josh, I added the config as you suggested, seems like the costs have been reduced, but, still higher than the normal. (double)
Does the collector takes the same config? any other component reports to CloudWatch by default?
Thank you
Hi @eko this was more to answer your question about the PutMetrics for those namespaces. Externally to that I am sorry there is not much more I can do to help there - it might be a good idea also to reach out to Amazon Support for help on this if you cannot figure out whats causing the hike in costs!
Hi @josh ,
Added “disableCloudWatch = true” to the configuration files for both apps yet PutMetrics requests are still high. (confirmed it with AWS support, the source of the requests are those apps)
We use stream-enrich v0.19.4 and s3-loader v0.6.0,
Could be this configuration is not supported by this versions?
Since which version of stream-enrich and s3-loader reporting PutMetrics is enabled by default ?
Thank you
Hi @eko - looking at the Git history for those projects that would be a no - neither of them supported that setting. You will need to upgrade to be able to take advantage of that setting.
For the S3 Loader v1.0.0 and for Stream Enrich v1.3.2 would be the versions I would recommend to start using.
Thank you @josh for the quick response
Currently in my Dockerfile:
FROM snowplow-docker-registry.bintray.io/snowplow/base-alpine:0.2.0
LABEL maintainer="Snowplow Analytics Ltd. <support@snowplowanalytics.com>"
# The version of stream enrich to download.
ENV ENRICH_VERSION="0.19.4"
But I think this whole setup is outdated…
Can you point me to an updated installation guide for Stream Enrich on Docker ?
Thank you!
Hi @eko,
I’d recommend you to use our pre-built Docker images. You can find them in docs. You still can built your own Docker image, but that would require assembling enrich asset from sources.
Also, to add to what Josh said - I’d recommend to use Enrich 1.4.2, not 1.3.2.
Hi @anton
As I was going to upgrade Stream Enrich to 1.4.2, I see that back in the days we used our own custom fork of stream-enrich in order to support text/plain requests.
(snowplow/EKO_README.md at feature/eko · Melatonin64/snowplow · GitHub)
Is text/plain requests supported by newer versions? or should I create a custom fork ?
Thanks
No afraid it’s not supported now. If you provide justification in the PR we can consider adding this in future releases.
Hey @eko
Older versions of Chromium had an issue with their Beacon implementation that meant they sent their payloads as text/plain and you couldn’t attach custom content-type headers. Luckily that is no longer the case and Chrome, Edge, etc will perform a proper CORS pre-flight now and allow content-type headers to be amended on the request (See Beacon implementation fails in Chrome and Safari 12+ due to browser bugs · Issue #751 · snowplow/snowplow-javascript-tracker · GitHub).
The official JavaScript Tracker should fallback to sending POST on those older browsers (as Chrome throws exceptions when trying to use custom headers on those older versions that we catch). We also made some additional improvements to using Beacon in 2.13.0 of the JavaScript Tracker, which benefit older versions of Safari, that might be worth upgrading to if you haven’t already.
If this is the only reason for the fork, then you should be ok to upgrade, I don’t think you should see any consequences assuming you are using JS Tracker 2.13.0+ (I’d use 2.16.3 or 2.17.0 personally as they have smaller filesizes), Collector 2.1.0+ and Enrich 1.4.2+.
Thanks ! will look into this