I’m encountering an issue where 2 out of 4 of my S3Loader pods are failing with the following error:
at java.base/java.lang.Thread.run(Thread.java:829)
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - No activities assigned
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Sleeping ...
[LeaseCoordinator-0000] ERROR com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator - Throwable encountered in lease taking thread
java.lang.IllegalStateException: Connection pool shut down
Does anyone have any idea why this is happening or how to resolve it?
What is the memory allocation of the pod and any JVM_OPTS you have assigned to these pods? Do you have any graphs showing memory utilization for these pods?
My initial hunch would be that with 64 shards that the pods might be getting overwhelmed and simply running out of memory.
I would suggest:
Maintaining a ratio of pod count to shard count of roughly 1:8 at a minimum (so 8 shards per pod)
Setting a specific heap size for the pod - for this application we tend to try and give 80-85% of the memory assigned directly to heap (which helps combat the OOM scenario documented above)
Would you be able to try that and see if starts working as expected?
One thing to add,
we have an auto scaling policy for s3 loader when ever memory or cpu usage is above 50% a new s3loader pod is added to the cluster. But unfortunately we never seen that autoscaling is triggered(out of 3 GB memory loader only 1 GB were used).
Hey @Sreenath I would bet that the pods were crashing before any auto-scaling could even register - with more base pods and a higher heap allocation it should become quite stable.
hi @josh
We have tried out your suggestions , s3loader with "-Xms1024m -Xmx2600m", total pod mem is 3 gb.
We have noticed that loader is not releasing memory,
After some research we found that Java’s garbage collector doesn’t always immediately release memory back to the operating system after it’s no longer needed. Instead, it often keeps the memory allocated within the heap for future use. This is generally efficient because reallocating memory from the OS can be expensive, but it can lead to high memory usage that doesn’t drop until the JVM exits.
If memory didnt go down scale-in of pod will not take place, do you have any solution to relase memory after usage
Hi @Sreenath no we don’t - for this app how it stands is that you should assign a fixed amount of heap to the application (85% or so of available is our recommendation).
This will normally let it work quite comfortably even under duress. You won’t be able to auto-scale on memory usage for this application though - scaling on CPU does work however.
@josh
We noticed that the S3 loader utilizes more memory compared to CPU. To enable autoscaling, we set triggers at 70% usage for both memory and CPU. The loader autoscaled when memory usage hit 70%, while CPU usage is only max 15% of a single core.
I think memory is utilized more compared to cpu, may be due the s3loader config issue.
Buffer i am using
“buffer”: {
# Maximum bytes to read before flushing
“byteLimit”: 67108864
# Maximum records to read before flushing
“recordLimit”: 10000
# Maximum time between flushes
“timeLimit”: 300000
}
The S3 Loader is much more memory bound than CPU bound given how it works is to essentially just buffer a stream into a single file and then write it to S3. You will see small spikes in CPU around the times when the buffer limit gets triggered.
You have also noticed that it holds onto memory during these flush periods which is why, again, when we run this application we do not scale on memory usage but rather our process is to:
Assign a large amount of heap and fully allocate it to the process
Scale on CPU > 60% (which captures and handles sudden extra load in the system)
Ensure that we set a healthy minimum number of pods for consumption to ensure no individual pod ends up with an OOM potential (generally based around a ratio of input shards)
This methodology works in production up to several billion events per day.
In terms of your config settings you want to set up the app to synchronize, generally, a few large files over many many small files as this is much more efficient for the application itself as well as any downstream consumers of these files.
In practice this means we use:
byteLimit: 67108864
recordLimit: 100000 (10x your setting)
time_limit: 180000 (3 minutes instead of 5 minutes)
This will mean that generally you are going to hit the byteLimit which will create 67mb files. Using 10k records you are likely going to be pushing to S3 very often and have smaller files closer to 20-30mb.
If the intention is to use the output with other downstream Snowplow systems like RDB Loader then Firehose will not work as it doesn’t produce the required format.
Of course you could look at our newer Streaming Loaders / Lake Loaders / Streaming Transformers and simply avoid using S3 Loader entirely!