Enrich-kinesis-3.5.0 does nothing

For engineers Enrichment
1

Trying to put together a PoC environment in AWS with Kinesis. But looks like enrich-kinesis does nothing. All components are in docker in an EC2 instance. ERnrich log:

[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.io.FileSystem - Files found in /snowplow/enrichments: /snowplow/enrichments/javascript_script_config.json, /snowplow/enrichments/pii_enrichment_config.json.tmpl, /snowplow/enrichments/currency_conversion_config.json.tmpl, /snowplow/enrichments/iab_spiders_and_robots_enrichment.json, /snowplow/enrichments/event_fingerprint_config.json, /snowplow/enrichments/cookie_extractor_config.json, /snowplow/enrichments/referer_parser.json, /snowplow/enrichments/api_request_enrichment_config.json.tmpl, /snowplow/enrichments/sql_query_enrichment_config.json.tmpl, /snowplow/enrichments/campaign_attribution.json, /snowplow/enrichments/yauaa_enrichment_config.json, /snowplow/enrichments/user_agent_utils_config.json, /snowplow/enrichments/disabled, /snowplow/enrichments/weather_enrichment_config.json.tmpl, /snowplow/enrichments/http_header_extractor_config.json
[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.config.ParsedConfigs - Parsed config file: ConfigFile(Kinesis(ati-test1-enrich,ati-test1-collector-raw,Some(us-east-1),TrimHorizon,Polling(10000),3,BackoffPolicy(100 milliseconds,10 seconds,Some(10)),None,None,None),Outputs(Kinesis(ati-test1-enrich-good,Some(us-east-1),None,BackoffPolicy(100 milliseconds,10 seconds,Some(10)),BackoffPolicy(100 milliseconds,1 second,None),500,5242880,None),None,Kinesis(ati-test1-enrich-bad,Some(us-east-1),None,BackoffPolicy(100 milliseconds,10 seconds,Some(10)),BackoffPolicy(100 milliseconds,1 second,None),500,5242880,None)),Concurrency(256,1),None,RemoteAdapterConfigs(10 seconds,45 seconds,10,List()),Monitoring(None,MetricsReporters(Some(StatsD(host.docker.internal,8125,Map(app -> enrich),2 seconds,Some(snowplow.enrich.))),None,true)),Telemetry(false,15 minutes,POST,collector-g.snowplowanalytics.com,443,true,None,None,None,None,None),FeatureFlags(false,false),None)
[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.config.ParsedConfigs - Parsed Iglu Client with following registries: Iglu Central, Uglu local
[pool-1-thread-2] WARN com.snowplowanalytics.snowplow.enrich.common.enrichments.registry.UserAgentUtilsEnrichmentConfig$ - user_agent_utils enrichment is deprecated. Please visit here for more information: https://github.com/snowplow/snowplow/wiki/user-agent-utils-enrichment
[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.config.ParsedConfigs - Parsed following enrichments: event_fingerprint_config, referer_parser, campaign_attribution, user_agent_utils_config
[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Run - Initialising resources for snowplow-enrich-kinesis 3.5.0
[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Assets - Initializing (downloading) enrichments assets
[pool-1-thread-2] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Assets - Downloading http://snowplow-hosted-assets-us-east-1.s3.amazonaws.com/third-party/referer-parser/referers-latest.json
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Run - Running enrichment stream
[pool-1-thread-1] INFO software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
[pool-1-thread-1] INFO software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Initialization attempt 1
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Initializing LeaseCoordinator
[cats-effect-blocker-1] INFO software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - Created new lease table for coordinator with initial read capacity of 10 and write capacity of 10.
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Lease table is still empty. Checking again in 3000 ms
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Lease table is still empty. Checking again in 3000 ms
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Lease table is still empty. Checking again in 3000 ms
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Lease table is still empty. Checking again in 3000 ms
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Worker atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is initiating the lease sync.
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Syncing Kinesis shard info for ati-test1-collector-raw
[cats-effect-blocker-1] INFO software.amazon.kinesis.leases.KinesisShardDetector - Stream ati-test1-collector-raw: listing shards with list shards request ListShardsRequest(StreamName=ati-test1-collector-raw, ShardFilter=ShardFilter(Type=AT_TRIM_HORIZON))
[cats-effect-blocker-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Starting lease cleanup thread.
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Starting LeaseCoordinator
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Scheduling periodicShardSync
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Initialization complete. Starting worker loop.
[LeaseCoordinator-0000] INFO software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker - Taking leases that have been expired for a long time: [Lease(leaseKey=shardId-000000000000, leaseOwner=null, leaseCounter=0, concurrencyToken=null, lastCounterIncrementNanos=0, checkpoint={SequenceNumber: TRIM_HORIZON,SubsequenceNumber: 0}, pendingCheckpoint=null, pendingCheckpointState=null, ownerSwitchesSinceCheckpoint=0, parentShardIds=[], childShardIds=[], hashKeyRangeForLease=HashKeyRangeForLease(startingHashKey=0, endingHashKey=340282366920938463463374607431768211455))]
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders:
[LeaseCoordinator-0000] INFO software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseTaker - Worker atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd successfully took 1 leases: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=0, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Created new shardConsumer for : ShardInfo(streamIdentifierSerOpt=Optional.empty, shardId=shardId-000000000000, concurrencyToken=f614ea37-dbc3-496e-a3ef-e653373d0450, parentShardIds=[], checkpoint={SequenceNumber: TRIM_HORIZON,SubsequenceNumber: 0})
[ShardRecordProcessor-0000] INFO software.amazon.kinesis.lifecycle.BlockOnParentShardTask - No need to block on parents [] of shard shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[ShardRecordProcessor-0000] INFO software.amazon.kinesis.retrieval.polling.PrefetchRecordsPublisher - ati-test1-collector-raw:shardId-000000000000 : Starting Prefetching thread and initializing publisher session.
[ShardRecordProcessor-0000] INFO software.amazon.kinesis.retrieval.polling.KinesisDataFetcher - Initializing shard ati-test1-collector-raw:shardId-000000000000 with TRIM_HORIZON
[ShardRecordProcessor-0001] INFO software.amazon.kinesis.leases.KinesisShardDetector - Stream ati-test1-collector-raw: listing shards with list shards request ListShardsRequest(StreamName=ati-test1-collector-raw)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - WorkerId atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is leader, running the periodic shard sync task
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Skipping shard sync for ati-test1-collector-raw due to the reason - Hash Ranges are complete for ati-test1-collector-raw
[pool-15-thread-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-15-thread-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - WorkerId atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is leader, running the periodic shard sync task
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Skipping shard sync for ati-test1-collector-raw due to the reason - Hash Ranges are complete for ati-test1-collector-raw
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - WorkerId atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is leader, running the periodic shard sync task
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Skipping shard sync for ati-test1-collector-raw due to the reason - Hash Ranges are complete for ati-test1-collector-raw
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-15-thread-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-15-thread-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - WorkerId atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is leader, running the periodic shard sync task
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Skipping shard sync for ati-test1-collector-raw due to the reason - Hash Ranges are complete for ati-test1-collector-raw
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - WorkerId atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is leader, running the periodic shard sync task
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Skipping shard sync for ati-test1-collector-raw due to the reason - Hash Ranges are complete for ati-test1-collector-raw
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - WorkerId atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd is leader, running the periodic shard sync task
[pool-16-thread-1] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Skipping shard sync for ati-test1-collector-raw due to the reason - Hash Ranges are complete for ati-test1-collector-raw
[pool-15-thread-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-15-thread-1] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Elected leaders: atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-17-thread-1] INFO software.amazon.kinesis.leases.LeaseCleanupManager - Number of pending leases to clean before the scan : 0
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647)
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Current stream shard assignments: shardId-000000000000
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
^C[pool-1-thread-1] WARN com.snowplowanalytics.snowplow.enrich.common.fs2.Enrich - Terminating enrich stream. Waiting 5 minutes for it to complete
[cats-effect-blocker-2] INFO software.amazon.kinesis.coordinator.Scheduler - Worker shutdown requested.
[cats-effect-blocker-2] INFO software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - Worker atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd has successfully stopped lease-tracking threads
[cats-effect-blocker-2] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Shutting down leader decider on worker atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[cats-effect-blocker-2] INFO software.amazon.kinesis.coordinator.DeterministicShuffleShardSyncLeaderDecider - Successfully stopped leader election on the worker
[cats-effect-blocker-2] INFO software.amazon.kinesis.coordinator.PeriodicShardSyncManager - Shutting down periodic shard sync task scheduler on worker atiutil.ati.sspinc.io:ad871f06-1bd7-49cf-bb43-385fb134dffd
[prefetch-cache-shardId-000000000000-0000] INFO software.amazon.kinesis.retrieval.polling.PrefetchRecordsPublisher - ati-test1-collector-raw:shardId-000000000000 :  Thread was interrupted, indicating shutdown was called on the cache.
[cats-effect-blocker-1] INFO software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=1, coreThreads=0, leasesOwned=0, largestPoolSize=2, maximumPoolSize=2147483647)
[pool-1-thread-2] INFO org.http4s.client.PoolManager - Shutting down connection pool: curAllocated=2 idleQueues.size=2 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[cats-effect-blocker-1] ERROR software.amazon.kinesis.coordinator.Scheduler - Worker.run caught exception, sleeping for 1000 milli seconds!
java.lang.InterruptedException: sleep interrupted
	at java.base/java.lang.Thread.sleep(Native Method)
	at software.amazon.kinesis.coordinator.Scheduler.runProcessLoop(Scheduler.java:433)
	at software.amazon.kinesis.coordinator.Scheduler.run(Scheduler.java:325)
	at fs2.aws.kinesis.Kinesis$GenericKinesis.$anonfun$readChunksFromKinesisStream$4(Kinesis.scala:88)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:108)
	at cats.effect.internals.IORunLoop$.restartCancelable(IORunLoop.scala:51)
	at cats.effect.internals.IOBracket$BracketStart.run(IOBracket.scala:100)
	at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67)
	at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35)
	at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90)
	at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90)
	at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
	at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42)
	at cats.effect.internals.IOBracket$BracketStart.apply(IOBracket.scala:80)
	at cats.effect.internals.IOBracket$BracketStart.apply(IOBracket.scala:58)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:192)
	at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:480)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:501)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:439)
	at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
	at cats.effect.internals.PoolUtils$$anon$2$$anon$3.run(PoolUtils.scala:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Iglu also doesn’t say anything:

[ioapp-compute-1] INFO com.snowplowanalytics.iglu.server.Server - Initializing server with following configuration: {"database":{"username":"root","host":"ati-ati-snowplow-test1.ctmikfo5p3gj.us-east-1.rds.amazonaws.com","enableStartupChecks":true,"dbname":"igludb","port":5432,"driver":"org.postgresql.Driver","maxPoolSize":null,"pool":{"connectionTimeout":"30 seconds","maxLifetime":"1800 seconds","minimumIdle":5,"maximumPoolSize":5,"connectionPool":{"size":4,"type":"fixed"},"transactionPool":"cached"},"password":"******"},"preTerminationUnhealthy":false,"webhooks":[],"debug":true,"superApiKey":"******","preTerminationPeriod":"1 second","swagger":{"baseUrl":"/custom/prefix"},"repoServer":{"interface":"0.0.0.0","port":8071,"idleTimeout":"60 seconds","maxConnections":1024,"threadPool":"cached"},"patchesAllowed":false}
[pool-2-thread-1] INFO com.zaxxer.hikari.HikariDataSource - iglu-hikaricp-pool - Starting...
[pool-2-thread-1] INFO com.zaxxer.hikari.HikariDataSource - iglu-hikaricp-pool - Start completed.
[ioapp-compute-0] INFO org.http4s.server.blaze.BlazeServerBuilder -
  _   _   _        _ _
 | |_| |_| |_ _ __| | | ___
 | ' \  _|  _| '_ \_  _(_-<
 |_||_\__|\__| .__/ |_|/__/
             |_|
[ioapp-compute-0] INFO org.http4s.server.blaze.BlazeServerBuilder - http4s v0.21.33 on blaze v0.14.18 started at http://[::]:8071/

enrich.hocon config: enrich.hocon - Pastebin.com

Docker images used:

snowplow/stream-enrich-kinesis            latest    b45dcb30a015   2 weeks ago     373MB
snowplow/snowplow-enrich-kinesis          3.6.0     464ade107e3f   2 weeks ago     365MB
snowplow/snowplow-enrich-kinesis          3.5.0     c7bce7833937   7 weeks ago     450MB
snowplow/snowplow-s3-loader               2.2.3     e88509bed5d8   2 months ago    303MB
snowplow/iglu-server                      latest    c84d157b72aa   2 months ago    328MB

IAM credentials are passed by environment variables when starting the container. IAM permissions are set up. Collector does write to the given stream, If I start an S3-loader it puts them into the given bucket so I guess collector is OK.
Any suggestions?
Thanks in advance,
-r

2

Hi @ati there are a lot of moving pieces to setting up the components from IAM Policies to network configuration that could all be at play here.

Are you deploying all of stream-enrich-kinesis:latest, snowplow-enrich-kinesis:3.6.0 and snowplow-enrich-kinesis:3.5.0 at the same time on the same box?

  • Its worth noting that when consuming from Kinesis if there is only 1 shard you can only have 1 consumer active so if you are running another enrichment process with the same appName then the lease on that shard will already be occupied.
  • Equally if your applications are overly memory constrained they very well might not behave as expected and will crash randomly - what memory is allocated to the process and how much is left on your server?

Have you tried comparing your setup to our quickstart-modules to see if you can spot any obvious differences in configuration that could be at play here? As a next step I recommend reviewing the configurations used in the OS module and the resource connections we make between apps and streams in quick-start to see where things have gone wrong for your deployment.

The current OS module is using v3.6.0: GitHub - snowplow-devops/terraform-aws-enrich-kinesis-ec2

3

Hi @josh
As I mentioned I’m trying to run them in docker, so some additional images are there. The main question is why enrich does not turn to iglu server when a message comes in via the raw stream. Memory is 8GB and I just can’t see JVMs OutOfMemoryError so I guess MEM is sufficient. At the moment there is only 1 consumer on the raw stream.
Any other hints?
Thanks in advance
-r

4

As I mentioned I’m trying to run them in docker, so some additional images are there

I understand that - all the quick-start AWS modules leverage an EC2 VM which then launches a docker container in the user-data chunk - this is why I am suggesting it as a reference for how to configure the application correctly.

The main question is why enrich does not turn to iglu server when a message comes in via the raw stream

In your first message you mentioned that it is doing nothing. Is it consuming messages but just sending them all to the bad-stream? If yes have you configured the Iglu Resolver config to use the Iglu Server you have deployed? Can you share that config?

Memory is 8GB and I just can’t see JVMs OutOfMemoryError so I guess MEM is sufficient. At the moment there is only 1 consumer on the raw stream.

8GB should be sufficient yes so likely not an issue.