I am attempting to get my Snowplow Enrich configuration set up.
Here are my configurations:
in {
raw = "good"
raw = ${?ENRICH_STREAMS_IN_RAW}
}
out {
enriched = "enriched"
enriched = ${?ENRICH_STREAMS_OUT_ENRICHED}
bad = "bad"
bad = ${?ENRICH_STREAMS_OUT_BAD}
pii = "outpii"
pii = ${?ENRICH_STREAMS_OUT_PII}
partitionKey = "user_ipaddress"
partitionKey = ${?ENRICH_STREAMS_OUT_PARTITION_KEY} }
sourceSink {
enabled = "kinesis"
enabled = ${?ENRICH_STREAMS_SOURCE_SINK_ENABLED}
region = "us-east-2"
region = ${?ENRICH_STREAMS_SOURCE_SINK_REGION}
AWS_SECRET_ACCESS_KEY
aws {
accessKey = "SECRET"
accessKey = ${?ENRICH_STREAMS_SOURCE_SINK_AWS_ACCESS_KEY}
secretKey = "SECRET"
secretKey = ${?ENRICH_STREAMS_SOURCE_SINK_AWS_SECRET_KEY}
}
# Maximum number of records to get from Kinesis per call to GetRecords
maxRecords = 10000
maxRecords = ${?ENRICH_MAX_RECORDS}
initialPosition = TRIM_HORIZON
initialPosition = ${?ENRICH_STREAMS_SOURCE_SINK_INITIAL_POSITION}
initialTimestamp = "2020-06-23T10:00:00Z"
initialTimestamp = ${?ENRICH_STREAMS_SOURCE_SINK_INITIAL_TIMESTAMP}
backoffPolicy {
minBackoff = 3000
minBackoff = ${?ENRICH_STREAMS_SOURCE_SINK_BACKOFF_POLICY_MIN_BACKOFF}
maxBackoff = 60000
maxBackoff = ${?ENRICH_STREAMS_SOURCE_SINK_BACKOFF_POLICY_MAX_BACKOFF}
}
}
buffer {
byteLimit = 5000
byteLimit = ${?ENRICH_STREAMS_BUFFER_BYTE_LIMIT}
recordLimit = 1000 # Not supported by Kafka; will be ignored
recordLimit = ${?ENRICH_STREAMS_BUFFER_RECORD_LIMIT}
timeLimit = 10000
timeLimit = ${?ENRICH_STREAMS_BUFFER_TIME_LIMIT}
}
appName = "snowplow-test"
appName = ${?ENRICH_STREAMS_APP_NAME} }
This is what is in resolver.json:
"schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1",
"data": {
"cacheSize": 500,
"repositories": [
{
"name": "Iglu Central",
"priority": 0,
"vendorPrefixes": [ "com.snowplowanalytics" ],
"connection": {
"http": {
"uri": "http://iglucentral.com"
}
}
},
{
"name": "Iglu Central - GCP Mirror",
"priority": 1,
"vendorPrefixes": [ "com.snowplowanalytics" ],
"connection": {
"http": {
"uri": "http://mirror01.iglucentral.com"
}
}
}
]
}
}
I get all the way through the process and when I run java -jar snowplow-stream-enrich-kinesis-1.1.3.jar --config=poc.enrich.config --resolver=file:resolver.json I receive the following output in the terminal:
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Using workerId: snowplow-collector.test.bi.int.liquidweb.com:313ee859-17f5-4d67-a589-f85b4431608d
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Running: snowplow-test.
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Processing raw input stream: good
[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for endpoint as https://kinesis.us-east-2.amazonaws.com, and region as us-east-2.
[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for endpoint as https://dynamodb.us-east-2.amazonaws.com, and region as us-east-2.
[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for region as us-east-2.
[main] INFO com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 1
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.fasterxml.jackson.databind.util.ClassUtil (file:/home/snowplow_poc/snowplow/snowplow-stream-enrich-kinesis-1.1.3.jar) to field java.lang.Throwable.cause
WARNING: Please consider reporting this to the maintainers of com.fasterxml.jackson.databind.util.ClassUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] ERROR com.amazonaws.services.kinesis.leases.impl.LeaseManager - Failed to get table status for snowplow-test
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 2
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] ERROR com.amazonaws.services.kinesis.leases.impl.LeaseManager - Failed to get table status for snowplow-test
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 3
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] ERROR com.amazonaws.services.kinesis.leases.impl.LeaseManager - Failed to get table status for snowplow-test
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 4
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] ERROR com.amazonaws.services.kinesis.leases.impl.LeaseManager - Failed to get table status for snowplow-test
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 5
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] ERROR com.amazonaws.services.kinesis.leases.impl.LeaseManager - Failed to get table status for snowplow-test
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Caught exception when initializing LeaseCoordinator
What might be the cause for the failed table status?