Snowplow Elasticsearch Loader version 0.10.2

For engineers
1

Hi,
I have configured Snowplow Elasticsearch Loader version 0.10.2 on one ec2 and that ec2 has aws kinesis full access.
Same Ec2 is having ELK setup.

Using this “https://github.com/snowplow/snowplow-elasticsearch-loader” i have configured
Snowplow Elasticsearch Loader

But when i run the Jar file with my.conf facing some issue.

[root@elk opt]# java -jar snowplow-elasticsearch-loader-http-0.10.1.jar --config my.conf
configuration error: ConfigReaderFailures(KeyNotFound(nsq,Some(ConfigValueLocation(file:/opt/my.conf,1)),Set()),List(KeyNotFound(elasticsearch.aws,Some(ConfigValueLocation(file:/opt/my.conf,32)),Set()), KeyNotFound(elasticsearch.cluster,Some(ConfigValueLocation(file:/opt/my.conf,32)),Set())))

Following is My “my.conf”
[root@elk opt]# cat my.conf

source = kinesis

sink {
good = elasticsearch
bad = elasticsearch
}

enabled = good

aws {
accessKey = iam
secretKey = iam
}

kinesis {
initialPosition = LATEST
maxRecords = 100
region = us-west-2
appName = enrich-stream-good-qa
}

streams {
inStreamName = enrich-stream-good
outStreamName = kinesis-s3-fail-qa
buffer {
byteLimit = 3000000
recordLimit = 200
timeLimit = 180000
}
}

elasticsearch {
client {
endpoint = “127.0.0.1”
port = “9200”
maxTimeout = 10
ssl = false
}
}

2

Hi @AWSDeven,

You need nsq configuration with dummy values (although it does not make any sense to have them), like:

nsq {
  channelName = "channel"
  nsqdHost = "host"
  nsqdPort = 0
  nsqlookupdHost = "host"
  nsqlookupdPort = 0
}

I believe this is because of unified config validator.
Cheers,
GE

3

@grzegorzewald Thanks for the reply… but unfortunately it wont worked. Then i added following config and it is working fine… without your reference it was not possible. Thanks

nsq {
channelName = raw
host = localhost
port = 4150
lookupPort = 4160
}

4

RESOLVED…

Hi All,

Following is the working config. I am able to setup Elasticsearch Loader

1: Download the Elasticsearch Loader jarfile
wget http://dl.bintray.com/snowplow/snowplow-generic/snowplow_elasticsearch_loader_http_0.10.2.zip

2:Unzip downloaded zip file
unzip snowplow_elasticsearch_loader_http_0.10.2.zip

3: Then configure my.conf with your config.
vi my.conf

#------------------------------------------------------------#
source = kinesis

sink {
good = elasticsearch
bad = stderr
}

enabled = good

aws {
accessKey = iam
secretKey = iam
}

kinesis {
initialPosition = LATEST
maxRecords = 100
region = us-west-2
initialTimestamp = “2019-05-08T13:00:00Z”
appName = enrich-stream-good
}

streams {
inStreamName = enrich-stream-good
outStreamName = collector-kinesis-stream
buffer {
byteLimit = 3000000
recordLimit = 200
timeLimit = 180000
}
}

nsq {
channelName = raw
host = localhost
port = 4150
lookupPort = 4160
}

elasticsearch {
client {
endpoint = “127.0.0.1”
port = “9200”
maxTimeout = 10
ssl = false
}

aws {
signing = false
region = us-west-2
}

cluster {
name = elasticsearch
index = snowplow
clusterType = enriched
}
}
#------------------------------------------------------------#

4: Start Elasticsearch Loader jarfile with configuration file

java -jar snowplow-elasticsearch-loader-http-0.10.2.jar --config my.conf

Bingo…
Thank you all of you…

1 Like
5

Hi @AWSDeven & @grzegorzewald , Where did you find the sample config? Are you sure the config that you are using is valid for the ES loader version you are using. As I can see from the documentation the configuration seems to be bit different of what you are using here. It does not have kinesis and nsq section present in the example config here - https://raw.githubusercontent.com/snowplow/snowplow-elasticsearch-loader/master/examples/config.hocon.sample
I am using snowplow_elasticsearch_loader_http_0.10.2.jar, I received the same error on running with which you have started this thread. I added the kinesis & nsq section in my config and it worked. But whenever I try to push data to kinesis stream it crashes, refer the link here - Unable to connect ELK stack on aws from Elasticsearch Loader

6

Hi @Tejas_Behra,

All the configs required are in the file you provided…

Note that file form github has always higher precedence than docs (as file is always compatible with code version).

Cheers

7

@grzegorzewald did you observed the kinesis & nsq section are not present in the file. I explicitly added in my configuration file after observing the config read error

8

If anyone else still has issues making this work: Note that the latest version hosted at https://github.com/snowplow/snowplow-docker is 0.10.2, and the current version of the loader is now at 1.0.0, so the example config that snowplow-docker links to is newer and incompatible. The config file has been through several format changes, so if you’re attempting to use 0.10.1 or 0.10.2 (or older) of the Dockerized loader, you have to go to https://github.com/snowplow/snowplow-elasticsearch-loader/, search the tags for that version, and then browse the repo at that tag. You’ll see the version of the config file that works only for that version of the loader.