Snowplow with AWS Elastic Search

For engineers AWS real-time pipeline
1

Hi All,

Below is my current setup, now i want to use AWS Elastic search for this.
Collector --> Stream-Enrich --> Kinesis-S3 --> EMR --> Redshift with ELK

I have use AWS Firehose and send all even from Kinesis stream and push to Elastic search.

But my events are not getting push to elastic search. I don’t want to loose or dont want to disturb my current data flow but additionally i want to get data from Kinesis stream an push into AWS Elastic search.

2

We have a similar use case, and we solve it by unloading data from Redshit to s3, then from s3 to Elastic Search using logstash

3

Hi Both,

You had better new consumer to enriched Kinesis stream, pushing data to Elasticsearch (Elasticsearch steram storage). This one supports AWS signed requests, so you can hide your hosted Elasticsearch a bit from the Internet.

Cheers