Kinesis + Elastic search

For engineers Storage targets
1

Hi all,

I am trying to use elastic search as a target. But I am using the Opensearch from AWS and I couldn’t install the elastic load balancer. Can I use the kinesis firehose to send to Elasticsearch? And just to understand, the Elastic Load balancer replaces firehose or is different approach?

Thanks a lot

2

You can use the Snowplow elasticsearch loader to load to elasticsearch.

Elastic Load Balancer isn’t anything to do with elasticsearch, it’s a separate AWS product (which distributes traffic across a network of resources).

3

Understood, but I can use this approach Kinesis Datastream > Firehose > ES instead using leastic load balancer to send data do ES?

4

instead using leastic load balancer to send data do ES?

Assuming this is a typo and you mean instead of the snowplow elasticsearch loader:

The enriched events in the kinesis stream are in TSV format, and the self-describing fields are JSON within that TSV.

So just using firehose to pipe them as-is into elasticsearch likely won’t work. You could try to create your own lambda function to run as part of the firehose job to transform the data to JSON using one of our analytics SDKs, and possibly get firehose to work (I don’t know much about firehose so I’m not sure how likely success is here).

However, all of that work is already done for you if you use our loader, so I would consider whether it’s worth the effort. :slight_smile:

2 Likes
5

Yeah pretty much this. Although you could pump Firehose => Lambda => Elasticsearch (you could not go Firehose => ES direct) you’d be reinventing the wheel.

2 Likes
6

Perfect, completly understood.

Thanks a lot

1 Like
7

Just coming back to this thread with the fact that we have now published a module for the AWS Elasticsearch Loader: https://registry.terraform.io/modules/snowplow-devops/elasticsearch-loader-kinesis-ec2/aws/latest

Big thanks to @nando_roz for help in beta testing this!

1 Like