Snowplow Elasticsearch Loader 2.0.0 released

Announcements New releases
1

We are pleased to announce Version 2.0.0 of the Elasticsearch Loader.

In this release, we are adding long-awaited Elasticsearch 7.x.x support to ES Loader. Also, we made some changes to the general config structure to make it more in line with other Snowplow products.

Elasticsearch 7.x.x support

Elasticsearch 7.x.x can be used with Elasticsearch Loader starting with this version. Most important thing to keep in mind is that mapping types are removed in Elasticsearch 7.x.x. Therefore, output.good.cluster.documentType field in the config shouldn’t be set when ES 7.x.x is used. We continue to support older versions as well. You should continue to use documentType with older versions.

Upgrading from 1.0.1

There are some changes on the config. Most notable changes are followings:

  • Removing AWS credentials section. We started to use DefaultAWSCredentialsProviderChain instead of custom credentials. If you are setting AWS credentials in the config, you should start to use one of the ways specified here.

  • Converting enabled field to purpose field. Also, possible values for it became BAD_ROWS, ENRICHED_EVENTS, JSON instead of good, bad, plain-json.

  • Adding chunk section under output.good.elasticsearch in the config. Chunk section specifies the maximum amount to send to Elasticsearch in one request. Chunking mechanism was in there previously too but it was set with buffer size values. We created separate section for it this time.

Other changes are mostly cosmetic to make the config structure similar to other apps. You can find example configs here.

4 Likes