Enrich Elastic Search Sink / Sink Bad

For engineers Enrichment
1

Hi Team,

I am facing challenge while performing Index rollover and delete functionality on elastic search.
Current version of Elastic Search : 7.4

Issue : While processing the event by snowplow-kinesis-elasticsearch-sink and snowplow-kinesis-elasticsearch-sink-bad , i am using the ‘enabled’ the configuration as
‘good’ and ‘bad’ respectively.

snowplow-kinesis-elasticsearch-sink –

“good” for a stream of successfully enriched events

“bad” for a stream of bad events

“plain-json” for writing plain json

enabled = good

snowplow-kinesis-elasticsearch-sink-bad –

“good” for a stream of successfully enriched events

“bad” for a stream of bad events

“plain-json” for writing plain json

enabled = bad

Error receiving writing to elastic search good indices (good-000001) -
" failed with message Rejecting mapping update to [good-000001] as the final mapping would have more than 1 type: [_doc, good] "

This is because in newer version (7.x) of elastic search doesn’t support multiple document types. By default the document type is ‘_doc’.

[ Elasticsearch 7.x
Specifying types in requests is deprecated. For instance, indexing a document no longer requires a document type. The new index APIs are PUT {index}/_doc/{id} in case of explicit ids and POST {index}/_doc for auto-generated ids. ]

Can you please help me to overcome this issue. Any help will be appreciated.
Thanks in advance.

2

Let me know in case I need to delete the indices and create the new one with custom root document type ‘good’ or ’ bad. If so please help me out with the syntax.

3

Hi @sp_user,

Current Elasticsearch loader version supports Elasticsearch 6.8.

The API that it uses requires to specify the documentType type when writing.

When looking at Elasticsearch documentation it seems that it’s not possible to replace _doc with anything else, as _doc is part of Elasticsearch API.

Would it be a possibility for you to use Elasticsearch 6.8 instead of 7.x ?

4

Hi Ben,

Thanks for the reply.

It not possible for me to downgrade the Elasticsearch version to 6.8 from 7.4, as PROD systems are already live running. :frowning:

Is there any way to bypass this in order to perform index rollover and cleanup ?

5

Hi @sp_user how are you creating the index mappings currently? Is it the loader that is creating them?

6

I wonder if we could specify _doc as the documentType in ES loader configuration (here), instead of good or bad. Could you try @sp_user ?

7

Sure Ben will try and let you know … !