The raw Snowplow events are Thrift encoded so this looks mostly correct to me. In order to get enriched events out you’ll need to make sure you also have the enrichment process setup as well to read from your raw Kafka topic and push into an enriched topic.
Have a enricher in place which is working fine initially, once i enabled the http_header_extractor_config.json enrichment then restart the enricher get the following log
Aug 20 06:16:49 ip-172-31-33-8 java[2542]: [main] WARN com.networknt.schema.JsonMetaSchema - Unknown keyword exclusiveMinimum - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Aug 20 06:16:49 ip-172-31-33-8 java[2542]: An error occured: NonEmptyList({“error”:“ValidationError”,“dataReports”:[{“message”:"[12].schema: is missing but it is required","path":"[12]",“keyword”:“required”,“targets”:[“schema”]},{“message”:"[12].data: is missing but it is required","path":"[12]",“keyword”:“required”,“targets”:[“data”]},{“message”:"[12].email: is not defined in the schema and the schema does not allow additional properties","path":"[12]",“keyword”:“additionalProperties”,“targets”:[“email”]},{“message”:"[12].unknown: is not defined in the schema and the schema does not allow additional properties","path":"[12]",“keyword”:“additionalProperties”,“targets”:[“unknown”]},{“message”:"[12].social: is not defined in the schema and the schema does not allow additional properties","path":"[12]",“keyword”:“additionalProperties”,“targets”:[“social”]},{“message”:"[12].paid: is not defined in the schema and the schema does not allow additional properties","path":"[12]",“keyword”:“additionalProperties”,“targets”:[“paid”]},{“message”:"[12].search: is not defined in the schema and the schema does not allow additional properties","path":"[12]",“keyword”:“additionalProperties”,“targets”:[“search”]}]})
After getting that error i have undo the changes but still getting the same error.
well when it comes to schema resolution, the default repo config is defined here - https://github.com/snowplow/enrich/blob/master/config/iglu_resolver.json - if your using standard events then they should pass schema validation with default iglu setup, but if your using custom schemas then you would need add your own iglu repo and change the config to point to it, for example:
