Unwanted Events in Bad Bucket

For engineers Enrichment
1

Hi Team,

  1. For real time scenario , I am seeing some of events continue flowing in bad bucket on daily basis. Can you please help me understand what are these events and why these events are getting triggered on regular basis. ?
    Errors

{ “level”: “error”, “message”: “Payload with vendor …; and version jkstatus; not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor .git and version config not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor .git and version index not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor .idea and version webServers.xml not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor .svn and version entries not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor .svn and version wc.db not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor %0D%0Ahrs:hrs and version %2e%2e not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor actuator and version env not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor actuator and version env.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor actuator and version heapdump not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor admin and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor adminer and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor build and version login not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor config and version database.yml not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor config and version databases.yml not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor CVS and version Root not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor data and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor hudson and version login not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor jenkins and version login not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor log and version access_log not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor log and version access.log not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor logs and version access_log not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor logs and version access.log not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version autoconfig not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version autoconfig.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version beans not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version beans.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version env not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version env.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version health not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version health.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version heapdump not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version mappings not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor manager and version mappings.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor mini-profiler-resources and version results not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor php and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor public and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor remote and version login not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor scripts and version elmah.axd not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor sym and version root not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor tools and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor users and version sign_in not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor vpn and version index_ghs.html not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor vpn and version tmindex.html not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor web and version adminer.php not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor wp-includes and version sftp-config.json not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor x%0ahrs:hrs%3Ax and version %2e%2e not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor x%0d%0ahrs:hrs%3Ax and version %2e%2e not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor x%0dhrs:hrs%3Ax and version %2e%2e not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor xxx and version x%23%0Dhrs:hrs not supported by this version of Scala Common Enrich” }
{ “level”: “error”, “message”: “Payload with vendor xxx and version x%3F%0Dhrs:hrs not supported by this version of Scala Common Enrich” }

  1. In case these events are dummy events .Please help me to understand how can we ignore these events from getting into bad bucket. As it is making extra effort to review bad events daily.
    Is there way to distinguish between functional bad events and non functional bad events in Kibana ?
2

@sp_user, those errors indicate that the HTTP requests were sent to the collector at the path /<vendor_name>/<version>. The Snowplow collector has many default endpoints and any endpoint the HTTP request was sent to that is not either default or not customary configured will be rejected at validation step with the error message “Payload with vendor_name and version not supported by this version of Scala Common Enrich”.

The default endpoints are related to the collector adaptor dedicated to the specific type of events. For example, POST events would have to be sent to /com.snowplowanalytics.snowplow/tp2 (vendor com.snowplowanalytics.snowplow, version tp2), redirects would be sent to /r/tp2, and so on.

You cannot stop that kind of bad data from reaching your bad bucket. Various bots searching for server vulnerability will be poking into different endpoints. However, this bad data could be easily ignored with your analytics tool based on the error description “Payload with vendor”. This is explained, for example, here:

1 Like
3

Thanks Ihor for the details.

One more concern , is there any possibilities by any chance events are getting lost post processing - neither coming to good or bad bucket.
We are facing the challenges that some the events are not available in kibana , so is there way to find any missing events using snowplow framework.

4

@sp_user, it is possible that events hitting your collector are not present in Kibana. This does not, however, mean the events got lost. It is likely the issue with Elasticsearch, which could disregard some events if the data contradicts the dynamic mapping Elasticsearch have built initially. This typically happens when there are conflicting data types. Say, you defined your JSON schema with some property as a numeric value but later on you patched the schema with a string instead. Elasticsearch will reject/drop such data as it does not match the initial mapping (string cannot be recorded as an integer).

Again, this would be Elasctisearch issue and the actual event would be processed further down your pipeline (assuming real-time pipeline and not Snowplow Mini). That is your data would still be available in Kinesis and S3 (if S3 Loader is used) while might be missing in Elasticsearch/Kibana.