Hi there, we switched from managed to self-hosted a while back and was wondering if there’s any guides out there to help make our bad rows mapping a little prettier in Kibana.
When we were on managed, we were able to pull out the different context properties and see individual values. Like what the error was and which context was affected. Currently we can only see the the entire payload and have to sift through the JSON to see what the issue is.
@ks107 , bad data has different structure depending on the rejection reason and point of rejection. Therefore, you cannot have a universal approach to all the bad data. Different bad data failure type would have different structure. The JSON schemas that describe those structures could be found in Iglu Central.
The visualization of the data in Kibana could be done as per their documentation by selecting the fields of interest - something like shown in their doc here. Additionally, you could use search query to narrow down the search. For better visibility, take a look at the screenshot below (very basic).
Hi Ihor,
Thanks for the reply. I understand the different structures of the different rejection reasons.
However, even the data.failure.messages is not shown as it’s own data field. It is currently within data.payload_str, which is why I’ve asked for help here parsing out the data.payload_str into more usable and viewable fields.
We are using the same mapping provided in here but I do not see the same fields being parsed out like in your screenshot. My screenshot below of usable fields
Then if I uncheck “hide missing fields” I get a lot of these fields that don’t seem to map to anything.
@ks107 , if you do not see all the fields, try to refresh the fields list. Note the issue you are dealing with is related to the product not created or managed by Snowplow. Please, refer to Kibana documentation.
Hi @ihor Thanks, I did refresh and I see no change in my bad rows.
I understand that this is a kibana/elastic search issue, I was just looking for guidance as to how to replicate the bad rows columns that we saw in our managed instance of snowplow into our self hosted instance to make debugging easier on the team. I also wondered if anyone in the discourse community faced similar issues and how they went about solving them. If there was anything we were missing in setting up our instance of spmini which is causing our bad row data to show up like this and not like the screenshot you initially sent.
@ks107 , to my knowledge, when you set up Snowplow Mini everything is done for you automatically - no need for creating the mapping manually as per the guides below:
@ihor Thanks for the links. That was my understanding as well. Which is why I was confused as to why our bad rows columns look so different from the screenshots in the examples.
This is the example image from the usage guide, where you can see all the different properties of the payload. In my example above, I only have on long JSON string within data.payload_str rather than data.payload_refererurl or data.failure.messages as shown in the screenshot below.
If everything is supposed to be mapped automatically, I’m trying to figure out why our bad row data is so limited to exploring unlike the example screenshots.
@ks107 , I do not see anything wrong with the last screenshot. Note that all the bad data displayed there is of “adapter_failures” type. As such, the data representation (fields) corresponds to that type. Once you get bad data of a different type the corresponding (new) fields will be added dynamically. If they do not appear in the fields panel once you have new bad data type captured, you can refresh that list as I mentioned earlier.
@ihor, that last screenshot is from the link you sent above, so of course it has no issues. Please review my earlier posts for the screenshots that I see in my bad rows instance. I will post them here again:
you’ll notice that the available fields list does not match the fields from the sample screenshot.
