Snowplow Mini - Not able to see previous month events in Kibana

For engineers Snowplow Mini
1

Hi ,

I am using snowplow mini to view the events in Kibana and observed that only 12 days of data is available (sep 1 to 12) in Kibana. Looks like some auto archival at elastic search is happening here. could you please help me to understand the issue.

2

Hi @anoop_kuruvayal,

Snowplow Mini is a small-scale verison of Snowplow intended for demo and testing purposes. By default, the retention period of data is 7 days - it looks like in your case this isn’t very strict.

There is an open issue to make this customisable, but in general if you want persistent storage, or are using the data for anything important, you need a full pipeline. (Aside from persistent storage, Mini isn’t built to scale with production volumes).

3

Hi Colm,

Can you explain me from where elastic search is fetching its data and loading into kibana.In Snowplow mini ,we configure end points of elastic search cluster,can i know where we configure data retention period .

4

Like I said it’s not currently configurable. Snowplow Mini is an out-of-the-box small scale instance of the Snowplow pipeline.

If you need persistent storage you need to set up a Snowplow pipeline.

5

Thanks Colm for the reply.Is there any way to get back of flushed out data.If not there then how can we archive it so that futher data is not lost.As of now in kibana i’m able to see events from sept 1st to sept 13th.How can i predict that when this data alos will not be visible in kibana like how now August month data is not visible

7

There’s no way that I know of to retrieve deleted data. Snowplow Mini does not allow you to archive data or persist it in any way. You can quite easily predict it - after a week the data is liable to be lost.

I’m sorry, but I can’t help you any further than I already have - Snowplow Mini is a testing tool. It is used for testing and demonstration purposes. By design it doesn’t persist data. If you need to store, archive or persist data then Snowplow Mini is not the tool to do that. The Snowplow pipeline is, and it’s open source.

You’re also free to fork the Mini Source code and have a bash at making it do whatever you want, but I really don’t see why when there’s an amazing open source technology available to you that’s been engineered to do exactly what you want.

The question has been answered more than once, I’m closing this topic.

If you have genuine questions on setting up a full pipeline or on using the products as intended, please don’t hesitate to open a new topic using your original account.

1 Like