Real-time visualization

Hi. We are exploring migrating from GA3 to our own Snowplow running in AWS. Our users want to have real-time stats: the most popular pages, how many visitors for each page, etc.

Has anyone implemented a visualization tool like opensearch dashboard, kibana, etc or implemented some other architecture to make this happen? If so how did you make it work?

Right now we’ve setup the AWS bootstrap which uses Kinesis. We have that loading into Redshift but need a different tool/arch for real-time.


Hi @yangabanga - loading Elasticsearch / Opensearch would likely be the best strategy here to deliver real-time dashboards. We also have an OS Terraform Module for this which you can use as a way to get started quickly.

The alternative if Elasticsearch / Kibana is not something you are interested in would be to setup the Postgres Loader which is also real-time / streaming. This one is not battle-tested for really high ingress rates however so how well this works might vary.

With Opensearch you get Kibana included so can easily spinup dashboards that way but with Postgres you are likely going to have an easier time plugging this in to your existing BI Tools and querying it in much the same syntax as Redshift.

Would either of these options work for what you are looking to implement?

Yes looks like opensearch/kibana could be the way forward. thanks!

@josh might be a dumb question but will this module work with aws opensearch and opensearch dashboard or is it elasticsearch only?

Internally we exclusively use the Elasticsearch Loader with AWS Opensearch clusters which then has Kibana out of the box as well. So yes it for sure works!

1 Like