Thanks for your help. Can we have two snowplow-minis connecting to one elasticsearch storage? i.e., two kinesis-elasticsearch-sinks from two different snowplow-minis moving data to one remote elasticsearch cluster? Kindly help. We wanted the failover set up for collectors, hence thinking of this route.
Yes, you can. The question is why - your setup looks a bit weird. Why not to use load balanced collectors without additional load. Than setup rest of stack on additional machine. Interconnections with Kinesis/Kafka. IMHO collectors should be as HR/HA as possible. Any extra load would make me sleepless…
To expand on this I would just like to highlight the fact that Snowplow Mini is not a production ready setup. It is designed and meant only for low-scale testing and nothing more.
For production I would recommend having a read of this post which highlights the lambda architecture: How to setup a Lambda architecture for Snowplow
Thank you both!
Sorry, can you please explain how best can this be done? I am using elastic load balancer to convert https to http. I have nginx router watching two collectors (in two different snowplow-minis) for failovers. Should I pass data from the collector to a single enrichment stream?
If there is a better way this could be done, please let me know. If possible we do not want to use Kinesis Shards.
@josh, thank you for the info on Lambda architecture.