Google Kubernetes Engine scaling

[Ltsrc]

Hello,

Our Snowplow open source pipeline runs on Google Kubernetes Engine, using PubSub for messages, and I’m wondering how each individual service can be scaled up horizontally, if that’s possible ?
Can each service be scaled horizontally without concurrency issue ?

Thanks

[BenB]

Hi @Ltsrc ,

@Ltsrc

and I’m wondering how each individual service can be scaled up horizontally, if that’s possible ?

Kubernetes handles horizontal auto-scaling, and the strategy for scaling up up down depends on how you configure it.

@Ltsrc

Can each service be scaled horizontally without concurrency issue ?

Yes. PubSub consumers from a same subscription automatically share the consumption of the records, each record going to one consumer only.

[Ltsrc]

Thanks for the reply !

That’s clear for the PubSub consumers services, but how about the collector ? I’m certain there is no issue scaling this service either, but I’d rather make sure now to avoid further issues