Two pubsub sinks from one enricher

For engineers Enrichment
1

Hi,

I have a snowplow pipeline setup on GCP. I was wondering if an enricher can split enriched messages between two different sinks. For example, sending events A to pubsub A and events B to pubsub B, essentially filtering and redirecting.

Is this possible?

Thank you.

2

Hi @siv

Thats not currently a feature of enrich. The enrich application is designed to write to a single enriched stream.

However, it can be configured to add pubsub message attributes using the output.good.attributes configuration option.

You can use these attributes to filter messages on your subscriptions.

See here for further docs: Filtrar mensajes  |  Cloud Pub/Sub  |  Google Cloud

2 Likes
3

Hey @PaulBoocock ,

Is it possible to load data into two different BigQuery Tables using a single BQLoader, with minor modification to the BQLoader config, such that we can duplicate only certain type of events into the second table?

4

Hi @siv, that’s not possible - but the same use case shouldn’t be too hard to achieve using SQL.

1 Like