Enrich PubSub running flaky with low throughput

For engineers Enrichment
1

Hi,
our enrich pubsub deployment (v3.2.3) on Kubernetes is not running properly: When a new pod starts, the CPU goes to its limit and drops after a few minutes to a low level. After a few minutes, this process repeats and CPU starts to ramp up and goes down again:

image
image1136×318 10.2 KB

Looking at the input / output message rates of PubSub, there are sharp spikes when reading events from the collector-good pubsub. On the other hand, the message rate written to enriched-good pubsub is constant, even if the CPU is low at that moment. Overall the message rate processed in this configuration is quite low, which is probably related to the low CPU usage on average.
collector-good PubSub / read:
image

enriched-good PubSub / write:
image

We run the enrich pubsub with default configs of:

{
  "input": {
    "type": "PubSub"
    "subscription": "projects/xxx/subscriptions/sp-coll-good-sub"
  }

  "output": {
    "good": {
      "type": "PubSub"
      "topic": "projects/xxx/topics/sp-enriched-good"
    }

    "bad": {
      "type": "PubSub"
      "topic": "projects/xxx/topics/sp-enriched-bad"
    }
  }

  "telemetry": {
    "disable": true
  }

  "monitoring": {
    "metrics": {
      "stdout": {
        "period": "1 minute"
        "prefix": "snowplow.enrich."
      }
    }
  }
}

Any idea how to fix this issue? We think it might be related to the config we use, however since we use the default config we are not sure what parameters to tweak.

Thanks in advance!

1 Like
2

Hi @tziegler ,

Thank you for the details.

Based on your description it would seem that enrich-pubsub consumes a new batch, enriches it (CPU high) and then sends it (CPU low), before consuming the next batch and repeating. But the app is designed to keep consuming and enriching while enriched events are being sent, so this should not be the case.

First of all, please note that we have released 3.3.0, I encourage you to upgrade. This new config parameter might solve your issue already. If not, I would try to decrease this parameter to something smaller (e.g. 1000).

Please keep us updated.