Snowplow Enrichment Stop Process Event From Google Pub/Sub

For engineers Enrichment
1

Hi

We are using Terraform to provision our Snowplow.
We host our infra on Google with VM running the docker and pub/sub for message communication.

We sometimes encounter issue where Enrichment stop processing message from Pub/Sub without any error emit from the log

From checking the VM the CPU uterize around 75 % person before it stop processing the message.

image
image921×302 31.4 KB

From checking the VM log it only output normal log message
This is the last log emit before VM restart before this there is also no error mention

image
image1690×327 48.8 KB

No error mention only normal logging meesage

In our pub/sub subscription unpacked message keep pipeline up

image
image929×377 20.2 KB

We have to restart the enrichment VM for it to continue process the event again.

This is the config setup in the enrichment

{
  "input": {
    "type": "PubSub"
    "subscription": "Subcription_value"
  }
  "output": {
    "good": {
      "type": "PubSub"
      "topic": "Output_value"
      "attributes": [ "app_id", "event_name" ]
    }
    "bad": {
      "type": "PubSub"
      "topic": "Bad_topicvalue"
    }
  }

  "assetsUpdatePeriod": "7 days"

  "monitoring": {
    "metrics": {
      "stdout": {
        "period": "60 seconds"
      }
    }
  }

  "telemetry": {
    "disable": true
    "interval": 15 minutes
    "method": "POST"
    "collectorUri": ""
    "collectorPort": 443
    "secure": true
    "userProvidedId": ""
    "autoGeneratedId": ""
    "moduleName": "enrich-pubsub-ce"
    "moduleVersion": "0.3.1"
    "instanceId": ${INSTANCE_ID}
  }

  "featureFlags" : {
    "acceptInvalid": false
  }
}

Any one have any clue what could be the issue?
I don’t really know if any setting have been miss or where else can I investigate further on this.

Thanks