Quick Start Pipeline on GCP: PostgreSQL DB does not receive test data

For engineers GCP pipeline
1

Hello all,

I need help to build a snowplow pipeline in GCP.

Goal

My goal is to build a Snowplow pipeline using the Quickstart method in GCP.

Situation

So far I have used the Quick Start Installation Guide on GCP. The installation of the pipeline seems to work fine. When I send a test event according to the Send a test event guide, I get back HTTP/1.1 200 OK. However, when I try to query the generated test events according to the instructions Query your Postgres data , I get back 0 rows from the PostgreSQL DB.

I have so far tried, unfortunately unsuccessfully, to find log entries with the GCP Log Explorer and on the individual Compute Engine instances, to explain at which point in the pipeline the event data is lost.

What can I do or check to solve this problem?

Thank you in advance and best regards
Richard

2

Hi,

check you pub/sub subscriptions. Google Cloud Platform

Are there some unacked messages in sp-enrich-server subscription?

check logs (Google Cloud Platform)
any errors?

1 Like
3

Hi @riwi
Welcome to the Snowplow Community!.
Kind regards,
Eddie

1 Like
4

Hi Jiri,

thanks for helping.

Unacked Messages

I get one unacked message from a few hours ago:

image
image1198×290 19.9 KB

At about that time, I actually sent some test events. I also sent test events again a few minutes ago. However, these do not show up in the chart.

Logs

Yes, we have two error logs from compute engine resources. After my research, I am not sure if these errors are related to the described problem. These are the error logs:

  1. Resource: sp-collector-server
    Method: v1.compute.instances.setMetadata
    Message: Supplied fingerprint does not match current metadata fingerprint.
  2. Resource: sp-enrich-server
    Method: v1.compute.instances.setMetadata
    Message: Supplied fingerprint does not match current metadata fingerprint.
5

That error sounds rather GCP-y rather than any Snowplow component error.

This is speculation at this stage, but I’m curious if you have appropriate permissions in your GCP environment?

I assume the Quick Start terraform completed successfully?

6

Hello Paul,

please excuse the late reply. I tried for a few weeks to get the Quick Start variant to work, but gave up on it at some point. I chose “owner” as access permission on project level early on during the setup phase to be able to exclude permission problems.
In the meantime, I’m more successful with manually setting up the individual components via scripts using the gcloud CLI. Basically, I like the Terraform approach more, but problems during the installation are much more intransparent and difficult to fix than with the “classic” approach. On top of that, unfortunately, some errors in the documentation (mostly unadapted passages, probably taken from the AWS Quick Start Guide, see for example in the screenshot) leave room for interpretation:

image
image801×745 121 KB

Kind regards
Richard

1 Like
7

Hey @riwi

Thanks for pointing these out, I’ll look to get them updated as soon as possible and I’ll give the docs a run through myself to see if I can spot any other copy and paste issues, or opportunities to make them clearer.

8

Hey @PaulBoocock

I would really appreciate that. Please let me know when you have revised the docs, then I would happily give the Quick Start Guide another chance. In the prerequisites under “A Google cloud service account” it would also be helpful to mention which role or permissions the service account should have at least to complete the setup successfully.

1 Like