Problem with selfdescribingevents and custom contexts

For engineers
1

I have made few jsonschemas and verified them using igluctl. I then uploaded them to iglu server using http api calls. Now when sending events and passing custom contexts or selfdescribingevents, no data is getting stored in postgresdb. I can see enriched objects in s3 bucket. No table is also created for those schemas. Should I create corresponding tables in atomic for those schemas manually?

Also I made one jsonschema earlier for selfdescribingevents which works perfectly fine.
I installed using Quick Start Guide for AWS and PostgresDB.

2

Hi @zameer can you please check the logs for the Postgres Loader to see if there are any errors in them that could help?

3

Hi @josh , I couldn’t find any errors in logs for both postgres-loader-enriched and postgres-loader-bad. I did find this unusual log:
[pool-14-thread-1] e[34mINFO e[0;39m e[36ms.a.k.c.PeriodicShardSyncManagere[0;39m - Skipping shard sync for sp-enriched-stream due to the reason - Hash Ranges are complete for sp-enriched-stream

Also in logs of iglu-server there is only log multiple times:

HTTP body not read to completion. Dropping connection.
[pool-1-thread-110] INFO org.http4s.server.blaze.Http1ServerStage$$anon$1 - HTTP body not read to completion. Dropping connection.
4

Postgres loader should take care of creating the tables for contexts and self-describing events.

Can you confirm that you’re not using the same value here and here please ?

Do you mean that you see it in Postgres?

5

Hi @BenB,

  1. OK
  2. I followed quick start guide (aws) for installation, these things were done automatically. I don’t know how I can access those config files.
  3. Yeah
6

You should be able to find them in an S3 bucket, as hocon files.

I wonder if the issue could be that you added new schemas to your registry, but enrich didn’t know about them and they weren’t in its cache, so they ended up as bad rows.

Could you please:

  1. Check in the bad rows table to see if you can see the events (probably as schema violations)

  2. Restart enrich and see if the problem persists

7

Hi,

  1. For some reason my bad rows don’t get populated, I mentioned it here and it seems to be an issue with previous version of snowplow. But since the data is present in enriched s3 bucket for that schema, I don’t think it is the case of schema violations.
  2. I restarted enrich, didn’t solve the issue, then re-run the terraform modules checking all the parameters, still no luck.
    I tried uploading other simple schemas to check if the problem is with the schema uploaded earlier, but simple schema is also not reflected in db. So I think the issue is not with schema.
    What other things I can test regarding this?

I found some warnings in collector server logs, these have multiple instances in logs:

[scala-stream-collector-akka.actor.default-dispatcher-7] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'host' header: Invalid input '{', expected 'EOI', ':', UPPER_ALPHA, lower-reg-name-char or pct-encoded (line 1, column 2): ${ip}

[scala-stream-collector-akka.actor.default-dispatcher-7] WARN akka.actor.ActorSystemImpl - Illegal request, responding with status '400 Bad Request': Request is missing required `Host` header


[scala-stream-collector-akka.actor.default-dispatcher-12] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'host' header: Invalid input '{', expected 'EOI', ':', UPPER_ALPHA, lower-reg-name-char or pct-encoded (line 1, column 2): ${ip}

[scala-stream-collector-akka.actor.default-dispatcher-12] WARN akka.actor.ActorSystemImpl - Illegal request, responding with status '400 Bad Request': Request is missing required `Host` header


[scala-stream-collector-akka.actor.default-dispatcher-11] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'user-agent' header: Invalid input ',', expected OWS, 'EOI', tchar, product-or-comment, comment or ws (line 1, column 5): xfa1,nvdorz


[scala-stream-collector-akka.actor.default-dispatcher-12] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'user-agent' header: Invalid input ',', expected OWS, 'EOI', tchar, product-or-comment, comment or ws (line 1, column 8): Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com
8

Hi @zameer ,

Alright that’s great information, so the problem comes from Postgres loader good for sure.

  1. In Postgres, can you see new enriched events getting inserted (in atomic)?
  2. If you can see them, can you see the ones that have the new schema?

Were you able to get this information ? If the same value is used, only one loader (so S3 loader it seems) will be able to consume enriched events.

9

Hi @BenB , my issue is resolved.
The problem was with schema itself, I made a property with name ‘schema_version’ which I think is not allowed, I removed that and now it works.
Also I was curious to know if I update a schema(not changing the version) will the db table be automatically updated by postgres loader or should I manually alter it?
Thank you!

1 Like
10

Hi @zameer ,

Great that you could solve your issue !

Also I was curious to know if I update a schema(not changing the version) will the db table be automatically updated by postgres loader or should I manually alter it?

Schemas should not be updated, instead a new version of the schema with the update should be created, respecting SchemaVer.

So you need to manually alter it.

1 Like