Problem with selfdescribingevents and custom contexts

I have made few jsonschemas and verified them using igluctl. I then uploaded them to iglu server using http api calls. Now when sending events and passing custom contexts or selfdescribingevents, no data is getting stored in postgresdb. I can see enriched objects in s3 bucket. No table is also created for those schemas. Should I create corresponding tables in atomic for those schemas manually?

Also I made one jsonschema earlier for selfdescribingevents which works perfectly fine.
I installed using Quick Start Guide for AWS and PostgresDB.

Hi @zameer can you please check the logs for the Postgres Loader to see if there are any errors in them that could help?

Hi @josh , I couldn’t find any errors in logs for both postgres-loader-enriched and postgres-loader-bad. I did find this unusual log:
[pool-14-thread-1] e[34mINFO e[0;39m e[36ms.a.k.c.PeriodicShardSyncManagere[0;39m - Skipping shard sync for sp-enriched-stream due to the reason - Hash Ranges are complete for sp-enriched-stream

Also in logs of iglu-server there is only log multiple times:

HTTP body not read to completion. Dropping connection.
[pool-1-thread-110] INFO org.http4s.server.blaze.Http1ServerStage$$anon$1 - HTTP body not read to completion. Dropping connection.

Postgres loader should take care of creating the tables for contexts and self-describing events.

Can you confirm that you’re not using the same value here and here please ?

Do you mean that you see it in Postgres?

Hi @BenB,

  1. OK
  2. I followed quick start guide (aws) for installation, these things were done automatically. I don’t know how I can access those config files.
  3. Yeah

You should be able to find them in an S3 bucket, as hocon files.

I wonder if the issue could be that you added new schemas to your registry, but enrich didn’t know about them and they weren’t in its cache, so they ended up as bad rows.

Could you please:

  1. Check in the bad rows table to see if you can see the events (probably as schema violations)

  2. Restart enrich and see if the problem persists

Hi,

  1. For some reason my bad rows don’t get populated, I mentioned it here and it seems to be an issue with previous version of snowplow. But since the data is present in enriched s3 bucket for that schema, I don’t think it is the case of schema violations.
  2. I restarted enrich, didn’t solve the issue, then re-run the terraform modules checking all the parameters, still no luck.
    I tried uploading other simple schemas to check if the problem is with the schema uploaded earlier, but simple schema is also not reflected in db. So I think the issue is not with schema.
    What other things I can test regarding this?

I found some warnings in collector server logs, these have multiple instances in logs:

[scala-stream-collector-akka.actor.default-dispatcher-7] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'host' header: Invalid input '{', expected 'EOI', ':', UPPER_ALPHA, lower-reg-name-char or pct-encoded (line 1, column 2): ${ip}

[scala-stream-collector-akka.actor.default-dispatcher-7] WARN akka.actor.ActorSystemImpl - Illegal request, responding with status '400 Bad Request': Request is missing required `Host` header


[scala-stream-collector-akka.actor.default-dispatcher-12] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'host' header: Invalid input '{', expected 'EOI', ':', UPPER_ALPHA, lower-reg-name-char or pct-encoded (line 1, column 2): ${ip}

[scala-stream-collector-akka.actor.default-dispatcher-12] WARN akka.actor.ActorSystemImpl - Illegal request, responding with status '400 Bad Request': Request is missing required `Host` header


[scala-stream-collector-akka.actor.default-dispatcher-11] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'user-agent' header: Invalid input ',', expected OWS, 'EOI', tchar, product-or-comment, comment or ws (line 1, column 5): xfa1,nvdorz


[scala-stream-collector-akka.actor.default-dispatcher-12] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal 'user-agent' header: Invalid input ',', expected OWS, 'EOI', tchar, product-or-comment, comment or ws (line 1, column 8): Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com