Postgres Loader Unable to Parse Schema List

For engineers Troubleshooting
1

Good evening, randomly our Postgres Loader began to throw the following error:

java.lang.RuntimeException: Couldn't initalise the state IgluError(SchemaKey(com.org,media_analytics_event,jsonschema,Full(1,0,2)),ResolutionError(TreeMap(Custom Iglu Server -> LookupHistory(Set(RepoFailure(DecodingFailure at : Cannot parse list of strings into SchemaList. Init schema iglu:com.org/media_analytics_event/jsonschema/1-0-1 is not 1-0-0)),1,2022-10-04T23:33:03.119Z), Iglu Central -> LookupHistory(Set(NotFound),1,2022-10-04T23:33:03.192Z), Iglu Central - Mirror 01 -> LookupHistory(Set(NotFound),1,2022-10-04T23:33:03.393Z), Iglu Client Embedded -> LookupHistory(Set(NotFound),1,2022-10-04T23:33:03.119Z))))

I have already verified that the loader does have access to our custom IGLU. I’ve also already verified that both schemas do in fact exist on our custom server.

This is a blocking error. It kills the java process and we’re unable to load any further data.

2

Hi @Katie62442 it looks like the loader is expecting 3 schemas not 2 - you are missing the 1-0-0 from the error message printed out.

For the loader to work it needs to have the full lineage of the schema evolution.

Hope this helps!

3

We do! Here is full output of the versions we have in iglu.

com.org	media_analytics_event	jsonschema	1	0	0	
com.org	media_analytics_event	jsonschema	1	0	1	
com.org	media_analytics_event	jsonschema	1	0	2	
com.org	media_analytics_event	jsonschema	2	0	0	
com.org	media_analytics_event	jsonschema	1	0	3
4

My next hunch would be that the evolution of some of these schemas is not following SchemaVer strictly which is why the table cannot be updated. Will ask internally if anyone has any other ideas!

5

Hi @Katie62442 , when the loader gets the list of schemas from Iglu, they are ordered by publish date (as in when they were inserted in the registry). Is it possible that 1-0-0 was deleted and then re-added? Or maybe you uploaded all the schemas in bulk (perhaps migrating from a different location) and they somehow got published out of order? This would explain why you can find 1-0-0 in the registry but the loaders claims the init schema is 1-0-1.

1 Like