Igluctl 0.4.1 - Linting throws error

For data modelers & consumers Iglu
1

Hey there,

I upgraded to Igluctl 0.4.1, while the version before was giving with --severityLevel 2 green light, I get now a warning for each property

2. Schema doesn't contain description property
3. Schema doesn't contain description property
4. Schema doesn't contain description property
5. Schema doesn't contain description property
6. Schema doesn't contain description property

While the schema contains a description. What am I missing here?

Best

1 Like
2

Hi @poberherr,

Have you checked Igluctl wiki page? --severityLevel option is deprecated and removed as of version 0.4.0. To skip description warnings;

$ /path/to/igluctl lint --skip-checks description /path/to/schema/registry/schemas

Can you check wiki page and try above then let us know if you still have issues?

Best,
Oguzhan

4

Hey there, yes I read it in the docu that it’s deprecated.
I also used the --skip-checks option, and it works with that.

My question is more regarding usability, I would want from the tool that I don’t have to chain extra parameters, or ideally : Understand what I have to fix to not get the error message. I didn’t find an example of how to fix this.

1 Like
5

This description linter will check that each field in your JSON schema has a description associated with it (rather than the description of the schema itself).

e.g., in the example below we have descriptions for both app_id and app_name.

{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Adjust install event",
  "self": {
          "vendor": "com.adjust",
          "name": "install",
          "format": "jsonschema",
          "version": "1-0-0"
  },
  "type": "object",
  "properties": {
    "app_id": {
          "description": "Store App ID or Google Play Store e.g 332193586; de.is24.android; C013FJP3WF",
          "type": "string",
          "maxLength": 1024
    },
    "app_name": {
          "description": "Name of the app e.g MyApp",
          "type": ["string", "null"],
          "maxLength": 1024
    }
  },
  "minProperties": 1,
  "required": ["app_id"],
  "additionalProperties": true
  }
1 Like
6

Ok, awesome!

One more follow up question: This would be just for the schema registry, so for people implementing it;
Is there also another way (feature?) where I could access this field description which I am not aware of ?

1 Like
7

Hi @poberherr,

Let me share the reasoning behind this.

Since igluctl has multiple checks, a.k.a linters, you’ll need one way or another to eliminate or include some of those linters. --severityLevel approach was about predefined and unmodifiable set of linters for each level and it is clear that each use case can require different combination of linters, hence the decision to move from --severityLevel to --skip-checks enabling one to discard unwanted specific checks. See the relevant section in our release post.

Although there is a dedicated table to linters and their explanations here, it could be more explicit so that one could understand which linter to skip after seeing linter messages. We’ll improve the docs on that.

Thanks for the valuable feedback.

Best,
Oguzhan

8

Yes - this is primarily for people using the schema registry in terms of implementing it but also works well as a reference for analyst.

At the moment there isn’t really another way to access this field description outside of the schema registry but I’d like to add in (hopefully in the near future) a feature to igluctl that takes the description from the schema and uses this when generating the Redshift DDL to add COMMENT ON COLUMN example IS 'this describes the column example'. This would help analysts/end users who may not consume or use JSON schemas directly to get a better understanding of the underlying tables.

1 Like
9

@oguzhanunlu thanks for the useful links, I checked on that.

I am heavily using plenty of self describing schemas, is there a automatic way of migrating those tables to the new compression type? I assume not, so basically this would be a manual job? (The regeneration of the schemas is not the problem, more the migration - but I guess this is clear.)

Having said that there are also more use-cases where I had the problem keeping the “should be” state ( aka the generated state in the schema registry ) aligned with the “is” state in the DB.
While a few of the cases probably can be covered with CI ( like the normal flow of deploying new schemas ) I am wondering if it is a more common problem or not.

@mike This totally makes sence and I agree, I also added a lot of comments before, but in my case I am using Redshift; In this case - for me - psql doesn’t allow to pull out comments from Redshift easily and you have to rely on queries like here: https://gist.github.com/alexanderlz/7302623 jesus!
But I am pretty sure this makes a lot more sence for plain PostgreSQL, or if you are using different tooling.