[snowplow-micro (0.1.0)] Issue with date-time validation

For engineers Other libraries and apps
1

Hello!

I ran into a small issue today while validating some tracking implementation using Snowplow Micro.

The problem I faced is related to unstructured event validation used in snowplow-micro compared to the current R118 release.

When sending an event with a schema such as:

{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
        "vendor": "com.acme",
        "name": "test",
        "format": "jsonschema",
        "version": "1-0-0"
    },
    "type": "object",
    "properties": {
        "test_tstamp": {
            "type": "string",
            "format": "date-time"
        }
    },
    "required": [
        "test_tstamp"
    ],
    "additionalProperties": false
}

So I am expecting a timestamp serialized in ISO8601 format (for instance using Instant.now().toString() in Java (since JDK9 the precision can be higher than milliseconds depending on the system clock).

With R118, the data will be validated to a timestamp with any precision, thanks to com.networknt.schema.DateTimeValidator.
However on snowplow-micro 0.1.0, the data validation is using: com.github.fge.jsonschema.format.common.DateTimeAttribute.

Due to the old version of the json-schema-validator (2.2.3), the timestamp field is considered invalid if its precision exceeds milliseconds.

See the json-schema-validator code.
This has been fixed in version 2.2.8 (see pull request).

I first wanted to rebuild snowplow-micro from source and upgrade these dependencies, however it seems like some of snowplow-micro build dependencies are not publicly available.

As an ugly test, I built a docker image based on snowplow-micro:0.1.0 and updated json-schema-validator to 2.2.13 and json-schema-core to 1.2.13 by replacing the existing files and the validation issue is then fixed by this dirty hack.

I was wondering if a new snowplow-micro release is planned soon?

Thanks in advance and keep up the good work at Snowplow!

PS: I also notice that snowplow-micro is released under license Apache 2.0 but sbt files build.sbt and project/* mentions the following

/**
  * PROPRIETARY AND CONFIDENTIAL
  *
  * Unauthorized copying of this file via any medium is strictly prohibited.
  *
  * Copyright (c) 2019-2019 Snowplow Analytics Ltd. All rights reserved.
  */
2

Hi @AcidFlow,

Thanks a lot for your feedback.

snowplow-micro is based on 2 components:

snowplow-micro was built on versions of these components that are prior to R118: it uses SSC 0.15.0 which itself uses SCE 0.31.0, whereas R118 uses version 1.0.0 of these components, which could explain the discrepancy that you observe.

Indeed we plan to extract the part of SSC that is used by snowplow-micro and put it in its own library but we haven’t done it yet, so for now you need to publish it locally manually, like here (with the right tag).

There is no release planned yet, but we’ll talk about it in our next planning meeting.
There are also new features that we would like to add.

Meanwhile feel free to open a PR to fix this issue. Please do not hesitate if you need any guidance.