How to make enricher work in Filesystem input mode?

For engineers
1

The docs mention you can run the enricher in FileSystem input mode like
and the sample hocon shows: enrich/config.file.extended.hocon at master · snowplow/enrich · GitHub

{
  "input": {
    "type": "FileSystem"

    # Directory containing collector payloads that are Thrift encoded
    "dir": "/var/collector-payloads"
  }

But what’s the expected format of input when running in this mode?
We tried to feed in the LZO compressed files that S3 loader produces when attached to the collector stream but it didn’t work.

2

Hi @anteup ,

The expected format is 1 Thrift payload (as emitted by the collector) per file, without any compression.

Please bear in mind that running Enrich with files is only for testing purposes.

3

Ok, we won’t be going down the filesystem path.

Related to what we wanted to do though - is there already a tool to read and replay S3 loader collector payloads into a stream?

4

Hi @anteup ,

Sorry for the silence.

We have a very experimental tool for this : GitHub - snowplow-incubator/blob2stream: Reads records from cloud blob storage and writes to cloud stream