RDB Loader Folder Monitoring

For engineers AWS real-time pipeline
1

Hi,

We are having issues setting up folder monitoring and the retry queue for our shredded files. Everything else seems to be running smoothly though.

Conf:
`

    folders: {
      "staging": "s3://snowplow-company/data/loader/",
      "period": "1 hour"
      "since": "14 days"
      "until": "7 days"
      "transformerOutput": "s3://snowplow-company/data/transformed/good/"
      "failBeforeAlarm": 3
    }

`

2022-10-11T14:50:40.052+02:00. INFO FolderMonitoring: Monitoring shredded folders
2022-10-11T14:50:41.328+02:00  INFO FolderMonitoring: Saved 0 folders from s3://snowplow-company/data/transformed/good/ in s3://snowplow-company/data/loader/shredded/2022-10-11-12-50-40/
2022-10-11T14:50:41.331+02:00  WARN FolderMonitoring: Folder monitoring has failed with unhandled exception for the 1 time

Am I missing something obvious?
Is folder monitoring supposed to work with the stream transformer?

Collector → Kinesis → Stream Enricher → Kinesis → Stream Transformer → S3+SQS → RDB Redshift Loader > Redshift

2

Hi @ddank,

Folder monitoring should be working with the stream transformer. Could you set failBeforeAlarm to 0, restart the application and post the error log here, please ?

3

Great thanks helped! Got an error now (permission problem) because I didn’t change a terraform variable.

2 Likes