Hi, in my data pipeline I successfully use SQS and redshift loader, which loads into Redshift with the COPY command. We are using RDB Loader version 4.2.1. Previously, we did not use reloading. After reviewing the documentation, I decided to add the retries parameter retries.* to the config as in the documentation
How it was tested: I sent previously successfully processed data from the shredding_complete.json file to SQS, but added an error to them to cause a COPY command to fail.
I was expecting the RDB loader to retry the download after the time I specified. But that doesn’t happen. Is the retries.* parameter not working as expected?
My config:
{
"region": "<%= ENV['SQS_REGION'] %>",
"messageQueue": "<%= ENV['SQS_NAME'] %>",
"storage" : {
"type": "redshift"
"host": "<%= ENV['REDSHIFT_HOST'] %>"
"database": "<%= ENV['REDSHIFT_DB_NAME'] %>"
"port": "<%= ENV['REDSHIFT_PORT'] %>"
"roleArn": "<%= ENV['REDSHIFT_AWS_ROLE'] %>"
"schema": "<%= ENV['REDSHIFT_SCHEMA_NAME'] %>"
"username": "<%= ENV['REDSHIFT_USERNAME'] %>"
"password": "<%= ENV['REDSHIFT_PASSWORD'] %>"
},
"schedules": {
# Periodic schedules to stop loading, e.g. for Redshift maintenance window
# Any amount of schedules is supported, but recommended to not overlap them
# The schedule works with machine's local timezone (and UTC is recommended)
"noOperation": [
{
# Human-readable name of the no-op window
"name": "Maintenance window",
# Cron expression with second granularity
"when": "<%= ENV['RDB_LOADER_MAINTENANCE_SCHEDULE'] %>"
# For how long the loader should be paused
"duration": "<%= ENV['RDB_LOADER_MAINTENANCE_DURATION'] %>"
}
]
}
# Observability and reporting options
"monitoring": {
"metrics": {
# Optional, send metrics to StatsD server
"statsd": {
"hostname": "<%= ENV['RDB_LOADER_STATSD_HOST'] %>",
"port": "<%= ENV['RDB_LOADER_STATSD_PORT'] %>",
# Any key-value pairs to be tagged on every StatsD metric
"tags": {
"app": "rdb-loader"
}
# Optional, override the default metric prefix
"prefix": "<%= ENV['RDB_LOADER_STATSD_PREFIX'] %>"
}
},
# Optional, configuration for periodic unloaded/corrupted folders checks
"folders": {
# Path where Loader could store auxiliary logs
# Loader should be able to write here, Redshift should be able to load from here
"staging": "<%= ENV['FOLDERS_MONITORING_AUXILIARY_DIR'] %>",
# How often to check
"period": "<%= ENV['FOLDERS_MONITORING_PERIOD'] %>"
# Path to transformer archive (must be same as Transformer's `output.path`)
"transformerOutput": "<%= ENV['FOLDERS_MONITORING_TRANSFORMER_OUTPUT'] %>"
# How many times the check can fail before generating an alarm instead of warning
"failBeforeAlarm": <%= ENV['FOLDERS_MONITORING_FAILS'] %>
}
},
# Immediate retries configuration
# Unlike retryQueue these retries happen immediately, without proceeding to another message
"retries": {
# Starting backoff period
"backoff": "5 minutes"
# A strategy to use when deciding on next backoff
"strategy": "CONSTANT"
}
}