StatsD not sending metrics in s3 loader 2.2.2

For engineers Troubleshooting
1

Hey all.
I have a problem with statsd metrics in s3-loader. According to the logs, everything works fine and the records are written in s3, but the metrics are not sent to prometheus, only restarting the s3-loader service helps. After a restart, the metrics are sent successfully again.
This problem is only with this service
We are using Version 2.2.2

The config looks like this:

{
  "region": "<%=ENV['S3_LOADER_REGION_KINESIS']%>",

  "purpose": "<%=ENV['S3_LOADER_PURPOSE_OPTION']%>",

  "input": {
    "appName": "<%=ENV['S3_LOADER_APPLICATION_NAME']%>",
    "streamName": "<%=ENV['S3_LOADER_STREAM_NAME_INPUT']%>",
    "position": "<%=ENV['S3_LOADER_STREAM_POSITION_OPTION']%>",
    "maxRecords": <%=ENV['S3_LOADER_STREAM_MAX_BATCH_SIZE']%>,
    "customEndpoint": "<%=ENV['S3_LOADER_CUSTOMENDPOINT']%>"
  },

  "output": {
    "s3": {
      "path": "<%=ENV['S3_LOADER_FULL_PATH_TO_OUTPUT_DATA']%>",
      partitionFormat: "<%=ENV['S3_LOADER_PARTITION_FORMAT']%>",
      "filenamePrefix": "<%=ENV['S3_LOADER_FILENAME_PREFIX']%>",
      "maxTimeout": <%=ENV['S3_LOADER_MAX_TIMEOUT']%>,
      "compression": "<%=ENV['S3_LOADER_OUTPUT_FORMAT']%>"
    },

    "bad": {
      "streamName": "<%=ENV['S3_LOADER_BAD_STREAM']%>",
      "customEndpoint": "<%=ENV['S3_LOADER_CUSTOMENDPOINT']%>"
    }
  },

  "buffer": {
    "byteLimit": <%=ENV['S3_LOADER_BUFFER_BYTE_LIMIT']%>,
    "recordLimit": <%=ENV['S3_LOADER_BUFFER_RECORD_LIMIT']%>,
    "timeLimit": <%=ENV['S3_LOADER_BUFFER_TIME_LIMIT']%>
  },

  # Optional
  "monitoring": {

    "metrics": {
      # StatsD-powered metrics; Optional
      "statsd": {
        "hostname": <%=ENV['S3_LOADER_STATSD_HOST']%>,
        "port": <%=ENV['S3_LOADER_STATSD_PORT']%>,
        "tags": {
          # It can resolve environment variables
          "input": "<%=ENV['S3_LOADER_STREAM_NAME_INPUT']%>",
          "output": "<%=ENV['S3_LOADER_FULL_PATH_TO_OUTPUT_DATA']%>"
        },
        # Optional
        "prefix": "<%=ENV['S3_LOADER_STATSD_PREFIX']%>"
      }
    }
  }
}

This is the second time this problem has happened in 2 months.

2

Hi @Timofey_Sorokin just to help debug this - around the time the metrics stopped arriving did the Prometheus endpoints IP addresses change at all? Either an update or a server rotation?

My hunch on this is that due to StatsD using UDP there is no delivery guarantee and it might be that there is an issue with re-resolving IP Addresses of the downstream host when changes occur.

3

Hi @josh , ip Prometheus does not change, that’s for sure, we just restart the service and everything is restored right away. There are also two more hosts on which s3-loader is raised and such problems do not occur there

4

Hi @Timofey_Sorokin and I am assuming you are running some middle-ware for relaying statsd metrics to prometheus like this? Did the exporter have any updates / changes around when the S3 Loader stopped emitting?

5

Hi @josh , yes we use statsd_exporter. At the time of the error, it continued to receive metrics from other s3 loaders and other services