Bad gateway error upon updating collector from 2.3.1 to 2.4.5

Hi,

we updated the scala-stream-collector-kinesis from 2.3.1. to 2.4.5. and get a 502 Bad Gateway Error upon sending events to our collector endpoint.

We are running the collector on AWS fargate and receive the following logs on cloudwatch which do not show an error:

|—|—|
|1641478641677 [main] WARN com.amazonaws.http.AmazonHttpClient - SSL Certificate checking for endpoints has been explicitly disabled.
|1641478643191,[main] INFO com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - Creating thread pool of size 10||
|1641478643194,[main] WARN com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - No SQS buffer for surge protection set up (consider setting a SQS Buffer in config.hocon).||
|1641478643203,[main] WARN com.amazonaws.http.AmazonHttpClient - SSL Certificate checking for endpoints has been explicitly disabled.||
|1641478643397,[main] INFO com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - Creating thread pool of size 10||
|1641478643397,[main] WARN com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - No SQS buffer for surge protection set up (consider setting a SQS Buffer in config.hocon).||
|1641478643888,[scala-stream-collector-akka.actor.default-dispatcher-6] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started||
|1641478644113,[main] INFO com.snowplowanalytics.snowplow.collectors.scalastream.telemetry.TelemetryAkkaService - Telemetry enabled||
|1641478646994,[scala-stream-collector-akka.actor.default-dispatcher-9] INFO com.snowplowanalytics.snowplow.collectors.scalastream.KinesisCollector$ - REST interface bound to /0:0:0:0:0:0:0:0:8000||
|1641478647012,[scala-stream-collector-akka.actor.default-dispatcher-8] INFO com.snowplowanalytics.snowplow.collectors.scalastream.KinesisCollector$ - REST interface bound to /0:0:0:0:0:0:0:0:9543||
|||

Our collector config is as follows:

# Copyright (c) 2013-2020 Snowplow Analytics Ltd. All rights reserved.
#
# This program is licensed to you under the Apache License Version 2.0, and
# you may not use this file except in compliance with the Apache License
# Version 2.0.  You may obtain a copy of the Apache License Version 2.0 at
# http://www.apache.org/licenses/LICENSE-2.0.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the Apache License Version 2.0 is distributed on an "AS
# IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.  See the Apache License Version 2.0 for the specific language
# governing permissions and limitations there under.

# This file (config.hocon.sample) contains a template with
# configuration options for the Scala Stream Collector.
#
# To use, copy this to 'application.conf' and modify the configuration options.

# 'collector' contains configuration options for the main Scala collector.
collector {
  # The collector runs as a web service specified on the following interface and port.
  interface = ${COLLECTOR_INTERFACE}
  port = 8000 #${COLLECTOR_PORT}

  # optional SSL/TLS configuration
  ssl {
    enable = ${COLLECTOR_SSL_ENABLE}
    # whether to redirect HTTP to HTTPS
    redirect = ${COLLECTOR_SSL_REDIRECT}
    port = ${COLLECTOR_SSL_PORT}
  }
# The collector responds with a cookie to requests with a path that matches the 'vendor/version' protocol.
  # The expected values are:
  # - com.snowplowanalytics.snowplow/tp2 for Tracker Protocol 2
  # - r/tp2 for redirects
  # - com.snowplowanalytics.iglu/v1 for the Iglu Webhook
  # Any path that matches the 'vendor/version' protocol will result in a cookie response, for use by custom webhooks
  # downstream of the collector.
  # But you can also map any valid (i.e. two-segment) path to one of the three defaults.
  # Your custom path must be the key and the value must be one of the corresponding default paths. Both must be full
  # valid paths starting with a leading slash.
  # Pass in an empty map to avoid mapping.
  paths {
    # "/com.acme/track" = "/com.snowplowanalytics.snowplow/tp2"
    # "/com.acme/redirect" = "/r/tp2"
    # "/com.acme/iglu" = "/com.snowplowanalytics.iglu/v1"
  }

  # Configure the P3P policy header.
  p3p {
    policyRef = "/w3c/p3p.xml"
    CP = "NOI DSP COR NID PSA OUR IND COM NAV STA"
  }

  # Cross domain policy configuration.
  # If "enabled" is set to "false", the collector will respond with a 404 to the /crossdomain.xml
  # route.
  crossDomain {
    enabled = false
    # Domains that are granted access, *.acme.com will match http://acme.com and http://sub.acme.com
    #enabled = ${?COLLECTOR_CROSS_DOMAIN_ENABLED}
    domains = [ "*" ]
    #domains = [ ${?COLLECTOR_CROSS_DOMAIN_DOMAIN} ]
    # Whether to only grant access to HTTPS or both HTTPS and HTTP sources
    secure = true
    #secure = ${?COLLECTOR_CROSS_DOMAIN_SECURE}
  }

  # The collector returns a cookie to clients for user identification
  # with the following domain and expiration.
  cookie {
    enabled = false #true  # false Track no cookies at all for now?
    #enabled = ${?COLLECTOR_COOKIE_ENABLED}
    expiration = 0 #{{cookieExpiration}} # e.g. "365 days"
    #expiration = ${?COLLECTOR_COOKIE_EXPIRATION}
    # Network cookie name
    name = 'sp' #{{collectorCookieName}}
    #name = ${?COLLECTOR_COOKIE_NAME}
    # The domain is optional and will make the cookie accessible to other
    # applications on the domain. Comment out these lines to tie cookies to
    # the collector's full domain.
    # The domain is determined by matching the domains from the Origin header of the request
    # to the list below. The first match is used. If no matches are found, the fallback domain will be used,
    # if configured.
    # If you specify a main domain, all subdomains on it will be matched.
    # If you specify a subdomain, only that subdomain will be matched.
    # Examples:
    # domain.com will match domain.com, www.domain.com and secure.client.domain.com
    # client.domain.com will match secure.client.domain.com but not domain.com or www.domain.com
    domains = [
        "{{cookieDomain1}}" # e.g. "domain.com" -> any origin domain ending with this will be matched and domain.com will be returned
        "{{cookieDomain2}}" # e.g. "secure.anotherdomain.com" -> any origin domain ending with this will be matched and secure.anotherdomain.com will be returned
        # ... more domains
    ]
    #domains += ${?COLLECTOR_COOKIE_DOMAIN_1}
    #domains += ${?COLLECTOR_COOKIE_DOMAIN_2}
    # ... more domains
    # If specified, the fallback domain will be used if none of the Origin header hosts matches the list of
    # cookie domains configured above. (For example, if there is no Origin header.)
    fallbackDomain = "{{fallbackDomain}}"
    #fallbackDomain = ${?FALLBACK_DOMAIN}
    secure = false
    #secure = ${?COLLECTOR_COOKIE_SECURE}
    httpOnly = false
    #httpOnly = ${?COLLECTOR_COOKIE_HTTP_ONLY}
    # The sameSite is optional. You can choose to not specify the attribute, or you can use `Strict`,
    # `Lax` or `None` to limit the cookie sent context.
    #   Strict: the cookie will only be sent along with "same-site" requests.
    #   Lax: the cookie will be sent with same-site requests, and with cross-site top-level navigation.
    #   None: the cookie will be sent with same-site and cross-site requests.
    sameSite = 'None' #"{{cookieSameSite}}"
    #sameSite = ${?COLLECTOR_COOKIE_SAME_SITE}
  }

  # If you have a do not track cookie in place, the Scala Stream Collector can respect it by
  # completely bypassing the processing of an incoming request carrying this cookie, the collector
  # will simply reply by a 200 saying "do not track".
  # The cookie name and value must match the configuration below, where the names of the cookies must
  # match entirely and the value could be a regular expression.
  doNotTrackCookie {
    enabled = false #false
    name = '' #{{doNotTrackCookieName}}
    value = '' #{{doNotTrackCookieValue}}
  }

  # When enabled and the cookie specified above is missing, performs a redirect to itself to check
  # if third-party cookies are blocked using the specified name. If they are indeed blocked,
  # fallbackNetworkId is used instead of generating a new random one.
  cookieBounce {
    enabled = false
    # The name of the request parameter which will be used on redirects checking that third-party
    # cookies work.
    name = "n3pc"
    # Network user id to fallback to when third-party cookies are blocked.
    fallbackNetworkUserId = "00000000-0000-4000-A000-000000000000"
    # Optionally, specify the name of the header containing the originating protocol for use in the
    # bounce redirect location. Use this if behind a load balancer that performs SSL termination.
    # The value of this header must be http or https. Example, if behind an AWS Classic ELB.
    forwardedProtocolHeader = "X-Forwarded-Proto"
  }

  # When enabled, redirect prefix `r/` will be enabled and its query parameters resolved.
  # Otherwise the request prefixed with `r/` will be dropped with `404 Not Found`
  # Custom redirects configured in `paths` can still be used.
  enableDefaultRedirect = true

  # When enabled, the redirect url passed via the `u` query parameter is scanned for a placeholder
  # token. All instances of that token are replaced withe the network ID. If the placeholder isn't
  # specified, the default value is `${SP_NUID}`.
  redirectMacro {
    enabled = false
    # Optional custom placeholder token (defaults to the literal `${SP_NUID}`)
    placeholder = "[TOKEN]"
  }

  # Customize response handling for requests for the root path ("/").
  # Useful if you need to redirect to web content or privacy policies regarding the use of this collector.
  rootResponse {
    enabled = false
    statusCode = 302
    # Optional, defaults to empty map
    headers = {
      Location = "https://127.0.0.1/",
      X-Custom = "something"
    }
    # Optional, defaults to empty string
    body = "302, redirecting"
  }

  # Configuration related to CORS preflight requests
  cors {
    # The Access-Control-Max-Age response header indicates how long the results of a preflight
    # request can be cached. -1 seconds disables the cache. Chromium max is 10m, Firefox is 24h.
    accessControlMaxAge = 5 seconds
  }

  # Configuration of prometheus http metrics
  prometheusMetrics {
    # If metrics are enabled then all requests will be logged as prometheus metrics
    # and '/metrics' endpoint will return the report about the requests
    enabled = false
    # Custom buckets for http_request_duration_seconds_bucket duration metric
    #durationBucketsInSeconds = [0.1, 3, 10]
  }
  streams {
    # Events which have successfully been collected will be stored in the good stream/topic
    good = ${COLLECTOR_GOOD_STREAM}
    # Events that are too big (w.r.t Kinesis 1MB limit) will be stored in the bad stream/topic
    bad = ${COLLECTOR_BAD_STREAM}

    # Whether to use the incoming event's ip as the partition key for the good stream/topic
    # Note: Nsq does not make use of partition key.

    useIpAddressAsPartitionKey = false

    # Enable the chosen sink by uncommenting the appropriate configuration
    sink {
      # Choose between kinesis, google-pub-sub, kafka, nsq, or stdout.
      # To use stdout, comment or remove everything in the "collector.streams.sink" section except
      # "enabled" which should be set to "stdout".
      enabled = kinesis

      # Region where the streams are located
      region = ${COLLECTOR_STREAMS_SINK_REGION}

      ## Optional endpoint url configuration to override aws kinesis endpoints,
      ## this can be used to specify local endpoints when using localstack
      #customEndpoint =  localstack:4566

      # Thread pool size for Kinesis API requests
      threadPoolSize = ${COLLECTOR_STREAMS_SINK_THREAD_POOL_SIZE}

      # The following are used to authenticate for the Amazon Kinesis sink.
      # If both are set to 'default', the default provider chain is used
      # (see http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html)
      # If both are set to 'iam', use AWS IAM Roles to provision credentials.
      # If both are set to 'env', use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
      aws {
        accessKey = default #iam
        secretKey = default #iam
      }

      # Minimum and maximum backoff periods, in milliseconds
      backoffPolicy {
        minBackoff = ${COLLECTOR_STREAMS_SINK_MIN_BACKOFF}
        maxBackoff = ${COLLECTOR_STREAMS_SINK_MAX_BACKOFF}
      }

    }

    # Incoming events are stored in a buffer before being sent to Kinesis/Kafka.
    # Note: Buffering is not supported by NSQ.
    # The buffer is emptied whenever:
    # - the number of stored records reaches record-limit or
    # - the combined size of the stored records reaches byte-limit or
    # - the time in milliseconds since the buffer was last emptied reaches time-limit
    buffer {
      byteLimit = ${COLLECTOR_STREAMS_BUFFER_BYTE_LIMIT}
      recordLimit = ${COLLECTOR_STREAMS_BUFFER_RECORD_LIMIT}
      timeLimit = ${COLLECTOR_STREAMS_BUFFER_TIME_LIMIT}
    }
  }

}

# Akka has a variety of possible configuration options defined at
# http://doc.akka.io/docs/akka/current/scala/general/configuration.html
akka {
  loglevel = DEBUG # 'OFF' for no logging, 'DEBUG' for all logging.
  loggers = ["akka.event.slf4j.Slf4jLogger"]

  # akka-http is the server the Stream collector uses and has configurable options defined at
  # http://doc.akka.io/docs/akka-http/current/scala/http/configuration.html
  http.server {
    # To obtain the hostname in the collector, the 'remote-address' header
    # should be set. By default, this is disabled, and enabling it
    # adds the 'Remote-Address' header to every request automatically.
    remote-address-header = off

    raw-request-uri-header = on

    # Define the maximum request length (the default is 2048)
    parsing {
      max-uri-length = 32768
      uri-parsing-mode = relaxed
    }
  }

  # By default setting `collector.ssl` relies on JSSE (Java Secure Socket
  # Extension) to enable secure communication.
  # To override the default settings set the following section as per
  # https://lightbend.github.io/ssl-config/ExampleSSLConfig.html
  ssl-config {
    debug = {
      ssl = true
    }

    keyManager = {
      stores = [
        {type = "PKCS12", classpath = false, path = "/opt/snowplow/ssl/collector.p12", password = ${CERT_PW} }
      ]
    }

    loose {
      disableHostnameVerification = true
    }
  }
}

```

and the corresponding Dockerfile:

```
FROM snowplow/scala-stream-collector-kinesis:2.4.0

ARG AWS_DEFAULT_REGION
ENV COLLECTOR_STREAMS_SINK_REGION $AWS_DEFAULT_REGION
ENV COLLECTOR_INTERFACE 0.0.0.0
ENV COLLECTOR_PORT 8000
ENV COLLECTOR_SSL_ENABLE true
ENV COLLECTOR_SSL_REDIRECT true
ENV COLLECTOR_SSL_PORT 9543
ENV COLLECTOR_STREAMS_SINK_THREAD_POOL_SIZE 10
ENV COLLECTOR_STREAMS_SINK_MIN_BACKOFF 5000
ENV COLLECTOR_STREAMS_SINK_MAX_BACKOFF 60000
ENV COLLECTOR_STREAMS_BUFFER_BYTE_LIMIT 10000
ENV COLLECTOR_STREAMS_BUFFER_RECORD_LIMIT 5
ENV COLLECTOR_STREAMS_BUFFER_TIME_LIMIT 60
ENV CERT_PW $CERT_PW
ENV SSL_DIR /opt/snowplow/ssl

WORKDIR /app
COPY src/ /app/

# hadolint ignore=DL3002
USER root

RUN sh generate_ssl_cert.sh

CMD ["--config", "oneapp_collector.conf", \
 "-Dcom.amazonaws.sdk.disableCertChecking", "-Dcom.amazonaws.sdk.disableCbor"]


Did the config template change in the latest collector version?

There are no breaking changes in the config between 2.3 to 2.4, only some new configuration options.

However, that config you’ve shared is for Enrich rather than the Collector. Just checking you’re definitely using your collector hocon for your collector and not your enrich hocon?

Sorry, my bad. I copied the wrong file. I changed it above. This is our correct collector config:

# Copyright (c) 2013-2020 Snowplow Analytics Ltd. All rights reserved.
#
# This program is licensed to you under the Apache License Version 2.0, and
# you may not use this file except in compliance with the Apache License
# Version 2.0.  You may obtain a copy of the Apache License Version 2.0 at
# http://www.apache.org/licenses/LICENSE-2.0.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the Apache License Version 2.0 is distributed on an "AS
# IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.  See the Apache License Version 2.0 for the specific language
# governing permissions and limitations there under.

# This file (config.hocon.sample) contains a template with
# configuration options for the Scala Stream Collector.
#
# To use, copy this to 'application.conf' and modify the configuration options.

# 'collector' contains configuration options for the main Scala collector.
collector {
  # The collector runs as a web service specified on the following interface and port.
  interface = ${COLLECTOR_INTERFACE}
  port = 8000 #${COLLECTOR_PORT}

  # optional SSL/TLS configuration
  ssl {
    enable = ${COLLECTOR_SSL_ENABLE}
    # whether to redirect HTTP to HTTPS
    redirect = ${COLLECTOR_SSL_REDIRECT}
    port = ${COLLECTOR_SSL_PORT}
  }
# The collector responds with a cookie to requests with a path that matches the 'vendor/version' protocol.
  # The expected values are:
  # - com.snowplowanalytics.snowplow/tp2 for Tracker Protocol 2
  # - r/tp2 for redirects
  # - com.snowplowanalytics.iglu/v1 for the Iglu Webhook
  # Any path that matches the 'vendor/version' protocol will result in a cookie response, for use by custom webhooks
  # downstream of the collector.
  # But you can also map any valid (i.e. two-segment) path to one of the three defaults.
  # Your custom path must be the key and the value must be one of the corresponding default paths. Both must be full
  # valid paths starting with a leading slash.
  # Pass in an empty map to avoid mapping.
  paths {
    # "/com.acme/track" = "/com.snowplowanalytics.snowplow/tp2"
    # "/com.acme/redirect" = "/r/tp2"
    # "/com.acme/iglu" = "/com.snowplowanalytics.iglu/v1"
  }

  # Configure the P3P policy header.
  p3p {
    policyRef = "/w3c/p3p.xml"
    CP = "NOI DSP COR NID PSA OUR IND COM NAV STA"
  }

  # Cross domain policy configuration.
  # If "enabled" is set to "false", the collector will respond with a 404 to the /crossdomain.xml
  # route.
  crossDomain {
    enabled = false
    # Domains that are granted access, *.acme.com will match http://acme.com and http://sub.acme.com
    #enabled = ${?COLLECTOR_CROSS_DOMAIN_ENABLED}
    domains = [ "*" ]
    #domains = [ ${?COLLECTOR_CROSS_DOMAIN_DOMAIN} ]
    # Whether to only grant access to HTTPS or both HTTPS and HTTP sources
    secure = true
    #secure = ${?COLLECTOR_CROSS_DOMAIN_SECURE}
  }

  # The collector returns a cookie to clients for user identification
  # with the following domain and expiration.
  cookie {
    enabled = false #true  # false Track no cookies at all for now?
    #enabled = ${?COLLECTOR_COOKIE_ENABLED}
    expiration = 0 #{{cookieExpiration}} # e.g. "365 days"
    #expiration = ${?COLLECTOR_COOKIE_EXPIRATION}
    # Network cookie name
    name = 'sp' #{{collectorCookieName}}
    #name = ${?COLLECTOR_COOKIE_NAME}
    # The domain is optional and will make the cookie accessible to other
    # applications on the domain. Comment out these lines to tie cookies to
    # the collector's full domain.
    # The domain is determined by matching the domains from the Origin header of the request
    # to the list below. The first match is used. If no matches are found, the fallback domain will be used,
    # if configured.
    # If you specify a main domain, all subdomains on it will be matched.
    # If you specify a subdomain, only that subdomain will be matched.
    # Examples:
    # domain.com will match domain.com, www.domain.com and secure.client.domain.com
    # client.domain.com will match secure.client.domain.com but not domain.com or www.domain.com
    domains = [
        "{{cookieDomain1}}" # e.g. "domain.com" -> any origin domain ending with this will be matched and domain.com will be returned
        "{{cookieDomain2}}" # e.g. "secure.anotherdomain.com" -> any origin domain ending with this will be matched and secure.anotherdomain.com will be returned
        # ... more domains
    ]
    #domains += ${?COLLECTOR_COOKIE_DOMAIN_1}
    #domains += ${?COLLECTOR_COOKIE_DOMAIN_2}
    # ... more domains
    # If specified, the fallback domain will be used if none of the Origin header hosts matches the list of
    # cookie domains configured above. (For example, if there is no Origin header.)
    fallbackDomain = "{{fallbackDomain}}"
    #fallbackDomain = ${?FALLBACK_DOMAIN}
    secure = false
    #secure = ${?COLLECTOR_COOKIE_SECURE}
    httpOnly = false
    #httpOnly = ${?COLLECTOR_COOKIE_HTTP_ONLY}
    # The sameSite is optional. You can choose to not specify the attribute, or you can use `Strict`,
    # `Lax` or `None` to limit the cookie sent context.
    #   Strict: the cookie will only be sent along with "same-site" requests.
    #   Lax: the cookie will be sent with same-site requests, and with cross-site top-level navigation.
    #   None: the cookie will be sent with same-site and cross-site requests.
    sameSite = 'None' #"{{cookieSameSite}}"
    #sameSite = ${?COLLECTOR_COOKIE_SAME_SITE}
  }

  # If you have a do not track cookie in place, the Scala Stream Collector can respect it by
  # completely bypassing the processing of an incoming request carrying this cookie, the collector
  # will simply reply by a 200 saying "do not track".
  # The cookie name and value must match the configuration below, where the names of the cookies must
  # match entirely and the value could be a regular expression.
  doNotTrackCookie {
    enabled = false #false
    name = '' #{{doNotTrackCookieName}}
    value = '' #{{doNotTrackCookieValue}}
  }

  # When enabled and the cookie specified above is missing, performs a redirect to itself to check
  # if third-party cookies are blocked using the specified name. If they are indeed blocked,
  # fallbackNetworkId is used instead of generating a new random one.
  cookieBounce {
    enabled = false
    # The name of the request parameter which will be used on redirects checking that third-party
    # cookies work.
    name = "n3pc"
    # Network user id to fallback to when third-party cookies are blocked.
    fallbackNetworkUserId = "00000000-0000-4000-A000-000000000000"
    # Optionally, specify the name of the header containing the originating protocol for use in the
    # bounce redirect location. Use this if behind a load balancer that performs SSL termination.
    # The value of this header must be http or https. Example, if behind an AWS Classic ELB.
    forwardedProtocolHeader = "X-Forwarded-Proto"
  }

  # When enabled, redirect prefix `r/` will be enabled and its query parameters resolved.
  # Otherwise the request prefixed with `r/` will be dropped with `404 Not Found`
  # Custom redirects configured in `paths` can still be used.
  enableDefaultRedirect = true

  # When enabled, the redirect url passed via the `u` query parameter is scanned for a placeholder
  # token. All instances of that token are replaced withe the network ID. If the placeholder isn't
  # specified, the default value is `${SP_NUID}`.
  redirectMacro {
    enabled = false
    # Optional custom placeholder token (defaults to the literal `${SP_NUID}`)
    placeholder = "[TOKEN]"
  }

  # Customize response handling for requests for the root path ("/").
  # Useful if you need to redirect to web content or privacy policies regarding the use of this collector.
  rootResponse {
    enabled = false
    statusCode = 302
    # Optional, defaults to empty map
    headers = {
      Location = "https://127.0.0.1/",
      X-Custom = "something"
    }
    # Optional, defaults to empty string
    body = "302, redirecting"
  }

  # Configuration related to CORS preflight requests
  cors {
    # The Access-Control-Max-Age response header indicates how long the results of a preflight
    # request can be cached. -1 seconds disables the cache. Chromium max is 10m, Firefox is 24h.
    accessControlMaxAge = 5 seconds
  }

  # Configuration of prometheus http metrics
  prometheusMetrics {
    # If metrics are enabled then all requests will be logged as prometheus metrics
    # and '/metrics' endpoint will return the report about the requests
    enabled = false
    # Custom buckets for http_request_duration_seconds_bucket duration metric
    #durationBucketsInSeconds = [0.1, 3, 10]
  }
  streams {
    # Events which have successfully been collected will be stored in the good stream/topic
    good = ${COLLECTOR_GOOD_STREAM}
    # Events that are too big (w.r.t Kinesis 1MB limit) will be stored in the bad stream/topic
    bad = ${COLLECTOR_BAD_STREAM}

    # Whether to use the incoming event's ip as the partition key for the good stream/topic
    # Note: Nsq does not make use of partition key.

    useIpAddressAsPartitionKey = false

    # Enable the chosen sink by uncommenting the appropriate configuration
    sink {
      # Choose between kinesis, google-pub-sub, kafka, nsq, or stdout.
      # To use stdout, comment or remove everything in the "collector.streams.sink" section except
      # "enabled" which should be set to "stdout".
      enabled = kinesis

      # Region where the streams are located
      region = ${COLLECTOR_STREAMS_SINK_REGION}

      ## Optional endpoint url configuration to override aws kinesis endpoints,
      ## this can be used to specify local endpoints when using localstack
      #customEndpoint =  localstack:4566

      # Thread pool size for Kinesis API requests
      threadPoolSize = ${COLLECTOR_STREAMS_SINK_THREAD_POOL_SIZE}

      # The following are used to authenticate for the Amazon Kinesis sink.
      # If both are set to 'default', the default provider chain is used
      # (see http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html)
      # If both are set to 'iam', use AWS IAM Roles to provision credentials.
      # If both are set to 'env', use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
      aws {
        accessKey = default #iam
        secretKey = default #iam
      }

      # Minimum and maximum backoff periods, in milliseconds
      backoffPolicy {
        minBackoff = ${COLLECTOR_STREAMS_SINK_MIN_BACKOFF}
        maxBackoff = ${COLLECTOR_STREAMS_SINK_MAX_BACKOFF}
      }

    }

    # Incoming events are stored in a buffer before being sent to Kinesis/Kafka.
    # Note: Buffering is not supported by NSQ.
    # The buffer is emptied whenever:
    # - the number of stored records reaches record-limit or
    # - the combined size of the stored records reaches byte-limit or
    # - the time in milliseconds since the buffer was last emptied reaches time-limit
    buffer {
      byteLimit = ${COLLECTOR_STREAMS_BUFFER_BYTE_LIMIT}
      recordLimit = ${COLLECTOR_STREAMS_BUFFER_RECORD_LIMIT}
      timeLimit = ${COLLECTOR_STREAMS_BUFFER_TIME_LIMIT}
    }
  }

}

# Akka has a variety of possible configuration options defined at
# http://doc.akka.io/docs/akka/current/scala/general/configuration.html
akka {
  loglevel = DEBUG # 'OFF' for no logging, 'DEBUG' for all logging.
  loggers = ["akka.event.slf4j.Slf4jLogger"]

  # akka-http is the server the Stream collector uses and has configurable options defined at
  # http://doc.akka.io/docs/akka-http/current/scala/http/configuration.html
  http.server {
    # To obtain the hostname in the collector, the 'remote-address' header
    # should be set. By default, this is disabled, and enabling it
    # adds the 'Remote-Address' header to every request automatically.
    remote-address-header = off

    raw-request-uri-header = on

    # Define the maximum request length (the default is 2048)
    parsing {
      max-uri-length = 32768
      uri-parsing-mode = relaxed
    }
  }

  # By default setting `collector.ssl` relies on JSSE (Java Secure Socket
  # Extension) to enable secure communication.
  # To override the default settings set the following section as per
  # https://lightbend.github.io/ssl-config/ExampleSSLConfig.html
  ssl-config {
    debug = {
      ssl = true
    }

    keyManager = {
      stores = [
        {type = "PKCS12", classpath = false, path = "/opt/snowplow/ssl/collector.p12", password = ${CERT_PW} }
      ]
    }

    loose {
      disableHostnameVerification = true
    }
  }
}
```

It sounds like you’re not able to reach the collector at all, the error is probably on the network configuration, not the collector.

Hi @BenB

good point but we checked that and the error does not occur on collector version 2.3.1 with exactly the same network settings.

We want to use 2.4.5 in order to overcome the log4j vulnerabilities.

Short update: We tried collector version 2.4.0 now and we get the same problem with 502 when trying to send data to the collector endpoint:

The logs differ only slightly 2.4.0:


The logs for 2.3.1:

The only difference that I can spot is the Warning about the disabled Hostname Verification. I am not sure whether this is relevant.
We will probably have to roll back to 2.3.1. but a vulnerability scan with AWS inspector marked our collector instance to have critical vulnerabilities last week so we would really like to move to 2.4.5.

Hi @mgloel I think this is because starting in 2.4.0, ssl configuration is done using jvm system properties, instead of via the config hocon file. This was a consequence of upgrading the akka-http lib to a newer version.

There are some details on our docs page about how to configure SSL on the newer versions. But in summary, you would remove the akka.ssl-config section from your configuration file, and instead set these system properties using command line arguments:

javax.net.ssl.keyStore
javax.net.ssl.keyStorePassword
javax.net.ssl.keyStoreType
jdk.tls.server.cipherSuites
jdk.tls.server.protocols
1 Like

Ah ok, thanks for the tip.

We just include them as RUN statements in the docker file?

I will check the docs.

For most use cases I suggest you use the provided docker image; I hope there isn’t a need for you to create your own docker file and build your own image.

You can run the docker image like this:

docker run  \
  -v /path/to/config.hocon:/snowplow/config.hocon \
  -p 8000:8000 \
  snowplow/scala-stream-collector-kinesis:2.4.5 \
  -Djavax.net.ssl.keyStore=???? \
  -Djavax.net.ssl.keyStorePassword=??? \
  -Djavax.net.ssl.keyStoreType=??? \
  -Djdk.tls.server.cipherSuites=??? \
  -Djdk.tls.server.protocols=??? \
  --config /snowplow/config.hocon
1 Like

Thanks a lot @istreeter.
Where do we retrieve the values for the ??? ? Could you share an example. As you can see above we did not specify them before in the config file akka section?
Is it possible to use non-custom default values here?

Hi @mgloel I think it’s like this:

  • javax.net.ssl.keyStore corresponds to the path option from your old config. So set it to /opt/snowplow/ssl/collector.p12
  • javax.net.ssl.keyStorePassword corresponds to the password option from your old config. So set it to ${CERT_PW}
  • javax.net.ssl.keyStoreType corresponds to the type option, which for you is PKCS12. But actually that’s the default, so you can omit this option.
  • For jdk.tls.server.cipherSuites and jdk.tls.server.protocols you can leave these as the defaults.
1 Like

Hey,

thanks a lot for pointing us into the right direction. Our dockerfile looks like this now:

FROM snowplow/scala-stream-collector-kinesis:2.4.5

ARG AWS_DEFAULT_REGION
ENV COLLECTOR_STREAMS_SINK_REGION $AWS_DEFAULT_REGION
ENV COLLECTOR_INTERFACE 0.0.0.0
ENV COLLECTOR_PORT 8000
ENV COLLECTOR_SSL_ENABLE true
ENV COLLECTOR_SSL_REDIRECT true
ENV COLLECTOR_SSL_PORT 9543
ENV COLLECTOR_STREAMS_SINK_THREAD_POOL_SIZE 10
ENV COLLECTOR_STREAMS_SINK_MIN_BACKOFF 5000
ENV COLLECTOR_STREAMS_SINK_MAX_BACKOFF 60000
ENV COLLECTOR_STREAMS_BUFFER_BYTE_LIMIT 10000
ENV COLLECTOR_STREAMS_BUFFER_RECORD_LIMIT 5
ENV COLLECTOR_STREAMS_BUFFER_TIME_LIMIT 60
ENV CERT_PW $CERT_PW
ENV SSL_DIR /opt/snowplow/ssl

WORKDIR /app
COPY src/ /app/

# hadolint ignore=DL3002
USER root

RUN sh generate_ssl_cert.sh

CMD ["--config", "oneapp_collector.conf", \
 "-Dcom.amazonaws.sdk.disableCertChecking", "-Dcom.amazonaws.sdk.disableCbor", \
 "-Djavax.net.ssl.keyStore=/opt/snowplow/ssl/collector.p12", \
 "-Djavax.net.ssl.keyStorePassword=${CERT_PW}", \
 "-Djavax.net.ssl.keyStoreType=PKCS12"]

However we are running into several errors now.

  1. Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

  2. Caused by: java.io.IOException: keystore password was incorrect

  3. Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

This is how we generate the ssl cert using openssl:


#!/bin/bash
mkdir -p "$SSL_DIR"

openssl req \
 -x509 \
 -newkey rsa:4096 \
 -keyout "$SSL_DIR/collector_key.pem" \
 -out "$SSL_DIR/collector_cert.pem" \
 -days 3650 \
 -nodes \
 -subj "/C=UK/O=Acme/OU=DevOps/CN=*.acme.com"

openssl pkcs12 \
 -export \
 -out "$SSL_DIR/collector.p12" \
 -inkey "$SSL_DIR/collector_key.pem" \
 -in "$SSL_DIR/collector_cert.pem" \
 -passout "pass:$CERT_PW"

chmod 644 "$SSL_DIR/collector.p12"

As suggested in this post: Enable https on collector; ALB cannot target ECS - #3 by josh

Should we remove this block in the config?
And should we add the telemetry block or is it optional?


 ssl-config {
    debug = {
      ssl = true
    }

    keyManager = {
      stores = [
        {type = "PKCS12", classpath = false, path = "/opt/snowplow/ssl/collector.p12", password = ${CERT_PW} }
      ]
    }

    loose {
      disableHostnameVerification = true
    }
  }

Just out of curiosity are there any log4j issues with 2.3.1? If there is no log4j issue with 2.3.1 we might wait with the update to 2.4.5.
We are still struggling to run it on 2.4.5 :confused:

@istreeter Sorry to bother you again but we are still stuck with this problem. Do we have to add jdk.tls.server.cipherSuites and jdk.tls.server.protocols explicitly and use the default values? What would they be?
The errors indicate that the keystore password is not correct. It is the one we use to generate the ssl certificate using openssl. :thinking:

Hi @mgloel please bear with me on this one. I don’t the answer myself but I will try to find someone who can help. I haven’t forgotten.

2 Likes

Hey @istreeter , that is no problem at all. :slight_smile: Thanks a lot for your help.

Hey @mgloel ,

We want to use 2.4.5 in order to overcome the log4j vulnerabilities.

You can use -Dlog4j2.formatMsgNoLookups=true JVM system property to patch apps with vulnerable log4j versions until the upgrade is complete.

Should we remove this block in the config?

Yes, Stream Collector 2.4.0 removed SSL configuration from config file.

And should we add the telemetry block or is it optional?

It is optional to configure and telemetry is enabled by default. You can check our docs for more details.

I’ll be back with another update soon to solve the initial problem.

Kind regards

1 Like

Hey @mgloel ,

Assuming that you don’t intend to generate an SSL cert per container, I’d suggest to make CERT_PW an ARG like AWS_DEFAULT_REGION and provide a non-empty password at image build time so that ssl cert generation could use it. Current Dockerfile will generate a cert using an empty password. Providing env var CERT_PW for containers might create an illusion that your cert would use that since it is defined at runtime, however it wouldn’t be the case as CERT_PW isn’t available in build time, when cert is generated.

I modified it as following

ARG CERT_PW
ENV CERT_PW $CERT_PW

and then run the following in the directory where I have my Dockerfile

$ docker build --no-cache --build-arg AWS_DEFAULT_REGION=eu-central-1 --build-arg CERT_PW=changeme -t collector-6204 .

but the problem didn’t go away completely.

2nd issue is about the way CERT_PW is provided. Current Dockerfile uses CMD’s exec form which does not perform variable substitution, hence invalid password. We need CMD’s shell form to have interpolation. e.g.

ENTRYPOINT ["/usr/bin/env"]
CMD /opt/snowplow/bin/snowplow-stream-collector --config oneapp_collector.conf -Dcom.amazonaws.sdk.disableCertChecking -Dcom.amazonaws.sdk.disableCbor -Djavax.net.ssl.keyStore=/opt/snowplow/ssl/collector.p12 -Djavax.net.ssl.keyStorePassword=${CERT_PW} -Djavax.net.ssl.keyStoreType=PKCS12

where I override base image’s entrypoint and use shell form of CMD.

After these 2 modifications, I was able to have a working collector 2.4.5 serving over HTTPS.

In case this is your full Docker setup, I recommend using the official docker image by Snowplow, where previously generated SSL certificate and config file could be attached as volumes and JVM properties could be provided as env var JAVA_OPTS.

In case any of the above isn’t clear, please let me know.

Kind regards

1 Like

Hi @oguzhanunlu , thank you so much for your comprehensive reply. I added your suggested changes and built the image as you suggested. This is what our Dockerfile looks like now:

FROM snowplow/scala-stream-collector-kinesis:2.4.5

ARG AWS_DEFAULT_REGION
ARG CERT_PW
ENV COLLECTOR_STREAMS_SINK_REGION $AWS_DEFAULT_REGION
ENV COLLECTOR_INTERFACE 0.0.0.0
ENV COLLECTOR_PORT 8000
ENV COLLECTOR_SSL_ENABLE true
ENV COLLECTOR_SSL_REDIRECT true
ENV COLLECTOR_SSL_PORT 9543
ENV COLLECTOR_STREAMS_SINK_THREAD_POOL_SIZE 10
ENV COLLECTOR_STREAMS_SINK_MIN_BACKOFF 5000
ENV COLLECTOR_STREAMS_SINK_MAX_BACKOFF 60000
ENV COLLECTOR_STREAMS_BUFFER_BYTE_LIMIT 10000
ENV COLLECTOR_STREAMS_BUFFER_RECORD_LIMIT 5
ENV COLLECTOR_STREAMS_BUFFER_TIME_LIMIT 60
ENV CERT_PW $CERT_PW
ENV SSL_DIR /opt/snowplow/ssl

WORKDIR /app
COPY src/ /app/

# hadolint ignore=DL3002
USER root

RUN sh generate_ssl_cert.sh

ENTRYPOINT ["/usr/bin/env"]

CMD ["/opt/snowplow/bin/snowplow-stream-collector", "--config", "oneapp_collector.conf", \
 "-Dcom.amazonaws.sdk.disableCertChecking", "-Dcom.amazonaws.sdk.disableCbor", \
 "-Djavax.net.ssl.keyStore=/opt/snowplow/ssl/collector.p12", \
 "-Djavax.net.ssl.keyStorePassword=${CERT_PW}", \
 "-Djavax.net.ssl.keyStoreType=PKCS12"]

Unfortunately we get the same “password incorrect error” (see Cloudwatch logs above) as before and therefore the endpoint is still not reachable.

What do you mean with the office docker image? We are using snowplow/scala-stream-collector-kinesis:2.4.5 currently. I am a bit confused. :slight_smile: