[Solved] - Iglu Server - RDS Postgres Server

For engineers
1

Hi team!

I trying to run a Iglu Server with a AWS Postgresql (RDS) as an instance database server to store internal tables such as APIKeys and JSON Schemas. In our pipeline we use a secrets vault in k8s and the connection string is built dynamically while the docker image is built, and we received this message (:
{
“data-file”:"/etc/vault/data.json",
“dry-run”:false,
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/cmd/kubevault/main.go:176",
“func”:“main.before”,
“level”:“info”,
“log-format”:“json”,
“log-level”:“info”,
“msg”:“starting kubevault”,
“time”:“2019-07-29T23:13:56Z”
}{
“address”:“https://vault.xxxxxxxxxx.com.br:443”,
“env-file”:"/etc/vault/env",
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/action/init.go:37",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/action.VaultInit.Run”,
“level”:“info”,
“msg”:“running kubevault init”,
“role”:“iglu5a-forno-k8s-sandbox-01”,
“time”:“2019-07-29T23:13:56Z”
}{
“data-file”:"/etc/vault/data.json",
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/client/vault.go:93",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/client.LoadVaultData”,
“level”:“info”,
“msg”:“loading vault data”,
“time”:“2019-07-29T23:13:56Z”
}{
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/client/vault.go:158",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/client.(*VaultWrapper).loadSecrets”,
“level”:“info”,
“msg”:“reading secret”,
“role”:“iglu5a-forno-k8s-sandbox-01”,
“secret-field”:“password”,
“secret-name”:“DB_IGLU_PASSWORD”,
“secret-path”:“database/creds/forno-migration-iglu”,
“time”:“2019-07-29T23:13:56Z”
}{
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/client/vault.go:158",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/client.(*VaultWrapper).loadSecrets”,
“level”:“info”,
“msg”:“reading secret”,
“role”:“iglu5a-forno-k8s-sandbox-01”,
“secret-field”:“username”,
“secret-name”:“DB_IGLU_USERNAME”,
“secret-path”:“database/creds/forno-migration-iglu”,
“time”:“2019-07-29T23:13:56Z”
}{
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/client/vault.go:211",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/client.(*VaultWrapper).saveEnvFile”,
“level”:“info”,
“msg”:“saving env file”,
“path”:"/etc/vault/env",
“time”:“2019-07-29T23:13:56Z”
}{
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/client/vault.go:226",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/client.(*VaultWrapper).saveEnvFile”,
“leaseID”:“database/creds/forno-migration-iglu/wraaZw3ST1HSQxALbxTk2Kws”,
“level”:“info”,
“msg”:“vault username info”,
“name”:“DB_IGLU_USERNAME”,
“time”:“2019-07-29T23:13:56Z”,
“username”:“v-k8s-sand-forno-mi-lzyF8KI6I9PVctx78xAW-1564442012”
}{
“data-file”:"/etc/vault/data.json",
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/client/vault.go:183",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/client.(*VaultWrapper).saveVaultData”,
“level”:“info”,
“msg”:“saving vault data”,
“time”:“2019-07-29T23:13:56Z”
}{
“args”:[
“java”,
“-jar”,
“iglu-server-0.4.0.jar”,
“–config”,
“/snowplow/config/application.conf”
],
“file”:"/go/src/github.com/xxxxxxxxxx/kubevault/internal/action/init.go:97",
“func”:“github.com/xxxxxxxxxx/kubevault/internal/action.runNextProcess”,
“level”:“info”,
“msg”:“Starting next process”,
“time”:“2019-07-29T23:13:56Z”
}[
DEBUG
][
07 /29/2019 23:13:57.783
][
main
][
EventStream
]StandardOutLogger started[
iglu-server-akka.actor.default-dispatcher-3
]INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started[
DEBUG
][
07 /29/2019 23:13:58.084
][
main
][
EventStream(akka://iglu-server)
]logger log1-Slf4jLogger started[
DEBUG
][
07 /29/2019 23:13:58.085
][
main
][
EventStream(akka://iglu-server)
]Default Loggers started
There is a problem with database initialization:The connection attempt failed. Check your credentials.

My Iglu configuration file:
postgres {
host = “{DB_IGLU_HOST}" port = 5432 dbname = "{DB_IGLU_NAME}”
username = “{DB_IGLU_USERNAME}" password = "{DB_IGLU_PASSWORD}”
driver = “org.postgresql.Driver”
}

Did I missed any detail? Does the iglu configuration file has to be build in docker entrypoint? I have no idea what’s happening.

Thanks

2

Hi @fbattestin ,

Could you provide us more details on how you build/run Iglu Server? I only see that you use version 0.4.0.

Does the iglu configuration file has to be build in docker entrypoint?

The entrypoint of Iglu Server 0.4.0’s docker image expects a configuration file, i.e. building Iglu Server image from scratch doesn’t require a configuration file while running the server requires one. See this on how you can run an Iglu Server in several examples, including in a kubernetes cluster.

Thanks,
Oguzhan

3

@oguzhanunlu
First of all, sorry for delay to reply!

Well, our image version is 0.4.0 and basically is the same from here (with some particular enviroment details).

My problem was about setting key vault variables to authenticate in postgresql. Our pipeline request secrets from vault like this:

  env:
    DB_HOSTNAME: db.sharedpsql.xxxxx.xxxxx.com
    DBNAME: iglu
  secrets:
    DB_USERNAME: database/creds/xxxx-xxxxx-iglu#username
    DB_PASSWORD: database/creds/xxxx-xxxxx-iglu#password
  cmd: [ "/bin/sh", "-c", "/usr/bin/java -jar /snowplow/bin/iglu-server-0.4.0.jar --config /snowplow/bin/application.conf" ]

and them assigned these vars in application.conf:

postgres {
  host = "${DB_HOSTNAME}"
  port = 5432
  dbname = "${DB_DBNAME}"
  username = "${DB_USERNAME}"
  password = "${DB_PASSWORD}"
  driver = "org.postgresql.Driver"
}

But, in runtime, these assigned dosen’t work. Because the Dockerfile is deployed first of my yaml file. That was the problem and the workaround comes form this

In my yaml file i setting up me enviroment variables as the same name of the code:

  val pgHost = env("**IGLU_PG_HOSTNAME**") getOrElse config.getString("postgres.host")
  val pgPort = env("**IGLU_PG_PORT**").map(_.toInt) getOrElse config.getInt("postgres.port")
  val pgDbName = env("**IGLU_PG_DBNAME**") getOrElse config.getString("postgres.dbname")
  val pgUsername = env("**IGLU_PG_USERNAME**") getOrElse config.getString("postgres.username")
  val pgPassword = env("**IGLU_PG_PASSWORD**") getOrElse config.getString("postgres.password")

So now my yaml file:

  env:
    **IGLU_PG_HOSTNAME**: db.sharedpsql.xxxxx.xxxxxxxxx.com
    **IGLU_PG_DBNAME**: iglu
  secrets:
    **IGLU_PG_USERNAME**: database/creds/xxxxxxx-xxxxxxx-iglu#username
    **IGLU_PG_PASSWORD**: database/creds/xxxxxxx-xxxxxxx-iglu#password
  cmd: [ "/bin/sh", "-c", "/usr/bin/java -jar /snowplow/bin/iglu-server-0.4.0.jar --config /snowplow/bin/application.conf" ]

In that case, we have a bunch of other options to achieve this goal, like working with configMaps or built-in application.conf, but, stay registred one of other strategies of deployment.

Tks!

4

In response to the email from @oguzhanunlu:

Hi @fbattestin , Could you provide us more details on how you build/run Iglu Server? I only see that you use version 0.4.0. Does the iglu configuration file has to be build in docker entrypoint? The entrypoint of Iglu Server 0.4.0’s docker image expects a configuration file, i.e. building Iglu …

more details on how you build/run Iglu Server
Our Iglu Server runs on a K8s cluster with external Postgresql RDS as a JSON schema repository. and is built in Drone.io pipiline + yaml composer.

Dockerfile:
FROM snowplow-docker-registry.bintray.io/snowplow/base-debian:0.1.0

# version of the server to download
ENV IGLU_SERVER_VERSION="0.4.0"
ENV IGLU_CONFIG_PATH="/snowplow/config"

# name of the archive to download
ENV ARCHIVE="iglu_server_${IGLU_SERVER_VERSION}.zip"

# run build pre-requirements
RUN mkdir -p /tmp/build && \
cd /tmp/build && \
wget -q http://dl.bintray.com/snowplow/snowplow-generic/${ARCHIVE} && \
unzip -d ${SNOWPLOW_BIN_PATH} ${ARCHIVE} && \
cd /tmp && \
rm -rf /tmp/build && \
chown -R snowplow:snowplow ${SNOWPLOW_BIN_PATH} && \
apt-get update && \
apt-get install -y procps && \
mkdir -p /usr/share/man/man7 && \
apt-get install -y postgresql-client-9.6

EXPOSE 8089

# run build post-requirements
ADD database/pgsql.sh ${SNOWPLOW_BIN_PATH}
ADD application.conf ${IGLU_CONFIG_PATH}/application.conf
ADD entrypoint/docker-entrypoint.sh /usr/local/bin/

RUN chmod 777 ${SNOWPLOW_BIN_PATH}/* && \
chown -R snowplow:snowplow ${SNOWPLOW_BIN_PATH}

ENTRYPOINT [ "docker-entrypoint.sh" ]

CMD [ "--help" ]

OBS.: We build a image of postgresql to had a client to insert of API master KEY and had any interaction with the DB (when/if necessary).

Configuration file:

# port on which the server will be running
repo-server {
  interface = "0.0.0.0"
  baseURL = "iglu5a.forno.xxxxxxxxxxxx.com.br"
  port = 8089
}

# 'postgres' contains configuration options for the postgre instance the server
# is using
postgres {
  host = "${IGLU_PG_HOSTNAME}"
  port = 5432
  dbname = "${IGLU_PG_DBNAME}"
  username = "${IGLU_PG_USERNAME}"
  password = "${IGLU_PG_PASSWORD}"
  driver = "org.postgresql.Driver"
}

akka {
    actor {
    debug {
      # enable DEBUG logging of all LoggingFSMs for events, transitions and timers
      fsm = on
    }
  }
  loggers = ["akka.event.slf4j.Slf4jLogger"]
  loglevel = "DEBUG"
  stdout-loglevel = "DEBUG"
  log-config-on-start = on
  logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
}

akka.http {
  server {
    request-timeout = 10 seconds
    remote-address-header = on
    parsing.uri-parsing-mode = relaxed
  }
}

OBS.: What if I need an extremely detailed logging level? What I need to do?

Entrypoint file:

#!/usr/bin/dumb-init /bin/sh
set -e

# if the config directory has been mounted through -v,  chown it.
if [ "$(stat -c %u ${SNOWPLOW_CONFIG_PATH})" != "$(id -u snowplow)" ]; then
  chown -R snowplow:snowplow ${SNOWPLOW_CONFIG_PATH}
fi

exec gosu snowplow:snowplow /usr/bin/java \
  $SP_JAVA_OPTS -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap \
  -jar ${SNOWPLOW_BIN_PATH}/iglu-server-${IGLU_SERVER_VERSION}.jar "$@"

yaml file:

app:
  name: iglu5a
  port: 8089
  public: true
  iamRoleName: iglu5a
  repository: gcr.io/xxxxxxhub/snowplow-iglu
  secrets:
    IGLU_PG_USERNAME: database/creds/forno-migration-iglu#username
    IGLU_PG_PASSWORD: database/creds/forno-migration-iglu#password
  env:
    IGLU_PG_HOSTNAME: db.sharedpsql.forno.xxxxxxxxx.com.br
    IGLU_PG_DBNAME: iglu
    PGPASSWORD: ${IGLU_PG_PASSWORD}
  cmd: [ "/bin/sh", "-c"]
  args: ["/snowplow/bin/pgsql.sh; rm /snowplow/bin/pgsql.sh; /usr/bin/java -jar /snowplow/bin/iglu-server-0.4.0.jar --config /snowplow/config/application.conf"]
  annotations:
    xxxx.to/squad: data
    xxxx.to/tribe: cross # 'top-of-funnel', 'conversion', 'scalability' or 'cross'
  livenessProbe:
    exec:
      command:
        - pgrep
        - -f
        - iglu
    initialDelaySeconds: 300
    timeoutSeconds: 5
  readinessProbe:
    exec:
      command:
        - pgrep
        - -f
        - iglu
    timeoutSeconds: 5
  resources:
    limits:
      cpu: 600m
      memory: 1024Mi
    requests:
      cpu: 400m
      memory: 1024Mi
  autoscaling:
    minReplicas: 1
    maxReplicas: 1
    targetCPUUtilizationPercentage: 70