Making updates to referers.yml in scala lib

For engineers
1

Hi!

We’re using scala-referer-parser as a part of one of our pipelines, but we noticed that we’re using an old 0.3.0 version from sbt. The referers.yml is out of date, and we’d like to add a few websites to it.

We have an old implementation of using scala-referer-parser that adds this to the build.sbt:

libraryDependencies ++= Seq(
...
 "com.snowplowanalytics" %% "referer-parser" % "0.3.0" excludeAll(
    ExclusionRule("org.apache.httpcomponents", "httpclient")
  ),
...
)
assemblyMergeStrategy in assembly := {
  case "referers.yml" => MergeStrategy.rename
  case "module-info.class" => MergeStrategy.discard
  case x => (assemblyMergeStrategy in assembly).value(x)
}

Should this enable us to have a local referers.yml that can get merged into library’s referers.yml at assembly time? If so, can you please help me understand why I’m not able to get referer-parser to use my entries in referers.yml?

Our referers.yml is at src/scala/src/main/resources/referers.yml, if that helps.

Thank you in advance :slight_smile:

2

Hi @Sid_Thota,

You always can get the latest version of the DB from S3:

Also if you want to add it to the resources it should be in a path like:

src/main/resources/referers.yml
3

Hey @anton, thank you for the pointers.

What’s the criteria for how Social referer list is filled, for example? I see that there’s no TikTok on it, for example, or Snapchat either. We’re having to maintain our own version of this list, and we’d much rather contribute to your list anyway.

I’m trying to understand your approach to maintaining this list, and if and how frequently it’s updated.

4

Hi @Sid_Thota ,

Sorry for taking some time to get back to you.

You’re very welcome to open a PR with your additions here.

Once merged the new version of the database gets automatically published every month as json and this json can be used at runtime by latest scala-referer-parser (instructions here) so that you don’t need to have it a compile time any more.