Proxy before Collector

For engineers
1

Hello humans!
I am testing a proxy (reverse) to send the events to the my collector.
The conf I have in my apache proxy is:

<VirtualHost *:80>
    ServerName x.x.x.x
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www/html

    SSLProxyVerify None
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerExpire Off

    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} OPTIONS
    RewriteRule ^(.*)$ $1 [R=200,L]

    ProxyRequests Off
    ProxyPreserveHost On
    ProxyAddHeaders Off

    ProxyPass "/endpoint" "http://com-collector"
    ProxyPassReverse  "/endpoint" "http://com-collector"

    Header always set Access-Control-Allow-Origin "http://my-nice-site.com"
    Header always set Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
    Header always set Access-Control-Allow-Headers "Content-Type, Authorization"
    Header always set Access-Control-Allow-Credentials "true"

    LogLevel debug
    ErrorLog "/var/log/apache2/test-error.log"
    TransferLog "/var/log/apache2/test-transfer-error.log"
</VirtualHost>

and the logs are saying:

[Thu Oct 12 11:17:37.953544 2023] [headers:debug] [pid 5285:tid 140481979680512] mod_headers.c(890): AH01503: headers: ap_headers_error_filter()
[Thu Oct 12 11:17:38.063726 2023] [authz_core:debug] [pid 5285:tid 140481962878720] mod_authz_core.c(843): [client happy.client:212] AH01628: authorization result: granted (no directives)
[Thu Oct 12 11:17:38.063798 2023] [proxy:debug] [pid 5285:tid 140481962878720] mod_proxy.c(1506): [client happy.client:212] AH01143: Running scheme http handler (attempt 0)
[Thu Oct 12 11:17:38.063809 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(2538): AH00942: http: has acquired connection for (happy-collector:80)
[Thu Oct 12 11:17:38.063819 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(2596): [client happy.client:212] AH00944: connecting http://happy-collector/endpoint to happy-collector:80
[Thu Oct 12 11:17:38.063825 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(2819): [client happy.client:212] AH00947: connected /endpoint to happy-collector:80
[Thu Oct 12 11:17:38.081945 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(3120): AH00951: http: backend socket is disconnected.
[Thu Oct 12 11:17:38.083052 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(3291): AH02824: http: connection established with happy-collector:80 (happy-collector:80)
[Thu Oct 12 11:17:38.083098 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(3480): AH00962: http: connection complete to happy-collector:80 (happy-collector)
[Thu Oct 12 11:17:38.105788 2023] [proxy:debug] [pid 5285:tid 140481962878720] proxy_util.c(2554): AH00943: http: has released connection for (happy-collector:80)
[Thu Oct 12 11:17:39.927753 2023] [headers:debug] [pid 5285:tid 140481946076928] mod_headers.c(890): AH01503: headers: ap_headers_error_filter()
[Thu Oct 12 11:17:39.996135 2023] [authz_core:debug] [pid 5285:tid 140481929275136] mod_authz_core.c(843): [client happy.client:212] AH01628: authorization result: granted (no directives)
[Thu Oct 12 11:17:39.996190 2023] [proxy:debug] [pid 5285:tid 140481929275136] mod_proxy.c(1506): [client happy.client:212] AH01143: Running scheme http handler (attempt 0)
[Thu Oct 12 11:17:39.996198 2023] [proxy:debug] [pid 5285:tid 140481929275136] proxy_util.c(2538): AH00942: http: has acquired connection for (happy-collector:80)
[Thu Oct 12 11:17:39.996206 2023] [proxy:debug] [pid 5285:tid 140481929275136] proxy_util.c(2596): [client happy.client:212] AH00944: connecting http://happy-collector/endpoint to happy-collector:80
[Thu Oct 12 11:17:39.996213 2023] [proxy:debug] [pid 5285:tid 140481929275136] proxy_util.c(2819): [client happy.client:212] AH00947: connected /endpoint to happy-collector:80
[Thu Oct 12 11:17:40.054803 2023] [proxy:debug] [pid 5285:tid 140481929275136] proxy_util.c(3291): AH02824: http: connection established with happy-collector:80 (happy-collector:80)
[Thu Oct 12 11:17:40.054850 2023] [proxy:debug] [pid 5285:tid 140481929275136] proxy_util.c(3480): AH00962: http: connection complete to happy-collector:80 (happy-collector)
[Thu Oct 12 11:17:40.076774 2023] [proxy:debug] [pid 5285:tid 140481929275136] proxy_util.c(2554): AH00943: http: has released connection for (happy-collector:80)

however I can’t find any event in my tables, not bad or good events.

Any Idea what I am missing here?

2

Hi @felipe_ache,

A few questions:

  • How did you deploy the pipeline?
  • What cloud are you on?
  • What warehouse/database/lake are you using (what loader)?
  • How are you sending the events?
  • Do you see events in the collector logs?
  • Have you tried sending the events prior to configuring the reverse proxy? If yes, can we assume it was successful?
3

hi @stanch , thanks for your answer.

  1. The Pipeline has been deployed by the SP folks into my organisation.
  2. GCP
  3. BQ
  4. I am using a JS Tracker. I am passing: myIP/endpoint as collector and then ProxyPassing the request to the real collector. I can see the payload in my the network tab of my website and I can see the proxy redirecting the request but nothing in the collectors log (this is weird because the last log’s entry is on Setpember)
  5. If, instead of my proxy endpoint, I use the collector url I see the events in my events table, but no entry in the collector log. (I am looking the “collector” pod log)
4

Hi @felipe_ache,

Thanks for those details. I didn’t realize this was for BDP — I suggest to follow up via Snowplow Support.

I am not an expert on Apache configuration, but the only thing that jumped out to me was this bit:

ProxyPass “/endpoint” “http://com-collector”

What’s /endpoint? Is it just the standard /com.snowplowanalytics.snowplow/tp2? Then perhaps it should be the following?

ProxyPass “/endpoint” “http://com-collector/endpoint”

Maybe others on the forum have more knowledge of Apache and better suggestions.