P1: HTTPS routing and source IP management

We're still unable to LOG We struggled trying to LOG original source-IP of https-client served by some of our containerized web-servers and we're searching for help.
Here is how we solved the problem !

1 - Context

We’re running a not-so-simple infrastructure, built on top of GARR-Cloud services.

Actually, we’re running resources on both the Catania and Palermo OpenStack regions of GARR-Cloud. In each region we’re running our own firewalling service between the public Internet and our own resources.

Within such infrastructure we’ve granted GARRLab students the possibility to hosts web-based services they would like to learn/test/analyze. In short, they are able to launch web-sites to serve contents and services at their wish. Such content is provided via containers hosted in a specific VM acting as a docker-host.

We decided, also that HTTPS connection must be terminated on student-managed resources and, as such, need to be properly proxied by the main firewall.

Also, at the moment, the firewall acting as the entry-point for HTTPS connection and the VM hosting the containers are located in different openstack regions.

With such an environment, source IP addresses as seen by student reverse-proxy is normally the one of the firewall, and not the one of the original HTTPS client.

In the following picture you can see a draft of the whole architecture: GARRLab schema

where you can see:

  1. remote client requesting HTTPS url

  2. GARRLab firewall running at the border of Region B

  3. GARRLab firewall running at the border of Region A

  4. haproxy running in TCP mode, so to ensure proper TLS forwarding, based on SNI HTTPS attribute

  5. VM hosted in Region A acting as docker-host

  6. traefik reverse-proxy, managed by students, receiving TLS connection and properly terminating it

  7. container running the web-server providing the requested content

2 - The problem

Source IP address of the HTTPS connection received and managed by traefik are the one of the Region B firewall and not the one of the remote client

3 - Our (failed!) solutions

We’ve been able to tell haproxy to forward HTTPS connection keeping track of the real source IP address.

In detail, we have used the source directive, in this way:

source 172.16.17.1 usesrc clientip

With such a directive, haproxy [4] take care to forward the request keeping the right source IP and the request is properly routed to trafik [6] and to the destination container [7].

Problem is that HTTPS response got routed back via default-gateway of region A, causing asymetric routing and…​ problems.

4 - Some elements we considered (…​but not tested)

There were several "hacks" that we could have implement to solve the problem, starting from hosting the docker-host [5] in the same region hosting the haproxy [4].

We could also have switched from tcp-mode to http-mode and use X-Forwarded-for header…​

…​but, you know, we really want to solve the problem the way we created it.

We’re also open to other approaches, providing the already-mentioned constraint:

  • TLS need to be terminated by student (currently, in a docker container running in their VM)

  • being able to log something at the border-firewall, out of control from students.

5 - How we solved the problem

We learned about the ProxyProtocol, and we discovered that it’s supported by both Traefik and HAProxy.

So we simply enabled it on the HaProxy side with the addition of the send-proxy attribute to related backend:

backend traefik
  server traefik vm-garristini.garr.lab:443 maxconn 2048 send-proxy

and telling Traefik to properly handle it, at the entry-point level:

--entrypoints.web-public.proxyProtocol=true
--entrypoints.web-public.proxyProtocol.trustedIPs=172.16.17.1,172.16.16.117,172.25.0.2

We also kindly asked Traefik to LOG requests, with this:

--accesslog=true
--accesslog.format=json

With above additions, what we got logged is a detailed JSON for each request. Something like this:

{
   "ClientAddr": "151.57.29.201:44857",
   "ClientHost": "151.57.29.201",
   "ClientPort": "44857",
   "ClientUsername": "-",
   "DownstreamContentSize": 0,
   "DownstreamStatus": 304,
   "Duration": 3147663,
   "OriginContentSize": 0,
   "OriginDuration": 2241521,
   "OriginStatus": 304,
   "Overhead": 906142,
   "RequestAddr": "www.garrlab.it",
   "RequestContentSize": 0,
   "RequestCount": 14372,
   "RequestHost": "www.garrlab.it",
   "RequestMethod": "GET",
   "RequestPath": "/fontawesome/webfonts/fa-solid-900.woff2",
   "RequestPort": "-",
   "RequestProtocol": "HTTP/2.0",
   "RequestScheme": "https",
   "RetryAttempts": 0,
   "RouterName": "webgarrlab@docker",
   "ServiceAddr": "172.25.0.15:8080",
   "ServiceName": "webgarrlab@docker",
   "ServiceURL": {
      "Scheme": "http",
      "Opaque": "",
      "User": null,
      "Host": "172.25.0.15:8080",
      "Path": "",
      "RawPath": "",
      "ForceQuery": false,
      "RawQuery": "",
      "Fragment": "",
      "RawFragment": ""
   },
   "StartLocal": "2022-07-05T10:41:01.336868924Z",
   "StartUTC": "2022-07-05T10:41:01.336868924Z",
   "TLSCipher": "TLS_AES_128_GCM_SHA256",
   "TLSVersion": "1.3",
   "downstream_Accept-Ranges": "bytes",
   "downstream_Cache-Control": "max-age=60",
   "downstream_Content-Type": "text/plain; charset=UTF-8",
   "downstream_Date": "Tue, 05 Jul 2022 10:41:01 GMT",
   "downstream_Expires": "Tue, 05 Jul 2022 10:42:01 GMT",
   "downstream_Last-Modified": "Thu, 30 Jun 2022 23:05:19 GMT",
   "downstream_Server": "thttpd/2.29 23May2018",
   "entryPointName": "web-public",
   "level": "info",
   "msg": "",
   "origin_Accept-Ranges": "bytes",
   "origin_Cache-Control": "max-age=60",
   "origin_Content-Type": "text/plain; charset=UTF-8",
   "origin_Date": "Tue, 05 Jul 2022 10:41:01 GMT",
   "origin_Expires": "Tue, 05 Jul 2022 10:42:01 GMT",
   "origin_Last-Modified": "Thu, 30 Jun 2022 23:05:19 GMT",
   "origin_Server": "thttpd/2.29 23May2018",
   "request_Accept": "application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8",
   "request_Accept-Encoding": "identity",
   "request_Accept-Language": "en-US,en;q=0.5",
   "request_Cookie": "***REDACTED***",
   "request_Dnt": "1",
   "request_If-Modified-Since": "Thu, 30 Jun 2022 23:05:19 GMT",
   "request_Referer": "https://www.garrlab.it/fontawesome/css/all.min.css",
   "request_Sec-Fetch-Dest": "font",
   "request_Sec-Fetch-Mode": "cors",
   "request_Sec-Fetch-Site": "same-origin",
   "request_Sec-Gpc": "1",
   "request_Te": "trailers",
   "request_User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0",
   "request_X-Forwarded-Host": "www.garrlab.it",
   "request_X-Forwarded-Port": "443",
   "request_X-Forwarded-Proto": "https",
   "request_X-Forwarded-Server": "8e3966b258ca",
   "request_X-Real-Ip": "151.57.29.201",
   "time": "2022-07-05T10:41:01Z"
}

…​and as you can see, we have the source IP address ( 151.57.29.210 in the above request).

Mission complete !

P.S.: if you’re still curious about other technicalities from us, feel free to give a look to how we deal with this very website !