I am trying to create a docker-compose setup for running FusionAuth behind an Nginx reverse proxy with LetsEncrypt certificates. Standard stuff, should be simple, right?
TL; DR; The problem is that FusionAuth drops the connection without even sending response headers when getting requests from a dockerized Nginx reverse proxy.
Now for the details:
I am running FusionAuth 1.42.1, all on a GCP VM with Ubuntu 22.04 LTS.
The certificate initialisation sequence is done, so I have the SSL proxy part running fine.
For the FusionAuth services I use the docker-compose file from https://fusionauth.io/docs/v1/tech/installation-guide/docker#docker-compose with some modifications and the nginx proxies added.
After having created the SSL cert the services come up fine without complaints:
$ docker-compose up -d
Starting fa-playground_certbot_1 ... done
Starting fa-playground_db_1 ... done
Starting search ... done
Starting fa-playground_fusionauth_1 ... done
Starting fa-playground_nginx_1 ... done
$
The system seems to have plenty of resources left over when everything is running:
$ free
total used free shared buff/cache available
Mem: 4014792 1643124 1225480 16044 1146188 1944304
I can access FusionAuth from the internet without issues on :9011 if I open that port in the firewall.
But accessing the site over https through the Nginx proxy fails.
$ curl -v https://auth.example.com/admin/setup-wizard
* Trying <ip>:443...
* Connected to auth.example.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=auth.example.com
* start date: Jan 17 13:25:15 2023 GMT
* expire date: Apr 17 13:25:14 2023 GMT
* subjectAltName: host "auth.example.com" matched cert's "auth.example.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
> GET /admin/setup-wizard HTTP/1.1
> Host: auth.example.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.22.1
< Date: Thu, 19 Jan 2023 07:34:34 GMT
< Content-Type: text/html
< Content-Length: 157
< Connection: keep-alive
<
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.22.1</center>
</body>
</html>
Because nginx is unhappy with the Fusionauth response:
nginx_1 | 2023/01/19 07:38:30 [error] 29#29: *8 upstream prematurely closed connection while reading response header from upstream, client: __ip__, server: auth.example.com, request: "GET /admin/setup-wizard HTTP/1.1", upstream: "http://192.168.0.4:9011/admin/setup-wizard", host: "auth.example.com"
nginx_1 | __ip__ - - [19/Jan/2023:07:38:30 +0000] "GET /admin/setup-wizard HTTP/1.1" 502 157 "-" "curl/7.79.1" "-"
If I log in to the Nginx docker container and try the request from that error message, it all works fine!
$ docker exec -it 7d3128bc8a17 bash
$ curl -v -s http://192.168.0.4:9011/admin/setup-wizard >/dev/null
* Trying 192.168.0.4:9011...
* Connected to 192.168.0.4 (192.168.0.4) port 9011 (#0)
> GET /admin/setup-wizard HTTP/1.1
> Host: 192.168.0.4:9011
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< set-cookie: fusionauth.flash-message=null; Max-Age=0; Path=/
< transfer-encoding: chunked
< content-type: text/html; charset=UTF-8
< connection: keep-alive
< cache-control: no-cache
<
{ [10878 bytes data]
(Those 10878 bytes is the proper FusionAuth installation setup page)
Here is the nginx config:
server {
listen 80;
listen [::]:80;
server_name auth.example.com;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
server_name auth.example.com;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Server auth.example.com;
proxy_pass http://fusionauth:9011;
proxy_read_timeout 10s;
proxy_connect_timeout 5s;
}
listen [::]:443 ssl ipv6only=on;
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/auth.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/auth.example.com/privkey.pem;
include /etc/letsencrypt/conf/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/conf/ssl-dhparams.pem;
}
There are no errors in the FusionAuth logs.
So I log in to the FusionAuth container and install tcpdump, which shows me the GET request coming in, but getting no response at all, just a closed connection:
# tcpdump port 9011 -s 0 -A
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:05:11.065637 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [S], seq 4186643246, win 64240, options [mss 1460,sackOK,TS val 2705402033 ecr 0,nop,wscale 7], length 0
E..<.r@.@.............#3.......................
.A(.........
15:05:11.065668 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [S.], seq 2717654938, ack 4186643247, win 65160, options [mss 1460,sackOK,TS val 3969280838 ecr 2705402033,nop,wscale 7], length 0
E..<..@.@..b........#3........./...............
..kF.A(.....
15:05:11.065739 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [.], ack 1, win 502, options [nop,nop,TS val 2705402033 ecr 3969280838], length 0
E..4.s@.@.............#3.../...............
.A(...kF
15:05:11.065793 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [P.], seq 1:718, ack 1, win 502, options [nop,nop,TS val 2705402033 ecr 3969280838], length 717
E....t@.@..)..........#3.../.........M.....
.A(...kFGET /admin/setup-wizard HTTP/1.0
Host: fusionauth
Connection: close
Cache-Control: max-age=0
sec-ch-ua: "Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,sv;q=0.7
15:05:11.065798 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [.], ack 718, win 504, options [nop,nop,TS val 3969280838 ecr 2705402033], length 0
E..4.D@.@..&........#3.....................
..kF.A(.
15:05:11.066377 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [F.], seq 1, ack 718, win 504, options [nop,nop,TS val 3969280839 ecr 2705402033], length 0
E..4.E@.@..%........#3.....................
..kG.A(.
15:05:11.066534 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [F.], seq 718, ack 2, win 502, options [nop,nop,TS val 2705402034 ecr 3969280839], length 0
E..4.u@.@.............#3...................
.A(...kG
15:05:11.066544 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [.], ack 719, win 504, options [nop,nop,TS val 3969280839 ecr 2705402034], length 0
E..4.F@.@..$........#3.....................
..kG.A(.
Any ideas on what could cause this?