Fusionauth drops requests from nginx reverse proxy

jesper

I am trying to create a docker-compose setup for running FusionAuth behind an Nginx reverse proxy with LetsEncrypt certificates. Standard stuff, should be simple, right?

TL; DR; The problem is that FusionAuth drops the connection without even sending response headers when getting requests from a dockerized Nginx reverse proxy.

Now for the details:

I am running FusionAuth 1.42.1, all on a GCP VM with Ubuntu 22.04 LTS.

The certificate initialisation sequence is done, so I have the SSL proxy part running fine.

For the FusionAuth services I use the docker-compose file from https://fusionauth.io/docs/v1/tech/installation-guide/docker#docker-compose with some modifications and the nginx proxies added.

After having created the SSL cert the services come up fine without complaints:

$ docker-compose up -d
Starting fa-playground_certbot_1 ... done
Starting fa-playground_db_1      ... done
Starting search                  ... done
Starting fa-playground_fusionauth_1 ... done
Starting fa-playground_nginx_1      ... done
$

The system seems to have plenty of resources left over when everything is running:

$ free
               total        used        free      shared  buff/cache   available
Mem:         4014792     1643124     1225480       16044     1146188     1944304

I can access FusionAuth from the internet without issues on :9011 if I open that port in the firewall.
But accessing the site over https through the Nginx proxy fails.

$ curl -v https://auth.example.com/admin/setup-wizard
*   Trying <ip>:443...
* Connected to auth.example.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=auth.example.com
*  start date: Jan 17 13:25:15 2023 GMT
*  expire date: Apr 17 13:25:14 2023 GMT
*  subjectAltName: host "auth.example.com" matched cert's "auth.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET /admin/setup-wizard HTTP/1.1
> Host: auth.example.com
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.22.1
< Date: Thu, 19 Jan 2023 07:34:34 GMT
< Content-Type: text/html
< Content-Length: 157
< Connection: keep-alive
< 
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.22.1</center>
</body>
</html>

Because nginx is unhappy with the Fusionauth response:

nginx_1        | 2023/01/19 07:38:30 [error] 29#29: *8 upstream prematurely closed connection while reading response header from upstream, client: __ip__, server: auth.example.com, request: "GET /admin/setup-wizard HTTP/1.1", upstream: "http://192.168.0.4:9011/admin/setup-wizard", host: "auth.example.com"
nginx_1        | __ip__ - - [19/Jan/2023:07:38:30 +0000] "GET /admin/setup-wizard HTTP/1.1" 502 157 "-" "curl/7.79.1" "-"

If I log in to the Nginx docker container and try the request from that error message, it all works fine!

$ docker exec -it 7d3128bc8a17 bash
$ curl -v -s http://192.168.0.4:9011/admin/setup-wizard >/dev/null
*   Trying 192.168.0.4:9011...
* Connected to 192.168.0.4 (192.168.0.4) port 9011 (#0)
> GET /admin/setup-wizard HTTP/1.1
> Host: 192.168.0.4:9011
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< set-cookie: fusionauth.flash-message=null; Max-Age=0; Path=/
< transfer-encoding: chunked
< content-type: text/html; charset=UTF-8
< connection: keep-alive
< cache-control: no-cache
< 
{ [10878 bytes data]

(Those 10878 bytes is the proper FusionAuth installation setup page)

Here is the nginx config:

server {
  listen 80;
  listen [::]:80;
  server_name auth.example.com;
  location /.well-known/acme-challenge/ {
    root /var/www/certbot;
  }

  location / {
      return 301 https://$host$request_uri;
  }
}

server {
    server_name auth.example.com;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Server auth.example.com;
        proxy_pass http://fusionauth:9011;
        proxy_read_timeout 10s;
        proxy_connect_timeout 5s;
    }

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/auth.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/auth.example.com/privkey.pem;
    include /etc/letsencrypt/conf/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/conf/ssl-dhparams.pem;
}

There are no errors in the FusionAuth logs.

So I log in to the FusionAuth container and install tcpdump, which shows me the GET request coming in, but getting no response at all, just a closed connection:

# tcpdump port 9011  -s 0 -A
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:05:11.065637 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [S], seq 4186643246, win 64240, options [mss 1460,sackOK,TS val 2705402033 ecr 0,nop,wscale 7], length 0
E..<.r@.@.............#3.......................
.A(.........
15:05:11.065668 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [S.], seq 2717654938, ack 4186643247, win 65160, options [mss 1460,sackOK,TS val 3969280838 ecr 2705402033,nop,wscale 7], length 0
E..<..@.@..b........#3........./...............
..kF.A(.....
15:05:11.065739 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [.], ack 1, win 502, options [nop,nop,TS val 2705402033 ecr 3969280838], length 0
E..4.s@.@.............#3.../...............
.A(...kF
15:05:11.065793 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [P.], seq 1:718, ack 1, win 502, options [nop,nop,TS val 2705402033 ecr 3969280838], length 717
E....t@.@..)..........#3.../.........M.....
.A(...kFGET /admin/setup-wizard HTTP/1.0
Host: fusionauth
Connection: close
Cache-Control: max-age=0
sec-ch-ua: "Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,sv;q=0.7


15:05:11.065798 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [.], ack 718, win 504, options [nop,nop,TS val 3969280838 ecr 2705402033], length 0
E..4.D@.@..&........#3.....................
..kF.A(.
15:05:11.066377 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [F.], seq 1, ack 718, win 504, options [nop,nop,TS val 3969280839 ecr 2705402033], length 0
E..4.E@.@..%........#3.....................
..kG.A(.
15:05:11.066534 IP fa-playground_nginx_1.fa-playground_fa_net.48144 > d762d8fca53b.9011: Flags [F.], seq 718, ack 2, win 502, options [nop,nop,TS val 2705402034 ecr 3969280839], length 0
E..4.u@.@.............#3...................
.A(...kG
15:05:11.066544 IP d762d8fca53b.9011 > fa-playground_nginx_1.fa-playground_fa_net.48144: Flags [.], ack 719, win 504, options [nop,nop,TS val 3969280839 ecr 2705402034], length 0
E..4.F@.@..$........#3.....................
..kG.A(.

Any ideas on what could cause this?

dan

@jesper Sorry to hear about the issues.

Can you try setting the http version, as outlined here:

https://fusionauth.io/community/forum/topic/2254/having-an-issue-with-nginx-in-front-of-fusionauth