An update.
So, I tried adding another IdP. This time with MS/Azure AD (using the tutorial https://fusionauth.io/docs/v1/tech/identity-providers/openid-connect/azure-ad). While going through the process, it seems that the port number was also added this redirect_uri here. So my guess is, it's hardcoded somewhere for the IdP stuff, and get inserted as part of the redirect without checking the domain/port FA is currenlty being run on.