We are on a paid plan but not able to run http within lambdas according to this in-app message. Is there a howto on how to enable this feature?

We are on a paid plan but not able to run http within lambdas according to this in-app message. Is there a howto on how to enable this feature?

Port 2525 worked... for some reason low port numbers cause issues in an AWS ECS Fargate cluster.
Sending via postman POST {{auth_api_endpoint}}/api/email/send/{{auth_email_template_id}} (with valid api key as an Authorization header) with request body
{
"toAddresses": [
{
"address": "nick@co6.com",
"display": "nick test"
}
]
}
here is the response:
400 Bad Request
{
"fieldErrors": {
"": [
{
"code": "[invalidJSON]",
"message": "Invalid JSON in the request body. The property was []. The error was [Possible conversion error]. The detailed exception was [No content to map due to end-of-input\n at [Source: (org.apache.catalina.connector.CoyoteInputStream); line: 1, column: 0]]."
}
]
}
}
and corresponding service logs:
2022-06-03T11:19:36.954+02:00 DEBUG: getProvider() returning jakarta.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle]
2022-06-03T11:19:36.958+02:00 DEBUG SMTP: need username and password for authentication
2022-06-03T11:19:36.958+02:00 DEBUG SMTP: protocolConnect returning false, host=smtp.postmarkapp.com, user=862684e6-27c8-4310-9a27-1b4d6e513a4e, password=<null>
2022-06-03T11:19:36.958+02:00 DEBUG SMTP: useEhlo true, useAuth true
2022-06-03T11:19:36.958+02:00 DEBUG SMTP: trying to connect to host "smtp.postmarkapp.com", port 25, isSSL false
2022-06-03T11:55:06.190+02:00
Copy
2022-06-03 9:55:06.190 AM WARN org.primeframework.mvc.action.DefaultActionMappingWorkflow - The action class [io.fusionauth.app.action.IndexAction] does not have a valid execute method for the HTTP method [POST]
2022-06-03 9:55:06.190 AM WARN org.primeframework.mvc.action.DefaultActionMappingWorkflow - The action class [io.fusionauth.app.action.IndexAction] does not have a valid execute method for the HTTP method [POST]
As i am debugging, i'm finding that certain email template tests will "work" showing the "Email sent" check mark (but same consistent error in service logs with emails not being sent, looks like password is not successfully being pulled from the tenant.data.emailConfiguration.password) and others will surface some FreeMarker exception looking like html template interpolation issues:
Threat Detection email template
{
"renderErrors": {
"html": "freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:\n==> event [in nameless template at line 3, column 6]\n\n----\nTip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use [#if myOptionalVar??]when-present[#else]when-missing[/#if]. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??\n----\n\n----\nFTL stack trace (\"~\" means nesting-related):\n\t- Failed at: #if event.type == \"UserLoginSuspicious\" [in nameless template at line 3, column 1]\n----",
"text": "freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:\n==> event [in nameless template at line 2, column 6]\n\n----\nTip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use [#if myOptionalVar??]when-present[#else]when-missing[/#if]. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??\n----\n\n----\nFTL stack trace (\"~\" means nesting-related):\n\t- Failed at: #if event.type == \"UserLoginSuspicious\" [in nameless template at line 2, column 1]\n----"
}
}
Upon inspection of the tenant's json data in the postgres db:
"emailConfiguration": {
"defaultFromEmail": "account-support@co6.com",
"defaultFromName": "Co6 FusionAuth",
"host": "smtp.postmarkapp.com",
"implicitEmailVerificationAllowed": true,
"password": "<REDACTED_MAIL_CLIENT_PASSWORD",
"port": 25,
"properties": "mail.debug=true",
"security": "TLS",
"unverified": {
"allowEmailChangeWhenGated": false,
"behavior": "Allow"
},
"username": "<REDACTED_MAIL_CLIENT_USERNAME>",
"verificationStrategy": "ClickableLink",
"verifyEmail": false,
"verifyEmailWhenChanged": false
},
Was also able to dig up an Event Log of possible interest when sending the test email:
A FreeMarker exception occurred.
Exception:
FreeMarker template error:
The following has evaluated to null or missing:
==> tenantId [in template "WEB-INF/templates/admin/tenant/edit.ftl" at line 16, column 159]
----
Tip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use [#if myOptionalVar??]when-present[#else]when-missing[/#if]. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??
----
----
FTL stack trace ("~" means nesting-related):
- Failed at: @layout.pageForm action="edit" method... [in template "WEB-INF/templates/admin/tenant/edit.ftl" at line 16, column 5]
~ Reached through: #nested [in template "WEB-INF/templates/_layouts/admin.ftl" in macro "body" at line 371, column 5]
~ Reached through: @layout.body [in template "WEB-INF/templates/admin/tenant/edit.ftl" at line 15, column 3]
~ Reached through: #nested [in template "WEB-INF/templates/_layouts/admin.ftl" in macro "html" at line 15, column 3]
~ Reached through: @layout.html [in template "WEB-INF/templates/admin/tenant/edit.ftl" at line 7, column 1]
----
Java stack trace (for programmers):
----
freemarker.core.InvalidReferenceException: [... Exception message was already printed; see it above ...]
at freemarker.core.InvalidReferenceException.getInstance(InvalidReferenceException.java:134)
at freemarker.core.EvalUtil.coerceModelToTextualCommon(EvalUtil.java:481)
at freemarker.core.EvalUtil.coerceModelToStringOrMarkup(EvalUtil.java:401)
at freemarker.core.EvalUtil.coerceModelToStringOrMarkup(EvalUtil.java:370)
at freemarker.core.DollarVariable.calculateInterpolatedStringOrMarkup(DollarVariable.java:100)
at freemarker.core.StringLiteral._eval(StringLiteral.java:103)
at freemarker.core.Expression.eval(Expression.java:101)
at freemarker.core.Expression.evalAndCoerceToPlainText(Expression.java:117)
at freemarker.core.HashLiteral$SequenceHash.<init>(HashLiteral.java:119)
at freemarker.core.HashLiteral._eval(HashLiteral.java:50)
at freemarker.core.Expression.eval(Expression.java:101)
at freemarker.core.Environment.setMacroContextLocalsFromArguments(Environment.java:1012)
at freemarker.core.Environment.invokeMacroOrFunctionCommonPart(Environment.java:858)
at freemarker.core.Environment.invokeMacro(Environment.java:812)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:84)
at freemarker.core.Environment.visit(Environment.java:370)
at freemarker.core.Environment.invokeNestedContent(Environment.java:620)
at freemarker.core.BodyInstruction.accept(BodyInstruction.java:60)
at freemarker.core.Environment.visit(Environment.java:370)
at freemarker.core.Environment.invokeMacroOrFunctionCommonPart(Environment.java:876)
at freemarker.core.Environment.invokeMacro(Environment.java:812)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:84)
at freemarker.core.Environment.visit(Environment.java:370)
at freemarker.core.Environment.invokeNestedContent(Environment.java:620)
at freemarker.core.BodyInstruction.accept(BodyInstruction.java:60)
at freemarker.core.Environment.visit(Environment.java:370)
at freemarker.core.Environment.invokeMacroOrFunctionCommonPart(Environment.java:876)
at freemarker.core.Environment.invokeMacro(Environment.java:812)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:84)
at freemarker.core.Environment.visit(Environment.java:334)
at freemarker.core.Environment.visit(Environment.java:340)
at freemarker.core.Environment.process(Environment.java:313)
at freemarker.template.Template.process(Template.java:383)
at org.primeframework.mvc.freemarker.DefaultFreeMarkerService.render(DefaultFreeMarkerService.java:76)
at org.primeframework.mvc.action.result.AbstractForwardResult.execute(AbstractForwardResult.java:93)
at org.primeframework.mvc.action.result.DefaultResultInvocationWorkflow.perform(DefaultResultInvocationWorkflow.java:108)
at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
at org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:44)
at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
at org.primeframework.mvc.scope.DefaultScopeStorageWorkflow.perform(DefaultScopeStorageWorkflow.java:62)
at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
at org.primeframework.mvc.workflow.DefaultErrorWorkflow.perform(DefaultErrorWorkflow.java:46)
at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
at io.fusionauth.app.primeframework.FusionAuthMVCWorkflow.perform(FusionAuthMVCWorkflow.java:91)
at org.primeframework.mvc.workflow.DefaultWorkflowChain.continueWorkflow(DefaultWorkflowChain.java:44)
at org.primeframework.mvc.servlet.FilterWorkflowChain.continueWorkflow(FilterWorkflowChain.java:50)
at org.primeframework.mvc.servlet.PrimeFilter.doFilter(PrimeFilter.java:78)
at com.inversoft.maintenance.servlet.MaintenanceModePrimeFilter.doFilter(MaintenanceModePrimeFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.inversoft.servlet.UTF8Filter.doFilter(UTF8Filter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:367)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:639)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:881)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1647)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
FusionAuth v1.36.4
I am trying to hook up an SMTP client to fusionauth via the admin ui.
When attempting to update the SMTP mail client password as described for a Tenant, i check the Change password toggle to reveal the Password field and save the form (posted request form data):
...
tenant.emailConfiguration.username: <REDACTED_MAIL_CLIENT_USERNAME>
__cb_editPasswordOption: useExisting
editPasswordOption: update
tenant.emailConfiguration.password: <REDACTED_MAIL_CLIENT_PASSWORD>
...
however any subsequent Send Test Email action yields a 504 Gateway Timeout in addition to the following form data response:
tenant.emailConfiguration.username: <REDACTED_MAIL_CLIENT_USERNAME>
__cb_editPasswordOption: useExisting
tenant.emailConfiguration.password:
Below app service logs (mail.debug=true) for additional context:
2022-06-02 18:23:43DEBUG SMTP: need username and password for authentication
2022-06-02 18:23:43DEBUG SMTP: protocolConnect returning false, host=smtp.postmarkapp.com, user=<REDACTED_MAIL_CLIENT_USERNAME>, password=<null>
2022-06-02 18:23:43DEBUG SMTP: useEhlo true, useAuth true
2022-06-02 18:23:43DEBUG SMTP: trying to connect to host "smtp.postmarkapp.com", port 25, isSSL false
2022-06-02 18:23:43DEBUG: getProvider() returning jakarta.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle]
2022-06-02 18:23:43DEBUG: Tables of loaded providers
2022-06-02 18:23:43DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPTransport=jakarta.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], com.sun.mail.imap.IMAPSSLStore=jakarta.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], com.sun.mail.pop3.POP3Store=jakarta.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], com.sun.mail.smtp.SMTPSSLTransport=jakarta.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], com.sun.mail.imap.IMAPStore=jakarta.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], com.sun.mail.pop3.POP3SSLStore=jakarta.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}
2022-06-02 18:23:43DEBUG: Providers Listed By Protocol: {imap=jakarta.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], smtp=jakarta.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], pop3=jakarta.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], imaps=jakarta.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], smtps=jakarta.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], pop3s=jakarta.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle]}d7b6237e079f4095b4d65e75229422fb
2022-06-02 18:23:43DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map
2022-06-02 18:23:43DEBUG: successfully loaded resource: /META-INF/javamail.default.providers
2022-06-02 18:23:43DEBUG: Jakarta Mail version 2.0.1
FusionAuth v1.36.4
@dan thanks in part to your past resolution "Hosting on RDS" in January 2022, i was able to resolve this in this way given we are trying to create a self-hosted clustered deployment of fusionauth:
FUSIONAUTH_APP_SILENT_MODE=falsepostgres.sql)DATABASE_ROOT_USERNAME and DATABASE_ROOT_PASSWORD with DATABASE_USERNAME and DATABASE_PASSWORD and connection was made possible 
Not sure if you have a recommendation on how to manage clustered fusion auth app upgrades requiring db migrations in a CI/CD pipeline?
Silent mode tripping up on a supposed recursive role relation
com.inversoft.maintenance.db.DatabaseSilentModeWorkflowTask - ERROR: role "fusionauth" is a member of role "fusionauth"
Not sure if this is a postgresql issue or a fusionauth issue as I am able to connect directly to the postgres instance from a local db client.
Full log and stack trace below for context:
No older events at this moment.
Retry
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
2022-06-01 10:17:04.808 AM INFO org.apache.catalina.core.AprLifecycleListener - The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
2022-06-01 10:17:05.806 AM INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-9011"]
2022-06-01 10:17:05.986 AM INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-127.0.0.1-9012"]
2022-06-01 10:17:05.988 AM INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["https-jsse-nio-9013"]
2022-06-01 10:17:06.667 AM INFO org.apache.coyote.ajp.AjpNioProtocol - Initializing ProtocolHandler ["ajp-nio-127.0.0.1-9019"]
2022-06-01 10:17:06.673 AM INFO org.apache.catalina.startup.Catalina - Initialization processed in 2864 ms
2022-06-01 10:17:06.686 AM INFO org.apache.catalina.core.StandardService - Starting service [Catalina]
2022-06-01 10:17:06.687 AM INFO org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/8.5.77]
2022-06-01 10:17:13.108 AM INFO org.apache.jasper.servlet.TldScanner - At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2022-06-01 10:17:14.261 AM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2022-06-01 10:17:14.282 AM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://10.0.1.239:9011]
2022-06-01 10:17:14.284 AM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
- Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true]
- Overriding default value of property [database.mysql.connector-version] with value [8.0.21]
- Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development]
- Overriding default value of property [SEARCH_TYPE] with value [database]
2022-06-01 10:17:14.464 AM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2022-06-01 10:17:14.465 AM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://10.0.1.239:9011]
2022-06-01 10:17:14.466 AM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
- Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true]
- Overriding default value of property [database.mysql.connector-version] with value [8.0.21]
- Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development]
- Overriding default value of property [SEARCH_TYPE] with value [database]
2022-06-01 10:17:14.682 AM INFO com.inversoft.maintenance.DefaultMaintenanceModeWorkflow -
---------------------------------------------------------------------------------------------------------
---------------------------------- Entering Silent Configuration Mode -----------------------------------
---------------------------------------------------------------------------------------------------------
2022-06-01 10:17:15.290 AM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [6BjtDA] Attempting to lock database to prevent multi-node issues
2022-06-01 10:17:15.306 AM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [6BjtDA] Obtained a database lock
2022-06-01 10:17:15.779 AM ERROR com.inversoft.maintenance.db.DatabaseSilentModeWorkflowTask - ERROR: role "fusionauth" is a member of role "fusionauth"
2022-06-01 10:17:15.780 AM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [6BjtDA] Attempting to unlock database to prevent multi-node issues
2022-06-01 10:17:15.782 AM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [6BjtDA] Unlock completed
2022-06-01 10:17:15.784 AM INFO org.primeframework.mvc.servlet.PrimeServletContextListener - Initializing Prime
2022-06-01 10:17:17.005 AM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties]
2022-06-01 10:17:17.006 AM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Dynamically set property [fusionauth-app.url] set to [http://10.0.1.239:9011]
2022-06-01 10:17:17.007 AM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration -
- Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true]
- Overriding default value of property [database.mysql.connector-version] with value [8.0.21]
- Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development]
- Overriding default value of property [SEARCH_TYPE] with value [database]
2022-06-01 10:17:17.009 AM INFO com.inversoft.maintenance.MaintenanceModePoller - Poller started to Wait for configuration to be completed.
2022-06-01 10:17:17.070 AM INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-9011"]
2022-06-01 10:17:17.091 AM INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-127.0.0.1-9012"]
2022-06-01 10:17:17.103 AM INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["https-jsse-nio-9013"]
2022-06-01 10:17:17.107 AM INFO org.apache.coyote.ajp.AjpNioProtocol - Starting ProtocolHandler ["ajp-nio-127.0.0.1-9019"]
2022-06-01 10:17:17.164 AM INFO org.apache.catalina.startup.Catalina - Server startup in 10490 ms