@dan
Thanx again,

The callback fails only when the user first login with google, later it works. Like my app therefore is receiving the redirect

Edit:
Also, the regular Username/Password login is working fine, so the AppAuth-Android catching redirect seems not to be the issue, i guess.

Hello Dan!

Aah! that should work, with still having the /signin-back button to take me to the app.

Thanx

@robotdan Thank you very much for your reply... Well, this is 1 time but although Please let me know where to purchase for the support and a direct link to the suited package shall be appreciated.

Moreover,

1. Why is just rendering the SSO page taking so long?, Password hashing is far story...
2. We have completely reduced crypto to Factor=2 with SHA-256 but still, an 8core CPU is reaching 100% for about 25-30 TPUs
3. We are trying the "creating nodes" way.
4. Also locally trying to profile FusionAuth Process Stack.
5. Also, please favour us by telling on a High level / Approximation if there is nothing running on the VM and it's only to load FusionAuth SSO, what should be the best Performance expected. I agree, there must be some configs, threads, workers into the equation. But If you were to optimize all those, what would you achieve on an approximation. This will help us understand if its the Limitation by the server resources (CPU/RAM/NODES) or its simply some misconfiguration somewhere.

Oh! Yes, we have tried that much earlier, sorry didn't tell you... we are out of Memory Heap... Right now its taking 100% CPU.

Java Process consumes 100% CPU when more than around 1000 users tries to login in at a time
Its

ONLY a max of 20 clients per second and 1000 users over a min with an average response time of 10secs!

We have checked all possible things what we could hunt over the internet. But with our limited knowledge, we are unable to solve this.

Hello @dan !

Any update on the Load test? Can you suggest what should be our server config & resources to handle 7000+ login requests at a time? We are still facing this issue? The fusionauth-app (https://login.gurukul.org/oauth2/authorize) Java Process consumes 100% CPU when more than around 1000 users tries to login in at a time.

Also, as we never had and are not having much regular concerns, we have not opted for support package. This issue if solved will the max we would need support actually.

Thank you...

@dan Surely... I shall try and revert.

However, Can I know one thing? What performance should we be expecting with this VM, like can you just give me approx. range? Or what best have you achieved as test load, how does that translate to our machine?

Thank you for your response, We could tryout something but its not improving performance

Somethings you Asked

• Fusionauth version 1.17.0
• Actions:
  • We are just using FA for Login & token exchange tasks,
  • However the during load test we only used SSO.
  • SSO: Only loading login page, Get request oauth2/authorize?client_id=
    {}&response_type=code&redirect_uri=%2Flogin&state={}
  • We combined API: /oauth2/login for 1 load test, (later I have mentioned when).
• Database is running under the same machine.
• Yes we are using elastic search.
• We observed
  • The process usage of Mysql it was using average 5% of CPU
  • Disk IOPS was around 70 (plenty of room was available for more IOs),
  • apache was also stable around 2-3% of CPU usage. During that time it was quickly serving other requests.
  • There were no errors in apache.

We found error in fusionauth

java.lang.OutOfMemoryError: Java heap space
at java.base/java.lang.reflect.Method.copy(Method.java:158)
at java.base/java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:102)
at java.base/jdk.internal.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:308)
at java.base/java.lang.Class.getDeclaredMethod(Class.java:2555)
at com.google.inject.internal.cglib.proxy.$Enhancer.getCallbacksSetter(Enhancer.java:809)
at com.google.inject.internal.cglib.proxy.$Enhancer.setCallbacksHelper(Enhancer.java:797)
at com.google.inject.internal.cglib.proxy.$Enhancer.setThreadCallbacks(Enhancer.java:791)
at com.google.inject.internal.cglib.proxy.$Enhancer.registerCallbacks(Enhancer.java:760)
at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:269)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)

• Thus we increased the memory footprint to 2GiB at fusionauth config file (fusionauth-app.memory property).
  • We ran the test again.
  • It was able to serve 3000 requests in a minute.
  • Test report https://bit.ly/3icn8rW.
  • But the CPU usage was more than 50% for fusionauth java process & Like 7-11% for FA Elastic Search.

Loadtest?

• Simply loading a GET SSO request with a login page was consuming half of the CPUs i.e. 4 cores without even login.
• With Login API combine, 1500 login across 1 min what we are able to serve with >50% cpu usage. https://bit.ly/35jESOp
• Going beyond that response time was increasing. https://bit.ly/3m8Nhum
• During the test we also reduced the PBKDF2 factor to 4000 from 20000. So that crypto is out of equation for bottleneck. We also tried with SHA256 with only 2 factor, confirming the crypto processing. The results were the same.
• And there are peak times where we expect such load. During all the above tests db and apache were stable.

Is this the performance we can expect from fusionauth with our CPU and memory config? Turning around 40TPS at max? Certainly there is something we are missing here. Please guide us.

Hoping...

Jay Swaminarayan!
Dear @dan , It's always great to have you as a silver lining.

Slightly long story as I am narrating over a week's working on performance. We are from Shree Swaminarayan Gurukul Organization (https://gurukul.org) a non-profit school-chain in India, moving almost whole schooling online during this pendamic situation. However, the following is our usecase & looking forward to know what are we missing in optimizing the performance.

Server Config

• Azure VM: Ubuntu Linux 18.04.4
• Linux 5.4.0-1023-azure on x86_64
• CPU: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz, 8 cores
• Memory: 31.36 GiB total
• Storage: 156.92 GiB total (SSD)
• Running 1 Wordpress site, multiple angular sites, .netcore small services
• FA installed on :9011 and reverse proxied from Apache2 Host

Use Case & Timeline:

• Intent: Scheduled online tests from 7th Sep (today) to 14th Sep, planned a month back as students are not coming to school due to CORONA.
• Users: Serve logins to 4000 - 5000 students for an external app (android app, backend not on the same server) and 7000 is the max users we have for this case
• Launch: 1st Sep - App Integration finishes to handle login flow with CODE Grant & Offline Access, using AppAuth-Android
• Onboarding Students on App: 1st & 2nd Sep - Distribution of Credentials and Tutorials for Students
• Trail#1: 3rd Sep - Planned Trail Test #1 for Testing Load
• Optimization#1: Resized Server from 2 Cores + 16gb to 4 Cores + 16gb
• Trail#2: 5th Sep - Planned Trail Test #2 for Testing Load
• Optimization#2: Optimized mpm_event apache2
• Real Exam: 7th Sep - Real Exam Day, still halted.
• Optimization#3: 7th Sep noon - Resized Server from 4 Cores + 16gb to 8 Cores + 32gb
• Optimization#4: 7th Sep mid day - PBKDF2 (f=24000) to SHA-256 (f=2)
• Seeking Support: Writing this topic for help.

Happenings:

1. During Launch itself, the server was overloaded with requests, and FA-SSO was not served to almost all users
2. We were almost not sure what went wrong and just rebooted the VM, simultaneously students waiting even up to 30 mins for the server to load dropped the hope for the day
3. Thus, we initially thought there must have been some blocking process and REBOOT would do each time there is a load.
4. On 3rd Sep trial #1, once again there was a flood, this day we already rebooted the machine before the test time.
5. However, this did not solve the problem and we resized VM, Optimization #1
6. We were anticipating, we are well to go for 5th Sep, but same issues, the request flooded, it is then we actually started researching on the matrices and with some local help found mpm_event is the culprit and had insufficient server limits set. After which we were almost confident about the load and declared that the issue is resolved completely to all school principals & students.
7. We were happy that this Trail-strategy actually benefited us to avoid any issues on the real day, i.e. today. Also, we checked out https://fusionauth.io/blog/2019/02/26/got-users-100-million and were assured about Good to Go on 7th, today.
8. Today, to our surprise yet again we faced the same issues, and students were not served FA-SSO. As this was happening, we firstly resized the VM again (Optimization #2), Parellaly we started digging deep into issues, FA was not actually in the our hit-list.
9. Still had the same issue. We observed load on Java & learned hashing could be a bottleneck thus changed PBKDF2 (f=24000) could be bottleneck and changed it to SHA-256 (f=2) with rehashing on login.
10. And also we quickly decided to divide users into different time slots to manage the load, and somehow finished the day, with many students failed to attempt, we will see that later.
11. Taking things really seriously, we started checking each process and to our surprise we found FA (JAVA) loading the VM even for small users, at least far smaller than 10M that too, time slotted.

Server Footprints:
1.

Here is the load test we tried later after applying all optimizations that we could think of Server Upgrade CPU + RAM, least cryptography SHA-256 (factor = 2), this is the footprint for 2000 clients load distributed across 1 min, you can see server HALTS after 15 secs.
https://bit.ly/33aarYk

Here is the load test of login API load for 3500 clients load distributed across 1 min, taking whooping 14 secs to serve.
https://bit.ly/2GwRbN2

End of the Story,
NEED HELP!

PS: For tomorrow we have divided the students in slots even till the evening. However, day after tomorrow we intend to take the test at a time.

Thanking you and hoping for the best.

Hello Dan!

Aah! that should work, with still having the /signin-back button to take me to the app.

Thanx

@dan: "...call each of your configured logout URLs per application."

• Do we have many logout URLs?

As you suggested, /api/logout removes cookies, what should I do to remove the SSO Session from the Native App itself?

Basically, from the Android App, when we call /api/logout & remove any locally saved state info, and then redirect to the Login page, as the FusionAuth session is still active, it just redirects back to the Consent screen and therefore back to the Application. Literally making the user not able to switch the accounts, until the fusionauth session is terminated.

If /api/logout doesn't end the fusionauth session, I am left with only 1 option (with as much I've understood)

• Logout Action in the app.
• Redirects to FusionAuth SSO Page
• Redirects back to the Custom Consent Screen (as FA session is active)
• I add a "LOGOUT COMPLETELY" button redirecting to /oauth2/logout page with post_logout_uri to /signin-back page (containing a single button "Sign In Again")
• FA Logout Progress
• Redirects to /signin-back
• Signin Back Button action to APP
• Finally APP opens, checks AuthState to be false
• Authroize() redirects to /oauth2/login

But I sincerely feel, this is too much, am I mistaken somewhere? There must be some best practices, which I want to know.

Thank you

Jay Swaminarayan! @dan

Something that I have learnt a hard way, after a week long troubleshooting was surprising silly, at least for you or other experienced members of fusionauth community.

However, little I knew the security concerns of the browser, It turns out that,

• A javascript cannot directly redirect to any com.android.app:/redirect_uri.

• There must be a User Manual Interactivity for the Redirect to complete, A Button Click or Ancher Link Click

• That was the Reason Chrome Cancelled the redirect from FusionAuth Screen

What I could find,

One of the reasons, that there are consent screens,

Basically Allow Button, redirect back to the app.

My conclusion:

• It would be Great if this Consent Screen Mechanism is available for Native Redirects after 1st Authorization within FusionAuth.
• Time-being I have made a Consent Screen of my own, which is the redirect_uri and than in turn that redirects back to the APP.

I don't know, if what I am doing is best practice or was there something else I should have done, at least this is working for me as now.

Thank you once again.

PS: There is another issue, calling /api/logout?global=true&refreshToken={refresh_token} only signs out of the app, but doesn't signout from the FusionAuth completely, making it redirecting back to the App instead of the Login Screen?