Posts made by maciej.wisniowski

@pacheco-eaguiar maybe you can use the Login API in the backend to verify user's credentials: https://fusionauth.io/docs/apis/login

Hi

We have an ongoing PCI DSS certification of our system and Qualys scanner reports issue with Apache Struts2 on the (self-hosted) FusionAuth 1.54.0 instance. I think it is a false positive but anyway, they want me to provide them with the Apache Struts version in use. So my question is if FusionAuth uses Apache Struts2 at all and if so, which version is this?

Issue details:

Apache Struts2 Multiple Vulnerabilities (S2-008).

Scanned URL:

GET /index.action?debug=command&expression=%23_memberAccess["allowStaticMethodAccess"]=true,@java.lang.Runtime@getRuntime().exec('0jWw997Z') HTTP/1.1

Validation logic:

QID Detection Logic (Unauthenticated): This QID sends specifically crafted payload with a random string command in the request to check for command execution in .action files. Vulnerable targets are expected to return string "null" in the respond.

As seen in the scanner logs, FusionAuth returns the login page for the above URL, with the JavaScript code containing 'null' text - which seems to trigger the false positive:

Prime.Document.query('.alert').each(function(e) {
var dismissButton = e.queryFirst('a.dismiss-button');
if (dismissButton !== null) {
new Prime.Widgets.Dismissable(e, dismissButton).initialize();
}

@duke have you tried the following (move username and password to another env variables):

DATABASE_URL=postgres://top2.nearest.of.host_name-db.internal:5432/db_name
DATABASE_USERNAME=the_name
DATABASE_PASSWORD=LtMvptwX

@joshua I've tried with the FA 1.32.1 and I think I've found the source of the problem.
Turns out I had my Forgot Password Email template a bit old (AFAIR based on the template from 1.27.2).

Because of this, the URL to reset the password (in the email template) was generated like:

https://#{FA_DOMAIN}/password/change/${changePasswordId}?tenantId=${user.tenantId}

In the newer versions of FA it was changed to:

[#assign url = "https://#{FA_DOMAIN}/password/change/${changePasswordId}?client_id=${(application.oauthConfiguration.clientId)!''}&tenantId=${user.tenantId}" /]
[#list state!{} as key, value][#if key != "tenantId" && key != "client_id" && value??][#assign url = url + "&" + key?url + "=" + value?url/][/#if][/#list]
${url}

The point is that with the new version we have extra parameters in the URL like client_id, redirect_uri and a few others. Everything works properly with the new syntax. If these new parameters are missing (I suppose the main issue is lack of the client_id) FA triggers error 500 after password reset (for the users that are not registered in the application that generated the password reset e-mail).

The most recent version of FA triggers a more detailed error message in the log than the previous one and this helped me a bit to track the problem:

2022-01-11 4:19:10.373 PM ERROR io.fusionauth.app.primeframework.error.ExceptionExceptionHandler - An unhandled exception was thrown
java.lang.NullPointerException: Cannot read field "oauthConfiguration" because "this.application" is null
   at io.fusionauth.app.action.oauth2.BaseOAuthAction.handleInteractiveLoginResponse(BaseOAuthAction.java:548)
   at io.fusionauth.app.action.oauth2.BaseOAuthAction.callLogin(BaseOAuthAction.java:447)
   at io.fusionauth.app.action.password.ChangeAction.post(ChangeAction.java:121)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   at org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:414)
   at org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:79)
   at org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:62)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:47)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:60)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:50)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:52)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:57)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:102)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:58)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:44)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:126)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.workflow.StaticResourceWorkflow.perform(StaticResourceWorkflow.java:97)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.parameter.RequestBodyWorkflow.perform(RequestBodyWorkflow.java:91)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:64)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at io.fusionauth.app.primeframework.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:51)
   at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
   at io.fusionauth.app.primeframework.FusionAuthMVCWorkflow.perform(FusionAuthMVCWorkflow.java:86)
   at org.primeframework.mvc.workflow.DefaultWorkflowChain.continueWorkflow(DefaultWorkflowChain.java:44)
   at org.primeframework.mvc.servlet.FilterWorkflowChain.continueWorkflow(FilterWorkflowChain.java:50)
   at org.primeframework.mvc.servlet.PrimeFilter.doFilter(PrimeFilter.java:78)
   at com.inversoft.maintenance.servlet.MaintenanceModePrimeFilter.doFilter(MaintenanceModePrimeFilter.java:63)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at com.inversoft.servlet.UTF8Filter.doFilter(UTF8Filter.java:27)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:196)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:624)
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1650)
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
   at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   at java.base/java.lang.Thread.run(Thread.java:833)

I'm not sure if this can be considered a bug in FA? What do you think?

BTW. Is this normal that if the kickstart file is used then no default email templates are generated at all?

@minhngocnguyenduy maybe using lambdas can help?

@joshua yes, I still have this issue on a few instances of FA while trying to change password for the user that is not registered in the specific application. If you're not able to reproduce this I can try to create a kickstart script to recreate the problem

@developers in your case the error message is pretty self-explanatory. The problem is just a lack of [[singleCase]user.password] message in your theme. Just add this message to your theme and it should work

@joshua any chance you had some time to look at this? Is this reproducible on your end?

@engineering-0 Try this:

users = []
for user in User.objects.all():
  user_data = {}
  ...
  encryption_scheme = "salted-pbkdf2-hmac-sha256"
  algorithm, iterations, salt, password_hash = user.password.split('$')
  salt = base64.b64encode(salt.encode('utf-8')).decode('utf-8')

  user_data['password'] = password_hash
  user_data['encryptionScheme'] = encryption_scheme
  user_data['factor'] = int(iterations)
  user_data['salt'] = salt
  users.append(user_data)

@psmiddy I've never heard about vitess but seems that your DATABASE_URL should point to vitess-test-8_0 not to localhost. Also, I think the port number can be just 33807 as it should be available internally in docker (ports: directive just publishes specific ports to localhost, eg. if you need to connect to docker from your desktop). Try:

DATABASE_URL: jdbc:mysql://vitess-test-8_0:33807/fusionauth

@fred-fred You can customize the templates (eg macro body in helpers.ftl) in your theme like this:

[#if application?? && application.name?? && application.name == "MyApp1"]
       this is myapp 1 login screen
[#elseif application?? && application.name?? && application.name == "FusionAuth"]
       this is FusionAuth login screen
[#else]
       this is something else
[/#if]

@joshua I think I've found the pattern to reproduce this issue.
Basically, it fails if the user I want to reset the password for is not registered in the application whose client_id is used during password reset flow. It works for the application where the user has a valid registration.

To reproduce:

1. Create new tenant: Tenant1
2. Create two Applications for Tenant1: application1 and application2 using https://example.com as redirect_uri
3. Create user: user1 for Tenant1
4. Create registration for user1 in application1
5. Open the authorize URL for application2 like:
   https://<FUSIONAUTH_URL>/oauth2/authorize?client_id=<CLIENT_ID_OF_APPLICATION2>&scope=openid%20offline_access&response_type=code&redirect_uri=https%3A%2F%2Fexample.com&tenantId=<TENANT_ID>
6. Click Forgot your password
7. Check you e-mail and click the password reset link
8. Try to change the password
   It should fail at this point.

You can now check the same for application1 (with the user registration) to see if it works.

I'm very curious if you can reproduce this too.

@joshua I think that yes, these users were imported and they're using salted-pbkdf2-hmac-sha256, but their imported passwords are useable so I don't think these are hashed incorrectly.

We have themed pages but I've also tried with the "Default" theme and the issue occurred as well. It is worth noting that even though I get an error from FA, the password is changed and the user is logged in automatically after the password change form is submitted. Seems that the password change procedure works but the problem is a bit later.

Using the API it works properly:

In [42]: res = client.forgot_password({"loginId": 'someuser@somedomain.com'})
In [43]: change_password_id = res.success_response["changePasswordId"]
In [44]: res2 = client.change_password(change_password_id, {'password': 'somepass2'})
In [45]: res2.was_successful()
Out[45]: True

Sorry for missing these in my previous post.

The issue occurs in both environments: local is a single docker instance and the server environment uses two FA nodes.

Database is PostgreSQL (9.6 in docker and 11 on the server - managed Postgres on DigitalOcean) + Elasticsearch.

No MFA is in use. Users were created long time ago. As I said previously, it doesn't happen all the time, even for the same users. For example, I tried to change the password to 'password123' and it was impossible but when I tried another password like 'someotherpass.123' it worked. Then, trying to change the password again to password123 fails. In the same time, I was able to use 'password123' to reset password for some other user. (note: sample password provided above are not real ones I've used).

@dan unfortunately it seems that I've encountered same, or similar error with the latest FA 1.30.2.
This happened to us on two separate FA instances (self hosted).

How to reproduce the issue is not very clear to me because it doesn't appear all the time but what I did is:

1. Start the password reset flow (eg. clicking the Forgot password link on the login screen)
2. Open the password change form using the link from the e-mail
3. Enter a new password and submit the form
4. The error appears

As I said, it doesn't happen all the time but I have a feeling it might be somehow related to using the same password as it was before the password reset.

I tried:

• disabling Reactor (one of the affected FA instances had it enabled)
• disabling login.successful webhook we're using
• disabling custom theme we have

Nothing helped

The FusionAuth log shows the following error:

2021-11-05 1:36:38.364 PM ERROR io.fusionauth.app.primeframework.error.ExceptionExceptionHandler - An unhandled exception was thrown
java.lang.NullPointerException: null
	at io.fusionauth.app.action.oauth2.BaseOAuthAction.handleInteractiveLoginResponse(BaseOAuthAction.java:545)
	at io.fusionauth.app.action.oauth2.BaseOAuthAction.callLogin(BaseOAuthAction.java:446)
	at io.fusionauth.app.action.password.ChangeAction.post(ChangeAction.java:117)
	at jdk.internal.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:414)
	at org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:79)
	at org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:62)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:47)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:60)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:50)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:52)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:57)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:102)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:58)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:44)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:126)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.workflow.StaticResourceWorkflow.perform(StaticResourceWorkflow.java:97)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.parameter.RequestBodyWorkflow.perform(RequestBodyWorkflow.java:91)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:64)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at io.fusionauth.app.primeframework.CORSFilter.doFilter(CORSFilter.java:237)
	at io.fusionauth.app.primeframework.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:49)
	at org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:51)
	at io.fusionauth.app.primeframework.FusionAuthMVCWorkflow.perform(FusionAuthMVCWorkflow.java:86)
	at org.primeframework.mvc.workflow.DefaultWorkflowChain.continueWorkflow(DefaultWorkflowChain.java:44)
	at org.primeframework.mvc.servlet.FilterWorkflowChain.continueWorkflow(FilterWorkflowChain.java:50)
	at org.primeframework.mvc.servlet.PrimeFilter.doFilter(PrimeFilter.java:78)
	at com.inversoft.maintenance.servlet.MaintenanceModePrimeFilter.doFilter(MaintenanceModePrimeFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.inversoft.servlet.UTF8Filter.doFilter(UTF8Filter.java:27)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:196)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:624)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1650)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:832)

The desired end state for me would be any of these you've brought up but I was wondering about the implementation of the first option (User would be logged out of App X If they're logged in to FA with a different set of credentials).
The possible solution that I was thinking of is sending a request to FA from App X using Ajax or iframe, that will tell me if the user is logged in to FA SSO and what is his current id. This way I'd be able to verify if the current App X user is the same as the FA SSO user and log him out if necessary. Unfortunately, there is no such endpoint in FA and it seems to be a bit overcomplicated so that's why I've asked about the other options.
Of course, another way to deal with it is to leave this as is It is rather not a very common case, but still... it is possible to happen and this poses a security risk of having one user accessing data of another one.

Regarding the session times, even if I have an FA SSO session valid for 12 months and App X session valid for 1 month only it will not solve the issue.
Imagine that after 11 months and 10 days since the FA SSO session was created, the user visits the App X (for the first time in a few months). This will create a brand new session for App X that will be valid for one month (so in this specific case it will last longer than SSO session that expires in next 20 days).

For local development, we're using FusionAuth's kickstart files so that every change is documented in kickstart.json (and templates if any).

Unfortunately, it is not so easy to apply such changes as the kickstart file is only executed on a clean FA instance.
This means that in order to update the local environment with the new kickstart it is necessary to remove the FA config and database and then start it again having new kickstart applied. This will remove all local users so you also need some way to have them recreated.

Recent versions of FA have a Reset option in the management panel but it is in early state and very limited (seems not to support any extra template files referenced from the kickstart.json file), but might be it will work for you.

Another option you might consider is using Terraform provider for FusionAuth. This can be used for production environments.

Regarding the use case - we're testing different scenarios and this just appears to be possible to log as a different user in this way.

I can imagine the following real use cases where two or more user credentials sets are in use:

• single computer but different users (eg. people working on different shifts or public computers used for training/presentations etc.)
• a testing environment where the tester keeps logging in/out as different users

Syncing the session times doesn't seem to be a solution because the application-specific sessions are created when the user accesses these applications. So, unless you log in to all the applications at (nearly) the same time you will not have the session times in sync.

If I have a FA and 2 applications, say X and Y, using SSO, then it is possible that:

1. User logs In to application X as user1 (SSO session is created in FA and session is created in application X)
2. SSO session expires in FA (but still persists in application X for user1, so the user is still logged in as user1)
3. User tries to access application Y and logs in as user2 (different credentials than before) - SSO session in FA is created and the session for user2 is created in application Y

As a result:
The user is logged in to application X as user1 and to application Y (and FA) as user Y.

This is obviously not a desired situation so do you have any clues on how to deal with that?