Migration

Evaluating FusionAuth

If you haven’t already done so, ensure FusionAuth will work with your application or applications. You can install it in about five minutes and built out a proof of concept (POC).

A POC is helpful in determining which login method you should use and how to theme the hosted login pages to maintain your application’s look and feel.

FusionAuth assigns users roles. A user’s roles are available in API responses and in the JWT (JSON Web Token) sent to client applications after successful user authentication. You may need to update your application to look at the roles claim to allow or disallow functionality within an application.

You may choose to use a language specific library to interface with FusionAuth’s standards compliant SAML, OAuth and OIDC endpoints. There are sample applications you can review to see examples of such integrations. You may also choose to use one of FusionAuth’s client libraries.

If you allow users to register with your application, modify your application to point to FusionAuth’s registration form and make sure you’re capturing the registration data you need.

If you want social sign-on, such as Google, or enterprise identity provider integration, such as SAML, configure and enable those providers as well.

Testing FusionAuth’s ability to integrate with your identity provides and existing applications before diving into the migration planning will ensure that when the time comes to cut over to FusionAuth, there won’t be any unpleasant surprises.

Next, let’s talk about migration planning.

An Example of Data Mapping

Let’s examine a data mapping example. Suppose an old auth system has the following user data model (let’s ignore email and password fields, as they won’t be necessarily be mapped):

• fname - string

• lname - string

• datebirth - string

• phone_num - string

• role - string

FusionAuth has a user object with these attributes and data types:

• firstName - string

• lastName - string

• birthDate - An ISO-8601 formatted date string

• mobilePhone - string

There are a number of mappings required.

The first is converting from fname to first_name and lname to last_name, which might seem trivial. It is, but you need to make sure you handle any field which needs to be renamed. Spreadsheets are your friend.

The second mapping task is parsing the datebirth field into an ISO-8601 formatted string, to be placed in the birthDate field. Depending on how clean the original data is, this could be simple or it could be painful. If the latter, use a date parsing library; it’ll handle edge cases.

Then, consider how to handle the phone_num field. If you know that every number in your original datastore is a mobile number, you could use the mobilePhone FusionAuth field. You could also store it in user.data, in the user.data.phoneNumber field, for example.

Finally, you need to map role values. Roles in FusionAuth are stored on the registration object, because they are associated with applications and users. When you are creating a user, you can create a registration at the same time, and then associate any roles for that application with that registration.

In general, any data associated with a user but which doesn’t change between different applications should be stored on the user object. Examples include a user’s name and their phone number. Here’s a sample FusionAuth User object:

{
  "user": {
    "active": true,
    "birthDate": "1976-05-30",
    "data": {
      "displayName": "Johnny Boy",
      "favoriteColors": [
        "Red",
        "Blue"
      ]
    },
    "email": "example@fusionauth.io",
    "encryptionScheme": "salted-sha256",
    "factor": 24000,
    "expiry": 1571786483322,
    "firstName": "John",
    "fullName": "John Doe",
    "imageUrl": "http://65.media.tumblr.com/tumblr_l7dbl0MHbU1qz50x3o1_500.png",
    "lastName": "Doe",
    "middleName": "William",
    "mobilePhone": "303-555-1234",
    "passwordChangeRequired": false,
    "preferredLanguages": [
      "en",
      "fr"
    ],
    "timezone": "America/Denver",
    "twoFactorEnabled": false,
    "usernameStatus": "ACTIVE",
    "username": "johnny123"
  }
}

Please consult the User API for detailed field descriptions.

Any data associated with a user’s use of an application, on the other hand, should be stored on the registration object. Examples include their roles or application specific profile data. Here’s a sample FusionAuth registration object:

{
  "registration": {
    "applicationId": "10000000-0000-0002-0000-000000000001",
    "data": {
      "displayName": "Johnny",
      "favoriteSports": [
        "Football",
        "Basketball"
      ]
    },
    "id": "00000000-0000-0002-0000-000000000000",
    "preferredLanguages": [
      "en",
      "fr"
    ],
    "roles": [
      "user",
      "community_helper"
    ],
    "timezone": "America/Chicago",
    "username": "johnny123"
  }
}

Similarly, consult the Registration API for full documentation of this object.

As this example shows, getting ready for a migration consists of many choices and design decisions. Understanding your user account data model and how it maps to FusionAuth’s before you write any code will prevent unpleasant surprises.

Configure FusionAuth

Prepare FusionAuth for your users and applications; while the exact configuration depends on your application needs, the following items should be considered.

Create one or more tenants. Multiple tenants are useful for allowing someone to have the same email but different passwords, or allowing different settings such as the theme or password rules. Add these via the API or navigating to Tenants and clicking the green plus sign.

Create one or many applications and add any roles needed for them. Create a FusionAuth application entity for each application whose users you are migrating. An application is anything a user can log in to, whether an API, a commercial product, or a custom web application. Add each of these via the API or navigating to Applications in the administrative user interface, then clicking the green plus sign to add the application.

Map user attributes, as discussed above, into the FusionAuth user or registration objects. If some of your data doesn’t fit into the FusionAuth model, add it to the appropriate data field.

All of this configuration can be done via the API or the administrative user interface. If you want to use the former, you’ll have to create an API key with appropriate permissions.

Now that FusionAuth is up and running, proceed to either the Big Bang Implementation, the Segment By Segment Implementation or the Slow Migration Implementation section.

Performance

Because you have downtime with this approach, you’re going to want to import users quickly. Tweak these FusionAuth settings and perform the following tasks to do so.

Switch to the database search engine. If you do so, fusionAuth won’t have to sync user data to Elasticsearch during the import. If you require Elasticsearch for advanced searching capabilities, you can switch back to Elasticsearch after the migration is complete.

Disable the user.bulk.create webhook unless you need FusionAuth to send an event with all the created users to another system.

Set the HTTP timeout to a large value on your API requests. Exactly how to do this varies based on the tool you’re using to make the HTTP request. The import API is currently a synchronous operation, though there are plans to make it asynchronous (see this GitHub issue for more).

If you only provide a password field, then FusionAuth will assume the password is in plaintext and hash it for you. This hashly negatively affects load time, performance and throughput. If you provide the salt, password, encryptionScheme and factor values when importing, then FusionAuth assumes the value in the password field is a hashed password, and it will not be hashed.

Deduplicate any emails. In FusionAuth, each email address may be associated with only one user account per tenant.

Stage your data by exporting current user data into JSON files. This will make debugging easier, since you can load one file at a time, and you can repeat a data load if there are issues. It will also be more performant than loading data across a network or from a database.

Building the Migration Scripts

To actually move the data, you’ll build out a series of scripts and programs. To begin this process, stand up a FusionAuth instance for testing. To start your FusionAuth instance in a known state every time, you may want to configure Kickstart. A Kickstart file can serve as a foundation for developers and CI processes in the future as well.

You’ll also want to create an API key. Make sure you give the key appropriate permissions. The minimum required are the POST method on the /api/user/import endpoint.

You can write the migration scripts in shell, any of the supported client library languages, or against the REST API in any language supporting HTTP requests. Iterate over all the users in the old system or systems. Build the JSON files. Add a registration for each application to which a user should have access.

If you can’t build JSON files on the filesystem for some reason, you may build the JSON in memory. This can be a good approach if you are dynamically merging two data sources, but will be tougher to troubleshoot.

Finally, import the JSON using the importUsers method of a FusionAuth client library or by calling the REST API directly. This is /docs/v1/tech/apis/users/#import-users[fully documented].

Here’s an example of an import API JSON request body:

{
  "users": [
    {
      "active": true,
      "birthDate": "1976-05-30",
      "data": {
        "displayName": "Johnny Boy",
        "favoriteColors": [
          "Red",
          "Blue"
        ]
      },
      "email": "example@fusionauth.io",
      "encryptionScheme": "salted-sha256",
      "expiry": 1571786483322,
      "factor": 24000,
      "firstName": "John",
      "fullName": "John Doe",
      "imageUrl": "http://65.media.tumblr.com/tumblr_l7dbl0MHbU1qz50x3o1_500.png",
      "insertInstant": 1331449200000,
      "lastName": "Doe",
      "memberships": [{
        "data": {
          "externalId": "cc6714c6-286c-411c-a6bc-ee413cda1dbc"
        },
        "groupId": "2cb5c83f-53ff-4d16-88bd-c5e3802111a5"
      }],
      "middleName": "William",
      "mobilePhone": "303-555-1234",
      "password": "5ac152b6f8bdb8bb12959548d542cb237c4a730064bf88bbb8dd6e204912baad",
      "passwordChangeRequired": false,
      "preferredLanguages": [
        "en",
        "fr"
      ],
      "registrations": [
        {
          "applicationId": "00000000-0000-0000-0000-000000000002",
          "data": {
            "birthplace": "Bremen"
          },
          "insertInstant": 1331449200000,
          "preferredLanguages": [
            "de"
          ],
          "roles": [
            "moderator"
          ],
          "username": "Mausebär",
          "verified": true
        }
      ],
      "salt": "NDdiYWZkZDMtYjk5ZC00ZmZkLWE1YmUtZTQxNGM4MDkwNWYw",
      "timezone": "America/Denver",
      "twoFactorEnabled": false,
      "usernameStatus": "ACTIVE",
      "username": "johnny123",
      "verified": true
    }
  ]
}

This JSON imports one user, but you can add multiple user objects to the users array to import multiple users.

Below is an example of a curl script to import JSON files. It times out in 10 minutes, iterates over files in a directory, and stops processing if it receives any non-200 status code. That would indicate there was an issue importing the users.

#!/bin/sh

API_KEY=...
JSON_FILE_DIR=...
FA_HOST=...

for file in $JSON_FILE_DIR/*.json; do
  echo "Processing $file";
  RES=`curl --max-time 600 \
       -s -w "%{http_code}" \
       -H "Authorization: $API_KEY" \
       -H "Content-type: application/json" \
       -XPOST \
       $FA_HOST/api/user/import \
       -d@$file`
  if [ "$RES" -ne "200" ]; then
    echo "Error: $RES";
    exit 1;
  fi
done

Consult the documentation for this API for more information.

If the original system hashes passwords using an algorithm other than those schemes FusionAuth supports, write and install a custom password hashing plugin. In either case, specify the encryption scheme in the user import JSON file.

In FusionAuth, duplicate emails are not allowed within the same tenant. If you may have duplicate emails, de-duplicate them before importing. If you don’t want to do so, set the validateDbConstraints property to true in the import JSON. When this is done, the import API will return a user friendly error message when duplicate addresses are found.

{
  "fieldErrors": {
    "user.email": [{
      "code": "[duplicate]user.email",
      "message": "A User with email [example@piedpiper.com] already exists."
    }]
  }
}

{
  "generalErrors": [{
    "code": "[ImportRequestFailed]",
    "message": "An error occurred during the import request. This is most likely due to a unique key constraint which would indicate one or more of the users in the import request already exist in FusionAuth. Re-attempt the request with additional validation by using the [validateDbConstraints] property. If you have already enabled the additional validation and you still receive this error, please open a bug report."
  }]
}

The extra validation comes at a performance cost, however, so you may want to run your import with validateDbConstraints equal to true to find the duplicate email addresses and remove or remediate them. After that, you can run an import with validateDbConstraints equal to false and reap the performance benefits.

Determine if you need to import refresh tokens. If you are using an existing system which presents such tokens to the identity provider when an existing JWT expires, importing refresh tokens ensures that your users enjoy an uninterrupted application experience. Consult the documentation for this API for more information.

Testing

Test this import process with as large of a dataset as possible. If you can, use your entire user dataset. Testing with a realistic sized load lets you know you how long your import will take. Real world data will reveal edge cases, such as duplicate emails or incorrectly formatted user attributes.

Even if you aren’t using multiple tenants in production, during the testing phase it is a good idea to create a Testing tenant and load all your users and applications into this tenant. You can drop a tenant with one API call or one click in the administrative user interface. All the users, applications, groups and settings of that tenant will be removed at that point, making it easy to iterate your import scripts.

However, you cannot drop the Default tenant containing the FusionAuth application. This why you must create a new tenant. Alternatively, you could also drop the entire database and perform a fresh FusionAuth install.

Performing the Migration

When your scripts work in your testbed environment, prepare to do a production migration. Inform all the internal stakeholders. Plan for downtime unless you can run your application with the original user store in a read only mode. How much downtime? You should know based on your testing of the import.

Run the migration on your production dataset, moving the data from the original system to FusionAuth. When the migration is finished, release your application changes. All applications should point to FusionAuth for authentication requests and related user flows, including, but not limited to:

• Log in

• Registration, if applicable

• Forgot password

• Password changes

Should you need to rollback, revert the changes to your application pointing it to FusionAuth. If users have updated profile data in FusionAuth, you’ll need to port those changes back to your legacy system. A script using the user API and searching on users with recent updates will be a good starting point, though the exact data rollback will be application dependent.

Determine Your Finish Line

Unlike a big bang migration, with a slow migration you need to set a migration completion goal. Slow migrations move accounts one user at a time, so it is unlikely you’ll migrate one hundred percent of your users through this approach. Some people log in to your application rarely while others may have abandoned their accounts. No matter how long a migration period you allow, some of your users will not log in during that timeframe, and therefore won’t be migrated.

So, with this approach, you need to decide what "done" means. Some factors to consider:

• How often do people log in?

• Is there a significant long tail of users who visit the application less frequently than the average user?

• Are there external events such as times of the year or holidays when greater or lesser numbers of users engage with your application?

• What are the ramifications of a user being unable to log in? Are there business, compliance, legal, or security concerns?

• Is loss of timely access to your application an annoyance or a disaster?

• You will have some users who are not migrated when the slow migration period is over. How will you handle those accounts?

• How painful is it to operate both FusionAuth and your current authentication system?

• How valuable is a customer who has not logged in to your application in six months? A year? Three years?

Based on these answers, set a goal for a number or proportion of migrated users, the duration of the migration period, or both. If you don’t have this, you don’t know when to stop the migration. Set a cadence for how often you’ll check the number of migrated users and compare it with your goal.

Make sure you can regularly query FusionAuth to know the number of migrated accounts. To do so, set a value on the user.data object, such as user.data.migrated, indicating successful migration. You’ll also need to ensure you are using the Elasticsearch search engine, but you can switch easily.

Migration Timeline

Communicate a timeline for migration to interested parties. You can calculate that by knowing the following:

• How frequently the average user logs in

• What the distribution of your users' login behavior is

• What your migration goal is

Determining these values precisely is beyond the scope of this guide. However, a good rule of thumb is to determine how often your average user logs in to your application. Then you can use this formula to find out when a certain percentage of users have migrated: S = [(1 - (1 - P)^D)*100]%.

Breaking that equation down, you have:

• P is the probability of any single user authenticating in a given day. So if your users login once a week on a business day, it is 0.2 (one out of five days). If they login once a year, it is 0.0027 (one out of 365 days).

• D is the number of days of the migration period.

• S is the percentage of users who have migrated.

For P = 0.2 and D = 10, S = 89%. Therefore, if your users log in one out of every five days, in ten days almost nine out of ten will be migrated to FusionAuth.

For P = 0.0027 and D = 370, S = 63.7%; if your users log in once a year on average, in about a year, almost two thirds of accounts will be migrated to FusionAuth.

Using the above calculations should help you determine how the correct length of a migration.

Connect To The Original System

With a slow migration, FusionAuth is connecting to your previous datastore every time a user not currently in FusionAuth authenticates. This connection requires either an HTTP API request or an LDAP call.

If your current user datastore is an LDAP directory, such as ActiveDirectory or OpenLDAP, then you don’t need to do anything special to enable the connection; FusionAuth knows how to communicate with LDAP servers.

If, on the other hand, your datastore is not LDAP, you’ll need to build an HTTP API and use a Generic Connector. This API must take a login request in JSON format and authenticate the user against the old datastore, but can be written in any language that can output JSON.

{
  "loginId": "example@fusionauth.io",
  "password": "password",
  "applicationId": "10000000-0000-0002-0000-000000000001",
  "noJWT" : false,
  "ipAddress": "192.168.1.42"
}

This API should return a FusionAuth login response, which includes the User object as well as a JWT:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0ODUxNDA5ODQsImlhdCI6MTQ4NTEzNzM4NCwiaXNzIjoiYWNtZS5jb20iLCJzdWIiOiIyOWFjMGMxOC0wYjRhLTQyY2YtODJmYy0wM2Q1NzAzMThhMWQiLCJhcHBsaWNhdGlvbklkIjoiNzkxMDM3MzQtOTdhYi00ZDFhLWFmMzctZTAwNmQwNWQyOTUyIiwicm9sZXMiOltdfQ.Mp0Pcwsz5VECK11Kf2ZZNF_SMKu5CgBeLN9ZOP04kZo",
  "user": {
    "active": true,
    "birthDate": "1976-05-30",
    "data": {
      "displayName": "Johnny Boy",
      "migrated" : true,
      "favoriteColors": [
        "Red",
        "Blue"
      ]
    },
    "email": "example@fusionauth.io",
    "expiry": 1571786483322,
    "firstName": "John",
    "fullName": "John Doe",
    "id": "00000000-0000-0001-0000-000000000000",
    "imageUrl": "http://65.media.tumblr.com/tumblr_l7dbl0MHbU1qz50x3o1_500.png",
    "lastLoginInstant": 1471786483322,
    "lastName": "Doe",
    "middleName": "William",
    "mobilePhone": "303-555-1234",
    "passwordChangeRequired": false,
    "passwordLastUpdateInstant": 1471786483322,
    "preferredLanguages": [
      "en",
      "fr"
    ],
    "registrations": [
      {
        "applicationId": "10000000-0000-0002-0000-000000000001",
        "data": {
          "displayName": "Johnny",
          "favoriteSports": [
            "Football",
            "Basketball"
          ]
        },
        "id": "00000000-0000-0002-0000-000000000000",
        "insertInstant": 1446064706250,
        "lastLoginInstant": 1456064601291,
        "preferredLanguages": [
          "en",
          "fr"
        ],
        "roles": [
          "user",
          "community_helper"
        ],
        "tokens": {
          "Facebook": "nQbbBIzDhMXXfa7iDUoonz5zS",
          "19544aa2-d634-4859-b193-e57af82b5d12": "eu1SsrjsiDf3h3LryUjxHIKTS0yyrbiPcsKF3HDp"
        },
        "username": "johnny123",
        "usernameStatus": "ACTIVE"
      }
    ],
    "timezone": "America/Denver",
    "twoFactorEnabled": false,
    "usernameStatus": "ACTIVE",
    "username": "johnny123",
    "verified": true
  }
}

This is an example of all the JSON data you could return, but you can omit most fields. The only requirement is that either the username or the email field must be returned.

The password is not included. That password will have been hashed according to your FusionAuth tenant password encryption scheme settings.

The generated JWT will be delivered to the client to present to any resource servers (other API servers, etc). For maximum compatibility, it should have FusionAuth claim values.

The JSON response above has a user.data.migrated value of true. This indicates that this user has been migrated. As mentioned above, adding this custom attribute allows you to query migration progress. You can read more about the Generic Connector requirements in the documentation.

Configure Your Connector

Navigate to Settings → Connectors and add a Connector. You can also configure Connectors by using the API; consult the Connector API documentation for more details.

Configuration varies depending on whether the original datasource is an HTTP API or an LDAP directory.

// This is an example LDAP Connector reconcile, modify this to your liking.
function reconcile(user, userAttributes) {

  // Un-comment this line to see the userAttributes object printed to the event log
  // console.info(JSON.stringify(userAttributes, null, 2));

  // This assumes the 'uid' attribute is a string form of a UUID in the format
  // `8-4-4-4-12`. It will be necessary to ensure an attribute is returned by your LDAP
  // connection that can be used for the FusionAuth user Id.
  user.id = userAttributes.uid;
  user.active = true;

  // if migrating users, tag them by uncommenting the below lines
  // user.data = {};
  // user.data.migrated = true;

  user.email = userAttributes.mail;
  user.fullName = userAttributes.cn;

  // In this example, the registration is hard coded, you may also build this
  // dynamically based upon the returned LDAP attributes.
  user.registrations = [{
    applicationId: "5d562fea-9ba9-4d5c-b4a3-e57bb254d6db",
    roles = ['user', 'admin']
  }];

}

Capturing Migration Progress

As discussed above, you want to make sure you can identify that a user has been migrated, rather than created directly in FusionAuth. Having this data allows you to determine progress toward your migration goal. Ensure that every migrated user has a user.data.migrated attribute set to true. Whichever connection you use, make sure you set this attribute, as illustrated above.

Testing

To test this, log some users in. If you have test users in your original auth system, you can use the Login API to automate the testing:

• Confirm a user does not exist in FusionAuth

• Log a user in

• Confirm the user now exists in FusionAuth with the expected values

Proxying Authentication

With FusionAuth, proxying authentication requests to the original datasource is easy. You’ve already set up the connection information when configuring the Connector. Now associate the Connector to the tenant. This is called a Connector policy. Navigate to Tenants → Your Tenant → Connectors and add a policy.

Make sure to check the Migrate user checkbox. Then, each user will be authenticated against the original user datastore the first time they are seen. Their data will then be migrated to FusionAuth. On subsequent logins, they’ll authenticate with FusionAuth. You can learn more about configuring Connector policies in the Connector documentation.

Modify Your Application

Unlike with a big bang approach, there is no protracted downtime. You simply need to release the changes required for users to authenticate against FusionAuth after configuring and testing the connectors. Make sure you record the number of accounts in the old system just before cutover.

Should you need to rollback, revert all changes made to your application which direct users to FusionAuth. If users updated profile data in FusionAuth, you’ll need to port those changes back to your original system. A script using the user API will be a good starting point.

After you release this modified version of your application, your users will begin their transparent migration into FusionAuth.

Monitor Progress

You can monitor your progress by comparing the number of users who have successfully migrated with the number of users in the original auth system. To query FusionAuth, run this shell script:

API_KEY=...
FA_HOST=...

curl -H "Authorization: $API_KEY" $FA_HOST'/api/user/search?queryString=data.migrated%3Atrue%0A'

This will return all the users who’ve been migrated as well as a count, subject to the limits of FusionAuth’s Elasticsearch integration. See this GitHub issue for more on those limits. Here’s an example of the output:

{"total":629,"users": [ ... ] }

If you are migrating more than 10,000 users, query Elasticsearch directly to retrieve the count:

curl 'https://elasticsearch.piedpiper.com/fusionauth_user/_count?q=data.migrated%3Atrue%0A'

{"count":62900,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0}}

However you retrieve the number of users migrated, regularly compare it to the number of accounts in the original system before the migration began. This ration will determine if it is time to end the slow migration.

Remove The Proxy

When it is time to stop the slow migration, remove the Connector policy. All future logins will take place against FusionAuth.

You may also optionally remove the Connector.

Handle Unmigrated Users

At this point, decide how to handle users who haven’t been migrated. You may be able to find these users by subtracting the set of migrated users from the users in the original system. You may also be able to query the original system directly and note who has not signed in since the slow migration started.

You considered this situation while planning your migration, but now, implement the decision. You could:

• Notify them and encourage them to log in. You can contact them with a message such as: "If you don’t log in by DATE, your account will be deleted".

• Archive or delete the accounts and their corresponding data. When one of these users comes to your site and tries to sign in, they won’t have an account and will be forced to re-register, having lost their data.

• Move them to FusionAuth via a big bang migration of all unmigrated users.

• Extend the time running both systems; that is, set a new goal and continue the slow migration.

You can mix and match these approaches. For example, you could migrate all paying customers, even those who haven’t signed in during the migration period. At the same time you could archive the data of free accounts; those potential customers may have been trialing your application and may even have forgotten they have an account.