Zero downtime migration of WSO2 API Manager along with Blue-Green concept
I hope you have already known about the default migration process of WSO2 API-Manager [1]. This blog post will guide to perform a blue-green migration for a live system with zero downtime in run-time.
First, let’s get some understanding of run-time and the design-time of the API-Manager.
Design-Time: This will include the actions related to the Publisher and Store. So the API designing in the Publisher and the testing phases in the Store are concluded as the API design time.
Run-Time: This will only include the actions related to the Gateway(GW), Key-Manager(KM) and the Traffic-Manager(TM). So this is will include the action related to the actual API invocation calls like routing through the GW, key-validation from the KM and throttle decisions according to the rate-limiting policies with TM.
The Blue-Green concept
The Blue-Green concept is introduced for new feature releases of current systems, updating the current systems and migrate the current systems with an effective manner such that without downtime.
For the current live environment, from here onwards we will call it as blue-environment. Then we need to create an environment identical to the blue-environment and it can be referred to as green-environment. Then the new actions will be performed (new features/updates/migration) on the green-environment. However, we need to thoroughly test all use-cases in this green-environment until we get confident along with the above new actions.
Still, the live traffic is serving by the blue-environment, and now we have tested green-environment with the above actions (new features/updates/migration). Then we need to route the traffic to the green-environment and then it will be the live environment. Then the blue-environment will become idle and we can remove the blue-environment.
Applying the blue-green concept for the WSO2 APIM migration
Now we are referring to the current live environment as the blue-environment. So we need to create another environment which is identical to the blue-environment. For that, we need to extract the database-dumps of the blue-environment and configure the green-environment. Before making the green-environment using the extracted database dumps we need do perform the below actions.
Read-only Publisher and Store
Before taking the database-dumps of the blue-environment, we need to move the Publisher and Store to the read-only mode. The reason was here we were only considering the zero downtime in run-time, not the design time. Also, we can expect that the design time of a production environment should be limited to a set of maintenance windows as, during the actual run-time, it is not a good practice to edit the available resources.
Database triggers and syncing mechanism
Although we extracted the database dumps of the blue-environment, we should allow the run-time of the blue-environment. So it will create new tokens, revoke old tokens and will expire the old tokens. So we need to create a mechanism to get these data to the green-environment as well. So we need to apply database triggers for the below three tables in the WSO2AM_DB in blue-environment.
IDN_OAUTH2_AUTHORIZATION_CODE
IDN_OAUTH2_ACCESS_TOKEN
IDN_OAUTH2_ACCESS_TOKEN_SCOPE
The triggers will extract the data from the original tables in the blue-environment and add those data to separate tables which are called sync tables. These sync tables have separate entries for the insertions and the update actions. If we insert a new row to the original tables, this will create a new row in the sync tables as well. Also, we update the same row(which we previously added) in the original table, the sync tables will have a separate row(entry) for that update action instead of updating the previously added row(entry). That will be useful for the synchronization process. Also, triggers will create an extra column in the sync tables to easily identify the sync-state of the entries.
We need to make a data synchronizing mechanism to add those data to the green-environment. The most common way is java based synchronizing the client to proceed with this. We will discuss more on this later in this blog.
Once after making the Publisher and Store as read-only mode and configuring the database-triggers, here, we can start setting-up the green environment using the database-dumps of the blue-environment. Then we need to perform the migration actions[1] for the green-environment. After all of those, we have migrated green-environment from a live blue-environment. So here we can start the testing phase of the green-environment. Once after the testing phase, now we have the most of the confidence to upgrade the green-environment to live. However before making it live, we have to do one more step which is the data synchronizing.
Data synchronizing
Now we have successfully migrated and tested green-environment, and live blue-environment with database-triggers enabled. There may several token generations and revocations that happened in the blue-environment even after the migration process started of green-environment as this is a zero down-time(run-time) migration. So we need to migrate those data to the green-environment just before make that as the live environment. This process is called data synchronizing. So data synchronization has extracted the data from the previously mentioned sync tables and add those to the green-environment. Normally this synchronization can be performed multiple times to reduce the time taken for synchronization. Because triggers created an extra column to add the sync-state.
As we performed the synchronization multiple times, now the green-environment and blue-environment are almost identical when considering the data they had. So we need to perform the make the green-environment as live (DNS route). Then live traffic will come to the green-environment. Just after routing the traffic to the green-environment, we need to perform one more data synchronization as we may miss the data entry related to one last millisecond.
Here we have done with the blue-green migration!!
Your valuable comments on this are most welcome. :)
[1] — https://medium.com/@madurangasaneth/simple-understanding-of-wso2-apim-migration-process-3b2624e200b4
