2023-07-15
Qlik Sense Server Migration
API and CURL Overview
There are a lot of discussions on how to perform a Qlik Sense server migration . One typical method is to use backup and restore the PostgreSQL database and then update the configuration. Instead of this traditional method, this article focuses the discussion on how to perform a Qlik Sense migration using QRS API with CURL to move QS items from on-premises servers to cloud servers.
This article provides an overview and provides an understanding about the basics for preparation. For detailed implementation, it will be discussed in future posts later (will provide the links later when they are ready). So, let's start.
Qlik Sense Migration Concept
There are a lot of discussions on how to perform a Qlik Sense server migration . One typical method is to use backup and restore the PostgreSQL database and then update the configuration. Instead of this traditional method, this article focuses the discussion on how to perform a Qlik Sense migration using QRS API with CURL to move QS items from on-premises servers to cloud servers.
This article provides an overview and provides an understanding about the basics for preparation. For detailed implementation, it will be discussed in future posts later (will provide the links later when they are ready). So, let's start.
Qlik Sense Migration Concept
Using QRS API to perform the migration means that the new QS server is first configured in place and then ready to accept the move-over QS items by QRS API. The new QS server should be readily configured:
a) Share location (the shared content location)
b) Node setup (join to the central node cluster)
c) Service Configuration (Proxy, Engine, Scheduler, etc)
After the new server is ready, in general, the below Qlik Sense items will be involved in the migration
a) Data Connection
b) Stream
c) Extension
d) Content Libraries
e) Applications
f) App Objects (Base/Private, Publish, Approved)
g) App Content
h) ODAG Links
i) Reload Task (including schema event, composite event)
j) System Rules (including Security Rules, license rules, etc)
k) more ... (like tag, custom properties and so on), depending on the usage
The above should cover majority of the usage pattern. More importantly, the QS items have inter-dependencies. For example, Stream must be ready before an application can be published to it. The above sequence can be used as a simple sequence for creation for reference.
In addition, all the active users should be synchronized from the user directory before the migration to start in order to preserve the ownership of the QS items.
Using QRS API to perform the migration means that the new QS server is first configured in place and then ready to accept the move-over QS items by QRS API. The new QS server should be readily configured:
a) Share location (the shared content location)
b) Node setup (join to the central node cluster)
c) Service Configuration (Proxy, Engine, Scheduler, etc)
After the new server is ready, in general, the below Qlik Sense items will be involved in the migration
a) Data Connection
b) Stream
c) Extension
d) Content Libraries
e) Applications
f) App Objects (Base/Private, Publish, Approved)
g) App Content
h) ODAG Links
i) Reload Task (including schema event, composite event)
j) System Rules (including Security Rules, license rules, etc)
k) more ... (like tag, custom properties and so on), depending on the usage
The above should cover majority of the usage pattern. More importantly, the QS items have inter-dependencies. For example, Stream must be ready before an application can be published to it. The above sequence can be used as a simple sequence for creation for reference.
In addition, all the active users should be synchronized from the user directory before the migration to start in order to preserve the ownership of the QS items.
Migration Control Master
Migration Control Master
Before migration, it is important to mark down the QS items that are going to be migrated. The easiest way is to make use of the QRS API to obtain the metadata directly from the Qlik Sense server. In general, it will be something similar to below, taking app as an example.
1) GET /qrs/app
This provides the condensed details
2) GET /qrs/app/full
This provides the full details.
If taking the aforementioned pattern, the below QRS API calls will be used to extract the metadata:
1) GET /qrs/dataconnection/full
The data connection to be used for loading data source data.
2) GET /qrs/stream/full
The stream that will be used to publish app.
3) GET /qrs/extension/full
The extension used for extra chart types.
4) GET /qrs/contentlibrary/full
The additional libraries to provide picture or media files.
5) GET /qrs/app/full
The QS application.
6) GET /qrs/app/object/full
The app objects including sheet, story, bookmark, etc.
7) GET /qrs/app/content/full
The in-app picture and media files.
8) GET /qrs/odaglink/full
The ODAG link that allows usage.
9) GET /qrs/odaglinkusage/full
The actual usage for the ODAG links.
10) GET /qrs/reloadtask/full
The reload task to trigger data reload in QS application.
11) GET /qrs/schemaevent/full
The reload task trigger by schedule.
12) GET /qrs/compositeevent/full
The reload task trigger by event.
13) GET /qrs/systemrule/full
The rule including security rules, license rules, etc.
14) GET /qrs/license/professionalaccessgroup/full
The group for professional license.
15) GET /qrs/license/analyzeraccessgroup/full
The group for analyzer license.
16) GET /qrs/user
The QS users.
Migration Process Overview
There are a few core steps required in the migration process. It includes:
1) Extraction
It extracts all the metadata from the on-premises server via QRS API. The result will be in JSON format and it is then required to convert into Excel as a metadata list.
2) Review
The metadata lists will then be reviewed by administrators to analyze what are required to migrate and what are redundant. Once this process is done, it becomes control master lists as a control to make sure all items marked "migrate" should be migrated to the new cloud server.
3) Prepare
Once the control master list is ready, it will then help to prepare the generation script for CURL batch file as well as the JSON for migration. This is tricky and a clear sequence of process is needed.
4) Migration
When all the CURL batches and JSON files are ready, it can then start the migration processes.
5) Verification
Once all items are then migrated to the cloud server, it is required to extract again both the on-premises metadata as well as the cloud metadata in addition to the control master. They are then matched against each other to make sure items:
1) should be left in on-premises
2) should be migrated to cloud
3) are new to cloud
Advantages of Using API
Before migration, it is important to mark down the QS items that are going to be migrated. The easiest way is to make use of the QRS API to obtain the metadata directly from the Qlik Sense server. In general, it will be something similar to below, taking app as an example.
1) GET /qrs/app
This provides the condensed details
2) GET /qrs/app/full
This provides the full details.
If taking the aforementioned pattern, the below QRS API calls will be used to extract the metadata:
1) GET /qrs/dataconnection/full
The data connection to be used for loading data source data.
2) GET /qrs/stream/full
The stream that will be used to publish app.
3) GET /qrs/extension/full
The extension used for extra chart types.
4) GET /qrs/contentlibrary/full
The additional libraries to provide picture or media files.
5) GET /qrs/app/full
The QS application.
6) GET /qrs/app/object/full
The app objects including sheet, story, bookmark, etc.
7) GET /qrs/app/content/full
The in-app picture and media files.
8) GET /qrs/odaglink/full
The ODAG link that allows usage.
9) GET /qrs/odaglinkusage/full
The actual usage for the ODAG links.
10) GET /qrs/reloadtask/full
The reload task to trigger data reload in QS application.
11) GET /qrs/schemaevent/full
The reload task trigger by schedule.
12) GET /qrs/compositeevent/full
The reload task trigger by event.
13) GET /qrs/systemrule/full
The rule including security rules, license rules, etc.
14) GET /qrs/license/professionalaccessgroup/full
The group for professional license.
15) GET /qrs/license/analyzeraccessgroup/full
The group for analyzer license.
16) GET /qrs/user
The QS users.
Migration Process Overview
There are a few core steps required in the migration process. It includes:
1) Extraction
It extracts all the metadata from the on-premises server via QRS API. The result will be in JSON format and it is then required to convert into Excel as a metadata list.
2) Review
The metadata lists will then be reviewed by administrators to analyze what are required to migrate and what are redundant. Once this process is done, it becomes control master lists as a control to make sure all items marked "migrate" should be migrated to the new cloud server.
3) Prepare
Once the control master list is ready, it will then help to prepare the generation script for CURL batch file as well as the JSON for migration. This is tricky and a clear sequence of process is needed.
4) Migration
When all the CURL batches and JSON files are ready, it can then start the migration processes.
5) Verification
Once all items are then migrated to the cloud server, it is required to extract again both the on-premises metadata as well as the cloud metadata in addition to the control master. They are then matched against each other to make sure items:
1) should be left in on-premises
2) should be migrated to cloud
3) are new to cloud
Advantages of Using API
One difficulty in migrating the Qlik Sense items over to the new servers is to maintain the GUID. It is required for Qlik Sense to link up all the components internally and work together. The integrity should be maintained. If the GUID is changed, it means the corresponding dependent items or references have to be updated. Otherwise, the relationship is broken, it creates lots of disfunctionality.
If the QS items are created via QMC console, a random GUID will be created. And a lot of effort is required to match the old/new GUID and it increases the tasks to re-configure all these dependencies.
Fortunately, using QRS API, a majority of the QS items can retain their GUID which can make the migration easier and faster. Although it is not possible to maintain the application ID (also the Application Object ID, please do not confuse with the Engine Object ID, will explain further during the detailed migration process), it already smoothens the migration process to reduce the changes required.
More discussion will be provided when discussing the migration details.
Thanks for reading and I hope you find it useful. See you in the next post! :)
One difficulty in migrating the Qlik Sense items over to the new servers is to maintain the GUID. It is required for Qlik Sense to link up all the components internally and work together. The integrity should be maintained. If the GUID is changed, it means the corresponding dependent items or references have to be updated. Otherwise, the relationship is broken, it creates lots of disfunctionality.
If the QS items are created via QMC console, a random GUID will be created. And a lot of effort is required to match the old/new GUID and it increases the tasks to re-configure all these dependencies.
Fortunately, using QRS API, a majority of the QS items can retain their GUID which can make the migration easier and faster. Although it is not possible to maintain the application ID (also the Application Object ID, please do not confuse with the Engine Object ID, will explain further during the detailed migration process), it already smoothens the migration process to reduce the changes required.
More discussion will be provided when discussing the migration details.
Thanks for reading and I hope you find it useful. See you in the next post! :)
No comments:
Post a Comment