Qlik Sense Server Migration - API and CURL Detail

Post Index

2023-07-19

How to use CURL and QRS API

Qlik Sense Server Migration Overview

Qlik Sense Server Migration

API and CURL Detail

Migrating Qlik Sense items using APIs

There are a number of QS items required to be migrated in the Qlik Sense server migration process.  This article focuses two core components to illustrate the concept and steps required for migration.  First, it will discuss about the App migration that involves App, App Object and App Content.  Then it will discuss about the Reload Task that involves Reload Task, System Event and Composite Event.

For other, the concept is similar.  I am happy to help if you have questions.

General QRS API Call

More technically, Qlik Sense Server Migration involves a few types of QRS API calls:

1) GET

To obtain the metadata in JSON and this JSON will be modified or trimmed off in order to POST/PUT into the new cloud server.

2) POST

POST command is generally to create a new item in the QS server.

3) PUT

PUT command is usually to update the QS item that is already created in the server.

For POST and PUT, usually, it requires details to be provided in the HTTP body.  It would be much easier to GET the metadata directly from the on-premises server and modify the necessary component in the JSON for the POST and PUT APIs.  One obvious example is the modified date and it must be larger than the last modified date.  More related to this article discussion is the app ID.  Since APP ID will be generated after upload, it should maintain a mapping in order to update the JSON for the POST and PUT APIs.

App Object ID vs Engine Object ID

These two IDs are tricky and easy to get confused.  When the app object is created, a unique GUID will be created and this ID is used internally in QS for QMC.  At the same time when the object is created, the Engine Object ID is also created.  It is an in-app unique ID to distinguish the object.  It is for the qvf used.

In the QRS API, app object ID is referring to the App Object ID.  However, in QMC or in the link, it is showing the Engine Object ID.

QS Application Migration

Before migration QS applications, firstly, some QS items should be migrated beforehand.  For instance, data connection, extension, content libraries, stream, etc.  This makes sure the application can make use of all these components.

In fact, to migrate an application, it is similar to do what manually do with the following steps:

1) Export the app with data and cover all the base, shared and private app objects.

2) Import the app with data with all the exported app objects and maintain the ownership

3) Publish the app to stream (if necessary)

4) Unapprove the app object

5) Unpublish the app object

6) Change the owner of the app object to maintain all the ownership

* If app is exported without data, the always one selected value setting in the field will be lost.  If you are certain that the application does not have this setting, it is recommended to export without data and do the reload afterwards.

Export App

There are three methods that are available to obtain the QS app with all the app objects:

1) using QRS API /qrs/app/exportapps (refer to May 2023 version)

This is only available in or above May 2023 version of Qlik Sense.  The export scope should be configured to "all".

2) copy the app directly in the Qlik shared folder, i.e. \\server-name\QlikShare\Apps

This folder contains all the Qlik Sense applications that are in the Qlik Sense server.  The files in this folder are named by the QS Application ID without extension.  These files are actually a binary qvf. 

3) Publish/Approve the sheets and then export the app.

This method has impact to the end users because the private and community app objects will be published and approved to base app objects.   After triggering the export, it can simultaneously unapprove and unpublish the app objects but it still take a while for these operations.  If the server is frozen, this method would be a good choice.

This method requires the following QRS API:

1) PUT /qrs/app/object/{id}/publish

To publish app objects into sharing state.

2) POST /qrs/app/object/{id}/approve

To approve app objects into base state.

3) POST /qrs/app/{id}/export

To create the download link and export the app

4) GET /qrs/download/app/{id}/{exportTicketId}/{fileName}

To download the app based on the download link

5) * if require to restore app object state, POST /qrs/app/object/{id}/unapprove

To unapprove the app objects back to sharing state.

6) * if require to restore app object state, PUT /qrs/app/object/{id}/unpublish

To unpublish the app objects back to private / personal state.

Import App

There are two APIs available for importing the QVF into the Qlik Sense server:

1) POST /qrs/app/import

This requires the qvf first located in the app folder.  C:\ProgramData\Qlik\Sense\App

2) POST /qrs/app/upload

This method is recommended.

Apparently, manual upload is also possible.

Publish App to Stream

The next step is to publish the app into stream with the QRS API.

PUT /qrs/app/{id}/publish

Some of the applications might be in the work stream that does not require this step.

Unapprove App and Change Ownership

This step is a bit tricky because we are not using the unapprove API.  Instead, 

the app object put is used.

PUT /qrs/app/object/{id}

As a result, we can update both the unapprove flag as well as the app object ownership in one-go.

Unpublish App

The last step is to unpublish the app object if they are private/personal.

PUT /qrs/app/object/{id}/unpublish

*** Note if binary QVF is used

The binary qvf files obtain all the app objects even it is deleted (reference).  So, there should be a steps to remove the redundant objects.  This can easily be cross-checked with the app object metadata and do a DELETE /qrs/app/object/{id}.

And there are a few more drawbacks:

The binary qvf files obtain all the app objects even it is deleted (reference).  So, there should be a steps to remove the redundant objects.  This can easily be cross-checked with the app object metadata and do a DELETE /qrs/app/object/{id}.

i) If the application has referenced to the In-App pictures, the link is broken.  One way to fix this is to make sure of the Engine JSON API to replace the on-premises app ID to the cloud app ID.

ii) the app properties are also lost.  This needs to be fixed by PUT /qrs/app/{id}.  Obviously, it requires a JSON that can be obtained by GET /qrs/app/{id}

iii) the app content is lost.  And it requires to POST /qrs/appcontent/{appid}/uploadfile.  The content physical files can be found in \\server-name\QlikShare\Static Content\App Content.

QS Reload Task Migration

The reload task migration is less complicated compared to the app migration.  The only tricky part is to make sure the reload task is pointing to the updated app ID.  And reload task always comes with trigger where

a) Schema event is based on the time schedule.

b) Composite event is used to chain up the app reload.

Overall, the below API calls will be used to obtain the metadata.

GET /qrs/reloadtask/full

GET /qrs/schemaevent/full

GET /qrs/compositeevent/full

The below API call will be used to create the reload task, schema event as well as the compositive event.

POST /qrs/reloadtask

POST /qrs/schemaevent

POST /qrs/compositeevent

Thank you for reading.  I hope you find it useful.  See you in the next post :)

Qlik Sense Server Migration - API and CURL Overview

Post Index

 2023-07-15

How to use CURL and QRS API

Qlik Sense Server Migration

API and CURL Overview

Migrating Qlik Sense items using APIs

There are a lot of discussions on how to perform a Qlik Sense server migration .  One typical method is to use backup and restore the PostgreSQL database and then update the configuration.  Instead of this traditional method, this article focuses the discussion on how to perform a Qlik Sense migration using QRS API with CURL to move QS items from on-premises servers to cloud servers.

This article provides an overview and provides an understanding about the basics for preparation.  For detailed implementation, it will be discussed in future posts later (will provide the links later when they are ready).  So, let's start.

Qlik Sense Migration Concept

Using QRS API to perform the migration means that the new QS server is first configured in place and then ready to accept the move-over QS items by QRS API.   The new QS server should be readily configured:

    a) Share location (the shared content location)

    b) Node setup (join to the central node cluster)

    c) Service Configuration (Proxy, Engine, Scheduler, etc)

After the new server is ready, in general, the below Qlik Sense items will be involved in the migration

    a) Data Connection

    b) Stream

    c) Extension

    d) Content Libraries

    e) Applications

    f) App Objects (Base/Private, Publish, Approved)

    g) App Content

    h) ODAG Links

    i) Reload Task (including schema event, composite event)

    j) System Rules (including Security Rules, license rules, etc)

    k) more ... (like tag, custom properties and so on), depending on the usage

The above should cover majority of the usage pattern.  More importantly, the QS items have inter-dependencies.  For example, Stream must be ready before an application can be published to it.  The above sequence can be used as a simple sequence for creation for reference.

In addition, all the active users should be synchronized from the user directory before the migration to start in order to preserve the ownership of the QS items.

Migration Control Master

Before migration, it is important to mark down the QS items that are going to be migrated.  The easiest way is to make use of the QRS API to obtain the metadata directly from the Qlik Sense server.  In general, it will be something similar to below, taking app as an example.

1) GET /qrs/app

This provides the condensed details

2) GET /qrs/app/full

This provides the full details.

If taking the aforementioned pattern, the below QRS API calls will be used to extract the metadata:

1) GET /qrs/dataconnection/full

The data connection to be used for loading data source data.

2) GET /qrs/stream/full

The stream that will be used to publish app.

3) GET /qrs/extension/full

The extension used for extra chart types.

4) GET /qrs/contentlibrary/full

The additional libraries to provide picture or media files.

5) GET /qrs/app/full

The QS application.

6) GET /qrs/app/object/full

The app objects including sheet, story, bookmark, etc.

7) GET /qrs/app/content/full

The in-app picture and media files.

8) GET /qrs/odaglink/full

The ODAG link that allows usage.

9) GET /qrs/odaglinkusage/full

The actual usage for the ODAG links.

10) GET /qrs/reloadtask/full

The reload task to trigger data reload in QS application.

11) GET /qrs/schemaevent/full

The reload task trigger by schedule.

12) GET /qrs/compositeevent/full

The reload task trigger by event.

13) GET /qrs/systemrule/full

The rule including security rules, license rules, etc.

14) GET /qrs/license/professionalaccessgroup/full

The group for professional license.

15) GET /qrs/license/analyzeraccessgroup/full

The group for analyzer license.

16) GET /qrs/user

The QS users.

Migration Process Overview

There are a few core steps required in the migration process.  It includes:

1) Extraction

It extracts all the metadata from the on-premises server via QRS API.  The result will be in JSON format and it is then required to convert into Excel as a metadata list.

2) Review

The metadata lists will then be reviewed by administrators to analyze what are required to migrate and what are redundant.  Once this process is done, it becomes control master lists as a control to make sure all items marked "migrate" should be migrated to the new cloud server.

3) Prepare

Once the control master list is ready, it will then help to prepare the generation script for CURL batch file as well as the JSON for migration.  This is tricky and a clear sequence of process is needed.

4) Migration

When all the CURL batches and JSON files are ready, it can then start the migration processes.

5) Verification

Once all items are then migrated to the cloud server, it is required to extract again both the on-premises metadata as well as the cloud metadata in addition to the control master.  They are then matched against each other to make sure items:

1) should be left in on-premises

2) should be migrated to cloud

3) are new to cloud

Advantages of Using API

One difficulty in migrating the Qlik Sense items over to the new servers is to maintain the GUID.  It is required for Qlik Sense to link up all the components internally and work together.  The integrity should be maintained.  If the GUID is changed, it means the corresponding dependent items or references have to be updated.  Otherwise, the relationship is broken, it creates lots of disfunctionality.

If the QS items are created via QMC console, a random GUID will be created.  And a lot of effort is required to match the old/new GUID and it increases the tasks to re-configure all these dependencies.

Fortunately, using QRS API, a majority of the QS items can retain their GUID which can make the migration easier and faster.  Although it is not possible to maintain the application ID (also the Application Object ID, please do not confuse with the Engine Object ID, will explain further during the detailed migration process), it already smoothens the migration process to reduce the changes required.

More discussion will be provided when discussing the migration details.

Thanks for reading and I hope you find it useful.  See you in the next post! :)