Qlik Sense Financial Reporting Extension

 Post Index  

2024-06-22

Qlik Sense Financial Reporting Extension

Customizing Table for Reporting Purpose

Using Qlik Sense Extension to customize Standard Report Layout

----------------------------------------------------------------------------------------------------------

Side Track

It was some time ago that I posted articles into different communities.   Now, I am trying to consolidate back all of my shared articles into my blog for easy reference.  Also, I will also review a bit of them and see if any new information to supplement.   As time goes, everything keeps changing.  In particular to technology, in most of the time, we can see a lot of old stuff to be depreciated and come with new ideas, tools, etc.  Let's ride on them and move forward!  Let's stay tuned and updated!

----------------------------------------------------------------------------------------------------------

Qlik Sense Financial Reporting Extension

This extension is my first Qlik Sense extension.  It was, in fact, previously achieved the same using QlikView.  Due to the wave to migrate QV to QS, the method done in QV is no longer possible to be applied in QS.  The only way is to make an extension to do it.

At first, I tried this because of curiosity.  However, later I found it very useful because it let me to have a deeper knowledge in QS and also got more familiar a lot of things in QS that were behind the scene.  Now, I revisit the post and share it again.  I do not think it is good to share the extension because it is a bit out of date and I don't see a big demand in using this (at least, not many people approach me and ask about this recent years).   Below is the original post.

https://community.qlik.com/t5/Integration-Extension-APIs/Financial-Reporting-Extension/m-p/1952079

Below is the content of the original post for reference.

=======================================================================

Financial Reporting Extension

Topic

Financial reports is actually a table with specific format. And each of the line is calculated from some accounts with some specific attribute filters. In QlikView, the frontend requirement is able to be achieved by customizing the straight table. And backend, it is also able to be achieved by associating the right amount to each of the reporting line. However, in Qlik Sense, because of its simplicity, it is no longer available. It is impossible to customize the table like cell border, font bold, font size, etc. While the world is migrating QlikView to Qlik Sense, it is still necessary for financial reporting.

Therefore, the only way to achieve this is to make use an extension to allow the reporting customization. The backend is still can still be the same to make use of the association.

Financial Reporting

Very Simple Sample

Settings Sample 1

Settings Sample 2

 

Keep Connected

Drop me a message for discussion and sharing if you find this interesting and useful.

=======================================================================

Qlik Sense Extension

Before ending this post, I would like to highlight a bit on using Qlik Sense Extension.  Extension is handy and easy to use and apply but in general, it is risky.   One typical issue is that it might be incompatible in future releases and often, it would not get an upgrade.  Also maintaining an extension is time-consuming.  Whenever there is a patch release or new version, regression test is needed.   Unless the extension is highly dependent, otherwise, it would not get enough resources to support it.   Being said, extension is still good to apply because the basic charts cannot offer.  In short, I would recommend you to try the basic charts, customize the basic charts first before thinking of an extension solution.

 

By the way, my extension is not the same the PNL table from Qlik Visualization bundle.  It is actually a free style HTML table allow you to customize each line.  Definitely, it requires a QS model to maximize its usage.  I will try to cover more about this later as the generic model design should be shared and learnt.

Thank you for reading.  I hope it help you.  Just leave a message if you have any discussion/share want to make.

Alteryx - Triggering Qlik Sense Task

Post Index  

2024-06-10

Alteryx - Triggering Qlik Sense Task

QS Reload Tool to CURL API Call

Integrating Alteryx and Qlik Sense

----------------------------------------------------------------------------------------------------------

Side Track

It was some time ago that I posted articles into different communities.   Now, I am trying to consolidate back all of my shared articles into my blog for easy reference.  Also, I will also review a bit of them and see if any new information to supplement.   As time goes, everything keeps changing.  In particular to technology, in most of the time, we can see a lot of old stuff to be depreciated and come with new ideas, tools, etc.  Let's ride on them and move forward!  Let's stay tuned and updated!

----------------------------------------------------------------------------------------------------------

Alteryx to Reload QS Task

Back to this post content, "Alteryx - Qlik Sense Reload Tool" is one of my first community posts in my life.  And previously, I only share concepts and roughly the ideas how to implement.  I would now try best also to share as much as I can.  You might find the original post below.

https://community.alteryx.com/t5/Alteryx-Designer-Desktop-Discussions/Alteryx-Qlik-Sense-Reload-Tool/td-p/939218

Unfortunately, the tool was developed using .NET SDK which is no longer supported by Alteryx (as of 2024-06-10) and therefore, it cannot be used anymore.  Nonetheless,  it is easy to replicate the details using CURL.  I will share more information at the end of this post.

Below is the content of the original post for reference.

=======================================================================

First Post Starts

Hello everyone. It is my first post to the Alteryx community. I start to write the posts because I would like to share the ideas on what I have done and experienced before. Might be there is someone who already applied similar solutions but I think it is good to share and keep improving with everyone in the community.  Let's evolve.

 

Topic

Nowadays, it is typical scenario that once the Alteryx workflow is completed, Alteryx is able to trigger the subsequent downstream systems to continue the data pipeline. For example, once Alteryx is finished the data preparation for Qlik Sense dashboard, it will trigger the reload of the dashboard.

 

In order to achieve this, it requires to call the Qlik Sense Repository Service API (https://help.qlik.com/en-US/sense-developer/February2022/Subsystems/RepositoryServiceAPI/Content/Sen...). Normally, we would get to use the Download tool to do so.  However, it does not support to pass the Qlik Sense certificate for authentication purpose.  The passive solution is to calculate the time required for the workflow to be finished and setup the schedule in Qlik Sense that will 100% to pick up the data.  This method is no guarantee because the workflow completion might be affected by a lot of factors like network, server resources, data volume, etc.  More practically, we could then look for the Run Command tool with CURL or postman, or simply write the python code in the Python tool. Either of them requires kind of coding/scripting to make it work.  It is not easy to maintain.

 

QS Reload Tool

In order to make it seamless for this kind of integration, I have created a QS Reload tool for this purpose.  Once it triggers the QS task and it will return and complete the workflow.  The is way to make it synchronous call, however, it is just a waste of resource to just keep waiting in the workflow without any processing.  The interface looks like below and the configurable parameters are illustrated below.

 

QS Server

This is where the server address would be supplied. The protocol does not need to be specified, only the server’s address

 

Certificate and Reload Certificate

This would be filled up when the “Reload Certificate” button is clicked. This is designed to get the certificates related to Qlik Client from your certificate store. This certificate would be passed as a header in the Http Request and is designed to make the request secure. Basically, this is also known as SSL or Secure Sockets Layer (SSL) or to encrypt the information sent from the requestor machine to the server machine. For further explanation, see Certificates Section of this document.

 

Get Task Name

Once the server address and the certificate are supplied with correct information, this button would get all the available tasks inside the Qlik Management Console inside the server.

 

Task Name

This would be filled with the tasks fetched from the server. This is possible through the use of Qlik Sense’s QRS API. The selected task would have the following format, ApplicationName(applicationID). Also whatever task id is selected in the combo box. It would be the one that would be triggered to start.

 

Messages

This is to inform the user about warnings, status and errors happening inside the GUI. This is designed to help the end user know what inputs are mandatory and the status if the request to get the task list succeed or failed. 

 

Keep Connected

I am unable to share the tool at the moment but if you need more information, kindly drop me a message.   Meanwhile, I will continue to share more stuff in the new posts.

=======================================================================

Modernizing to use CURL API 

A lot of people might query because the depreciation of the .NET SDK, it should then come into the Python SDK to continue the development as recommended by Alteryx for SDK.  Why I now provide and discuss the CURL method instead of providing a new Python replacement is because the tool itself is not a supported product and it is really difficult for people and companies to use.   No matter how native is the underlying code is, it is not possible to make it widely available where is regulatory, compliance, security, etc.  Plenty of rules that are not able to overcome.  Even if it can be passed, when Alteryx upgrades, it also faces other compatibility issues.

As a result, nowadays, API would be a more appropriate solution.  API is using widely in the word and it is one of the major method to integrate systems together.   API would be easier to be captured and used for everyone.   And natively, Alteryx has a Command tool that is possible for you to use.   That means you do not need to rely my tool and you can integrate as freely as you can.

OK.  Let's see how to use CURL API.  In fact, previously, I have shared a post before discussing about CURL and Qlik API.  For more detailed understanding of QS API call, please refer to the link below.

https://kongsoncheung.blogspot.com/2023/06/curl-qrs-api.html

To be specific, we now focus on two API methods to trigger the reload of the QS Task including:

• post /task/{id}/start/synchronous

https://help.qlik.com/en-US/sense-developer/February2022/APIs/RepositoryServiceAPI/index.html?page=1110

This method is to trigger the reload and wait for its completion.  It seldom use and waste of Alteryx worker resource in server environment.  This holds a worker without doing anything until getting a response that the QS task reload is completed. 

• post /task/{id}/start

https://help.qlik.com/en-US/sense-developer/February2022/APIs/RepositoryServiceAPI/index.html?page=1330

This is to trigger the QS reload task asynchronously, i.e. to trigger and complete.   Since the reload is inside Qlik Sense, if there is any issue, it could be reported from the Qlik Sense monitoring setup.  And more importantly, it can free up the Alteryx worker for other important tasks.

 

 A sample CURL command is illustrated as below to trigger QS Task asynchronously is as below:

"E:\Qlik Sense API\CURL\bin\curl.exe" -X POST --cert "E:\Qlik Sense API\Certificate\client.pem" --key "E:\Qlik Sense API\Certificate\client_key.pem" --insecure -k "https://[Server FQDN]:4242/qrs/task/[Task_ID]/start/?xrfkey=0123456789abcdef" --header "x-qlik-xrfkey: abcdefgh0123456" --header "X-Qlik-User: UserDirectory=internal;UserId=sa_repository" --header "Content-Type:application.json" --header "Content-Length:0"

The parameters are explained in the following:

-X

It is to specify the HTTP request is using which HTTP method for request.  The API call is required to use POST.

--cert 

The root certificate or the exported certificate from Qlik Sense server.

--key

The root key.  *Please use securely.

--insecure

Just to avoid SSL issue.   It is recommended to remove this if all underlying infrastructure is working fine.

-k "https://[Server FQDN]:4242/qrs/task/[Task_ID]/start/?xrfkey=0123456789abcdef"

To specify the API endpoint like the server address, server port, the QS Reload Task ID.  The xrfkey is to be provided both in the query as well as in the HTTP header.

--header "X-Qlik-User: UserDirectory=internal;UserId=sa_repository"

This is to impersonate the account to execute the reload.  Normally, it should be internal as the user directory and sa_repository as the user ID.

--header "Content-Type:application/json"

To specify the content provided in body is using JSON.   Even no content, we specify as JSON because all QS API is using JSON.

--header "Content-Length:0"

Since this request is a POST and there is not content required, 0 length is provided to the HTTP header.

The remaining is to put the API command into the Alteryx Command tool and let it trigger CURL to call the Qlik Sense Repository Service API.   I don't go into detail this time.  If you need help on it, leave me a message.

Thank you for reading.  I hope it help you.  Just leave a message if you have any discussion/share want to make.

 Post Index

2024-05-05

Qlik Sense Penalty Card

Share Knowledge Share Fun

Performance Tips in Qlik Sense

First of all, thank you for our team to put their effort and passion in developing the cards with fun, with heart and with the knowledge to share about the Qlik Sense best practices.  

@Sim

@Leo

@Anthony

You might know some of them but let's see if you know them all.  Performance, in fact, is $$$.  With better performance, the usage of CPU, RAM, Network and Disk is dropped and in the clod era, it saves your pocket.  Take the card wisely and most importantly, have fun with them.

The original post can be found in the Qlik Sense Community.

https://community.qlik.com/t5/New-to-Qlik-Analytics/Qlik-Sense-Penalty-Card/m-p/2448393#M250926

The cards are not perfect but we have put our passion into it.  So, enjoy the cards and have fun together.  Kindly share your views with us and let's continue with it.  Learning should be interesting and fun.

And you can find the penalty cards below.  How much do you know already?

Alteryx QVD Output Tool - Prototype

 Post Index

2023-08-17

Alteryx QVD Output Tool

Alteryx Custom Tool

Writing QVD data from Alteryx

In the previous post, the Alteryx QVD Input Tool prototype is shared and following in this post, the Alteryx QVD Output Tool will be introduced.

It reads the data from Alteryx and then convert into QVD, i.e. convert each column into a list of symbols and symbol indexes and then compact each record  by the symbol index into the least bits required to store the record data.  XML information is saved in memory during the processes.  Once everything is ready, it flushes out the XML, symbols as well as the records.

Alteryx QVD Output Tool

The Alteryx QVD Input Tool is very simple.  It just takes in a QVD file and read all the content then convert it as an Alteryx output stream.  The input UI is as below.

* the new Alteryx SDK is now using reactjs where it is not possible to get through the security to get the full path.  Thus, there is no button to pop up a dialog to ask for file location.  Instead, there is only a textfield for inputting the path.  If you have any clue to get this through, it is welcome.  The prototype is hoping to show the possibility to integrate with QVD files.

The prototype

If you hope to try it, you can download it in my github.  https://github.com/kongson-cheung/Alteryx-QVD-Tools/blob/main/yxi/QVD%20Tools_v1.1.yxi

I have share the core files to create this Alteryx QVD Output Tool.  Since the SDK includes a large number of files, I did not upload them all.  If you need any help, feel free to drop me a message.

* Note: this is still very early version of prototype.  It still requires a number of improvements for intensive use.

Next

I will try summarize how to develop the Custom Alteryx Tool.

Thank you for reading.  I hope it help you.  Appreciated your sharing if you have any discussion/share want to make.

Alteryx QVD Input Tool - Prototype

 Post Index

2023-08-07

Alteryx QVD Input Tool

Alteryx Custom Tool

Reading QVD data into Alteryx

With the findings in the previous post (QlikView Data File (QVD) - Reverse Engineering), I have developed a prototype of the Alteryx QVD input tool.  Alteryx is an extremely good tool for data wrangling and contains a bundle of tools that allow simple data transformation and predictive analysis.  However, it does not have the ability to deal with QVD.  It only allows to read and write QVX.

Alteryx, in fact, is a good upfront data stream for QlikView and Qlik Sense.  It can help business to make clear use of data with drag and drop capability to explore, transform and try new business logics.  QVD integration would be benefitial for Qlik at the lower data stream in the data cycle.   Sound like advertisement but it is real project experience to conclude this.  QVX does not work great with heavy usage.  The performance is similar to CSV.  Still, QVD is the best for Qlik.

Alteryx QVD Input Tool

The Alteryx QVD Input Tool is very simple.  It just takes in a QVD file and read all the content then convert it as an Alteryx output stream.  The input UI is as below.

* the new Alteryx SDK is now using reactjs where it is not possible to get through the security to get the full path.  Thus, there is no button to pop up a dialog to ask for file location.  Instead, there is only a textfield for inputting the path.  If you have any clue to get this through, it is welcome.  The prototype is hoping to show the possibility to integrate with QVD files.

An Example

Taking a QVD file as an example.

The QVD file contains 2 columns named Num and Text.  It has total number of 4 records.  In Alteryx, the result runs as below.

If you hope to try it, you can download it in my github.  https://github.com/kongson-cheung/Alteryx-QVD-Tools/blob/main/yxi/QVD%20Tools_v1.0.yxi

I have share the core files to create this Alteryx QVD Input Tool.  Since the SDK includes a large number of files, I did not upload them all.  If you need any help, feel free to drop me a message.

* Note: this is still very early version of prototype.  It still requires a number of improvements for intensive use.

Next

The Alteryx QVD Input Tool is made up by Platform SDK, UI SDK, Python SDK.  This is a first prototype for reading QVD.  I am exploring a prototype of Alteryx QVD Output Tool that write QVD as output.

Thank you for reading.  I hope it help you.  Appreciated your sharing if you have any discussion/share want to make.

QlikView Data File (QVD) - Reverse Engineering

Post Index 

2023-07-31

QlikView Data File (QVD)

Reverse Engineering

Revealing what is inside a QVD file

QVD is a Qlik proprietary format that is widely use in the Qlik products.  And this format works very well within Qlik environment but other than using Qlik products, it is not possible to convert data into QVD format.

QVD is famous on its performance and compression.  Comparing to CSV, Excel and other formats, it has 10x performance gain because the format can, in fact, directly be loaded into memory and directly be used by the Qlik products.

From technically perspective, this article will try to discuss the QVD in detail so that we can understand the amazing elements in this QVD file.

Note: I was analyzing this because I was trying to do a project that hopes to read/write QVD using Alteryx.  I then put some effort for this reverse engineering and developed some prototypes but then a simpler method is used and the information and effort seem deemed.  Hopefully, with this article, I think it is good for every to applause for the design and also learn how to do reverse engineering a bit.  If I get enough time, I will release the prototype in github.

QVD Component

QVD is a file that contains of three major parts:

1) XML

The XML is providing the metadata information about the QVD.   It describes the QVD table and the QVD field with a number of internal used elements.

2) Symbol

Symbol means the unique value of a field.  Each field has a list of symbols.  This is why QVD is highly compressed because each unique value in a field is only saved once.  Each symbol is indexed by a unique number, i.e. 0, 1, 2, ....

3) Record Data

Record data is stored in binary format.  Each record is stored with the size of a record byte size.  With the record bytes, each field value in a record indicated of offset and length.  This portion of binary data can then be converted into the index to get the symbol value.  The entire method makes uses of of bit operations and bit masks to reduce the byte required.  This is one of the main reasons why the data volume is highly compressed.

XML

The QVD XML is illustrated below:

<QvdTableHeader>

    <QvBuildNo>...</QvBuildNo>

    <CreatorDoc>...</CreatorDoc>

    <CreateUtcTime>...</CreateUtcTime>

    <SourceCreateUtcTime>...</SourceCreateUtcTime>

    <SourceFileUtcTime>...</SourceFileUtcTime>

    <SourceFileSize>...</SourceFileSize>

    <StaleUtcTime>...</StaleUtcTime>

    <Fields>

        <QvdFieldHeader>

            <FieldName>...</FieldName>

            <BitOffset>...<BitOffset>

            <BitWidth>...<BitWidth>

            <Bias>...</Bias>

            <NumberFormat>

                <Type>...</Type>

                <nDec>...</nDec>

                <UseThou>...</UseThou>

                <Fmt>...</Fmt>

                <Dec>...</Dec>

                <Thou>...</Thou>

            </NumberFormat>

            <NoOfSymbols>...</NoOfSymbols>

            <Offset>...</Offset>

            <Length>...</Length>

            <Comment>...</Comment>

            <Tags>

                <String>...</String>

                <String>...</String>

            </Tags>

        </QvdFieldHeader>

    </Fields>

    <Compression>...</Compression>

    <RecordByteSize>...</RecordByteSize>

    <NoOfRecords>...</NoOfRecords>

    <Offset>...</Offset>

    <Length>...</Length>

    <Lineage>

        <LineageInfo>

            <Discriminator>...</Discriminator>

            <Statement>...</Statement>

        </LineageInfo

    </Lineage>

    <Comment>...</Comment>

</QvdTableHeader>

Some of the core tags are explained:

QvBuildNo

The QVD version.

CreateUtcTime

The QVD file created date time.

TableName

The QVD table name.

QvdFieldHeader

The details about each field for the QVD parser

QvdFieldHeader/FieldName

The Field Name

QvdFieldHeader/BitOffset

In the record byte, which starting bit to start extract the symbol index.

QvdFieldHeader/BitWidth

In the record byte, how many bits to extract starting from bit offset in order to get the symbol index.

QvdFieldHeader/Bias

It is a special indicator for special handling.

QvdFieldHeader/NoOfSymbols

The number of symbols in the field.

RecordByteSize

The size required to store a record in this QVD dataset.

NoOfRecords

The number of records in the QVD

Symbol

In each fields, there is a list of symbols stored with the pattern of symbol type and symbol data.

Symbol Type is 1 byte to indicate the what kind of data and pattern to parse.

Symbol data is the data content stored in the file.  It is unique in each field for each symbol.

For symbol type 5 and 6, it is where DUAL data type is introduced.  Dual is a special data type that how Qlik stores data.  It is a pair of number and text in the form of (Text, Number).  All data in QVD, in fact, are in dual form.  For example, integer 25, it is stored as (NULL, 25).  A text "Hello" is stored as ("Hello", NULL).  A date is special that it stores ("DATE-STRING", DATE_INT).  A datetime is stored as ("DATETIME-STRING", DATE_NUMBER).  In general, it can be any (Text, Number) pair but generally, DUAL is tackling date and datetime. Sometime, color code will also make use of dual, e.g. (RED, 1), (GREEN, 2), etc.

The known (as result of reverse engineering) symbol types are:

1. Symbol Type = 1

4-byte integer is in this type.

2. Symbol Type = 2

8-byte number is in this type.  This also include decimal point numbers.

3. Symbol Type = 4

Text is in this type.  And a NULL char is at the end to indicate the end of the text.

4. Symbol Type = 5

4-byte integer along with text with a NULL end.  It is date with the form (Text, 4-byte integer).  In fact, other than date, it is possible to store any text/integer pair.

5. Symbol Type = 6

8-byte number along with text with a NULL end.  It is datetime with the form (Text, 8-byte number)  In fact, other than date, it is possible to store any text/number pair.

The order of how these symbols are read indicates the corresponding symbol index.  For example in a field, "c" is the first read for the field, it will have a index =0, the "a" is the second symbol read, it will have index = 1.   It does not require a proper sorting.

Moreover, it is also required special attention on how it manipulates NULL.  Using an example will be easier to understand.  With a table with three fields Num, Text, Dummy with the following data:

Num,Text,Dummy

2, A,

1, B,

1, A,

2, B,

,,

Num has 3 symbols including values of 2, 1 and NULL.

Text has 3 symbols including values of A, B and NULL

Dummy has 1 symbol including values of i.e. NULL

The symbol stored of Num will be

[Symbol Type =5][1 and "1"]  => index =0

[Symbol Type =5][2, "2"] => index = 1

[Symbol Type =5][NULL, NULL] => index = 2

It requires 8 bytes + 2 bytes (utf-8, 2 bytes for a char) for the symbol data.  To indicate 3 symbol indexes, it requires 2 bits.

As a result in the QVDFieldHeader, BitOffset is 0 and BitWidth is 2 and Bias is 0.

* it will treat as 5 as it does not know the data type well.  It happens when the data is coming from a CSV without data type specification.  If it is coming from DB, it has a mapping between the DB type and the type to be used in QVD.

The symbol stored of Text will be

[Symbol Type =4][A] => index = 0

[Symbol Type =4][B] => index = 1

[Symbol Type =4][NULL] => index = 2

It requires 2 bytes (utf-8, 2 bytes for a char) for the symbol data.  To indicate 3 symbol indexes, it requires 2 bits.

As a result in the QVDFieldHeader, BitOffset is 2 and BitWidth is 6 and Bias is 0.  Since it is the last column with data, it will take up all bits to form a full byte, i.e. 6-bits even the smallest and required is just 2 bits.  And funny is that the last column means the last column with data.  If it is all null, it would not treat as the last column.

The symbol stored of Dummy will be

[Symbol Type =4][NULL] => index = 0

It requires nothing for data storage.  But the symbol type 4 is still required to store 1 bytes, i.e. NULL byte.

As a result in the QVDFieldHeader, BitOffset is 0 and BitWidth is 0 and Bias is 0.  Offset, width and bias are zeros indicate no bytes are required for record. 

Record Data

In QVD, each record is not storing the exact field values.  Indeed, it stores the symbol indexes of the all fields.

Taking the same example used in the symbol illustration, the table below:

Num,Text,Dummy

2, A,

1, B,

1, A,

2, B,

,,

First record: require Num[index=0] and Text[index=0] and dummy=nothing, the record represents as [0000], 2 bits for Num, 2 bits for Text.

Second record: require Num[index=1] and Text[index=1] and dummy=nothing, 3 bits represents as [0101]

Third record: require Num[index=1] and Text[index=0] and dummy=nothing, 2 bits represents as [0001]

Forth record: require Num[index=0] and Text[index=1] and dummy=nothing, 2 bits represents as [0100]

Fifth record: require Num[index=2] and Text[index=2] and dummy=nothing, 2 bits represents as [1010]

The first field will be stored in the rightmost bits while the last field will be stored in the left most bits.

Thus, the 5 four records will be stored as [0000], [0101] [0001] [0100], [1010].  The just enough bytes are required to hold these binary data, i.e. 8 bits, 1 byte.  Thus, they become [0000 0000], [0000 0101], [0000 0100], [0000 0001], [00001010] => 0, 5, 1, 4, 10.  These 5 integers are used to represent total of 15 values.

To complete the QvdTableHeader description, the RecordByteSize is 1.  And NoOfRecords are 5 because there are 5 rows.

* Bias, still, needs more investigation on the exact usage.

Reverse Engineering - What have been done?

In order to try further understanding the format, a simple way to is generate a simple QVD file and look into the details.  For example a single column with 2 rows with integer only.  Keep iterating with different data and review the details, it is easy to spot the changes.  It might require a notepad editor that can show the invisible byte like NULL, EOT, etc.  A good notepad editor is notepad++.

Obviously, there might be still more handling in QVD but it is already showcase the beauty of it and why it can process that fast and compress that much.

Personally, it terms of file operation, I seriously hope that Qlik can further expand this usage because nowadays the data usage is huge and file is everywhere.  A proper and manageable file is important.  In particular to support the cloud computing, an enhanced of QVD might do the trick as well.  If it can break through the area and become open-source to use, it will greatly beneficial to everyone dealing with data.