Monday, May 8, 2023

Analytical Calendar - 001 - The Story

Post Index

Analytical Calendar The Story | The Index Page

2023-05-08

Analytical Calendar



Analytical Calendar is a new calendar that is designed specifically for analytics.  It redefines the new practice of how to use date and calendar in analytics.  Analytical Calendar not only helps users to clearly define the period for analysis and comparison, but also reduces the development effort for developers because of its generic design.  Overall, the visualization is also improved with the change of less charts and a clearer concept of date usage.

The idea for Analytical Calendar came about when I was working on a project that required me to provide a variety of period-based analysis with flexible selections and visualizations.  I quickly realized that the traditional methods of using date and calendar in analytics were not sufficient.  I needed a tool, or even re-design the calendar, that would allow everyone more easily and efficiently to analyze the data.

The journey to create Analytical Calendar was not easy. It took many iterative attempts before the first draft was complete. While working against the clock to complete the project, I designed and iterated on the calendar in parallel. The first release was difficult to make, and the overall feedback was negative. There were many questions about what the calendar was and how it could be used. It took several weeks of iterative experiments and explaining the usage and advantages of the calendar before people could see its real power and potential. Nothing is perfect at the beginning, and Analytical Calendar is still on a journey of continuous improvement with positive feedback.

I would like to thank you my team for the success of this analytical calendar design and implementation.  The key to our success was our open communication and willingness to discuss ideas.  I appreciate the hard work and dedication and I am proud of what we have accomplished together.  My team includes:

Before I provide more details about the calendar, I would like to share some of the thoughts and motivations behind it. The result is important, but the process is equally important because it is how we learn and grow. We must be determined and believe in what we are doing, even when there are no clear paths forward.

This valuable experience will be helpful to everyone who faces challenges that have never been tackled before. I hope to share as much as possible so that we can all become better professionals and create a better, more knowledgeable world for the future.

I hope this is helpful!



Motivation

The motivation for this project came from a Qlik Sense (QS) project. The client needed a way to flexibly implement a number of charts with a variety of comparison requirements. The traditional way to implement a master calendar (or auto calendar in QS) is to write a lot of expressions and set analysis. However, this can be difficult to maintain and can lead to performance issues.

When a user tries to perform a date selection, such as picking a week, the month expression for comparison can be broken. This is because the traditional method of implementing a master calendar does not take into account user interactions. As a result, the more expressions that are written, the more difficult it is to maintain the QS application.

Additionally, the more expressions that are written, the more difficult it is to create new visualizations. This is because the overall design of the QS application becomes more complex and difficult to understand. When a user selects a week, a month, or a year, it is not always clear what periods should be used for analysis and comparison in the chart. Is it a fixed chart? Is it changed based on selection? How should this be determined?

As a result, it is difficult for both users and developers to strike a balance between flexibility and maintainability. This project aims to address this challenge by providing an innovative solution that allows for flexible implementation of charts with a variety of comparison requirements while maintaining performance and ease of maintenance.



Iterative Attempts

We are running out of time on this project. If we cannot find a way to speed things up, we'll have to abandon our plans for a new design and stick with the traditional approach.



(1) Master Calendar 

A master calendar is a common term in analytics that contains all the date components and the series of dates. When a selection is made, the period is filtered out to present insights in all the charts.

The master calendar is made up of single date components, such as year, quarter, month, week, and day. It also includes composite components, such as year-quarter, year-month, year-week, and date. To use the master calendar, it is simply to click on one or more of the elements.

However, there are a few problems with the master calendar design

  • First, it requires manually selecting the date components one by one. For example, if it is to see data for a specific date, it needs to first select the year, then the month, and then the day.
  • Second, it is not always clear what data will be shown in the chart when selecting certain date components.  For example, if selecting a year-month, will the chart show data for the entire year or just for the month?
  • Third, the master calendar design does not support comparisons. To compare data from different periods, it is required to create two separate master calendars.

Comparing to just using a date field, the master calendar design is more powerful.  It is the traditional ways of manipulating dates.


(2) As-of Calendar

A few years ago, I was inspired by my colleague Simon Wong of his new type of calendar design called the as-of calendar.  It was designed for financial reporting and was built on top of a master calendar.

The idea behind the as-of calendar is that it is required to first select a date, and then select a period type, such as YTD (year-to-date), YTM (year-to-month), YTW (year-to-week), or MTD (month-to-date).  When selecting both a date and a period type, it means that it is selecting a range of dates.

The as-of calendar does not require complicated expressions or set analysis. All of this is built into the associative data model. When a selection is made, the association filters out the required dates for analysis.

However, there are a few issues with the as-of calendar.

  • First, it must always select a period type. If it is neglected, the results will not be accurate.  For example, if a date is selected without a period type, the results will be just the selected date instead of YTD if it is desired.
  • Second, users often forget to select a period type.  This is because they are used to the traditional calendar design, which only allows you to select year, month, date, or week.
  • Third, the as-of calendar still requires another calendar for comparison. It is not possible to compare data from different periods using the as-of calendar alone. It must create another calendar, or use set analysis, to control the analysis behavior for the period.
  • Finally, the as-of calendar does not include any comparison component. This means that you can't compare data from different periods side-by-side.

The analytical calendar is, in fact, based on the core concepts and design of as-of calendar with further thinking about date representation as well as the inter-relationship between all date components.


(3) From-to Calendar

The as-of calendar has been refined to allow for a more flexible representation of date ranges. This is done by representing all date ranges as a from-date and a to-date. This change has been made to the associative data model in the backend and is now using a from-to notation design.  The front-end is still remaining the same.

In other words, the backend of the system has been improved, but the front-end has not been updated to take advantage of these changes. This means that users will not see any immediate benefits from the new design.


(4) Analysis Calendar and Comparison Calendar

The from-to calendar has been renamed to analysis calendar to make its purpose more clear. The new version introduces the comparison unit and comparison range. The comparison unit can be year, quarter, month, week, or day, and it indicates how many units are included in the comparison range. For example, if the comparison unit is year and the comparison range is 2, it means that the analysis will compare data from two years ago.

As a result, there are now four main components required for an analysis: date, period type, comparison unit, and comparison range. All complicated expressions are now built into the associative data model, so users do not need to write the complicated set analysis and expressions.

The backend of the system has been improved, but the front-end is still not user-friendly. The user interface is similar to the old as-of calendar, so users may not be familiar with the new design.


(5) Analytical Calendar (Re-designed of Analysis Calendar and Comparison Calendar)

The calendar front end has been redesigned to include the following features:
  • Date Range Type
This allows users to specify the granularity of the dates as well as the date range such D-YTD, M-Actual, M-YTM.
  • Unique Date
This is the specific date in its representational format.  If date range type is Y-Actual, it is like YYYY, e.g. 2022.  If date range type is M-YTM, it is YYYY-MM, e.g. 2023-01.  It is a unique representation of date.
  • Rolling Range
This allows specifying how many previous dates in addition to the unique date.
  • Comparison Unit
This allows specifying the unit of time for comparison, such as year, quarter, month, week, or day.
  • Comparison Range
This allows specifying how many previous periods against the unique date.
  • Custom Date
This allows specifying a custom date range for comparison.

The new design was met with some initial negative feedback, as it changed the way users were used to interacting with the calendar.  However, after explaining the benefits of the new design and how it would make it easier to use, users eventually came around and the new design was a success and it was the first launch of the analytical calendar.


(6) Analytical Calendar (Enhancement of date sequences and including/excluding full of year, quarter, month, week options)

After the first release of the analytical calendar, there were many pieces of feedback received.
  • Date Range Type
Some users found it difficult to understand. They were unsure of the exact meaning.
  • Unique Date
Some users also found it difficult to understand.  In this field, it has all unique formats of date and it has a dependency to the date range type.
  • Rolling Range
Some users found it to be confusing.  It is only useful when date range type is selected with the rolling related values such a D-Rolling, M-Rolling, etc.
  • Comparison Unit and Comparison Range
Some users also found them to be difficult to interpret because both need to be selected to make a meaning.
  • Date sequence

          The date sequence was not fully implemented and some date sorting is not achievable.

  • Full period
User feedback the unique date should reflect full period.  For example, today is 2023-05-08, the last available year-month should be 2023-04 instead of 2023-05.
  • Cross year
For week, there should be flexibility on maintaining full week even crossing the year or just starting a new week when it is on 1-Jan.

These changes are intended to address the feedback that was received and to make the calendar front end easier to use.

This version is targeted to enhance Date sequence, full period and cross year.



(7) Analytical Calendar (Latest version)

The front-end and backend have been redesigned to address feedback from users. The new design includes the following fields only:
  • Date Perspective
This allows users to select the granularity of the data they want to analyze, such as year, quarter, month, week, or date.
  • Unique Date
This is a combination of all the unique dates, including year, year quarter, year month, week year week, and date. When the date perspective is selected, the relevant unique date of the same granularity will be displayed.
  • Analysis Perspective
This allows selecting the date range to analyze, such as actual, YTD, YTM, etc.
  • Comparison Perspective
This allows users to compare the selected date range to a previous date, such as 1 year before, 2 months before, 3 weeks before, or a custom date.


This new design makes it easier for analysts to define their date analysis strategy. The four key questions that analysts need to answer are:
  1. What date granularity to use?
  2. Which date to analyze?
  3. To analyze a range of dates or just the actual date?
  4. To compare by a period before or by just selecting the date.
This new version is still in experimental use, but it has the potential to make data analysis much easier for analysts.


Thank You

I hope you enjoyed reading about the story of analytical calendar. In the next few sections, I will walk through the design thinking process, the core implementation requirements, and some actual implementation samples. I hope you find this information interesting and inspiring.














No comments:

Post a Comment