22 June 2023

Developing a custom data provider for TimeXtender

Share this message
Learn the basic steps of implementing a custom data provider in TimeXtender

We were recently asked to migrate a custom data provider from a Qlik environment to TimeXtender. The data source is a custom-built SOAP API. Extracting the data for a single table involves calling multiple endpoints. It also involves obscure pagination logic and splitting string data into multiple rows and columns. While the supplier of the API still maintains it, it’s not actively supported or developed. We wouldn’t be getting any help or changes there. A nice challenge!

While TimeXtender supports data connections for SOAP, because of all the steps involved this could not be used. However, TimeXtender supports creating custom data connectors. This lets you develop a connector that is caters to your specific needs. It’s not restricted by the limitations of built-in connectors. We decided to take this route and learn some new things in the process.

But first, the result:

TimeXtender custom data provider - previewing data
(click to enlarge)

The task

Custom connectors in TimeXtender work as follows: using a reference DLL provided by TimeXtender, you develop a DLL that holds the logic for your connector. This DLL is then installed on the server where the TimeXtender client is used, after which you configure TimeXtender to add a custom data source. After that, there’s no difference between adding any other data source. The connector holds all the logic for any function or feature that is usually handled by TimeXtender, up and until the moment the data is loaded into TimeXtender. This means that creating configuration files, testing the connection to the data source, making a selection of the tables and fields you want, previewing the table, fetching the data and implementing any data selection rules created by users all needs to be handled in the connector.

The implementation

There is little official documentation regarding custom connectors and how to build them, but fortunately TimeXtender has been very helpful in supporting Bitmetric throughout the development. The connector has to be built in .Net 4.8, and is divided into several parts: the configuration properties, the data table, the data source and finally the provider that ties it all together. The data table and provider are kept to the default code as most of the logic is in the settings that you create to configure the connector and the data source where you actually connect to the source you want to fetch the data from and perform all the other logic mentioned earlier. While I won’t go into actual code, to give an impression of the required logic here is a list of the functions you’ll need to implement:

  • Creating, loading and saving the connection settings
  • Connecting to the data source
  • Fetching a list of the tables and columns
  • Fetching data from the tables selected in TimeXtender
  • Formatting the data into an object that TimeXtender can ingest
  • Performing data selection based on the rules created in TimeXtender per field
  • Returning the data to TimeXtender

After developing the connector you end up with a custom DLL file. This file is installed on the TX server where you want to use the connector. PowerShell is used to add it to the GAC (Global Assembly Cache). This allows TimeXtender to find the DLL and support it as a custom data connector.

The reference DLL and our custom-built DLL, located in the TimeXtender location folder
Successfully registering the DLL in the GAC

The result

The connector is added as a Custom Data Source in a Business Unit. From there, the workflow is the same as adding any other data source. You configure it, select the tables and columns you want, add any data selection rules, then Execute to load the data into TimeXtender. The animations below show the main functionality of the connector.

Adding and testing the new custom data provider for TimeXtender

TimeXtender custom data provider - previewing data
(click to enlarge)

Reading the custom data source and selecting tables and fields

TimeXtender custom data provider - reading the data source and selecting tables and fields
(click to enlarge)

Loading data with the custom data provider for TimeXtender

TimeXtender custom data connector - loading data into TimeXtender
(click to enlarge)

Viewing the loaded data

TimeXtender custom data provider - viewing the data that was loaded using the custom connector
(click to enlarge)

Conclusion

Thanks to TimeXtender supporting creating your own connectors we’ve been able to support a data source that we otherwise would’ve had to leave out of TimeXtender.

Of course, if you run into any obscure data sources that you want added to TimeXtender then do not hesitate to get in touch. We’ll gladly help you out!

Take your Qlik skills to the next level!

Since 2013, the Masters Summit for Qlik is the premier advanced training for Qlik. Join us in Vienna and take your Qlik skills to the next level.

Join the team!

Do you want to work within a highly-skilled, informal team where craftsmanship, ingenuity, knowledge sharing and personal development are valued and encouraged? Check out our job openings.

Client Case ETL TimeXtender

How can we help?

Barry has over 20 years experience as a Data & Analytics architect, developer, trainer and author. He will gladly help you with any questions you may have.