Table recreation enhancements

In some cases, a change in the source data might require the target tables to be recreated (for example, if you need to change the target schema). Qlik Talend Cloud pipelines provide the ability to recreate the target datasets through the Recreate tables option. Previously, executing a recreate would also require tables in downstream data tasks to be recreated. This behavior has changed to allow recreation of tables in a data pipeline without needing to recreate tables in the downstream task. When recreating a table, the downstream task will react as if a truncate and reload action occurred on the source datasets.

Troubleshooting a storage data task

Target-based dataset model for transform and data mart tasks

To make pipeline design more flexible, we are introducing a new dataset model for transform and data mart tasks: the target-based model.

With this new model, dataset metadata is now decoupled from source metadata changes. Transform and data mart tasks are more self-contained, letting you control the effect of source metadata changes.

Target datasets are now explicitly mapped from the source datasets. If a column is added from the source, or removed, it is not automatically propagated to the next downstream task anymore. Instead, you will be able to adjust the mappings according to your needs. You can also change the column order and replace the data source as needed.

Ingestion-type tasks, such as landing and storage, remain on the source-based dataset model.

Data models

Re-designed view for transform tasks

A new view is now available for transform tasks, providing visible lineage.

This view lets you see the full dataset lineage, including transformation flows with multiple output datasets. Filters and display options are also available to adjust the view to your needs.

Using the Transform view

Layout container improvements

Charts are now highlighted in the container when they are selected in the list in the property panel.

Layout container