New streaming ingestion capabilities for Qlik Open Lakehouse

Qlik Open Lakehouse now supports high-throughput streaming ingestion, allowing you to land millions of events per second from Apache Kafka, Amazon Kinesis, and Amazon S3 directly into optimized Apache Iceberg tables in your own AWS environment.

What’s new:

Three new streaming source connectors: Kafka, Amazon Kinesis, and Amazon S3, with support for JSON, Avro, Parquet, CSV, and more.

Two new task types:

• Streaming Landing Task: Continuously stages raw streaming events on S3, acting as a buffer before transformation into Iceberg, with configurable retention policies.

• Streaming Transform Task: Transforms landed data into query-ready Iceberg tables with support for:

  • Append-only and Apply changes (with SCD Type 2 historical tracking) modes

  • Transformation capabilities including STRUCT unnesting and ARRAY flattening for complex hierarchical payloads

  • Iceberg partitioning, column sorting, and snapshot expiration settings for performance and cost control

  • Configurable and automatic schema evolution

• Support for downstream mirroring to cloud data warehouses.