Introducing Knowledge Marts

Qlik Talend Cloud is introducing knowledge mart tasks to automate the transformation, vectorization and loading of data into a vector store to support custom semantic search and retrieval-augmented-generative (RAG) AI applications.

Two new tasks have been introduced with this release:

• Knowledge Mart task: Enables users to generate documents from structured data and publish to a vector database. The document design process leverages a data model, similar to data mart tasks, to automate the enrichment and transformation of data into domain-oriented documents. The task processes data and automates the vectorization and incremental updates to the defined vector store. Knowledge mart tasks are supported in Qlik Talend Cloud Enterprise Edition running on Snowflake.
• File-based Knowledge Mart: Enables users to upload unstructured documents to a knowledge mart. The task can source data from file shares, FTP, OneDrive or Sharepoint and automates the chunking, embedding and loading of the data to aa vector store index. File-based Knowledge mart tasks are supported in Qlik Talend Cloud Enterprise Edition.

  

The Test Assistant supports testing vector store content via natural language semantic search for both styles of Knowledge Marts.

Knowledge Mart and File-based Knowledge Mart tasks support:

• Delivery to vector stores:
  • Snowflake Cortex
  • ElasticSearch
  • OpenSearch
  • PineCone Vector Search
• Embedding via customer-provided LLMs:
  • Snowflake Corte. You must use it when Snowflake is your vector store.
  • Amazon Bedrock
  • Azure OpenAI
  • OpenAI

Creating knowledge marts