- The migration process requires downtime of approximately 60–90 minutes, depending on the number of stored Knowledge Chunks and the number of projects configured to use Knowledge AI. However, this downtime applies to Knowledge AI features only; all other features will remain available.
- The migration process alters the
distance
value. Decision-making in Flow will be ineffective post-migration if distance is used.
Introduction
Cognigy.AI releases 4.72.0 and earlier uses Weaviate as the vector database for Knowledge AI. With release 4.73.0, Cognigy.AI transitions to Qdrant as the vector database instead of Weaviate. This change aims to provide Cognigy.AI users with a significantly improved Knowledge AI experience. This guide explains how to switch from using Weaviate to Qdrant as the vector database for Knowledge AI.Test Migration
You can perform a test migration from Weaviate to Qdrant without causing any downtime. This means there will be no disruptions to the Knowledge AI traffic in the system. A test migration can be used to determine the following:- The approximate downtime that will be required for a proper migration.
- The review of the report generated by the migration job to determine which projects, if any, will be affected.
Terminology
- This guide uses the file name
cognigy-ai-values.yaml
to refer to the Helm values file with your customizations for the Cognigy.AI Helm chart. Replace this with the file name you use in your environment. - This guide assumes that Cognigy.AI is deployed in the
cognigy-ai
namespace. If you’ve deployed Cognigy.AI in another namespace, then adjust the namespace parameters in all the commands accordingly. - This guide uses the term project, which corresponds to the term Agent (as a Project) as specified in the glossary.
Prerequisites
- Cognigy.AI v4.73 is deployed.
- Kubernetes v1.21-1.27.
- The
kubectl
utility is installed locally on a Linux or macOS client host. Note that this guide does not support Windows client hosts. - Helm v3.9.0–v3.12.3 installed on the client host.
- Kubernetes cluster meets general Cognigy.AI prerequisites, including hardware resources.
Pre-Migration Steps
Execute the pre-migration steps to proceed to the migration process.Calculate the Amount of Memory Required for Qdrant
The following formula can be used to calculate the memory required by each Qdrant pod in megabytes:number_of_knowledge_chunks
is the maximum number of Knowledge Chunks permitted under your Knowledge AI license.vector_dimension
is the dimension of the embedding vectors used in your environment, which depends on the embedding model used for the Knowledge Search use case. Since thetext-embedding-ada-002
is the only embedding model supported for Knowledge AI at this time,vector_dimension
is always1536
.0.000004
megabytes (4 bytes) is the size of each 32-bit single-precision floating point element in the embedding vector.1.5
is a factor that accounts for the space needed to store data in addition to the vectors, for example, indexes.number_of_projects
is to use Knowledge AI. Each project using Knowledge AI creates a collection in Qdrant, requiring 10 Megabytes of memory space, even if the collection is empty.
Example
Let’s consider a scenario where thetext-embedding-ada-002
model is used to compute the embedding vectors.
Then, to store 100,000 Chunks in just one project, the amount of memory we need is calculated as follows:
text-embedding-ada-002
vectors in one project:
(Optional) Create a Test Project for Knowledge AI
This step isn’t required if you’re already running a test migration.
Migration Steps
Pay close attention to the migration steps, ensuring that you don’t skip any.Step 1. Enable Qdrant
Perform the preliminary steps and proceed to enable Qdrant.Preliminary Steps
- Before proceeding with the setup, ensure that your cluster has enough CPU and memory resources available to support three Qdrant pods.
- To prevent any missing data in Qdrant during setup, disable the ingestion of new Knowledge AI data while Qdrant is being populated. This entails scaling down Weaviate, effectively stopping all ingestion. This note doesn’t apply if you’re conducting a test migration; Weaviate must remain unscaled for such a scenario.
Enable Qdrant
To enable Qdrant, follow these steps:-
In the
cognigy-ai-values.yaml
file, add the following sections at the root level: -
Deploy the Cognigy.AI Helm chart with the above changes:
-
The
qdrant-0
,qdrant-1
, andqdrant-2
pods should start in tandem. The following command can be used to list the Qdrant pods:
Step 2. Populate Qdrant with Existing Knowledge AI Data
Before starting to populate Qdrant, make sure all three Qdrant pods are up and running and Weaviate has been scaled down. If you are running a test migration, Weaviate should not be scaled down. To populate Qdrant with the existing Knowledge AI data, aKubernetes Job
must be run in the cluster.
The time it takes for this job to finish depends on the amount of Knowledge AI data that needs to be copied,
as well as the total number of projects that are configured to use Knowledge AI.
The migration job provided below should only be used with an empty Qdrant database.
In case you need to run the job again, you must remove all collections from Qdrant first.
Refer to the Delete Data from Qdrant section for detailed instructions on how to empty Qdrant.
To populate Qdrant with Existing Knowledge AI Data, follow these steps:
-
Copy the following job description to a file named
job-populate-qdrant.yaml
. If you are using an image registry other than Cognigy’s, replacecognigy-registry-token
underimagePullSecrets
with the correct value. -
Apply the job in the cluster:
A pod will be created to correspond with the applied job. The logs for this pod will continue reporting the number of Knowledge Sources copied compared to the total number of sources to be copied.
- Once the job is complete, copy the pod logs for your records. At the end of the logs, a brief report is included that highlights any Sources and Chunks that could not be copied. For more information about the report interpretation, referer to the Migration Report Overview section.
Broker connection closed: undefined
and Error while connecting to RabbitMQ. Shutting down
. Ignore both of them.
If you have completed a test migration, the process is already finished. You can record the duration of the migration job and schedule the necessary downtime for the proper migration later. Additionally, you can review the logs from the job to see a list of Sources and Chunks that will not be included in the migration.
Migration Report Overview
The migration job logs, generated in the previous step, contain a report at the end. This report lists, for each organization and project, the number of Knowledge Sources and Chunks ignored during migration. The following example helps in interpreting the report:- In the first project, two entire sources were ignored; none of the Chunks in these sources were ingested into Qdrant. Note that only those Knowledge Sources are ignored entirely that are already corrupt and therefore cannot be used. This action should not cause any issues for production AI Agents.
- In the second project, the number corresponding to
missingEmbedding
refers to the number of Chunks ignored because they didn’t have corresponding embedding vectors stored. This error is similar to the one described previously, except that in this case, the ignored Chunks may be from more than one source. Also, these Chunks are already corrupt, and ignoring them should not disrupt the existing AI Agents.
Delete Data from Qdrant
Deleting all data from the Qdrant database may be necessary in the following cases:- You ran a test migration that populated Qdrant. Since the migration job only works with an empty Qdrant database, you need to remove all data from Qdrant before starting a proper migration.
- As part of a proper migration, you finished running the migration job but have not yet started Step 3. Due to a potential issue with the migration job, you would like to run it again before starting Step 3.
-
Delete the PVCs:
-
Use the following command to delete each Qdrant PV, replacing
Qdrant PV
with the name of each specific PV in the command:
Step 3. Instruct Knowledge AI Components to Use Qdrant instead of Weaviate
Before proceeding with the steps below, ensure that Qdrant has been successfully populated. To instruct Knowledge AI components to use Qdrant instead of Weaviate, follow these steps:-
In the
cognigy-ai-values.yaml
file, add or update the following sections: -
Deploy the Cognigy.AI Helm chart with the above changes:
Step 4. Test Knowledge AI
If you created a new small Knowledge AI project as described in the (Optional) Create a Test Knowledge AI Project section, you can use it to test search, ingestion, and other Knowledge AI data manipulation operations on existing data. Otherwise, you can use any of your existing projects containing Knowledge AI data for testing these operations. Additionally, we recommend creating a new project with Knowledge AI enabled to test all Knowledge AI operations.Post-Migration Steps
Complete the post-migration steps.Delete the Weaviate PVC and PV
After successful migration and testing of Knowledge AI features with Qdrant as the vector database, the Weaviate PVC and PV can be deleted.-
Delete the Weaviate PVC:
-
Delete the Weaviate PV using the name of the Weaviate PV in your system: