Migration from Weaviate to Qdrant

​

Introduction

​

Test Migration

1. The approximate downtime that will be required for a proper migration.
2. The review of the report generated by the migration job to determine which projects, if any, will be affected.

​

Terminology

• This guide uses the file name cognigy-ai-values.yaml to refer to the Helm values file with your customizations for the Cognigy.AI Helm chart. Replace this with the file name you use in your environment.
• This guide assumes that Cognigy.AI is deployed in the cognigy-ai namespace. If you’ve deployed Cognigy.AI in another namespace, then adjust the namespace parameters in all the commands accordingly.
• This guide uses the term project, which corresponds to the term Agent (as a Project) as specified in the glossary.

​

Prerequisites

• Cognigy.AI v4.73 is deployed.
• Kubernetes v1.21-1.27.
• The kubectl utility is installed locally on a Linux or macOS client host. Note that this guide does not support Windows client hosts.
• Helm v3.9.0–v3.12.3 installed on the client host.
• Kubernetes cluster meets general Cognigy.AI prerequisites, including hardware resources.

​

Pre-Migration Steps

​

Calculate the Amount of Memory Required for Qdrant

memory_size = (number_of_knowledge_chunks * vector_dimension * 0.000004 * 1.5) + (number_of_projects * 10)

• number_of_knowledge_chunks is the maximum number of Knowledge Chunks permitted under your Knowledge AI license.
• vector_dimension is the dimension of the embedding vectors used in your environment, which depends on the embedding model used for the Knowledge Search use case. Since the text-embedding-ada-002 is the only embedding model supported for Knowledge AI at this time, vector_dimension is always 1536.
• 0.000004 megabytes (4 bytes) is the size of each 32-bit single-precision floating point element in the embedding vector.
• 1.5 is a factor that accounts for the space needed to store data in addition to the vectors, for example, indexes.
• number_of_projects is to use Knowledge AI. Each project using Knowledge AI creates a collection in Qdrant, requiring 10 Megabytes of memory space, even if the collection is empty.

​

Example

memory_size = (100000 * 1536 * 0.000004 * 1.5) + (1 * 10) = 931.6 MB`

memory_size = (1000000 * 1536 * 0.000004 * 1.5) + (1 * 10) = 9226 MB = 9.226 GB

​

(Optional) Create a Test Project for Knowledge AI

​

Migration Steps

​

Step 1. Enable Qdrant

​

Preliminary Steps

• Before proceeding with the setup, ensure that your cluster has enough CPU and memory resources available to support three Qdrant pods.
• To prevent any missing data in Qdrant during setup, disable the ingestion of new Knowledge AI data while Qdrant is being populated. This entails scaling down Weaviate, effectively stopping all ingestion. This note doesn’t apply if you’re conducting a test migration; Weaviate must remain unscaled for such a scenario.

​

Enable Qdrant

1. In the cognigy-ai-values.yaml file, add the following sections at the root level:
   Report incorrect code

   Copy

   # Enable Qdrant. Add a ‘resources’ section if needed.
   qdrant:
     enabled: true
     metrics:
       serviceMonitor:
         # You can enable this only if are running the Cognigy Monitoring Stack or kube-prometheus-stack in your cluster.
         enabled: false
   
   # Enable the storage class for qdrant
   # Set the following for AWS
   storageClass:
     aws:
       ebs:
         qdrant:
           enabled: true
   
     # Or, set the following for Azure
     storageClass:
       azure:
         disk:
           qdrant:
             enabled: true
   
   # Scale down weaviate (not for test migration), so that no new Knowledge AI data can be ingested while qdrant is being populated.
   weaviate:
     replicas: "0"
   

2. Deploy the Cognigy.AI Helm chart with the above changes:
   Report incorrect code

   Copy

   helm registry login cognigy.azurecr.io \
   --username <your-username> \
   --password <your-password>
   
   helm upgrade --install --namespace cognigy-ai cognigy-ai oci://cognigy.azurecr.io/helm/cognigy.ai --version 4.73.0 --values cognigy-ai-values.yaml
   

3. The qdrant-0, qdrant-1, and qdrant-2 pods should start in tandem. The following command can be used to list the Qdrant pods:
   Report incorrect code

   Copy

   kubectl get pods -n=cognigy-ai | grep qdrant
   

​

Step 2. Populate Qdrant with Existing Knowledge AI Data

1. Copy the following job description to a file named job-populate-qdrant.yaml. If you are using an image registry other than Cognigy’s, replace cognigy-registry-token under imagePullSecrets with the correct value.
   Report incorrect code

   Copy

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: job-populate-qdrant
     namespace: cognigy-ai
   spec:
     backoffLimit: 0 # No retries
     activeDeadlineSeconds: 3600 # 60 minutes
     ttlSecondsAfterFinished: 3600 # 60 minutes
     template:
       spec:
         restartPolicy: Never
         containers:
           - name: job-populate-qdrant
             image: cognigy.azurecr.io/job-sync-knowledge-data:release-5dd849b-1710242738
             resources:
               requests:
                 cpu: '0.3'
                 memory: 300M
               limits:
                 cpu: '0.5'
                 memory: 800M
             env:
               - name: REDIS_ENABLE_RECONNECT
                 value: "true"
               - name: RABBITMQ_ENABLE_RECONNECT
                 value: "true"
               - name: KAI_VECTOR_DB
                 value: "qdrant"
               - name: SYNC_KAI_CHUNKS_BATCH_SIZE
                 value: "1000"
               - name: IMPORT_TASK_ARGUMENTS
                 value: "eyJhY3Rpb24iOiJyZXN0b3JlVmVjdG9yREIiLCJvcmdhbmlzYXRpb25JZHMiOltdLCJwcm9qZWN0SWRzIjpbXSwic3RvcmVJZHMiOltdLCJ0YXNrSWQiOiIiLCJ0cmFjZUlkIjoidGVzdCIsInVwZGF0ZUJhdGNoU2l6ZSI6NDB9"
             envFrom:
               - configMapRef:
                   name: cognigy-env
             volumeMounts:
               - name: mongodb-connection-string
                 mountPath: /var/run/secrets/mongodbConnectionString
                 subPath: mongodbConnectionString
               - name: rabbitmq-connection-string
                 mountPath: /var/run/secrets/rabbitmqConnectionString
                 subPath: rabbitmqConnectionString
               - name: redis-password
                 mountPath: /var/run/secrets/redis-password.conf
                 subPath: redis-password.conf
               - name: qdrant-api-key
                 mountPath: /var/run/secrets/qdrant-api-key
                 subPath: qdrant-api-key
               - name: vector-db-config
                 mountPath: /vector-db-config/qdrant/collection/config.json
                 subPath: qdrant-collection-config
         volumes:
           - name: mongodb-connection-string
             secret:
               secretName: cognigy-service-resources
               items:
                 - key: connection-string
                   path: mongodbConnectionString
           - name: rabbitmq-connection-string
             secret:
               secretName: cognigy-rabbitmq
               items:
                 - key: connection-string
                   path: rabbitmqConnectionString
           - name: redis-password
             secret:
               secretName: cognigy-redis-password
           - name: qdrant-api-key
             secret:
               secretName: qdrant-apikey
               items:
                 - key: api-key
                   path: qdrant-api-key
           - name: vector-db-config
             configMap:
               name: vector-db-config
         imagePullSecrets:
           - name: cognigy-registry-token
   

2. Apply the job in the cluster:
   Report incorrect code

   Copy

   kubectl apply –n cognigy-ai –f job-populate-qdrant
   

   A pod will be created to correspond with the applied job. The logs for this pod will continue reporting the number of Knowledge Sources copied compared to the total number of sources to be copied.
3. Once the job is complete, copy the pod logs for your records. At the end of the logs, a brief report is included that highlights any Sources and Chunks that could not be copied. For more information about the report interpretation, referer to the Migration Report Overview section.

​

Migration Report Overview

{
  "Organisation: <org-id>": {
    "Project: <proj-id>": {
      "ignoredSource": 2, // Total number of ignored sources
      "Source: <source-id>": <Number of chunks ignored in this source>,
      "Source: <source-id> ": <Number of chunks ignored in this source>
    },
    "Project: <proj-id>": {
      "missingEmbedding": <Number of chunks ignored>
    }
  }
}

• In the first project, two entire sources were ignored; none of the Chunks in these sources were ingested into Qdrant. Note that only those Knowledge Sources are ignored entirely that are already corrupt and therefore cannot be used. This action should not cause any issues for production AI Agents.
• In the second project, the number corresponding to missingEmbedding refers to the number of Chunks ignored because they didn’t have corresponding embedding vectors stored. This error is similar to the one described previously, except that in this case, the ignored Chunks may be from more than one source. Also, these Chunks are already corrupt, and ignoring them should not disrupt the existing AI Agents.

​

Delete Data from Qdrant

• You ran a test migration that populated Qdrant. Since the migration job only works with an empty Qdrant database, you need to remove all data from Qdrant before starting a proper migration.
• As part of a proper migration, you finished running the migration job but have not yet started Step 3. Due to a potential issue with the migration job, you would like to run it again before starting Step 3.

1. Delete the PVCs:
   Report incorrect code

   Copy

   for pvc in {0..2}; do kubectl delete pvc -n cognigy-ai qdrant-storage-qdrant-$pvc; done
   

2. Use the following command to delete each Qdrant PV, replacing Qdrant PV with the name of each specific PV in the command:
   Report incorrect code

   Copy

   kubectl delete pv <pv-name>
   

​

Step 3. Instruct Knowledge AI Components to Use Qdrant instead of Weaviate

1. In the cognigy-ai-values.yaml file, add or update the following sections:
   Report incorrect code

   Copy

   # Make sure KAI_VECTOR_DB is set to "qdrant". WEAVIATE_SCHEMA_REPLICATION_FACTOR is no longer needed.
   serviceSearchOrchestrator:
     extraEnvVars:
       - name: KAI_VECTOR_DB
         value: "qdrant"
   
   # Make sure KAI_VECTOR_DB is set to "qdrant", and the batch size is set to 100.
   taskIngestKnowledgeChunks:
     extraEnvVars:
       - name: TASK_INGEST_KNOWLEDGE_CHUNKS_BATCH_SIZE
         value: "100"
       - name: KAI_VECTOR_DB
         value: "qdrant"
   
   # Disable the CronJob and make sure KAI_VECTOR_DB is set to "qdrant".
   jobSyncKnowledgeData:
     enabled: false
     extraEnvVars:
       - name: KAI_VECTOR_DB
         value: "qdrant"
   

2. Deploy the Cognigy.AI Helm chart with the above changes:
   Report incorrect code

   Copy

   helm registry login cognigy.azurecr.io \
   --username <your-username> \
   --password <your-password>
   
   helm upgrade --install --namespace cognigy-ai cognigy-ai oci://cognigy.azurecr.io/helm/cognigy.ai --version 4.73.0 --values cognigy-ai-values.yaml
   

​

Step 4. Test Knowledge AI

​

Post-Migration Steps

​

Delete the Weaviate PVC and PV

1. Delete the Weaviate PVC:
   Report incorrect code

   Copy

   delete pvc -n cognigy-ai weaviate-data-weaviate-0
   

2. Delete the Weaviate PV using the name of the Weaviate PV in your system:
   Report incorrect code

   Copy

   kubectl delete pv <pv-name>