> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Fusion 5 Upgrades export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => { const [isLoaded, setIsLoaded] = useState(false); useEffect(() => { const timer = setTimeout(() => { setIsLoaded(true); }, 500); return () => clearTimeout(timer); }, []); return
{isLoaded && ` }} />} Powered by Lucidworks Agent Studio
; }; [localhost link]: http://localhost:3000/docs/5/fusion/operations/fusion-5-upgrades [mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/fusion-5-upgrades This guide describes how to perform Fusion 5 upgrades. The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade. Fusion 5.9.x must be upgraded one maintenance release at a time. If you are on version 5.9.12 and want to reach 5.9.14, you are required to upgrade to 5.9.13 first, then upgrade to 5.9.14. Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading to Fusion 5.9.5, follow the [Fusion 5.9.5 upgrade instructions](#to-5-9-5). Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions. {/* [#general-upgrade-process] */} ## General upgrade process Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE. Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments. Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice. Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any. You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads). Check for any running jobs prior to upgrading to maintain data integrity and prevent job state corruption. 1. Verify job status with the Connectors API: ```bash theme={"dark"} curl GET https://FUSION_HOST/api/connectors/jobs?state=RUNNING ``` 2. Complete document processing. 1. Wait for all pending documents to be indexed. 2. Verify no retry operations are pending. 3. Confirm index pipeline statistics show completion. ### Natively supported deployment upgrades | Deployment type | Platform | | ------------------------------------------- | -------- | | **Azure Kubernetes Service (AKS)** | `aks` | | **Amazon Elastic Kubernetes Service (EKS)** | `eks` | | **Google Kubernetes Engine (GKE)** | `gke` | Fusion includes upgrade scripts for natively supported deployment types. To upgrade: 1. Open the `___upgrade_fusion.sh` upgrade script file for editing. 2. Update the `CHART_VERSION` to your target Fusion version, and save your changes. 3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option. After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do: ``` kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq ``` #### Configset template upgrades Introduced in Fusion 5.9.12 During upgrades, Fusion automatically updates Solr configset templates (system templates with names beginning with `_fusion_`) to ensure they match the version you're upgrading to. The upgrade process uses an ownership file (`_fusion_owner.txt`) in each configset to determine whether it's safe to replace the template. If you've customized a configset template and want to preserve your changes during upgrades, you have two options: * Remove the `_fusion_owner.txt` file from the configset in ZooKeeper. * Modify the `_fusion_owner.txt` file to contain a different value than the Fusion docker image owner. When Fusion encounters a configset template with a missing or mismatched `_fusion_owner.txt` file, it skips that template and logs a warning, preserving your customizations. For complete details about configset templates and how they work, see [Configset templates](/docs/5/fusion/intro/fusion-stack/solr/configset-templates). {/* [#other-kube-upgrades] */} ### Other Kubernetes deployment upgrades To update an existing installation, do: ``` RELEASE=f5 NAMESPACE=default helm repo update helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}" ``` Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy: ```yaml theme={"dark"} strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate ``` ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example: ``` kubectl delete pod f5-zookeeper-0 ``` Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod. You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file: ```yaml theme={"dark"} solr: ... zookeeper: updateStrategy: type: "RollingUpdate" ``` #### Upgrades with Helm v3 One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service. When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions. Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version. Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion. **How to upgrade Fusion** 1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already. 2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform. 3. Run the script with the `--upgrade` option. To see what would be upgraded, pass the `--dry-run` option to the script. The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks. #### Helm upgrade script Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files. Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options. If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do. The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script. ## Learn more This article includes instructions for upgrading Fusion from one version to another. In some cases, the instructions do not vary. Other upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the [General upgrade process](#general-upgrade-process). Remember, upgrade instructions may vary between deployment types too. Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any. You can download the latest files at [V2 Connectors Downloads](https://doc.lucidworks.com/fusion-connectors/n0s114/v-2-connectors-downloads). Fusion values change between releases. Check the [example values](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) and update values as needed. ## Upgrades from 5.8.x ### to 5.9.y Upgrading from 5.8.x to 5.9.y involves using the [General upgrade process](#general-upgrade-process). ## General upgrade process Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE. Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments. Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice. Whenever you upgrade Fusion, you must also update your remote connectors. You can download the latest files at [V2 Connectors Downloads](https://doc.lucidworks.com/fusion-connectors/n0s114/v-2-connectors-downloads). ### Natively supported deployment upgrades | Deployment type | Platform | | ------------------------------------------- | -------- | | **Azure Kubernetes Service (AKS)** | `aks` | | **Amazon Elastic Kubernetes Service (EKS)** | `eks` | | **Google Kubernetes Engine (GKE)** | `gke` | Fusion includes upgrade scripts for natively supported deployment types. To upgrade: 1. Open the `___upgrade_fusion.sh` upgrade script file for editing. 2. Update the `CHART_VERSION` to your target Fusion version, and save your changes. 3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option. After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do: ```bash wrap theme={"dark"} kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq ``` {/* // Details on the upgrade process and why current vs. target versions matter. */} This article includes instructions for upgrading Fusion from one version to another. Some upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the [General upgrade process](#general-upgrade-process). Fusion 5.9.x must be upgraded one maintenance release at a time. If you are on version 5.9.12 and want to reach 5.9.14, you are required to upgrade to 5.9.13 first, then upgrade to 5.9.14. When you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any. You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads). Fusion values change between releases. Check the [example values](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) and update values as needed. {/* [#59x] */} ## Upgrades from 5.9.x {/* [#59x-59y] */} ### to 5.9.y Upgrading from 5.9.x to 5.9.y involves using the [General upgrade process](#general-upgrade-process). If you are using horizontal pod autoscaling, follow the [steps in the Fusion 5.8.1 release notes](/docs/5/fusion/release-notes/more/5.8.1-release-notes). If you have already done this as part of a previous upgrade, you can skip this process. ### to 5.9.16 If you use Seldon models or Smart Answers, which depends on Seldon, and override `JAVA_TOOL_OPTIONS` or `javaToolOptions` in your Helm values for `ml-model-service`, `fusion-query`, or `fusion-indexing`, the required protobuf flag is lost when you upgrade to Fusion 5.9.16. Without this flag, Seldon model serving and Smart Answers will fail. To avoid this: 1. Locate where you set `JAVA_TOOL_OPTIONS` or `javaToolOptions` in your Helm values. 2. Add `-Dcom.google.protobuf.use_unsafe_pre22_gencode` to your Java options. 3. Validate Seldon models or Smart Answers after upgrade. {/* [#59x-5912] */} ### to 5.9.12 When upgrading to Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully: ```yaml theme={"dark"} kuberay-operator: crd: create: true ``` {/* [#59x-595] */} ### to 5.9.5 Upgrading to Fusion 5.9.5 has special considerations, due to changes introduced with Solr 9.6.1 and Lucene 9.10.0. All upgrades are susceptible to issues from these changes. Follow the upgrade procedures closely to avoid issues with the upgrade. See the following sections for an overview of the issues, or [skip to the upgrade process](#upgrade-process). #### Solr 9.6.1 changes Prior to Fusion 5.9.5, Fusion utilized Solr 9.1.1 or earlier. Due to changes in Solr 9.3, some Solr configuration and collection configurations are no longer compatible. As Fusion 5.9.5 leverages Solr 9.6.1, it’s imperative to address these compatibility issues during the upgrade process. To address Solr and collection configuration issues, a Docker utility called `fm-upgrade-apps-to-solr-9`, also known as the Fusion migration script, is included in the Fusion 5.9.5 release. This utility performs the following tasks: * Removes the unused configuration, ``, from `solrconfig.xml`. Solr no longer supports this configuration. * Removes the query response writer of class `solr.XSLTResponseWriter`. * Comments out processors of type `solr.StatelessScriptUpdateProcessorFactory`. * Removes `` element from request handler. * Changes cache class attribute of elements `"filterCache"`, `"cache"`, `"documentCache"`, `"queryResultCache"` to `solr.search.CaffeineCache`. * Removes `keepShortTerm` attribute from filter of class `solr.NGramFilterFactory`. * Updates collection configurations, as needed. #### Lucene 9.10.0 changes A Lucene update to 9.10.0 in Fusion 5.9.5 may cause issues with certain collections in Solr. The change to the FST posting format codec (from `Lucene90PostingsWriterDoc` to `Lucene99PostingsWriterDoc`) in Lucene is incompatible with Solr in Fusion. As a result, Solr will not open a new searcher for collections using the `FST50` postings format. To identify collections potentially affected by the Lucene codec change, examine the field definitions within your Solr schema. Look for fields that specify the `postingsFormat` attribute with a value of `FST50`. Collections containing such fields may experience compatibility issues. For example: ```xml theme={"dark"} ``` The following log excerpt demonstrates a typical error message encountered when an upgrade is impacted by the codec change: ```java wrap expandable theme={"dark"} Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene90PostingsWriterDoc vs expected codec=Lucene99PostingsWriterDoc (resource=ByteBufferIndexInput(path="/var/solr/data/acme_query_rewrite_staging_shard1_replica_t9/data/index/_cn_FST50_0.doc")) at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:205) ~[?:?] at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[?:?] at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254) ~[?:?] at org.apache.lucene.codecs.lucene99.Lucene99PostingsReader.(Lucene99PostingsReader.java:80) ~[?:?] at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:60) ~[?:?] at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:330) ~[?:?] at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:392) ~[?:?] at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:119) ~[?:?] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:96) ~[?:?] at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178) ~[?:?] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[?:?] at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542) ~[?:?] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[?:?] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:604) ~[?:?] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112) ~[?:?] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91) ~[?:?] at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) ~[?:?] at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2399) ~[?:?] ``` {/* [#run-the-fm-upgrade-query-rewrite-docker-utility] */} To account for the `postingsFormat="FST50"` codec issue, a Docker utility called `run-the-fm-upgrade-query-rewrite-docker-utility` is provided alongside the Fusion 5.9.5 release. You can pull this image from Docker using `docker pull lucidworks/run-the-fm-upgrade-query-rewrite-docker-utility:2.x`. This utility performs two actions: `prepare` and `restore`. Use the `prepare` action before the Fusion 5.9.5 upgrade begins. At a high level, the `prepare` action performs the following actions: 1. Removes `postingsFormat="FST50"` from all collections in the environment. 2. Re-indexes documents to new, temporary collections. 3. Compares the original collections to the new, temporary collections to ensure data integrity. Use the `restore` action after the Fusion 5.9.5 upgrade finishes, which must include the Solr 9.6.1 upgrade. The `restore` action performs the following actions: 1. Restores `postingsFormat="FST50"` to all collections in the environment that were changed with the `prepare` action. 2. Re-indexes documents to new, permanent collections. These collections match the original collections that were in place prior to the `prepare` action. 3. Compares the restored collections to the temporary collections to ensure data integrity. {/* [#upgrade-process] */} #### Upgrade process This section provides a high-level overview of the steps involved in upgrading to Fusion 5.9.5. Follow each step in the order given: 1. [Create a full backup of all Fusion collections](#back-up-your-solr-collections). These backups are intended as an emergency failsafe only. 2. [Run the `fm-upgrade-apps-to-solr-9` Docker utility](#run-the-fm-upgrade-apps-to-solr-9-docker-utility). This updates the Solr configuration and collections for compatibility with Solr 9.6.1. 3. [Run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility](#run-the-fm-upgrade-query-rewrite-docker-utility). Use the `prepare` action to address potential collection compatibility issues with Lucene 9.10.0 codecs. 4. [Upgrade your Fusion environment to version 5.9.5](#general-upgrade-process). Use the upgrade scripts or your own upgrade process. 5. [Re-run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility](#re-run-the-fm-upgrade-query-rewrite-docker-utility). Use the `restore` action to restore collections to their original state. 6. [Validate the upgrade was successful](#validate-the-upgrade). In addition to your usual validations, there are some extra things to check. To mitigate potential upgrade issues, adhere to the following procedures. {/* [#backup] */} #### Back up your Solr collections Back up all Solr collections in each environment before continuing with the upgrade. For this upgrade, backups are intended as an emergency failsafe. Performing a rollback after encountering the issue described is a difficult, time-consuming process and is not recommended. {/* [#run-the-fm-upgrade-apps-to-solr-9-docker-utility] */} #### Run the `fm-upgrade-apps-to-solr-9` Docker utility Use the `fm-upgrade-apps-to-solr-9` Docker utility to mitigate issues related to the change from Solr 9.1.1 and earlier to Solr 9.6.1. To begin, run the `fm-upgrade-apps-to-solr-9` Docker utility using the `DRY_RUN` environmental variable: ```bash theme={"dark"} docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e DRY_RUN=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0 ``` The `DRY_RUN` variable prints the changes that would occur to the console without performing those actions. Review the changes thoroughly. If the changes look correct, run the `fm-upgrade-apps-to-solr-9` Docker utility again without using the `DRY_RUN` environmental variable. The updated config files are saved to the `/upgrade-work/updated-configs` directory. The utility also creates backups for all configs in the `/upgrade-work/backup-configs`. The `fm-upgrade-apps-to-solr-9` Docker utility has another environmental variable, `REVERT`, that allows you to revert any changes you made. To revert your changes, run: ```bash theme={"dark"} docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e REVERT=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0 ``` {/* [#run-the-fm-upgrade-query-rewrite-docker-utility] */} #### Run the `fm-upgrade-query-rewrite` Docker utility Next, mitigate codec issues related to the Lucene 9.10.0 update. Run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility `prepare` action: ```bash theme={"dark"} kubectl run \ --image="lucidworks/fm-upgrade-query-rewrite:2.x" \ --restart=Never \ --env="HELM_RELEASE=FUSION_NAMESPACE" \ --env="TARGET_SOLR_VERSION=9.6.1" \ --env="ACTION=prepare" prepare-595-upgrade --namespace=FUSION_NAMESPACE ``` Change `FUSION_NAMESPACE` to the name of your application namespace for the Fusion installation. You can find this value using `helm list` against your Fusion namespace. Locate the release using the `fusion` chart, and find the value in the `name` column. Typically, the release name is the same as your namespace name. Including `--namespace=FUSION_NAMESPACE` lets the update pod runs in the correct application namespace. The `prepare` action removes `postingsFormat="FST50"` from all collections in the environment before re-indexing data to temporary collections. When the `prepare-595-upgrade` pod shows the status `Completed`, the process is finished. {/* [#general-upgrade-process] */} #### Upgrade your Fusion environment Upgrade Fusion to version 5.9.5. Before beginning, ensure the Fusion admin is running and all collections are healthy. Then, [complete the General upgrade process](#general-upgrade-process) before returning to the [next step in the process](#re-run-the-fm-upgrade-query-rewrite-docker-utility). Alternatively, your organization may use a custom upgrade process. In either case, ensure you have successfully upgraded to Solr 9.6.1 as part of the Fusion upgrade. **Do not** make changes to the `signals` collection with the Rules Editor during the upgrade process. For production clusters, upgrade during a maintenance window. {/* [#re-run-the-fm-upgrade-query-rewrite-docker-utility] */} #### Re-run the `fm-upgrade-query-rewrite` Docker utility Use the `run-the-fm-upgrade-query-rewrite-docker-utility` utility’s `restore` action to restore the data from the temporary collections created by the `prepare` action. Before you begin, verify all collections appended with `_temp_fix` are online and healthy. ```bash theme={"dark"} kubectl run \ --image="lucidworks/fm-upgrade-query-rewrite:2.x" \ --restart=Never \ --env="HELM_RELEASE=FUSION_NAMESPACE" \ --env="TARGET_SOLR_VERSION=9.6.1" \ --env="ACTION=restore" restore-595-upgrade --namespace=FUSION_NAMESPACE ``` Change `FUSION_NAMESPACE` to the name of your application namespace for the Fusion installation. You can find this value using `helm list` against your Fusion namespace. Locate the release using the `fusion` chart, and find the value in the `name` column. Typically, the release name is the same as your namespace name. Including `--namespace=FUSION_NAMESPACE` lets the update pod runs in the correct application namespace. When the `restore-595-upgrade` pod shows the status `Completed`, the process is finished. For a complete summary of what this action does, refer to [Upgrade Utility](#run-the-fm-upgrade-query-rewrite-docker-utility). {/* [#validate-the-upgrade] */} #### Validate the upgrade In addition to your typical validation process, ensure Solr collections are healthy: 1. Log into Fusion as the admin. 2. Access the Solr Admin UI at `https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/`. 3. Watch for error messages. For example, the following message reports errors for the query rewrite staging collections Acme1, Acme2, Acme3, and Acme4: Codec error 4. Navigate to the **Cloud graph** screen at `https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/~cloud?view=graph`. 5. Review each collection for errors. After you have completed validations, delete the temporary `prepare` and `restore` pods that were created by the upgrade utility: ```bash theme={"dark"} kubectl delete po prepare-595-upgrade kubectl delete po restore-595-upgrade ``` #### Resolving post-upgrade issues If you followed the previous instructions, and you are still experiencing issues with the codec, you need to re-feed the affected data. Contact [Lucidworks Support](https://support.lucidworks.com/hc/en-us) for further support.