Autoscaling with KEDA

KEDA (Kubernetes Event-Driven Autoscaling) provides flexible, event-driven autoscaling for Fusion workloads as an alternative to the default Kubernetes Horizontal Pod Autoscaler (HPA).

Traditional HPA-based autoscaling relies primarily on CPU and memory utilization. KEDA extends this by enabling you to scale workloads based on events, schedules, and business metrics, allowing you to align infrastructure capacity more closely with actual demand.

Autoscaling Fusion with KEDA

The course for Autoscaling Fusion with KEDA focuses on enabling and tuning event-driven, scheduled, and lag-based autoscaling for the api-gateway, query-pipeline, and fusion-indexing services. It covers cron, CPU, and Kafka consumer lag triggers, troubleshooting, and how Fusion 5.19 keeps indexing consistent while pods scale in and out.

Take this course on the LucidAcademy.

Benefits

KEDA provides these advantages for Fusion workloads:

• Event-driven autoscaling - Scale in response to external signals such as Prometheus metrics, queue depth, or pipeline execution load.
• Scheduled scaling - Automatically scale workloads based on predictable traffic patterns, such as business hours or peak usage windows.
• Scale-to-zero capability - Reduce infrastructure costs by scaling api-gateway and query-pipeline down to zero during off-hours or periods of inactivity.
• Improved operational efficiency - Align infrastructure capacity with real business demand instead of relying solely on CPU or memory thresholds.

KEDA combines scheduled scaling (cron trigger) with reactive scaling (CPU trigger) to align infrastructure capacity with actual demand throughout the day.

Supported services

The following Fusion services support KEDA autoscaling:

• api-gateway
• query-pipeline
• fusion-indexing

Solr pods use a standard HPA. For how that affects Solr collections, see Solr autoscaling limitations.

Don’t scale fusion-indexing to zero. It serves synchronous APIs, such as push indexing and parser testing, and no KEDA trigger can wake it in response to API traffic. In Fusion 5.19 and later, the Helm chart rejects a minReplicas or idleReplicaCount value of 0 for this service.

Before you begin

Before configuring KEDA for Fusion services, ensure you have:

• KEDA version 2.18.3 or later installed in your Kubernetes cluster
• Access to the Fusion Helm charts
• Ability to provide a custom values file for your Fusion deployment

If KEDA isn’t already installed in your cluster, install it using Helm:

# Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts

# Update your local Helm chart repository cache
helm repo update

# Install KEDA
helm install keda kedacore/keda --version 2.18.3 --namespace keda --create-namespace

For more information, see the official KEDA documentation.

How KEDA autoscaling works

Fusion Helm charts support two mutually exclusive autoscaling mechanisms per workload:

1. Kubernetes HPA (default) - Scales based on CPU and memory metrics.
2. KEDA ScaledObject - Scales based on events, schedules, and custom metrics.

Only one autoscaling mechanism can be active per workload at a time. You can’t use both HPA and KEDA for the same service.

However, you can mix autoscaling approaches across different services. For example, use HPA for api-gateway and KEDA for query-pipeline.

Autoscaling parameters

Each Fusion service has three key autoscaling parameters:

Parameter	Description	Default
autoscaling.enabled	Master switch to enable autoscaling	false
autoscaling.hpa.enabled	Enables HPA autoscaling	true
autoscaling.keda.enabled	Enables KEDA autoscaling	false

Default behavior

When autoscaling.enabled is true and you don’t modify other settings, Fusion creates an HPA by default. This preserves backward compatibility with existing deployments.

Mutual exclusion

The Helm chart enforces these rules:

• If autoscaling.enabled is false → No autoscaling (neither HPA nor KEDA)
• If hpa.enabled is true and keda.enabled is false → HPA autoscaling
• If hpa.enabled is false and keda.enabled is true → KEDA autoscaling
• If both hpa.enabled and keda.enabled are true → No autoscaling (conflict resolution)

When KEDA creates a ScaledObject with certain trigger types, it may create its own HPA prefixed with keda-hpa-. This is expected behavior and indicates that KEDA is managing autoscaling correctly.

Enable KEDA

You can enable KEDA for the api-gateway, query-pipeline, or fusion-indexing services by editing the Fusion values file before a Fusion deployment or upgrade, as explained in the next section below.

Update the Fusion values file

The configuration for each Fusion service is the same, as shown below:

api-gateway

api-gateway:
  autoscaling:
    enabled: true
    hpa:
      enabled: false  # Disable HPA
    keda:
      enabled: true   # Enable KEDA

Deploy or upgrade Fusion

After preparing your custom values file (for example, fusion-values.yaml), deploy or upgrade Fusion:

helm upgrade --install <RELEASE_NAME> <FUSION_CHART_PATH> -f fusion-values.yaml

Replace <RELEASE_NAME> and <FUSION_CHART_PATH> with your specific values.

To find your existing Fusion release name, run helm list -n <namespace>. The chart path can be a repository reference (for example, lucidworks/fusion) or a local path to the chart directory.

Verify the configuration

After deployment, verify that KEDA is active for the configured services.

Verify api-gateway

Check for the ScaledObject:

kubectl get scaledobject -n <FUSION_NAMESPACE> api-gateway

The api-gateway ScaledObject should appear in the output.

Verify that no conflicting HPA exists:

kubectl get hpa -n <FUSION_NAMESPACE> api-gateway

No HPA should exist for api-gateway, except those prefixed with keda-hpa- (which are managed by KEDA).

Verify query-pipeline

Check for the ScaledObject:

kubectl get scaledobject -n <FUSION_NAMESPACE> query-pipeline

The query-pipeline ScaledObject should appear in the output.

Verify that no conflicting HPA exists:

kubectl get hpa -n <FUSION_NAMESPACE> query-pipeline

No HPA should exist for query-pipeline, except those prefixed with keda-hpa- (which are managed by KEDA).

Verify fusion-indexing

Check for the ScaledObject:

kubectl get scaledobject -n <FUSION_NAMESPACE> fusion-indexing

The fusion-indexing ScaledObject should appear in the output.

Verify that no conflicting HPA exists:

kubectl get hpa -n <FUSION_NAMESPACE> fusion-indexing

No HPA should exist for fusion-indexing, except those prefixed with keda-hpa- (which are managed by KEDA).

KEDA may create an HPA prefixed with keda-hpa- followed by your release name and service name. For example: keda-hpa-fusion-query-pipeline. This is expected when KEDA uses certain trigger types.

Custom configuration parameters

You can customize the scaling behavior, metadata, and the triggers KEDA monitors using the parameters in the table below.

For the api-gateway, query-pipeline, or fusion-indexing services, replace [service-name] with the name of the service.

Parameter	Description	Default
[service-name].autoscaling.keda.labels	Labels to add to the ScaledObject.	{}
[service-name].autoscaling.keda.annotations	Annotations to add to the ScaledObject.	{}
[service-name].autoscaling.keda.pollingInterval	Interval in seconds at which KEDA checks triggers.	30
[service-name].autoscaling.keda.cooldownPeriod	Time in seconds KEDA waits after scale-down before re-evaluating.	300
[service-name].autoscaling.keda.initialCooldownPeriod	Time in seconds KEDA waits after scale-out before re-evaluating.	0
[service-name].autoscaling.keda.idleReplicaCount	Replicas to maintain when triggers are inactive (enables scale-to-zero). Must not be 0 for fusion-indexing.	Disabled
[service-name].autoscaling.keda.minReplicas	Minimum number of replicas. Must be 1 or greater for fusion-indexing.	1
[service-name].autoscaling.keda.maxReplicas	Maximum number of replicas.	5
[service-name].autoscaling.keda.advanced	Advanced HPA behavior configuration.	{}
[service-name].autoscaling.keda.fallback	Fallback HPA configuration if KEDA encounters errors.	{}
[service-name].autoscaling.keda.triggers	List of KEDA triggers (required).	[]

At least one trigger is required when KEDA is enabled. Without triggers, KEDA can’t determine when to scale your workload. For fusion-indexing, enabling the built-in kafka or prometheus trigger (see Triggers for fusion-indexing) satisfies this on its own; triggers is only required for api-gateway and query-pipeline, which have no built-in trigger.

This example demonstrates a custom KEDA configuration for query-pipeline with these components:

• Cron trigger - Scales to 6 replicas during business hours (Monday-Friday, 7:00 AM - 6:00 PM Central Time).
• CPU trigger - Scales up to 15 replicas when CPU utilization exceeds 60%.

query-pipeline:
  autoscaling:
    enabled: true
    hpa:
      enabled: false
    keda:
      enabled: true
      pollingInterval: 40  # Check triggers every 40 seconds
      minReplicas: 2
      maxReplicas: 15
      triggers:
        # Scheduled scaling
        - type: cron
          metadata:
            timezone: "America/Chicago"
            start: "0 7 * * 1-5"      # 7:00 AM weekdays
            end: "0 18 * * 1-5"       # 6:00 PM weekdays
            desiredReplicas: "6"
        # Reactive scaling
        - type: cpu
          metricType: Utilization
          metadata:
            value: "60"  # Scale when CPU exceeds 60%

For more information about available KEDA triggers, see the KEDA scalers documentation.

Triggers for fusion-indexing

This section applies to Fusion 5.19 and later.

fusion-indexing includes two built-in triggers that scale on Kafka consumer lag, so you don’t need to write your own. For details about how each trigger evaluates lag, see the KEDA Apache Kafka scaler and Prometheus scaler documentation.

Parameter	Description	Default
fusion-indexing.autoscaling.keda.kafka.enabled	Enables the kafka trigger, which reads consumer group lag directly from the Kafka broker.	false
fusion-indexing.autoscaling.keda.kafka.bootstrapServers	Kafka bootstrap servers for the fusion-indexing consumer group. Required when enabled.	""
fusion-indexing.autoscaling.keda.kafka.consumerGroup	The fusion-indexing consumer group to monitor. Required when enabled.	""
fusion-indexing.autoscaling.keda.kafka.topic	The Kafka topic to monitor. Required when enabled.	""
fusion-indexing.autoscaling.keda.kafka.lagThreshold	Consumer lag at which KEDA scales out.	"1000"
fusion-indexing.autoscaling.keda.kafka.activationLagThreshold	Applies only when scaling from 0 replicas, which fusion-indexing doesn’t allow, so it has no effect.	"0"
fusion-indexing.autoscaling.keda.kafka.offsetResetPolicy	Offset policy applied when the consumer group has no committed offset yet. latest keeps a new consumer group’s whole topic from counting as lag.	"latest"
fusion-indexing.autoscaling.keda.kafka.excludePersistentLag	Set to true to leave a partition out of the lag calculation when its consumer offset hasn’t changed since the previous polling interval, such as when its consumer is stuck.	Not set
fusion-indexing.autoscaling.keda.kafka.authenticationRef	Name of a keda.sh/v1alpha1 TriggerAuthentication, if the broker requires SASL or TLS.	""
fusion-indexing.autoscaling.keda.prometheus.enabled	Enables the prometheus trigger, which reads consumer lag from the kafka_consumer_fetch_manager_records_lag_max metric. Requires a scrape target.	false
fusion-indexing.autoscaling.keda.prometheus.serverAddress	Address of the Prometheus server to query. Required when enabled.	""
fusion-indexing.autoscaling.keda.prometheus.consumerGroup	The fusion-indexing consumer group to monitor. Required when enabled.	""
fusion-indexing.autoscaling.keda.prometheus.lagThreshold	Consumer lag at which KEDA scales out.	"1000"
fusion-indexing.autoscaling.keda.prometheus.activationThreshold	Applies only when scaling from 0 replicas, which fusion-indexing doesn’t allow, so it has no effect.	"0"

Use the kafka trigger unless Prometheus already scrapes fusion-indexing. The kafka trigger depends only on the Kafka broker.

Scrape targets for the prometheus trigger

The prometheus trigger reads a metric that Prometheus must collect from fusion-indexing. The chart can create the scrape target for you. Enable the option that matches your Prometheus installation:

Parameter	Description	Default
fusion-indexing.metrics.podMonitoring.enabled	Creates a PodMonitoring resource for Google Cloud Managed Service for Prometheus.	false
fusion-indexing.metrics.podMonitoring.interval	Scrape interval. Keep this at or below pollingInterval so KEDA doesn’t act on stale lag.	"30s"
fusion-indexing.metrics.podMonitoring.keepRegex	Regular expression for the metrics to keep. Set to "" to keep all metrics.	"kafka_consumer_fetch_manager_records_lag_max|indexing_.*"
fusion-indexing.metrics.podMonitoring.keepRegexUnchecked	Set to true to skip the chart’s check that keepRegex keeps the lag metric. Use this only if your keepRegex matches the metric with a pattern instead of its full name.	false
fusion-indexing.metrics.serviceMonitor.enabled	Creates a ServiceMonitor resource for the Prometheus Operator.	false
fusion-indexing.metrics.serviceMonitor.interval	Scrape interval. Keep this at or below pollingInterval so KEDA doesn’t act on stale lag.	"30s"
fusion-indexing.metrics.serviceMonitor.additionalLabels	Labels your Prometheus Operator uses to select ServiceMonitors, such as release: <PROMETHEUS_RELEASE>.	{}

If Prometheus doesn’t collect kafka_consumer_fetch_manager_records_lag_max, KEDA reads the lag as 0 and never scales out. If prometheus.enabled is true, the chart fails to render unless keepRegex is empty, contains the full metric name, or keepRegexUnchecked is true.

This example scales fusion-indexing between 1 and 8 replicas using the kafka trigger:

fusion-indexing:
  autoscaling:
    enabled: true
    hpa:
      enabled: false
    keda:
      enabled: true
      minReplicas: 1
      maxReplicas: 8
      kafka:
        enabled: true
        bootstrapServers: "<RELEASE_NAME>-kafka-headless:9092"
        consumerGroup: "<CONSUMER_GROUP>"
        topic: "<TOPIC>"
        lagThreshold: "1000"

Replace the placeholders with these values:

• <RELEASE_NAME>: The Helm release name of your Fusion deployment.
• <CONSUMER_GROUP> and <TOPIC>: The consumer group and topic of the indexing workload you want to scale on, typically your highest-volume datasource. Each trigger monitors one consumer group.

To list the consumer groups and see the topic each one reads, run these commands:

# List consumer groups
kubectl exec -n <FUSION_NAMESPACE> <RELEASE_NAME>-kafka-0 -- \
  /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

# Show the topic and current lag for a consumer group
kubectl exec -n <FUSION_NAMESPACE> <RELEASE_NAME>-kafka-0 -- \
  /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <CONSUMER_GROUP>

Index pipeline consumer groups follow the pattern index-pipeline--<PIPELINE_ID>--<TOPIC>. For example, index-pipeline--example_Products--fusion.connectors.datasource-products_S3_Load reads the topic fusion.connectors.datasource-products_S3_Load.

Troubleshooting

This section provides troubleshooting steps for some common issues with KEDA configuration.

No ScaledObject created

1. Verify that autoscaling.enabled is true.
2. Verify that autoscaling.keda.enabled is true.
3. Verify that autoscaling.hpa.enabled is false.
4. Check that KEDA is installed: kubectl get pods -n keda.
5. Review Helm deployment logs for errors.

Both HPA and ScaledObject exist for the same workload

1. Check your configuration for conflicting settings.
2. Ensure that both hpa.enabled and keda.enabled aren’t set to true.
3. Delete the unwanted resource manually if necessary.

KEDA not scaling as expected

1. Verify that triggers are configured correctly.
2. Check KEDA operator logs: kubectl logs -n keda -l app=keda-operator
3. Describe the ScaledObject to see its status: kubectl describe scaledobject <name> -n <namespace>
4. Verify that the trigger source (metrics endpoint, queue, and so on) is accessible.

autoscaling.keda.minReplicas must be >= 1 or idleReplicaCount must not be 0

The Helm chart returns these errors because fusion-indexing doesn’t support scale-to-zero. Set fusion-indexing.autoscaling.keda.minReplicas to 1 or greater. Remove idleReplicaCount, or set it to a value from 1 up to, but not including, minReplicas.