> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LWAI Batch Vectorize Index Stage

> Lucidworks AI

export const schema = {
  "type": "object",
  "title": "LWAI Batch Vectorize",
  "description": "Invokes a Lucidworks AI embedding model to convert a string field into a dense vector representation, processing documents in configurable batches for throughput efficiency. Specify the source field containing the text and the destination field to receive the vector output. Skips documents where the source field is absent or `null`.",
  "required": ["accountName", "modelType", "sourceFieldName", "destinationFieldName"],
  "properties": {
    "skip": {
      "type": "boolean",
      "title": "Skip This Stage",
      "description": "Controls whether this stage executes during pipeline processing at runtime. When set to `true`, the stage is completely bypassed and documents pass through unchanged to the next stage. Useful for A/B testing, gradual rollouts, or temporarily disabling a stage without removing it from the pipeline.",
      "default": false,
      "hints": ["advanced"]
    },
    "label": {
      "type": "string",
      "title": "Label",
      "description": "Human-readable identifier displayed in the Fusion Admin UI, monitoring dashboards, and log messages. Use descriptive labels like `Parse Product PDFs` to aid debugging and team collaboration. Labels appear in performance metrics and error reports, making it easier to identify which stage failed.",
      "hints": ["advanced"],
      "maxLength": 255
    },
    "condition": {
      "type": "string",
      "title": "Condition",
      "description": "JavaScript expression evaluated at runtime on each document to conditionally execute this stage. The expression must return `true` to execute or `false` to skip. Access document fields using `doc.getFieldValue('fieldName')` and request parameters via `request.getFirstParam('paramName')`. For example, `doc.getFieldValue('type') === 'premium'` executes this stage only for premium content.",
      "hints": ["code", "code/javascript", "advanced"]
    },
    "accountName": {
      "type": "string",
      "title": "Account Name",
      "description": "Specifies the Lucidworks AI account name registered in the Lucidworks AI Gateway Service. This value must match an existing account. An invalid name causes all vectorization calls to fail. Use the account selector to choose from configured accounts.",
      "hints": ["enumUrl:/api/query-stages/lwai-accounts"]
    },
    "modelType": {
      "type": "string",
      "title": "Model",
      "description": "Specifies the Lucidworks AI embedding model to invoke for text-to-vector encoding. The model must be configured in the Lucidworks AI platform under the selected account. Only embedding-type models are supported for vectorization.",
      "hints": ["enumUrl:/api/query-stages/lwai-model?account=${accountName}&useCase=embedding"]
    },
    "sourceFieldName": {
      "type": "string",
      "title": "Source Field",
      "description": "Specifies the document field whose string value is sent to the embedding model for vectorization. Supports template expressions such as `<doc.title_t>` and `<ctx.myVar>` for dynamic field resolution. The stage skips processing if the field is absent or `null`."
    },
    "destinationFieldName": {
      "type": "string",
      "title": "Destination Field",
      "description": "Specifies the document field where the dense vector returned by the model is stored. The field must be defined as a dense vector type in the Solr schema to be indexed correctly. Storing the vector in an incompatible field type causes indexing errors."
    },
    "useCaseConfig": {
      "type": "array",
      "title": "Use Case Configuration",
      "default": [{
        "key": "dataType",
        "value": "passage"
      }],
      "items": {
        "type": "object",
        "required": ["key"],
        "properties": {
          "key": {
            "type": "string",
            "title": "Parameter Name"
          },
          "value": {
            "type": "string",
            "title": "Parameter Value"
          }
        }
      }
    },
    "modelConfig": {
      "type": "array",
      "title": "Model Configuration",
      "items": {
        "type": "object",
        "required": ["key"],
        "properties": {
          "key": {
            "type": "string",
            "title": "Parameter Name"
          },
          "value": {
            "type": "string",
            "title": "Parameter Value"
          }
        }
      },
      "description": "Sends additional key-value configuration parameters to the Lucidworks AI model alongside the vectorization request. Use this to supply model-specific options not covered by the standard configuration. Each entry requires a `key` and an optional `value`."
    },
    "maxBatchSize": {
      "type": "integer",
      "title": "Maximum Batch Size",
      "description": "Sets the maximum number of documents grouped into a single batch request sent to the Lucidworks AI model. Larger batches improve throughput but must not exceed the model's supported batch limit. Reduce this value if the model returns errors about oversized requests.",
      "default": 16,
      "maximum": 32,
      "exclusiveMaximum": false,
      "minimum": 1,
      "exclusiveMinimum": false
    },
    "maxBatchDelay": {
      "type": "integer",
      "title": "Maximum Batch Delay (ms)",
      "description": "Sets the maximum time in milliseconds the stage waits to accumulate a full batch before sending available documents for vectorization. When the delay elapses, the current partial batch is sent regardless of size. Use a lower value to reduce indexing latency at the cost of smaller, less efficient batches.",
      "default": 1000,
      "minimum": 0,
      "exclusiveMinimum": false
    },
    "failOnError": {
      "type": "boolean",
      "title": "Fail on Error",
      "description": "Controls whether a vectorization error causes the entire pipeline to fail. When `true`, any model error throws an exception and stops the document. When `false`, errors are logged and the document continues. Set to `true` in production to surface model connectivity issues immediately.",
      "default": false
    }
  },
  "category": "AI",
  "categoryPriority": 10,
  "unsafe": false
};

export const SchemaParamFields = ({schema}) => {
  const sanitize = str => {
    if (typeof str !== "string") return str;
    return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'");
  };
  const renderMd = str => {
    const s = sanitize(str);
    const text = (/[.!?]\)*$/).test(s) ? s : `${s}.`;
    return text.split(/(\*\*[^*]+\*\*|_[^_]+_|`[^`]+`)/g).map((part, i) => {
      if (part.startsWith("**")) return <strong key={i}>{part.slice(2, -2)}</strong>;
      if (part.startsWith("_")) return <em key={i}>{part.slice(1, -1)}</em>;
      if (part.startsWith("`")) return <code key={i}>{part.slice(1, -1)}</code>;
      return part;
    });
  };
  const {description, properties = {}, required: requiredProps = []} = schema;
  const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]);
  const renderProp = ([name, prop]) => {
    const isRequired = requiredProps.includes(name);
    const hasDefault = prop.default !== undefined;
    const rawDefault = prop.default;
    const hints = prop.hints || [];
    const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"')));
    const postBadges = [];
    if (prop.title) {
      postBadges.push(<><span className="text-stone-400 dark:text-stone-500">API property: </span>{name}</>);
    }
    const constraints = [];
    if (prop.minimum !== undefined && prop.maximum !== undefined) {
      constraints.push(`Range: ${prop.minimum} – ${prop.maximum}`);
    } else if (prop.minimum !== undefined) {
      constraints.push(`Min: ${prop.minimum}`);
    } else if (prop.maximum !== undefined) {
      constraints.push(`Max: ${prop.maximum}`);
    }
    if (prop.minLength !== undefined && prop.maxLength !== undefined) {
      constraints.push(`Length: ${prop.minLength} – ${prop.maxLength}`);
    } else if (prop.minLength !== undefined) {
      constraints.push(`Min length: ${prop.minLength}`);
    } else if (prop.maxLength !== undefined) {
      constraints.push(`Max length: ${prop.maxLength}`);
    }
    const fieldProps = {
      key: name,
      body: prop.title || name,
      type: prop.type,
      ...postBadges.length > 0 && ({
        post: postBadges
      }),
      ...isRequired && ({
        required: true
      }),
      ...!isComplexDefault && hasDefault ? {
        default: sanitize(String(rawDefault))
      } : {}
    };
    const isObject = prop.type === "object" && prop.properties;
    const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties;
    return <ParamField {...fieldProps}>
        {prop.description && <p>{renderMd(prop.description)}</p>}

        {prop.enum && <p>
            Allowed values: 
            {prop.enum.map((v, i) => <>{i > 0 && ", "}<code key={i}>{String(v)}</code></>)}
          </p>}

        {constraints.length > 0 && <p className="text-stone-500 dark:text-stone-400 text-sm">
            {constraints.join(" · ")}
          </p>}

        {isComplexDefault && <div className="flex">
            <p>
              <strong>Default:</strong>
            </p>
            <pre className="!my-0">
              <code>
                {JSON.stringify(rawDefault, null, 2)}
              </code>
            </pre>
          </div>}

        {isArrayOfObjects && <Expandable title="item properties">
            <SchemaParamFields schema={{
      properties: prop.items.properties,
      required: prop.items.required
    }} />
          </Expandable>}

        {isObject && <Expandable title="properties">
            <SchemaParamFields schema={{
      properties: prop.properties,
      required: prop.required
    }} />
          </Expandable>}
      </ParamField>;
  };
  return <div>
      {description && <p>{renderMd(description)}</p>}

      {visibleProps.map(renderProp)}
    </div>;
};

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lwai-batch-vectorize

[mintlify link]: https://doc.lucidworks.com/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/lwai-batch-vectorize

<Note>
  This feature is only available in Fusion 5.9.x for versions 5.9.15 and later.
</Note>

The LWAI Batch Vectorize Index Stage is an integration between Lucidworks Search and Lucidworks AI to enrich your index with [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) predictions. This stage processes your documents in batches before sending them to the next stage in your index pipeline. Use this stage instead of the LWAI Vectorize Field index stage if you have many documents to process.

All the field names and values for the LWAI Vectorize Field index stage are maintained in the LWAI Batch Vectorize index stage, so you can switch from the LWAI Vectorize Field stage while maintaining your existing index pipeline.

To use this stage, non-admin Lucidworks Search users must be granted the `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**` permission in Lucidworks Search, which is the Lucidworks AI API Account Name defined in [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway) when this stage is configured.

<LwTemplate />

## Configurable batch length and waiting time

In order to use the LWAI Batch Vectorize index stage, you must configure two fields in addition to the fields that are also included in the LWAI Vectorize Field stage.

The **Maximum Batch Size** field allows you to configure how many documents to include in each batch when batch processing documents. The value should not exceed the maximum value supported by your model. The Lucidworks AI maximum value is 32.

The **Maximum Batch Delay** field allows you to configure the maximum time to wait before sending a batch of documents to be vectorized. If the batch is not full within this time, the current batch will be sent regardless of size. This setting helps to balance latency and throughput while batch processing documents. The default value is 1000 milliseconds, or 1 second.

## Configuration

<Tip>
  When entering configuration values in the UI, use *unescaped* characters, such as `\t` for the tab character. When entering configuration values in the API, use *escaped* characters, such as `\\t` for the tab character.
</Tip>

<SchemaParamFields schema={schema} />