> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Detect Language

> Index pipeline stage configuration specifications

export const schema = {
  "type": "object",
  "title": "Detect Language",
  "description": "Detects the natural language of document text fields using statistical models and character n-gram analysis. Supports 50+ languages and provides confidence scores for each detection result. Enables language-aware routing, appropriate analyzer selection, and multilingual content organization.",
  "required": ["source", "outputType"],
  "properties": {
    "skip": {
      "type": "boolean",
      "title": "Skip This Stage",
      "description": "Controls whether this stage executes during pipeline processing at runtime. When set to `true`, the stage is completely bypassed and documents pass through unchanged to the next stage. Useful for A/B testing, gradual rollouts, or temporarily disabling a stage without removing it from the pipeline.",
      "default": false,
      "hints": ["advanced"]
    },
    "label": {
      "type": "string",
      "title": "Label",
      "description": "Human-readable identifier displayed in the Fusion Admin UI, monitoring dashboards, and log messages. Use descriptive labels like `Parse Product PDFs` to aid debugging and team collaboration. Labels appear in performance metrics and error reports, making it easier to identify which stage failed.",
      "hints": ["advanced"],
      "maxLength": 255
    },
    "condition": {
      "type": "string",
      "title": "Condition",
      "description": "JavaScript expression evaluated at runtime on each document to conditionally execute this stage. The expression must return `true` to execute or `false` to skip. Access document fields using `doc.getFieldValue('fieldName')` and request parameters via `request.getFirstParam('paramName')`. For example, `doc.getFieldValue('type') === 'premium'` executes this stage only for premium content.",
      "hints": ["code", "code/javascript", "advanced"]
    },
    "source": {
      "type": "array",
      "title": "Source",
      "description": "Specifies the field names or context keys containing the text to analyze for language detection. Supports String Template syntax for dynamic field references.",
      "minItems": 1,
      "items": {
        "type": "string"
      }
    },
    "languages": {
      "type": "array",
      "title": "Languages",
      "description": "Specifies the language profiles (as BCP 47 language codes) to use during detection, limiting detection to the listed languages. Leave empty to detect from all available language profiles.",
      "items": {
        "type": "object",
        "required": ["code"],
        "properties": {
          "code": {
            "type": "string",
            "title": "Language code ",
            "enum": ["af", "an", "ar", "ast", "be", "br", "ca", "bg", "bn", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", "fr", "ga", "gl", "gu", "he", "hi", "hr", "ht", "hu", "id", "is", "it", "ja", "km", "kn", "ko", "lt", "lv", "mk", "ml", "mr", "ms", "mt", "ne", "nl", "no", "oc", "pa", "pl", "pt", "ro", "ru", "sk", "sl", "so", "sq", "sr", "sv", "sw", "ta", "te", "th", "tl", "tr", "uk", "ur", "vi", "wa", "yi", "zh-CN", "zh-TW"]
          }
        }
      }
    },
    "outputKey": {
      "type": "string",
      "title": "Output Key",
      "description": "Specifies the pipeline context key where detection results are stored when `outputType` is set to `context`. The value is a map of source field name to detected language code.",
      "default": "languages"
    },
    "documentPostfix": {
      "type": "string",
      "title": "Document Postfix",
      "description": "Specifies the suffix appended to the source field name when storing detection results on the document. For example, a postfix of `_lang` stores the result in `body_lang` for source field `body`.",
      "default": "_lang"
    },
    "outputType": {
      "type": "string",
      "title": "Output Type",
      "description": "Specifies whether detection results are stored on the document or in the pipeline context. Use `document` to add a language field to the indexed document, or `context` to pass results to downstream stages without indexing.",
      "enum": ["document", "context"],
      "default": "document"
    },
    "minimumConfidence": {
      "type": "number",
      "title": "Minimum confidence",
      "description": "Sets the minimum confidence score (0–1) required for a language to be reported. Detections below this threshold are discarded, reducing false positives in ambiguous or short text. For example, `0.7` requires 70% confidence.",
      "default": 0.5
    },
    "returnAllMatchedWithConfidenceScores": {
      "type": "boolean",
      "title": "Return all detected languages and their confidence scores.",
      "description": "Controls whether all languages exceeding the minimum confidence score are returned, along with their scores. When `true`, multiple candidate languages are reported. When `false`, only the top-scoring language is returned.",
      "default": false
    }
  },
  "category": "Document Filtering and Enrichment",
  "categoryPriority": 8,
  "unsafe": false
};

export const SchemaParamFields = ({schema}) => {
  const sanitize = str => {
    if (typeof str !== "string") return str;
    return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'");
  };
  const renderMd = str => {
    const s = sanitize(str);
    const text = (/[.!?]\)*$/).test(s) ? s : `${s}.`;
    return text.split(/(\*\*[^*]+\*\*|_[^_]+_|`[^`]+`)/g).map((part, i) => {
      if (part.startsWith("**")) return <strong key={i}>{part.slice(2, -2)}</strong>;
      if (part.startsWith("_")) return <em key={i}>{part.slice(1, -1)}</em>;
      if (part.startsWith("`")) return <code key={i}>{part.slice(1, -1)}</code>;
      return part;
    });
  };
  const {description, properties = {}, required: requiredProps = []} = schema;
  const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]);
  const renderProp = ([name, prop]) => {
    const isRequired = requiredProps.includes(name);
    const hasDefault = prop.default !== undefined;
    const rawDefault = prop.default;
    const hints = prop.hints || [];
    const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"')));
    const postBadges = [];
    if (prop.title) {
      postBadges.push(<><span className="text-stone-400 dark:text-stone-500">API property: </span>{name}</>);
    }
    const constraints = [];
    if (prop.minimum !== undefined && prop.maximum !== undefined) {
      constraints.push(`Range: ${prop.minimum} – ${prop.maximum}`);
    } else if (prop.minimum !== undefined) {
      constraints.push(`Min: ${prop.minimum}`);
    } else if (prop.maximum !== undefined) {
      constraints.push(`Max: ${prop.maximum}`);
    }
    if (prop.minLength !== undefined && prop.maxLength !== undefined) {
      constraints.push(`Length: ${prop.minLength} – ${prop.maxLength}`);
    } else if (prop.minLength !== undefined) {
      constraints.push(`Min length: ${prop.minLength}`);
    } else if (prop.maxLength !== undefined) {
      constraints.push(`Max length: ${prop.maxLength}`);
    }
    const fieldProps = {
      key: name,
      body: prop.title || name,
      type: prop.type,
      ...postBadges.length > 0 && ({
        post: postBadges
      }),
      ...isRequired && ({
        required: true
      }),
      ...!isComplexDefault && hasDefault ? {
        default: sanitize(String(rawDefault))
      } : {}
    };
    const isObject = prop.type === "object" && prop.properties;
    const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties;
    return <ParamField {...fieldProps}>
        {prop.description && <p>{renderMd(prop.description)}</p>}

        {prop.enum && <p>
            Allowed values: 
            {prop.enum.map((v, i) => <>{i > 0 && ", "}<code key={i}>{String(v)}</code></>)}
          </p>}

        {constraints.length > 0 && <p className="text-stone-500 dark:text-stone-400 text-sm">
            {constraints.join(" · ")}
          </p>}

        {isComplexDefault && <div className="flex">
            <p>
              <strong>Default:</strong>
            </p>
            <pre className="!my-0">
              <code>
                {JSON.stringify(rawDefault, null, 2)}
              </code>
            </pre>
          </div>}

        {isArrayOfObjects && <Expandable title="item properties">
            <SchemaParamFields schema={{
      properties: prop.items.properties,
      required: prop.items.required
    }} />
          </Expandable>}

        {isObject && <Expandable title="properties">
            <SchemaParamFields schema={{
      properties: prop.properties,
      required: prop.required
    }} />
          </Expandable>}
      </ParamField>;
  };
  return <div>
      {description && <p>{renderMd(description)}</p>}

      {visibleProps.map(renderProp)}
    </div>;
};

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/detect-language

[mintlify link]: https://doc.lucidworks.com/docs/lucidworks-search/09-developer-documentation/config-specs/index-pipeline-stages/detect-language

[old doc.lw link]: https://doc.lucidworks.com/managed-fusion/5.9/o704ak

The Detect Language index stage operates over one or more fields in the Pipeline Document.
The contents of each field are analyzed using the
[Language Detection Library for Java](https://github.com/optimaize/language-detector), which is an open source project hosted on GitHub.
The analyzer returns the ID of the language which best matches the contents of that field, if any.
The stage can return these IDs as annotations on the Pipeline Document context, or as annotation on each field analyzed.

The language identification algorithm breaks the text in each source field into n-grams and compares them to sets of n-grams compiled from all the different language versions of the Wikipedia.
This library only produces reasonable results for document fields which are comparable in length, vocabulary, and style to the known texts compiled from the Wikipedia.
Caveats are discussed below.

If a positive language identification is made, the stage writes the detected language information to the Pipeline Document based on the **Output Type** configuration property.
If the language annotation is added to the Pipeline Document context object, the name of the context key string is specified by the **Output Key** configuration property.

<Note>
  Although `documentPostfix` is primarily associated with document output, it must also be set when `outputType` is set to `context`. If `documentPostfix` is not specified, the detected language information may not be captured in the context output, even when `outputKey` is configured.
</Note>

For Output Type configuration property **Document**, per-field language annotations are added to the document using a parallel naming convention where the name of the language identification field starts with the name of the analyzed field and has an additional suffix string, default value `_lang`.
For example, if a document contains fields named `plot_summary_txt` and `user_reviews_txt` to be analyzed,
if the software can detect the language, it adds the fields `plot_summary_txt_lang` and `user_reviews_txt_lang`.

There is also an option to allow detection of multiple languages. This can be achieved by setting **Return all detected languages and their confidence scores** to `true`.
In this case, the detected languages is either set as document fields in a form of `Field Name_Document Postfix.Language:Confidence`, or as a field with name **Output Key** in the context having a dictionary of following form `{ "language":"probability" }` as a value.
Example Document fields could look like this: `plot_summary_txt_lang.pl_: [0.99]`, `plot_summary_txt_lang.en_: [0.99]` when languages `pl` and `en` would be detected.

<LwTemplate />

## Languages

The Language Detection Library for Java has build-in profiles for [many languages](https://github.com/optimaize/language-detector/blob/master/README.md#71-built-in-language-profiles). These are the language profiles that can be used as object attributes in the `languages` array. If there is a set of Wikipedia entries written in a language, it is likely that the Language Detection Library can identify texts written in this language.

## Caveats

This library should produce reasonable results on document fields which are comparable in length, vocabulary, and style to the known texts compiled from the Wikipedia.

The documentation lists the following [challenges](https://github.com/optimaize/language-detector/blob/master/README.md#challenges):

* This software does not work as well when the input text to analyze is short, or unclean. For example tweets.
* When a text is written in multiple languages, the default algorithm of this software is not appropriate. You can try to split the text (by sentence or paragraph) and detect the individual parts. Running the language guesser on the whole text will just tell you the language that is most dominant, in the best case.
* This software cannot handle it well when the input text is in none of the expected (and supported) languages.
* Detection of unwanted languages (for example the stage might detect some language that is not even used in the input data because of some language similarities). By default, the stage uses a full array of available languages for detection ([List here](https://github.com/optimaize/language-detector)). If one wants to only use selected languages, this can be configured

## Configuration

<Tip>
  When entering configuration values in the UI, use *unescaped* characters, such as `\t` for the tab character. When entering configuration values in the API, use *escaped* characters, such as `\\t` for the tab character.
</Tip>

<SchemaParamFields schema={schema} />