> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# HTML

> Parser stage configuration specifications

export const schema = {
  "type": "object",
  "title": "HTML",
  "description": "Parses HTML content with support for element extraction via CSS selectors, link detection, metatag extraction, and content filtering using Jsoup. Maps specific HTML elements and attributes to document fields via CSS selector rules. Can extract body text, anchor links, metatags, and structured data from web pages.",
  "required": ["charset", "type"],
  "properties": {
    "id": {
      "type": "string",
      "title": "Parser ID",
      "default": "ea1f6c9b-e5e1-4f15-a887-7d0a52833d23"
    },
    "label": {
      "type": "string",
      "title": "Label",
      "description": "Human-readable identifier displayed in the Fusion Admin UI, monitoring dashboards, and log messages. Use descriptive labels like `Parse Product PDFs` to aid debugging and team collaboration. Labels appear in performance metrics and error reports, making it easier to identify which stage failed.",
      "maxLength": 255
    },
    "enabled": {
      "type": "boolean",
      "title": "Enable this Parser Stage",
      "default": true,
      "description": "Controls whether this parser stage is active and available for use. When `false`, the stage is completely inactive regardless of other settings. When `true`, the stage runs according to its other configuration options."
    },
    "mediaTypes": {
      "type": "array",
      "title": "Media Types to match",
      "description": "Specifies the media types this parser stage handles. Documents with a matching media type are routed to this stage for parsing. See `inheritMediaTypes` to combine this list with the stage's built-in defaults.",
      "items": {
        "type": "string",
        "pattern": "^[^\\/]+\\/[^\\/]+$",
        "format": "rfc2646"
      }
    },
    "inheritMediaTypes": {
      "type": "boolean",
      "title": "Match default media types in this Parser Stage",
      "description": "Controls whether this stage combines its built-in default media types with those in `mediaTypes`. When `true`, both lists are merged. When `false`, only the `mediaTypes` list is used and must contain at least one entry. Set to `false` to override the default media types entirely.",
      "default": true
    },
    "ignoredMediaTypes": {
      "type": "array",
      "title": "Media Types to ignore",
      "description": "Specifies media types this parser stage excludes from processing. Documents matching an ignored media type are skipped even if they match `mediaTypes`. Use this to carve out exceptions from a broadly matched media type set.",
      "items": {
        "type": "string",
        "pattern": "^[^\\/]+\\/[^\\/]+$",
        "format": "rfc2646"
      }
    },
    "pathPatterns": {
      "type": "array",
      "title": "File names to parse",
      "description": "Restricts this parser stage to files whose names match the specified pattern. Use forward slashes (`/`) to join archive names with entry names when matching files inside archives. If no pattern is specified, the stage applies to all matching media types.",
      "items": {
        "type": "object",
        "properties": {
          "syntax": {
            "type": "string",
            "title": "Pattern type",
            "description": "glob uses bash shell-style wildcards and regex uses Java (PCRE-style) regex.",
            "enum": ["glob", "regex"],
            "default": "glob"
          },
          "pattern": {
            "type": "string",
            "title": "File name or pattern",
            "description": "glob examples are \"z.txt\" or \"*.md\" or \"/a/*/b/f.txt\". regex examples are \"z.txt$\" or \".*\\.txt$\" or \"^/a/[^\\/]*/b/f.txt$\"."
          }
        }
      }
    },
    "errorHandling": {
      "type": "string",
      "title": "Error Handling",
      "enum": ["ignore", "log", "fail", "mark"],
      "default": "mark"
    },
    "outputFieldPrefix": {
      "type": "string",
      "title": "Prefix parsed fields with",
      "description": "Sets a string prefix applied to all fields extracted by this parser, useful for namespacing or avoiding field name collisions. For example, `tika_` produces fields like `tika_title` and `tika_author`. Leave empty to apply no prefix.",
      "maxLength": 20,
      "pattern": "^$|^[A-Za-z_][A-Za-z0-9_\\-\\.]+$"
    },
    "charset": {
      "type": "string",
      "title": "Character Set",
      "description": "Specifies the character encoding used to read file content. Common values include `UTF-8`, `ISO-8859-1`, and `Windows-1252`. If incorrect, text may appear garbled or cause parsing errors.",
      "default": "detect"
    },
    "recordSelector": {
      "type": "string",
      "title": "Record Selector"
    },
    "keepParent": {
      "type": "boolean",
      "title": "Keep Parent Document?",
      "description": "Controls whether the original parent document is indexed when a Record Selector splits HTML into multiple child documents. When `true`, both the parent and each child document are indexed. When `false`, only child documents are indexed. Set to `false` when the parent document would duplicate content already captured in the children.",
      "default": true
    },
    "excludeFilters": {
      "type": "array",
      "title": "Exclude filters",
      "description": "Specifies CSS/Jsoup selector expressions identifying HTML elements to remove before processing. Use this to exclude navigation menus, advertisements, footers, and other boilerplate content that should not appear in field mappings, link extraction, or body text. For example, `nav, footer, .ads` removes those elements from the parsed DOM.",
      "items": {
        "type": "string"
      }
    },
    "filterBeforeMapping": {
      "type": "boolean",
      "title": "Filter before mapping",
      "description": "Controls whether `excludeFilters` are applied before running field mappings. When `true`, excludeFilters clean the HTML first, then mappings run on the filtered result. When `false`, mappings run on the original HTML before filters are applied. Enable this to prevent excluded elements from appearing in mapped fields.",
      "default": false
    },
    "filterBeforeExtractingLinks": {
      "type": "boolean",
      "title": "Filter before extracting links",
      "description": "Controls whether `excludeFilters` are applied before extracting links. When `true`, excludeFilters clean the HTML first, then links are extracted from the filtered result. When `false`, links are extracted from the original HTML before filters are applied. Enable this to prevent links in excluded elements from being collected.",
      "default": false
    },
    "mappings": {
      "type": "array",
      "title": "HTML Element Mappings",
      "items": {
        "type": "object",
        "required": ["selectRule", "field"],
        "properties": {
          "selectRule": {
            "type": "string",
            "title": "Select Rule",
            "description": "A jsoup selection rule, for example 'div#foo' to select '<div id=\"foo\">...</div>'."
          },
          "attribute": {
            "type": "string",
            "title": "Attribute to map",
            "description": "What attribute of the selected element to map. For example 'href' to get the link URL from an '<a>' tag. Special values are '.outerText', '.html', '.outerHtml' and '.data'."
          },
          "field": {
            "type": "string",
            "title": "Target Field",
            "description": "Specifies the field name(s) for target field operations in the document processing pipeline."
          },
          "multivalued": {
            "type": "boolean",
            "title": "Multi-valued",
            "description": "Set to `true` to map multiple elements if there is more than one match for the select rule.",
            "default": false
          }
        }
      },
      "description": "Defines CSS/Jsoup selector-based rules that map specific HTML elements or attributes to named document fields. Each mapping specifies a selector to locate content and a target field name to store the extracted value. Configure multiple mappings to extract title, headings, custom attributes, or structured content into separate searchable fields."
    },
    "extractHtmlLinks": {
      "type": "boolean",
      "title": "Extract HTML links",
      "description": "Extract hyperlinks from HTML anchor tags (<a href=\"...\">) and related link elements in the document. When enabled, URLs are collected and stored for downstream processing such as web crawling or link analysis. When disabled (default), links are not extracted.",
      "default": false
    },
    "extractBodyText": {
      "type": "boolean",
      "title": "Extract body as a text",
      "description": "Extract plain text content from the HTML document body, stripping all HTML tags and formatting. When enabled (default), the visible text from the HTML is stored in the document for full-text search indexing. When disabled, only explicitly mapped fields via mappings configuration are extracted.",
      "default": true
    },
    "contentExtractionConfig": {
      "type": "object",
      "title": "Content Extraction (Experimental)",
      "description": "Experimental feature that uses heuristics to identify and extract the main content from HTML pages while filtering out navigation, sidebars, advertisements, and other boilerplate elements. Useful for web pages with significant non-content HTML, but accuracy varies by website structure. May not work reliably for all sites.",
      "properties": {
        "extractContent": {
          "type": "boolean",
          "title": "Extract page content",
          "default": true
        },
        "extractMetadata": {
          "type": "boolean",
          "title": "Extract metadata",
          "default": true
        },
        "preserveContentFormat": {
          "type": "boolean",
          "title": "Store readable html version of body content",
          "default": false
        },
        "metadataPrefix": {
          "type": "string",
          "title": "Optional prefix for rich content and extracted metadata",
          "description": "String prepended to optional prefix for rich content and extracted metadata for namespacing, avoiding collisions, or creating hierarchical structures. Empty string means no prefix."
        }
      }
    },
    "metatagsPrefix": {
      "type": "string",
      "title": "Optional prefix for metatags captured html document",
      "description": "Sets a string prefix applied to field names extracted from HTML metatags, useful for namespacing or avoiding field name collisions. For example, `meta_` produces fields like `meta_description` and `meta_keywords`. An empty string stores metatag fields without a prefix.",
      "default": "false"
    },
    "type": {
      "type": "string",
      "enum": ["html"],
      "default": "html"
    }
  },
  "additionalProperties": false,
  "category": "Other",
  "categoryPriority": 1,
  "unsafe": false
};

export const SchemaParamFields = ({schema}) => {
  const sanitize = str => {
    if (typeof str !== "string") return str;
    return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'");
  };
  const renderMd = str => {
    const s = sanitize(str);
    const text = (/[.!?]\)*$/).test(s) ? s : `${s}.`;
    return text.split(/(\*\*[^*]+\*\*|_[^_]+_|`[^`]+`)/g).map((part, i) => {
      if (part.startsWith("**")) return <strong key={i}>{part.slice(2, -2)}</strong>;
      if (part.startsWith("_")) return <em key={i}>{part.slice(1, -1)}</em>;
      if (part.startsWith("`")) return <code key={i}>{part.slice(1, -1)}</code>;
      return part;
    });
  };
  const {description, properties = {}, required: requiredProps = []} = schema;
  const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]);
  const renderProp = ([name, prop]) => {
    const isRequired = requiredProps.includes(name);
    const hasDefault = prop.default !== undefined;
    const rawDefault = prop.default;
    const hints = prop.hints || [];
    const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"')));
    const postBadges = [];
    if (prop.title) {
      postBadges.push(<><span className="text-stone-400 dark:text-stone-500">API property: </span>{name}</>);
    }
    const constraints = [];
    if (prop.minimum !== undefined && prop.maximum !== undefined) {
      constraints.push(`Range: ${prop.minimum} – ${prop.maximum}`);
    } else if (prop.minimum !== undefined) {
      constraints.push(`Min: ${prop.minimum}`);
    } else if (prop.maximum !== undefined) {
      constraints.push(`Max: ${prop.maximum}`);
    }
    if (prop.minLength !== undefined && prop.maxLength !== undefined) {
      constraints.push(`Length: ${prop.minLength} – ${prop.maxLength}`);
    } else if (prop.minLength !== undefined) {
      constraints.push(`Min length: ${prop.minLength}`);
    } else if (prop.maxLength !== undefined) {
      constraints.push(`Max length: ${prop.maxLength}`);
    }
    const fieldProps = {
      key: name,
      body: prop.title || name,
      type: prop.type,
      ...postBadges.length > 0 && ({
        post: postBadges
      }),
      ...isRequired && ({
        required: true
      }),
      ...!isComplexDefault && hasDefault ? {
        default: sanitize(String(rawDefault))
      } : {}
    };
    const isObject = prop.type === "object" && prop.properties;
    const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties;
    return <ParamField {...fieldProps}>
        {prop.description && <p>{renderMd(prop.description)}</p>}

        {prop.enum && <p>
            Allowed values: 
            {prop.enum.map((v, i) => <>{i > 0 && ", "}<code key={i}>{String(v)}</code></>)}
          </p>}

        {constraints.length > 0 && <p className="text-stone-500 dark:text-stone-400 text-sm">
            {constraints.join(" · ")}
          </p>}

        {isComplexDefault && <div className="flex">
            <p>
              <strong>Default:</strong>
            </p>
            <pre className="!my-0">
              <code>
                {JSON.stringify(rawDefault, null, 2)}
              </code>
            </pre>
          </div>}

        {isArrayOfObjects && <Expandable title="item properties">
            <SchemaParamFields schema={{
      properties: prop.items.properties,
      required: prop.items.required
    }} />
          </Expandable>}

        {isObject && <Expandable title="properties">
            <SchemaParamFields schema={{
      properties: prop.properties,
      required: prop.required
    }} />
          </Expandable>}
      </ParamField>;
  };
  return <div>
      {description && <p>{renderMd(description)}</p>}

      {visibleProps.map(renderProp)}
    </div>;
};

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/lucidworks-search/09-developer-documentation/config-specs/parsers/html-parser

[mintlify link]: https://doc.lucidworks.com/docs/lucidworks-search/09-developer-documentation/config-specs/parsers/html-parser

[old doc.lw link]: https://doc.lucidworks.com/managed-fusion/5.9/rcjl69

This parser stage processes the following HTML elements:

* `<title>`
* `<body>` (with tags removed)
* `<meta>`
* `<a>` and `<link>`

Additionally, you can configure [JSoup selectors](https://jsoup.org/cookbook/extracting-data/) to extract specific HTML and CSS elements from a document and map them to PipelineDocument fields.
For example, you could use this to process navigational `DIV` elements one way, then process content-ful `DIV` elements another way.

HTML Parser Stage supports jsoup selector methods by allowing the following special attribute values starting with `.` in `HtmlMappingRule`:

* `.text`: [jsoup Element text method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#text%28%29)
* `.ownText`: [jsoup Element ownText method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#ownText%28%29)
* `.html`: [jsoup Element html method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#html%28%29)
* `.outerHtml`: [jsoup Element outerHtml method](https://jsoup.org/apidocs/org/jsoup/nodes/Node.html#outerHtml%28%29)
* `.data`: [jsoup Element data method](https://jsoup.org/apidocs/org/jsoup/nodes/Element.html#data%28%29)

<Note>
  The HTML Transformation index pipeline stage is deprecated in favor of this parser stage.
</Note>

HTML and CSS elements can be selected for extraction into new documents or fields:

* To create new documents from selected elements, configure `recordSelector`.
* To create new fields from selected elements, configure `mappings`.

Title, body, metadata, and links are only populated in the parent document. Both of these parameters support [JSoup selectors](https://jsoup.org/cookbook/extracting-data/), which provides a rich syntax for selecting HTML and CSS elements.

<Tip>
  When entering configuration values in the UI, use *unescaped* characters, such as `\t` for the tab character. When entering configuration values in the API, use *escaped* characters, such as `\\t` for the tab character.
</Tip>

<LwTemplate />

## HTML content extraction

The Context Extraction setting in the HTML Parser has a special meaning.

By default, when unchecked or false, the HTML parser will essentially attempt to extract the text of the entire HTML page as the text that will be used in the Solr document.

However, when Content Extract is checked, or true, a set of heuristic rules are applied to attempt to automatically determine what node in the entire page is most likely to be the page content, based on the tree of nodes inside it and the text component of all the sub-nodes.

This may be helpful when you have a variety of different page formats, and also significant text on the pages which is not useful to add to the documents, such as significant text in the page header or footer.

However, since the algorithm is based on heuristics , it is possible that the results could change as the site is altered. When you need a high degree of certainty, we recommend that you use explicit rules of which nodes to extract to match your local configuration.

<SchemaParamFields schema={schema} />
