Regex Field Replacement Index Stage

The Regex Field Replacement index stage lets you match against a source field and append or overwrite that field or another one with a replacement string. It uses the java.util.regex library.

One notable feature is support for flags in the prefix of the pattern, such as (?iu)\w+ where the i flag means case-insensitive and u means Unicode-aware case folding. You can also configure the desired behavior when there is no match. Additionally, you can use capture groups in your replacement configuration using $1 to denote the first capture group.

Examples

Remove special characters from product SKUs

This example replaces each run of non-alphanumeric characters in a product_sku field with a hyphen:

{
  "id": "clean-product-skus",
  "type": "regex-field-replacement",
  "rules": [
    {
      "source": ["product_sku"],
      "pattern": "[^a-zA-Z0-9]+",
      "replacement": "-"
    }
  ]
}

• Input: product_sku: "ABC@123#XYZ"
• Output: product_sku: "ABC-123-XYZ"

Reformat phone numbers using capture groups

This example uses capture groups ($1, $2, $3) to reformat phone numbers from 5551234567 to [phone]:

{
  "id": "format-phone-numbers",
  "type": "regex-field-replacement",
  "rules": [
    {
      "source": ["phone"],
      "pattern": "(\\d{3})(\\d{3})(\\d{4})",
      "replacement": "($1) $2-$3"
    }
  ]
}

• Input: phone: "5551234567"
• Output: phone: "[phone]"

Case-insensitive URL normalization

This example uses the case-insensitive flag (?i) to normalize URLs by converting HTTP or http to https, and stores the result in a new field:

{
  "id": "normalize-urls",
  "type": "regex-field-replacement",
  "rules": [
    {
      "source": ["url"],
      "target": "secure_url",
      "pattern": "(?i)^http://",
      "replacement": "https://"
    }
  ]
}

• Input: url: "HTTP://example.com/page"
• Output: url: "HTTP://example.com/page", secure_url: "https://example.com/page"

Because target is set, the stage writes the result to secure_url and leaves the original url field unchanged.

Configuration

When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.