> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Query-to-Query Session-Based Similarity Jobs

export const schema = {
  "type": "object",
  "title": "Query-to-Query Session Based Similarity",
  "description": "Batch-computes query-to-query similarities using a co-occurrence approach based on shared clicked documents within sessions.",
  "required": ["id", "trainingCollection", "fieldToVectorize", "dataFormat", "docIdField", "type"],
  "properties": {
    "id": {
      "type": "string",
      "title": "Spark Job ID",
      "description": "The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_).",
      "maxLength": 63,
      "pattern": "[a-zA-Z][_\\-a-zA-Z0-9]*[a-zA-Z0-9]?"
    },
    "sparkConfig": {
      "type": "array",
      "title": "Spark Settings",
      "description": "Spark configuration settings.",
      "hints": ["advanced"],
      "items": {
        "type": "object",
        "required": ["key"],
        "properties": {
          "key": {
            "type": "string",
            "title": "Parameter Name"
          },
          "value": {
            "type": "string",
            "title": "Parameter Value"
          }
        }
      }
    },
    "trainingCollection": {
      "type": "string",
      "title": "Input Collection",
      "description": "Specifies the collection containing queries, document IDs, and event counts. Can be a signal aggregation collection or a signals collection."
    },
    "fieldToVectorize": {
      "type": "string",
      "title": "Query Field Name",
      "description": "Specifies the field containing query strings.",
      "default": "query_s",
      "minLength": 1
    },
    "dataFormat": {
      "type": "string",
      "title": "Data format",
      "description": "Specifies the Spark-compatible input data format, such as `solr`, `parquet`, or `orc`.",
      "default": "solr",
      "minLength": 1
    },
    "trainingDataFrameConfigOptions": {
      "type": "object",
      "title": "Dataframe Config Options",
      "description": "Sets additional key-value configuration options passed to the Spark DataFrame reader at load time.",
      "properties": {},
      "additionalProperties": {
        "type": "string"
      },
      "hints": ["advanced"]
    },
    "trainingDataFilterQuery": {
      "type": "string",
      "title": "Data filter query",
      "description": "Filters the input collection using a Solr query or SQL expression.",
      "default": "*:*",
      "hints": ["dummy"]
    },
    "sparkSQL": {
      "type": "string",
      "title": "Spark SQL filter query",
      "description": "Filters the loaded DataFrame using a Spark SQL query before processing. The input data is registered as a temporary table named `spark_input`.",
      "default": "SELECT * from spark_input",
      "hints": ["code/sql", "advanced"]
    },
    "trainingDataSamplingFraction": {
      "type": "number",
      "title": "Training data sampling fraction",
      "description": "Sets the fraction of training data to sample before processing. Use a value between `0` and `1`. `1.0` uses all available data.",
      "default": 1,
      "hints": ["advanced"],
      "maximum": 1,
      "exclusiveMaximum": false
    },
    "randomSeed": {
      "type": "integer",
      "title": "Random seed",
      "description": "Sets the random seed for deterministic operations including sampling and initialization. Fix this value to reproduce identical results across runs.",
      "default": 1234,
      "hints": ["advanced"]
    },
    "outputCollection": {
      "type": "string",
      "title": "Output collection",
      "description": "Specifies the collection where synonym and similar query pairs are stored.",
      "hints": ["dummy"]
    },
    "overwriteOutput": {
      "type": "boolean",
      "title": "Overwrite Output",
      "description": "When enabled, overwrites the output collection before writing new results.",
      "default": true,
      "hints": ["hidden", "advanced"]
    },
    "dataOutputFormat": {
      "type": "string",
      "title": "Data output format",
      "description": "Specifies the Spark-compatible output format, such as `solr`, `parquet`, or `orc`.",
      "default": "solr",
      "hints": ["advanced"],
      "minLength": 1
    },
    "sourceFields": {
      "type": "string",
      "title": "Fields to Load",
      "description": "Specifies the document fields to load from the input source.",
      "hints": ["dummy", "hidden"]
    },
    "partitionCols": {
      "type": "string",
      "title": "Partition fields",
      "description": "Specifies a comma-delimited list of column names used to partition output when writing to non-Solr sinks.",
      "hints": ["advanced"]
    },
    "writeOptions": {
      "type": "array",
      "title": "Write Options",
      "description": "Sets additional key-value options passed to the Spark writer when writing output to Solr or other sinks.",
      "hints": ["advanced"],
      "items": {
        "type": "object",
        "required": ["key"],
        "properties": {
          "key": {
            "type": "string",
            "title": "Parameter Name"
          },
          "value": {
            "type": "string",
            "title": "Parameter Value"
          }
        }
      }
    },
    "readOptions": {
      "type": "array",
      "title": "Read Options",
      "description": "Sets additional key-value options passed to the Spark reader when loading input from Solr or other sources.",
      "hints": ["advanced"],
      "items": {
        "type": "object",
        "required": ["key"],
        "properties": {
          "key": {
            "type": "string",
            "title": "Parameter Name"
          },
          "value": {
            "type": "string",
            "title": "Parameter Value"
          }
        }
      }
    },
    "specialCharsFilterString": {
      "type": "string",
      "title": "Special characters to be filtered out",
      "description": "Specifies a string of special characters to remove from query strings before processing.",
      "default": "~!@#$^%&*\\(\\)_+={}\\[\\]|;:\"'<,>.?`/\\\\-",
      "hints": ["advanced"]
    },
    "minQueryLength": {
      "type": "integer",
      "title": "Minimum query length",
      "description": "Sets the minimum query length in characters. Queries shorter than this are excluded from similarity computation.",
      "default": 3,
      "minimum": 1,
      "exclusiveMinimum": false
    },
    "maxQueryLength": {
      "type": "integer",
      "title": "Maximum query length",
      "description": "Sets the maximum query length in characters. Queries longer than this are excluded from similarity computation.",
      "default": 50,
      "minimum": 1,
      "exclusiveMinimum": false
    },
    "countField": {
      "type": "string",
      "title": "Event Count Field Name",
      "description": "Specifies the field name(s) for event count field name operations in the document processing pipeline. Default value is `count_i`.",
      "default": "count_i"
    },
    "docIdField": {
      "type": "string",
      "title": "Document id Field Name",
      "description": "Specifies the field name(s) for document id field name operations in the document processing pipeline. Default value is `doc_id_s`.",
      "default": "doc_id_s"
    },
    "overlapThreshold": {
      "type": "number",
      "title": "Query Similarity Threshold",
      "description": "Minimum threshold value for Query-to-Query Session Based Similarity. Values below this threshold are excluded or trigger alternative processing. Numeric value with valid range depending on context.",
      "default": 0.3,
      "hints": ["advanced"],
      "maximum": 1,
      "exclusiveMaximum": false
    },
    "minQueryCount": {
      "type": "integer",
      "title": "Query Clicks Threshold",
      "description": "The minimum number of clicked documents needed for comparing queries.",
      "default": 1,
      "hints": ["advanced"],
      "minimum": 1,
      "exclusiveMinimum": false
    },
    "overlapEnabled": {
      "type": "boolean",
      "title": "Boost on token overlap",
      "description": "Maximize score for query pairs with overlapping tokens by setting score to 1.",
      "default": true,
      "hints": ["advanced"]
    },
    "tokenOverlapValue": {
      "type": "number",
      "title": "Minimum match for token overlap",
      "description": "Minimum amount of overlap to consider for boosting. To specify overlap in terms of ratio, specify a value in (0, 1). To specify overlap in terms of exact count, specify a value >= 1.",
      "default": 1,
      "hints": ["advanced"]
    },
    "sessionIdField": {
      "type": "string",
      "title": "Session/User ID field",
      "description": "Specifies the field name(s) for session/user id field operations in the document processing pipeline. Default value is `session_id_s`.",
      "default": "session_id_s"
    },
    "minPairOccCount": {
      "type": "integer",
      "title": "Minimum query-recommendation pair occurrence count",
      "description": "Minimum number of times a query pair must be generated to be considered valid.",
      "default": 2,
      "hints": ["advanced"],
      "minimum": 1,
      "exclusiveMinimum": false
    },
    "stopwordsBlobName": {
      "type": "string",
      "title": "Stopwords Blob Store",
      "description": "Name of the stopwords blob resource. This is a .txt file with one stopword per line. By default the file is called stopwords/stopwords_nltk_en.txt however a custom file can also be used.",
      "default": "stopwords/stopwords_en.txt",
      "reference": "blob",
      "blobType": "file:spark"
    },
    "type": {
      "type": "string",
      "title": "Spark Job Type",
      "enum": ["similar_queries"],
      "default": "similar_queries",
      "hints": ["readonly"]
    }
  },
  "additionalProperties": true,
  "category": "Other",
  "categoryPriority": 1,
  "propertyGroups": [{
    "label": "Input/Output Parameters",
    "properties": ["trainingCollection", "outputCollection", "dataFormat", "trainingDataFilterQuery", "readOptions", "writeOptions", "trainingDataFrameConfigOptions", "trainingDataSamplingFraction", "randomSeed"]
  }, {
    "label": "Field Parameters",
    "properties": ["fieldToVectorize", "sourceFields", "countField", "docIdField", "sessionIdField"]
  }, {
    "label": "Model Tuning Parameters",
    "properties": ["minQueryLength", "maxQueryLength", "specialCharsFilterString", "stopwordsBlobName", "overlapThreshold", "overlapEnabled", "tokenOverlapValue", "minQueryCount", "minPairOccCount"]
  }]
};

export const SchemaParamFields = ({schema}) => {
  const sanitize = str => {
    if (typeof str !== "string") return str;
    return str.replace(/^"(.*)"$/s, "$1").replace(/\\/g, "").replace(/"/g, "'");
  };
  const renderMd = str => {
    const s = sanitize(str);
    const text = (/[.!?]\)*$/).test(s) ? s : `${s}.`;
    return text.split(/(\*\*[^*]+\*\*|_[^_]+_|`[^`]+`)/g).map((part, i) => {
      if (part.startsWith("**")) return <strong key={i}>{part.slice(2, -2)}</strong>;
      if (part.startsWith("_")) return <em key={i}>{part.slice(1, -1)}</em>;
      if (part.startsWith("`")) return <code key={i}>{part.slice(1, -1)}</code>;
      return part;
    });
  };
  const {description, properties = {}, required: requiredProps = []} = schema;
  const visibleProps = useMemo(() => Object.entries(properties).filter(([, prop]) => !prop.hints?.includes("hidden")), [properties]);
  const renderProp = ([name, prop]) => {
    const isRequired = requiredProps.includes(name);
    const hasDefault = prop.default !== undefined;
    const rawDefault = prop.default;
    const hints = prop.hints || [];
    const isComplexDefault = hasDefault && (typeof rawDefault === "object" || typeof rawDefault === "string" && (rawDefault.length > 20 || rawDefault.includes('"')));
    const postBadges = [];
    if (prop.title) {
      postBadges.push(<><span className="text-stone-400 dark:text-stone-500">API property: </span>{name}</>);
    }
    const constraints = [];
    if (prop.minimum !== undefined && prop.maximum !== undefined) {
      constraints.push(`Range: ${prop.minimum} – ${prop.maximum}`);
    } else if (prop.minimum !== undefined) {
      constraints.push(`Min: ${prop.minimum}`);
    } else if (prop.maximum !== undefined) {
      constraints.push(`Max: ${prop.maximum}`);
    }
    if (prop.minLength !== undefined && prop.maxLength !== undefined) {
      constraints.push(`Length: ${prop.minLength} – ${prop.maxLength}`);
    } else if (prop.minLength !== undefined) {
      constraints.push(`Min length: ${prop.minLength}`);
    } else if (prop.maxLength !== undefined) {
      constraints.push(`Max length: ${prop.maxLength}`);
    }
    const fieldProps = {
      key: name,
      body: prop.title || name,
      type: prop.type,
      ...postBadges.length > 0 && ({
        post: postBadges
      }),
      ...isRequired && ({
        required: true
      }),
      ...!isComplexDefault && hasDefault ? {
        default: sanitize(String(rawDefault))
      } : {}
    };
    const isObject = prop.type === "object" && prop.properties;
    const isArrayOfObjects = prop.type === "array" && prop.items?.type === "object" && prop.items.properties;
    return <ParamField {...fieldProps}>
        {prop.description && <p>{renderMd(prop.description)}</p>}

        {prop.enum && <p>
            Allowed values: 
            {prop.enum.map((v, i) => <>{i > 0 && ", "}<code key={i}>{String(v)}</code></>)}
          </p>}

        {constraints.length > 0 && <p className="text-stone-500 dark:text-stone-400 text-sm">
            {constraints.join(" · ")}
          </p>}

        {isComplexDefault && <div className="flex">
            <p>
              <strong>Default:</strong>
            </p>
            <pre className="!my-0">
              <code>
                {JSON.stringify(rawDefault, null, 2)}
              </code>
            </pre>
          </div>}

        {isArrayOfObjects && <Expandable title="item properties">
            <SchemaParamFields schema={{
      properties: prop.items.properties,
      required: prop.items.required
    }} />
          </Expandable>}

        {isObject && <Expandable title="properties">
            <SchemaParamFields schema={{
      properties: prop.properties,
      required: prop.required
    }} />
          </Expandable>}
      </ParamField>;
  };
  return <div>
      {description && <p>{renderMd(description)}</p>}

      {visibleProps.map(renderProp)}
    </div>;
};

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity

[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity

[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8802

This recommender is based on co-occurrence of queries in the context of clicked documents and sessions. It is useful when your data shows that users tend to search for similar items in a single search session. This method of generating query-to-query recommendations is faster and more reliable than the Query-to-Query Similarity recommender job, and is session-based unlike the similar queries previously generated as part of the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).

|                      |                                                                                                                                                                                           |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Default job name** | `COLLECTION_NAME_query_recs`                                                                                                                                                              |
| **Input**            | Raw signals (the `_<collection_signals` collection by default).                                                                                                                           |
| **Output**           | [Queries-for-query recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query) (the `COLLECTION_NAME_queries_query_recs` collection by default) |

|                          | query | count\_i | type              | timestamp\_tdt    | user\_id          | doc\_id | session\_id | fusion\_query\_id |
| ------------------------ | ----- | -------- | ----------------- | ----------------- | ----------------- | ------- | ----------- | ----------------- |
| Required signals fields: | ✅     | ✅        | See note 1 below. | See note 2 below. | See note 2 below. | ✅       | ✅           |                   |

**Note 1:** Required if you want to weight types differently.

**Note 2:** Required if you want to use time decay.

**Note 3:** Required if no `session_id` field is available. Either `user_id` or `session_id` is needed on response signals if doing any click path analysis from signals.

The job generates two sets of recommendations based on the two approaches described below, then merges and de-duplicates them to present unique query-recommender pairs.

| Similar queries based on documents clicked                                                                                                                                                                                                                                                                                                              | Similar queries based on co-occurrence in sessions                                                                                                                                                                                                                                                                |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Queries are considered for recommendation if two queries have similar sets of document IDs clicked according to the signals data. This is directly implemented from the similar queries portion of the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).  This approach can work on both raw and aggregated signals. | Queries are considered for recommendation if two queries have co-occurred in the same session based on the assumption that users search for similar items in a single search session (this may or may not hold true depending on the data).  This approach, based on session/user IDs, needs raw signals to work. |

<Frame caption="Query-to-Query Session-Based Similarity job dataflow">
  <img src="https://mintcdn.com/lucidworks/iN-DD0xMOO3PKUmX/assets/images/5.2/similar-queries-rec-dataflow-diagram.png?fit=max&auto=format&n=iN-DD0xMOO3PKUmX&q=85&s=d34f9feb81587b6d46097d4548d569b3" alt="Query-to-Query Session-Based Similarity job dataflow" width="841" height="89" data-path="assets/images/5.2/similar-queries-rec-dataflow-diagram.png" />
</Frame>

A default Query-to-Query Session-Based Similarity job (`COLLECTION_NAME_query_recs`) and a dedicated [collection](#the-similar-queries-collection) and [pipeline](#the-query-pipeline) are created when you [enable recommendations for a collection](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started).

At a minimum, you must configure these:

* an ID for this job
* the input collection containing the signals data, usually `COLLECTION_NAME_signals`
* the data format, usually `solr`
* the query field name, usually `query_s`
* the document ID field name, usually `doc_id_s`
* optionally, the user session ID or user ID field name

  <Note>
    If this field is not specified, then the job generates click-based recommendations only, without session-based recommendations.
  </Note>

<LwTemplate />

## Data tips

* Running the job on other types of data than signals is not recommended and may yield unexpected results.
* To get about 90% query coverage with the query pipeline, we recommend a raw signals dataset of about \~170k unique queries. More signals will generally improve coverage.
* On a raw signal dataset of about 3 million records, the job finishes execution in about 7-8 minutes on two executor pods with one CPU and about 3G of memory each. Your performance may vary depending on your configuration.

## Boosting recommendations

Generally if a query and recommendation has some token overlap, then they’re closely related and we want to highlight these. Therefore, query-recommendation pair similarity scores can be boosted based on token overlap. This overlap is calculated in terms of the number or fractions of tokens that overlap.

For example, consider the pair (“a red polo shirt”, “red polo”). If the minimum match parameter is set to 1, then there should be 1 token in common. For this example there is 1 token in common and therefore it is boosted. If it is set to 0.5, then at least half of the tokens from the shorter string (in terms of space separated tokens) should match. Here, the shorter string is “red polo” which is 2 tokens long. Therefore, to satisfy the boosting requirement, at least 1 token should match.

## Tuning tips

These tuning tips pertain to the advanced Model Tuning Parameters:

* **Special characters to be filtered out.** Special characters can cause problems with matching queries and are therefore removed in the job.

  <Note>
    Only the characters are removed, not the queries, so a query like `ps3$` becomes `ps3`.
  </Note>

* **Query similarity threshold.** This is for use by the similar queries portion of the job and is the same as that used in the Synonym and Similar Queries Detection job.

* **Boost on token overlap.** This enables or disables boosting of query recommendation pairs where all or some tokens match. How much match is required to boost can be configured using the next parameter.\
  For example, if this is enabled, then a query-recommendation pair like `(playstation 3, playstation console)` is boosted with a similarity score of 1, provided the minimum match is set to 1 token or 0.5.

* **Minimum match for token overlap.** Similar to the `mm` param in Solr, this defines the number/fraction of tokens that should overlap if boosting is enabled. Queries and recommendations are split by “ “ (space) and each part is considered a token. If using a less-than sign (\<), it must be escaped using a backslash.\
  The value can be an integer, such as 1, in which case that many tokens should match. So in the previous example, pair is boosted because the term “playstation” is common to both and the `mm` value is set to 1.\
  The value can also be a fraction, in which case that fraction of the shorter member of the query and recommendation pair should match. For example, if the value is set to 0.5 and query is 4 tokens long and recommendation is 6 tokens long then there should be at least 2 common tokens between query and recommendation.\
  Here the stopwords specified in the [list of stopwords](/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files) are ignored while calculating the overlap.

* **Query clicks threshold.** The minimum number of clicked documents needed for comparing queries.

* **Minimum query-recommendation pair occurrence count.** Minimum limit for the number of times a query-recommendation pair needs to be generated to make it to the final similar query recommendation list. Default is set to 2. Higher value will improve quality but reduce coverage.

## The similar queries collection

The following fields are stored in the `COLLECTION_NAME_queries_query_recs` collection:

* `query_t`
* `recommendation_t`
* `similarity_d`, the similarity score
* `source_s`, the approach that generated this pair, one of the following: `SessionBased` or `ClickedDocumentBased`
* `query_count_l`, the number of times the query occurred in signals
* `recommendation_count_l`, the number of times recommendations occurred in signals
* `pair_count_l`, the number of instances of the pair generated in the final recommendations using either of the approaches
* `type_s`, always set to `similar_queries`

## The query pipeline

When you enable recommendations, a default query pipeline, `COLLECTION_NAME_queries_query_recs`. is created.

## Configuration properties

<SchemaParamFields schema={schema} />