> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Tokenization by MODEL_ID > The tokenization request for the pre-trained and custom embedding use cases and specified embedding `modelId` (model name) sends text to return results in formats supported by embedding models. ## OpenAPI ````yaml /api-reference/saas/machine-learning-platform-tokenization.json post /ai/tokenization/{MODEL_ID} openapi: 3.0.1 info: title: Lucidworks AI Tokenization API version: v0 description: >- The Lucidworks AI Tokenization API returns Prediction API `embedding` use case tokens before being sent to the model. This API is used to help debug `embedding` use case tokens to ensure the input to the pre-trained or custom embedding model is valid, and within the model's processing limits. NOTE: Before the tokens are passed to the embedding model, they may be formatted, truncated, expanded, or modified in other ways to meet that model's requirements so the API call is successful. The `tokenization` endpoint requires an authentication token with scope `machinelearning.predict`. contact: name: Lucidworks url: https://lucidworks.com/ email: support@lucidworks.com termsOfService: https://lucidworks.com/legal/developer-license-agreement/ license: name: Lucidworks url: https://lucidworks.com/legal/developer-license-agreement/ servers: - url: https://APPLICATION_ID.applications.lucidworks.com description: Production security: [] tags: - name: Get tokens paths: /ai/tokenization/{MODEL_ID}: parameters: - schema: type: string example: e5-small-v2 name: MODEL_ID in: path required: true description: The name of the pre-trained or custom embedding model. post: tags: - Get tokens summary: Tokenization by MODEL_ID description: >- The tokenization request for the pre-trained and custom embedding use cases and specified embedding `modelId` (model name) sends text to return results in formats supported by embedding models. operationId: post-ai-tokenization-modelId parameters: - schema: type: string in: header name: 'Authorization: Bearer ACCESS_TOKEN' description: The authentication and authorization access token. - schema: type: string example: application/json in: header name: Content-Type description: application/json requestBody: content: application/json: schema: $ref: '#/components/schemas/TokenizationRequest' responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/TokenizationResponse' examples: {} components: schemas: TokenizationRequest: title: Tokenization request type: object x-examples: {} properties: batch: type: array description: The text used as the input for the request. items: type: object properties: text: type: string description: The content sent in the request. example: >- Mr. and Mrs. Dursley and O'\''Malley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much useCaseConfig: $ref: '#/components/schemas/UseCaseConfig' modelConfig: $ref: '#/components/schemas/ModelConfig' TokenizationResponse: title: TokenizationResponse type: object properties: generatedTokens: type: array items: type: object properties: tokens: type: array description: >- The array of tokens derived from the text submitted in the request. For example: "generatedTokens": [ { "tokens": [ "[CLS]", "query", ":", "mr", ".", "and", "mrs", ".", "du", "##rs", "##ley", "and", "o", "'", "malley", ",", "of", "number", "four", ",", "pri", "##vet", "drive", ",", "were", "proud", "to", "say", "that", "they", "were", "perfectly", "normal", ",", "thank", "you", "very", "much", ".", "[SEP]" ], items: type: string example: >- "[CLS]", "query", ":", "mr", ".", "and", "mrs", ".", "du", "##rs", "##ley", "and", "o", "'", "malley", ",", "of", "number", "four", ",", "pri", "##vet", "drive", ",", "were", "proud", "to", "say", "that", "they", "were", "perfectly", "normal", ",", "thank", "you", "very", "much", ".", "[SEP]" tokensUsed: $ref: '#/components/schemas/Token' UseCaseConfig: title: UseCaseConfig type: object properties: dataType: type: string description: >- This optional parameter enables model-specific handling in the Prediction API to help improve model accuracy. Use the most applicable fields based on available dataTypes and the dataType value that best aligns with the text sent to the Prediction API. The two string values to use for embedding models are: * `"dataType": "query"` for the query. For query-to-query pairing, best practice is to use `dataType=query` on both API calls. * `"dataType": "passage"` for fields searched at query time. For example, if questions and answers from a FAQ are indexed, the value for questions is `"dataType": "query"` and the value for the answers is `"dataType": "passage"`. default: query example: passage ModelConfig: title: ModelConfig type: object description: '' properties: vectorQuantizationMethod: type: string example: max-scale description: >- Quantization is implemented by converting float vectors into integer vectors, allowing for byte vector search using 8-bit integers. Float vectors, while very precise, are often a bit of a burden to compute and store, especially as they grow in dimensionality. One solution to this issue is to convert the vector floats into integers after inference, making byte vectors which are lower consumers of memory space and faster to compute with minimal loss in accuracy or quality. The following are the available options: * The min-max method creates tensors of embeddings and converts them to uint8 by normalizing them to the range [0, 255]. * The max-scale method finds the maximum absolute value along each embedding, normalizes the embeddings by scaling them to a range of -127 to 127, and returns the quantized embeddings as an 8-bit integer tensor. dimReductionSize: type: integer example: 256 description: >- Vector dimension reduction is the process of making the default vector size of a model smaller. The purpose of this reduction is to lessen the burden of storing large vectors while still achieving the good quality of a larger model. Token: type: object properties: inputTokens: type: integer description: The number of tokens created from the text input into the model. example: 40 promptTokens: type: integer format: int32 description: >- The number of tokens generated to prompt the model to continue generating results. example: 148 completionTokens: type: integer format: int32 description: >- The number of tokens used until the model completes. This value is always zero (0). example: 0 default: 0 totalTokens: type: integer format: int32 description: The sum of the prompt and completion tokens used in the model. example: 175 ````