Whitespace

whitespace tokenizer 在单词之间有空格时将文本分割为术语。

配置

whitespace tokenizer 是内置在 Milvus 中的。要使用它，只需在 analyzer_params 内的 tokenizer 部分中指定其名称即可。

PythonJavaNodeJSGocURL

analyzer_params = {
    "tokenizer": "whitespace",
}

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "whitespace");

const analyzer_params = {
    "tokenizer": "whitespace"
};

analyzerParams = map[string]any{"tokenizer": "whitespace"}

# restful
analyzerParams='{
  "tokenizer": "whitespace"
}'

The whitespace tokenizer can work in conjunction with one or more filters. For example, the following code defines an analyzer that uses the whitespace tokenizer and lowercase filter: whitespace tokenizer 可以与一个或多个 filter 结合使用。例如，以下代码定义了一个使用 whitespace tokenizer 和 lowercase filter 的 analyzer：

PythonJavaNodeJSGocURL

analyzer_params = {
    "tokenizer": "whitespace",
    "filter": ["lowercase"]
}

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "whitespace");
analyzerParams.put("filter", Collections.singletonList("lowercase"));

const analyzer_params = {
    "tokenizer": "whitespace",
    "filter": ["lowercase"]
};

analyzerParams = map[string]any{"tokenizer": "whitespace", "filter": []any{"lowercase"}}

# restful
analyzerParams='{
  "tokenizer": "whitespace",
  "filter": [
    "lowercase"
  ]
}'

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Milvus to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use. 定义 analyzer_params 后，您可以在定义 collection schema 时将其应用于 VARCHAR field。这允许 Milvus 使用指定的 analyzer 处理该 field 中的文本，以实现高效的分词和过滤。有关详细信息，请参阅 示例使用。

示例

在将 analyzer 配置应用到您的 collection schema 之前，使用 run_analyzer 方法验证其行为。

Analyzer 配置

PythonJavaNodeJSGocURL

analyzer_params = {
    "tokenizer": "whitespace",
    "filter": ["lowercase"]
}

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "whitespace");
analyzerParams.put("filter", Collections.singletonList("lowercase"));

// javascript

analyzerParams = map[string]any{"tokenizer": "whitespace", "filter": []any{"lowercase"}}

# restful

使用 run_analyzer 验证

PythonJavaNodeJSGocURL

# Sample text to analyze
sample_text = "The Milvus vector database is built for scale!"

# Run the standard analyzer with the defined configuration
result = MilvusClient.run_analyzer(sample_text, analyzer_params)
print(result)

// java

// javascript

// go

# restful

预期输出

['Milvus', 'is', 'a', 'high-performance,', 'scalable', 'vector', 'database.']