WordDelimiterFilter
- Namespace
- Rowles.LeanCorpus.Analysis.Filters
- Assembly
- Rowles.LeanCorpus.dll
Splits compound tokens on word delimiters, case transitions, and letter-digit boundaries.
public sealed class WordDelimiterFilter : ISpanTokenFilter- Inheritance
-
WordDelimiterFilter
- Implements
Remarks
For example, "WiFi4Schools_test" splits into
"Wi", "Fi", "4", "Schools", "test".
With CatenateWords the filter also emits
"WiFi", "Schools", and "WiFiSchools".
With PreserveOriginal the original token is emitted too.
Sub-tokens are emitted at the same position
(positionIncrement == 0) so downstream filters see them as
alternatives.
WordDelimiterFilter(char[]?)
Initialises a new WordDelimiterFilter.
CatenateAll
When true, all word and number parts are concatenated and emitted.
CatenateNumbers
When true, runs of number parts are concatenated and emitted.
CatenateWords
When true, runs of word parts are concatenated and emitted.
GenerateNumberParts
When true, number sub-tokens are emitted.
GenerateWordParts
When true, word sub-tokens are emitted.
PreserveOriginal
When true, emit the original token text alongside the split parts.
SplitOnCaseChange
When true, split on lowercase-to-uppercase transitions within a token.
SplitOnNumerics
When true, split between letters and digits.
StemEnglishPossessive
When true, strip trailing English possessives ('s or ')
before splitting.
Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)
Applies the filter to a token and emits the transformed token into sink.