NGramTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Splits text into all contiguous character substrings of length in [MinGram, MaxGram]. Useful for partial-word matching and CJK text.

When SplitOnWhitespace is true the tokeniser first splits on whitespace (via IsWhiteSpace(char)) and applies n-grams per word only, which avoids cross-word-boundary grams.

Thread-safety: the span path and enumerator are thread-safe for concurrent use on the same instance. No per-instance mutable state is retained across calls.

public sealed class NGramTokeniser : ISpanTokeniser

Inheritance

object

NGramTokeniser

Implements

ISpanTokeniser

NGramTokeniser(int, int, bool)

Initialises a new NGramTokeniser with the specified gram size range.

MaxGram

Gets the maximum n-gram length (inclusive).

MinGram

Gets the minimum n-gram length (inclusive).

SplitOnWhitespace

Gets whether the tokeniser splits on whitespace before applying n-grams. When true, no gram spans a word boundary.

EnumerateTokens(ReadOnlySpan<char>)

Returns a stack-only NGramTokeniser.Enumerator that yields n-gram tokens one at a time without materialising a List<T> or token text strings. When SplitOnWhitespace is true, n-grams are generated per word; otherwise they span the full input. Use in a foreach loop for zero-list-allocation enumeration.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.