EdgeNGramTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Splits text into character substrings of length [MinGram, MaxGram] anchored at the start of each whitespace-delimited token (edge n-grams), using IsWhiteSpace(char) for Unicode-aware whitespace detection.

Thread-safety: the span path and enumerator are thread-safe for concurrent use on the same instance. No per-instance mutable state is retained across calls.

public sealed class EdgeNGramTokeniser : ISpanTokeniser

Inheritance

object

EdgeNGramTokeniser

Implements

ISpanTokeniser

EdgeNGramTokeniser(int, int)

Initialises a new EdgeNGramTokeniser with the specified gram size range.

MaxGram

Gets the maximum n-gram length (inclusive).

MinGram

Gets the minimum n-gram length (inclusive).

EnumerateTokens(ReadOnlySpan<char>)

Returns a stack-only EdgeNGramTokeniser.Enumerator that yields edge n-gram tokens one at a time without materialising a List<T> or token text strings. Use in a foreach loop when early termination or zero-list-allocation enumeration is desired.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.