Tokeniser
- Namespace
- Rowles.LeanCorpus.Analysis.Tokenisers
- Assembly
- Rowles.LeanCorpus.dll
Slices input text into tokens at word boundaries, splitting on whitespace and punctuation whilst tracking character offsets.
public sealed class Tokeniser : ISpanTokeniser- Inheritance
-
Tokeniser
- Implements
Tokenise(ReadOnlySpan<char>, ISpanTokenSink)
Splits the input text and emits tokens into the supplied sink.
TokeniseOffsets(ReadOnlySpan<char>, List<(int Start, int End)>)
Emits token offsets without allocating any strings. Used by StandardAnalyser to defer string materialisation until after filtering.