PatternTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Tokenises input text by splitting on a regex pattern, emitting each match as a token.

public sealed class PatternTokeniser : ISpanTokeniser

Inheritance

object

PatternTokeniser

Implements

ISpanTokeniser

Remarks

Uses Regex.EnumerateMatches on the input span, which is allocation-free and returns a ref struct enumerator. Only the matched spans are forwarded to the sink; the sink may choose to copy them.

Matches of zero length are skipped.

PatternTokeniser(string, RegexOptions?)

Initialises a new PatternTokeniser from a regex pattern string. The regex uses the interpreter (not Compiled) for Native AOT compatibility and applies CultureInvariant by default.

PatternTokeniser(Regex)

Initialises a new PatternTokeniser with a pre-compiled Regex.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.