WhitespaceTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Splits input text into tokens separated only by whitespace.

public sealed class WhitespaceTokeniser : ISpanTokeniser

Inheritance

object

WhitespaceTokeniser

Implements

ISpanTokeniser

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.