KeywordTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Treats the complete input as a single token.

public sealed class KeywordTokeniser : ISpanTokeniser

Inheritance

object

KeywordTokeniser

Implements

ISpanTokeniser

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.