KeywordTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Treats the complete input as a single token.

public sealed class KeywordTokeniser : ITokeniser

KeywordTokeniser

Implements

ITokeniser

Methods

Tokenise(ReadOnlySpan<char>)

Splits the input text into a list of tokens at word boundaries.