LetterTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Splits input text into letter-only tokens, discarding digits and punctuation.

public sealed class LetterTokeniser : ISpanTokeniser

Inheritance

object

LetterTokeniser

Implements

ISpanTokeniser

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.