Uax29UrlEmailTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Unicode-aware tokeniser that preserves URLs, email addresses, hashtags, and mentions as single tokens. Thai segmentation is opt-in via the constructor.

public sealed class Uax29UrlEmailTokeniser : ISpanTokeniser

Inheritance

object

Uax29UrlEmailTokeniser

Implements

ISpanTokeniser

Uax29UrlEmailTokeniser()

Initialises a new Uax29UrlEmailTokeniser without Thai segmentation. Thai characters are treated as regular word characters.

Uax29UrlEmailTokeniser(ISpanTokeniser?)

Initialises a new Uax29UrlEmailTokeniser with an optional Thai tokeniser. When supplied, contiguous Thai runs are delegated to thaiTokeniser.

EmailType

Token type emitted for email addresses.

HashtagType

Token type emitted for hashtags.

MentionType

Token type emitted for at-mentions.

UrlType

Token type emitted for URLs.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.