IcuTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Lightweight Unicode-aware tokeniser that segments text using Unicode character classes. Thai segmentation is opt-in: pass a ThaiTokeniser to the constructor to enable dictionary-based Thai word splitting.

public sealed class IcuTokeniser : ISpanTokeniser

Inheritance

object

IcuTokeniser

Implements

ISpanTokeniser

IcuTokeniser()

Initialises a new IcuTokeniser without Thai segmentation support. Thai characters are treated as regular word characters.

IcuTokeniser(ISpanTokeniser?)

Initialises a new IcuTokeniser with an optional Thai tokeniser. When supplied, contiguous Thai runs are delegated to thaiTokeniser.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.