CJKBigramTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Tokeniser for CJK (Chinese, Japanese, Korean) text using overlapping bigrams on CJK unified ideographs. Non-CJK text is tokenised by standard word boundaries. CJK ideograph runs produce overlapping 2-character tokens; single isolated ideographs are emitted as unigrams.

public sealed class CJKBigramTokeniser : ISpanTokeniser

Inheritance

object

CJKBigramTokeniser

Implements

ISpanTokeniser

Remarks

Hiragana, Katakana, and Hangul are not bigrammed. They are syllabaries or composed syllables and are word-tokenised instead. Supplementary-plane CJK ideographs (Extension B+) are supported via surrogate pair decoding.

CjkType

Token type emitted for CJK ideograph tokens.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.