ChineseLexiconTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Chinese word segmentation tokeniser using greedy longest-match against a user-supplied lexicon. Unknown CJK characters fall back to unigrams. Non-CJK text is tokenised by standard word boundaries.

public sealed class ChineseLexiconTokeniser : ISpanTokeniser

Inheritance

object

ChineseLexiconTokeniser

Implements

ISpanTokeniser

Remarks

The lexicon must be provided via the constructor, FromFile(string), or FromStream(Stream). A lexicon file is available as an optional download. The format is one word per line, UTF-8, with # comments.

ChineseLexiconTokeniser(IEnumerable<string>)

Initialises a new ChineseLexiconTokeniser with the supplied lexicon.

CjkType

Token type emitted for CJK ideograph tokens.

FromFile(string)

Loads a UTF-8 text lexicon from disk, using one word per line. Lines starting with # are ignored.

FromStream(Stream)

Loads a UTF-8 text lexicon from a stream, using one word per line. Lines starting with # are ignored. The stream is not disposed.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.