Section navigation

Public namespace Rowles.LeanCorpus.Analysis.Tokenisers

Classes

Public class CJKBigramTokeniser

Tokeniser for CJK (Chinese, Japanese, Korean) text using overlapping bigrams on CJK unified ideographs. Non-CJK text is tokenised by standard word boundaries. CJK ideograph runs produce overlapping 2-character tokens; single isolated ideographs are emitted as unigrams.

Public class ChineseLexicon

Provides the default Chinese lexicon for the ChineseLexiconTokeniser. Searches parent directories for lexicons/chinese-dict.txt. If the file is not present, the factory falls back to a minimal built-in list.

Public class ChineseLexiconTokeniser

Chinese word segmentation tokeniser using greedy longest-match against a user-supplied lexicon. Unknown CJK characters fall back to unigrams. Non-CJK text is tokenised by standard word boundaries.

Public class EdgeNGramTokeniser

Splits text into character substrings of length [MinGram, MaxGram] anchored at the start of each whitespace-delimited token (edge n-grams), using IsWhiteSpace(char) for Unicode-aware whitespace detection.

Thread-safety: the span path and enumerator are thread-safe for concurrent use on the same instance. No per-instance mutable state is retained across calls.

Public class IcuTokeniser

Lightweight Unicode-aware tokeniser that segments text using Unicode character classes. Thai segmentation is opt-in: pass a ThaiTokeniser to the constructor to enable dictionary-based Thai word splitting.

Public class JapaneseTokeniser

Japanese morphological tokeniser using a dictionary-backed least-cost Viterbi search.

Public class KeywordTokeniser

Treats the complete input as a single token.

Public class LetterTokeniser

Splits input text into letter-only tokens, discarding digits and punctuation.

Public class MediaWikiTokeniser

Tokeniser for common MediaWiki markup including headings, links, categories, emphasis markup, citations, URLs, and plain body text.

Public class NGramTokeniser

Splits text into all contiguous character substrings of length in [MinGram, MaxGram]. Useful for partial-word matching and CJK text.

When SplitOnWhitespace is true the tokeniser first splits on whitespace (via IsWhiteSpace(char)) and applies n-grams per word only, which avoids cross-word-boundary grams.

Thread-safety: the span path and enumerator are thread-safe for concurrent use on the same instance. No per-instance mutable state is retained across calls.

Public class PathTreeTokeniser

Path hierarchy tokeniser. Emits compound tokens from root to leaf (or leaf to root in suffix mode).

Public class PatternTokeniser

Tokenises input text by splitting on a regex pattern, emitting each match as a token.

Public class ThaiTokeniser

Lightweight Thai tokeniser using greedy longest-match segmentation with a user-supplied lexicon and grapheme-cluster fallback for unknown runs.

Public class Tokeniser

Slices input text into tokens at word boundaries, splitting on whitespace and punctuation whilst tracking character offsets.

Public class Uax29UrlEmailTokeniser

Unicode-aware tokeniser that preserves URLs, email addresses, hashtags, and mentions as single tokens. Thai segmentation is opt-in via the constructor.

Public class WhitespaceTokeniser

Splits input text into tokens separated only by whitespace.

Structs

Public struct EdgeNGramTokeniser.Enumerator

Stack-only edge n-gram enumerator. Each call to MoveNext() advances to the next edge n-gram. Current exposes the yielded token.

Public struct NGramTokeniser.Enumerator

Stack-only n-gram enumerator. Each call to MoveNext() yields the next n-gram in increasing start-offset order.

Interfaces

Public interface ISpanTokeniser

Splits input text into span-backed raw tokens without materialising token text strings.