ChineseStemmer
- Namespace
- Rowles.LeanCorpus.Analysis.Stemmers
- Assembly
- Rowles.LeanCorpus.dll
Chinese stemmer — identity implementation.
public sealed class ChineseStemmer : ISpanStemmer- Inheritance
-
ChineseStemmer
- Implements
Remarks
Mandarin Chinese is an isolating language: words do not inflect via suffixes, so suffix-stripping stemming is linguistically inappropriate. The morphological unit in Chinese is the character (字) or multi-character word (词), not a stem produced by affix removal.
Meaningful normalisation for Chinese search involves:
- Word segmentation with jieba, a CJK analyser, or a dictionary-based tokeniser
- Simplified ↔ Traditional character conversion
- Full-width → half-width normalisation
This class is provided so the ISpanStemmer pipeline compiles uniformly
across all supported languages. Wire up proper segmentation as a pre-tokenisation
step before passing tokens here.
Stem(ReadOnlySpan<char>, Span<char>)
Stems word into output.
Returns the length of the stemmed result, or -1 if output is too small.
The caller must ensure output.Length >= word.Length for guaranteed success.
Stem(string)
Convenience overload returning the stemmed string.