Token filters

Token filters run after tokenisation and before stemming. They rewrite, drop, or add tokens.

Build a pipeline

using Rowles.LeanCorpus.Analysis;
using Rowles.LeanCorpus.Analysis.Tokenisers;
using Rowles.LeanCorpus.Analysis.Filters;

var analyser = new Analyser(
    tokeniser: new Tokeniser(),
    new LowercaseFilter(),
    new StopWordFilter(StopWords.English),
    new PorterStemmerFilter());

Filters by role

| Role | Filters | | Normalisation | LowercaseFilter, AccentFoldingFilter, ClassicFilter, DecimalDigitFilter, HyphenatedWordsFilter, PatternReplaceFilter, ReverseStringFilter, WordDelimiterFilter, TruncateTokenFilter | | Selection | StopWordFilter, LengthFilter, UniqueTokenFilter, KeepWordFilter, TypeTokenFilter, LimitTokenCountFilter | | Stemming | PorterStemmerFilter, HunspellStemFilter, KeywordMarkerFilter | | Synonyms | SynonymGraphFilter, FlattenGraphFilter, CommonGramsFilter, ShingleFilter | | Language | ElisionFilter | | Phonetics | MetaphoneFilter, PhoneticAlternatesFilter | | Caching | CachingTokenFilter |

KeywordMarkerFilter marks tokens that compatible stemmers should skip.

Char filters

Char filters mutate the input before tokenisation:

HtmlStripCharFilter
MappingCharFilter
PatternReplaceCharFilter

Attach them through IndexWriterConfig.CharFilters.

Order matters

Lowercase before stop words. Stemming after both. Synonym expansion after normalisation.

See also

Edit this page