Section navigation

Public namespace Rowles.LeanCorpus.Analysis.Filters

Classes

Public class AccentFoldingFilter

Normalises accented/diacritic characters to their ASCII base form (e.g., é→e, ñ→n, ü→u) for language-neutral matching. Uses Unicode canonical decomposition followed by stripping combining marks.

Public class CachingTokenFilter

Captures tokens as they pass through the filter chain, enabling multiple passes over the same token stream without re-running the analysis pipeline.

Public class ClassicFilter

Removes possessive endings and periods from acronyms, replicating Lucene's ClassicFilter.

Public class CommonGramsFilter

Produces bigrams of consecutive common words to improve phrase query recall.

Public class DecimalDigitFilter

Normalises Unicode decimal digits to ASCII digits.

Public class ElisionFilter

Removes configured elided articles before straight or curly apostrophes.

Public class FlattenGraphFilter

Normalises token position increments so same-position alternates remain explicit and the stream stays consumable by LeanCorpus's linear postings model.

Public class HtmlStripCharFilter

Strips HTML/XML tags and HTML entities from input text, leaving only text content. Uses a span-based scanner to avoid regex allocations.

Public class HunspellDictionary

Lightweight Hunspell dictionary that supports short-form flags and simple prefix and suffix rules.

Public class HunspellStemFilter

Stems tokens using a pre-parsed HunspellDictionary.

Public class HyphenatedWordsFilter

Joins consecutive tokens that share the same position into a single hyphenated token.

Public class KeepWordFilter

Removes any token whose text is not present in the configured keep-word set.

Public class KeywordMarkerFilter

Identifies tokens that should be treated as keywords by compatible analysers.

Public class LengthFilter

Removes tokens whose text length falls outside an inclusive range.

Public class LimitTokenCountFilter

Truncates the token stream after a fixed number of emitted tokens.

Public class LowercaseFilter

Performs an in-place lowercase transformation on tokens or a character buffer.

Public class MappingCharFilter

Maps specific characters or strings to replacements using a lookup table. Useful for normalising special characters (e.g., smart quotes → straight quotes).

Public class MetaphoneFilter

Emits Metaphone encodings for tokens.

Public class PatternReplaceCharFilter

Replaces text matching a regex pattern with a replacement string.

Public class PatternReplaceFilter

Applies a regex replacement to each token's text.

Public class PhoneticAlternatesFilter

Emits bounded Latin-name phonetic alternates at the same token position.

Public class PorterStemmerFilter

Porter Stemming Algorithm implementation as an ISpanTokenFilter. Based on the Porter 1980 specification for English stemming. Operates on tokens in-place, replacing text with stemmed form.

Public class ReverseStringFilter

Reverses the characters in each token.

Public class ShingleFilter

Emits contiguous token shingles for phrase-oriented analysis.

Public class StemTokenFilter

Applies an ISpanStemmer to each token in the list. Useful as a drop-in filter in the composable Analyser pipeline.

Public class StopWordFilter

Removes common English stop words from a token list using a frozen set for fast, allocation-free lookups.

Public class SynonymGraphFilter

Token filter that supports multi-token synonym expansion using a trie-based SynonymMap. Uses longest-match lookahead for multi-word synonyms and inserts replacement tokens at the same position offsets.

Public class SynonymMap

Trie-based synonym map supporting multi-token source phrases. Used by SynonymGraphFilter for longest-match multi-token synonym expansion.

Public class TruncateTokenFilter

Truncates token text to a maximum character length.

Public class TypeTokenFilter

Filters tokens by extensible token type.

Public class UniqueTokenFilter

Removes duplicate tokens at the same position, keeping the first occurrence.

Public class WordDelimiterFilter

Splits compound tokens on word delimiters, case transitions, and letter-digit boundaries.

Interfaces

Public interface ICharFilter

Interface for character-level filters that transform raw text before tokenisation. Char filters run before the tokeniser, operating on the entire input string.