StopWordFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Removes common English stop words from a token list using a frozen set for fast, allocation-free lookups.

public sealed class StopWordFilter : ISpanTokenFilter

Inheritance

object

StopWordFilter

Implements

ISpanTokenFilter

StopWordFilter()

Initialises a new StopWordFilter using the default English stop word list.

StopWordFilter(IEnumerable<string>?)

Initialises a new StopWordFilter with a custom stop word list.

DefaultStopWords

The classic 33-word English stop word list used by the default analyser. This is the default list used by StandardAnalyser and aligns with the external benchmark baseline behaviour for maximum compatibility.

ExtendedStopWords

An extended English stop word list that removes a broader range of function words, prepositions, pronouns, modal verbs, and adverbs. This list is more aggressive than DefaultStopWords and will suppress indexing of terms such as after, before, could, how, when, etc. Use only when reduced index size matters more than full-text compatibility. Pass this to IndexWriterConfig.StopWords to opt in.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.