CommonGramsFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Produces bigrams of consecutive common words to improve phrase query recall.

public sealed class CommonGramsFilter : ISpanTokenFilter

Inheritance

object

CommonGramsFilter

Implements

ISpanTokenFilter

Remarks

When two consecutive tokens are both in the common words set, this filter emits a bigram token (e.g. "the_quick") in addition to the individual common words. The bigram appears at the same position as the first common word (positionIncrement = 0), enabling phrase queries to match across common-word boundaries.

This filter is stateful — it buffers the previous token across calls — and relies on Finish(ISpanTokenSink) to flush the final buffered token at end-of-stream.

Common-word lookup uses FrozenSet<T> with AlternateLookup<ReadOnlySpan<char>> for zero-allocation membership tests, matching the pattern used by StopWordFilter.

CommonGramsFilter(IEnumerable<string>, string)

Initialises a new CommonGramsFilter.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.