UniqueTokenFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Removes duplicate tokens at the same position, keeping the first occurrence.

public sealed class UniqueTokenFilter : ISpanTokenFilter

Inheritance

object

UniqueTokenFilter

Implements

ISpanTokenFilter

Remarks

When multiple tokens land at the same position (signalled by positionIncrement == 0), only the first instance of each distinct token text is emitted. Tokens at different positions are never deduplicated against each other.

Uses a per-position HashSet<T> of token text. The set is cleared on Finish(ISpanTokenSink) for reuse across documents.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.