ShingleFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Emits contiguous token shingles for phrase-oriented analysis.

public sealed class ShingleFilter : ISpanTokenFilter

Inheritance

object

ShingleFilter

Implements

ISpanTokenFilter

Remarks

Buffers tokens during application and generates shingles covering MinShingleSize to MaxShingleSize tokens in Finish(ISpanTokenSink). When OutputUnigrams is true, original tokens are emitted alongside shingles.

ShingleFilter(int, int, bool, string)

Initialises a new ShingleFilter.

MaxShingleSize

Gets the maximum number of source tokens in a shingle.

MinShingleSize

Gets the minimum number of source tokens in a shingle.

OutputUnigrams

Gets whether original tokens are emitted alongside shingles.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Apply(ReadOnlySpan<char>, int, int, string, int, int, byte[]?, ISpanTokenSink)

Applies the filter to a token edge with an explicit position length.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.