ISpanTokenFilter

Namespace

Rowles.LeanCorpus.Analysis

Assembly

Rowles.LeanCorpus.dll

Transforms span-backed tokens without requiring token text strings.

public interface ISpanTokenFilter

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Apply(ReadOnlySpan<char>, int, int, string, int, int, byte[]?, ISpanTokenSink)

Applies the filter to a token edge with an explicit position length.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.