CachingTokenFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Captures tokens as they pass through the filter chain, enabling multiple passes over the same token stream without re-running the analysis pipeline.

public sealed class CachingTokenFilter : ISpanTokenFilter

Inheritance

object

CachingTokenFilter

Implements

ISpanTokenFilter

Remarks

Place this filter in the analysis chain, run Analyser.Analyse(...), then read Tokens to inspect the captured stream. Call Reset() before the next Analyse call to clear the capture.

Each captured token's text is materialised as a string because the original span is transient. This is the only allocation this filter introduces.

Tokens are forwarded to the downstream sink unchanged — the filter is transparent to the pipeline.

Tokens

The tokens captured during the most recent Analyse call. Safe to enumerate multiple times without re-running the pipeline.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.

Reset()

Clears the captured token list so the filter is ready for the next Analyse call.