LimitTokenCountFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Truncates the token stream after a fixed number of emitted tokens.

public sealed class LimitTokenCountFilter : ISpanTokenFilter

Inheritance

object

LimitTokenCountFilter

Implements

ISpanTokenFilter

LimitTokenCountFilter(int)

Initialises a new LimitTokenCountFilter.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.