HyphenatedWordsFilter

Namespace

Rowles.LeanCorpus.Analysis.Filters

Assembly

Rowles.LeanCorpus.dll

Joins consecutive tokens that share the same position into a single hyphenated token.

public sealed class HyphenatedWordsFilter : ISpanTokenFilter

Inheritance

object

HyphenatedWordsFilter

Implements

ISpanTokenFilter

Remarks

When a tokeniser splits hyphenated compounds into separate tokens at the same position (signalled by positionIncrement == 0), this filter reverses the split by joining those tokens back together with a separator character between them.

For example, if "state-of-the-art" is tokenised as four tokens at the same position — "state", "of", "the", "art" — this filter emits a single token "state-of-the-art".

Uses ArrayPool<T> for the accumulation buffer and calls Finish(ISpanTokenSink) to flush the final buffered token at end-of-stream.

HyphenatedWordsFilter(char, int)

Initialises a new HyphenatedWordsFilter.

Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)

Applies the filter to a token and emits the transformed token into sink.

Clone()

Creates an independent copy of this filter with the same configuration but fresh state. The default returns this, which is safe for stateless filters. Stateful filters must override to return a new instance.

Finish(ISpanTokenSink)

Called after all tokens have been processed, allowing stateful filters to flush buffered tokens into the pipeline. The default implementation is a no-op.