PatternReplaceFilter
- Namespace
- Rowles.LeanCorpus.Analysis.Filters
- Assembly
- Rowles.LeanCorpus.dll
Applies a regex replacement to each token's text.
public sealed class PatternReplaceFilter : ISpanTokenFilter- Inheritance
-
PatternReplaceFilter
- Implements
Remarks
This operates on individual tokens (unlike PatternReplaceCharFilter which operates on the raw character input before tokenisation).
The regex uses the interpreter (not Compiled) for
Native AOT compatibility and applies CultureInvariant by default;
callers may override via the constructor's options parameter, which is merged with the defaults
via bitwise OR. Callers requiring compilation should use the
PatternReplaceFilter(Regex, string) constructor with a pre-compiled regex.
Tokens whose text does not match the pattern are forwarded unchanged with a single zero-allocation IsMatch(ReadOnlySpan<char>) check.
PatternReplaceFilter(string, string, RegexOptions?)
Initialises a new PatternReplaceFilter with the specified regex pattern and replacement string.
PatternReplaceFilter(Regex, string)
Initialises a new PatternReplaceFilter with a pre-compiled Regex and replacement string. The caller retains full control over regex options and lifetime.
Apply(ReadOnlySpan<char>, int, int, string, int, byte[]?, ISpanTokenSink)
Applies the filter to a token and emits the transformed token into sink.