LanguageAnalyser
- Namespace
- Rowles.LeanCorpus.Analysis.Analysers
- Assembly
- Rowles.LeanCorpus.dll
Configurable analyser that chains a tokeniser, lowercase normalisation, stop-word removal, and optional stemming. Used by AnalyserFactory for language-specific analysis pipelines.
public sealed class LanguageAnalyser : IAnalyser- Inheritance
-
LanguageAnalyser
- Implements
Remarks
Each call to Analyse(ReadOnlySpan<char>, ISpanTokenSink) clones the inner analyser so filter state is independent per call, making instances safe for concurrent use provided the tokeniser and stemmer are thread-safe.
LanguageAnalyser(ISpanTokeniser, IEnumerable<string>?, ISpanStemmer?, KeywordMarkerFilter?)
Initialises a new LanguageAnalyser with the specified tokeniser, stop words, and optional stemmer.
Analyse(ReadOnlySpan<char>, ISpanTokenSink)
Analyses the input text and emits tokens synchronously into sink.
Callers that require a materialised List<T> should use
CountingTokenSink as the sink argument.