StandardAnalyser

Namespace

Rowles.LeanCorpus.Analysis.Analysers

Assembly

Rowles.LeanCorpus.dll

Default analyser combining tokenisation, lowercase normalisation, and stop-word removal into a single pipeline. Passes span-backed tokens directly to the sink without per-token string allocations.

Thread-safety: This class maintains instance-level buffers (_lowerBuf, _offsetBuf) for performance. Each instance should be used by a single thread, or callers should create separate instances per thread (as IndexWriter does in AddDocumentsConcurrent).

public sealed class StandardAnalyser : IAnalyser

Inheritance

object

StandardAnalyser

Implements

IAnalyser

StandardAnalyser(int, IEnumerable<string>?)

Initialises a new StandardAnalyser with the specified stop words.

Analyse(ReadOnlySpan<char>, ISpanTokenSink)

Analyses the input text and emits tokens synchronously into sink. Callers that require a materialised List<T> should use CountingTokenSink as the sink argument.