Section navigation

Public namespace Rowles.LeanCorpus.Analysis.Analysers

Classes

Public class Analyser

Composable analyser that runs a tokeniser followed by a chain of span filters.

Public class AnalyserFactory

Factory for creating language-specific analysers.

Public class CountingTokenSink

An ISpanTokenSink that counts accepted tokens without allocating any strings or Token objects. Use for performance measurement of the analysis pipeline when only throughput matters, not the tokens themselves.

Public class IcuAnalyser

Default Unicode-aware analyser built on IcuTokeniser, lowercase normalisation, and stop-word removal.

Public class KeywordAnalyser

Analyser that treats the complete input as a single token.

Public class LanguageAnalyser

Configurable analyser that chains a tokeniser, lowercase normalisation, stop-word removal, and optional stemming. Used by AnalyserFactory for language-specific analysis pipelines.

Public class SimpleAnalyser

Analyser that splits text into letter-only tokens and lowercases them without stop-word removal.

Public class StandardAnalyser

Default analyser combining tokenisation, lowercase normalisation, and stop-word removal into a single pipeline. Passes span-backed tokens directly to the sink without per-token string allocations.

Thread-safety: This class maintains instance-level buffers (_lowerBuf, _offsetBuf) for performance. Each instance should be used by a single thread, or callers should create separate instances per thread (as IndexWriter does in AddDocumentsConcurrent).

Public class StemmedAnalyser

Extends the standard analysis pipeline with Porter stemming for improved recall. Pipeline: tokenise → lowercase → stop-word removal → Porter stem. Uses the composable Analyser pipeline for zero-allocation streaming.

Public class StemmerAnalyser

Generic analyser that runs tokenise → lowercase → stop-word removal → stem using the supplied ISpanStemmer.

Public class WhitespaceAnalyser

Analyser that splits text only on whitespace and applies no token filters.

Interfaces

Public interface IAnalyser

Analyses input text and emits span-backed tokens into the supplied sink.