IcuAnalyser

Namespace

Rowles.LeanCorpus.Analysis.Analysers

Assembly

Rowles.LeanCorpus.dll

Default Unicode-aware analyser built on IcuTokeniser, lowercase normalisation, and stop-word removal.

public sealed class IcuAnalyser : IAnalyser

Inheritance

object

IcuAnalyser

Implements

IAnalyser

IcuAnalyser(IEnumerable<string>?, ISpanTokeniser?, params ISpanTokenFilter[])

Initialises a new IcuAnalyser.

Analyse(ReadOnlySpan<char>, ISpanTokenSink)

Analyses the input text and emits tokens synchronously into sink. Callers that require a materialised List<T> should use CountingTokenSink as the sink argument.