Section navigation

Public namespace Rowles.LeanCorpus.Analysis

Namespaces

Public namespace Rowles.LeanCorpus.Analysis.Analysers
Public namespace Rowles.LeanCorpus.Analysis.Filters
Public namespace Rowles.LeanCorpus.Analysis.Stemmers
Public namespace Rowles.LeanCorpus.Analysis.Tokenisers

Classes

Public class StopWords

Stop word lists for common languages.

Public class TokenBudgetExceededException

Thrown when Reject is active and a document exceeds the token budget.

Structs

Public struct SpanToken

Represents token data whose text is a span over the analysed input.

Public struct Token

Represents a single token produced by the analysis pipeline, carrying the term text, offsets, token-type metadata, and positional metadata.

Interfaces

Public interface ISpanTokenFilter

Transforms span-backed tokens without requiring token text strings.

Public interface ISpanTokenSink

Consumes span-backed tokens synchronously without requiring token text strings.

Enums

Public enum TokenBudgetPolicy

Determines what happens when a document exceeds its configured token limit.