Section navigation

Public structRead-only Token

Namespace
Rowles.LeanCorpus.Analysis
Assembly
Rowles.LeanCorpus.dll

Represents a single token produced by the analysis pipeline, carrying the term text, offsets, token-type metadata, and positional metadata.

public readonly struct Token

Public constructor Token(string, int, int, string, int, byte[]?, int)

Initialises a new Token with an extensible token type.

Public field DefaultType

The default token type used for ordinary term tokens.

Public property EndOffset

Gets the exclusive end character offset of the token in the original input.

Public property Payload

Gets the optional payload bytes associated with this token position.

Public property PositionIncrement

Gets the position increment for this token relative to the previous emitted token.

Public property PositionLength

Gets the number of positions spanned by this token edge. The default value of one represents an ordinary linear token.

Public property StartOffset

Gets the start character offset of the token in the original input.

Public property Text

Gets the normalised term text of the token.

Public property Type

Gets the token type.

Edit this page