Section navigation

Public structRead-only SpanToken

Namespace
Rowles.LeanCorpus.Analysis
Assembly
Rowles.LeanCorpus.dll

Represents token data whose text is a span over the analysed input.

public readonly ref struct SpanToken

Remarks

The token text is only valid while the source input remains in scope. Use this type for allocation-aware tokenisation pipelines that consume tokens immediately.

Public constructor SpanToken(ReadOnlySpan<char>, int, int, string, int, byte[]?)

Initialises a new SpanToken.

Public property EndOffset

Gets the exclusive end character offset in the original input.

Public property Payload

Gets the optional payload bytes for this token.

Public property PositionIncrement

Gets the position increment relative to the previous emitted token.

Public property StartOffset

Gets the start character offset in the original input.

Public property Text

Gets the token text span.

Public property Type

Gets the token type.

Edit this page