Section navigation

Public classSealed PathTreeTokeniser

Namespace
Rowles.LeanCorpus.Analysis.Tokenisers
Assembly
Rowles.LeanCorpus.dll

Path hierarchy tokeniser. Emits compound tokens from root to leaf (or leaf to root in suffix mode).

public sealed class PathTreeTokeniser : ISpanTokeniser
Inheritance
PathTreeTokeniser
Implements

Public field PathType

Public property EmitDepthPayloads

Public property Lowercase

Public property SkipRoot

Public property SuffixMode

Public method Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.

Edit this page