TokenBudgetPolicy

Namespace

Rowles.LeanCorpus.Analysis

Assembly

Rowles.LeanCorpus.dll

Determines what happens when a document exceeds MaxTokensPerDocument.

public enum TokenBudgetPolicy

Fields

Reject = 2

Throw an TokenBudgetExceededException to reject the document.

Truncate = 0

Silently discard tokens beyond the limit.

Warn = 1

Log a warning and continue indexing with all tokens.