JapaneseTokeniser
- Namespace
- Rowles.LeanCorpus.Analysis.Tokenisers
- Assembly
- Rowles.LeanCorpus.dll
Japanese morphological tokeniser using a dictionary-backed least-cost Viterbi search.
public sealed class JapaneseTokeniser : ISpanTokeniser, IDisposable- Inheritance
-
JapaneseTokeniser
- Implements
Remarks
Dictionary data is loaded lazily from a LeanCorpus .jlc file. The
default dictionary is shared for the process lifetime. Instances created
with a custom dictionary path own that mapping and should be disposed.
JapaneseTokeniser()
Initialises a tokeniser using the shared default Japanese dictionary.
JapaneseTokeniser(string)
Initialises a tokeniser using a custom Japanese language codec.
JapaneseType
Token type emitted for Japanese dictionary tokens.
DefaultDictionaryPath
Default path for the Japanese language codec.
Dispose()
Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
Tokenise(ReadOnlySpan<char>, ISpanTokenSink)
Splits the input text and emits tokens into the supplied sink.