MediaWikiTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Tokeniser for common MediaWiki markup including headings, links, categories, emphasis markup, citations, URLs, and plain body text.

public sealed class MediaWikiTokeniser : ISpanTokeniser

Inheritance

object

MediaWikiTokeniser

Implements

ISpanTokeniser

MediaWikiTokeniser(IcuTokeniser?, Uax29UrlEmailTokeniser?)

Initialises a new MediaWikiTokeniser.

BoldType

Token type emitted for MediaWiki bold markup.

CategoryType

Token type emitted for MediaWiki categories.

CitationType

Token type emitted for MediaWiki citations.

ExternalLinkType

Token type emitted for MediaWiki external links.

HeadingType

Token type emitted for MediaWiki headings.

InternalLinkType

Token type emitted for MediaWiki internal links.

ItalicType

Token type emitted for MediaWiki italic markup.

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.