tokenizer_word.hpp File Reference

Exported functions of the strus analyzer word tokenization library. More...

Go to the source code of this file.

Namespaces
	strus
	strus toplevel namespace

Functions
TokenizerFunctionInterface *	strus::createTokenizer_word (ErrorBufferInterface *errorhnd)
	Get the tokenizer type that creates the tokenization of words in the input. More...
TokenizerFunctionInterface *	strus::createTokenizer_whitespace (ErrorBufferInterface *errorhnd)
	Get the tokenizer type that creates the tokenization as splitting of the input by whitespaces. More...
TokenizerFunctionInterface *	strus::createTokenizer_langtoken (ErrorBufferInterface *errorhnd)
	Get the tokenizer type that creates the tokenization as splitting of all tokens, returning sequnces of language characters as tokens and word boundary delimiters as single character. More...

Detailed Description

Exported functions of the strus analyzer word tokenization library.