Skip to main content

tokenize

Function tokenize 

Source
fn tokenize(body: &str) -> impl Iterator<Item = String> + Send + '_
Expand description

Splits a string into tokens used as keys in the search inverted index

This may be used to tokenize both message bodies (for indexing) or search queries (for querying).