An implementation of the Porter stemming algorithm in Rust. It operates over grapheme clusters rather than characters, so your input stream can mixed content.
use unicode_segmentation::UnicodeSegmentation;
use porter_stemmer::stem;
let original = "Almost forty years later, these fair information practices have become the standard for privacy protection around the world. And yet, over that same time period, we have seen an exponential growth in the use of surveillance technologies, and our daily interactions are now routinely captured, recorded, and manipulated by small and large institutions alike.";
let tokenised_sentence = original.clone().unicode_words();
println!("Original:\n{}", original);
println!("Stemmed:\n{}", tokenised_sentence.map(stem).fold(String::new(), |last, next| { format!("{}{} ", last, next)}));
Original:
Almost forty years later, these fair information practices have become
the standard for privacy protection around the world. And yet, over
that same time period, we have seen an exponential growth in the use
of surveillance technologies, and our daily interactions are now
routinely captured, recorded, and manipulated by small and large institutions
alike.
Stemmed:
Almost forti year later these fair inform practic have becom the standard for
privaci protect around the world And yet over that same time period we have
seen an exponenti growth in the us of surveil technologi and our daili interact
ar now routin captur record and manipul by small and larg institut alik
Passage of text from Lessons from the Identity Trail used only as an example - License: https://creativecommons.org/licenses/by-nc-nd/2.5/ca/
MPL-2.0