-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why deacc? #19
Comments
Hey @emillundhcodemill, thanks for your contributions! I actually don't recall why the deaccent is always set to true. In our mother tongue (Spanish) few words can be written with or without the accent (solo, for instance). In most cases, though, the accent mark is used consistently to show intonation or to differentiate meaning between one syllable words that are spelled the same. I would suggest to default |
I totally agree with @fbarrios. Honestly we don't remember why we left that with True. @emillundhcodemill if you could just set it to False by default, and add it as an optional parameter in the keywords method, we would be more than happy to merge your contribution. Thanks in advance! |
OK, done. #22 |
I see that the parameter
deacc
is set to True for all languages whentokenize
is called fromclean_text_by_word
andtokenize_by_word
. This means that in Nordic languages, characters with umlaut like å, ä, ö are rendered without umlaut as a, a, o; this drastically reduces the quality of the keyword extraction for Swedish which makes heavy use of umlauts and where the umlauted characters count as characters in their own right (For example snö=snow, sno=twist; äga=own, aga=spank).AFAICS, it's easy to set deacc=False and obtain much nicer results for Swedish; it should probably be done on a per-language basis (I suggest False for danish, finnish, german, norwegian, and swedish).
I'll be happy to file a PR but I'd like to hear if you have any comments?
The text was updated successfully, but these errors were encountered: