FIX #6 by encoding UTF-8 into UTF-16LE #42
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I ran into #6 on Ubuntu 16.04 with g++ 5.4.0.
Debugging reveals that Tokenizer.Process(trainText) just treats entire text as single sentence because text gets encoded in big-endian:
First 20 characters on machine with reproduced bug:
First 20 characters on machine without bug:
I can't reproduce this on more recent Ubuntu 18.04 with g++ 7.3.0.
This is fixable by explicitly converting text to UTF-16LE.