Core: Fix UnicodeUtil#truncateStringMax returns malformed string. #11161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@amogh-jahagirdar @nastra can you please help review this? thanks. |
| // surrogate code points are not Unicode scalar values, | ||
| // any UTF-8 byte sequence that would otherwise map to code points U+D800..U+DFFF is ill-formed. | ||
| // see https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G27288 | ||
| Preconditions.checkArgument( |
There was a problem hiding this comment.
Just a minor point here, but shouldn't this only be relevant if we somehow get non-unicode binary in a unicode string? Shouldn't be possible in a Java string right?
| @@ -274,4 +274,17 @@ public void testTruncateStringMax() { | |||
| "Test input with multiple 4 byte UTF-8 character where the first unicode character should be incremented") | |||
| .isEqualTo(0); | |||
| } | |||
|
|
|||
| @Test | |||
| public void testTruncateStringMaxUpperBound() { | |||
There was a problem hiding this comment.
Could we add these to the test above? I'm also fine if there is a specific reason to have them somewhere else but it seems like these would fit into the test above as just other examples. The test cases for +1 and MAX_CODE_POINT are there already right?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We encountered an exception while writing data, and the stack trace is as follows. It occurred during the collection of Parquet column metrics:
Exception stack:
Investigation
After some investigation, I found that when collecting Parquet column metrics, string metrics are truncated by default to a length of 16 characters. When truncating the max metric, if the truncated length is less than the length of the original max value, the last character will be incremented by 1 to ensure that the truncated value is greater than the max value. However, this increment operation does not consider skipping illegal UTF-8 Unicode code points, which may lead to the following exception.
In the scenario where we encountered this issue, there is a Parquet file with a column's max metric length exceeding 16, and the code point of its 16th character is '\uD7FF', which is
Character.MIN_SURROGATE - 1. Adding 1 to this resulted inCharacter.MIN_SURROGATE, which is not a valid Unicode scalar value. Therefore, whenConversions.toByteBufferattempted to encode it in UTF-8 format, aMalformedInputExceptionwas thrown.This fix specifically skips illegal code points when incrementing the last character to avoid this issue.
To reproduce