Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect result for IS NOT NULL predicate over UNION ALL query #14352

Open
Tracked by #14375
findepi opened this issue Jan 29, 2025 · 1 comment · May be fixed by #14356
Open
Tracked by #14375

Incorrect result for IS NOT NULL predicate over UNION ALL query #14352

findepi opened this issue Jan 29, 2025 · 1 comment · May be fixed by #14356
Labels
bug Something isn't working

Comments

@findepi
Copy link
Member

findepi commented Jan 29, 2025 •
edited
Loading

Describe the bug

Incorrect result for IS NOT NULL predicate over UNION ALL query

To Reproduce

SELECT
    a,
    a IS NOT NULL
FROM (
    -- second column, even though it's not selected, is necessary to reproduce this bug
    SELECT 'foo' AS a, 3 AS b
    UNION ALL
    SELECT NULL AS a, 4 AS b
)

DF

+------+---------------+
| a    | a IS NOT NULL |
+------+---------------+
| NULL | true  😱       |
| foo  | true          |
+------+---------------+

Expected behavior

NULL valued checked with IS NOT NULL should return false

trino> SELECT
    ->     a,
    ->     a IS NOT NULL
    -> FROM (
    ->     SELECT 'foo' AS a, 3 AS b
    ->     UNION ALL
    ->     SELECT NULL AS a, 4 AS b
    -> );
  a   | _col1
------+-------
 foo  | true
 NULL | false

Additional context

No response
@findepi findepi added the bug Something isn't working label Jan 29, 2025
@findepi findepi linked a pull request Jan 29, 2025 that will close this issue
Open
@findepi
Copy link
Member Author

findepi commented Jan 29, 2025

This is actually two separate bugs:

  • initial plan for the Union already has the field marked as not-nullable, because the plan builder simply takes first input schema as the union schema (thus ignoring actual nullability). This alone maybe is corrected somewhere later, as the query without b projection does not reproduce the bug.
  • even if the plan is constructed correctly (by whatever means), the column pruning optimizer will do the same, replace union schema with the first input's schema (thus ignoring actual nullability).

Both bugs are being fixed in #14356.
The second led me to believe the idea that we can "reconstruct schema", while very convenient, is not solid, thus #14357.

@alamb alamb mentioned this issue Jan 30, 2025
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix UNION field nullability tracking findepi/datafusion
1 participant
@findepi