feat(spark)!: Transpile ANY to EXISTS #4305
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #4298
In Hive hierarchy
ANY
is an aggregate function forBOOLEAN
expressions and not an array/subquery operator as is in other dialects, meaning that the following transpiled queries do not work:However, Spark2+ supports the
EXISTS()
function which can process both ARRAY expressions and subqueries in a similar fashion. This PR:EXISTS
class to also inherit fromFunc
, making it possible to parse & construct it as a proper function.As step (2) is not required per se (could also build an anonymous function, check first commit), it's added as a standalone commit on top of (1); If we don't want to keep it it's trivial to drop it at once.
Docs
Postgres ANY for ARRAY | Postgres ANY for subqueries | Spark/Databricks ANY | Spark/Databricks EXISTS