Skip to content

Apache Airflow 2.3.0
Compare
Choose a tag to compare
@ephraimbuddy ephraimbuddy released this 01 May 00:37
· 10750 commits to main since this release
2.3.0
6b772f6

New Features

  • Add dynamic task mapping (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22)
  • New Grid View replaces Tree View (#18675)
  • Templated requirements.txt in Python Operators (#17349)
  • Allow reuse of decorated tasks (#22941)
  • Move the database configuration to a new section (#22284)
  • Add SmoothOperator (#22813)
  • Make operator's execution_timeout configurable (#22389)
  • Events Timetable (#22332)
  • Support dag serialization with custom ti_deps rules (#22698)
  • Support log download in task log view (#22804)
  • support for continue backfill on failures (#22697)
  • Add dag-processor cli command (#22305)
  • Add possibility to create users in LDAP mode (#22619)
  • Add ignore_first_depends_on_past for scheduled jobs (#22491)
  • Update base sensor operator to support XCOM return value (#20656)
  • Add an option for run id in the ui trigger screen (#21851)
  • Enable JSON serialization for connections (#19857)
  • Add REST API endpoint for bulk update of DAGs (#19758)
  • Add queue button to click-on-DagRun interface. (#21555)
  • Add list-import-errors to airflow dags command (#22084)
  • Store callbacks in database if standalone_dag_processor config is True. (#21731)
  • Add LocalKubernetesExecutor (#19729)
  • Add celery.task_timeout_error metric (#21602)
  • Airflow db downgrade cli command (#21596)
  • Add ALL_SKIPPED trigger rule (#21662)
  • Add db clean CLI command for purging old data (#20838)
  • Add celery_logging_level (#21506)
  • Support different timeout value for dag file parsing (#21501)
  • Support generating SQL script for upgrades (#20962)
  • Add option to compress Serialized dag data (#21332)
  • Branch python operator decorator (#20860)
  • Add Audit Log View to Dag View (#20733)
  • Add missing StatsD metric for failing SLA Callback notification (#20924)
  • Add ShortCircuitOperator configurability for respecting downstream trigger rules (#20044)
  • Allow using Markup in page title in Webserver (#20888)
  • Add Listener Plugin API that tracks TaskInstance state changes (#20443)
  • Add context var hook to inject more env vars (#20361)
  • Add a button to set all tasks to skipped (#20455)
  • Cleanup pending pods (#20438)
  • Add config to warn public deployment exposure in UI (#18557)
  • Log filename template records (#20165)
  • Added windows extensions (#16110)
  • Showing approximate time until next dag_run in Airflow (#20273)
  • Extend config window on UI (#20052)
  • Add show dag dependencies feature to CLI (#19985)
  • Add cli command for 'airflow dags reserialize` (#19471)
  • Add missing description field to Pool schema(REST API) (#19841)
  • Introduce DagRun action to change state to queued. (#19353)
  • Add DAG run details page (#19705)
  • Add role export/import to cli tools (#18916)
  • Adding dag_id_pattern parameter to the /dags endpoint (#18924)

Improvements

  • Show schedule_interval/timetable description in UI (#16931)
  • Added column duration to DAG runs view (#19482)
  • Enable use of custom conn extra fields without prefix (#22607)
  • Initialize finished counter at zero (#23080)
  • Improve logging of optional provider features messages (#23037)
  • Meaningful error message in resolve_template_files (#23027)
  • Update ImportError items instead of deleting and recreating them (#22928)
  • Add option --skip-init to db reset command (#22989)
  • Support importing connections from files with ".yml" extension (#22872)
  • Support glob syntax in .airflowignore files (#21392) (#22051)
  • Hide pagination when data is a single page (#22963)
  • Support for sorting DAGs in the web UI (#22671)
  • Speed up has_access decorator by ~200ms (#22858)
  • Add XComArg to lazy-imported list of Airflow module (#22862)
  • Add more fields to REST API dags/dag_id/details endpoint (#22756)
  • Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812)
  • No need to load whole ti in current_state (#22764)
  • Pickle dag exception string fix (#22760)
  • Better verification of Localexecutor's parallelism option (#22711)
  • log backfill exceptions to sentry (#22704)
  • retry commit on MySQL deadlocks during backfill (#22696)
  • Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637)
  • Use timetable to generate planned days for current year (#22055)
  • Disable connection pool for celery worker (#22493)
  • Make date picker label visible in trigger dag view (#22379)
  • Expose try_number in airflow vars (#22297)
  • Add generic connection type (#22310)
  • Add a few more fields to the taskinstance finished log message (#22262)
  • Pause auto-refresh if scheduler isn't running (#22151)
  • Show DagModel details. (#21868)
  • Add pip_install_options to PythonVirtualenvOperator (#22158)
  • Show import error for airflow dags list CLI command (#21991)
  • Pause auto-refresh when page is hidden (#21904)
  • Default args type check (#21809)
  • Enhance magic methods on XComArg for UX (#21882)
  • py files don't have to be checked is_zipfiles in refresh_dag (#21926)
  • Fix TaskDecorator type hints (#21881)
  • Add 'Show record' option for variables (#21342)
  • Use DB where possible for quicker airflow dag subcommands (#21793)
  • REST API: add rendered fields in task instance. (#21741)
  • Change the default auth backend to session (#21640)
  • Don't check if py DAG files are zipped during parsing (#21538)
  • Switch XCom implementation to use run_id (#20975)
  • Action log on Browse Views (#21569)
  • Implement multiple API auth backends (#21472)
  • Change logging level details of connection info in get_connection() (#21162)
  • Support mssql in airflow db shell (#21511)
  • Support config worker_enable_remote_control for celery (#21507)
  • Log memory usage in CgroupTaskRunner (#21481)
  • Modernize DAG-related URL routes and rename "tree" to "grid" (#20730)
  • Move Zombie detection to SchedulerJob (#21181)
  • Improve speed to run airflow by 6x (#21438)
  • Add more SQL template fields renderers (#21237)
  • Simplify fab has access lookup (#19294)
  • Log context only for default method (#21244)
  • Log trigger status only if at least one is running (#21191)
  • Add optional features in providers. (#21074)
  • Better multiple_outputs inferral for @task.python (#20800)
  • Improve handling of string type and non-attribute template_fields (#21054)
  • Remove un-needed deps/version requirements (#20979)
  • Correctly specify overloads for TaskFlow API for type-hinting (#20933)
  • Introduce notification_sent to SlaMiss view (#20923)
  • Rewrite the task decorator as a composition (#20868)
  • Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798)
  • Rewrite DAG run retrieval in task command (#20737)
  • Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722)
  • Make native environment Airflow-flavored like sandbox (#20704)
  • Better error when param value has unexpected type (#20648)
  • Add filter by state in DagRun REST API (List Dag Runs) (#20485)
  • Prevent exponential memory growth in Tasks with custom logging handler (#20541)
  • Set default logger in logging Mixin (#20355)
  • Reduce deprecation warnings from www (#20378)
  • Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002)
  • Add specific warning when Task asks for more slots than pool defined with (#20178)
  • UI: Update duration column for better human readability (#20112)
  • Use Viewer role as example public role (#19215)
  • Properly implement DAG param dict copying (#20216)
  • ShortCircuitOperator push XCom by returning python_callable result (#20071)
  • Add clear logging to tasks killed due to a Dagrun timeout (#19950)
  • Change log level for Zombie detection messages (#20204)
  • Better confirmation prompts (#20183)
  • Only execute TIs of running DagRuns (#20182)
  • Check and run migration in commands if necessary (#18439)
  • Log only when Zombies exists (#20118)
  • Increase length of the email and username (#19932)
  • Add more filtering options for TI's in the UI (#19910)
  • Dynamically enable "Test Connection" button by connection type (#19792)
  • Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842)
  • Renamed Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849)
  • Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431)
  • Speed up webserver boot time by delaying provider initialization (#19709)
  • Configurable logging of XCOM value in PythonOperator (#19378)
  • Minimize production js files (#19658)
  • Add hook_params in BaseSqlOperator (#18718)
  • Add missing "end_date" to hash components (#19281)
  • More friendly output of the airflow plugins command + add timetables (#19298)
  • Add sensor default timeout config (#19119)
  • Update taskinstance REST API schema to include dag_run_id field (#19105)
  • Adding feature in bash operator to append the user defined env variable to system env variable (#18944)
  • Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161)

Bug Fixes

  • Use inherited 'trigger_tasks' method (#23016)
  • In DAG dependency detector, use class type instead of class name (#21706)
  • Fix tasks being wrongly skipped by schedule_after_task_execution (#23181)
  • Fix X-Frame enabled behaviour (#23222)
  • Allow extra to be nullable in connection payload as per schema(REST API). (#23183)
  • Fix dag_id extraction for dag level access checks in web ui (#23015)
  • Fix timezone display for logs on UI (#23075)
  • Include message in graph errors (#23021)
  • Change trigger dropdown left position (#23013)
  • Don't add planned tasks for legacy DAG runs (#23007)
  • Add dangling rows check for TaskInstance references (#22924)
  • Validate the input params in connection CLI command (#22688)
  • Fix trigger event payload is not persisted in db (#22944)
  • Drop "airflow moved" tables in command db reset (#22990)
  • Add max width to task group tooltips (#22978)
  • Add template support for external_task_ids. (#22809)
  • Allow DagParam to hold falsy values (#22964)
  • Fix regression in pool metrics (#22939)
  • Priority order tasks even when using pools (#22483)
  • Do not clear XCom when resuming from deferral (#22932)
  • Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898)
  • Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850)
  • Fixed backfill interference with scheduler (#22701)
  • Support conf param override for backfill runs (#22837)
  • Correctly interpolate pool name in PoolSlotsAvailableDep statues (#22807)
  • Fix email_on_failure with render_template_as_native_obj (#22770)
  • Fix processor cleanup on DagFileProcessorManager (#22685)
  • Prevent meta name clash for task instances (#22783)
  • remove json parse for gantt chart (#22780)
  • Check for missing dagrun should know version (#22752)
  • Fixes ScheduleInterval spec (#22635)
  • Fixing task status for non-running and non-committed tasks (#22410)
  • Do not log the hook connection details even at DEBUG level (#22627)
  • Stop crashing when empty logs are received from kubernetes client (#22566)
  • Fix bugs about timezone change (#22525)
  • Fix entire DAG stops when one task has end_date (#20920)
  • Use logger to print message during task execution. (#22488)
  • Make sure finalizers are not skipped during exception handling (#22475)
  • update smart sensor docs and minor fix on is_smart_sensor_compatible() (#22386)
  • Fix run_id k8s and elasticsearch compatibility with Airflow 2.1 (#22385)
  • Allow to except_skip None on BranchPythonOperator (#20411)
  • Fix incorrect datetime details (DagRun views) (#21357)
  • Remove incorrect deprecation warning in secrets backend (#22326)
  • Remove RefreshConfiguration workaround for K8s token refreshing (#20759)
  • Masking extras in GET /connections/<connection> endpoint (#22227)
  • Set queued_dttm when submitting task to directly to executor (#22259)
  • Addressed some issues in the tutorial mentioned in discussion #22233 (#22236)
  • Change default python executable to python3 for docker decorator (#21973)
  • Don't validate that Params are JSON when NOTSET (#22000)
  • Add per-DAG delete permissions (#21938)
  • Fix handling some None parameters in kubernetes 23 libs. (#21905)
  • Fix handling of empty (None) tags in bulk_write_to_db (#21757)
  • Fix DAG date range bug (#20507)
  • Removed request.referrer from views.py (#21751)
  • Make DbApiHook use get_uri from Connection (#21764)
  • Fix some migrations (#21670)
  • [de]serialize resources on task correctly (#21445)
  • Add params dag_id, task_id etc to XCom.serialize_value (#19505)
  • Update test connection functionality to use custom form fields (#21330)
  • fix all "high" npm vulnerabilities (#21526)
  • Fix bug incorrectly removing action from role, rather than permission. (#21483)
  • Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296)
  • Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238)
  • Ensure on_task_instance_running listener can get at task (#21157)
  • Return to the same place when triggering a DAG (#20955)
  • Fix task ID deduplication in @task_group (#20870)
  • Add downgrade to some FAB migrations (#20874)
  • Only validate Params when DAG is triggered (#20802)
  • Fix airflow trigger cli (#20781)
  • Fix task instances iteration in a pool to prevent blocking (#20816)
  • Allow depending to a @task_group as a whole (#20671)
  • Use original task's start_date if a task continues after deferral (#20062)
  • Disabled edit button in task instances list view page (#20659)
  • Fix a package name import error (#20519) (#20519)
  • Remove execution_date label when get cleanup pods list (#20417)
  • Remove unneeded FAB REST API endpoints (#20487)
  • Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700)
  • Sanity check for MySQL's TIMESTAMP column (#19821)
  • Allow using default celery command group with executors subclassed from Celery-based executors. (#18189)
  • Move class_permission_name to mixin so it applies to all classes (#18749)
  • Adjust trimmed_pod_id and replace '.' with '-' (#19036)
  • Pass custom_headers to send_email and send_email_smtp (#19009)
  • Ensure catchup=False is used in example dags (#19396)
  • Edit permalinks in OpenApi description file (#19244)
  • Navigate directly to DAG when selecting from search typeahead list (#18991)
  • [Minor] Fix padding on home page (#19025)

Doc only changes

  • Update doc for DAG file processing (#23209)
  • Replace changelog/updating with release notes and towncrier now (#22003)
  • Fix wrong reference in tracking-user-activity.rst (#22745)
  • Remove references to rbac = True from docs (#22725)
  • Doc: Update description for executor-bound dependencies (#22601)
  • Update check-health.rst (#22372)
  • Stronger language about Docker Compose customizability (#22304)
  • Update logging-tasks.rst (#22116)
  • Add example config of sql_alchemy_connect_args (#22045)
  • Update best-practices.rst (#22053)
  • Add information on DAG pausing/deactivation/deletion (#22025)
  • Add brief examples of integration test dags you might want (#22009)
  • Run inclusive language check on CHANGELOG (#21980)
  • Add detailed email docs for Sendgrid (#21958)
  • Add docs for db upgrade / db downgrade (#21879)
  • Update modules_management.rst (#21889)
  • Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887)
  • Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446)
  • Fix doc - replace decreasing by increasing (#21805)
  • Add another way to dynamically generate DAGs to docs (#21297)
  • Add extra information about time synchronization needed (#21685)
  • Update debug.rst docs (#21246)
  • Replaces the usage of postgres:// with postgresql:// (#21205)
  • Fix task execution process in CeleryExecutor docs (#20783)

Misc/Internal

  • Bring back deprecated security manager functions (#23243)
  • Replace usage of DummyOperator with EmptyOperator (#22974)
  • Deprecate DummyOperator in favor of EmptyOperator (#22832)
  • Remove unnecessary python 3.6 conditionals (#20549)
  • Bump moment from 2.29.1 to 2.29.2 in /airflow/www (#22873)
  • Bump prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823)
  • Bump nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803)
  • Bump minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798)
  • Remove dag parsing from db init command (#22531)
  • Update our approach for executor-bound dependencies (#22573)
  • Use Airflow.Base.metadata in FAB models (#22353)
  • Limit docutils to make our documentation pretty again (#22420)
  • Add Python 3.10 support (#22050)
  • [FEATURE] add 1.22 1.23 K8S support (#21902)
  • Remove pandas upper limit now that SQLA is 1.4+ (#22162)
  • Patch sql_alchemy_conn if old postgres scheme used (#22333)
  • Protect against accidental misuse of XCom.get_value() (#22244)
  • Order filenames for migrations (#22168)
  • Don't try to auto generate migrations for Celery tables (#22120)
  • Require SQLAlchemy 1.4 (#22114)
  • bump sphinx-jinja (#22101)
  • Add compat shim for SQLAlchemy to avoid warnings (#21959)
  • Rename xcom.dagrun_id to xcom.dag_run_id (#21806)
  • Deprecate non-JSON conn.extra (#21816)
  • Bump upper bound version of jsonschema to 5.0 (#21712)
  • Deprecate helper utility days_ago (#21653)
  • Remove :type lines now sphinx-autoapi supports type hints (#20951)
  • Silence deprecation warning in tests (#20900)
  • Use DagRun.run_id instead of execution_date when updating state of TIs (UI & REST API) (#18724)
  • Add Context stub to Airflow packages (#20817)
  • Update Kubernetes library version (#18797)
  • Rename PodLauncher to PodManager (#20576)
  • Removes Python 3.6 support (#20467)
  • Add deprecation warning for non-json-serializable params (#20174)
  • Rename TaskMixin to DependencyMixin (#20297)
  • Deprecate passing execution_date to XCom methods (#19825)
  • Remove get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961)
  • Remove postgres 9.6 support (#19987)
  • Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639)
  • add kubernetes 1.21 support (#19557)
  • Add FAB base class and set import_name explicitly. (#19667)
  • Removes unused state transitions to handle auto-changing view permissions. (#19153)
  • Chore: Use enum for __var and __type members (#19303)
  • Use fab models (#19121)
  • Consolidate method names between Airflow Security Manager and FAB default (#18726)
  • Remove distutils usages for Python 3.10 (#19064)
  • Removing redundant max_tis_per_query initialisation on SchedulerJob (#19020)
  • Remove deprecated usage of init_role() from API (#18820)
  • Remove duplicate code on dbapi hook (#18821)
Assets 11
Loading