Task Framework IdealState Dependency Removal Progression

Overview

This document outlines the progression of removing IdealState dependency from Task Framework. The process is segmented to different phases in order to maintain backward compatibility and ensure that every work item is of a reasonable size. This document will also be updated as more progress is made.

Progression Phases

Phase 1

Status

Merged: https://github.com/apache/helix/pull/1326

Change Logs

• In ResourceComputationStage, Resources are computed based on WorkflowConfigs and JobConfigs. This section of new code is placed after IdealState reading to overwrite the IdealState result; this allows the controller to function correctly without client side update;
• In TaskSchedulingStage, the legacy pipeline logic is removed. The legacy pipeline was for the workflows that rely purely on IdealStates and not WorkflowConfigs; with the new change, this is strictly contradicting with how Workflows should be created, and therefore this legacy logic is no longer supported;
• In TaskDriver, logics that create IdealStates for workflows are removed;
• In TaskUtil, logics that remove IdealStates are marked as deprecated. This is for backward compatibility reasons - leaving them ensures that any customer created IdealStates are cleaned up properly. These logics will be removed later.

Phase 2

Status

Planned.

Proposed Changes

• With code changes in relevant stages, it is possible to remove RESOURCES_TO_REBALANCE attribute. This is made possible because Task Framework no longer relies on IdealState and resource filtering is no longer necessary - meaning the code doesn't have to separate resource into RESOURCES map and RESOURCES_TO_REBALANCE map. This may also mean that in ResourceComputationStage, we no longer need to compute Resources based on TaskConfigs, as Resources wouldn't be used in the task pipeline.