Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow manual control of tasks #17005

Open
kent-ps opened this issue Feb 6, 2025 · 0 comments
Open

allow manual control of tasks #17005

kent-ps opened this issue Feb 6, 2025 · 0 comments
Labels
enhancement An improvement of an existing feature

Comments

@kent-ps
Copy link

kent-ps commented Feb 6, 2025

Describe the current behavior

Currently the only (official/documented) way to define a task is adding @task decorator to functions. For jobs that involve large numbers of long interwoven concurrent tasks, they are often wrongly nested together in the Prefect Cloud UI.

Image

Describe the proposed behavior

I'd like to have the ability to:

  • manually define a task's parent, so I can choose where to nest a task
  • even better: manually create an empty task, which acts as a logical group of smaller tasks. This wrapping task will end when no more subtasks are manually added to it.

Example Use

def parse_sitemap(self, response, **cb_kwargs):
  group = new_task()
  ... 
  for listing in listings:
    yield Request(... cb_kwargs={'group_id' = group.id})

@staticmethod
def get_listing_parent():
    parameters = task_run.parameters
    group_id= parameters["cb_kwargs"].get('group_id')

    return group_id

@task(name="Parse listing", parent=get_listing_parent)
def parse_listing(self, response, **cb_kwargs):
  ...

Additional context

No response

@kent-ps kent-ps added the enhancement An improvement of an existing feature label Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

1 participant