How to force ProcessWorker to use all concurrency limits?

[dinya]

Hello, team.

I start previously defined ProcessWorker:

prefect worker start --pool "my-work-pool" --name "my-process-worker" --limit 10

And ask to run 100 flows with it (each flow starts number crunching).

But according to "Work Pools / my-work-pool" and Process Explorer no more than 4 (maybe 5) parallel processes are used.

The same thing happens using Prefect UI for set limit with "Work Pools / my-work-pool / Edit -> Flow Run Concurrency".

I haven't set any global concurrency limits.

Which extra settings should I set to force worker to use all 10 slots?

My env: Windows 10 / Python 3.12.9 / prefect 3.2.7 (self-hosted) / my CPU has 20 cores.

[dinya]

Related question. Should starting

prefect worker start --pool "my-work-pool" --name "my-process-worker" --limit 10

or

prefect worker start --pool "my-work-pool" --name "my-process-worker" --limit 10 --type process

result in any records in the tables concurrency_limit or concurrency_limit_v2?

[dinya]

I found the case when the ProcessWorker concurrency works in unexpected manner

---

1 When I set the limits for both:

1. Work pool (it works for any value except UNLIMITED as my experiments showed):

   prefect work-pool set-concurrency-limit "my-work-pool" 10

   (or set Flow Run Concurrency (Optional) in Prefect UI)

2. and then start the worker with limits:

   prefect worker start --pool "my-work-pool" --name "my-process-worker" --limit 10

When I execute

prefect work-pool clear-concurrency-limit "my-work-pool"

(or set Flow Run Concurrency (Optional) to UNLIMITED in Prefect UI)

and start

prefect worker start --pool "my-work-pool" --name "my-process-worker" --limit 10

then it works as expected -- 10 parallel (forked) processes.

(It works too if I clear concurrency limit while worker was started previously.)

---

2 When I start the worker without limits, but concurrency is set for work pool

When I start my worker without limits

prefect worker start --pool "my-work-pool" --name "my-process-worker"

and set (before/after)

prefect work-pool set-concurrency-limit "my-work-pool" 10

the worker doesn't start any flows.

---

But if I start the worker without limits and clear concurrency limit for the work pool the worker starts processes without any limits (this is quite expected).

---

Is it is a bug or a feature?