You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: sections/parallel-programming.qmd
+7-2
Original file line number
Diff line number
Diff line change
@@ -402,7 +402,7 @@ In addition, Parsl supports a lot of different kinds of [providers](https://pars
402
402
Similarly to before, we start by configuring an executor in parsl, and loading it. We'll use multiprocessing by configuring the `HighThroughputExecutor` to use our local resources as a cluster, and we'll activate our virtual environment to be sure we're executing in a consistent environment.
403
403
404
404
```{python}
405
-
#| eval: true
405
+
#| eval: false
406
406
407
407
# Required packages
408
408
import parsl
@@ -430,6 +430,7 @@ parsl.load(htex_local)
430
430
We now have a live parsl executor (`htex_local`) that is waiting to execute processes. We tell it to execute processes by annotating functions with decorators that indicate which tasks should be parallelized. Parsl then handles the scheduling and execution of those tasks based on the dependencies between them. In the simplest case, we'll decorate our previous function for downloading a file with the `@python_app` decorator, which tells parsl that any function calls with this function should be run on the default executor (in this case, `htex_local`).
431
431
432
432
```{python}
433
+
#! eval: false
433
434
# Decorators seem to be ignored as the first line of a cell, so print something first
434
435
print("Create decorated function")
435
436
@@ -448,7 +449,7 @@ def download_file_parsl(row):
448
449
Now we just write regular python code that calls that function, and parsl handles the scheduling. Parsl app executors return an [`AppFuture`](https://parsl.readthedocs.io/en/stable/userguide/futures.html#appfutures), and we can call the `AppFuture.done()` function to determine when the future result is ready without blocking. Or, we can just block on `AppFuture.result()` which waits for each of the executions to complete and then returns the result.
When we're done, be sure to clean up and shutdown the `htex_local` executor, or it will continue to persist in your environment and utilize resources. Generally, an executor should be created when setting up your environment, and then it can be used repeatedly for many different tasks.
473
474
474
475
```{python}
476
+
#! eval: false
475
477
htex_local.executors[0].shutdown()
476
478
parsl.clear()
477
479
```
@@ -483,6 +485,7 @@ Parsl and other concurrency libraries generally provide both **blocking** and **
483
485
In practice this means that we can either 1) wait for all async calls to complete, and then process them using the blocking methods, or 2) query with a non-blocking method to see if each async call is complete, and only then retrieve the results for that method. We illustrate this approach below with parsl.
484
486
485
487
```{python}
488
+
#! eval: false
486
489
# Required packages
487
490
import parsl
488
491
from parsl import python_app
@@ -509,6 +512,7 @@ parsl.load(htex_local)
509
512
Define the task we want to run.
510
513
511
514
```{python}
515
+
#! eval: false
512
516
@python_app
513
517
def do_stuff(x):
514
518
import time
@@ -519,6 +523,7 @@ def do_stuff(x):
519
523
And now execute the tasks with some sleeps to see which are blocking and which are completed.
0 commit comments