Skip to content

Commit 728c052

Browse files
committedFeb 6, 2025·
turn off some chunks
1 parent 503f207 commit 728c052

File tree

1 file changed

+7
-2
lines changed

1 file changed

+7
-2
lines changed
 

‎sections/parallel-programming.qmd

+7-2
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,7 @@ In addition, Parsl supports a lot of different kinds of [providers](https://pars
402402
Similarly to before, we start by configuring an executor in parsl, and loading it. We'll use multiprocessing by configuring the `HighThroughputExecutor` to use our local resources as a cluster, and we'll activate our virtual environment to be sure we're executing in a consistent environment.
403403

404404
```{python}
405-
#| eval: true
405+
#| eval: false
406406
407407
# Required packages
408408
import parsl
@@ -430,6 +430,7 @@ parsl.load(htex_local)
430430
We now have a live parsl executor (`htex_local`) that is waiting to execute processes. We tell it to execute processes by annotating functions with decorators that indicate which tasks should be parallelized. Parsl then handles the scheduling and execution of those tasks based on the dependencies between them. In the simplest case, we'll decorate our previous function for downloading a file with the `@python_app` decorator, which tells parsl that any function calls with this function should be run on the default executor (in this case, `htex_local`).
431431

432432
```{python}
433+
#! eval: false
433434
# Decorators seem to be ignored as the first line of a cell, so print something first
434435
print("Create decorated function")
435436
@@ -448,7 +449,7 @@ def download_file_parsl(row):
448449
Now we just write regular python code that calls that function, and parsl handles the scheduling. Parsl app executors return an [`AppFuture`](https://parsl.readthedocs.io/en/stable/userguide/futures.html#appfutures), and we can call the `AppFuture.done()` function to determine when the future result is ready without blocking. Or, we can just block on `AppFuture.result()` which waits for each of the executions to complete and then returns the result.
449450

450451
```{python}
451-
#! eval: true
452+
#! eval: false
452453
print("Define our download_futures function")
453454
454455
@timethis
@@ -472,6 +473,7 @@ wait_for_futures(file_table[0:5])
472473
When we're done, be sure to clean up and shutdown the `htex_local` executor, or it will continue to persist in your environment and utilize resources. Generally, an executor should be created when setting up your environment, and then it can be used repeatedly for many different tasks.
473474

474475
```{python}
476+
#! eval: false
475477
htex_local.executors[0].shutdown()
476478
parsl.clear()
477479
```
@@ -483,6 +485,7 @@ Parsl and other concurrency libraries generally provide both **blocking** and **
483485
In practice this means that we can either 1) wait for all async calls to complete, and then process them using the blocking methods, or 2) query with a non-blocking method to see if each async call is complete, and only then retrieve the results for that method. We illustrate this approach below with parsl.
484486

485487
```{python}
488+
#! eval: false
486489
# Required packages
487490
import parsl
488491
from parsl import python_app
@@ -509,6 +512,7 @@ parsl.load(htex_local)
509512
Define the task we want to run.
510513

511514
```{python}
515+
#! eval: false
512516
@python_app
513517
def do_stuff(x):
514518
import time
@@ -519,6 +523,7 @@ def do_stuff(x):
519523
And now execute the tasks with some sleeps to see which are blocking and which are completed.
520524

521525
```{python}
526+
#! eval: false
522527
import time
523528
all_futures = []
524529
for x in range(0,10):

0 commit comments

Comments
 (0)
Please sign in to comment.