Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider refactoring I/O parts #211

Open
JCGoran opened this issue Sep 17, 2024 · 0 comments
Open

Consider refactoring I/O parts #211

JCGoran opened this issue Sep 17, 2024 · 0 comments

Comments

@JCGoran
Copy link
Contributor

JCGoran commented Sep 17, 2024

Doing some benchmarks, I've noticed that the I/O parts are done in the main loop, which causes an increase in the total runtime.
As an example, we have:

time python -m data_morph --seed 42 --start-shape panda --target-shape star --iterations 10000
13.81s user 0.79s system 109% cpu 13.310 total

but if we remove this block (which is in the loop):

frame_number = record_frames(
data=morphed_data,
count=frame_numbers.count(i),
frame_number=frame_number,
)

we achieve a major speed-up:

time python -m data_morph --seed 42 --start-shape panda --target-shape star --iterations 10000
5.41s user 0.28s system 137% cpu 4.149 total

I'd propose that instead of doing the I/O in the main loop, the frames that would be written to disk are simply stored in some internal list, and then I/O is done after all the computations. This has several benefits:

  1. if keep_frames=False, we don't output anything other than the final GIF, so we spare the disk from unnecessary writes
  2. if keep_frames=True, since the task is probably I/O bound, we can take advantage of the concurrent.futures module to do it concurrently, so the speed-up is probably still significant
  3. less error-prone since we don't need to find the files. There's also no guarantee that the files won't change on disk while the sim is running
  4. the most obvious one, speed! If using keep_frames=False, which is the default, we just output one file instead of potentially hundreds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant