optimize memory view using lttb sample #776
Conversation
|
@chaekit @aaronenyeshi I checked the failed test (test_tensorboard_end2end.py). This test compares the response from the tensorboard frontend with the manually generated and verified Since the memory curve has been sampled, it cannot match the previous manually generated result. As a result, the test fails. Shall I regenerate the |
|
@lh-ycx , yes please feel free to re-generate the file. can you please share screenshots of the before and after views of this change? |
|
OK, I'll work on it this week. |
|
Hi, @aaronenyeshi. Sorry for the late response. CI failures I've carefully checked the test case and found that the test would always fail in Python 3.9 and 3.10 (please refer to #785). I also find my commit is consistent with the content in Screenshots of the before and after views I use a 329M trace to show the before and after views as well as the memory footprint. (A larger trace (up to GBs) can be selected but the unoptimized view would crash.) Before view: After view: Before memory footprint: After memory footprint: |
|
@aaronenyeshi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@aaronenyeshi merged this pull request in 170d45a. |
Enhancement for Issue #760.
Hey guys, I've optimized the speed of the memory view using the LTTB sampling (Largest-Triangle-Three-Buckets sampling, which is able to downsample time series–like data while retaining the overall shape and variability in the data).
I've tested this using a PyTorch profiler trace of 2G, and the memory view page will not get crashed and the scaling operation is smooth and rather acceptable to me.