Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ring-based reduce-scatter pipelining, ATen implementation #2950

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

ring-based reduce-scatter pipelining, ATen implementation #2950

wants to merge 27 commits into from

Commits on Sep 17, 2024

  1. duplication of test and class

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ffc7b19 View commit details
    Browse the repository at this point in the history

  2. WIP ATen ring RS with hang

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    d9c842f View commit details
    Browse the repository at this point in the history

  3. reproducer for hang in send/recv

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    4afcda1 View commit details
    Browse the repository at this point in the history

  4. still hanging

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    86a0967 View commit details
    Browse the repository at this point in the history

  5. not hanging in nccl, hanging in ucc

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0977cc5 View commit details
    Browse the repository at this point in the history

  6. hangs for both. Conclusion: must be posted in the right order

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    e34ff63 View commit details
    Browse the repository at this point in the history

  7. hang with all backends (nccl hangs at posting... which therefore seem… 

    …s blocking) with 3 ranks which all first post send and then recv
    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0743101 View commit details
    Browse the repository at this point in the history

  8. no hang with alternating order of posting send and recv

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c36b8cc View commit details
    Browse the repository at this point in the history

  9. hangs when all rank first post recv

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    532955e View commit details
    Browse the repository at this point in the history

  10. error with ucc: symbol lookup error: /usr/local/ucx/lib/ucx/libuct_cu… 

    …da_gdrcopy.so.0: undefined symbol: gdr_get_info_v2
    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    b828755 View commit details
    Browse the repository at this point in the history

  11. working with S=1

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ae31616 View commit details
    Browse the repository at this point in the history

  12. add multistreaming and number_of_iterations

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    017b36f View commit details
    Browse the repository at this point in the history

  13. fixed for 4 devices. moving coalescing outside the main for loop

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    47c21f4 View commit details
    Browse the repository at this point in the history

  14. working chkpt with non trivial S

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1512842 View commit details
    Browse the repository at this point in the history

  15. number_of_repetition

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    d8dbc80 View commit details
    Browse the repository at this point in the history

  16. clean

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    af789bb View commit details
    Browse the repository at this point in the history

  17. clean

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0487a80 View commit details
    Browse the repository at this point in the history

  18. refactor test case classes

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    94d683d View commit details
    Browse the repository at this point in the history

  19. lintrunner

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    fd134a4 View commit details
    Browse the repository at this point in the history

  20. remove hang reproducer with send/recv p2p

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1fe3dfb View commit details
    Browse the repository at this point in the history

  21. clean

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ee8f59c View commit details
    Browse the repository at this point in the history

  22. fix stream round robin

    @samnordmann
    samnordmann committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    dff90d0 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. fix recv rank, and add option whether to use coalescence

    @samnordmann
    samnordmann committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    f81d903 View commit details
    Browse the repository at this point in the history

  2. lintrunner

    @samnordmann
    samnordmann committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b1cf578 View commit details
    Browse the repository at this point in the history

  3. fix number of streams

    @samnordmann
    samnordmann committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b56c1b9 View commit details
    Browse the repository at this point in the history

  4. use option to change backend and not coalesce

    @samnordmann
    samnordmann committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b7e7d56 View commit details
    Browse the repository at this point in the history

  5. do not test UCC

    @samnordmann
    samnordmann committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    582613a View commit details
    Browse the repository at this point in the history