Skip to content

Latest commit

 

History

History
170 lines (84 loc) · 4.84 KB

README.md

File metadata and controls

170 lines (84 loc) · 4.84 KB
Raw

GPT-home-private

Setup:

  • Use AWS Deep Learning Base AMI

  • Install PyTorch env (old):

    pip3 install torch==1.9.0+cu111 torchtext -f https://download.pytorch.org/whl/torch_stable.html
pip3 install torch==1.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch==1.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

# Magic, not sure why cupy-cuda111 would not work, it seems that cupy-cuda111 will use different PTX from torch.
pip3 install cupy-cuda111==8.6.0
pip3 install transformers

  • Install PyTorch env (latest):

    pip3 install --pre torch==1.12.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install cupy-cuda11x==11.0.0
python3 -m cupyx.tools.install_library --cuda 11.x --library nccl
pip3 install transformers

  • Install PyTorch env (CPU-latest):

    pip3 install --pre torch==1.12.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

  • Install deepspeed for some micro-benchmark (optional)

    pip install deepspeed

  • Clone this repo:

    git clone https://github.com/BinhangYuan/GPT-home-private.git

  • set the github cache (Optional):

    git config credential.helper 'cache --timeout=30000'

  • Download a tiny dataset:

    wget https://binhang-language-datasets.s3.us-west-2.amazonaws.com/glue_qqp_dataset/data.tar.xz -P ./glue_dataset/

tar -xvf ./glue_dataset/data.tar.xz -C ./glue_dataset/

  • Setup network configuration:

    export GLOO_SOCKET_IFNAME=ens3

export NCCL_SOCKET_IFNAME=ens3

  • Use TC scripts to control network delay and bandwidth:

Run Distributed Gpipe:

  • On each node, run:

    python dist_pipeline_runner.py --dist-url tcp://XXX.XXX.XXX.XXX:9000 --world-size N --rank i (i=0,...,N-1)

Run deepspeed benchmark:

  • Update public-ip and hostname in the ./scripts/ip_list.sh file

  • Update the host name with slots (number of GPUs) in ./scripts/ds_hostnames.sh file

  • Sync code to all nodes

  • Setup password free ssh cluster by executing (under the ./script/ directory):

    bash ssh_pass_free.sh

  • SSH to the rank-0 node, on that node run:

    source activate pytorch_p38

  • A sample run:

    deepspeed --hostfile=./scripts/ds_hostnames dist_deepspeed_zero_s3.py --embedding-dim 2048 --seq-length 2048 --batch-size 1024 --num-layers 40 --micro-batch-size 4

  • Batch run all settings:

     cd ./scripts
 bash local_run_deepspeed_batch_on_rank0.sh #CASE

Run with Advanced Scripts (under scripts directory):

  • First update the public IPs and private IP of the rank-0 node in ip_list.sh.

  • Allow SSH connects:

    bash accept_ssh_keys.sh

  • Update local repository:

    bash aws_sync_code.sh #GIT_TOKEN

  • Enable environment: (This is optional but load conda env seems to be slow for the first time)

    bash aws_foo_load_lib.sh

  • Setup heterogeneous network (update the private IPs in ./scheduler/generate_heterogeneous_tc.py, sync the code to AWS!):

    bash aws_generate_heter_tc.sh #HETER_CASE (3/4/5)

  • Optional(Play with it by starting TC, this is not needed if you use the next block of cmds to start benchmarks):

    bash aws_start_heter_tc.sh #HETER_CASE (3/4/5)

  • Run Schedulers (under scheduler/heuristic_evolutionary_solver directory) to get assignments and estimated cost

    python scheduler.py

  • Update correspond aws_run_gpt3_*_training.sh with the above output:

  • Run Tasks (e.g.,):

    bash aws_run_gpt3_training.sh gpt3_small_pp3_dp4.sh
bash aws_run_gpt3_training.sh gpt3_small_pp3_dp4.sh #HETER_CASE
bash aws_run_gpt3_training.sh gpt3_small_pp3_dp4.sh #DELAY #BANDWIDTH

  • Clear logs:

    bash aws_clear_logs.sh

  • Copy training logs from Rank-0 node (For my implementation the benchmark result is on the rank-0 node.)

    bash copy_rank0_logs.sh

  • Download and generate trace:

    bash copy_traces.sh #PREFIX
bash generate_traces.sh #PREFIX

Run the System on Euler with Coordinator

  • First log in to ETH Zurich Euler cluster (need to use VPN if not on campus), our working directory:

    /nfs/iiscratch-zhang.inf.ethz.ch/export/zhang/export/fm/

  • Get the Euler ssh client's node IP, and start the coordinator server (under ./coordinator directory), e.g.:

    python coordinate_server.py --coordinator-server-ip 129.132.93.88 --coordinator-type inference/train

  • Alternatively, we can use the following script to submit multiple jobs:

    bash multi_inference_jobs_submit.sh 129.132.93.88 lsf_gptJ_inf_4RTX2080Ti 3 10

  • On a different terminal, start a job submission client to submit a job:

    python job_submit_client.py --coordinator-server-ip 129.132.93.88 --submit-job inference --job-name lsf_gptJ_inf_4RTX2080Ti

  • So far, we need to manually change the IP in the job-submit templates, e.g. change lsf_gptJ_inf_4gpu.bsub file.

  • Check submitted job states:

    bjobs