Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BE] github actions migration #2422

Merged
merged 86 commits into from
Nov 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
aa5da48
add github actions and workflows test for basic setup, linting, and i…
aclegg3 Jul 2, 2024
3ceaeb0
add cmake install
aclegg3 Jul 3, 2024
9951088
add shell for cmake build
aclegg3 Jul 3, 2024
0f39852
add ssh debug
aclegg3 Jul 3, 2024
c7b01b8
try sudo
aclegg3 Jul 3, 2024
bda25cb
add dependency installs and adjust python bin path
aclegg3 Jul 3, 2024
ec0152f
try activating conda env correctly
aclegg3 Jul 3, 2024
67b4e14
try changing the permissions and not building with sudo
aclegg3 Jul 3, 2024
c67695a
refresh PATH to find cmake?
aclegg3 Jul 3, 2024
7adb2a3
more details
aclegg3 Jul 3, 2024
db96b48
try again
aclegg3 Jul 3, 2024
d006968
try opening permissions to /usr/local/bin
aclegg3 Jul 3, 2024
282ba8d
open ssh again
aclegg3 Jul 10, 2024
79626d9
remove failing cmake call
aclegg3 Jul 10, 2024
e87bee2
changing permissions broke sudo
aclegg3 Jul 10, 2024
5f92e90
fix build process and download test data before ssh
aclegg3 Jul 10, 2024
50dc339
move ssh back up
aclegg3 Jul 10, 2024
f758149
reset owner recursively for cmake dir
aclegg3 Jul 10, 2024
4e0e20d
setup conda env activation uniformly
aclegg3 Jul 10, 2024
79545f2
Include Ubuntu NVIDIA dependencies to CI.
0mdc Nov 18, 2024
1e39fa8
Merge remote-tracking branch 'origin/main' into alex-07_02-github_act…
aclegg3 Nov 18, 2024
2169f43
remove JS req cmake20 and move the git-lfs install to the deps instal…
aclegg3 Nov 18, 2024
f813cf3
add testing to the workflow
aclegg3 Nov 19, 2024
33333fb
use correct conda activate style
aclegg3 Nov 19, 2024
ab2622f
add some tests and try updating smake permissions to clear ctest bug
aclegg3 Nov 19, 2024
1caa62b
fix workflow config typo
aclegg3 Nov 19, 2024
36ba584
ssh debugging
aclegg3 Nov 19, 2024
9ad32ad
symblink the ctest binary also
aclegg3 Nov 19, 2024
c103a41
move debugging
aclegg3 Nov 19, 2024
6981188
remove a broken test and move the audio testing back into the primary…
aclegg3 Nov 19, 2024
a653561
try pyest without cuda earlier in the process as a sanity check
aclegg3 Nov 19, 2024
7e045e1
try with different pythonpath
aclegg3 Nov 19, 2024
38252bd
debugging
aclegg3 Nov 20, 2024
e4c54f4
try moving all cuda/gl installation into the action runner and inform…
aclegg3 Nov 20, 2024
85c3dbd
remove debugging prompt to let the tests run
aclegg3 Nov 20, 2024
8e5956f
add conda binary build test
aclegg3 Nov 20, 2024
ac6e70f
try without new section to check for yaml error
aclegg3 Nov 20, 2024
fe51770
try adding back the obvious
aclegg3 Nov 20, 2024
700b694
remove more questionable lines
aclegg3 Nov 20, 2024
8dcff67
try toy example
aclegg3 Nov 20, 2024
337e42f
try a copy example
aclegg3 Nov 20, 2024
037fe53
add the env variables
aclegg3 Nov 20, 2024
53d69ef
add a secrete env variable
aclegg3 Nov 20, 2024
0473615
remove test build job and add more complexity
aclegg3 Nov 20, 2024
20d8c8e
try again
aclegg3 Nov 20, 2024
677c041
correct format
aclegg3 Nov 20, 2024
d0c1f8b
try a few more edits
aclegg3 Nov 20, 2024
0219afb
try again
aclegg3 Nov 20, 2024
e21ee95
try removing more
aclegg3 Nov 20, 2024
16deed7
back to something that works?
aclegg3 Nov 20, 2024
4b5823a
one step
aclegg3 Nov 20, 2024
ae8d2c5
try to conditional setter
aclegg3 Nov 20, 2024
e4147d8
try again
aclegg3 Nov 20, 2024
b3ed1c7
another way
aclegg3 Nov 20, 2024
1007934
expand
aclegg3 Nov 20, 2024
b47e178
try another
aclegg3 Nov 20, 2024
26829e3
setup the actual trigger set
aclegg3 Nov 20, 2024
6780cfb
try again
aclegg3 Nov 20, 2024
554b438
fix if then typo
aclegg3 Nov 20, 2024
710ace6
try to create a nightly var
aclegg3 Nov 20, 2024
52f18a5
try adding the docker build
aclegg3 Nov 20, 2024
eb93eea
fill the empty space with echo
aclegg3 Nov 20, 2024
0699dd8
try to add the ci conda build and pre-commit job
aclegg3 Nov 20, 2024
f6d0fc1
typo in conda build command
aclegg3 Nov 20, 2024
673dc11
add a debug ssh session to test out the conda command
aclegg3 Nov 20, 2024
1a1da75
remove the debugging and try without -it
aclegg3 Nov 20, 2024
e255dc1
add back testing, add osx conda build, add ubuntu conda deployment
aclegg3 Nov 21, 2024
e679379
try setting up python 3.9 before osx install
aclegg3 Nov 21, 2024
67de84c
use 3.9.20 for arm64 compat
aclegg3 Nov 21, 2024
5ddf6ba
use python 3.9.13 for arm64 darwin compat
aclegg3 Nov 21, 2024
9ff4abf
install miniconda with python 3.9
aclegg3 Nov 21, 2024
a198536
add ssh debugging to figure out the python issue
aclegg3 Nov 21, 2024
8f46c78
remove the ssh and try conda init
aclegg3 Nov 21, 2024
ca25eb1
always need a username/password
aclegg3 Nov 21, 2024
e4fabb8
move test data download to action file, add start of lab build job, t…
aclegg3 Nov 21, 2024
0c5dee3
add back the cmake permissions call
aclegg3 Nov 21, 2024
644c027
add --yes
aclegg3 Nov 21, 2024
0dcd196
add the rest of the lab testing blocks
aclegg3 Nov 22, 2024
ae01cd0
remove the touch command
aclegg3 Nov 22, 2024
3a2c6f3
add doc deployment step
aclegg3 Nov 22, 2024
9567633
try to fix odd yaml issue
aclegg3 Nov 22, 2024
6dfc86d
cleanup
aclegg3 Nov 25, 2024
b7f7c61
try to fix downgrade issue in nvidia packages
aclegg3 Nov 25, 2024
70f6ce3
address reviews: try streamlined cmake install, remove some comments
aclegg3 Nov 25, 2024
b7f1f2a
revert to previous cmake install style
aclegg3 Nov 25, 2024
1ee62e9
trigger another build attempt
aclegg3 Nov 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/actions/download_test_data/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: download_test_data
runs:
using: composite
steps:
- name: download_test_data
run: |
# Disable clone protection for git lfs
export GIT_CLONE_PROTECTION_ACTIVE=false

git --version
git-lfs --version
export PATH=$HOME/miniconda/bin:/usr/local/cuda/bin:$PATH
conda init
source ~/.bashrc
conda activate habitat
conda install -y gitpython git-lfs
cd habitat-sim
git lfs install
python src_python/habitat_sim/utils/datasets_download.py --uids ci_test_assets ycb rearrange_dataset_v2 --replace --data-path data/ --no-prune
ls -la data/scene_datasets/habitat-test-scenes/
shell: bash -el {0}
61 changes: 61 additions & 0 deletions .github/actions/install_ubuntu_deps/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: install_ubuntu_deps
runs:
using: composite
steps:
- name: install cmake
# OpenEXR requires CMake 3.12. We also download CMake 3.20, which
# we'll use later for the JS build only.
run: |
echo $(date +%F) > ./date
echo $(git ls-remote https://github.com/facebookresearch/habitat-lab.git HEAD | awk '{ print $1}') > ./hablab_sha
cat ./hablab_sha
wget https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh
sudo mkdir /opt/cmake312
sudo sh ./cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake312 --skip-license
sudo ln -s /opt/cmake312/bin/cmake /usr/local/bin/cmake
sudo ln -s /opt/cmake312/bin/ctest /usr/local/bin/ctest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would simplify this with sudo apt install -y cmake.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can try this and see if it works. 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, cmake isn't available on apt. I think that's the reason for this installation method initially.
Anyway, feel free to fiddle with it as a follow-up.

shell: bash
- name: Install dependencies
run: |-
echo "Install dependencies"
sudo apt-get update || true
sudo apt-get install -y --no-install-recommends \
build-essential \
git \
git-lfs \
curl \
vim \
ca-certificates \
libjpeg-dev \
libglm-dev \
libegl1-mesa-dev \
ninja-build \
xorg-dev \
freeglut3-dev \
pkg-config \
wget \
zip \
lcov\
libhdf5-dev \
libomp-dev \
unzip || true
shell: bash
- name: Setup miniconda
uses: conda-incubator/[email protected]
with:
miniconda-version: "latest"
python-version: "3.9"
activate-environment: "habitat"
- name: Install conda and dependencies
run: |-
echo "Install conda and dependencies"
conda install -y pytorch==1.12.1 torchvision==0.13.1 -c pytorch -c nvidia
conda install -y -c conda-forge ninja numpy pytest pytest-cov ccache hypothesis pytest-mock
pip install pytest-sugar pytest-xdist pytest-benchmark opencv-python cython mock
shell: bash -el {0}
- name: Validate Pytorch Installation
run: |-
echo "Validate Pytorch Installation"
# Check that pytorch is installed with CUDA.
python -c 'import torch; torch.cuda.set_device(0)'
shell: bash -el {0}
24 changes: 24 additions & 0 deletions .github/actions/install_ubuntu_gpu_deps/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# This action installs required Ubuntu NVIDIA dependencies.
# It assumes that `nvidia-smi` is available on the system.
name: "Install GPU Dependencies"
runs:
using: composite
steps:
- name: Install GPU Dependencies
shell: bash
run: |
NVIDIA_SMI_OUTPUT=$(nvidia-smi)
# Extract CUDA version (e.g., 11.7)
CUDA_VERSION=$(echo "$NVIDIA_SMI_OUTPUT" | grep -oP 'CUDA Version:\s+\K[\d.]+')

FULL_DRIVER_VERSION=$(echo "$NVIDIA_SMI_OUTPUT" | grep -oP 'Driver Version:\s+\K[\d.]+')
MAJOR_DRIVER_VERSION=$(echo "$NVIDIA_SMI_OUTPUT" | grep -oP 'Driver Version:\s+\K\d+')
UBUNTU_PACKAGE_SUFFIX="${MAJOR_DRIVER_VERSION}=${FULL_DRIVER_VERSION}-0ubuntu1"
sudo apt-get update
sudo apt-get install -y cuda-toolkit-${CUDA_VERSION}
sudo apt-get install -y nvidia-gds-${CUDA_VERSION}
sudo apt-get install -y libnvidia-common-${UBUNTU_PACKAGE_SUFFIX}
sudo apt-get install -y libnvidia-gl-${UBUNTU_PACKAGE_SUFFIX}
sudo apt-get install -y libnvidia-compute-${UBUNTU_PACKAGE_SUFFIX} --allow-downgrades
sudo apt-get install -y libnvidia-decode-${UBUNTU_PACKAGE_SUFFIX}
sudo apt-get install -y libnvidia-encode-${UBUNTU_PACKAGE_SUFFIX}
Loading
Loading