-
Notifications
You must be signed in to change notification settings - Fork 452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BE] github actions migration #2422
Merged
Merged
Changes from all commits
Commits
Show all changes
86 commits
Select commit
Hold shift + click to select a range
aa5da48
add github actions and workflows test for basic setup, linting, and i…
aclegg3 3ceaeb0
add cmake install
aclegg3 9951088
add shell for cmake build
aclegg3 0f39852
add ssh debug
aclegg3 c7b01b8
try sudo
aclegg3 bda25cb
add dependency installs and adjust python bin path
aclegg3 ec0152f
try activating conda env correctly
aclegg3 67b4e14
try changing the permissions and not building with sudo
aclegg3 c67695a
refresh PATH to find cmake?
aclegg3 7adb2a3
more details
aclegg3 db96b48
try again
aclegg3 d006968
try opening permissions to /usr/local/bin
aclegg3 282ba8d
open ssh again
aclegg3 79626d9
remove failing cmake call
aclegg3 e87bee2
changing permissions broke sudo
aclegg3 5f92e90
fix build process and download test data before ssh
aclegg3 50dc339
move ssh back up
aclegg3 f758149
reset owner recursively for cmake dir
aclegg3 4e0e20d
setup conda env activation uniformly
aclegg3 79545f2
Include Ubuntu NVIDIA dependencies to CI.
0mdc 1e39fa8
Merge remote-tracking branch 'origin/main' into alex-07_02-github_act…
aclegg3 2169f43
remove JS req cmake20 and move the git-lfs install to the deps instal…
aclegg3 f813cf3
add testing to the workflow
aclegg3 33333fb
use correct conda activate style
aclegg3 ab2622f
add some tests and try updating smake permissions to clear ctest bug
aclegg3 1caa62b
fix workflow config typo
aclegg3 36ba584
ssh debugging
aclegg3 9ad32ad
symblink the ctest binary also
aclegg3 c103a41
move debugging
aclegg3 6981188
remove a broken test and move the audio testing back into the primary…
aclegg3 a653561
try pyest without cuda earlier in the process as a sanity check
aclegg3 7e045e1
try with different pythonpath
aclegg3 38252bd
debugging
aclegg3 e4c54f4
try moving all cuda/gl installation into the action runner and inform…
aclegg3 85c3dbd
remove debugging prompt to let the tests run
aclegg3 8e5956f
add conda binary build test
aclegg3 ac6e70f
try without new section to check for yaml error
aclegg3 fe51770
try adding back the obvious
aclegg3 700b694
remove more questionable lines
aclegg3 8dcff67
try toy example
aclegg3 337e42f
try a copy example
aclegg3 037fe53
add the env variables
aclegg3 53d69ef
add a secrete env variable
aclegg3 0473615
remove test build job and add more complexity
aclegg3 20d8c8e
try again
aclegg3 677c041
correct format
aclegg3 d0c1f8b
try a few more edits
aclegg3 0219afb
try again
aclegg3 e21ee95
try removing more
aclegg3 16deed7
back to something that works?
aclegg3 4b5823a
one step
aclegg3 ae8d2c5
try to conditional setter
aclegg3 e4147d8
try again
aclegg3 b3ed1c7
another way
aclegg3 1007934
expand
aclegg3 b47e178
try another
aclegg3 26829e3
setup the actual trigger set
aclegg3 6780cfb
try again
aclegg3 554b438
fix if then typo
aclegg3 710ace6
try to create a nightly var
aclegg3 52f18a5
try adding the docker build
aclegg3 eb93eea
fill the empty space with echo
aclegg3 0699dd8
try to add the ci conda build and pre-commit job
aclegg3 f6d0fc1
typo in conda build command
aclegg3 673dc11
add a debug ssh session to test out the conda command
aclegg3 1a1da75
remove the debugging and try without -it
aclegg3 e255dc1
add back testing, add osx conda build, add ubuntu conda deployment
aclegg3 e679379
try setting up python 3.9 before osx install
aclegg3 67de84c
use 3.9.20 for arm64 compat
aclegg3 5ddf6ba
use python 3.9.13 for arm64 darwin compat
aclegg3 9ff4abf
install miniconda with python 3.9
aclegg3 a198536
add ssh debugging to figure out the python issue
aclegg3 8f46c78
remove the ssh and try conda init
aclegg3 ca25eb1
always need a username/password
aclegg3 e4fabb8
move test data download to action file, add start of lab build job, t…
aclegg3 0c5dee3
add back the cmake permissions call
aclegg3 644c027
add --yes
aclegg3 0dcd196
add the rest of the lab testing blocks
aclegg3 ae01cd0
remove the touch command
aclegg3 3a2c6f3
add doc deployment step
aclegg3 9567633
try to fix odd yaml issue
aclegg3 6dfc86d
cleanup
aclegg3 b7f7c61
try to fix downgrade issue in nvidia packages
aclegg3 70f6ce3
address reviews: try streamlined cmake install, remove some comments
aclegg3 b7f1f2a
revert to previous cmake install style
aclegg3 1ee62e9
trigger another build attempt
aclegg3 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
name: download_test_data | ||
runs: | ||
using: composite | ||
steps: | ||
- name: download_test_data | ||
run: | | ||
# Disable clone protection for git lfs | ||
export GIT_CLONE_PROTECTION_ACTIVE=false | ||
|
||
git --version | ||
git-lfs --version | ||
export PATH=$HOME/miniconda/bin:/usr/local/cuda/bin:$PATH | ||
conda init | ||
source ~/.bashrc | ||
conda activate habitat | ||
conda install -y gitpython git-lfs | ||
cd habitat-sim | ||
git lfs install | ||
python src_python/habitat_sim/utils/datasets_download.py --uids ci_test_assets ycb rearrange_dataset_v2 --replace --data-path data/ --no-prune | ||
ls -la data/scene_datasets/habitat-test-scenes/ | ||
shell: bash -el {0} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
name: install_ubuntu_deps | ||
runs: | ||
using: composite | ||
steps: | ||
- name: install cmake | ||
# OpenEXR requires CMake 3.12. We also download CMake 3.20, which | ||
# we'll use later for the JS build only. | ||
run: | | ||
echo $(date +%F) > ./date | ||
echo $(git ls-remote https://github.com/facebookresearch/habitat-lab.git HEAD | awk '{ print $1}') > ./hablab_sha | ||
cat ./hablab_sha | ||
wget https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh | ||
sudo mkdir /opt/cmake312 | ||
sudo sh ./cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake312 --skip-license | ||
sudo ln -s /opt/cmake312/bin/cmake /usr/local/bin/cmake | ||
sudo ln -s /opt/cmake312/bin/ctest /usr/local/bin/ctest | ||
shell: bash | ||
- name: Install dependencies | ||
run: |- | ||
echo "Install dependencies" | ||
sudo apt-get update || true | ||
sudo apt-get install -y --no-install-recommends \ | ||
build-essential \ | ||
git \ | ||
git-lfs \ | ||
curl \ | ||
vim \ | ||
ca-certificates \ | ||
libjpeg-dev \ | ||
libglm-dev \ | ||
libegl1-mesa-dev \ | ||
ninja-build \ | ||
xorg-dev \ | ||
freeglut3-dev \ | ||
pkg-config \ | ||
wget \ | ||
zip \ | ||
lcov\ | ||
libhdf5-dev \ | ||
libomp-dev \ | ||
unzip || true | ||
shell: bash | ||
- name: Setup miniconda | ||
uses: conda-incubator/[email protected] | ||
with: | ||
miniconda-version: "latest" | ||
python-version: "3.9" | ||
activate-environment: "habitat" | ||
- name: Install conda and dependencies | ||
run: |- | ||
echo "Install conda and dependencies" | ||
conda install -y pytorch==1.12.1 torchvision==0.13.1 -c pytorch -c nvidia | ||
conda install -y -c conda-forge ninja numpy pytest pytest-cov ccache hypothesis pytest-mock | ||
pip install pytest-sugar pytest-xdist pytest-benchmark opencv-python cython mock | ||
shell: bash -el {0} | ||
- name: Validate Pytorch Installation | ||
run: |- | ||
echo "Validate Pytorch Installation" | ||
# Check that pytorch is installed with CUDA. | ||
python -c 'import torch; torch.cuda.set_device(0)' | ||
shell: bash -el {0} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# This action installs required Ubuntu NVIDIA dependencies. | ||
# It assumes that `nvidia-smi` is available on the system. | ||
name: "Install GPU Dependencies" | ||
runs: | ||
using: composite | ||
steps: | ||
- name: Install GPU Dependencies | ||
shell: bash | ||
run: | | ||
NVIDIA_SMI_OUTPUT=$(nvidia-smi) | ||
# Extract CUDA version (e.g., 11.7) | ||
CUDA_VERSION=$(echo "$NVIDIA_SMI_OUTPUT" | grep -oP 'CUDA Version:\s+\K[\d.]+') | ||
|
||
FULL_DRIVER_VERSION=$(echo "$NVIDIA_SMI_OUTPUT" | grep -oP 'Driver Version:\s+\K[\d.]+') | ||
MAJOR_DRIVER_VERSION=$(echo "$NVIDIA_SMI_OUTPUT" | grep -oP 'Driver Version:\s+\K\d+') | ||
UBUNTU_PACKAGE_SUFFIX="${MAJOR_DRIVER_VERSION}=${FULL_DRIVER_VERSION}-0ubuntu1" | ||
sudo apt-get update | ||
sudo apt-get install -y cuda-toolkit-${CUDA_VERSION} | ||
sudo apt-get install -y nvidia-gds-${CUDA_VERSION} | ||
sudo apt-get install -y libnvidia-common-${UBUNTU_PACKAGE_SUFFIX} | ||
sudo apt-get install -y libnvidia-gl-${UBUNTU_PACKAGE_SUFFIX} | ||
sudo apt-get install -y libnvidia-compute-${UBUNTU_PACKAGE_SUFFIX} --allow-downgrades | ||
sudo apt-get install -y libnvidia-decode-${UBUNTU_PACKAGE_SUFFIX} | ||
sudo apt-get install -y libnvidia-encode-${UBUNTU_PACKAGE_SUFFIX} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would simplify this with
sudo apt install -y cmake
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can try this and see if it works. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nah, cmake isn't available on apt. I think that's the reason for this installation method initially.
Anyway, feel free to fiddle with it as a follow-up.