first commit

Author: Haotianz94

.gitignore

@@ -0,0 +1,7 @@
+*.pyc
+*.pth
+.vs/
+__pycache__/
+data
+results
+out

LICENSE.txt

@@ -0,0 +1,35 @@
+NVIDIA License
+
+1. Definitions
+
+“Licensor” means any person or entity that distributes its Work.
+“Work” means (a) the original work of authorship made available under this license, which may include software, documentation, or other files, and (b) any additions to or derivative works  thereof  that are made available under this license.
+The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this license, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work.
+Works are “made available” under this license by including in or with the Work either (a) a copyright notice referencing the applicability of this license to the Work, or (b) a copy of this license.
+
+2. License Grant
+
+2.1 Copyright Grant. Subject to the terms and conditions of this license, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to use, reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form.
+
+3. Limitations
+
+3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this license, (b) you include a complete copy of this license with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work.
+
+3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this license (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself.
+
+3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use non-commercially. Notwithstanding the foregoing, NVIDIA Corporation and its affiliates may use the Work and any derivative works commercially. As used herein, “non-commercially” means for research or evaluation purposes only.
+
+3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this license from such Licensor (including the grant in Section 2.1) will terminate immediately.
+
+3.5 Trademarks. This license does not grant any rights to use any Licensor’s or its affiliates’ names, logos, or trademarks, except as necessary to reproduce the notices described in this license.
+
+3.6 Termination. If you violate any term of this license, then your rights under this license (including the grant in Section 2.1) will terminate immediately.
+
+4. Disclaimer of Warranty.
+
+THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF 
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE. 
+
+5. Limitation of Liability.
+
+EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

README.md

@@ -0,0 +1,100 @@
+# Learning Physically Simulated Tennis Skills from Broadcast Videos
+
+<strong>Haotian zhang</strong>, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, Kayvon Fatahalian
+
+SIGGRAPH 2023 (best paper honorable mention) 
+
+[Paper](https://research.nvidia.com/labs/toronto-ai/vid2player3d/data/tennis_skills_main.pdf) |
+[Project](https://research.nvidia.com/labs/toronto-ai/vid2player3d/) |
+[Video](https://youtu.be/ZZVKrNs7_mk) 
+
+<img src="doc/teaser.png"/>
+
+# The current release provides the implementation of the controller, including the low-level imitation policy, motion embedding and the high-level planning policy, as well as the environment setup in IsaacGym. Unfortunately, the demo can NOT run because the trained models are currently not available due to legal issues. 
+
+# Environment setup
+
+### 1. Download IsaacGym and create python virtual env
+You can download IsaacGym Preview Release 4 from the official [site](https://developer.nvidia.com/isaac-gym).
+Then download Miniconda3 from [here](https://repo.anaconda.com/miniconda/Miniconda3-py37_23.1.0-1-Linux-x86_64.sh).
+Create a conda virtual env named `rlgpu` by running `create_conda_env_rlgpu.sh` from IsaacGym, either python3.7 or python3.8 works.
+Note you might need to run the following command or add it to your `.bashrc` if you encounter the error `ImportError: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory` when running IsaacGym.
+```
+export LD_LIBRARY_PATH=<YOUR CONDA PATH>envs/rlgpu/lib/
+``` 
+
+### 2. Install dependencies 
+Enter the created virtual env and run the install script.
+```
+conda activate rlgpu
+bash install.sh
+```
+
+### 3. Install [smpl_visualizer](https://github.com/Haotianz94/smpl_visualizer) for visualizing results
+Git clone and then run 
+```
+bash install.sh
+```
+
+### 4. Download data/checkpoints
+
+Download [data](https://drive.google.com/drive/folders/1kkM9tl1T3dXZbvh5oYHSerL0JkgaL1Mi?usp=sharing) into `vid2player3d/data`.
+
+Download checkpoints of motion VAE and trained polices into `vid2player3d/results` (currently unavailable).
+
+Download SMPL [models](https://download.is.tue.mpg.de/download.php?domain=smpl&sfile=SMPL_python_v.1.0.0.zip) (male and female models) into `smpl_visualizer/data/smpl` after registering and rename the files as `SMPL_MALE.pkl` and `SMPL_FEMALE.pkl`.
+
+# Test with trained policies
+### Single player
+In the single player setting, the player will react to consecutive incoming tennis balls from the other side.
+The script below runs the simulation and renders the result online. The simulation will be reset after 300 frames. You can change the player by chaning`--cfg` to `djokovic` or `nadal`. 
+```
+python vid2player/run.py --cfg federer --rl_device cuda:0 --test --num_envs 1 --episode_length 300 --seed 0 --checkpoint latest --enable_shadow
+```
+
+The script below will run the simulations in batch and render the result videos offline and saved into `out/video`. You can also change `--record` to `--record_scenepic`, which will save the result into an interactive html file under `out/html`. Note that the saved html file is large and may take seconds to load.
+```
+python vid2player/run.py --cfg federer --rl_device cuda:0 --test --num_envs 8192 --episode_length 300 --seed 0 --checkpoint latest --select_best --enable_shadow --num_rec_frames 300 --num_eg 5 --record --headless
+```
+
+### Dual player
+In the dual player setting, the two players will play tennis rally against each other.
+The script below runs the simulation and renders the result online. The simulation will be reset if the ball is missed or out. You can change the players by changing `--cfg` to `nadal_djokovic`. More player settings will be added soon. 
+```
+python vid2player/run.py --cfg federer_djokovic --rl_device cuda:0 --test --num_envs 2 --episode_length 10000 --seed 0 --checkpoint latest --enable_shadow
+```
+
+The script below will run the simulations in batch and render the result videos offline and saved into `out/video`.
+```
+python vid2player/run.py --cfg federer_djokovic --rl_device cuda:0 --test --num_envs 8192 --episode_length 10000 --seed 0 --checkpoint latest --enable_shadow --headless --num_rec_frames 600 --num_eg 5 --record
+```
+
+
+# Retrain the high-level policy
+We also provide code for training the high-level policy using Federer's motion VAE. As described in the paper, we design a curriculum trained in three stages. You can run the following script to execute the curriculum training.
+The script assumes a GPU with memory >= 24GB and it takes about 2 days to finish on a single A5000.
+```
+python vid2player/run.py --cfg federer_train_stage_1 --rl_device cuda:0 --headless
+python vid2player/run.py --cfg federer_train_stage_2 --rl_device cuda:0 --headless
+python vid2player/run.py --cfg federer_train_stage_3 --rl_device cuda:0 --headless
+```
+
+
+# Citation
+```
+@article{
+  zhang2023vid2player3d,
+  author = {Zhang, Haotian and Yuan, Ye and Makoviychuk, Viktor and Guo, Yunrong and Fidler, Sanja and Peng, Xue Bin and Fatahalian, Kayvon},
+  title = {Learning Physically Simulated Tennis Skills from Broadcast Videos},
+  journal = {ACM Trans. Graph.},
+  issue_date = {August 2023},
+  numpages = {14},
+  doi = {10.1145/3592408},
+  publisher = {ACM},
+  address = {New York, NY, USA},
+  keywords = {physics-based character animation, imitation learning, reinforcement learning},
+}
+```
+
+# Contact
+For any question regarding this project, please contact Haotian Zhang via [email protected].

doc/teaser.png

@@ -0,0 +1,9 @@
+# Addtional python libs
+python -m pip install rl-games==1.1.4 pyvista==0.34.2 pyglet==1.5.27 tqdm
+
+# The default install may install `vtk>=9.2.0` which will lead to program crash when running pyvista. 
+# A temporary fix is to install vtk in an older version
+python -m pip install --ignore-installed vtk==9.1.0
+
+# Downgrade numpy
+conda install numpy=1.23.5 --force -c conda-forge