Skip to content

Commit e732321

Browse files
wenjwenj
committed
first commit
1 parent 99068a9 commit e732321

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+9063
-0
lines changed

‎.gitignore

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
*.pyc
2+
__pycache__/
3+
log
4+
log/
5+
data
6+
data/
7+
8+
submit_sbatch.sh
9+
sbatch_gpu.sh

‎README.md

+145
Original file line numberDiff line numberDiff line change
@@ -1 +1,146 @@
11
# GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh
2+
3+
CVPR 2024
4+
5+
[Paper]() | [Project Page](https://wenj.github.io/GoMAvatar/)
6+
7+
```bibtex
8+
@inproceedings{wen2024gomavatar,
9+
title={{GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh}},
10+
author={Jing Wen and Xiaoming Zhao and Zhongzheng Ren and Alex Schwing and Shenlong Wang},
11+
booktitle={CVPR},
12+
year={2024}
13+
}
14+
```
15+
16+
## Requirements
17+
18+
Our codes are tested in
19+
* CUDA 11.6
20+
* PyTorch 1.13.0
21+
* PyTorch3D 0.7.0
22+
23+
Install the required packages:
24+
```Shell
25+
conda create -n GoMAvatar
26+
conda activate GoMAvatar
27+
28+
conda install pytorch==1.13.0 torchvision==0.14.0 pytorch-cuda=11.6 -c pytorch -c nvidia
29+
pip install -r requirements.txt
30+
31+
# install pytorch3d
32+
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
33+
conda install pytorch3d -c pytorch3d
34+
35+
# install gaussian splatting
36+
pip install git+"https://github.com/graphdeco-inria/diff-gaussian-rasterization"
37+
```
38+
39+
## Data preparation
40+
### Prerequisites
41+
Download SMPL v1.0.0 models from [here](https://smpl.is.tue.mpg.de/download.php) and put the `.pkl` files under `utils/smpl/models`.
42+
You may need to remove the Chumpy objects following [here](https://github.com/vchoutas/smplx/tree/main/tools).
43+
44+
### ZJU-MoCap
45+
46+
First download the [ZJU-MoCap](https://github.com/zju3dv/neuralbody/blob/master/INSTALL.md#zju-mocap-dataset) dataset and save the raw data under `data/zju-mocap`.
47+
48+
Run the following script to preprocess the dataset:
49+
```Shell
50+
cd scripts/prepare_zju-mocap
51+
python prepare_dataset.py --cfg "$SCENE".yaml
52+
```
53+
Change `$SCENE` to one of 377, 386, 387, 392, 393, 394.
54+
55+
The folder will be in the following structure:
56+
```Shell
57+
├── data
58+
├── zju-mocap
59+
├── 377
60+
├── 386
61+
├── ...
62+
├── CoreView_377
63+
├── CoreView_386
64+
├── ...
65+
```
66+
Folders named after scene ID only are preprocessed training data, while those prefixed with `CoreView_` are raw data.
67+
68+
### PeopleSnapshot
69+
70+
Download the [PeopleSnapshot](https://graphics.tu-bs.de/people-snapshot) dataset and save the files under `data/snapshot`.
71+
72+
Download the refined training poses from [here](https://github.com/tijiang13/InstantAvatar/tree/master/data/PeopleSnapshot).
73+
74+
Run the following script to preprocess the training and test set.
75+
```Shell
76+
cd scripts/prepare_snapshot
77+
python prepare_dataset.py --cfg "$SCENE".yaml # training set
78+
python prepare_dataset.py --cfg "$SCENE"_test.yaml # test set
79+
```
80+
`$SCENE` is one of `female-3-casual`, `female-4-casual`, `male-3-casual` and `male-4-casual`.
81+
82+
After the preprocessing, the folder will be in the following structure:
83+
```Shell
84+
├── data
85+
├── snapshot
86+
├── f3c_train
87+
├── f3c_test
88+
├── f4c_train
89+
├── f4c_test
90+
├── ...
91+
├── female-3-casual
92+
├── female-4-casual
93+
├── ...
94+
├── poses # refined training poses
95+
├── female-3-casual
96+
├── poses
97+
├── anim_nerf_test.npz
98+
├── anim_nerf_train.npz
99+
├── anim_nerf_val.npz
100+
├── ...
101+
102+
```
103+
Folders ended with `_train` or `_test` are preprocessed data.
104+
105+
## Rendering and evaluation
106+
107+
We provide the pretrained checkpoints in this [link](https://uofi.box.com/s/onwfp29ej03sr2ci7mm59nu74v6i0ip3). To reproduce the rendering results in the paper, run
108+
```Shell
109+
# ZJU-MoCap novel view synthesis
110+
python eval.py --cfg exps/zju-mocap_"$SCENE".yaml --type view
111+
112+
# ZJU-MoCap novel pose synthesis
113+
python eval.py --cfg exps/zju-mocap_"$SCENE".yaml --type pose
114+
```
115+
On the PeopleSnapshot dataset, we follow [Anim-NeRF](https://github.com/JanaldoChen/Anim-NeRF) and [InstantAvatar](https://github.com/tijiang13/InstantAvatar) to refine test poses:
116+
```Shell
117+
python train_pose.py --cfg exps/snapshot_"$SCENE".yaml
118+
```
119+
Please check `exps/` for detailed configuration files.
120+
121+
You can run 360 degree freeview rendering using the following command
122+
```Shell
123+
python eval.py --cfg exps/zju-mocap_"$SCENE".yaml --type freeview
124+
```
125+
Use `--frame_idx` to specify the training frame id and `--n_frames` to set the number of views.
126+
127+
Or you can render novel poses from [MDM](https://guytevet.github.io/mdm-page/):
128+
```Shell
129+
python eval.py --cfg exps/zju-mocap_"$SCENE".yaml --type pose_mdm --pose_path data/mdm_poses/sample.npy
130+
```
131+
We provide an example of pose trajectory in `data/pose_mdm/sample.npy`.
132+
133+
## Training
134+
135+
Run the following command to train from scratch:
136+
```Shell
137+
# ZJU-MoCap
138+
python train.py --cfg exps/zju-mocap_"$SCENE".yaml
139+
140+
# PeopleSnapshot
141+
python train.py --cfg exps/snapshot_"$SCENE".yaml
142+
```
143+
144+
## Acknowledgements
145+
146+
This project builds upon [HumanNeRF](https://github.com/chungyiweng/humannerf) and [MonoHuman](https://github.com/Yzmblog/MonoHuman/tree/main). We appreciate the authors for their great work!

‎configs/__init__.py

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
import os
2+
import argparse
3+
4+
import torch
5+
6+
from .yacs import CfgNode as CN
7+
8+
9+
# pylint: disable=redefined-outer-name
10+
11+
12+
def make_cfg(cfg_filename):
13+
cfg = CN()
14+
cfg.merge_from_file('configs/default.yaml')
15+
if cfg_filename is not None:
16+
cfg.merge_from_file(cfg_filename)
17+
18+
log_root = 'log' if not hasattr(cfg, 'save_dir') else cfg.save_dir
19+
cfg.save_dir = os.path.join(log_root, cfg.exp_name)
20+
21+
return cfg

‎configs/default.yaml

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
exp_name: "default"
2+
3+
bgcolor: [255., 255., 255.]
4+
random_bgcolor: true
5+
img_size: [512, 512]
6+
7+
dataset:
8+
train:
9+
dataset_path: "dataset/zju_mocap/387"
10+
batch_size: 1
11+
num_workers: 1
12+
13+
skip: 1
14+
maxframes: -1
15+
split_for_pose: false # if true, leave 1/5 frames for novel pose evaluation (monohuman's split)
16+
17+
crop_size: [-1, -1]
18+
prefetch: false
19+
20+
test_view:
21+
name: "zju-mocap"
22+
dataset_path: "dataset/zju_mocap/387"
23+
raw_dataset_path: "dataset/zju_mocap/CoreView_387"
24+
batch_size: 1
25+
num_workers: 1
26+
27+
skip: 1
28+
maxframes: -1
29+
30+
exclude_view: 0
31+
32+
test_pose:
33+
dataset_path: "dataset/zju_mocap/387"
34+
raw_dataset_path: "dataset/zju_mocap/CoreView_387"
35+
batch_size: 1
36+
num_workers: 1
37+
38+
test_pose_mdm:
39+
dataset_path: "dataset/zju_mocap/387"
40+
pose_path: "data/pose_mdm/row00_col00_smpl_params.npy"
41+
batch_size: 1
42+
num_workers: 1
43+
format: 'mdm'
44+
45+
test_on_train:
46+
batch_size: 1
47+
num_workers: 1
48+
49+
test_freeview:
50+
dataset_path: "dataset/zju_mocap/387"
51+
batch_size: 1
52+
num_workers: 1
53+
src_type: 'zju_mocap'
54+
55+
model:
56+
# basic info, override all modules
57+
img_size: [512, 512]
58+
subdivide_iters: []
59+
eval_mode: false
60+
61+
appearance:
62+
face_color: true
63+
color_init: 0.5
64+
65+
canonical_geometry:
66+
radius_scale: 1.0
67+
deform_scale: false
68+
deform_so3: false
69+
70+
sigma: 0.001
71+
72+
lbs_weights:
73+
refine: false
74+
75+
renderer:
76+
name: 'gaussian'
77+
78+
pose_refinement:
79+
name: 'none'
80+
81+
non_rigid:
82+
name: 'none'
83+
84+
normal_renderer:
85+
name: 'none'
86+
87+
shadow_module:
88+
name: 'none'
89+
90+
train:
91+
optim: 'adam'
92+
lr:
93+
lbs_weights: 0.0
94+
appearance: 0.005
95+
canonical_geometry: 0.005
96+
canonical_geometry_xyz: 0.005
97+
non_rigid: 0.005
98+
pose_refinement: 0.0005
99+
shadow: 0.005
100+
101+
losses:
102+
rgb:
103+
coeff: 1.0
104+
105+
mask:
106+
coeff: 5.0
107+
108+
lpips:
109+
coeff: 1.0
110+
111+
laplacian:
112+
coeff_canonical: 0.0
113+
coeff_observation: 0.0
114+
115+
normal:
116+
coeff_consist: 0.0
117+
118+
mask_dilate: false
119+
coeff_mask: 0.0
120+
121+
color_consist:
122+
coeff: 0.0
123+
124+
total_iters: 30000
125+
lr_update_exp: true
126+
lr_decay_steps: 100000
127+
128+
log_freq: 10
129+
tb_freq: 100
130+
save_freq: 1000
131+
eval_freq: 50000

0 commit comments

Comments
 (0)