|
| 1 | +[](https://raw.githubusercontent.com/NVIDIA/FastPhotoStyle/master/LICENSE.md) |
| 2 | + |
| 3 | + |
| 4 | +## FastPhotoStyle Tutorial |
| 5 | + |
| 6 | +In this short tutorial, we will guide you through setting up the system environment for running the FastPhotoStyle software and then show several usage examples. |
| 7 | + |
| 8 | +### Background |
| 9 | + |
| 10 | +Existing style transfer algorithms can be divided into categories: artistic style transfer and photorealistic style transfer. |
| 11 | +For artistic style transfer, the goal is to transfer the style of a reference painting to a photo so that the stylized photo looks like a painting and carries the style of the reference painting. |
| 12 | +For photorealistic style transfer, the goal is to transfer the style of a reference photo to a photo so that the stylized photo preserves the content of the original photo but carries the style of the reference photo. |
| 13 | +The FastPhotoStyle algorithm is in the category of photorealistic style transfer. |
| 14 | + |
| 15 | +### Algorithm |
| 16 | + |
| 17 | +FastPhotoStyle takes two images as input where one is the content image and the other is the style image. Its goal is to transfer the style of the style photo to the content photo for creating a stylized image as shown below. |
| 18 | + |
| 19 | +<img src="https://raw.githubusercontent.com/NVIDIA/FastPhotoStyle/master/teaser.png" width="800" title="Teaser results"> |
| 20 | + |
| 21 | +FastPhotoStyle divides the photorealistic stylization process into two steps. |
| 22 | + 1. **PhotoWCT:** Generate a stylized image with visible distortions by applying a whitening and coloring transform to the deep features extracted from the content and style images. |
| 23 | + 2. **Photorealistic Smoothing:** Suppress the distortion in the stylized image by applying an image smoothing filter. |
| 24 | + |
| 25 | +The output is a photorealistic image as it were captured by a camera. |
| 26 | + |
| 27 | +### Requirements |
| 28 | + |
| 29 | +- Hardware: PC with NVIDIA Titan GPU. |
| 30 | +- Software: *Ubuntu 16.04*, *CUDA 9.1*, *Anaconda3*, *pytorch 0.4.0* |
| 31 | +- Environment variables. |
| 32 | + - export ANACONDA=PATH-TO-YOUR-ANACONDA-LIBRARY |
| 33 | + - export CUDA_PATH=/usr/local/cuda |
| 34 | + - export PATH=${ANACONDA}/bin:${CUDA_PATH}/bin:$PATH |
| 35 | + - export LD_LIBRARY_PATH=${ANACONDA}/lib:${CUDA_PATH}/bin64:$LD_LIBRARY_PATH |
| 36 | + - export C_INCLUDE_PATH=${CUDA_PATH}/include |
| 37 | +- System package |
| 38 | + - `sudo apt-get install -y axel imagemagick` (Only used for demo) |
| 39 | +- Python package |
| 40 | + - `conda install pytorch=0.4.0 torchvision cuda91 -y -c pytorch` |
| 41 | + - `pip install scikit-umfpack` |
| 42 | + - `pip install -U setuptools` |
| 43 | + - `pip install cupy` |
| 44 | + - `pip install pynvrtc` |
| 45 | + - `conda install -c menpo opencv3` (OpenCV is only required if you want to use the approximate version of the photo smoothing step.) |
| 46 | + |
| 47 | +### Examples |
| 48 | + |
| 49 | +In the following, we will provide 3 usage examples. |
| 50 | +In the 1st example, we will run the FastPhotoStyle code without using |
| 51 | +segmentation mask. |
| 52 | +In the 2nd example, we will show how to use a labeling tool to create the segmentation masks and use them for stylization. |
| 53 | +In the 3rd example, we will show how to use a pretrained segmetnation network to automatically generate the segmetnation masks and use them for stylization. |
| 54 | + |
| 55 | +#### Example 1: Transfer style of a style photo to a content photo without using segmentation masks. |
| 56 | + |
| 57 | +You can simply type `./demo_example1.sh` to run the demo or follow the steps below. |
| 58 | +- Create image and output folders and make sure nothing is inside the folders. `mkdir images && mkdir results` |
| 59 | +- Go to the image folder: `cd images` |
| 60 | +- Download content image 1: `axel -n 1 http://freebigpictures.com/wp-content/uploads/shady-forest.jpg --output=content1.png` |
| 61 | +- Download style image 1: `axel -n 1 https://vignette.wikia.nocookie.net/strangerthings8338/images/e/e0/Wiki-background.jpeg/revision/latest?cb=20170522192233 --output=style1.png` |
| 62 | +- These images are huge. We need to resize them first. Run |
| 63 | + - `convert -resize 25% content1.png content1.png` |
| 64 | + - `convert -resize 50% style1.png style1.png` |
| 65 | +- Go back to the root folder: `cd ..` |
| 66 | +- Test the photorealistic image stylization code `python demo.py --output_image_path results/example1.png` |
| 67 | +- You should see output messages like |
| 68 | +- ``` |
| 69 | + Resize image: (803,538)->(803,538) |
| 70 | + Resize image: (960,540)->(960,540) |
| 71 | + Elapsed time in stylization: 0.398996 |
| 72 | + Elapsed time in propagation: 13.456573 |
| 73 | + Elapsed time in post processing: 0.202319 |
| 74 | + ``` |
| 75 | +- You should see an output image like |
| 76 | + |
| 77 | +| Input Style Photo | Input Content Photo | Output Stylization Result | |
| 78 | +|-------------------|---------------------|---------------------------| |
| 79 | +|<img src="https://vignette.wikia.nocookie.net/strangerthings8338/images/e/e0/Wiki-background.jpeg" height="300" title="content 1"> | <img src="http://freebigpictures.com/wp-content/uploads/shady-forest.jpg" height="300" title="content 1"> |<img src="https://raw.githubusercontent.com/NVIDIA/FastPhotoStyle/master/demo_result_example1.png" height="300" title="demo_result_example1.png"> | |
| 80 | + |
| 81 | +- As shown in the output messages, the computational bottleneck of FastPhotoStyle is the propagation step (the photorealistic smoothing step). We find that we can make this step much faster by using the guided image filtering algorithm as an approximate. To run the fast version of the demo, you can simply type `./demo_example1_fast.sh` or run. |
| 82 | +- `python demo.py --fast --output_image_path results/example1_fast.png` |
| 83 | +- You should see output messages like |
| 84 | +- ``` |
| 85 | + Resize image: (803,538)->(803,538) |
| 86 | + Resize image: (960,540)->(960,540) |
| 87 | + Elapsed time in stylization: 0.342203 |
| 88 | + Elapsed time in propagation: 0.039506 |
| 89 | + Elapsed time in post processing: 0.203081 |
| 90 | + ``` |
| 91 | +- Check out the stylization result computed by the fast approximation step in `results/example1_fast.png`. It should look very similar to `results/example1.png` from the full algorithm. |
| 92 | + |
| 93 | +#### Example 2: Transfer style of a style photo to a content photo with manually generated semantic label maps. |
| 94 | + |
| 95 | +When segmentation masks of content and style photos are available, FastPhotoStyle can utilize content–style |
| 96 | +correspondences obtained by matching the semantic labels in the segmentation masks for generating better stylization effects. |
| 97 | +In this example, we should how to manually create segmentation masks of content and style photos and use them for photorealistic style transfer. |
| 98 | + |
| 99 | +##### Prepare label maps |
| 100 | + |
| 101 | +- Install the tool [labelme](https://github.com/wkentaro/labelme) and run the following command to start it: `labelme` |
| 102 | +- Start labeling regions (drawing polygons) in the content and style image. The corresponding regions (e.g., sky-to-sky) should have the same label. |
| 103 | +- The labeling result is saved in a ".json" file. By running the following command, you will get the `label.png` under `path/example_json`, which is the label map used in our code. `label.png` is a 1-channel image (usually looks totally black) consists of consecutive labels starting from 0. |
| 104 | + |
| 105 | +``` |
| 106 | +labelme_json_to_dataset example.json -o path/example_json |
| 107 | +``` |
| 108 | + |
| 109 | +##### Stylize with label maps |
| 110 | + |
| 111 | +``` |
| 112 | +python demo.py \ |
| 113 | + --content_image_path PATH-TO-YOUR-CONTENT-IMAGE \ |
| 114 | + --content_seg_path PATH-TO-YOUR-CONTENT-LABEL \ |
| 115 | + --style_image_path PATH-TO-YOUR-STYLE-IMAGE \ |
| 116 | + --style_seg_path PATH-TO-YOUR-STYLE-LABEL \ |
| 117 | + --output_image_path PATH-TO-YOUR-OUTPUT |
| 118 | +``` |
| 119 | + |
| 120 | +Below is a 3-label transferring example (images and labels are from the [DPST](https://github.com/luanfujun/deep-photo-styletransfer) work by Luan et al.): |
| 121 | + |
| 122 | + |
| 123 | + |
| 124 | +#### Example 3: Transfer the style of a style photo to a content photo with automatically generated semantic label maps. |
| 125 | + |
| 126 | +In this example, we will show how to use segmentation masks of content and style photos generated by a pretrained segmentation network to achieve better stylization results. |
| 127 | +We will use the segmentation network provided from [CSAILVision/semantic-segmentation-pytorch](https://github.com/CSAILVision/semantic-segmentation-pytorch) in this example. |
| 128 | +To setup up the segmentation network, do the following steps: |
| 129 | +- Clone the CSAIL segmentation network from this fork of [CSAILVision/semantic-segmentation-pytorch](https://github.com/CSAILVision/semantic-segmentation-pytorch) using the following command |
| 130 | + `git clone https://github.com/mingyuliutw/semantic-segmentation-pytorch segmentation` |
| 131 | +- Run the demo code in [CSAILVision/semantic-segmentation-pytorch](https://github.com/CSAILVision/semantic-segmentation-pytorch) to download the network and make sure the environment is set up properly. |
| 132 | + - `cd segmentation` |
| 133 | + - `./demo_test.sh` |
| 134 | + - You should see output messages like |
| 135 | + ``` |
| 136 | + 2018-XX-XX XX:XX:XX-- http://sceneparsing.csail.mit.edu//data/ADEChallengeData2016/images/validation/ADE_val_00001519.jpg |
| 137 | + Resolving sceneparsing.csail.mit.edu (sceneparsing.csail.mit.edu)... 128.30.100.255 |
| 138 | + Connecting to sceneparsing.csail.mit.edu (sceneparsing.csail.mit.edu)|128.30.100.255|:80... connected. |
| 139 | + HTTP request sent, awaiting response... 200 OK |
| 140 | + Length: 62271 (61K) [image/jpeg] |
| 141 | + Saving to: ‘./ADE_val_00001519.jpg’ |
| 142 | + |
| 143 | + ADE_val_00001519.jpg 100%[=====================================>] 60.81K 366KB/s in 0.2s |
| 144 | + |
| 145 | + 2018-07-25 16:55:00 (366 KB/s) - ‘./ADE_val_00001519.jpg’ saved [62271/62271] |
| 146 | + |
| 147 | + Namespace(arch_decoder='ppm_bilinear_deepsup', arch_encoder='resnet50_dilated8', batch_size=1, fc_dim=2048, gpu_id=0, imgMaxSize=1000, imgSize=[300, 400, 500, 600], model_path='baseline-resnet50_dilated8-ppm_bilinear_deepsup', num_class=150, num_val=-1, padding_constant=8, result='./', segm_downsampling_rate=8, suffix='_epoch_20.pth', test_img='ADE_val_00001519.jpg') |
| 148 | + Loading weights for net_encoder |
| 149 | + Loading weights for net_decoder |
| 150 | + Inference done! |
| 151 | + ``` |
| 152 | + - Go back to the root folder `cd ..` |
| 153 | +
|
| 154 | +- Now, we are ready to use the segmentation network trained on the ADE20K for automatically generating the segmentation mask. |
| 155 | +- To run the fast version of the demo, you can simply type `./demo_example3.sh` or run. |
| 156 | +- Create image and output folders and make sure nothing is inside the folders. `mkdir images && mkdir results` |
| 157 | +- Go to the image folder: `cd images` |
| 158 | +- Download content image 3: `axel -n 1 https://pre00.deviantart.net/f1a6/th/pre/i/2010/019/0/e/country_road_hdr_by_mirre89.jpg --output=content3.png` |
| 159 | +- Download style image 3: `axel -n 1 https://nerdist.com/wp-content/uploads/2017/11/Stranger_Things_S2_news_Images_V03-1024x481.jpg --output=style3.png;` |
| 160 | +- These images are huge. We need to resize them first. Run |
| 161 | + - `convert -resize 50% content3.png content3.png` |
| 162 | + - `convert -resize 50% style3.png style3.png` |
| 163 | +- Go back to the root folder: `cd ..` |
| 164 | +- **Update the python library path by** `export PYTHONPATH=$PYTHONPATH:segmentation` |
| 165 | +- We will now run the demo code that first computing the segmentation masks of content and style images and then performing photorealistic style transfer. |
| 166 | + `python demo_with_ade20k_ssn.py --output_visualization` or `python demo_with_ade20k_ssn.py --fast --output_visualization` |
| 167 | +- You should see output messages like |
| 168 | + ``` |
| 169 | + Loading weights for net_encoder |
| 170 | + Loading weights for net_decoder |
| 171 | + Resize image: (546,366)->(546,366) |
| 172 | + Resize image: (485,273)->(485,273) |
| 173 | + Elapsed time in stylization: 0.890762 |
| 174 | + Elapsed time in propagation: 0.014808 |
| 175 | + Elapsed time in post processing: 0.197138 |
| 176 | + ``` |
| 177 | +- You should see an output image like |
| 178 | +
|
| 179 | +| Input Style Photo | Input Content Photo | Output Stylization Result | |
| 180 | +|-------------------|---------------------|---------------------------| |
| 181 | +|<img src="https://nerdist.com/wp-content/uploads/2017/11/Stranger_Things_S2_news_Images_V03-1024x481.jpg" height="300" title="content 3"> | <img src="https://pre00.deviantart.net/f1a6/th/pre/i/2010/019/0/e/country_road_hdr_by_mirre89.jpg" height="300" title="content 3"> |<img src="https://raw.githubusercontent.com/NVIDIA/FastPhotoStyle/master/demo_result_example1.png" height="300" title="demo_result_example3.png"> | |
| 182 | +
|
| 183 | +- We can check out the segmentation results in the `results` folder. |
| 184 | +
|
| 185 | +| Segmentation of the Style Photo | Segmentation of the Content Photo | |
| 186 | +|---------------------------------|-----------------------------------| |
| 187 | +|<img src="https://raw.githubusercontent.com/NVIDIA/FastPhotoStyle/master/demo_result_style3_seg.pgm.visualization.jpg" height="300" title="demo_result_style3_seg.png"> | <img src="https://raw.githubusercontent.com/NVIDIA/FastPhotoStyle/master/demo_result_content3_seg.pgm.visualization.jpg" height="300" title="demo_result_content3_seg.png"> | |
| 188 | +
|
| 189 | +
|
| 190 | +### Use docker image |
| 191 | +
|
| 192 | +We provide a docker image for testing the code. |
| 193 | +
|
| 194 | + 1. Install docker-ce. Follow the instruction in the [Docker page](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1) |
| 195 | + 2. Install nvidia-docker. Follow the instruction in the [NVIDIA-DOCKER README page](https://github.com/NVIDIA/nvidia-docker). |
| 196 | + 3. Build the docker image `docker build -t your-docker-image:v1.0 .` |
| 197 | + 4. Run an interactive session `docker run -v YOUR_PATH:YOUR_PATH --runtime=nvidia -i -t your-docker-image:v1.0 /bin/bash` |
| 198 | + 5. `cd YOUR_PATH` |
| 199 | + 6. `./demo.sh` |
| 200 | +
|
| 201 | +## Acknowledgement |
| 202 | +
|
| 203 | +- We express gratitudes to the great work [DPST](https://www.cs.cornell.edu/~fujun/files/style-cvpr17/style-cvpr17.pdf) by Luan et al. and their [Torch](https://github.com/luanfujun/deep-photo-styletransfer) and [Tensorflow](https://github.com/LouieYang/deep-photo-styletransfer-tf) implementations. |
0 commit comments