initial

.gitattributes

@@ -0,0 +1,2 @@
+# Auto detect text files and perform LF normalization
+* text=auto

.gitignore

@@ -0,0 +1,102 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+images/continuous_lighting_1/.DS_Store
+images/continuous_lighting_2/.DS_Store
+.DS_Store

LICENSE

@@ -0,0 +1,19 @@
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,80 @@
+# Learning to Shadow Hand-drawn Sketches
+
+Official Implementation of "Learning to Shadow Hand-drawn Sketches" (CVPR 2020).
+
+[Project Site](https://cal.cs.umbc.edu/Papers/Zheng-2020-Shade) | [Paper](https://arxiv.org/abs/2002.11812) | Training Code, Online Demo and Dataset (coming soon)
+
+## Overview
+
+![Image of flowchart](https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/overview.png)
+
+We present a fully automatic method to generate detailed and accurate artistic shadows from pairs of line drawing sketches and lighting directions. We also contribute a new dataset of one thousand examples of pairs of line drawings and shadows that are tagged with lighting directions. Remarkably, the generated shadows quickly communicate the underlying 3D structure of the sketched scene. Consequently, the shadows generated by our approach can be used directly or as an excellent starting point for artists. We demonstrate that the deep learning network we propose takes a hand-drawn sketch, builds a 3D model in latent space, and renders the resulting shadows. The generated shadows respect the hand-drawn lines and underlying 3D space and contain sophisticated and accurate details, such as self-shadowing effects. Moreover, the generated shadows contain artistic effects, such as rim lighting or halos appearing from back lighting, that would be achievable with traditional 3D rendering methods.
+
+## Prerequisites
+
+- python3
+- tensorflow
+- numpy
+- opencv-python
+- argparse
+
+```
+sudo pip3 install -r requirements.txt
+```
+
+## How to Use
+
+**Predict from arbitary lighting direction:**
+```
+python3 main.py --image-size=320 --direction=810
+```
+--direction choice: 001, 002 or xy0, where x={1,2,3,4,5,6,7,8} and y={1,2,3}.
+
+**Predict gif:**
+```
+python3 main_gif.py --image-size=320 --dir='[image_name].png'
+```
+This command will result 80 frames in ./[image_name] folder. Copy makegif.py into ./[image_name] folder. Then:
+```
+cd [image_name]/
+python3 makegif.py
+```
+## Dataset
+
+The dataset will be released in the future. 
+
+## Gallery
+
+<img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/26_710.png" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/2_810.png" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/47_710.png" width="150"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/5_410.png" width="200">
+
+<img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/girl-210.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/girl-810.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/b.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/c.gif" width="200">
+
+<img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/11-top.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/continuous_lighting_1/810-830.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/continuous_lighting_1/front-lighting2.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/continuous_lighting_1/side-lighting2.gif" width="200">
+
+<img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/13-top.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/continuous_lighting_2/810-830-41600.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/continuous_lighting_2/front-lighting-41600.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/continuous_lighting_2/side-lighting-41600.gif" width="200">
+
+<img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/4-top-25800.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/4_front_41600.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/4_side_25800.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/60-top.gif" width="200">
+
+<img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/42-top.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/42-front-lighting-25800.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/42-side-lighting-25800.gif" width="200"><img src="https://github.com/qyzdao/Learn_to_Shade_Sketch/blob/master/images/60_front_25800.gif" width="200">
+
+## License
+
+Models is available under Creative Commons BY-NC 4.0 license. You can use, redistribute the models for **non-commercial purposes**.
+
+## Citation
+
+If you use our work for your research, please cite our paper
+```
+@InProceedings{Zheng2020LSHS,
+title = {Learning to Shadow Hand-drawn Sketches},
+author = {Qingyuan Zheng, Zhuoru Li and Adam W. Bargteil},
+booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
+year = {2020}
+}
+```
+
+## Credits
+
+Sketches were collected from nico-opendata and web. animation sketches (c) Yoshinari Yo.
+
+For training line normalization model, please see [LineNormalizer](https://github.com/hepesu/LineNormalizer).

main.py

@@ -0,0 +1,108 @@
+import tensorflow as tf
+import numpy as np
+import cv2
+import os
+import argparse
+
+parser = argparse.ArgumentParser(description='Shade Sketches')
+parser.add_argument('--image-size', type=int, default=320,
+                    help='input image size (default: 320)')
+parser.add_argument('--direction', type=str, default='810',
+                    help='lighting directions (suggest to choose 810, 210, 710)')
+args = parser.parse_args()
+
+
+def cond_to_pos(cond):
+    cond_pos_rel = {
+        '002': [0, 0, -1],
+        '110': [0, 1, -1], '210': [1, 1, -1], '310': [1, 0, -1], '410': [1, -1, -1], '510': [0, -1, -1],
+        '610': [-1, -1, -1], '710': [-1, 0, -1], '810': [-1, 1, -1],
+        '120': [0, 1, 0], '220': [1, 1, 0], '320': [1, 0, 0], '420': [1, -1, 0], '520': [0, -1, 0], '620': [-1, -1, 0],
+        '720': [-1, 0, 0], '820': [-1, 1, 0],
+        '130': [0, 1, 1], '230': [1, 1, 1], '330': [1, 0, 1], '430': [1, -1, 1], '530': [0, -1, 1], '630': [-1, -1, 1],
+        '730': [-1, 0, 1], '830': [-1, 1, 1],
+        '001': [0, 0, 1]
+    }
+    return cond_pos_rel[cond]
+
+if not os.path.exists('norm/'):
+    os.makedirs('norm/')
+
+if not os.path.exists('out/'):
+    os.makedirs('out/')
+
+# Line norm
+with tf.Graph().as_default():
+    output_graph_def = tf.GraphDef()
+
+    with open("linenorm.pb", "rb") as f:
+        output_graph_def.ParseFromString(f.read())
+        tensors = tf.import_graph_def(output_graph_def, name="")
+
+    with tf.Session() as sess:
+        init = tf.global_variables_initializer()
+        sess.run(init)
+
+        op = sess.graph.get_operations()
+
+        for i, m in enumerate(op):
+            print('op{}:'.format(i), m.values())
+
+        inputs = sess.graph.get_tensor_by_name('input_1:0')
+        outputs = sess.graph.get_tensor_by_name('conv2d_9/Sigmoid:0')
+        s = args.image_size
+
+        for root, dirs, files in os.walk('val/', topdown=False):
+            for name in files:
+                line_path = os.path.join(root, name)
+                print(line_path)
+
+                img = cv2.imread(line_path, cv2.IMREAD_GRAYSCALE)
+                img = cv2.resize(img, (s, s))
+                img = img.astype(np.float32) / 255.
+
+                img_out = sess.run(outputs, {inputs: np.reshape(img, (1, img.shape[0], img.shape[1], 1))})
+                cv2.imwrite(os.path.join('norm/', name), np.squeeze(img_out) * 255.)
+
+
+# Line shade
+with tf.Graph().as_default():
+    output_graph_def = tf.GraphDef()
+
+    with open("lineshader.pb", "rb") as f:
+        output_graph_def.ParseFromString(f.read())
+        tensors = tf.import_graph_def(output_graph_def, name="")
+
+    with tf.Session() as sess:
+        init = tf.global_variables_initializer()
+        sess.run(init)
+
+        inputs1 = sess.graph.get_tensor_by_name('input_1:0')
+        inputs2 = sess.graph.get_tensor_by_name('input_2:0')
+        outputs = sess.graph.get_tensor_by_name('conv2d_139/Tanh:0')
+        s = args.image_size
+
+        for root, dirs, files in os.walk('norm/', topdown=False):
+            for name in files:
+                norm_path = os.path.join(root, name)
+                print(norm_path)
+
+                img = cv2.imread(norm_path, cv2.IMREAD_GRAYSCALE)
+                img = 1 - img.astype(np.float32) / 255. #inverse black-in-white lines to white-in-black
+
+                cond = cond_to_pos(args.direction) # lighting direction
+
+                img_out = sess.run(
+                    outputs, {
+                        inputs1: np.expand_dims(cond, 0),
+                        inputs2: np.reshape(img, (1, s, s, 1)),
+                    }
+                )
+
+                line = cv2.imread(os.path.join('val/', name), cv2.IMREAD_GRAYSCALE)
+                line = cv2.resize(line, (s, s))
+
+                shade = (1 - (np.squeeze(img_out) + 1) / 2) * 255. # inverse white-in-black shadow to black-in-white
+                final_output = 0.8 * line + 0.2 * shade # composite line drawing and shadow
+                cv2.imwrite(os.path.join('out/', name), final_output)
+