07_autoencoders_and_gans.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 17 – Autoencoders and GANs**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "❗️ **This will be very slow, unless you have a GPU (Check the instruction in https://github.com/UAPH4582/PH482_582/blob/main/INSTALL.md).**\n",
    "\n",
    "❗️ **If you do not, then you should run this notebook in Colab, using a GPU runtime**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/UAPH4582/PH482_582/blob/main/07_autoencoders_and_gans.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. LSTMs and CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"autoencoders\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple utility functions to plot grayscale 28x28 image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_image(image):\n",
    "    plt.imshow(image, cmap=\"binary\")\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PCA with a linear Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build 3D dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(4)\n",
    "\n",
    "def generate_3d_data(m, w1=0.1, w2=0.3, noise=0.1):\n",
    "    angles = np.random.rand(m) * 3 * np.pi / 2 - 0.5\n",
    "    data = np.empty((m, 3))\n",
    "    data[:, 0] = np.cos(angles) + np.sin(angles)/2 + noise * np.random.randn(m) / 2\n",
    "    data[:, 1] = np.sin(angles) * 0.7 + noise * np.random.randn(m) / 2\n",
    "    data[:, 2] = data[:, 0] * w1 + data[:, 1] * w2 + noise * np.random.randn(m)\n",
    "    return data\n",
    "\n",
    "X_train = generate_3d_data(60)\n",
    "X_train = X_train - X_train.mean(axis=0, keepdims=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's build the Autoencoder..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "encoder = keras.models.Sequential([keras.layers.Dense(2, input_shape=[3])])\n",
    "decoder = keras.models.Sequential([keras.layers.Dense(3, input_shape=[2])])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: \n",
    "#### a) Build an autoencoder model with the two subcomponents: the encoder and the decoder.\n",
    "#### b) Compile the model with `mse` loss and SGD optimizer with lr=1 \n",
    "#### c) Train the model (10 epochs) on a simple generated 3D dataset and use it to encode that same dataset. \n",
    "#### d) Look at the plot and explain in the report the purpose of this encoder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#autoencoder = \n",
    "#autoencoder.compile(...)\n",
    "#history     = \n",
    "#predictions = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(4,3))\n",
    "plt.plot(codings[:,0], codings[:, 1], \"b.\")\n",
    "plt.xlabel(\"$z_1$\", fontsize=18)\n",
    "plt.ylabel(\"$z_2$\", fontsize=18, rotation=0)\n",
    "plt.grid(True)\n",
    "save_fig(\"linear_autoencoder_pca_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stacked Autoencoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use MNIST:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full.astype(np.float32) / 255\n",
    "X_test = X_test.astype(np.float32) / 255\n",
    "X_train, X_valid = X_train_full[:-5000], X_train_full[-5000:]\n",
    "y_train, y_valid = y_train_full[:-5000], y_train_full[-5000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train all layers at once"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a stacked Autoencoder with 3 hidden layers and 1 output layer (i.e., 2 stacked Autoencoders)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rounded_accuracy(y_true, y_pred):\n",
    "    return keras.metrics.binary_accuracy(tf.round(y_true), tf.round(y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "stacked_encoder = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(100, activation=\"selu\"),\n",
    "    keras.layers.Dense(30, activation=\"selu\"),\n",
    "])\n",
    "stacked_decoder = keras.models.Sequential([\n",
    "    keras.layers.Dense(100, activation=\"selu\", input_shape=[30]),\n",
    "    keras.layers.Dense(28 * 28, activation=\"sigmoid\"),\n",
    "    keras.layers.Reshape([28, 28])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: \n",
    "#### a) Build an autoencoder model with the two subcomponents: the encoder and the decoder.\n",
    "#### b) Compile the model using binary cross entropy,  SGD optimizer with lr=1.0 and `rounded_accuracy` as metrics.\n",
    "#### c) Train the model using X_train (10 epochs) as both the inputs and the targets (and similarly, use X_valid as both the validation inputs and targets). \n",
    "#### d) Look at the reconstructed images and comment the results in the report. Could be improved? How? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#stacked_ae = \n",
    "#stacked_ae.compile(...)\n",
    "#history = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function processes a few test images through the autoencoder and displays the original images and their reconstructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_reconstructions(model, images=X_valid, n_images=5):\n",
    "    reconstructions = model.predict(images[:n_images])\n",
    "    fig = plt.figure(figsize=(n_images * 1.5, 3))\n",
    "    for image_index in range(n_images):\n",
    "        plt.subplot(2, n_images, 1 + image_index)\n",
    "        plot_image(images[image_index])\n",
    "        plt.subplot(2, n_images, 1 + n_images + image_index)\n",
    "        plot_image(reconstructions[image_index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_reconstructions(stacked_ae)\n",
    "save_fig(\"reconstruction_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Convolutional Layers Instead of Dense Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a stacked Autoencoder with 3 hidden layers and 1 output layer (i.e., 2 stacked Autoencoders)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "conv_encoder = keras.models.Sequential([\n",
    "    keras.layers.Reshape([28, 28, 1], input_shape=[28, 28]),\n",
    "    keras.layers.Conv2D(16, kernel_size=3, padding=\"SAME\", activation=\"selu\"),\n",
    "    keras.layers.MaxPool2D(pool_size=2),\n",
    "    keras.layers.Conv2D(32, kernel_size=3, padding=\"SAME\", activation=\"selu\"),\n",
    "    keras.layers.MaxPool2D(pool_size=2),\n",
    "    keras.layers.Conv2D(64, kernel_size=3, padding=\"SAME\", activation=\"selu\"),\n",
    "    keras.layers.MaxPool2D(pool_size=2)\n",
    "])\n",
    "conv_decoder = keras.models.Sequential([\n",
    "    keras.layers.Conv2DTranspose(32, kernel_size=3, strides=2, padding=\"VALID\", activation=\"selu\",\n",
    "                                 input_shape=[3, 3, 64]),\n",
    "    keras.layers.Conv2DTranspose(16, kernel_size=3, strides=2, padding=\"SAME\", activation=\"selu\"),\n",
    "    keras.layers.Conv2DTranspose(1, kernel_size=3, strides=2, padding=\"SAME\", activation=\"sigmoid\"),\n",
    "    keras.layers.Reshape([28, 28])\n",
    "])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_encoder.summary()\n",
    "conv_decoder.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3:\n",
    "#### a) Build an autoencoder model with the two subcomponents: the encoder and the decoder.\n",
    "#### b) Compile the model using binary cross entropy,  SGD optimizer with lr=1.0 and `rounded_accuracy` as metrics.\n",
    "#### c) Train the model using X_train (10 epochs) as both the inputs and the targets (and similarly, use X_valid as both the validation inputs and targets). \n",
    "#### d) Is the CNN autoencoder better than dense layer's autoencoders? Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#conv_ae = \n",
    "#conv_ae.compile(...)\n",
    "#history = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_reconstructions(conv_ae)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stacked denoising Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Gaussian noise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "denoising_encoder = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.GaussianNoise(0.2),\n",
    "    keras.layers.Dense(100, activation=\"selu\"),\n",
    "    keras.layers.Dense(30, activation=\"selu\")\n",
    "])\n",
    "denoising_decoder = keras.models.Sequential([\n",
    "    keras.layers.Dense(100, activation=\"selu\", input_shape=[30]),\n",
    "    keras.layers.Dense(28 * 28, activation=\"sigmoid\"),\n",
    "    keras.layers.Reshape([28, 28])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4:\n",
    "#### a) Build an autoencoder model with the two subcomponents: the encoder and the decoder.\n",
    "#### b) Compile the model using binary cross entropy,  SGD optimizer with lr=1.0 and `rounded_accuracy` as metrics.\n",
    "#### c) Train the model using X_train (10 epochs) as both the inputs and the targets (and similarly, use X_valid as both the validation inputs and targets). \n",
    "#### d) Explain why adding Gaussian noise to the input would help the autoencoder to learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#denoising_ae = \n",
    "#denoising_ae.compile(...)\n",
    "#history = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "noise = keras.layers.GaussianNoise(0.2)\n",
    "show_reconstructions(denoising_ae, noise(X_valid, training=True))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variational Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5:\n",
    "#### a) Why are the variational autoencoders different from the previous autoencoders?\n",
    "#### b) Explain in the report how the variational autoencoder is implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "kl_divergence = keras.losses.kullback_leibler_divergence\n",
    "\n",
    "class KLDivergenceRegularizer(keras.regularizers.Regularizer):\n",
    "    def __init__(self, weight, target=0.1):\n",
    "        self.weight = weight\n",
    "        self.target = target\n",
    "    def __call__(self, inputs):\n",
    "        mean_activities = K.mean(inputs, axis=0)\n",
    "        return self.weight * (\n",
    "            kl_divergence(self.target, mean_activities) +\n",
    "            kl_divergence(1. - self.target, 1. - mean_activities))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Sampling(keras.layers.Layer):\n",
    "    def call(self, inputs):\n",
    "        mean, log_var = inputs\n",
    "        return K.random_normal(tf.shape(log_var)) * K.exp(log_var / 2) + mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "codings_size = 10\n",
    "\n",
    "inputs = keras.layers.Input(shape=[28, 28])\n",
    "z = keras.layers.Flatten()(inputs)\n",
    "z = keras.layers.Dense(150, activation=\"selu\")(z)\n",
    "z = keras.layers.Dense(100, activation=\"selu\")(z)\n",
    "codings_mean = keras.layers.Dense(codings_size)(z)\n",
    "codings_log_var = keras.layers.Dense(codings_size)(z)\n",
    "codings = Sampling()([codings_mean, codings_log_var])\n",
    "variational_encoder = keras.models.Model(\n",
    "    inputs=[inputs], outputs=[codings_mean, codings_log_var, codings])\n",
    "\n",
    "decoder_inputs = keras.layers.Input(shape=[codings_size])\n",
    "x = keras.layers.Dense(100, activation=\"selu\")(decoder_inputs)\n",
    "x = keras.layers.Dense(150, activation=\"selu\")(x)\n",
    "x = keras.layers.Dense(28 * 28, activation=\"sigmoid\")(x)\n",
    "outputs = keras.layers.Reshape([28, 28])(x)\n",
    "variational_decoder = keras.models.Model(inputs=[decoder_inputs], outputs=[outputs])\n",
    "\n",
    "_, _, codings = variational_encoder(inputs)\n",
    "reconstructions = variational_decoder(codings)\n",
    "variational_ae = keras.models.Model(inputs=[inputs], outputs=[reconstructions])\n",
    "\n",
    "latent_loss = -0.5 * K.sum(\n",
    "    1 + codings_log_var - K.exp(codings_log_var) - K.square(codings_mean),\n",
    "    axis=-1)\n",
    "variational_ae.add_loss(K.mean(latent_loss) / 784.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "variational_ae.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\", metrics=[rounded_accuracy])\n",
    "history = variational_ae.fit(X_train, X_train, epochs=25, \n",
    "                             validation_data=(X_valid, X_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "show_reconstructions(variational_ae)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Fashion Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_multiple_images(images, n_cols=None):\n",
    "    n_cols = n_cols or len(images)\n",
    "    n_rows = (len(images) - 1) // n_cols + 1\n",
    "    if images.shape[-1] == 1:\n",
    "        images = np.squeeze(images, axis=-1)\n",
    "    plt.figure(figsize=(n_cols, n_rows))\n",
    "    for index, image in enumerate(images):\n",
    "        plt.subplot(n_rows, n_cols, index + 1)\n",
    "        plt.imshow(image, cmap=\"binary\")\n",
    "        plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's generate a few random codings, decode them and plot the resulting images:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6:\n",
    "#### Generate few random codings (https://www.tensorflow.org/api_docs/python/tf/random/normal), decode them and plot the resulting images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "#codings = \n",
    "#images  = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_multiple_images(images, 4)\n",
    "save_fig(\"vae_generated_images_plot\", tight_layout=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's perform semantic interpolation between these images:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative Adversarial Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7:\n",
    "#### Look at the variational autoencoder implementation and explain in the report:\n",
    "#### a) What is the purpose of the generator and discriminator neural networks? \n",
    "#### b) Why are these two NN used together to build the GANs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "codings_size = 30\n",
    "\n",
    "generator = keras.models.Sequential([\n",
    "    keras.layers.Dense(100, activation=\"selu\", input_shape=[codings_size]),\n",
    "    keras.layers.Dense(150, activation=\"selu\"),\n",
    "    keras.layers.Dense(28 * 28, activation=\"sigmoid\"),\n",
    "    keras.layers.Reshape([28, 28])\n",
    "])\n",
    "discriminator = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(150, activation=\"selu\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\"),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gan = keras.models.Sequential([generator, discriminator])\n",
    "discriminator.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\")\n",
    "discriminator.trainable = False\n",
    "gan.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "dataset = tf.data.Dataset.from_tensor_slices(X_train).shuffle(1000)\n",
    "dataset = dataset.batch(batch_size, drop_remainder=True).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_gan(gan, dataset, batch_size, codings_size, n_epochs=50):\n",
    "    generator, discriminator = gan.layers\n",
    "    for epoch in range(n_epochs):\n",
    "        print(\"Epoch {}/{}\".format(epoch + 1, n_epochs))              # not shown in the book\n",
    "        for X_batch in dataset:\n",
    "            # phase 1 - training the discriminator\n",
    "            noise = tf.random.normal(shape=[batch_size, codings_size])\n",
    "            generated_images = generator(noise)\n",
    "            X_fake_and_real = tf.concat([generated_images, X_batch], axis=0)\n",
    "            y1 = tf.constant([[0.]] * batch_size + [[1.]] * batch_size)\n",
    "            discriminator.trainable = True\n",
    "            discriminator.train_on_batch(X_fake_and_real, y1)\n",
    "            # phase 2 - training the generator\n",
    "            noise = tf.random.normal(shape=[batch_size, codings_size])\n",
    "            y2 = tf.constant([[1.]] * batch_size)\n",
    "            discriminator.trainable = False\n",
    "            gan.train_on_batch(noise, y2)\n",
    "        plot_multiple_images(generated_images, 8)                     # not shown\n",
    "        plt.show()                                                    # not shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_gan(gan, dataset, batch_size, codings_size, n_epochs=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "noise = tf.random.normal(shape=[batch_size, codings_size])\n",
    "generated_images = generator(noise)\n",
    "plot_multiple_images(generated_images, 8)\n",
    "save_fig(\"gan_generated_images_plot\", tight_layout=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_gan(gan, dataset, batch_size, codings_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Convolutional GAN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8:\n",
    "#### Compare the results from Deep Convolutional GAN with the ones from Task 9. Comment in the report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "codings_size = 100\n",
    "\n",
    "generator = keras.models.Sequential([\n",
    "    keras.layers.Dense(7 * 7 * 128, input_shape=[codings_size]),\n",
    "    keras.layers.Reshape([7, 7, 128]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Conv2DTranspose(64, kernel_size=5, strides=2, padding=\"SAME\",\n",
    "                                 activation=\"selu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Conv2DTranspose(1, kernel_size=5, strides=2, padding=\"SAME\",\n",
    "                                 activation=\"tanh\"),\n",
    "])\n",
    "discriminator = keras.models.Sequential([\n",
    "    keras.layers.Conv2D(64, kernel_size=5, strides=2, padding=\"SAME\",\n",
    "                        activation=keras.layers.LeakyReLU(0.2),\n",
    "                        input_shape=[28, 28, 1]),\n",
    "    keras.layers.Dropout(0.4),\n",
    "    keras.layers.Conv2D(128, kernel_size=5, strides=2, padding=\"SAME\",\n",
    "                        activation=keras.layers.LeakyReLU(0.2)),\n",
    "    keras.layers.Dropout(0.4),\n",
    "    keras.layers.Flatten(),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "gan = keras.models.Sequential([generator, discriminator])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "discriminator.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\")\n",
    "discriminator.trainable = False\n",
    "gan.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_dcgan = X_train.reshape(-1, 28, 28, 1) * 2. - 1. # reshape and rescale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "dataset = tf.data.Dataset.from_tensor_slices(X_train_dcgan)\n",
    "dataset = dataset.shuffle(1000)\n",
    "dataset = dataset.batch(batch_size, drop_remainder=True).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_gan(gan, dataset, batch_size, codings_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "noise = tf.random.normal(shape=[batch_size, codings_size])\n",
    "generated_images = generator(noise)\n",
    "plot_multiple_images(generated_images, 8)\n",
    "save_fig(\"dcgan_generated_images_plot\", tight_layout=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  },
  "nav_menu": {
   "height": "381px",
   "width": "453px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}