SoftMax and Cross Entropy Loss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8f0fd5b4",
   "metadata": {},
   "source": [
    "## SoftMax and Cross Entropy Loss\n",
    "------\n",
    "Softmax and cross-entropy loss are commonly used together in machine learning for multi-class classification tasks. Let's understand each concept individually and then explore how they are related.\n",
    "\n",
    "**Softmax Function:** The softmax function is a mathematical function that takes a vector of real numbers as input and transforms it into a probability distribution. It is typically used in multi-class classification problems to convert raw scores or logits into probabilities. The softmax function calculates the exponential of each element in the input vector, normalizes the results, and ensures that the sum of the probabilities adds up to 1. This normalization allows us to interpret the outputs as probabilities of belonging to different classes.\n",
    "\n",
    "***The softmax function is defined as follows for an input vector x of length K:***\n",
    "- softmax(x_i) = exp(x_i) / (sum(exp(x_j)) for j=1 to K)\n",
    "\n",
    "***Cross-Entropy Loss:*** Cross-entropy loss, or simply \"cross-entropy,\" is a loss function commonly used in classification tasks to measure the dissimilarity between predicted probabilities and true class labels. It quantifies the difference between the predicted probability distribution (obtained from softmax) and the actual target distribution (one-hot encoded vector representing the true class).\n",
    "\n",
    "Given a set of N training samples, where each sample has K classes, and denoting the predicted probability distribution for the i-th sample as p_i and the true class labels as y_i, the cross-entropy loss is computed as follows:\n",
    "- CrossEntropyLoss = - (1/N) * sum(sum(y_i * log(p_i)))\n",
    "\n",
    "The inner sum is taken over all classes, and the outer sum is taken over all training samples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "508c1c9f",
   "metadata": {},
   "source": [
    "## Relation between Softmax and Cross-Entropy Loss\n",
    "--------\n",
    "\n",
    "The softmax function is typically applied to the logits or raw scores obtained from the final layer of a neural network before making predictions. It transforms these scores into probabilities that sum up to 1. The cross-entropy loss, on the other hand, measures the difference between these predicted probabilities and the true class labels.\n",
    "\n",
    "During training, the softmax function is used to obtain the predicted probabilities, and then the cross-entropy loss is computed based on these probabilities and the true labels. The goal is to minimize the cross-entropy loss, which effectively encourages the model to predict higher probabilities for the correct classes and lower probabilities for the incorrect classes.\n",
    "\n",
    "In summary, the softmax function is used to transform logits into probabilities, while the cross-entropy loss measures the dissimilarity between predicted probabilities and true class labels, providing a gradient for training the model in a supervised learning setting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b618dfa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "import torch.nn as nn\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5061b3c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    return np.exp(x) / np.sum(np.exp(x), axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8bd84753",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "softmax numpy: [0.65900114 0.24243297 0.09856589]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([2.0, 1.0, 0.1])\n",
    "outputs = softmax(x)\n",
    "print('softmax numpy:', outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5aab36e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "softmax numpy: tensor([0.6590, 0.2424, 0.0986])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([2.0, 1.0, 0.1])\n",
    "outputs = torch.softmax(x, dim = 0)\n",
    "print('softmax numpy:', outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "480720ea",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}