packaging-distribution.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "title: \"Packaging and distributing your Python code\"\n",
    "author: \"Kyle Niemeyer\"\n",
    "date: \"29 January 2025\"\n",
    "institute: \"Oregon State University\"\n",
    "format:\n",
    "  revealjs:\n",
    "    output-file: packaging-distribution-revealjs.html\n",
    "    incremental: true\n",
    "    fig-align: center\n",
    "    fig-responsive: true\n",
    "    toc: false\n",
    "  html: default\n",
    "jupyter: python3\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook to installable package\n",
    "\n",
    "## Hypothetical workflow for a researcher {.smaller}\n",
    "\n",
    "1. Work on idea for paper with collaborators\n",
    "2. Do exploratory analysis in scripts and Jupyter ecosystem\n",
    "3. As research progresses, need to write more-complicated functions and workflows\n",
    "3. Code begins to sprawl across multiple directories\n",
    "4. Software dependencies begin to become more complicated\n",
    "4. The code \"works on my machine\", but what about your collaborators?\n",
    "\n",
    "::: {.r-fit-text .fragment}\n",
    "**People heroically press forward, but this is painful, and not reusable**\n",
    ":::\n",
    "\n",
    "## {.smaller}\n",
    "\n",
    "Imagine you start with a Jupyter notebook that looks like this:\n",
    "\n",
    "```{.python .fragment}\n",
    "import numpy as np\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "# Rosenbrock function\n",
    "def rosen(x):\n",
    "    \"\"\"The Rosenbrock function\"\"\"\n",
    "    return sum(100.0 * (x[1:] - x[:-1] ** 2.0) ** 2.0 + (1 - x[:-1]) ** 2.0)\n",
    "\n",
    "def rosen_der(x):\n",
    "    \"\"\"Gradient of the Rosenbrock function\"\"\"\n",
    "    xm = x[1:-1]\n",
    "    xm_m1 = x[:-2]\n",
    "    xm_p1 = x[2:]\n",
    "    der = np.zeros_like(x)\n",
    "    der[1:-1] = 200 * (xm - xm_m1**2) - 400 * (xm_p1 - xm**2) * xm - 2 * (1 - xm)\n",
    "    der[0] = -400 * x[0] * (x[1] - x[0] ** 2) - 2 * (1 - x[0])\n",
    "    der[-1] = 200 * (x[-1] - x[-2] ** 2)\n",
    "    return der\n",
    "\n",
    "# Minimization of the Rosenbrock function with some initial guess\n",
    "x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])\n",
    "result = minimize(rosen, x0, method=\"BFGS\", jac=rosen_der, options={\"disp\": True})\n",
    "optimized_params = result.x\n",
    "print(optimized_params)\n",
    "```\n",
    "\n",
    "## Reusable science, step by step\n",
    "\n",
    "We can convert our notebook code into a simple importable module an and example calling it:\n",
    "\n",
    "``` bash\n",
    "$ tree edit-sys-path \n",
    "edit-sys-path\n",
    "├── code\n",
    "│   └── utils.py\n",
    "└── example.py\n",
    "\n",
    "2 directories, 2 files\n",
    "```\n",
    "\n",
    "## Reusable science, step by step\n",
    "\n",
    "``` python\n",
    "# example.py\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "# Make ./code/utils.py visible to sys.path\n",
    "# sys.path position 1 should be after cwd and before activated virtual environment\n",
    "sys.path.insert(1, str(Path().cwd() / \"code\"))\n",
    "from utils import rosen, rosen_der\n",
    "\n",
    "x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])\n",
    "result = minimize(rosen, x0, method=\"BFGS\", jac=rosen_der, options={\"disp\": True})\n",
    "optimized_params = result.x\n",
    "print(optimized_params)\n",
    "```\n",
    "\n",
    "## Reusable science, step by step\n",
    "\n",
    "- This is **already better** than having everything in a single massive file!\n",
    "- However, now things are tied to this relative path on your computer:\n",
    "\n",
    "``` {.python .fragment}\n",
    "# Make ./code/utils.py visible to sys.path\n",
    "sys.path.insert(1, str(Path(__file__).parent / \"code\"))\n",
    "from utils import rosen, rosen_der\n",
    "```\n",
    "\n",
    "::: fragment\n",
    "and are brittle to refactoring and change; plus, not very portable to others.\n",
    "::: \n",
    "\n",
    "- But we can do better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fundamentals\n",
    "\n",
    "## What is a Python *package*?\n",
    "\n",
    "::: fragment\n",
    "First, let's define **module**.\n",
    ":::\n",
    "\n",
    "::: fragment\n",
    "From <https://docs.python.org/3/glossary.html#term-module>:\n",
    ":::\n",
    "\n",
    "**module**\n",
    ": An object that serves as an organizational unit of Python code. Modules have a namespace containing arbitrary Python objects. Modules are loaded into Python by the process of [importing](https://docs.python.org/3/glossary.html#term-importing).\n",
    "\n",
    ": See also [package](https://docs.python.org/3/glossary.html#term-package).\n",
    "\n",
    "## What is a Python *package*? {.smaller}\n",
    "\n",
    "Deconstructing that definition:\n",
    "\n",
    "- When I write `import modulename` I am loading a module\n",
    "- But we also talk about \"importing a package\"! This seems to imply a package is a kind of module.\n",
    "- The module can contain \"arbitrary Python objects\", which could include other modules.\n",
    "- It's almost like we need a special name for a \"module that can contain other modules\"...\n",
    "- When loading a module, its name gets added to the *namespace*, a collection of currently defined names and objects they reference: `numpy.random.default_rng`\n",
    "\n",
    "## Aside: why does Python have modules?\n",
    "\n",
    "::: fragment\n",
    "Because it's good for your code to be *modular*.\n",
    ":::\n",
    "\n",
    "::: aside\n",
    "Source: <https://realpython.com/python-modules-packages/>\n",
    ":::\n",
    "\n",
    "## Where do modules come from?\n",
    "\n",
    "- The standard library\n",
    "  - Python code\n",
    "  - Compiled C code\n",
    "- A local file that ends in `.py (confusingly, also called a \"module\")\n",
    "- Third-party libraries that you `pip install` or `conda install` (i.e., packages)\n",
    "- A local package\n",
    "\n",
    "## What is a Python *package*?\n",
    "\n",
    "We spent all that time defining **module**, now we can define **package**:\n",
    "\n",
    "::: fragment\n",
    "From <https://docs.python.org/3/glossary.html#term-package>:\n",
    ":::\n",
    "\n",
    "**package**\n",
    ": A Python module which can contain submodules or recursively, subpackages.\n",
    "\n",
    "## \"Package\" can have multiple meanings!\n",
    "\n",
    "[Two main usages](https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/):\n",
    "\n",
    "1. import package: the one we talked about, when you write `import packagename`. We'll start by making this.\n",
    "2. distribution package: the actual artifact that gets downloaded off the internet and stored somewhere, like when you run `pip install package`. We'll make these too.\n",
    "\n",
    "## When should I turn my code into a package?\n",
    "\n",
    "Two common cases for research code:\n",
    "\n",
    "1. Code that goes with a research article; mainly used to reproduce the results, AKA a (computational) project or a research compendium.\n",
    "2. A generalized tool or **library** that other researchers can use\n",
    "\n",
    "::: fragment\n",
    "Not all packages intended to reproduce a paper's results need the full infrastructure we will discuss (documentation website, continuous integration, etc.).\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Packaging your code\n",
    "\n",
    "## Next steps: packaging your code\n",
    "\n",
    "- The goal is for your code to be installable, and distributable.\n",
    "- Per the Zen of Python, this should be straightforward, right?\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ python3 -c 'import this' | grep obvious\n",
    "There should be one-- and preferably only one --obvious way to do it.\n",
    "Although that way may not be obvious at first unless you're Dutch.\n",
    "```\n",
    "\n",
    "## Next steps: packaging your code\n",
    "\n",
    "Unfortunately, not so much. 😔\n",
    "\n",
    "::: fragment\n",
    "[![](./images/python-packaging-options.png){width=60% fig-alt=\"List of Python packaging options\"}](https://github.com/scientific-python/cookie)\n",
    ":::\n",
    "\n",
    "::: fragment\n",
    "You might be asking: Why is there more than one thing?\n",
    ":::\n",
    "\n",
    "## Next steps: packaging your code {.smaller}\n",
    "\n",
    "The [good news]{style=\"color:blue;\"}: Python packaging has improved dramatically in the last 5 years\n",
    "\n",
    "- It has never been easier to just point your package manager to some code locally, or on the internet, and get working Python code installed and running on your machine regardless of operating system or architecture\n",
    "- This is a small miracle!\n",
    "\n",
    "::: fragment\n",
    "The [bad news]{style=\"color:red;\"}: Python packaging has expanded dramatically in the last 5 years\n",
    ":::\n",
    "\n",
    "::: fragment\n",
    "- By creating standards the PyPA allowed for an ecosystem of packaging backends to be created to tackle various problems (this is good!)\n",
    "- This means that our The Zen of Python expectations are violated and we need to make design choices (hard for beginners)\n",
    ":::\n",
    "\n",
    "## Next steps: packaging your code\n",
    "\n",
    "The [okay news]{style=\"color:green;\"}: You can probably default to the simplest thing.\n",
    "\n",
    "- pure Python: Probably `hatch`\n",
    "- compiled extensions: Probably [`setuptools` + `pybind11`](https://pybind11.readthedocs.io/) or [`scikit-build-core`](https://scikit-build-core.readthedocs.io/) + [`pybind11`](https://github.com/pybind/pybind11)\n",
    "\n",
    ":::: {.columns}\n",
    "\n",
    "::: {.fragment .column width=\"50%\"}\n",
    "[![Python Packaging User Guide, Packaging Python Projects Tutorial](./images/python-packaging-guide.png)](https://packaging.python.org/en/latest/tutorials/packaging-projects/)\n",
    ":::\n",
    "\n",
    "::: {.fragment .column width=\"50%\"}\n",
    "[![Scientific Python Library Development Guide](./images/scientific-python-guide.png)](https://learn.scientific-python.org/development/guides/packaging-simple/)\n",
    ":::\n",
    "\n",
    "::::\n",
    "\n",
    "## Simple packaging example\n",
    "\n",
    "Modern [PEP 518](https://peps.python.org/pep-0518/) compliant build backends just need a single file: [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/)\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ tree simple_packaging\n",
    "simple-packaging\n",
    "├── LICENSE\n",
    "├── README.md\n",
    "├── pyproject.toml\n",
    "├── src\n",
    "│   └── rosen\n",
    "│       ├── __init__.py\n",
    "│       ├── _version.py\n",
    "│       └── example.py\n",
    "└── tests\n",
    "    └── test_example.py\n",
    "\n",
    "4 directories, 7 files\n",
    "```\n",
    "\n",
    "## Simple packaging example: `pyproject.toml` {.smaller}\n",
    "\n",
    "What is `.toml?`\n",
    "\n",
    "> \"TOML aims to be a minimal configuration file format that's easy to read due to obvious semantics. TOML is designed to map unambiguously to a hash table. TOML should be easy to parse into data structures in a wide variety of languages.\" — [https://toml.io/](https://toml.io/) (emphasis mine)\n",
    "\n",
    "In recent years TOML has seen a rise in popularity for configuration files and lock files. Things that need to be easy to read (humans) and easy to parse (machines).\n",
    "\n",
    "## Simple packaging example: `pyproject.toml`\n",
    "\n",
    "Defining how your project should get **built**:\n",
    "\n",
    "``` {.toml .fragment}\n",
    "[build-system]\n",
    "requires = [\n",
    "    \"hatchling>=1.13.0\",\n",
    "    \"hatch-vcs>=0.3.0\",\n",
    "]\n",
    "build-backend = \"hatchling.build\"\n",
    "...\n",
    "```\n",
    "\n",
    "## Simple packaging example: `pyproject.toml`\n",
    "\n",
    "Defining project **metadata** and **requirements**/**dependencies**:\n",
    "\n",
    "``` {.toml .fragment}\n",
    "[project]\n",
    "name = \"rosen\"\n",
    "dynamic = [\"version\"]\n",
    "description = \"Example package for demonstration\"\n",
    "readme = \"README.md\"\n",
    "license = { text = \"BSD-3-Clause\" }  # SPDX short identifier\n",
    "authors = [\n",
    "  { name = \"Kyle Niemeyer\", email = \"kyle.niemeyer@oregonstate.edu\" },\n",
    "]\n",
    "requires-python = \">=3.8\"\n",
    "\n",
    "dependencies = [\n",
    "    \"scipy>=1.6.0\",\n",
    "    \"numpy\",  # compatible versions controlled through scipy\n",
    "]\n",
    "...\n",
    "```\n",
    "\n",
    "## Simple packaging example: `pyproject.toml`\n",
    "\n",
    "Configuring **tooling options** and **interactions** with other tools:\n",
    "\n",
    "``` {.toml .fragment}\n",
    "[tool.hatch.version]\n",
    "source = \"vcs\"\n",
    "\n",
    "[tool.hatch.version.raw-options]\n",
    "local_scheme = \"no-local-version\"\n",
    "# Need to give root as we aren't at the same level as the git repo\n",
    "root = \"..\"\n",
    "\n",
    "[tool.hatch.build.hooks.vcs]\n",
    "version-file = \"src/rosen/_version.py\"\n",
    "...\n",
    "```\n",
    "\n",
    "## Simple packaging example: Installing your code {.smaller}\n",
    "\n",
    "You can now locally install your package into your Python virtual environment!\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ cd simple_packaging\n",
    "$ pip install --upgrade pip wheel\n",
    "$ pip install .\n",
    "Successfully built rosen\n",
    "Installing collected packages: rosen\n",
    "Successfully installed rosen-0.0.2.dev1\n",
    "$ pip show rosen\n",
    "Name: rosen\n",
    "Version: 0.0.2.dev1\n",
    "Summary: Example package for demonstration\n",
    "Home-page: https://github.com/SoftwareDevEngResearch/packaging-examples\n",
    "Author: \n",
    "Author-email: Kyle Niemeyer <kyle.niemeyer@oregonstate.edu>\n",
    "License: BSD-3-Clause\n",
    "Location: ***/.venv/lib/python3.13/site-packages\n",
    "Requires: numpy, scipy\n",
    "Required-by: \n",
    "```\n",
    "\n",
    "## Simple packaging example: Installing your code\n",
    "\n",
    "and use it anywhere\n",
    "\n",
    "``` python\n",
    "# example.py\n",
    "import numpy as np\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "# We can now import our code\n",
    "from rosen.example import rosen, rosen_der\n",
    "\n",
    "x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])\n",
    "result = minimize(rosen, x0, method=\"BFGS\",\n",
    "                  jac=rosen_der, options={\"disp\": True})\n",
    "optimized_params = result.x\n",
    "# array([1.00000004, 1.0000001 , 1.00000021, 1.00000044, 1.00000092])\n",
    "```\n",
    "\n",
    "## Packaging doesn't slow down development\n",
    "\n",
    "[PEP 518](https://peps.python.org/pep-0518/) compliant build backends allow for \"[editable installs](https://pip.pypa.io/en/latest/topics/local-project-installs/#editable-installs)\"\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ python -m pip install --upgrade --editable .\n",
    "$ python -m pip show rosen | grep --ignore-case 'location'\n",
    "Location: ***/lib/python3.12/site-packages\n",
    "Editable project location: ***/examples/simple_packaging\n",
    "```\n",
    "\n",
    "::: fragment\n",
    "Editable installs add the files in the development directory to Python’s import path. (Only need to re-installation if you change the project metadata.)\n",
    ":::\n",
    "\n",
    "::: fragment\n",
    "Can **develop** your code under `src/` and have **immediate** access to it.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packaging compiled extensions\n",
    "\n",
    "With modern packaging infrastructure, packaging compiled extensions requires small extra work!\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ tree compiled_packaging\n",
    "compiled-packaging\n",
    "├── CMakeLists.txt\n",
    "├── LICENSE\n",
    "├── README.md\n",
    "├── pyproject.toml\n",
    "├── src\n",
    "│   ├── basic_math.cpp\n",
    "│   └── rosen_cpp\n",
    "│       ├── __init__.py\n",
    "│       └── example.py\n",
    "└── tests\n",
    "    └── test_example.py\n",
    "\n",
    "4 directories, 8 files\n",
    "```\n",
    "\n",
    "## Packaging compiled extensions\n",
    "\n",
    "In `pyproject.toml`:\n",
    "\n",
    "Swap build system to [`scikit-build-core`](https://scikit-build-core.readthedocs.io/) + [`pybind11`](https://github.com/pybind/pybind11)\n",
    "\n",
    "``` {.toml .fragment}\n",
    "[build-system]\n",
    "requires = [\n",
    "  \"scikit-build-core\",\n",
    "  \"pybind11\"\n",
    "  ]\n",
    "build-backend = \"scikit_build_core.build\"\n",
    "...\n",
    "```\n",
    "\n",
    "## Packaging compiled extensions\n",
    "\n",
    "[`CMakeLists.txt`](https://cliutils.gitlab.io/modern-cmake/):\n",
    "\n",
    "``` {.cmake .fragment}\n",
    "# Specify CMake version and project language\n",
    "cmake_minimum_required(VERSION 3.15...3.30)\n",
    "project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)\n",
    "\n",
    "# Setup pybind11\n",
    "set(PYBIND11_FINDPYTHON ON)\n",
    "find_package(pybind11 CONFIG REQUIRED)\n",
    "\n",
    "# Add the pybind11 module to build targets\n",
    "pybind11_add_module(basic_math MODULE src/basic_math.cpp)\n",
    "install(TARGETS basic_math DESTINATION ${SKBUILD_PROJECT_NAME})\n",
    "```\n",
    "\n",
    "## Packaging compiled extensions\n",
    "\n",
    "`src/basic_math.cpp`:\n",
    "\n",
    "``` {.cpp .fragment}\n",
    "#include <pybind11/pybind11.h>\n",
    "\n",
    "int add(int i, int j) { return i + j; }\n",
    "\n",
    "namespace py = pybind11;\n",
    "\n",
    "PYBIND11_MODULE(basic_math, m) {\n",
    "  m.def(\"add\", &add, R\"pbdoc(\n",
    "      Add two numbers\n",
    "  )pbdoc\");\n",
    "...\n",
    "\n",
    "}\n",
    "```\n",
    "\n",
    "## Packaging compiled extensions: Installing\n",
    "\n",
    "Installing locally is the same as for the pure-Python example:\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ cd simple_packaging\n",
    "$ pip install --upgrade pip wheel\n",
    "$ pip install .\n",
    "Successfully built rosen-cpp\n",
    "Installing collected packages: rosen-cpp\n",
    "Successfully installed rosen-cpp-0.0.1\n",
    "```\n",
    "\n",
    "Module name is that given in C++:\n",
    "\n",
    "``` {.python .fragment}\n",
    "from rosen_cpp import basic_math\n",
    "basic_math.add(1, 2)\n",
    "# 3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributing packages\n",
    "\n",
    "## Going further: Distributing packages {.smaller}\n",
    "\n",
    "If your code is publicly available on the WWW in a Git repository, you've already done a version of distribution!\n",
    "\n",
    "``` {.bash .fragment}\n",
    "# pip install \"git+https://example.com/repo/project.git@branch#subdirectory=path\"\n",
    "$ pip install \\\n",
    "  \"git+https://github.com/SoftwareDevEngResearch/packaging-examples.git#subdirectory=simple_packaging\"\n",
    "```\n",
    "\n",
    "``` {.bash .fragment}\n",
    "# Works for pure-Python packages\n",
    "$ pip install --upgrade \"git+https://github.com/scikit-hep/pyhf.git\"\n",
    "# as well as packages with compiled extensions\n",
    "$ pip install --upgrade \"git+https://github.com/scikit-hep/iminuit.git\"\n",
    "```\n",
    "\n",
    "## Going further: Distributing packages\n",
    "\n",
    "Ideally, we'd prefer a more organized approach: distribution through a **package index**.\n",
    "\n",
    "::: fragment\n",
    "First, we need to create **distributions** of our packaged code.\n",
    "::: \n",
    "\n",
    "::: fragment\n",
    "Distributions that `pip` can install:\n",
    "\n",
    "- [source distribution (sdist)](https://packaging.python.org/en/latest/glossary/#term-Source-Distribution-or-sdist): A tarfile (`.tar.gz`) of the source files of our package (subset of all the files in the repository)\n",
    "- [wheel](https://packaging.python.org/en/latest/glossary/#term-Built-Distribution): A zipfile (`.whl`) of the file system structure and package metadata with any dependencies prebuilt. No arbitrary code execution, only decompressing and copying of files\n",
    "::: \n",
    "\n",
    "## Going further: Distributing packages\n",
    "\n",
    "To create these **distributions** from source code, rely on our package **build backend** (e.g., [`hatchling`](https://hatch.pypa.io/)) and **build frontend** tool like [`build`](https://pypa-build.readthedocs.io/en/stable/)\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ pip install --upgrade build\n",
    "$ python -m build .\n",
    "* Creating venv isolated environment...\n",
    "* Installing packages in isolated environment... (hatch-vcs>=0.3.0, hatchling>=1.13.0)\n",
    "* Getting build dependencies for sdist...\n",
    "* Building sdist...\n",
    "* Building wheel from sdist\n",
    "* Creating venv isolated environment...\n",
    "* Installing packages in isolated environment... (hatch-vcs>=0.3.0, hatchling>=1.13.0)\n",
    "* Getting build dependencies for wheel...\n",
    "* Building wheel...\n",
    "Successfully built rosen-0.0.1.tar.gz and rosen-0.0.1-py3-none-any.whl\n",
    "$ ls dist\n",
    "rosen-0.0.1-py3-none-any.whl  rosen-0.0.1.tar.gz\n",
    "```\n",
    "\n",
    "## Going further: Distributing packages\n",
    "\n",
    "We can now securely upload the distributions under `./dist/` to any package index that understands how to use them.\n",
    "\n",
    "::: fragment\n",
    "The most common is the [Python Package Index (PyPI)](https://pypi.org), which serves as the default package index for `pip`.\n",
    "\n",
    "[![](./images/pypi-page.png){width=50% fig-alt=\"PyPI landing page\"}](https://pypi.org/)\n",
    ":::\n",
    "\n",
    "## Distributing packages: [conda-forge](https://conda-forge.org/)\n",
    "\n",
    "The conda family of package managers ([`conda`](https://docs.conda.io/), [`mamba`](https://mamba.readthedocs.io/), [`micromamba`](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html), [`pixi`](https://prefix.dev/docs/pixi/overview)) take an alternative approach from `pip`.\n",
    "\n",
    "::: fragment\n",
    "Instead of installing Python packages, they act as general purpose package managers and install **all dependencies** (including Python) as OS and architecture specific **built binaries** (.`conda` files — `zip`file containing compressed `tar` files) hosted on conda-forge.\n",
    ":::\n",
    "\n",
    "::: fragment\n",
    "This allows an additional level of runtime environment specification not possible with just `pip`, though getting environment solves right can become more complicated.\n",
    ":::\n",
    "\n",
    "## Distributing packages: [conda-forge](https://conda-forge.org/)\n",
    "\n",
    "Popular in scientific computing as arbitrary binaries can be hosted, including compilers (e.g., `gcc`, Fortran) and even the [full NVIDIA CUDA stack](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda-installation)!\n",
    "\n",
    "::: fragment\n",
    "With the change to full binaries only this also requires that specification of the environment being installed is important.\n",
    ":::\n",
    "\n",
    "::: fragment\n",
    "With sdists and wheels, if there is no compatible wheel available, `pip` will automatically **fall back** to trying to locally build from the sdist. Can't do that if there is no matching `.conda` binary!\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Versioning\n",
    "\n",
    "## Semantic Versioning\n",
    "\n",
    "Given a version number `MAJOR.MINOR.PATCH`, increment the:\n",
    "\n",
    "- `MAJOR` version when you make incompatible API changes,\n",
    "- `MINOR` version when you add functionality in a backwards-compatible manner, and\n",
    "- `PATCH` version when you make backwards-compatible bug fixes.\n",
    "\n",
    "::: fragment\n",
    "To start: initial development release starts at 0.0.1, and increment minor version for subsequent releases.\n",
    ":::\n",
    "\n",
    "## Versioning with `hatch` & `git`\n",
    "\n",
    "We actually already set up our `hatch` build system to automatically version our package based on `git`: \n",
    "\n",
    "``` {.toml .fragment}\n",
    "[tool.hatch.version]\n",
    "source = \"vcs\"\n",
    "\n",
    "[tool.hatch.version.raw-options]\n",
    "local_scheme = \"no-local-version\"\n",
    "# Need to give root as we aren't at the same level as the git repo\n",
    "root = \"..\"\n",
    "\n",
    "[tool.hatch.build.hooks.vcs]\n",
    "version-file = \"src/rosen/_version.py\"\n",
    "...\n",
    "```\n",
    "\n",
    "::: fragment\n",
    "This tells `hatch` to look at the latest `git` tag and use that as the version number, stored in the automatically generated file `src/rosen/_version.py`.\n",
    ":::\n",
    "\n",
    "## Versioning with `hatch` & `git`\n",
    "\n",
    "To document a new version, simply create a new `git` tag:\n",
    "\n",
    "``` {.bash .fragment}\n",
    "$ git tag v0.x.y -m \"version 0.x.y\"\n",
    "$ git push --tags\n",
    "```\n",
    "\n",
    "::: fragment\n",
    "This will automatically update the version number in `src/rosen/_version.py` when next building the package.\n",
    ":::\n",
    "\n",
    "## Good idea: keep a `CHANGELOG` {.smaller}\n",
    "\n",
    "Use a [`CHANGELOG`](https://keepachangelog.com/) file to document changes in your package over time.\n",
    "\n",
    "``` {.markdown .fragment}\n",
    "# Changelog\n",
    "All notable changes to this project will be documented in this file.\n",
    "\n",
    "The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)\n",
    "and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).\n",
    "\n",
    "## [Unreleased]\n",
    "\n",
    "## [0.0.2] - 2014-07-10\n",
    "### Added\n",
    "- Explanation of the recommended reverse chronological release ordering.\n",
    "\n",
    "## 0.0.1 - 2014-05-31\n",
    "### Added\n",
    "- This CHANGELOG file to hopefully serve as an evolving example of a\n",
    "  standardized open source project CHANGELOG.\n",
    "- CNAME file to enable GitHub Pages custom domain\n",
    "- README now contains answers to common questions about CHANGELOGs\n",
    "- Good examples and basic guidelines, including proper date formatting.\n",
    "- Counter-examples: \"What makes unicorns cry?\"\n",
    "\n",
    "[Unreleased]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.2...HEAD\n",
    "[0.0.2]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.1...v0.0.2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Takeaways\n",
    "\n",
    "- Packaging your code is a good idea\n",
    "- Packaging and distributing your code is pretty easy\n",
    "- Later, we'll see how continuous integration can help automate this process"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}