managing-environment.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "a5a8ad7c-d322-4c70-9d35-d56392fca217",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"Managing your computational environment\"\n",
    "author: \"Kyle Niemeyer\"\n",
    "date: \"6 January 2025\"\n",
    "institute: \"Oregon State University\"\n",
    "format:\n",
    "  revealjs:\n",
    "    output-file: managing-environment-revealjs.html\n",
    "    incremental: true\n",
    "  html: default\n",
    "jupyter: python3\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861cc532-b340-4d06-9b75-ddf6dc421e6a",
   "metadata": {},
   "source": [
    "# Introduction to virtual environments\n",
    "\n",
    "## Power of Python for research comes from rich ecosystem of third-party packages\n",
    "\n",
    "::: {.fragment}\n",
    "These include general numerical and scientific tools like NumPy, SciPy, and Pandas, and domain-specific software like Cantera. You should rely on mature packages that provide useful features when possible—don't reinvent the wheel!\n",
    ":::\n",
    "\n",
    "::: {.fragment}\n",
    "But, each software package has its own set of dependencies, and might require a specific version of some other package (like when a newer version breaks or changes some functionality).\n",
    ":::\n",
    "\n",
    "## Your results may depend on the specific versions and combination of packages you have installed. {.center}\n",
    "\n",
    "::: {.fragment}\n",
    "This is particularly important for research analysis.\n",
    ":::\n",
    "\n",
    "## 🚨 As a rule, don't install packages into your global environment! {.center}\n",
    "\n",
    "::: {.fragment}\n",
    "### ... even if the documentation says `pip install x`\n",
    ":::\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74f5800-fa88-474c-8061-d8a299adc14c",
   "metadata": {},
   "source": [
    "## Best practice: use virtual environments\n",
    "\n",
    "- Virtual environments are lightweight installations of packages in a particular location (like in your working directory or project project) that build on top of your system Python installation.\n",
    "\n",
    "- Virtual environments isolate the set of dependencies for a project or package from each other, and from your system configuration.\n",
    "\n",
    "- These are intended to be easy to set up, disposable, and replaceable.\n",
    "\n",
    "---\n",
    "\n",
    "- On a Mac or Linux machine, I recommend using the built-in [`venv` module](https://docs.python.org/3/library/venv.html) to create a virtual environment for your work, then install inside that using the `pip` installer.\n",
    "\n",
    "- On Windows, your best bet is probably the [Anaconda Distribution](https://www.anaconda.com/download). Anaconda comes with hundreds of scientific packages already, and also lets you set up system-wide virtual environments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c3e395-811e-490f-a019-00b21a653556",
   "metadata": {},
   "source": [
    "# Using venv to manage virtual environments\n",
    "\n",
    "## Creating and activating a virtual environment\n",
    "\n",
    "```{.bash .fragment}\n",
    "python3 -m venv .venv\n",
    "```\n",
    "\n",
    "::: {.fragment}\n",
    "This creates a new directory `.venv` with:\n",
    ":::\n",
    "\n",
    "- `.venv/bin` with link to Python, `pip`, and activation scripts; and\n",
    "- `.venv/lib/site-packages` where packages will be installed.\n",
    "\n",
    "---\n",
    "\n",
    "::: {.fragment}\n",
    "To activate the environment, source the activation script:\n",
    ":::\n",
    "\n",
    "```{.bash .fragment}\n",
    "source .venv/bin/activate\n",
    "```\n",
    "\n",
    "::: {.callout-tip .fragment}\n",
    "Try this yourself. `.venv/bin` will be added to your PATH, and your shell will usually indicate you are \"in\" a virtual environment.\n",
    ":::\n",
    "\n",
    "::: {.fragment}\n",
    "To leave the virtual environment, run the deactivate function:\n",
    ":::\n",
    "\n",
    "```{.bash .fragment}\n",
    "deactivate\n",
    "```\n",
    "\n",
    "::: {.callout-note .fragment}\n",
    "Closing/terminating your shell will automatically deactivate the virtual environment, and you'll need to manually activate whenever you start a new session.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d9dbf81-d3fa-4a42-aaed-8106b84a5594",
   "metadata": {},
   "source": [
    "## Installing packages\n",
    "\n",
    "```{.bash .fragment}\n",
    "pip install <package>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066721d7-3183-42e5-9131-87cadc64ccc9",
   "metadata": {},
   "source": [
    "# Alternatives for managing virtual environments\n",
    "\n",
    "## pip family\n",
    "\n",
    "Tools that all work similarly to `venv`:\n",
    "\n",
    "- `venv`: comes with Python, simple\n",
    "- [`virtualenv`](https://virtualenv.pypa.io/): installable package with same interface as `venv`, but faster and with more options\n",
    "- [`uv`](https://docs.astral.sh/uv/): written in Rust and extremely fast\n",
    "\n",
    "---\n",
    "\n",
    "In addition, there are project management tools that both handle environments, packaging, production, etc.:\n",
    "\n",
    "- [Hatch](https://hatch.pypa.io/)\n",
    "- [Poetry](https://python-poetry.org)\n",
    "- [Pipenv](https://pipenv.pypa.io/en/)\n",
    "\n",
    "::: {.fragment}\n",
    "These work a bit differently, but can be quite powerful particularly for locking specific versions of dependencies. Consider returning to these when we are thinking about packaging and distribution.\n",
    ":::\n",
    "\n",
    "## Conda family\n",
    "\n",
    "::: {.fragment}\n",
    "Conda is an open-source tool for managing virtual environments across your system, originally designed for Python package management but now supports a wide range of software (e.g., compiled C++ dependencies).\n",
    "::: \n",
    "\n",
    "::: {.fragment}\n",
    "Virtual environments created with Conda are not location-dependent like `venv`, but the same approach applies: do not install in the base/default environment.\n",
    "::: \n",
    "\n",
    "::: {.fragment}\n",
    "Conda is recommended if you expect to need non-Python dependencies.\n",
    "::: \n",
    "\n",
    "\n",
    "## {.center}\n",
    "\n",
    "```bash\n",
    "conda config --set auto_activate_base false  # turn off the default environment\n",
    "conda env create -n some_name  # or use paths with `-p`\n",
    "conda activate some_name\n",
    "conda deactivate\n",
    "```\n",
    "\n",
    "## Conda-based package managers\n",
    "\n",
    "Some alternatives to conda exist and offer some advantages, including speed:\n",
    "\n",
    "- [Miniconda](https://docs.anaconda.com/miniconda/): minimal Conda installation without all the packages that come with the full Anaconda distribution\n",
    "- [Pixi](https://pixi.sh/latest/): fast, Rust-based package manager and workflow tool\n",
    "Some other\n",
    "- [Micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html): tiny version of Mamba, statically linked C++ binary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "604c19cd-bb60-4f32-826d-d844f0d9cc2a",
   "metadata": {},
   "source": [
    "# References and further reading\n",
    "\n",
    "---\n",
    "\n",
    "Schreiner, H., Niemeyer, K., Holland, J. G., Dang, G., & Restrepo, M. I. (2024). \"INTERSECT: Packaging\" [Data set]. <https://github.com/INTERSECT-training/packaging>\n",
    "\n",
    "[Scientific Python Library Development Guide](https://learn.scientific-python.org/development/)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5014316f-73f5-4b57-a39a-9548cd4185f2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}