Differentiable programming for Differential equations: a review

⚠️ New preprint available! 📖 ⚠️

The review paper is now available as a preprint on arXiv: https://arxiv.org/abs/2406.09699

If you want to cite this work, please use this BibTex citation:

@misc{sapienza2024differentiable,
      title={Differentiable Programming for Differential Equations: A Review}, 
      author={Facundo Sapienza and Jordi Bolibar and Frank Schäfer and Brian Groenke and Avik Pal and Victor Boussange and Patrick Heimbach and Giles Hooker and Fernando Pérez and Per-Olof Persson and Christopher Rackauckas},
      year={2024},
      eprint={2406.09699},
      archivePrefix={arXiv},
      primaryClass={id='math.NA' full_name='Numerical Analysis' is_active=True alt_name='cs.NA' in_archive='math' is_general=False description='Numerical algorithms for problems in analysis and algebra, scientific computation'}
}

---

This respository contains all the text, code and figures used for the review paper about sentitivity methods for differential equations. This topic received different names in different communities, but the core problem is quite simple. Given a system of differential equations

$$\frac{du}{dt} = f(u, \theta, t),$$

with $u \in \mathbb R^n$ the unknow solution and $\theta \in \mathbb R^p$ a vector of parameters, how do we compute the gradient of a loss function

$$\mathcal L (\theta) = L ( u(\cdot, \theta) )$$

with respect to the parameters $\theta$ of the dynamical model?

There are different methods that try to solve this problem, and we had roughly classify them in the following scheeme.

Schematic classification of different methods to compute gradients of functions involving solutions of differential equations.

The goal of this review is to revisit all this methods and compare them.

Contributors

Facundo Sapienza 💻 📖 🤔 🚧 📆 🔬	Jordi Bolibar 📖 🤔 🔬	frankschae 💻 📖 🤔 🔬	Brian Groenke 💻 📖 🤔	Avik Pal 💻 📖 ⚠️	vboussange 📖 🤔 🔬	Patrick Heimbach 📖 🤔 🧑‍🏫 🔬
gileshooker 📖 🤔 🧑‍🏫	Fernando Pérez 📖 💵 🤔 🧑‍🏫	Christopher Rackauckas 💻 📖 💵 🤔 🧑‍🏫

Code 💻

We provide code for the different simulations and examples we exhibit in the project. Most of this code is provided in the Julia programming language, given that many of the libraries to perform sensitivity analysis are supported there, plus Julia solvers for differential equations are currently the state of the art in scientific computing.

How to use this repository ❓

This repository is organized in a way that contains most of the important elements of modern scientific workflow. We have included the following elements:

• tex: This is the folder where all the latex text belongs and what we use to compile main.pdf.
• code: Folder with both Jupyter notebooks and Julia scripts.
• CI: Continuous integration with GitHub actions to automatically compile and commit the manuscript.
• Makefile: make file that automatizes all the commands that can be executed within this repository.

Continuous integration

This repository has a workflow implemented that automatically compiles the latex files into the file main.pdf and then commits this file directy to the repository. If you are working from your fork, this action should also work and you should be able to generate the pdf file automatically using GitHub actions. In order to trigger the GitHub action to compile and commit the Latex file, you just need to inclide the word latex in your commit message.

This repository also includes an action to automatically merge all *.bib files into one single bibliography file tex/bibliography.bib that includes just the references that are cited in any of the *.tex files. In order to trigger this action, just include the word bib-merge in your commit message.

Makefile

Make is an old but very useful technology that allows automation of computing processes. From the directory where you have this respository you can enter make help from a terminal to display the different functionalities currently supported in our Makefile. We currently support the following operations:

• make tex: Compiles the main.tex latex file inside the folder tex with its respective bibliografy, deletes auxiliaries files in the process and them move the generated pdf file to the home directory.

Open Science from Scratch: contribute to the project! 👋

This review started with some of the authors willing to understand this tools in a comprehensive way and gathering references from fields like statistics, applied mathematics and computer science. Unhappy with the lack of a general compendium of the different methods that exists to address this problem, we had decided to make a single document where all the methods can coexists under common ground and can be compareded under their different scopes and domains of applications.

We are driving by learning and undersranding and the original authors of this report decided to manage this as an open science project from the beginning. What does this entitles? Anyone interested in participating and contrubuting is welcome to join. We beleive that science will benefit for more open collaborations happening under the basis of people trying to understand a topic.

We encourage contributors to participate in this project! If you are interested in contributing, there are many ways in which you can help build this:

• 💥 Report bugs in the code. You can report problems with the code by oppening issues under the Issues tab in this repository. Please explain the problem you encounter and try to give a complete description of it so we can follow up on that.
• 📚 Suggest new bibliography. If you are aware of references that may be useful to explore and expand this review, you can report it by creating an Issue in this repository, with the title of the issue being the title of the paper and adding the label paper to the issue.
• 💡 Request new features and explanations. If there is an important topic or example that you feel falls under the scope of this review and you would like us to include it, please request it! We are looking for new insights into what the community wants to learn.

Contact

If you have any questions or want to reach out, feel free to send us an email to [email protected].

License

The content of this project itself is licensed under the Creative Commons Attribution 3.0 Unported license, and the underlying source code used to format and display that content is licensed under the MIT license.