Making virtualenv setup transparent via import statements

[andyk]

As we work on our demo, I keep coming back to the same observation about the extra work of the developer having to declare their components in a yaml file, which feels cumbersome and like it may end up being an inhibitor to adoption.

So I've been thinking about an alternate approach to the problem of making Virtualenv setup transparent to the developer. And I'm wondering if we can use existing Python language mechanisms for declaring dependencies:

• import statements declare a dependency on another module or a class within that module, and can be used at the module, class, or class instance level
• classes can declare other classes as parent classes that they inherit from or declare other classes as their metaclass
• class instances take arguments to their constructor __init__() function

At a high level, library support in Python is done via modules and those modules are dynamically linked and interpreted at runtime.

Developers use the language constructs listed above to declare a namespace's (i.e., a module/class/class instance) dependencies on a symbol and they then assume that symbol conforms to a particular API.

Then, as is typical with dynamically linked languages, using some other process/language that is separate from their source code (e.g., pip install at the command line), it is up to the developer to install the source files that the Python runtime will load and map to those library symbols.

What I'm proposing is that we could provide a new but (hopefully) backwards compatible mechanism that would allow a developer to declare in their code, in addition to the library (i.e. dependency) symbol, optional extra information that the runtime would use to select a specific implementation of the library to link.

As a straw-person proposal for one way to do this: we we could make import statements assume that if a symbol is not found locally within the project directory (i.e. if it isn't part of this project) then it is a pypi dependency. Furthermore, the import statement could be augmented to support (1) version information, (2) where to fetch the versioned python code from (3) how to fetch it (e.g. download it via https from pypi servers, git clone a repo, etc.).

Some examples of potential valid import statements:

# The following would fetch the project dependency from PyPI
import numpy  # this would do something similar to `pip install numpy` under the hood
import numpy==1.22.3
import numpy>=1.0.0

# Also support non-pypi sources such as fetching python projects direct from github
import git+https://github.com/numpy/numpy.git

# Also support installing a PCS HashDAG
import pcs+URL_TO_PCS_REGISTRY:MODULE_COMPONENT_IDENTIFIER

[nickjalbert]

Yeah! I've also had thoughts along these lines (e.g. see the Just-in-time venv doc's "Goal" section).

One idea to ease the path-to-adoption is to initially specify reqs in the comments so that the programs could still be run by the vanilla interpreter, e.g.

I like the addition of supporting hashdag (and the broader PCS/AOS ecosystem) in the same manner. I think we may have talked about this before, but it looks a bit like how Go and Rust (maybe?) handle can handle imports from non-local sources.