add support for pipenv

[goern]

Could supporting pipenv be added? :)

[mcyprian]

Pipenv is higher-level Python packaging tool officially recommended from Python.org. Pip and virtualenv are integrated into this tools and it supports Pipfile which intends to replace requirements.txt in the future.
Support of pipenv is in my opinion a valid request. If you agree @torsava @hhorak @GrahamDumpleton, I will prepare a PR.

[hhorak]

It's still not clear to me what changes are required in order to address this request. We don't have it packaged in RPMs, right? Would it be installed using pip and then used instead of pip?

[torsava]

@hhorak We don't have it packaged in RPMs, right? Would it be installed using pip and then used instead of pip?

Yes, that's the way we would have to go.

---

I'm not opposed to the idea, pipenv does bring some ease-of-use for developers. However, it might take some finesse to make it work here, since we're explicitly using virtualenv and pipenv wants to manage that on its own.

@mcyprian Maybe you can look into it and figure out how troublesome it would be to incorporate here. If there's a clean and easy solution without bad hacks, I'm all for it.

[hhorak]

I agree, let's see how it could be implemented and then we can decide..

As a side-note, there can be more requirements like this that might be not easily solved in the generally available assemble script, and we'll simply need to offer users to extend functionality of the assemble scripts on their own -- one example of such functionality is #216. So, IOW, all the s2i images should have assemble script that allows extending by users. Once we have it and the pipenv support would be too messy, we can just document how to do it by extending the assemble script.

[mcyprian]

Thanks for your comments, I will try to make something up.

[GrahamDumpleton]

For whoever looks at this, one thing you will need to contend with is the decision that $HOME is /opt/app-root/src. IMHO this was a bad design choice for various reasons. Either way, pipenv will want to create stuff under $HOME, so you will want to override that behaviour for a start and have it use a directory under /opt/app-root instead otherwise it will not play nice with local development workflows involving oc cluster up where you mount a local file system directory on top of /opt/app-root/src to allow live code updates.

[GrahamDumpleton]

So $HOME is not an issue as pipenv will use the virtual environment which already exists. You do need to be very careful though. Due to what I would regard as a bug in pipenv, if you run pipenv --two or pipenv --three when pipenv is installed inside of a virtual environment, it will first say, okay I will use the virtual environment. It then though goes on to delete the virtual environment, which for us is /opt/app-root, thus destroying the virtual environment and all the application code under /opt/app-root/src. Once it does that it decides, wait, I didn't create that virtual environment in the first place, so then aborts. Already too late though and it has destroyed everything.

Anyway, for now, the following can be used to handle a Pipfile.

Create executable script .s2i/bin/assemble which includes:

#!/bin/bash

pip install --no-cache-dir powershift-cli[image]

exec powershift image assemble

Create executable script .s2i/bin/run which includes:

#!/bin/bash

exec powershift image run

Create executable script .s2i/action_hooks/pre_build which includes:

#!/bin/bash

pip install --no-cache-dir pipenv

Create executable script .s2i/action_hooks/build which includes:

#!/bin/bash

pipenv install

Yes you could have done it all in custom assemble, albeit would need to add fix-permissions at end of custom assemble as well.

I used the powershift-image package to show benefit of having action hooks once more. ;-)

[GrahamDumpleton]

One issue with this approach will be that pipenv wants to install lots of its own packages. If these get installed into existing virtual environment, there could end up being a clash with versions of packages that the user itself wants, or what may get installed by a requirements.txt file if for some reason that still exists. Having pipenv installed outside of the virtual environment outside of the main virtual environment we use gets a bit messy though. You can't use the per user Python installation area as that may conflict with packages from SCL that pipenv itself may need, and that would have to be done without the main virtual environment being activated or used as can't use per user Python installation area at same time as virtual environment. So this is all going to need a bit more research to come up with solution that avoids issues. For now, the above will most likely work in most cases.

[torsava]

@GrahamDumpleton Thanks for the extensive rundown! We might want to just recommend using powershift like described, but I'll leave that up to Michal.

As for pipenv dependencies having different versions than the versions developer desires, I believe if you try to pipenv install a different version of a package that pipenv depends on, it will happily comply and install it, so I think that might not be a problem.

[GrahamDumpleton]

The problem with pipenv versions is that because it is using the same virtual environment as pipenv is installed in, if pipenv pulls in a pinned version of a package initially, if a user then has same package name in requirements.txt but doesn't specify a version, it will use whatever version is already installed and will not necessarily pull down the latest as the user might expect. This is the default behaviour of pip which pipenv would use under the covers and this would be what will happen unless pipenv is using -U or --update option to pip to force it to always update to latest version of packages. Will need to look further into pipenv to confirm what it does.

[torsava]

Ah, I was thinking people using pipenv would most likely have a Pipfile.lock file, but you're right that in other cases it would create an unintuitive behaviour. If that can indeed be mitigated by use of the --update flag, however, then it should be fine.

[mcyprian]

I've tried a couple of different approaches and pipenv locations. The best solution in my opinion is to install pipenv to per user Python installation area before the main virtualenv is activated, pipenv will have /opt/rh/rh-python36/root/usr/bin/python3 shebang, so activation of virtualenv won't affect it. Pipenv will than use existing virtualenv to install dependencies of the application without recreating it.

• per user Python installation area is reserved for pipenv and its dependencies
• main virtualenv is reserved for application dependencies
• there is no danger of application code loss

In case SCL pip is too old for pipenv (image <= centos/python-35-centos7) newer version will be installed to per user area and will be preferred in sys.path of SCL python binary (/opt/rh/rh-python36/root/usr/bin/python), but per user area is not included sys.path of python binary inside virtualenv (/opt/app-root/bin/python). When virtualenv is activated which pip is /opt/app-root/bin/pip - original version of pip copied from SCL pip, so I don't think this is an issue. My proof of concept implementation is here.

[torsava]

@mcyprian So if we needed a newer pip for pipenv, we would first create the virtualenv, then update pip outside of it and install pipenv to the per user directory?

I don't see a problem with this approach, so 👍 from me.

[mcyprian]

@torsava No, newer pip is installed automatically to per user directory when we run pip install --user pipenv in the Dockerfile, installation procedure is always the same, but there is extra pip in user site if pipenv needs it.

[GrahamDumpleton]

Using per user site-packages has the potential for problems. I am not confident that it is something we want to use because of possible conflicts with existing packages installed with SCL. I have blogged about this issue before at:

• http://blog.dscpl.com.au/2016/01/python-virtual-environments-and-docker.html

Anyway, see:

• https://github.com/kennethreitz/pipenv/issues/819

for related discussion of how to bootstrap pipenv so can be used. This is the bug report I created for the problem I found.

[mcyprian]

The only possible conflict here is pip, setuptools or virtualenv and as I mentioned before, newer version of pip is installed to user site in case SCL pip can't be used by pipenv, I don't see any big problem here, but I agree that installing it to the separate venv and symlinking the executable (similar to pipsi approach) is even cleaner solution.

[goern]

Thanks guys!! 👍