Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Packaged Numpy wheel in macOS installer is only OS X 10.14+ compatible #2120

Closed
carlosperate opened this issue Mar 14, 2022 · 5 comments
Closed

Packaged Numpy wheel in macOS installer is only OS X 10.14+ compatible #2120

carlosperate opened this issue Mar 14, 2022 · 5 comments
Labels
high priority OS: macOS
Milestone
1.2.0

Comments

@carlosperate
Copy link
Member

carlosperate commented Mar 14, 2022 •
edited
Loading

Problem

The main Mu dependency restricting macOS compatibility is PyQt5. As we are currently using v5.13, so it should be compatible with macOS Sierra 10.12 and newer: https://doc.qt.io/archives/qt-5.13/macos.html

Currently the Mu installers download a numpy wheel for the user venv as a dependency for pygamezero and qtconsole.

With Python 3.8 in a modern macOS pip will download the latest Numpy which is only compatible wit macOS 10.14 Mojave and newer. So users with older OS versions are seeing errors like this one:

2022-03-13 16:03:57,169 - root:171(excepthook) ERROR: Unrecoverable error
Traceback (most recent call last):
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/app.py", line 159, in run
    raise ex
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/app.py", line 146, in run
    venv.ensure_and_create(self.display_text)
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 603, in ensure_and_create
    self.create()
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 772, in create
    self.install_baseline_packages()
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 885, in install_baseline_packages
    self.install_from_zipped_wheels(zipped_wheels_filepath)
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 860, in install_from_zipped_wheels
    self.pip.install(wheel, deps=False, index=False)
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 270, in install
    return self.run(
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 246, in run
    result = self.process.run_blocking(
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 122, in run_blocking
    return self.wait(wait_for_s=wait_for_s)
  File "/Applications/Mu Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/virtual_environment.py", line 186, in wait
    raise VirtualEnvironmentError(
mu.virtual_environment.VirtualEnvironmentError: Process finished but with error code 1:
ERROR: numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl is not a supported wheel on this platform.

User Workaround

We can delete the packaged wheels included with Mu, which will force Mu to download them again.

To do this run the following command on a terminal (assuming Mu was installed in the Applications folder):

rm /Applications/Mu\ Editor.app/Contents/Resources/Python/lib/python3.8/site-packages/mu/wheels/1.1.1.zip

Then launch Mu and this time it should work.
If Mu crashes again we would really appreciated if you could submit a new Crash Report letting us know you tried this workaround, as that will help us revole the issue. Thanks!
@carlosperate carlosperate changed the title Packaged Numpy wheel in macOS installer only OS X 10.9+ compatible Packaged Numpy wheel in macOS installer is only OS X 10.9+ compatible Mar 14, 2022
@carlosperate carlosperate changed the title Packaged Numpy wheel in macOS installer is only OS X 10.9+ compatible Packaged Numpy wheel in macOS installer is only OS X 10.14+ compatible Mar 14, 2022
@carlosperate
Copy link
Member Author

carlosperate commented Mar 14, 2022 •
edited
Loading

Related crash reports:

@carlosperate
Copy link
Member Author

carlosperate commented Mar 15, 2022 •
edited
Loading

Possible Solutions

❌ Would deleting the wheel zip file inside the Mu installation directory be a possible workaround?

If the wheels zip file is deleted Mu tries to download the latest numpy version from PyPI via pip. As there isn't a wheel available in the latest release it download the source package and tries to build numpy.

❌ What if we add the --prefer-binary flag to the pip donwload steps?

Looks like this feature gives wheels preference, going back to older compatible versions with wheels, but if a package doesn't have any wheels, it fails the pip download step. So it looks like this flag would never download a source distribution, which currently is needed for Mu to build a couple of pure python packages into wheels.

❌ Can we add numpy to the user venv dependencies with a version clamp to ensure a compatible version is downloaded and installed in the user venv?

Currently pip doesn't check if a "compatible" dependency has already been downloaded in the destination directory (if the exact same wheel it tries to download is already present it does not download it again).
So first it would download the older and compatible numpy wheel, then it would download the pgzero and its dependencies, pip sees that that numpy (which doesn't have any version clamp) is a required dependency and then download the latest numpy release wheel as well.
So we end up with a zip file with two versions of numpy. Then when the user venv is created the version that prevails is the last wheel that was installed.

❌ Can we use the --platform flag to ensure any downloaded wheels is compatible with older OS releases?

In this case the value for the --platform flag would be macosx_10_12_x86_64, and pip can then download wheels marked for 10.12 or lower (like an older macosx_10_9_x86_64 numpy wheel), so that's good.

However, this flag requires the --only-binary or --no-deps options.
We cannot use --no-deps unless we list all dependencies inside the mu/wheels/__init__.py file (almost like a requirements.txt file). And if we use --only-binary then we hit the same problem as --prefer-binary, where it fails to download packages without wheels.

❌ Can we add numpy to the user venv dependencies with the --platform flag?

We would have the same issue as clamping the numpy version, and end up with two wheels in the zip file.

🤷‍♂️ Can we use a constraints.txt file?

Maybe this could work?

🤷‍♂️ Should we use some pip freeze-like list of dependencies for the user venv?

This might be worth discussion or exploring.
This could also resolve the issue that is guaranteed that in the future some subdependency is going to break the pip download or venv creation step and therefore break old versions of Mu and make them unusable.
Some discussion here, but basically this has happened before and it will happen again:

One disadvantage is that it will likely include quite a few environmental markers for dependencies that are platform specific, python version specific, etc. So it will also be harder to generate and will take some time and crash reports to refine it.

The opposite problem might also be true, where having all dependencies locked means they might not work in future OS releases. However this issue is much less troublesobe, as people on newer OS versions will likely want to use the newest version of Mu, compared to the other problem where users in old OS versions cannot use newer versions of Mu.

🤷‍♂️ Is there something in pip-tools (or something else) that could help us?

Good question! Is anybody aware of something that we could input a list of packages, some restrictions, and outputs a resolved full listed of packages (including dependencies)?

@carlosperate carlosperate added this to the 1.1.2 milestone May 11, 2022
This was referenced May 25, 2022
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@carlosperate carlosperate modified the milestones: 1.2.0, 1.2.1 Jun 6, 2022
@carlosperate carlosperate mentioned this issue Jun 9, 2022
Closed
@carlosperate carlosperate mentioned this issue Jun 24, 2022
Closed
@carlosperate carlosperate mentioned this issue Jul 8, 2022
Closed
This was referenced Jul 16, 2022
Closed
Closed
This was referenced Aug 15, 2022
Closed
Closed
This was referenced Aug 22, 2022
Closed
Closed
@akayer19 akayer19 mentioned this issue Aug 26, 2022
Closed
@carlosperate carlosperate mentioned this issue Aug 29, 2022
Closed
@carlosperate carlosperate mentioned this issue Sep 12, 2022
Closed
@carlosperate carlosperate mentioned this issue Sep 20, 2022
Closed
This was referenced Oct 17, 2022
Closed
Closed
This was referenced Oct 25, 2022
Closed
Closed
Closed
Closed
This was referenced Nov 2, 2022
Closed
Closed
Closed
Closed
@carlosperate carlosperate modified the milestones: 1.2.1, 1.2.0 Nov 9, 2022
@carlosperate
Copy link
Member Author

carlosperate commented Nov 9, 2022 •
edited
Loading

Okay, so I think we have two options:

  1. make a requirement.txt-like full list of pinned dependencies that we know work everywhere
    • This single list could still use environmental markers to adapt some dependencies depending on the platforms
  2. Ensure everything installed in the user venv has wheels available and use the --platform flag

I think option 1) is a significant change on how the user venv dependencies are picked and installed, perhaps too much of a change to rush it in the last minute before v1.2.0. Not because the change is large in number of lines, but because it's a different way to specify dependency versions (pinning vs. min version has a lot of pros and cons).

For option 2) the only dependency that haven't got wheels in PyPI is esptool and its dependencies, and taking in consideration the esptool dependencies have been a headache for a really log time (see #1952), I think we can save ourselves some trouble by vendoring the esptool.py script (we only need a single script from the entire package) and that will also reduce the dependency list, which remove some of the most annoying packages to install.

So my proposal is:

  • Remove esptool as user venv dependency, pick the single file script from esptool we need and vendor it inside Mu
    • We only use esptool.py, which only requires pyserial, which is already included
    • All the other dependencies listed for esptool are for the other scripts, which we don't need
    • This does change the category of the dependency from being a "user venv dependency" to a "Mu runtime dependency". I think this is okay as esptool is used in the settings window, so it's closer to the "Mu as an app dependency" than "a dependency to run user code"
    • This would also cover: Avoid installing cryptography (for esptool) module in the user venv #1952
  • Add the --platform flag to the pip download commands from mu.wheels
    • We should instrument mu.wheels to only add the --platform flag when we are packaging Mu, not when the user is running Mu from a pip/pipx/python-source installation

An alternative proposal, which I think it's still worth pursuing, but not for tonight, would be to update Mu so that if installing wheels from the zip file fails, it can try again but this time ignoring the zip file and downloading the wheels again.

This was referenced Nov 10, 2022
Merged
Merged
@carlosperate
Copy link
Member Author

PRs #2352 and #2351 have been created:

@carlosperate
Copy link
Member Author

This has been fixed with PRs #2351 and #2352 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high priority OS: macOS
Projects
None yet
Milestone
1.2.0
Development

No branches or pull requests
1 participant
@carlosperate