Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pydantic v2 #1017

Open
bitxeno opened this issue Nov 6, 2023 · 16 comments · May be fixed by #1328
Open

pydantic v2 #1017

bitxeno opened this issue Nov 6, 2023 · 16 comments · May be fixed by #1328
Labels
packages

Comments

@bitxeno
Copy link

bitxeno commented Nov 6, 2023

I follow this link #657 to build pydantic v2.4.2，but download package step report error:

build-wheel.py: python3.10 -m pip --disable-pip-version-check install cffi==1.15.1 setuptools-rust==1.2.0
Requirement already satisfied: cffi==1.15.1 in /home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages (1.15.1)
Requirement already satisfied: setuptools-rust==1.2.0 in /home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages (1.2.0)
Requirement already satisfied: pycparser in /home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages (from cffi==1.15.1) (2.21)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages (from setuptools-rust==1.2.0) (4.8.0)
Requirement already satisfied: setuptools>=46.1 in /home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages (from setuptools-rust==1.2.0) (67.0.0)
Requirement already satisfied: semantic-version<3,>=2.8.2 in /home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages (from setuptools-rust==1.2.0) (2.10.0)
build-wheel.py: cd /mnt/d/dev/chaquopy/server/pypi/packages/pydantic/build/2.4.2
build-wheel.py: rm -rf /mnt/d/dev/chaquopy/server/pypi/packages/pydantic/build/2.4.2/cp310-cp310-android_21_x86_64
build-wheel.py: mkdir -p /mnt/d/dev/chaquopy/server/pypi/packages/pydantic/build/2.4.2/cp310-cp310-android_21_x86_64
build-wheel.py: python3.10 -m pip --disable-pip-version-check download --no-deps --no-binary pydantic --no-build-isolation pydantic==2.4.2
Collecting pydantic==2.4.2
  Using cached pydantic-2.4.2.tar.gz (654 kB)
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [53 lines of output]
      Traceback (most recent call last):
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_wheel'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/build.py", line 56, in build_wheel
          return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 93, in build
          self.metadata.validate_fields()
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/metadata/core.py", line 243, in validate_fields
          _ = self.version
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/metadata/core.py", line 128, in version
          self._version = self._get_version()
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/metadata/core.py", line 222, in _get_version
          core_metadata = self.core
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/metadata/core.py", line 166, in core
          metadata_hooks = self.hatch.metadata.hooks
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/metadata/core.py", line 1539, in hooks
          metadata_hook = self.plugin_manager.metadata_hook.get(hook_name)
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/plugin/manager.py", line 97, in get
          return self.collect().get(name)
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/plugin/manager.py", line 67, in collect
          self.third_party_plugins.load()
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatchling/plugin/manager.py", line 106, in load
          self.manager.load_setuptools_entrypoints('hatch')
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/pluggy/_manager.py", line 398, in load_setuptools_entrypoints
          plugin = ep.load()
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
          module = import_module(match.group('module'))
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/importlib/__init__.py", line 126, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatch_fancy_pypi_readme/hooks.py", line 12, in <module>
          from ._builder import build_text
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatch_fancy_pypi_readme/_builder.py", line 7, in <module>
          from ._fragments import Fragment
        File "/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/hatch_fancy_pypi_readme/_fragments.py", line 14, in <module>
          from jsonschema import Validator
      ImportError: cannot import name 'Validator' from 'jsonschema' (/home/apps/miniconda3/envs/build-wheel-310/lib/python3.10/site-packages/jsonschema/__init__.py)
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
build-wheel.py: Warning: pip download returned exit status 1
build-wheel.py: Error: Can't find downloaded source archive. Does the name and version in the package's meta.yaml match the filename shown above?

how to fix this?
@IsakTheHacker
Copy link
Contributor

Related: #995

@Saeppo
Copy link

Saeppo commented Nov 10, 2023 •
edited by mhsmith
Loading

This worked for me.

python {
            version 3.8
            pip {
                install "pydantic<2"
            }
 }

This was referenced Nov 28, 2023
Closed
Closed
@mhsmith
Copy link
Member

mhsmith commented Nov 28, 2023

Originally posted by @caarmen in #995 (comment)

Would like to be able to use pydantic 2. Some libs seem to require it (ex: ariadne-codegen: mirumee/ariadne-codegen#186 )

@mhsmith
Copy link
Member

mhsmith commented Nov 28, 2023 •
edited
Loading

Originally posted by @IsakTheHacker in #995

I need the pydantic package for my app which depends on the native pydantic-core package written in Rust. I have tried following the guide for building Cryptography version 39 and read #657. It is hard to understand exactly what I need to do to set up Rust support when there is no official documentation. For example, do I need to cross-compile Python as described here or is that specific to the Cryptogaphy package?

@mhsmith
Copy link
Member

mhsmith commented Nov 28, 2023

@IsakTheHacker: I'll try to centralize all the notes about Rust in a single place: #1030.

@mhsmith mhsmith mentioned this issue Jan 11, 2024
Closed
@svdgoor
Copy link

svdgoor commented May 30, 2024 •
edited
Loading

Is there a method to manually build and insert the module? I don't need the latest version each time, but since openai depends on it I cannot build my app.

I need to build for Android and iOS. Neither are currently functioning.

@mhsmith mhsmith mentioned this issue Jun 3, 2024
Closed
@mhsmith
Copy link
Member

mhsmith commented Jun 3, 2024 •
edited
Loading

Since openai is still compatible with pydantic version 1, there is an easy workaround for that:

            pip {
                install("pydantic<2")
                install("openai")
            }

@mhsmith mhsmith changed the title Build pydantic v2 report error pydantic v2 Jun 23, 2024
@mhsmith mhsmith mentioned this issue Jun 23, 2024
Closed
@mhsmith mhsmith mentioned this issue Aug 19, 2024
Closed
@mhsmith mhsmith mentioned this issue Sep 10, 2024
Closed
This was referenced Nov 12, 2024
Closed
Open
@mhsmith
Copy link
Member

mhsmith commented Nov 19, 2024

Originally posted by @maciekglowka in #1261 (comment)

Would this approach [from #1261] work for crates that don't support ABI3 ? I am currently trying to build pydantic core, but due to a jiter dependency I can't use this feature on PyO3.
[otherwise I keep getting linking errors with libgcc as previously]

Originally posted by @emanuele-f in #1261 (comment)

Hello, no, for non-ABI3 modules currently you will still need to cross compile the Python interpreter as explained in #657

Originally posted by @maciekglowka in #1261 (comment)

Shoot....thanks for the quick answer though. I was worried that'd be the case (after spending half a day again trying different configurations).
I managed to even work around the jiter thing (it has cfg flags). But then pydantic itself import loads of non-abi3 types, so...

@mhsmith mhsmith mentioned this issue Nov 19, 2024
Merged
@mhsmith
Copy link
Member

mhsmith commented Nov 19, 2024

@maciekglowka: Are the libgcc errors the same as in #1142? If so, that probably indicates that the build is using the Linux compiler rather than the Android one. The current NDK has replaced GCC with Clang, so I don't think libgcc even exists for Android anymore.

@maciekglowka
Copy link

@mhsmith that's the issue. No libgcc to link to (as it's not in the android toolchain anymore).

I've tried to use rust's nightly feature to build the std lib from source (https://doc.rust-lang.org/cargo/reference/unstable.html#build-std) but no success yet.
The other approach would be to compile libgcc for the target. I think numba package does it, but I do not fully understand it's shell script yet :)

@mhsmith
Copy link
Member

mhsmith commented Nov 19, 2024

I've tried to use rust's nightly feature to build the std lib from source (https://doc.rust-lang.org/cargo/reference/unstable.html#build-std) but no success yet.

You mean the Rust std lib? Why would you need to rebuild that? If Rust has Android support, doesn't it provide a pre-compiled copy?

The other approach would be to compile libgcc for the target. I think numba package does it, but I do not fully understand it's shell script yet :)

No, that's the wrong approach. Libgcc is very closely coupled to the exact compiler version, and it can't be used with a different one. Clang has its own equivalent called libcompiler_rt.

But all this is a compiler implementation detail which you shouldn't normally be aware of. These are static libraries, included implicitly by the linker, so we don't even need to worry about distributing them.

Numba is an exception because it does low-level JIT compilation, so its recipe will need to be significantly changed to work with the current NDK.

Like I said, the fact that you're seeing a reference to libgcc anywhere in the build probably indicates that a Linux compiler has been used at some point. I suggest you look through the build log to try and find out how this happened. It's often caused by build scripts which hard-code "cc" or "gcc" rather than using the CC environment variable.

@maciekglowka
Copy link

You mean the Rust std lib? Why would you need to rebuild that? If Rust has Android support, doesn't it provide a pre-compiled copy?

Thought that was something worth a shot (to avoid linking to the standard std-lib):
rust-mobile/ndk#149 (comment)

Ok, thanks - I will look for the wrong toolchain used by PyO3

@emanuele-f
Copy link
Contributor

You should provide the env variables for the cross compilation, either by editing the build-wheel script or by creating a patch under the pydantic package.

The following worked before, but may need adaptations now that the abi3 modules compilation support was merged:

os.environ["RUSTFLAGS"] = f"-C linker={os.environ['CC']}"
os.environ["CARGO_BUILD_TARGET"] = os.environ['CHAQUOPY_TRIPLET']

# https://pyo3.rs/v0.15.2/building_and_distribution.html
os.environ["PYO3_PYTHON"] = f"python{os.environ['CHAQUOPY_PYTHON']}"
os.environ["PYO3_CROSS_PYTHON_VERSION"] = os.environ['CHAQUOPY_PYTHON']
os.environ["PYO3_CROSS_LIB_DIR"] = f"{os.environ['RECIPE_DIR']}/../../../../build/{os.environ['CHAQUOPY_ABI']}/sysroot/usr/lib"
#print(os.environ)

@maciekglowka
Copy link

Thanks for the help!
However I think I'd give up for now, that doesn't feel feasible for me. I've tested so many configurations that I am confused what gave which output :)
[anyway the furthest I could ever get was the dreaded sys config data or whatever it's properly called)

This was referenced Dec 13, 2024
Closed
Closed
@Jzhenli
Copy link

Jzhenli commented Jan 13, 2025

anything update for this topic? I need to use the pydantic v2, but seems it's not possible for android.

@mhsmith
Copy link
Member

mhsmith commented Jan 13, 2025

As far as I know, nobody has made any further progress on this package.

If anyone else needs this package, please don't post a comment unless it provides some actual help in building it. Instead, register your support by clicking the thumbs up button below the top comment.

Meanwhile, you may be able to use Pydantic version 1 as shown in the workaround above, and/or use an older version of the package that requires it.

@mhsmith mhsmith linked a pull request Feb 2, 2025 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
packages
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add pydantic core FCare/chaquopy
8 participants
@mhsmith @emanuele-f @maciekglowka @Jzhenli @Saeppo @svdgoor @IsakTheHacker @bitxeno