Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix linux PGO wheel build #170

Merged
merged 3 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/actions/build-pgo-wheel/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: Build PGO wheel
description: Builds a PGO-optimized wheel
inputs:
interpreter:
description: 'Interpreter to build the wheel for'
required: true
rust-toolchain:
description: 'Rust toolchain to use'
required: true
outputs:
wheel:
description: 'Path to the built wheel'
value: ${{ steps.find_wheel.outputs.path }}
runs:
using: "composite"
steps:
- name: prepare profiling directory
shell: bash
# making this ahead of the compile ensures that the local user can write to this
# directory; the maturin action (on linux) runs in docker so would create as root
run: mkdir -p ${{ github.workspace }}/profdata

- name: build initial wheel
uses: PyO3/maturin-action@v1
with:
manylinux: auto
args: >
--release
--out pgo-wheel
--interpreter ${{ inputs.interpreter }}
rust-toolchain: ${{ inputs.rust-toolchain }}
docker-options: -e CI
working-directory: crates/jiter-python
env:
RUSTFLAGS: '-Cprofile-generate=${{ github.workspace }}/profdata'

- name: detect rust host
run: echo RUST_HOST=$(rustc -Vv | grep host | cut -d ' ' -f 2) >> "$GITHUB_ENV"
shell: bash

- name: generate pgo data
run: |
pip install -U pip
pip install -r tests/requirements.txt
pip install jiter --no-index --no-deps --find-links pgo-wheel --force-reinstall
python bench.py jiter jiter-cache
RUST_HOST=$(rustc -Vv | grep host | cut -d ' ' -f 2)
rustup run ${{ inputs.rust-toolchain }} bash -c 'echo LLVM_PROFDATA=$RUSTUP_HOME/toolchains/$RUSTUP_TOOLCHAIN/lib/rustlib/$RUST_HOST/bin/llvm-profdata >> "$GITHUB_ENV"'
shell: bash
working-directory: crates/jiter-python

- name: merge pgo data
run: ${{ env.LLVM_PROFDATA }} merge -o ${{ github.workspace }}/merged.profdata ${{ github.workspace }}/profdata
shell: pwsh # because it handles paths on windows better, and works well enough on unix for this step

- name: build pgo-optimized wheel
uses: PyO3/maturin-action@v1
with:
manylinux: auto
args: >
--release
--out dist
--interpreter ${{ inputs.interpreter }}
rust-toolchain: ${{inputs.rust-toolchain}}
docker-options: -e CI
working-directory: crates/jiter-python
env:
RUSTFLAGS: '-Cprofile-use=${{ github.workspace }}/merged.profdata'

- name: find built wheel
id: find_wheel
run: echo "path=$(ls dist/*.whl)" | tee -a "$GITHUB_OUTPUT"
shell: bash
working-directory: crates/jiter-python
48 changes: 4 additions & 44 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -380,8 +380,6 @@ jobs:
with:
components: llvm-tools

- run: rustc --version --verbose

# linux: builds are done inside manylinux docker images, need to help maturin-action
# find the right path
#
Expand All @@ -401,49 +399,11 @@ jobs:
'3.13t': 'cp313-cp313t',
}['${{ matrix.interpreter }}'])")/bin/python >> $GITHUB_OUTPUT

- name: build initial wheel
uses: PyO3/maturin-action@v1
- name: build pgo wheel
uses: ./.github/actions/build-pgo-wheel
with:
manylinux: auto
args: >
--release
--out pgo-wheel
--interpreter ${{ matrix.os == 'linux' && steps.resolve-interpreter.outputs.python-path || steps.setup-python.outputs.python-path }}
rust-toolchain: stable
working-directory: crates/jiter-python
env:
RUSTFLAGS: "-Cprofile-generate=${{ github.workspace }}/profdata"

- name: detect rust host
run: echo RUST_HOST=$(rustc -Vv | grep host | cut -d ' ' -f 2) >> "$GITHUB_ENV"
shell: bash

- name: generate pgo data
run: |
cd crates/jiter-python
pip install -U pip
pip install -r tests/requirements.txt
pip install jiter --no-index --no-deps --find-links pgo-wheel --force-reinstall
python bench.py jiter jiter-cache
rustup run stable bash -c 'echo LLVM_PROFDATA=$RUSTUP_HOME/toolchains/$RUSTUP_TOOLCHAIN/lib/rustlib/${{ env.RUST_HOST }}/bin/llvm-profdata >> "$GITHUB_ENV"'

- name: merge pgo data
run: |
cd crates/jiter-python
${{ env.LLVM_PROFDATA }} merge -o ${{ github.workspace }}/merged.profdata ${{ github.workspace }}/profdata

- name: build pgo-optimized wheel
uses: PyO3/maturin-action@v1
with:
manylinux: auto
args: >
--release
--out dist
--interpreter ${{ matrix.os == 'linux' && steps.resolve-interpreter.outputs.python-path || steps.setup-python.outputs.python-path }}
rust-toolchain: stable
working-directory: crates/jiter-python
env:
RUSTFLAGS: "-Cprofile-use=${{ github.workspace }}/merged.profdata"
interpreter: ${{ matrix.os == 'linux' && steps.resolve-interpreter.outputs.python-path || steps.setup-python.outputs.python-path }}
rust-toolchain: ${{ steps.rust-toolchain.outputs.name }}

- run: ${{ matrix.ls || 'ls -lh' }} crates/jiter-python/dist/

Expand Down
Loading