CI: Build debian packages and run autopackage tests #130

3v1n0 · 2023-12-05T21:22:57Z

Add a workflow to build debian packages and run autopkgtests.

Requires: canonical/desktop-engineering#58

UDENG-2343

codecov-commenter · 2023-12-05T21:27:24Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.09%. Comparing base (36511cd) to head (0642fbb).
Report is 230 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
- Coverage   83.43%   83.09%   -0.34%     
==========================================
  Files          83       96      +13     
  Lines        8689     9603     +914     
  Branches       74       74              
==========================================
+ Hits         7250     7980     +730     
- Misses       1111     1239     +128     
- Partials      328      384      +56

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

didrocks

There is a lot of assumptions on that PR, in particular in which distro to runs. Also, the containers in polluted with preparing the build package and end up having network connection when building it, which is not what a real ppa or builder is doing.

I think it will be better to reuse https://github.com/canonical/desktop-engineering/blob/main/gh-actions/common/build-debian/action.yml, amend it for anything needed without extending too much complexity. Please check with @EduardGomezEscandell who is the primary author.

Some of the changes that can be made for instance it to not rely on

    - name: Build package
      uses: jtdor/build-deb-action@v1

For instance, which should be easy enough to do.

That way, we can decide on which releases to build and ensure we have something closer to a real build environement. Adding autopkgtests as a separate action is interesting there too.

.github/workflows/build-deb.yaml

3v1n0 · 2023-12-11T17:30:32Z

There is a lot of assumptions on that PR, in particular in which distro to runs.

Consider this was done as a quick thing to test #120 (and actually caught few issues during the final testing of that PR :)).

So, there are assumptions but all depending on variables, so ideally can be multiplexed to multiple jobs, although this was not a requirement yet.

Also, the containers in polluted with preparing the build package and end up having network connection when building it, which is not what a real ppa or builder is doing.

Yeah, this has to be changed indeed.

I think it will be better to reuse https://github.com/canonical/desktop-engineering/blob/main/gh-actions/common/build-debian/action.yml, amend it for anything needed without extending too much complexity. Please check with @EduardGomezEscandell who is the primary author.

However I was looking at the code of that action and I'm not sure how internet is prevented there too, since it seems that certificates are still installed. However I'll dig on that more further.

EduardGomezEscandell · 2023-12-12T09:08:49Z

Hi, thanks for taking a look :)

However I was looking at the code of that action and I'm not sure how internet is prevented there too, since it seems that certificates are still installed. However I'll dig on that more further.

The action works in two phases, each in a separate container.

The first one builds the source package, with certificates installed
The second one builds the package, with no certificates.

3v1n0 · 2023-12-12T14:20:18Z

name: Build package
uses: jtdor/build-deb-action@v1

Something I don't like of that is that it's running the build process as root, which is what other builders do, but it should not be the case. I can probably patch that to fix this though.

3v1n0 · 2024-02-25T17:20:58Z

I've tried again using canonical/desktop-engineering/gh-actions/common/build-debian but still this doesn't work as per this error:

  dh_auto_configure --buildsystem=cargo
  debian cargo wrapper: options, profiles, parallel, lto: ['parallel=4'] ['noudeb'] ['-j4'] 0
  debian cargo wrapper: rust_type, gnu_type: x86_64-unknown-linux-gnu, x86_64-linux-gnu
  error: current package believes it's in a workspace when it's not:
  current:   /github/workspace/tmp.Vj6UHjMoxL/source/vendor_rust/tokio/Cargo.toml
  workspace: /github/workspace/Cargo.toml
  this may be fixable by adding `tmp.Vj6UHjMoxL/source/vendor_rust/tokio` to the `workspace.members` array of the manifest located at: /github/workspace/Cargo.toml
  Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
  malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/cargo/bin/dh-cargo-vendored-sources line 23.
  dh_auto_configure: error: /usr/share/cargo/bin/dh-cargo-vendored-sources returned exit code 255
  make[1]: Leaving directory '/github/workspace/tmp.Vj6UHjMoxL/source'
  make[1]: *** [debian/rules:68: override_dh_auto_configure] Error 25
  make: *** [debian/rules:50: binary] Error 2
  dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

It's quite weird since it does seems to run in the same environment.... And funny thing: there's no relevant difference in source packages as per debdiff.

So.... I'm not really sure what it is. I'll check in future, but I think that for now the safest way would just go with this option (once I address the comments on building based on the source and disabling internet access), instead of continuing living without a CI action that ensures we're still packaging authd properly.

3v1n0 · 2024-02-26T05:46:54Z

Opening this again, even though now it's using canonical/desktop-engineering#28

.github/workflows/build-deb.yaml

…s and add tests (#28) As underlined in ubuntu/authd#130 (comment) using `jtdor/build-deb-action` for building debian packages has some problems that are yet not clear, and it also does not seem to properly support building non-native packages. As per this, given that the code needed for handling the build inside a simpler docker container is quite small, I've moved the build to manual handling which allows us some more control over the steps at the cost of being a bit more verbose. As bonus: - We now only build as user (not the sources, since there's no much point) - All the network traffic is totally blocked for the builder user (not just HTTPS) - ~~`DEB_BUILD_OPTIONS` can be customized for special builds (e.g. `nocheck`)~~ - Version uses actual distro version I've added some commits from #22, but not included the lintian build yet (that I've ready [here](3v1n0@1b72205)) not to make this PR harder to check, but also because I think those could instead be part of another action to allow more parallelization when used. By doing this change, not only both test cases pass, but also authd is correctly built: ubuntu/authd#130 (e.g. https://github.com/ubuntu/authd/actions/runs/8044158864) UDENG-2439

denisonbarbosa

Thanks for handling this!

didrocks

I have a couple of questions/notes/requests. Once you are done merging with the other PR on the desktop-engineering repo, we can have a look at this together :)

.github/workflows/build-deb.yaml

debian/rules

Ensure that autopkgtests works with authd at every run

Uniform the same code that we've in debian rules with debian tests

Those are so far the targets we care, so let's build and test packages in both scenarios

didrocks

One last comment to add I think before we release. Looks good otherwise!

.github/workflows/build-deb.yaml

didrocks

Excellent, thanks!

…ified

As per lintian report

Thanks lintian!

Newer versions require new cargo that is not in noble, so we don't try to use the newest version. Even though using a bit more advanced matrix we could do that...

3v1n0 · 2025-01-22T18:23:26Z

@didrocks sorry, I had to re-iterate since I noticed that indeed the modified thingy didn't work (was removed long time ago)...

So I re-implemented it myself: https://github.com/ubuntu/authd/compare/c0052923fc0b17f645abe0e2d79beed3877d1409..0642fbb04e3be78ace0057a8654c5bf58d4b5110

3v1n0 requested a review from a team as a code owner December 5, 2023 21:22

GabrielNagy force-pushed the debian-packaging branch from 1a4eecc to aef2d42 Compare December 5, 2023 21:26

3v1n0 force-pushed the debian-packaging branch from e92325b to f2aa14a Compare December 5, 2023 21:28

3v1n0 changed the base branch from debian-packaging to main December 5, 2023 21:30

3v1n0 force-pushed the debian-packaging branch 2 times, most recently from 9df4ea2 to 7281cd3 Compare December 6, 2023 02:09

didrocks previously requested changes Dec 7, 2023

View reviewed changes

.github/workflows/build-deb.yaml Outdated Show resolved Hide resolved

.github/workflows/build-deb.yaml Show resolved Hide resolved

3v1n0 mentioned this pull request Dec 13, 2023

tests(pam/testutils): Integration tests for the PAM CLI #109

Merged

3v1n0 marked this pull request as draft January 15, 2024 13:21

3v1n0 force-pushed the debian-packaging branch 2 times, most recently from 96257cf to dd5f794 Compare February 26, 2024 05:28

3v1n0 mentioned this pull request Feb 26, 2024

feat(build-debian): Use docker to manually run build and prepare steps and add tests canonical/desktop-engineering#28

Merged

3v1n0 marked this pull request as ready for review February 26, 2024 05:46

3v1n0 force-pushed the debian-packaging branch from dd5f794 to 0a23351 Compare February 26, 2024 06:17

denisonbarbosa requested changes Feb 26, 2024

View reviewed changes

.github/workflows/build-deb.yaml Outdated Show resolved Hide resolved

3v1n0 marked this pull request as draft December 13, 2024 16:30

3v1n0 force-pushed the debian-packaging branch 2 times, most recently from 95bab82 to 032a1f9 Compare December 14, 2024 02:18

3v1n0 mentioned this pull request Dec 14, 2024

feat(gh-actions/build-debian): Support running a workflow script at build-source phase canonical/desktop-engineering#54

Merged

3v1n0 force-pushed the debian-packaging branch 5 times, most recently from f2aecf5 to dee7b3a Compare December 19, 2024 15:41

denisonbarbosa approved these changes Dec 19, 2024

View reviewed changes

3v1n0 force-pushed the debian-packaging branch from dee7b3a to c386833 Compare January 9, 2025 03:00

3v1n0 requested a review from didrocks January 9, 2025 03:03

didrocks reviewed Jan 15, 2025

View reviewed changes

.github/workflows/build-deb.yaml Show resolved Hide resolved

.github/workflows/build-deb.yaml Outdated Show resolved Hide resolved

debian/rules Outdated Show resolved Hide resolved

3v1n0 added 3 commits January 17, 2025 16:08

CI: Add workflow to build debian package as part of build

5b1713b

CI: Add workflow to run autopkgtests on the previously built deb package

d3c221d

Ensure that autopkgtests works with authd at every run

CI/build-deb: reuse desktop actions to run autopkgtests

a35b553

3v1n0 force-pushed the debian-packaging branch from c386833 to bc3c687 Compare January 17, 2025 16:17

debian/tests/run-tests: Try to use the backported go version

a2abdb9

Uniform the same code that we've in debian rules with debian tests

3v1n0 force-pushed the debian-packaging branch from bc3c687 to 256957a Compare January 17, 2025 16:21

3v1n0 requested a review from didrocks January 17, 2025 16:22

CI/build-deb: Use a matrix to run package builds in latest LTS and devel

f1bd88c

Those are so far the targets we care, so let's build and test packages in both scenarios

3v1n0 force-pushed the debian-packaging branch 2 times, most recently from 69da712 to e5b42a8 Compare January 17, 2025 23:54

didrocks requested changes Jan 22, 2025

View reviewed changes

.github/workflows/build-deb.yaml Show resolved Hide resolved

3v1n0 requested a review from didrocks January 22, 2025 13:00

didrocks approved these changes Jan 22, 2025

View reviewed changes

3v1n0 added 4 commits January 22, 2025 18:14

ci: Only run autpkgtests on main, on tags and if a debian file is mod…

3fd53e7

…ified

debian/changelog: Drop trailing whitespace

628967f

As per lintian report

debian/copyright: Drop superflus files patterns

2a78294

Thanks lintian!

build-deb: Lock the cargo-filterer-version to the noble one

0642fbb

Newer versions require new cargo that is not in noble, so we don't try to use the newest version. Even though using a bit more advanced matrix we could do that...

3v1n0 force-pushed the debian-packaging branch from c005292 to 0642fbb Compare January 22, 2025 18:17

didrocks approved these changes Jan 23, 2025

View reviewed changes

3v1n0 merged commit 2244650 into ubuntu:main Jan 23, 2025
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Build debian packages and run autopackage tests #130

CI: Build debian packages and run autopackage tests #130

3v1n0 commented Dec 5, 2023 •

edited

Loading

codecov-commenter commented Dec 5, 2023 •

edited

Loading

didrocks left a comment •

edited

Loading

3v1n0 commented Dec 11, 2023

EduardGomezEscandell commented Dec 12, 2023 •

edited

Loading

3v1n0 commented Dec 12, 2023

3v1n0 commented Feb 25, 2024

3v1n0 commented Feb 26, 2024

denisonbarbosa left a comment

didrocks left a comment

didrocks left a comment

didrocks left a comment

3v1n0 commented Jan 22, 2025

CI: Build debian packages and run autopackage tests #130

CI: Build debian packages and run autopackage tests #130

Conversation

3v1n0 commented Dec 5, 2023 • edited Loading

codecov-commenter commented Dec 5, 2023 • edited Loading

Codecov Report

didrocks left a comment • edited Loading

Choose a reason for hiding this comment

3v1n0 commented Dec 11, 2023

EduardGomezEscandell commented Dec 12, 2023 • edited Loading

3v1n0 commented Dec 12, 2023

3v1n0 commented Feb 25, 2024

3v1n0 commented Feb 26, 2024

denisonbarbosa left a comment

Choose a reason for hiding this comment

didrocks left a comment

Choose a reason for hiding this comment

didrocks left a comment

Choose a reason for hiding this comment

didrocks left a comment

Choose a reason for hiding this comment

3v1n0 commented Jan 22, 2025

3v1n0 commented Dec 5, 2023 •

edited

Loading

codecov-commenter commented Dec 5, 2023 •

edited

Loading

didrocks left a comment •

edited

Loading

EduardGomezEscandell commented Dec 12, 2023 •

edited

Loading