Parse Library.uk files

[nderjung]

Feature request summary

This issue tracks the parsing of a top-level file called Library.uk within the context of an external Unikraft-compatible microlibrary repository in KraftKit and following the resolution of the RFC process for challenges of microlibrary versioning. The goal of the Library.uk file is intended to both simplify the discovery process as well as the internal realization of an external microlibrary within the context of KraftKit.

Please see the RFC for additional context, definitions and goals.

Background and Challenges

Thanks to Unikraft's modular architecture, which is ultimately a combination of its GNU Make-based build system and the use of the KConfig configuration language, it is possible to plug in external "microlibraries" which represent alternative implementations for common APIs (whether plugging into the Linux syscall shim layer or directly into Unikraft's internal APIs through masking). This powerful primitive makes it possible to swap implementations for previously impossible OS features such as the core memory allocator, scheduler and network stack) without having to perform shortcuts or hacks.

To provide such functionality into Unikraft, and during the GNU Make build process, ultimately an external directory is referenced as part of the L¹² global make variable. These directory repositories must contain several core files which are referenced by Unikraft's build system: a mandatory Makefile.uk file, a mandatory Config.uk file, and an optional exportsysms.uk file.

The Makefile.uk and Config.uk files are particularly interesting because they contain both metadata information about the library. These two files, independently along with a remote reference to the repository, provide enriching data to KraftKit and allow a user to select a remote repository to add to their unikernel project.

On aspect of the problem, however, is that the Makefile.uk, being essentially a Makefile, is prone to non-standard layout and composition. Across all of Unikraft's official external microlibrary repositories (which are prefixed with lib-), the contents of Makfile.uk varies. In Unikraft's earlier days, before KraftKit, there was no problem with the composition of these files (because it was make performing most of the heavy lifting). However, during the construction of KraftKit and today, in order to help discover, retrieve and plug-in these external microlibraries at the user's convenience, the composition of this file began to matter. Effort was made to at least clean up some the Makefile.uk to make it slightly easier to parse. This is accomplished with some tricks and by ultimately manually parsing these files.

Whilst it is possible that the make program itself can read it and work has been made to clean up these files in order to make them slightly easier to parse by KraftKit, it does not safely guarantee recognition, and therefore discovery and retrieval, whether by Unikraft contributors, maintainers and staff, or external developers who work privately on their own microlibrary implementations.

Another aspect of the problem is that an external microlibrary is essentially wrapper code around origin code. This in itself posits a versioning problem, making it both difficult to a). maintain the external microlibrary with regard to new version of the code it may wrap and b). makes it hard to discover new versions of said origin code. An additional goal of the Library.uk is to provide a list of possible versions.

Proposal

Towards better facilitating the discovery of microlibraries, the introduction of a third mandatory Library.uk is that is both GNU Make parsable and explicitly parsable by external tools such as KraftKit is underway. Library.uk has a simple structure and aims to provide all the necessary relevant metadata attributes as well as more in order to enrich the discovery process for a user:

name        := "lwip"
description := "A Lightweight TCP/IP stack."
homepage    := "https://savannah.nongnu.org/projects/lwip/"
license     := "BSD-3-Clause"
version     := 2.1.x sha256:544d82aa663165dcecb6626b00035f6d7fe0b8d39d3eec63a8635be8aa8f5d62 https://github.com/unikraft/fork-lwip/archive/STABLE-2_1_x.tar.gz
version     += 2.1.2 sha256:c6b10d0f5392d507f27a5a79bc43e706cb20df66bba86da56f557a53b68f2da8 https://github.com/unikraft/fork-lwip/archive/STABLE-2_1_2_RELEASE.tar.gz
version     += 2.1.x-unikraft sha256:478d5aab535444de2f71debb212b1d1574be69f1f889fcb2becf251db62e87f2 https://github.com/unikraft/fork-lwip/archive/UNIKRAFT-2_1_x.tar.gz

The Library.uk file has since been introduced into all external microlibrary repositories in Unikraft's GitHub organization where the above snippet represents Library.uk from lib-lwip. The snippet contains essential information which is parity to parsing the Config.uk and Makefile.uk which is its name and description. However, new features include its license in SDPX format, an a list of versions which are space-separated by its canonical representation exposed by the project, a checksum which is prefixed by its algorithm, and the remote source of the external microlibrary.

In order to achieve discovery of these files, the Manifest Package Manager needs to checkout only a single file during its discovery phase in order to retrieve enough metadata information in order to enrich the Manifest structure.

Background reading material

• Unikraft Build System Internals
• Packaging Unikernels with KraftKit
• RFC: Unikraft Microlibrary Versioning

Footnotes

1. https://github.com/unikraft/kraftkit/blob/staging/unikraft/core/args.go#L49 ↩

2. https://unikraft.org/docs/internals/build-system#makefile ↩