REP: 132 Title: Incorporation of Changelogs into Package Source Tree Author: Tully Foote, Thibault Kruse, Mirza Shah, Dirk Thomas, William Woodall Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-Jan-2013 Post-History: ?-?-2013
Contents
This REP describes incorporating package changelogs (i.e. a list of changes made to the package in each release) as part of the package source tree rather than being maintained separately on the ROS wiki. This is to address shortcomings with maintaining a separate changelog list.
In ROS, package changelogs have traditionally been maintained as their own subpage within a package's ROS wiki page. These changelogs have had to be maintained separately from their codebases. This approach has had some drawbacks:
- Hard to Find - As changelogs are isolated from code, they can be difficult to find and correlate with particular versions of code. By making them part of the source tree, you can determine what has changed by inspecting the package itself.
- Lack of Maintenance - As changelogs are isolated from code and are optional, the motivation to maintain the lists is very weak. The result is that only a minority of package maintainers keep good changelogs. Especially due to packaging packages instead of stacks the number of changelogs which need to be maintained has increased. The hope is that when changelogs are kept with the package source, developers will be more likely to update the changelog file.
- Syntactic Inconsistency - Even among well-maintained changelogs there is an inconsistency in the layout of these lists across the ROS wiki. The lack of consistency makes it harder for both people and machines to parse the data in these logs. This is an opportunity to constrain the structural and syntactic format of the changelogs so that they are consistent across packages.
- Needed for Packaging - Virtually all package formats including .deb and .rpm require changelogs as part of the package. Scraping this information from the wiki is not feasible and therefore not performed. A changelog file with a specified parsable format can be automatically converted into these package formats.
The format of the changelog should have the following properties:
- In Source - The changelog is part of the package source rather than its own wiki page.
- Wiki Integration - The changelog is displayed on the wiki as in the past, but now pulled from SCM rather than being directly edited -- just like package headers.
- Syntax - Simple, easily parseable syntax. Markup languages like YAML are likely overkill.
- Package Compatible - Satisfies the required fields for .deb/.rpm changelogs.
- Non-redundant - Does not contain redundant information found elsewhere such as package.xml.
- Optional but Recommended - Should not be required, but highly recommended with tools such as bloom giving warnings.
- Generated Documentation - It is good practice to also include changelogs in generated static documentation. As we use Sphinx, a format that it can process would be ideal.
- GitHub Integration - Since ROS packages are being moved to GitHub (and GitHub practices are copied by other hosting solutions), it would be good to comply with GitHub display of files. [5]
- Flexibility - While we like standard syntax, developers should get as much freedom to do changelogs their own way as we can provide.
- Each changelog will be specified in ReStructuredText (RST) with additional constraints.
- Each package (i.e. every chunk of code that has a package.xml) may contain a single file used to list changelog entries. The recommended filename is CHANGELOG.rst.
- A changelog file ending with .rst has to be be a valid ReStructuredText [4] document. (Other formats may be supported in the future)
- The file must be in the same folder as the package.xml file, namely in the root folder of a package.
- The file shall contain sections for versions in descending order. Such changelog sections are distinguished from non-changelog sections (like TOC) via naming convention on the section title.
- The library will allow extraction of version numbers and respective text blocks from the document.
- bloom will emit a warning during package release if no changelog is found or if the changelog does not contain an entry for the released version. Future versions of bloom can force changelogs as a requirement.
For now, we will only support RST syntax for changelogs as this format is already widely used in places including ROS, GitHub, and Python. Other markup support may be added in the future if requested.
A document containing the changelogs must therefore be a valid RST document. Such a document will be processed by an RST parser (ignoring file inclusion directives, as those also do not work on GitHub).
From the RST specification [4] we will use one section without subsections for each changelog entry, and within that we will allow
- paragraphs
- transitions
- blank lines
- references
- comments
- bullet lists
- enumeration lists
- inline markup
- directives (may be invisible in processed changelog)
Inline markup transformation rules may just use the raw source when transforming the log entries for deb/rpm format. We explicitly forbid the following elements within version sections, as changelog entries should be well writable without those, and useful transformation into condensed deb/rpm formats would be difficult. For details on these elements see [4].
- definition lists
- field lists
- option Lists
- literal blocks
- line blocks
- block quotes
- doctest blocks
- tables
- subsections
A changelog entry is defined by a heading which contains a version number followed by an optional timestamp in parenthesis. The version number consists of three positive integers separate by single dots, i.e. 1.2.3 as specified in [8]. The timestamp must be parsable by the Python function dateutil.parser.parse. Futher constraints on the timestamp require it to contain at least a full date (YYYY-MM-DD, ISO 8601 format) and if hours are specified also the minutes (in order to keep the timestamp readable by humans). So the time as well as seconds and a timezone offset are optional. If not time and/or timezone is specified midnight
A changelog entry may not contain subsections. Version sections must follow each other in the document in descending version number order. However they do not need to be all in the same parent section nor on the same section level.
The following example shows different valid changelog entries:
1.2.3 (2013-01-23) ------------------ * Change 1 Optional Text . . . * Change n Optional Text 1.2.2 (2013-01-22 08:35) ------------------------ * Change 1 * Change 2 1.2.1 (2013-01-22 06:35 +0100) ------------------------------ * Change
But many variations are possible with the elements allowed. The following example uses a lot of variations to show what possibilities exist. The header (enclosed by ^ characters) and the sections with only MAJOR.MINOR are additional visual markup to make the changelog more readable / structure for humans to read and optional.
Likewise the section 0.1.27 (forthcoming) is only considered additional free form text as it does not match the pattern of a changelog entry. This is recommended practice to state changes which are already committed but not yet released.
^^^^^^^^^^^^^^^^^^^^^^^^^ Changelog for package foo ^^^^^^^^^^^^^^^^^^^^^^^^^ 0.1 === Free form text about this minor release. 0.1.27 (forthcoming) -------------------- * Great new feature 0.1.26 (2012-12-26) ------------------- * Utilizes caching to improve query performance (fix https://github.com/ros/ros_comm/pull/2) * Simplified API calls based on (https://github.com/ros/robot_model) Note that these changes are based on REP 192 * Fixed synchronization issue on startup .. not mentioning secret feature on purpose 0.1.25 (2012-11-25) ------------------- - Added thread safety - Replaced custom XML parser with `TinyXML <http://www.grinninglizard.com/tinyxml/>`_. - Fixed regression introduced in 0.1.22 ---- The library should now compile under Win32 0.1.0 (2012-10-01) ------------------ First public *stable* release 0.0 === 0.0.1 (2012-01-31) ------------------ 1. Initial release 2. Initial bugs
At the very least, the ROS wiki should link to the changelog in its source repository if publicly available. However, it is preferable if a custom wiki macro is written to pull the changelog from the repository and render it directly on the wiki.
The proposed format has the following properties that help to meet the design requirements:
- Changelogs will be in-source while remaining optional.
- Wiki integration is possible with simple solutions.
- Simple markup and very similar to how changelogs are typically written on the wiki and other open source projects.
- Can reuse RST parsers. See [6]
- Can be embedded in sphinx docs via include directive.
- The use of RST for markup allows us to automatically generated documentation without changes.
- Markup allows many different ways of writing changes as long as this can be transformed into brief format for deb/rpm content.
- When combined with the corresponding package.xml, enough information is provided to meet the full requirements of .deb and .rpm changelog formats (timestamp, package name, etc.).
- No redundant information from package.xml
Concerns have been discussed on ros-developers ([3]) and in the Buildsystem SIG ([7]).
- Can the timestamp of a changelog entry be optional?
While the toolchain could use the current time when a release is made the information is missing when reading the changelog in the source repository. Without a timestamp it is also not clear if the version has already been release (but the maintainer did not provided a timestamp) or is forthcoming (where the maintainer did not add an annotation for that).
- Can we allow free form text in the changelog entry headline?
This would make the decision if a headline is a valid changelog entry more difficult. On the other hand free form text could be either placed before a changelog entry or inside the changelog entry which should be sufficient to add additional information. Therefore the specification does not allow that.
How can a full changelog with multiple versions be generated?
The available information from the current package.xml and changelog file can be used to generate the changelog for the current version. Older changelog can not be generated since the information from the package.xml file at that point in time might have been different (i.e. other maintainer). Anyway a full changelog can be constructed based on the changelog of the previous changelog and the changelog of the current version if desired.
How to link to tickets/issues in bug tracker without having to give full URL?
Would be nice if GitHub did this for us on their website, but currently it does not.
- How much of RST should be supported?
- Outside section entries, no reason to forbid full RST
- Inside section entries, we only want to support things that can easily be transformed into deb/rpm format, though some loss of quality might be acceptable. Things to consider:
- Substitutions http://docutils.sourceforge.net/docs/ref/rst/directives.html#replacement-text
- References http://docutils.sourceforge.net/docs/ref/rst/directives.html#references
- Inclusion of other files (disabled on GitHub)
- Nested lists
- Definition lists (could also be used for version!)
- Directives, such as . note:: foo
REP now states some definitely allowed and forbidden elements. More may be allowed if users demand that and they can be easily supported.
- Other markup language support. See [5]
Not urgent, leave out for now.
- Name and placement
An early suggestion "ChangeList.txt" was rejected due to similarity to CMake "CMakeLists.txt".
The RST extension makes it possible for GitHub to render the file, and allows us to later possibly also support other markup flavors.
The package root is a common default way for such meta information, a "doc" subfolder is useful for static documentation. Sphinx does not allow to refer to documents outside the doc folder via toc-trees, but it does allow inclusion of files like this:
.. include:: ../CHANGELOG.rstSo we went for CHANGELOG.rst in root as ideal place. Alternatives are not planned to have a single location to check for the existance of a changelog.
- README.rst fallback: When users have a small package, it may be more convenient to put changelog into the README.rst. Could changelog tooling(bloom) fall back to try README.rst for changelog entries?
The prototype library could handle such complex README files. Though no technical reason is known that would prevent this, there was too much doubt on possible unknown problems with that approach, and user confusion over multiple alternatives, so for now it was decided to not go ahead with this.
- inline markup transformation rules: When creating deb/rpm changelogs from RST, a problem is how to deal with unicode and complex inline markup. Alternatives:
- Forbid all inline markup
- Support some inline markup nicely, forbid all that we do not transform
- Support some inline markup nicely, treat other markup as raw source
- Support all inline markup nicely
The actual transformations to happen are for other tools to decide. For now, we shall support some markup nicely (references), and treat other markup as raw source.
- Wiki display: We could display the changelog in the wiki as raw text, try to render the RST, display what goes into the deb, or merely link to the source file in its home repo.
- raw display is quickest for the users and easiest for us, maybe
- rendered display is nicer to the eye, allows following embedded references
- link to the source location is a bit worse for the users (navigating separate sites, but may be least effort)
- rosbuild stacks and package support?
For rosbuild stacks the CHANGELOG.rst file could be placed beside the stack.xml file. However, that won't be a priority to implement in the near future and might require contributions from the community.
For reference, here are the changelog formats for .deb [1] and .rpm [2] packages. Both package formats expect a changelog as prerequisite to creating a package.
package (version) distribution(s); urgency=urgency
[optional blank line(s), stripped]
* change details
more change details
[blank line(s), included in output of dpkg-parsechangelog]
* even more change details
[optional blank line(s), stripped]
-- maintainer name <email address>[two spaces] date
* Fri Jun 23 2006 Jesse Keating <[email protected]> - 0.6-4 - And fix the link syntax. * Fri Jun 23 2006 Jesse Keating <[email protected]> 0.6-4 - And fix the link syntax. * Fri Jun 23 2006 Jesse Keating <[email protected]> - 0.6-4 - And fix the link syntax. * Wed Jun 14 2003 Joe Packager <joe at gmail.com> - 1.0-2 - Added README file (#42).
A prototype implementation of a library that parses any RST document and extracts changelog entries as described here is provided as ongoing effort here [6].
| [1] | Debian Package Changelog Requirements (http://www.debian.org/doc/debian-policy/ch-source.html) |
| [2] | Fedora RPM Package Changelog Requirements (http://fedoraproject.org/wiki/Packaging:Guidelines#Changelogs) |
| [3] | Tully Foote Proposal for Stack Changelogs (9-03-2010) (http://code.ros.org/lurker/message/20100903.213420.d959fddc.en.html) |
| [4] | (1, 2, 3) reStructuredText (RST) (http://docutils.sourceforge.net/rst.html) |
| [5] | (1, 2) Github Markup languages (https://github.com/github/markup) |
| [6] | (1, 2) Prototype python script (https://github.com/tkruse/changelog_rst.git) |
| [7] | Buildsystem SIG discussion (https://groups.google.com/d/msg/ros-sig-buildsystem/L3nE9X0T2Jk/ML_1JsHLuF0J) |
| [8] | REP 127 Specification of package manifest format (https://github.com/ros-infrastructure/rep/blob/master/rep-0127.rst#version) |
This document has been placed in the public domain.