Proposal - New Doc Comments

[wekempf]

So, from the beginning, DocFX chose to do something I never understood. The project doesn't use the XML produced from the doc comments by the compiler, but instead chose to use Roslyn to parse the original doc comments straight from the source code. This means the source is parsed twice and the comments transformed twice, once to the XML that other tooling such as Visual Studio understands and a second time to Yaml which DocFX understands. Even if there's a valid reason why DocFX wants to work with Yaml exclusively here, it would have made a lot more sense to transform the XML to Yaml rather than parsing the source a second time.

Since DocFX is parsing the source, it's failed to meet parity with XML doc comments. There's tags and attributes that it's failed to understand, such as . There's also Sandcastle extensions that projects have relied on for years, such as extensions to the tag that allow you to reference code from other files, especially code within a real source file that gets compiled to verify the code, with the ability to just include the code within a #region in that file. Yes, I understand some work is being done to reach parity in many of these cases, but it probably would have been simpler to do if you'd stuck to the XML produced by the compiler.

Worse, though, is that DocFX actually extended these doc comments, but did so not with new XML tags/attributes but instead by adding support for Markdown within these comments. Swashbuckle/Swagger have done the same thing, and users of both projects are finding this to be a pain. Markdown expects certain indentation patterns to mean something, while XML ignores whitespace entirely. This means the two DO NOT MIX. At best lots of existing doc comments will have to be reformatted to make them compatible with DocFX, and that reformatting is non-obvious or expected. At worst, it may be that certain scenarios simply won't work due to the mix of Markdown and XML. It's not a pit of success scenario.

Another problem with mixing Markdown and XML? Older tooling doesn't understand the Markdown. Try to bold a word in the summary and DocFX will produce nice documentation, but Visual Studio will include the Markdown in the IntelliSense, a not very pleasing presentation.

So, what I'm proposing, is since DocFX is making a slight break from XML Doc Comments anyway, it should make a cleaner break. In fact, all tooling for .NET should move to a more modern concept here. Let's replace XML based doc comments with Markdown based doc comments.

/**
# Summary
Formats the specified resource string using @CultureInfo.CurrentCulture.

# Parameters
* format - A composite format string.
* args - An object array that contains zero or more objects to format.

# Returns
The formatted string.
**/
internal static string Format(string format, params object[] args)

I took this example from aspnetwebstack so that you can compare the two.

Unlike XML based doc comments, Markdown based doc comments are much easier to both read and write (by humans, that is). That's been a complaint about the XML doc comments from the very beginning... they make the source code hard to read. While including documentation with the source may still be considered "clutter" by some, the readability of Markdown does improve things a lot.

There's lots of details to be figured out here. For instance, in my example I chose to use multi-line comments rather than single line comments. That's a break from XML doc comments, but one I think makes it much easier to write the comments, and any friction you can remove from documentation creation, no matter how small, the better off we'll be. I chose to use /* as the trigger much as /// is used by traditional doc comments. There is a likelyhood this will cause conflicts with existing source code, so some other syntax, like perhaps /% would be safer, though less pleasant, choice. The @ references would need to be carefully defined, and langword references such as using @true should be supported. I've made up the concept of a "definition list" here by using a bulleted list with term and description divided by a hyphen. That's natural to me, but a parser would need to be aware of such a convention. I'm sure there's plenty of other details to work through as well.

This is great for moving us forward to a more modern set of tooling, but there still needs to be a way to interop with existing tooling. I'd suggest that DocFX should be able to produce traditional XML documentation from these comments, so that it can be used by Visual Studio for IntelliSense and by other tooling that understands the XML. Eventually all other tooling can be made to understand the Yaml produced by DocFX and this backwards compatibility can be dropped.

[InariTheFox]

I feel that DocFX shouldn't be taking a step backwards and having people need to rewrite the entirety of their XML documentation to suit the tool. I am relying on DocFX to generate html for a rather huge project with thousands of lines of XML documentation. Myself and others included would likely feel that this would be an insane task to have to go and manually change all of the documentation to suit the tool, instead of getting the tool to work right from the get-go.

[wilkinson-steve]

As other people have commented, this is in fact more a question for the VS team, whether they would consider supporting Markdown-based documentation, presumably alongside "legacy" XML documentation. I took a look at the F# implementation described below and it seems pretty nice, and I have to say I share a lot of the sentiments raised by @wekempf and others on the subject.

As a point of order, if the VS team were to start to support Markdown-based documentation, then it would be a logical step to include a "Convert all projects in this solution from XML comments to Markdown" facility - that doesn't look like it would be a particularly tricky thing to implement, so I don't think a manual conversion task (which I agree would be insane) should be a concern.

(One thing that has been a mystery to me is why Docfx uses YAML - I am sure one of the originators could comment, but it feels like just another layer of complexity/opaqueness in the implementation.)

[wekempf]

I guess I didn't explain myself very well, despite the lengthy post. :)

DocFX, by adding Markdown support within the XML Doc comments, has already caused some existing code to not work. What I'm proposing is a way out of that. I guess I only made clear half of the idea, however, which is my bad. So, let me elaborate.

Existing XML comments should be supported, but as is. DocFX should not add Markdown support inside such comments. It should also fully understand and support all of the XML tags and attributes that were understood by Sandcastle.

Then, as an alternative to the XML comments, new Markdown comments should be supported as well. This allows for the transition to a more modern way of writing documentation, while maintaining backwards compatibility.

The benefits to moving towards using Markdown comments:

1. Compatibility with modern tools, which are all choosing to use Markdown over other formats. An example here is Swashbuckle/Swagger.
2. Improve the readability of the source code (XML is machine readable, but not really human readable).
3. Remove friction in writing documentation. (Writing XML is tedious in comparison to writing Markdown.)

So, no, I'm not suggesting backwards compatibility to be broken. Quite the contrary, I want to fix the existing backwards compatibility issues, while still providing a way forward.

[kzu]

The challenge with the proposal, however, is that unless VS adopts the same markdown-based source for the XML intellisense file generation, you'll end up having to write the docs twice: once for the nice rendering markdown would give you, and once to drive VS intellisense with meaningful info.

[yufeih]

The project doesn't use the XML produced from the doc comments by the compiler, but instead chose to use Roslyn to parse the original doc comments straight from the source code. it would have made a lot more sense to transform the XML to Yaml rather than parsing the source a second time

This I agree. A custom source parser leads to feature gaps and causes performance problems.

Replace XML based doc comments with Markdown based doc comments

There are similar discussions in csharplang. It seems people generally likes markdown, but a completely new document syntax is C# is not very likely to happen soon.

The proposed syntax looks very similar to our design of markdown fragment, a slightly different overwrite markdown format. Markdown fragment is a separate markdown file that overrides some part of XML comments, it is not directly written in C# source code and work across languages.

[wilkinson-steve]

To the subject of full fidelity, it's my assumption that dotnet build uses Roslyn is used to compile C# code, so is there a good reason why the Roslyn team wouldn't expand its capabilities to include compilation of C# comments into metadata structures rather than (or as well as) XML output files?

[TheAngryByrd]

The F# community has taken to the markdown flavor pretty heavily. Ionide supports markdown documentation generation already

[yufeih]

It is unlikely that Docfx will embrace a radically different approach to documentation commenting unless Visual Studio adopts it. Therefore, this issue is being converted into a discussion for people to continue sharing their ideas.

[wekempf]

What Visual Studio support is wanted? There's not a tremendous amount of support provided by Visual Studio for XML document comments. The only real support one would want is the display of the documentation in tool tips... but that does NOT come from the comments but instead from the generated XML. It shouldn't be hard to generate that XML file from new comment formats. But since DocFX doesn't use the XML but rather reads the comments from source DocFX would have to understand it. If you read the XML file instead, a third party could create other ways of producing the XML from whatever source/format they desired, and I'd be happy to create a Markdown Comments solution in a separate project.

[KalleOlaviNiemitalo]

that does NOT come from the comments but instead from the generated XML

For types or members defined in the same solution, Visual Studio displays the documentation from the XML comment in the tooltip, even if the referenced project is configured not to generate an XML file and contains errors so that it cannot even be compiled (e.g. the XML comment is on a property whose type is not defined anywhere).

[wekempf]

Interesting. I had to verify this, as I recall in the past that I didn't get tooltip support for code in the solution regardless of whether or not the XML was generated. In any case, it doesn't change much of what I said. If DocFX used the XML, instead of parsing the source, then "a radically different approach to documentation commenting" could be made to work by third parties. This includes support inside Visual Studio via an extension.