[lex] Better specify whitespace characters #7359
Conversation
AlisdairM
force-pushed
the
define_whitespace_character
branch
from
8675494
to
912443c
Compare
There was a problem hiding this comment.
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p2348r3.pdf
has a paper touching the same area. It seems the paper was abandoned by its author.
AlisdairM
force-pushed
the
define_whitespace_character
branch
2 times, most recently
from
7abebf8
to
bbc7afd
Compare
I refer to this paper in my commit message, and how I leave room for it by deliberately not replacing textual "new-line" with U+000A LINE FEED, in order to leave room for that paper's specific treatment of new-lines and line-breaks. This PR is entirely editorial, or there is an error in the PR. I believe P2348 intends to make small but normative changes in our handling of whitespace. P2348 should be a much simpler paper to update if this PR lands. Paging @cor3ntin for further comments. |
| \begin{bnf} | ||
| \nontermdef{whitespace-character}\br | ||
| \unicode{0009}{character tabulation}\br | ||
| \textnormal{new-line}\br | ||
| \unicode{000b}{line tabulation}\br | ||
| \unicode{000c}{form feed}\br | ||
| \unicode{0020}{space}\br | ||
| \end{bnf} | ||
|
|
There was a problem hiding this comment.
Please move that to the very top of [lex.pptoken]. While the grammar is used here first, whitespace-characters are much more relevant for separating pptokens than for comments. Yes, we'll have a forward reference then. Hm. Not good, either.
Alternatively, a new subclause under [lex.char] would be appropriate: It's a kind of character (set), after all.
But having a section that contains just grammar is not good. Can we add some more meat? Maybe not editorially.
There was a problem hiding this comment.
My first draft (not submitted) did indeed add the grammar to the top of [lex.pptoken], but it felt even more wrong there.
I think a real solution, leaning more in the direction of P2348 but remaining editorial, would be to add a new level above [lex.comment] for "Whitespace" the covers both whitespace characters, comprising the existing [lex.comment], and a new subclause with the grammar and some text describing whitespace characters. This would be a sibling to [lex.char] and [lex.pptoken], similar to when we introduced [lex.char] for character sets.
There was a problem hiding this comment.
I have pushed a more substantial update that creates a new Whitespace clause for both whitespace tokens and comments. With appropriate cross-references this feels like a better approach, and closer to P2348. The main difference with that paper is that it defines a full grammar for whitespace, whereas my PR is limited to whitespace-character. The full whitespace grammar in P2348 is more detailed, and has a special treatment for line-breaks that my PR deliberately leaves room for.
I believe my changes remain purely editorial with no semantic/technical changes, although Jens correctly points out that introducing a new grammar term should be seen in Core. And while I would like to make this change for C++26, introducing the new Whitespace clause after our main clause re-org might make this C++29 material.
I have also retained my original branch that this replaces, should we wish to revert to the earlier approach.
source/lex.tex
Outdated
| The appearance of either of the characters \unicode{0027}{apostrophe} or \unicode{005c}{reverse solidus} or of | ||
| either of the character sequences \tcode{/*} or \tcode{//} in a | ||
| \grammarterm{q-char-sequence} or an \grammarterm{h-char-sequence} | ||
| is conditionally-supported with \impldef{meaning of \tcode{'}, \tcode{\textbackslash}, | ||
| \tcode{/*}, or \tcode{//} in a \grammarterm{q-char-sequence} or an | ||
| \grammarterm{h-char-sequence}} semantics, as is the appearance of the character | ||
| \tcode{"} in an \grammarterm{h-char-sequence}. | ||
| \unicode{0022}{quotation mark} in an \grammarterm{h-char-sequence}. |
There was a problem hiding this comment.
I don't think we want to use unicode notation for printable basic characters (that introduces inconsistencies even within that paragraph)
There was a problem hiding this comment.
I concur regarding this paragraph. Please revert.
There was a problem hiding this comment.
Reverted and force-pushed.
There was a problem hiding this comment.
Note that this change seems to be an accidental repetition of the same change in #7336 which is marked as approved. Should we revisit that PR too?
There was a problem hiding this comment.
On one hand, having lone special characters such as ' appear in text as (just) the glyph is a bit non-obvious, so the \unicode stuff helps for that; on the other hand, we don't make an attempt to present anything but glyphs for // or /*, which feels totally fine to me in isolation.
I'm not really sure which way we should go. @tkoeppe ?
AlisdairM
force-pushed
the
define_whitespace_character
branch
2 times, most recently
from
00f3471
to
9b755df
Compare
This commit defines a grammar term for _whitespace-character_ and uses it consistently where the plain text term whitespace character is used. A whitespace character is defined as one of the five characters that are mentioned in the text closest to provifing a defifinition. The unicode character name is (mostly) consistently used to name these characters, and for consistency, similar changes were made to name unicode characters rather than render specified characters in code font throughout [lex]. The one exception is backslash, which is retained as-is to avoid making more issues for P2348. Note that this commit is not a replacement for P2348, merely a clearer statement of the existing specification without any normative changes.
AlisdairM
force-pushed
the
define_whitespace_character
branch
from
9b755df
to
35ec5fe
Compare
This commit defines a grammar term for whitespace-character and uses it consistently where the plain text term whitespace character is used. A whitespace character is defined as one of the five characters that are mentioned in the text closest to provifing a defifinition. The unicode character name is (mostly) consistently used to name these characters, and for consistency, similar changes were made to name unicode characters rather than render specified characters in code font throughout [lex]. The one exception is backslash, which is retained as-is to avoid making more issues for P2348. Note that this commit is not a replacement for P2348, merely a clearer statement of the existing specification without any normative changes.