Topic: Defect report: [lex.key] and [lex.operators] contradict each


Author: Alberto Ganesh Barbati <AlbertoBarbati@libero.it>
Date: Sun, 5 Aug 2007 17:17:05 CST
Raw View
James Kanze ha scritto:
>> int main(){
>>    void * memory = operator new(1000);
>>    mytype* = new mytype;
>> }
>
>> It is worth noting that many of the suggested preprocessor
>> substitutions seem to assume that the programmer only uses the
>> second case.
>
> And also that he doesn't use any placement new.
>

In fact there is a technique that I used a couple of times to replace
new, that works for placement new also:

#define new SetDebugInfo(__FILE__, __LINE__) ? 0 : new

Sure, it doesn't work in every context where you can use "new", but it
never failed me in real-life code.

Just my two eurocent,

Ganesh

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: jdennett@acm.org (James Dennett)
Date: Mon, 6 Aug 2007 16:11:39 GMT
Raw View
James Kanze wrote:
> On Aug 6, 1:17 am, Alberto Ganesh Barbati <AlbertoBarb...@libero.it>
> wrote:
>> James Kanze ha scritto:
>
>>>> int main(){
>>>>    void * memory = operator new(1000);
>>>>    mytype* = new mytype;
>>>> }
>
>>>> It is worth noting that many of the suggested preprocessor
>>>> substitutions seem to assume that the programmer only uses the
>>>> second case.
>
>>> And also that he doesn't use any placement new.
>
>> In fact there is a technique that I used a couple of times to replace
>> new, that works for placement new also:
>
>> #define new SetDebugInfo(__FILE__, __LINE__) ? 0 : new
>
>> Sure, it doesn't work in every context where you can use "new", but it
>> never failed me in real-life code.
>
> Not even in multi-threaded code:-)?

In multi-threaded code, one might assume that SetDebugInfo
would use thread-local storage; such techniques are one
argument for including TLS in standard C++0x once thread
support is added.

-- James

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: jdennett@acm.org (James Dennett)
Date: Tue, 31 Jul 2007 17:32:10 GMT
Raw View
James Kanze wrote:
> [lex.key] and [lex.operators] give contradictory information as
> to the status of new and delete: in [lex.key], they are
> keywords, but in [lex.operators], they are listed as
> preprocessing-op-or-punc.  This affects the legality of programs
> such as:
>
>     #define new 0
>     int
>     main()
>     {
>         return new ;
>     }
>
> If new is a keyword, the above is a legal C++ program; if new is
> a preprocessing-op-or-punc, it is not.

When it comes to specifying types of tokens, I find the
standard to be very flawed; this is the tip of the iceberg.

There are, in some sense, two categories of token types
in C++: one for preprocessing tokens, and one for "proper"
tokens, with a conversion from pptokens to tokens at the
end of preprocessing.

Keywords are meaningless to the preprocessor; so far as
it is concerned they are just identifiers.  (I believe
James Kanze knows this.)  pp-op-or-punc tokens, however,
aren't identifiers, even if they look like them, which
is the crux of the example above.

My own interpretation of the standard was that during
preprocessing "new" is pp-op-or-punc (as keywords aren't
relevant), so the program above is ill-formed, and gives
a diagnostic

Error: Error at file {standard input}, line 1: #define directive
followed by token 'new' of type pp_op_or_punc.   #define must be
followed by an identifier (as per 16/1)

My bigger complain with the token specification is that
the conversions from pp-op-or-punc to actual tokens are
underspecified, or more generally that classification
into tokens (as opposed to pptokens) is underspecified.
However, most aspects of the later phases of translation
make no reference to the types of tokens.  In particular
I don't know anywhere where operators or punctuators
need to be correctly classified.

-- James

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: Greg Herlihy <greghe@pacbell.net>
Date: Tue, 31 Jul 2007 22:36:05 CST
Raw View
On 7/31/07 8:19 AM, in article
1185873775.132554.287510@l70g2000hse.googlegroups.com, "James Kanze"
<james.kanze@gmail.com> wrote:

> [lex.key] and [lex.operators] give contradictory information as
> to the status of new and delete: in [lex.key], they are
> keywords, but in [lex.operators], they are listed as
> preprocessing-op-or-punc.  This affects the legality of programs
> such as:
>
>     #define new 0
>     int
>     main()
>     {
>         return new ;
>     }
>
> If new is a keyword, the above is a legal C++ program; if new is
> a preprocessing-op-or-punc, it is not.

I think the answer is that "new" can be be either a keyword or an operator -
depending on the program. For example, the "new" in the above program is not
an operator or punctuator (which is the reason why the program compiles).

On the other hand, the "new" in the program below is an operator:

    int main()
    {
        return new ;
    }

and - as you observed - an operator new in this context is not legal C++
(and in fact this program fails to compile).

So whether the preprocessor treats "new" as an identifier or as an operator
depends upon the stage of preprocessing that the preprocessor has reached -
when the "new" token is encountered. In stages 1-6, a "new" token is an
identifier - but during stage 7, the preprocessor recognizes "new" as an
operator and replaces it with a suitable operator token (whatever that might
be).

Now, in the original program the "new" identifier is replaced (through macro
expansion) with the literal 0 before stage 7 is reached. So the preprocessor
encounters no "new" token in Stage 7, so there is no "new" operator in the
original program - it just looks like there is one.

Greg


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: AlbertoBarbati@libero.it (Alberto Ganesh Barbati)
Date: Wed, 1 Aug 2007 13:48:26 GMT
Raw View
Greg Herlihy ha scritto:
> So whether the preprocessor treats "new" as an identifier or as an operator
> depends upon the stage of preprocessing that the preprocessor has reached -
> when the "new" token is encountered. In stages 1-6, a "new" token is an
> identifier - but during stage 7, the preprocessor recognizes "new" as an
> operator and replaces it with a suitable operator token (whatever that might
> be).

You are missing the point. [lex-pptoken] provide a classification that
lasts for phases 3 through 6. Such classification, as further specified
in [lex.name] and [lex.operators], is contradictory, as it would require
"new" to be *both* an identifier and a preprocessing-op-or-punc. This
all happens, I repeat, in phases 3 through 6. What happens from phase 7
onwards is irrelevant here.

Ganesh

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: greghe@pacbell.net (Greg Herlihy)
Date: Thu, 2 Aug 2007 15:00:57 GMT
Raw View
On 8/1/07 6:48 AM, in article urYri.79797$U01.722622@twister1.libero.it,
"Alberto Ganesh Barbati" <AlbertoBarbati@libero.it> wrote:

> Greg Herlihy ha scritto:
>> So whether the preprocessor treats "new" as an identifier or as an operator
>> depends upon the stage of preprocessing that the preprocessor has reached -
>> when the "new" token is encountered. In stages 1-6, a "new" token is an
>> identifier - but during stage 7, the preprocessor recognizes "new" as an
>> operator and replaces it with a suitable operator token (whatever that might
>> be).
>
> You are missing the point. [lex-pptoken] provide a classification that
> lasts for phases 3 through 6. Such classification, as further specified
> in [lex.name] and [lex.operators], is contradictory, as it would require
> "new" to be *both* an identifier and a preprocessing-op-or-punc. This
> all happens, I repeat, in phases 3 through 6. What happens from phase 7
> onwards is irrelevant here.

Where's the contradiction? The token "new" -is- both an identifier and an
operator (since "new" appears on both the list of keywords and the list of
operators, after all). The fact that "new" is an operator matters only in
phase 7 - when the preprocessor replaces it with a token.

See items #189 and #369 on the C++ Active Language Issues list.

Greg

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: AlbertoBarbati@libero.it (Alberto Ganesh Barbati)
Date: Thu, 2 Aug 2007 21:19:40 GMT
Raw View
James Kanze ha scritto:
>
> (If someone wanted to ban defining keywords as macros entirely,
> I'd be all in favor as well.  But that's another issue, and a
> definite change.)
>

As a matter of fact, the standard already does it in [macro.names],
paragraph 2: "A translation unit shall not #define or #undef names
lexically identical to keywords."

Such paragraph is part of clause [library], which makes sense because if
you define a macro with the same name as a keyword and then you #include
a header file then you are bound to get rubbish. However, I found it
amusing that the requirement forbids the redefinition unconditionally,
that is even if the program puts the #define after the #include or even
if it doesn't include any library header at all.

Ganesh

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: AlbertoBarbati@libero.it (Alberto Ganesh Barbati)
Date: Fri, 3 Aug 2007 02:54:16 GMT
Raw View
Greg Herlihy ha scritto:
>
> Where's the contradiction? The token "new" -is- both an identifier and an
> operator (since "new" appears on both the list of keywords and the list of
> operators, after all). The fact that "new" is an operator matters only in
> phase 7 - when the preprocessor replaces it with a token.

In other places in the standard, where two grammar production rules can
parse the same input, disambiguation rules are always provided. In this
case the standard fails to provide one. I don't know any other place
where this happens, therefore I consider this ambiguity a defect.

>
> See items #189 and #369 on the C++ Active Language Issues list.
>

Thanks for the reference. Item #369 was precisely what I was looking
for. The "drafting" status seems to indicate that committee has already
agreed that this is indeed a defect.

Ganesh

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]





Author: Francis Glassborow <francis.glassborow@btinternet.com>
Date: Fri, 3 Aug 2007 09:37:21 CST
Raw View
>> <james.ka...@gmail.com> wrote:
>>> [lex.key] and [lex.operators] give contradictory information as
>>> to the status of new and delete: in [lex.key], they are
>>> keywords, but in [lex.operators], they are listed as
>>> preprocessing-op-or-punc.  This affects the legality of programs
>>> such as:
>
>>>     #define new 0
>>>     int
>>>     main()
>>>     {
>>>         return new ;
>>>     }
>
>>> If new is a keyword, the above is a legal C++ program; if new is
>>> a preprocessing-op-or-punc, it is not.

It has just crossed my mind that the problem may have arisen because new
and delete in C++ have two distinct meanings. We have them as operators
where they are user replaceable and overloadable. We also have them as
non-operators where they are neither replaceable nor overloadable.

int main(){
   void * memory = operator new(1000);
   mytype* = new mytype;
}

It is worth noting that many of the suggested preprocessor substitutions
seem to assume that the programmer only uses the second case.

--
Note that robinton.demon.co.uk addresses are no longer valid.

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]