Topic: C++ test suite available from FSF


Author: pkt@lpi.liant.com (Scott Turner)
Date: Tue, 16 Nov 1993 18:30:14 GMT
Raw View
In article <KANZE.93Nov16133037@slsvhdt.us-es.sel.de>, kanze@us-es.sel.de (James Kanze) writes:

> On the other hand, it will be a standard, and not a reference manual.
> And I am confident that, while there is still much discussion and
> disagreement within the standards committee, the final standard that
> comes out will be at least as good as the C standard.
>
> Regretfully, this procedure takes time.  Perhaps more time, even, than
> the people involved realize.

The standards committee has produced some good results, adding needed
extensions, resolving matters left open by the ARM, and stating clearly
what errors a compiler should diagnose.   But for a potential
standard, the working paper's editorial precision is not much improved
over what was available 4 years ago when the committee started out.
At this point, the committee is still working on serious problems with
the definitions (or lack thereof) of terms like "object", "name", and
"definition".  Yet the plan is to have a draft standard a year from now,
ready for public review (correct me if the schedule has changed recently).

I expect the committee eventually to focus its attention on editorial
quality and precision, though perhaps not until after the first public
review wakes us up to what we've got.  In view of the committee's tendency
to defer editorial work, and the more complex structure of the C++ language
itself, "as good as the C standard" looks unachievable, ever.
--
Prescott K. Turner, Jr.
Liant Software Corp. (developers of LPI languages)
959 Concord St., Framingham, MA 01701 USA    (508) 872-8700
UUCP: uunet!lpi!pkt                          Internet: pkt@lpi.liant.com




Author: kanze@us-es.sel.de (James Kanze)
Date: 18 Nov 93 17:18:52
Raw View
In article <1993Nov16.183014.2554@lpi.liant.com> pkt@lpi.liant.com
(Scott Turner) writes:

|> In article <KANZE.93Nov16133037@slsvhdt.us-es.sel.de>, kanze@us-es.sel.de (James Kanze) writes:

|> > On the other hand, it will be a standard, and not a reference manual.
|> > And I am confident that, while there is still much discussion and
|> > disagreement within the standards committee, the final standard that
|> > comes out will be at least as good as the C standard.
|> >
|> > Regretfully, this procedure takes time.  Perhaps more time, even, than
|> > the people involved realize.

|> The standards committee has produced some good results, adding needed
|> extensions, resolving matters left open by the ARM, and stating clearly
|> what errors a compiler should diagnose.   But for a potential
|> standard, the working paper's editorial precision is not much improved
|> over what was available 4 years ago when the committee started out.
|> At this point, the committee is still working on serious problems with
|> the definitions (or lack thereof) of terms like "object", "name", and
|> "definition".  Yet the plan is to have a draft standard a year from now,
|> ready for public review (correct me if the schedule has changed recently).

Agreed so far.  Which is why I say that some people may not realize
how much time this will take.  (It is no secret that I consider the
current planning unrealistic in view of the work yet to be done.)

|> I expect the committee eventually to focus its attention on editorial
|> quality and precision, though perhaps not until after the first public
|> review wakes us up to what we've got.  In view of the committee's tendency
|> to defer editorial work, and the more complex structure of the C++ language
|> itself, "as good as the C standard" looks unachievable, ever.

I think most of the people involved are aware that there is much
editorial work yet to be done.  My feeling from recent meetings is
that there is a gradual shift of emphisis toward these problems,
although I still think it is generally underestimated just how much
work it will entail.  Since some of the people leading this move are
the same people who were in part responsible for the C standard, I
have high hopes concerning the quality of the final standard.  And I
think that there is enough awareness of the problem within the
committee that the problem will be addressed before the first public
review.

The only real worry I have is the schedule.  Software engineers are
reknown for underestimating schedules, and in my experience, the more
qualified the engineer in general, the more he tends to underestimate.
And some of the people in influential positions in the committee are
among the most qualified people I've ever met.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: pete@borland.com (Pete Becker)
Date: Fri, 5 Nov 1993 17:05:16 GMT
Raw View
In article <KANZE.93Nov5173114@slsvhdt.us-es.sel.de>,
James Kanze <kanze@us-es.sel.de> wrote:
>
>When we did our compiler evaluation about a year back, in addition to
>running a test suite through the compilers, we coded all of the
>examples in the ARM.  Of course, we didn't expect to find much with
>this; any compiler vendor would have at least done as much.  Ha.  Most
>(or was it all, I forget) of the compilers failed to compile some of
>the examples in the ARM correctly.
>
>One sometimes gets the feeling that the compiler vendors aren't even
>trying.

 Another possibility is that some of the examples in the ARM are
wrong.
 -- Pete




Author: ark@alice.att.com (Andrew Koenig)
Date: 3 Nov 93 23:28:59 GMT
Raw View
In article <matth.752290227@extro.ucc.su.OZ.AU> matth@extro.ucc.su.OZ.AU (Matthew Hannigan) writes:

> I would _really_ like to see the result of the testsuite on some
> commercial compilers.  Not that I think that they aren't better
> (at the moment), but how much better?

In my experience, the ability of a compiler to pass a test suite
is only slightly related to its usability.  It is part of the story,
but only part -- and a smaller part than one might think.
--
    --Andrew Koenig
      ark@research.att.com




Author: daniels@biles.com (Brad Daniels)
Date: Fri, 5 Nov 1993 20:13:30 GMT
Raw View
In article <1993Nov5.154236.1437@kth.se>,
Jon W   tte <d88-jwa@mumrik.nada.kth.se> wrote:
[ Attributions missing, but I believe >>> is nagle@netcom.com, and >> is
  rfg@netcom.com...]
>>>Nobody can produce an even marginally bug-free C++ compiler.
>>I know of at least one counter-example.
>
>Name it. It ain't g++.

While i'm not sure which compiler RFG was referring to, I am aware of one
surprisingly-close-to-bug-free compiler, and that is DEC C++.  Even things
like nested classes (inside other classes or functions), exceptions, and
templates work.  I haven't pounded on exceptions too heavily, but the template
and nesting support is much better than I've seen elsewhere.  I've had to
rewrite many sections of my code when moving from DEC C++ to HP C++ because of
bugs in the HP compiler, while I've had almost no bug problems moving from
HP C++ to DEC C++.  Most if not all of the problems I've had with DEC C++
come from very deliberate decisions made by DEC as to how the language
should be interpreted, which differ from my interpretation or an emerging
interpretation of the ARM and working document.  A particular nuisance is
DEC's approach to template instantiation (I have to tell it where to put
template code), but a real advantage of DEC's approach is that links are
hugely faster than Cfront links with templates.

I found g++ 2.4.5 to be too buggy to be usable for my application.  I have
hopes that future versions will be better, though, since gcc really is an
outstanding C compiler.

- Brad
----------------------------------------------------------------------
+ Brad Daniels   | "Let others praise ancient times;  +
+ Biles and Associates  |  I am glad I was born in these."   +
+ These are my views, not B&A's |   - Ovid(43 B.C - 17 A.D)    +
----------------------------------------------------------------------




Author: jbuck@synopsys.com (Joe Buck)
Date: Sat, 6 Nov 1993 01:51:30 GMT
Raw View
James Kanze <kanze@us-es.sel.de> wrote:
>>One sometimes gets the feeling that the compiler vendors aren't even
>>trying.

pete@borland.com (Pete Becker) writes:
> Another possibility is that some of the examples in the ARM are
>wrong.

Yes.  For instance, consider the example on page 326:

f(int);
f(int ...);
f(int, char* ...);

void g()
{
 f(1);  // error, ambiguous: f(int) or f(int ...)
 ....
}

But on the previous page we read as the lowest possible priority match:
"[5] Match with ellipsis: Sequences that involve matches with the ellipsis
are worse than all others".

Note that the comment in the example program claims that an exact match
is ambiguous with a match that has just been declared "worse than all
others".  If this isn't an error in the ARM, someone please explain it to
me.

Strictly speaking, the example is in the commentary, not the standard.

[ Ron, you might remember that this was a case you, Mike Tiemann and I
kicked around a while back.  ]



--
-- Joe Buck jbuck@synopsys.com
Posting from but not speaking for Synopsys, Inc.
Formerly jbuck@<various-hosts>.eecs.berkeley.edu




Author: joe@bftsi0.UUCP (Joe Foster of Borg)
Date: 6 Nov 93 22:13:37 GMT
Raw View
In article <27078@alice.att.com>, ark@alice.att.com (Andrew Koenig) writes:
> In article <matth.752290227@extro.ucc.su.OZ.AU> matth@extro.ucc.su.OZ.AU (Matthew Hannigan) writes:

: > I would _really_ like to see the result of the testsuite on some
: > commercial compilers.  Not that I think that they aren't better
: > (at the moment), but how much better?

: In my experience, the ability of a compiler to pass a test suite
: is only slightly related to its usability.  It is part of the story,
: but only part -- and a smaller part than one might think.

Hee hee. This reminds me of some of the first Ada compilers.
They'd compile the programs in the validation suite just fine,
but they wouldn't correctly compile anything else! The only
sure way to tell is to try them out with your own programs,
but this can be pretty time-consuming. Oh, well.

--
Joe Foster (joe@bftsi0.uucp)
WARNING: I cannot be held responsible for the above        They're   coming  to
because  my cats have  apparently  learned to type.        take me away, ha ha!




Author: david.binderman@pmsr.philips.co.uk (Dave Binderman)
Date: Fri, 5 Nov 1993 11:45:13 +0000
Raw View
Joe Buck (jbuck@synopsys.com) wrote:
: kanze@us-es.sel.de (James Kanze) writes:
: >We actually ran a number of commercial compilers (and g++) through a
: >test suite some time back.  Although I don't remember the exact
: >figures any longer, basically, g++ had roughly 10 times as many errors
: >as the commercial products (which were all more or less equivalent).

: It would be interesting if you re-ran the tests with a more recent version
: of g++.  From private correspondence, it appears that James has no
: experience with g++ versions newer than 2.2.2.

One of the reasons why James has no experience of GNU G++ after 2.2.2
is that the later versions (2.3.* + Reno) didn't do as well as 2.2.2 in
the test suite.

Although to be fair, the 2.2.2 James describes was a modified ( mostly
by me ) version of GNU G++ 2.2.2. The original G++ 2.2.2 didn't do
as well as only 10 times worse.

Given the "three steps forward, two steps back" nature of the g++
compiler, I'm not sure that a post 2.3.* version of G++ would score much
better.

I'd like a g++ that just had the "one step forward" nature. Save a lot
of time, if g++ users didn't have to re-debug previously working
features.

I'm sure a large test suite will help here. Certainly 1000 test
cases cannot be regarded as large. That is only about 50 days of
gnu.g++.bug Didn't Ada Verification Test have 20,000 tests ?

Given that the bug report mountain for g++ was 10 times larger than an
average C++ compiler, and the evidence that the software process used
by Cygnus/FSF/gnu.g++.bug is erratic, to say the least, then I find
it hard to recommend G++ as a useable commercial compiler.

Leaving aside that I was better able than FSF/Cygnus at making G++
more ARM conformant.

: Also, the following flags should be given to g++: "-ansi
: -pedantic -fansi-overloading".

Expect bugs in -fansi-overloading., since it is new code.

: In gcc, trigraphs are not implemented by default: I think "-ansi" is
: required.  g++ may be the same way

yes, or use -trigraphs.

: As someone who uses both g++ and cfront a lot: the gap has narrowed
: considerably because cfront hasn't been having its bugs fixed at nearly
: the same rate.

So maybe g++ is now only 5 times more buggy than cfront ?

> (BOTH compilers are too buggy for my taste).

Agreed, try Lucid, its better than both g++ and cfront

Regards

--
David C Binderman BSc MSc     Academy Programming Services   +44 293 544364
                  dcb@pmsr.philips.co.uk   +44 293 654149
5 years C++ / 8 years C / 12 years UNIX / telecomms / compilers / real time




Author: kanze@us-es.sel.de (James Kanze)
Date: 8 Nov 93 13:50:25
Raw View
In article <1993Nov6.015130.29198@Synopsys.Com> jbuck@synopsys.com
(Joe Buck) writes:

|> James Kanze <kanze@us-es.sel.de> wrote:
|> >>One sometimes gets the feeling that the compiler vendors aren't even
|> >>trying.

|> pete@borland.com (Pete Becker) writes:
|> > Another possibility is that some of the examples in the ARM are
|> >wrong.

I agree, you cannot exclude this possibility.

|> Yes.  For instance, consider the example on page 326:

|> f(int);
|> f(int ...);
|> f(int, char* ...);

|> void g()
|> {
|>  f(1);  // error, ambiguous: f(int) or f(int ...)
|>  ....
|> }

|> But on the previous page we read as the lowest possible priority match:
|> "[5] Match with ellipsis: Sequences that involve matches with the ellipsis
|> are worse than all others".

|> Note that the comment in the example program claims that an exact match
|> is ambiguous with a match that has just been declared "worse than all
|> others".  If this isn't an error in the ARM, someone please explain it to
|> me.

The sentence quoted is talking about matching an argument with the
ellipsis.  In the above example, *no* argument is being matched to the
ellipsis, so this rule simply doesn't apply.

|> Strictly speaking, the example is in the commentary, not the standard.

According to the C standard, at least, *all* examples are to be
considered commentary, even if they are not presented as such:-).

In conjunction with the ARM, however, I think some common sense is in
order.  If the wording in the reference part can be interpreted two
ways, but the commentary makes it clear which way the authors had in
mind, the it would seem perverse to interpret it the other way.
(Obviously, this is *not* true for an ISO standard, but this is
precisely the difference between a reference manual and a standard.)

Before everyone starts flaming, I have just reread the passage in the
ARM in context.  I recognize that the wording is somewhat ambiguous;
in fact, I am somewhat surprised that the interpretation Joe Buck
gives never occurred to me.  But in conjunction with the examples, I'm
fairly sure what the *intent* was.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: kanze@us-es.sel.de (James Kanze)
Date: 8 Nov 93 13:56:08
Raw View
In article <1993Nov5.154236.1437@kth.se> d88-jwa@mumrik.nada.kth.se
(Jon W   tte) writes:

|> >>A test suite is somewhat simpler to produce - you read the ARM,
|> >>and code a special case for each of the interesting things you
|> >>find in there...

|> >(Yea.  Right.  That's all there is to it. :-)  :-)
|> >I think this is essentially what the folks at AT&T/USL did.  The result of
|> >course was that they produced a perfectly bug-free C++ compiler... NOT.


|> Uh... we said "test suite" here, not "C++ compiler" A test suite
|> that is made by "only" following the above statement will work
|> better than not having any test suite, and writing such a test
|> suite is simpler than writing a bug-free C++ compiler. It seems
|> to me you disagree? Do you mean it's simpler to write a bug-free
|> C++ compiler than a reasonable test suite?

I think what Ron was trying to say is that if you have a reasonable
test suite, then (supposing you are into writing compilers) you will
also have a reasonably bug-free compiler.

Either that, or you know your compiler is shit, but the marketing
people say: "ship," and they have the final word.  (Before anyone gets
the wrong idea, I don't think that this is the case with either CFront
or g++.)
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: kanze@us-es.sel.de (James Kanze)
Date: 8 Nov 93 13:58:22
Raw View
In article <27078@alice.att.com> ark@alice.att.com (Andrew Koenig)
writes:

|> In article <matth.752290227@extro.ucc.su.OZ.AU> matth@extro.ucc.su.OZ.AU (Matthew Hannigan) writes:

|> > I would _really_ like to see the result of the testsuite on some
|> > commercial compilers.  Not that I think that they aren't better
|> > (at the moment), but how much better?

|> In my experience, the ability of a compiler to pass a test suite
|> is only slightly related to its usability.  It is part of the story,
|> but only part -- and a smaller part than one might think.

I concur completely.  On the other hand, the inability of a compiler
to pass a test suite might be a strong indication of its unusability.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: pkt@lpi.liant.com (Scott Turner)
Date: Mon, 8 Nov 1993 20:08:03 GMT
Raw View
In article <1993Nov6.015130.29198@Synopsys.Com>, jbuck@synopsys.com (Joe Buck) writes:
> pete@borland.com (Pete Becker) writes:
> > Another possibility is that some of the examples in the ARM are
> >wrong.
>
> Yes.  For instance, consider the example on page 326:
>
> f(int);
> f(int ...);
> f(int, char* ...);
>
> void g()
> {
>  f(1);  // error, ambiguous: f(int) or f(int ...)
>  ....
> }
>
> But on the previous page we read as the lowest possible priority match:
> "[5] Match with ellipsis: Sequences that involve matches with the ellipsis
> are worse than all others".
>
> Note that the comment in the example program claims that an exact match
> is ambiguous with a match that has just been declared "worse than all
> others".  If this isn't an error in the ARM, someone please explain it to
> me.

It's not covered by [5] that you quote, because those priorities are for
matching individual arguments.  The argument to f(1) matches the 'int'
parameter.  No argument matches the ellipsis.

As an example to which [5] applies,
     f(1,2);     // matches f(int, char* ...)
because in f(int ...) the argument 2 involves a "Match with ellipsis".

--
Prescott K. Turner, Jr.
Liant Software Corp. (developers of LPI languages)
959 Concord St., Framingham, MA 01701 USA    (508) 872-8700
UUCP: uunet!lpi!pkt                          Internet: pkt@lpi.liant.com




Author: nagle@netcom.com (John Nagle)
Date: Mon, 8 Nov 1993 20:41:44 GMT
Raw View
daniels@biles.com (Brad Daniels) writes:
>In article <1993Nov5.154236.1437@kth.se>,
>Jon W   tte <d88-jwa@mumrik.nada.kth.se> wrote:
>[ Attributions missing, but I believe >>> is nagle@netcom.com, and >> is
>  rfg@netcom.com...]
>>>>Nobody can produce an even marginally bug-free C++ compiler.
>>>I know of at least one counter-example.

       Not me.

     John Nagle




Author: kanze@us-es.sel.de (James Kanze)
Date: 9 Nov 93 11:50:22
Raw View
In article <1993Nov8.200803.23733@lpi.liant.com> pkt@lpi.liant.com
(Scott Turner) writes:

|> As an example to which [5] applies,
|>      f(1,2);     // matches f(int, char* ...)
|> because in f(int ...) the argument 2 involves a "Match with ellipsis".

Don't you mean 'f( 1 , 0 )'?  There is no conversion from 2 -> char*.

So we have:

 f( 1 , 0 ) ; // matches f( int , char* , ... )
 f( 1 , 2 ) ; // matches f( int , ... )

Interestingly, neither is ambiguous.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: hendrik@vedge.com (Hendrik Boom)
Date: Tue, 09 Nov 1993 14:22:54 GMT
Raw View
kanze@us-es.sel.de (James Kanze) writes:
: In article <1993Nov4.203217.18579@Synopsys.Com> jbuck@synopsys.com
: (Joe Buck) writes:
:
: |> kanze@us-es.sel.de (James Kanze) writes:
...
...
: |> As someone who uses both g++ and cfront a lot: the gap has narrowed
: |> considerably because cfront hasn't been having its bugs fixed at nearly
: |> the same rate.  (BOTH compilers are too buggy for my taste).
:
: Agreed.  Except that from my experience, you could replace 'both' with
: 'all' in the last sentance.

Really? *all*?  I'd believe *most*. from personal experisnce.

But have you tried IBM's xlC on their workstations?  Or their CSET++ on OS/2?
These are two ports of essentially the same compiler, and seem pretty
good to me.

It seems to me that the hard part of implementing C++ is figuring out
what the language is.

: --
: James Kanze                             email: kanze@us-es.sel.de
: GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
: Conseils en informatique industrielle --
:                    -- Beratung in industrieller Datenverarbeitung
--
-------------------------------------------------------
Try one or more of the following addresses to reply.
at work: hendrik@vedge.com,  iros1!vedge!hendrik
at home: uunet!ozrout!topoi!hendrik




Author: pkt@lpi.liant.com (Scott Turner)
Date: Tue, 9 Nov 1993 18:09:49 GMT
Raw View
In article <KANZE.93Nov9115022@slsvhdt.us-es.sel.de>, kanze@us-es.sel.de (James Kanze) writes:
> In article <1993Nov8.200803.23733@lpi.liant.com> pkt@lpi.liant.com
> (Scott Turner) writes:
>
> |> As an example to which [5] applies,
> |>      f(1,2);     // matches f(int, char* ...)
> |> because in f(int ...) the argument 2 involves a "Match with ellipsis".
>
> Don't you mean 'f( 1 , 0 )'?  There is no conversion from 2 -> char*.

I erred.  Thanks for the correction.
Actually, what I had in mind was more like
      f(1,2);     // matches f(int, char ...)
--
Prescott K. Turner, Jr.
Liant Software Corp. (developers of LPI languages)
959 Concord St., Framingham, MA 01701 USA    (508) 872-8700
UUCP: uunet!lpi!pkt                          Internet: pkt@lpi.liant.com




Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Wed, 10 Nov 1993 08:28:54 GMT
Raw View
In article <KANZE.93Nov8135608@slsvhdt.us-es.sel.de> kanze@us-es.sel.de (James Kanze) writes:
>In article <1993Nov5.154236.1437@kth.se> d88-jwa@mumrik.nada.kth.se
>(Jon W   tte) writes:
>
>|> >>A test suite is somewhat simpler to produce - you read the ARM,
>|> >>and code a special case for each of the interesting things you
>|> >>find in there...
>
>|> >(Yea.  Right.  That's all there is to it. :-)  :-)
>|> >I think this is essentially what the folks at AT&T/USL did.  The result of
>|> >course was that they produced a perfectly bug-free C++ compiler... NOT.
>
>
>|> Uh... we said "test suite" here, not "C++ compiler" A test suite
>|> that is made by "only" following the above statement will work
>|> better than not having any test suite, and writing such a test
>|> suite is simpler than writing a bug-free C++ compiler. It seems
>|> to me you disagree? Do you mean it's simpler to write a bug-free
>|> C++ compiler than a reasonable test suite?
>
>I think what Ron was trying to say is...

Ahem...  (The horse's mouth will now speak. :-)

What I really was trying to say was simply that I disagree with the
original poster.

He said you could produce a good test suite by simply reading the ARM,
and coding a test case for everything in there.  That is simply wrong.
The best way to write a real ball-buster test suite is to read the ARM,
think about it, and then code up a test case for everything which you
*DON'T* find in there.  (Speaking from personal experience, this is THE
single most sure fire way to make a compiler, *any* compiler, crash.
Generally, the implementors tend to do most of the stuff in the ARM at
least well enough to avoid crashes.  As soon as you set foot outside of
the confined circle which is illuminated by the ARM however, you almost
immediately enter regions of compiler code which have been less well
considered, and typically less well implemented.)

Regarding the question of which is harder to write, a test suite or a
compiler, a compiler is definitely easier.  If you are lucky, and if
the language is well defined, writing a compiler is a fairly finite
task.  There is an endpoint.  There is however, so such thing as having
too many compiler tests.  (That's because implementors, in general, have
historically shown truly astonishing degrees of creativity in developing
new bugs... even in areas where one might have thought that none were
possible. (1/2 :-)

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------




Author: d88-jwa@dront.nada.kth.se (d88-jwa)
Date: Wed, 10 Nov 1993 16:17:25 GMT
Raw View
In <rfgCG9ow6.9uD@netcom.com> rfg@netcom.com (Ronald F. Guilmette) writes:

>What I really was trying to say was simply that I disagree with the
>original poster.

That was me.

>He said you could produce a good test suite by simply reading the ARM,
>and coding a test case for everything in there.  That is simply wrong.
>The best way to write a real ball-buster test suite is to read the ARM,

I said a "reasonable" test suite. I leave ball-busters for the pros :-)

>Generally, the implementors tend to do most of the stuff in the ARM at
>least well enough to avoid crashes.  As soon as you set foot outside of

Well, except for the Zortech C++ (now SC++) case...

>immediately enter regions of compiler code which have been less well
>considered, and typically less well implemented.)

And are typically more ambiguously defined, if at all.

>Regarding the question of which is harder to write, a test suite or a
>compiler, a compiler is definitely easier.  If you are lucky, and if

It's not "a test suite or a compiler," it's a "C++ test suite or
a C++ compiler."

>the language is well defined, writing a compiler is a fairly finite
>task.  There is an endpoint.  There is however, so such thing as having

Yes, but that's not the case with a C++ compiler. The language is just
too big, over-featured and clunky. I want a good Dylan implementation.

--
 -- Jon W{tte, h+@nada.kth.se, Mac Hacker Deluxe --

  Clearly, most humans are not rational beings; they are rationalizing beings.
                     -- Mel Walker




Author: peter@ncrpda.curtin.edu.au (Peter N Lewis)
Date: 12 Nov 1993 09:45:31 +0800
Raw View
rfg@netcom.com (Ronald F. Guilmette) writes:

>think about it, and then code up a test case for everything which you
>*DON'T* find in there.  (Speaking from personal experience, this is THE
>single most sure fire way to make a compiler, *any* compiler, crash.

I disagree - the surest way to make a compiler crash (IME) is to feed it
computer generated code.  Write a simple compiler that compiles some high
level code into the compiler's source file, and then try to compile that.

>Generally, the implementors tend to do most of the stuff in the ARM at
>least well enough to avoid crashes.  As soon as you set foot outside of

Yeah, but we're talking about Symantec right? :-)

>Regarding the question of which is harder to write, a test suite or a
>compiler, a compiler is definitely easier.  If you are lucky, and if

Well, I know which one I'd rather right!
   Peter.
--
_______________________________________________________________________
Peter N Lewis <peter.lewis@info.curtin.edu.au>       Ph: +61 9 368 2055




Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Mon, 15 Nov 1993 09:44:26 GMT
Raw View
In article <KANZE.93Nov5173114@slsvhdt.us-es.sel.de> kanze@us-es.sel.de (James Kanze) writes:
>In article <rfgCG07xo.8rq@netcom.com> rfg@netcom.com (Ronald F.
>Guilmette) writes:
>
>|> In article <1993Nov1.111346.27775@kth.se> d88-jwa@mumrik.nada.kth.se (Jon W   tte) writes:
>
>|> >Nobody can produce an even marginally bug-free C++ compiler.
>
>|> I know of at least one counter-example.
>
>Would you care to say who?

No.  I wouldn't.

Several of my clients and customers are competing compiler vendors.

(My moma didn't raise any sons stupid enough to piss off their clients &
customers by recommending someone else's products in a public forum.)

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------




Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Mon, 15 Nov 1993 10:05:34 GMT
Raw View
In article <CG1C6J.FBJ@biles.com> daniels@biles.com (Brad Daniels) writes:
>
>While i'm not sure which compiler RFG was referring to, I am aware of one
>surprisingly-close-to-bug-free compiler, and that is DEC C++.  Even things
>like nested classes (inside other classes or functions), exceptions, and
>templates work.

As it happens, I recently had occasion to run some tests on the DEC C and C++
compilers for the Alpha.  I'm sure that these compilers have earned their
good reputations, but (needless to say) I *did* find a number of bugs in
both compilers.  I hope to provide a sampling of those here later on.
For the moment however, I'll just mention one obvious bug in the C compiler.

In C, the following is a valid expression:

  &"xyz"

The DEC C compiler (on Alpha/OSF-1.3) disagrees however.

(I'll post some descriptions of C++ specific bugs in the DEC C++ compiler
here later on... if no one objects that is.)

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------




Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Mon, 15 Nov 1993 10:25:24 GMT
Raw View
In article <KANZE.93Nov8135025@slsvhdt.us-es.sel.de> kanze@us-es.sel.de (James Kanze) writes:
>In article <1993Nov6.015130.29198@Synopsys.Com> jbuck@synopsys.com
>|> Yes.  For instance, consider the example on page 326:
>
>|> f(int);
>|> f(int ...);
>|> f(int, char* ...);
>
>|> void g()
>|> {
>|>  f(1);  // error, ambiguous: f(int) or f(int ...)
>|>  ....
>|> }
>
>|> But on the previous page we read as the lowest possible priority match:
>|> "[5] Match with ellipsis: Sequences that involve matches with the ellipsis
>|> are worse than all others".
>
>|> Note that the comment in the example program claims that an exact match
>|> is ambiguous with a match that has just been declared "worse than all
>|> others".  If this isn't an error in the ARM, someone please explain it to
>|> me.
>
>The sentence quoted is talking about matching an argument with the
>ellipsis.  In the above example, *no* argument is being matched to the
>ellipsis, so this rule simply doesn't apply.

The sentence quoted is just talking about "matching" with an ellipsis.  It
is less than clear if a set of arguments consisting of ZERO arguments "matches"
an ellipsis, but I think we would have to conclude that it does.  Thus,
"matching" is occuring.

>In conjunction with the ARM, however, I think some common sense is in
>order.

One man's common sense is another man's irrationality.  (That's why we
write language standards and also civil and criminal laws.)

>If the wording in the reference part can be interpreted two
>ways, but the commentary makes it clear which way the authors had in
>mind, the it would seem perverse to interpret it the other way.
>(Obviously, this is *not* true for an ISO standard, but this is
>precisely the difference between a reference manual and a standard.)
>
>Before everyone starts flaming, I have just reread the passage in the
>ARM in context.  I recognize that the wording is somewhat ambiguous;

Good.  That's a start.

>in fact, I am somewhat surprised that the interpretation Joe Buck
>gives never occurred to me.  But in conjunction with the examples, I'm
>fairly sure what the *intent* was.

I still cling to some hope that in the final approved international C++
language standard, it will not be necessary to infer intents... either
from non-normative commentary material, or from seances, or from other
sources.  Contrary to popular belief, the English language is an extra-
ordinarily precise tool (when in the proper hands) and I hope that all
of its potential for precision will be brought to bear upon this difficult
problem of accurately describing the C++ language.

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------




Author: kanze@us-es.sel.de (James Kanze)
Date: 16 Nov 93 13:30:37
Raw View
In article <rfgCGJ3MC.1nq@netcom.com> rfg@netcom.com (Ronald F.
Guilmette) writes:

|> In article <KANZE.93Nov8135025@slsvhdt.us-es.sel.de> kanze@us-es.sel.de (James Kanze) writes:

|> >In conjunction with the ARM, however, I think some common sense is in
|> >order.

|> One man's common sense is another man's irrationality.  (That's why we
|> write language standards and also civil and criminal laws.)

This is regretfully true.  And as Ron points out, this is why we need
a *standard*.  As reference manuals go, I think that the ARM is pretty
good.  Obviously, it could be better, but that is true of all text.
But the ARM is *not* a standard, and cannot be read with the same
rigor as a standard.

|> >If the wording in the reference part can be interpreted two
|> >ways, but the commentary makes it clear which way the authors had in
|> >mind, the it would seem perverse to interpret it the other way.
|> >(Obviously, this is *not* true for an ISO standard, but this is
|> >precisely the difference between a reference manual and a standard.)

|> >Before everyone starts flaming, I have just reread the passage in the
|> >ARM in context.  I recognize that the wording is somewhat ambiguous;

|> Good.  That's a start.

|> >in fact, I am somewhat surprised that the interpretation Joe Buck
|> >gives never occurred to me.  But in conjunction with the examples, I'm
|> >fairly sure what the *intent* was.

|> I still cling to some hope that in the final approved international C++
|> language standard, it will not be necessary to infer intents... either
|> from non-normative commentary material, or from seances, or from other
|> sources.  Contrary to popular belief, the English language is an extra-
|> ordinarily precise tool (when in the proper hands) and I hope that all
|> of its potential for precision will be brought to bear upon this difficult
|> problem of accurately describing the C++ language.

Agreed 100%.

It is almost certain that there will be some ambiguity in the final
standard.  When the C standard was approved, it was commended by ANSI
(or ISO?) for its preciseness.  But there have still been defect
reports and requests for interpretation.  It would be unrealistic to
suppose that the C++ standard will be different.

On the other hand, it will be a standard, and not a reference manual.
And I am confident that, while there is still much discussion and
disagreement within the standards committee, the final standard that
comes out will be at least as good as the C standard.

Regretfully, this procedure takes time.  Perhaps more time, even, than
the people involved realize.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: ark@alice.att.com (Andrew Koenig)
Date: 2 Nov 93 02:32:22 GMT
Raw View
In article <1993Nov1.111346.27775@kth.se> d88-jwa@mumrik.nada.kth.se (Jon Wtte) writes:

> Nobody can produce an even marginally bug-free C++ compiler.
> It's an impossibility inherent in the language, and is the reason
> why C++ will fail in the end (i e when the industry decides that
> there are better alternatives)

Well, I should hope so!

No one should be so presumptuous as to believe that they know the
best possible solutions to engineering problems for all time.

Meanwhile, most of us will do what we can with what we have.
--
    --Andrew Koenig
      ark@research.att.com




Author: kanze@us-es.sel.de (James Kanze)
Date: 4 Nov 93 17:12:44
Raw View
In article <matth.752290227@extro.ucc.su.OZ.AU>
matth@extro.ucc.su.OZ.AU (Matthew Hannigan) writes:

|> I would _really_ like to see the result of the testsuite on some
|> commercial compilers.  Not that I think that they aren't better
|> (at the moment), but how much better?

We actually ran a number of commercial compilers (and g++) through a
test suite some time back.  Although I don't remember the exact
figures any longer, basically, g++ had roughly 10 times as many errors
as the commercial products (which were all more or less equivalent).

But any such figures must be taken with a grain of salt.  For example,
on the first pass, g++ was closer to 100 times more errors.  Then
someone noticed that there was a large group of tests concerning
tri-graphs, and g++ was failing them all.  Since like most people, we
don't use tri-graphs, we removed this section of the test suite.  Any
other non-implemented feature will have a similar effect.

Before anyone starts posting defenses of g++, etc., please note that
this comparison was done some time ago, and that there are newer
versions of most of the compilers tested.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: jbuck@synopsys.com (Joe Buck)
Date: Thu, 4 Nov 1993 20:32:17 GMT
Raw View
kanze@us-es.sel.de (James Kanze) writes:
>We actually ran a number of commercial compilers (and g++) through a
>test suite some time back.  Although I don't remember the exact
>figures any longer, basically, g++ had roughly 10 times as many errors
>as the commercial products (which were all more or less equivalent).

It would be interesting if you re-ran the tests with a more recent version
of g++.  From private correspondence, it appears that James has no
experience with g++ versions newer than 2.2.2.

>Before anyone starts posting defenses of g++, etc., please note that
>this comparison was done some time ago, and that there are newer
>versions of most of the compilers tested.

Any re-run should use the latest versions of *all* compilers, of course,
not just g++.  Also, the following flags should be given to g++: "-ansi
-pedantic -fansi-overloading".  I expect that g++ will still come out
the buggiest, but it will no longer be by a factor of 10.

In gcc, trigraphs are not implemented by default: I think "-ansi" is
required.  g++ may be the same way (this is because RMS despises them).

As someone who uses both g++ and cfront a lot: the gap has narrowed
considerably because cfront hasn't been having its bugs fixed at nearly
the same rate.  (BOTH compilers are too buggy for my taste).


--
-- Joe Buck jbuck@synopsys.com
Posting from but not speaking for Synopsys, Inc.
Formerly jbuck@<various-hosts>.eecs.berkeley.edu




Author: hendrik@vedge.com (Hendrik Boom)
Date: Thu, 04 Nov 1993 14:27:34 GMT
Raw View
rfg@netcom.com (Ronald F. Guilmette) writes:
: In article <nagleCFq4pI.6K@netcom.com> nagle@netcom.com (John Nagle) writes:
: >      The Free Software Foundation has just released a C++ test suite
: >intended for "g++".  Details in "gnu.g++.announce".  Files at
: >"prep.ai.mit.edu:pub/gnu/g++-2.5.0-testsuite.tar.gz".
: >
: >      Applying this test suite to commercial compilers should be helpful
: >in locating their defects.  Obviously, vendors will want to do this before
: >users, and reviewers, do.
: >
: >           John Nagle
:
: Oh yea.  Definitely.  Every compiler vendor and every reviewer who wants
: to establish bug-for-bug compatability with the zillions of bugs in the
: g++ compiler ought to rush right out and get this test suite immediately.

The trouble with their test suite (I've used an earlier version)
is that it tests g++, not c++. That means it will fault a compiler
for not implementing, for example, the unary && operator....

:
: Get real.  What makes you think that anybody who can't handle producing an
: even marginally bug-free C++ compiler is going to have anymore success at
: producing an even marginally bug-free C++ test suite?
:

I disagree, g++ is marginally bug-free. It correctly implements
every specification printed in the margin of the ARM.

: --
:
: -- Ronald F. Guilmette, Sunnyvale, California -------------------------------
: ------ domain address: rfg@netcom.com ---------------------------------------
: ------ uucp address: ...!uunet!netcom.com!rfg -------------------------------
--
-------------------------------------------------------
Try one or more of the following addresses to reply.
at work: hendrik@vedge.com,  iros1!vedge!hendrik
at home: uunet!ozrout!topoi!hendrik




Author: jlr1801@eagle.tamu.edu (Jeff Rife)
Date: 5 Nov 1993 04:32:34 GMT
Raw View
In article <matth.752290227@extro.ucc.su.OZ.AU> matth@extro.ucc.su.OZ.AU (Matthew Hannigan) writes:
}jimlynch@netcom.com (Jim Lynch) writes:
}
}>In article <rfgCFsI2F.BqI@netcom.com> rfg@netcom.com (Ronald F. Guilmette) writes:
}>>Oh yea.  Definitely.  Every compiler vendor and every reviewer who wants
}>>to establish bug-for-bug compatability with the zillions of bugs in the
}>>g++ compiler ought to rush right out and get this test suite immediately.
}>>
}>>Get real.  What makes you think that anybody who can't handle producing an
}>>even marginally bug-free C++ compiler is going to have anymore success at
}>>producing an even marginally bug-free C++ test suite?
}
}
}>Care to back that statement up with a complete list? If not, <ho hum> your
}>comment isn't worth much.
}
}I would _really_ like to see the result of the testsuite on some
}commercial compilers.  Not that I think that they aren't better
}(at the moment), but how much better?


In article <1993Nov3.203051.29868@outback.ny.tpc.com> gregg@outback.ny.tpc.com (Greg Gershowitz) writes:
}goblec@physc1.byu.edu. (Clark Goble) writes:
}
}>   Is g++ buggier than Symantic C++?
}
}   Yes and no.  It's differently buggier.

I, too, would like to see some results on commercial compilers.  Since I
work in the PC world, I was going to test Microsoft C++ 8.0, Symantec C++
6.0, and Borland C++ 3.1 (no 4.0 here, yet).  But, I can't.

It seems that the filenames on the test suite are *so* far from being usable
by a PC namespace, even with auto mangling, (things like operators1.cpp,
operators2.cpp, etc.), that I don't care to waste the time on even trying to
get it to work.  The *directories* are even worse.

My suspicion is that they don't want it run on a commercial compiler, so
that the comparison to the horribly buggy g++ will never be made.

--
Jeff Rife   P.O. Box 3836   |   "Because he was human; because he had goodness; College Station, TX 77844   |    because he was moral they called him insane.
(409) 696-4252 or 847-8775  |    Delusions of grandeur; visons of splendor;     nabs@tamu.edu               |    A manic-depressive, he walks in the rain."




Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Fri, 5 Nov 1993 05:44:10 GMT
Raw View
In article <1993Nov1.111346.27775@kth.se> d88-jwa@mumrik.nada.kth.se (Jon W   tte) writes:
>
>Nobody can produce an even marginally bug-free C++ compiler.

I know of at least one counter-example.

>A test suite is somewhat simpler to produce - you read the ARM,
>and code a special case for each of the interesting things you
>find in there...

(Yea.  Right.  That's all there is to it. :-)  :-)

I think this is essentially what the folks at AT&T/USL did.  The result of
course was that they produced a perfectly bug-free C++ compiler... NOT.

(The funny thing is, I don't think that cfront even does everything shown
in the ARM perfectly... let alone all of the stuff that *not* quite in
there.)

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------




Author: kanze@us-es.sel.de (James Kanze)
Date: 5 Nov 93 15:26:34
Raw View
In article <1993Nov4.203217.18579@Synopsys.Com> jbuck@synopsys.com
(Joe Buck) writes:

|> kanze@us-es.sel.de (James Kanze) writes:
|> >We actually ran a number of commercial compilers (and g++) through a
|> >test suite some time back.  Although I don't remember the exact
|> >figures any longer, basically, g++ had roughly 10 times as many errors
|> >as the commercial products (which were all more or less equivalent).

|> It would be interesting if you re-ran the tests with a more recent version
|> of g++.  From private correspondence, it appears that James has no
|> experience with g++ versions newer than 2.2.2.

Correct.  This is why I hesitate to make too overt statements against
GNU at present.  Of course, the other compilers I currently have
access to are not that recent either: Sun CC 2.1 and an old Zortech
(the last version distributed by Zortech, I think).

As for re-running the tests, I agree that it would be interesting.
But it takes time.  I didn't actually run them myself the first time,
but they were run as part of an evaluation for a purchase; the
purchase has since been made, so no one here is going to be paid to
run them again.

|> >Before anyone starts posting defenses of g++, etc., please note that
|> >this comparison was done some time ago, and that there are newer
|> >versions of most of the compilers tested.

|> Any re-run should use the latest versions of *all* compilers, of course,
|> not just g++.  Also, the following flags should be given to g++: "-ansi
|> -pedantic -fansi-overloading".  I expect that g++ will still come out
|> the buggiest, but it will no longer be by a factor of 10.

|> In gcc, trigraphs are not implemented by default: I think "-ansi" is
|> required.  g++ may be the same way (this is because RMS despises them).

Regretfully, I have no idea what flags were used for the tests.  I do
know that all of the tests concerning tri-graphs were removed, as were
anything concerning templates and exceptions, and I believe that
anything with nested classes was also removed.  (Some of the compilers
being evaluated still used the old semantics for nested classes.)

Note that the factor 10 is not an exact figure.  For some compilers,
it may have only been 7 or 8, for others 12 or more.

Also, note that test suites are a lot like benchmarks; they can easily
be tilted in any desired direction.  (I don't think that there was any
concious attempt to tilt the one we were using, however.)  I'm sure I
could write a test suite (apparently objective) which showed that g++
had the least bugs, or a benchmark which showed that CFront had the
fastest compile times.  But that wouldn't mean that you will run into
less bugs with *your* programs, or that *your* programs will compile
any faster.

|> As someone who uses both g++ and cfront a lot: the gap has narrowed
|> considerably because cfront hasn't been having its bugs fixed at nearly
|> the same rate.  (BOTH compilers are too buggy for my taste).

Agreed.  Except that from my experience, you could replace 'both' with
'all' in the last sentance.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: d88-jwa@mumrik.nada.kth.se (Jon W tte)
Date: Fri, 5 Nov 1993 15:42:36 GMT
Raw View
>>Nobody can produce an even marginally bug-free C++ compiler.
>I know of at least one counter-example.

Name it. It ain't g++.

>>A test suite is somewhat simpler to produce - you read the ARM,
>>and code a special case for each of the interesting things you
>>find in there...

>(Yea.  Right.  That's all there is to it. :-)  :-)
>I think this is essentially what the folks at AT&T/USL did.  The result of
>course was that they produced a perfectly bug-free C++ compiler... NOT.


Uh... we said "test suite" here, not "C++ compiler" A test suite
that is made by "only" following the above statement will work
better than not having any test suite, and writing such a test
suite is simpler than writing a bug-free C++ compiler. It seems
to me you disagree? Do you mean it's simpler to write a bug-free
C++ compiler than a reasonable test suite?

--
 -- Jon W{tte, h+@nada.kth.se, Mac Hacker Deluxe --

  "From now on I will re-label the EQ on the deck as Fizz and Wobble
   instead of HF and LF."




Author: kanze@us-es.sel.de (James Kanze)
Date: 5 Nov 93 17:31:14
Raw View
In article <rfgCG07xo.8rq@netcom.com> rfg@netcom.com (Ronald F.
Guilmette) writes:

|> In article <1993Nov1.111346.27775@kth.se> d88-jwa@mumrik.nada.kth.se (Jon W   tte) writes:

|> >Nobody can produce an even marginally bug-free C++ compiler.

|> I know of at least one counter-example.

Would you care to say who?

|> >A test suite is somewhat simpler to produce - you read the ARM,
|> >and code a special case for each of the interesting things you
|> >find in there...

|> (Yea.  Right.  That's all there is to it. :-)  :-)

|> I think this is essentially what the folks at AT&T/USL did.  The result of
|> course was that they produced a perfectly bug-free C++ compiler... NOT.

|> (The funny thing is, I don't think that cfront even does everything shown
|> in the ARM perfectly... let alone all of the stuff that *not* quite in
|> there.)

When we did our compiler evaluation about a year back, in addition to
running a test suite through the compilers, we coded all of the
examples in the ARM.  Of course, we didn't expect to find much with
this; any compiler vendor would have at least done as much.  Ha.  Most
(or was it all, I forget) of the compilers failed to compile some of
the examples in the ARM correctly.

One sometimes gets the feeling that the compiler vendors aren't even
trying.
--
James Kanze                             email: kanze@us-es.sel.de
GABI Software, Sarl., 8 rue du Faisan, F-67000 Strasbourg, France
Conseils en informatique industrielle --
                   -- Beratung in industrieller Datenverarbeitung




Author: d88-jwa@mumrik.nada.kth.se (Jon W tte)
Date: Mon, 1 Nov 1993 11:13:46 GMT
Raw View
In <rfgCFsI2F.BqI@netcom.com> rfg@netcom.com (Ronald F. Guilmette) writes:

>Get real.  What makes you think that anybody who can't handle producing an
>even marginally bug-free C++ compiler is going to have anymore success at
>producing an even marginally bug-free C++ test suite?

Nobody can produce an even marginally bug-free C++ compiler.
It's an impossibility inherent in the language, and is the reason
why C++ will fail in the end (i e when the industry decides that
there are better alternatives)

A test suite is somewhat simpler to produce - you read the ARM,
and code a special case for each of the interesting things you
find in there. A suite like this is probably a very good debugging
tool for g++ as well - if they did it right, neither g++ nor any
other C++ compiler passes all parts.

Cheers,

    / h+
--
 -- Jon W{tte, h+@nada.kth.se, Mac Hacker Deluxe --

   This article printed on 100% recycled electrons.




Author: jimlynch@netcom.com (Jim Lynch)
Date: Tue, 2 Nov 1993 10:57:32 GMT
Raw View
In article <rfgCFsI2F.BqI@netcom.com> rfg@netcom.com (Ronald F. Guilmette) writes:
>In article <nagleCFq4pI.6K@netcom.com> nagle@netcom.com (John Nagle) writes:
>>      The Free Software Foundation has just released a C++ test suite
>>intended for "g++".  Details in "gnu.g++.announce".  Files at
>>"prep.ai.mit.edu:pub/gnu/g++-2.5.0-testsuite.tar.gz".
>>
>>      Applying this test suite to commercial compilers should be helpful
>>in locating their defects.  Obviously, vendors will want to do this before
>>users, and reviewers, do.
>>
>>           John Nagle
>
>Oh yea.  Definitely.  Every compiler vendor and every reviewer who wants
>to establish bug-for-bug compatability with the zillions of bugs in the
>g++ compiler ought to rush right out and get this test suite immediately.
>
>Get real.  What makes you think that anybody who can't handle producing an
>even marginally bug-free C++ compiler is going to have anymore success at
>producing an even marginally bug-free C++ test suite?
>
>--
>
>-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
>------ domain address: rfg@netcom.com ---------------------------------------
>------ uucp address: ...!uunet!netcom.com!rfg -------------------------------

Care to back that statement up with a complete list? If not, <ho hum> your
comment isn't worth much.

-Jim




Author: harrison@sp1.csrd.uiuc.edu (Luddy Harrison)
Date: Tue, 2 Nov 93 16:20:47 GMT
Raw View
jimlynch@netcom.com (Jim Lynch) writes:

>Care to back that statement up with a complete list? If not, <ho hum> your
>comment isn't worth much.

Are you saying that one must find every bug in the compiler in order
to conclude that it is buggy?  It seems to me sufficient to find so many
bugs that no useful progress can be made, or even one bug so serious that no
workaround can be found.  Besides, if the same bug is repeatedly reported
but is not fixed, what's the use of looking for more?

-Luddy Harrison




Author: matth@extro.ucc.su.OZ.AU (Matthew Hannigan)
Date: Wed, 3 Nov 1993 01:30:27 GMT
Raw View
[ mac groups removed ]

jimlynch@netcom.com (Jim Lynch) writes:

>In article <rfgCFsI2F.BqI@netcom.com> rfg@netcom.com (Ronald F. Guilmette) writes:
>>Oh yea.  Definitely.  Every compiler vendor and every reviewer who wants
>>to establish bug-for-bug compatability with the zillions of bugs in the
>>g++ compiler ought to rush right out and get this test suite immediately.
>>
>>Get real.  What makes you think that anybody who can't handle producing an
>>even marginally bug-free C++ compiler is going to have anymore success at
>>producing an even marginally bug-free C++ test suite?


>Care to back that statement up with a complete list? If not, <ho hum> your
>comment isn't worth much.

I would _really_ like to see the result of the testsuite on some
commercial compilers.  Not that I think that they aren't better
(at the moment), but how much better?

Here's the results for gcc from the announcement, in case anyone missed
it.

}   This suite is still under development, and has some known problems.
} The biggest problem is that not all test cases have their expectedness
} set right yet.  When run on a SunOS 4.1.3 SPARC machine using a
} prerelease version of gcc 2.5.0, I get:
}
}                 === g++ Summary ===
}
} # of expected passes       733
} # of expected failures     130
} # of unexpected successes  1
} # of unexpected failures   146
}
} You should see numbers similar to this.  If not, then something is
} wrong.  In the future, the number of unexpected failures will be much
} closer to zero.





Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Wed, 3 Nov 1993 01:36:19 GMT
Raw View
In article <jimlynchCFv2Fx.Hw0@netcom.com> jimlynch@netcom.com (Jim Lynch) writes:
>In article <rfgCFsI2F.BqI@netcom.com> rfg@netcom.com (Ronald F. Guilmette) writes:
>[... comments about buggyness of the g++ compiler deleted for brevity...]
>
>Care to back that statement up with a complete list? If not, <ho hum> your
>comment isn't worth much.

I sure as heck don't have time to compile a COMPLETE list of g++ bugs.
Nobody does.  If you are interested in seeing examples of about a ton
or two of g++ bugs, I suggest that you subscribe to the gnu.g++.bugs
newsgroup.  You should be able to gleen about a hundred or so new ones
a week from that newsgroup.

(A multi-megabyte collection of old g++ bugs reports is available somewhere
on the net also, but I have forgotten where.)

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------




Author: rfg@netcom.com (Ronald F. Guilmette)
Date: Mon, 1 Nov 1993 01:42:14 GMT
Raw View
In article <nagleCFq4pI.6K@netcom.com> nagle@netcom.com (John Nagle) writes:
>      The Free Software Foundation has just released a C++ test suite
>intended for "g++".  Details in "gnu.g++.announce".  Files at
>"prep.ai.mit.edu:pub/gnu/g++-2.5.0-testsuite.tar.gz".
>
>      Applying this test suite to commercial compilers should be helpful
>in locating their defects.  Obviously, vendors will want to do this before
>users, and reviewers, do.
>
>           John Nagle

Oh yea.  Definitely.  Every compiler vendor and every reviewer who wants
to establish bug-for-bug compatability with the zillions of bugs in the
g++ compiler ought to rush right out and get this test suite immediately.

Get real.  What makes you think that anybody who can't handle producing an
even marginally bug-free C++ compiler is going to have anymore success at
producing an even marginally bug-free C++ test suite?

--

-- Ronald F. Guilmette, Sunnyvale, California -------------------------------
------ domain address: rfg@netcom.com ---------------------------------------
------ uucp address: ...!uunet!netcom.com!rfg -------------------------------