James Kanze wrote:
>
> I'll still stand by my original evaluation, however.  Fragmentation is
> not a problem for most programs, and probably shouldn't be addressed
> until it is shown to be a problem.
>

I agree. If I remember correctly, Knuth found that once a program gets
into a steady state (where it's freeing just about as many bytes as it's
allocating) the heap settles into a pattern where allocated blocks are
about twice as large as free blocks. (that's with a heap manager that
merges adjacent free blocks). Most compilers these days come with more
sophisticated memory managers which are much faster (merging freed
blocks in constant time instead of linear time) and that produce less
fragmentation (multiple free lists, one for each small allocation size,
for example). For most purposes this is good enough (i.e. it's far
better than most people could do on their own, and in practice doesn't
lead to performance problems).

--
Pete Becker
Dinkumware, Ltd. (http://www.dinkumware.com)

      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

Matthias D   rfel <doerfel@gmx.de> wrote
> Why not use alloca() for the memory allocation? It allocates memory on the
> stack frame of the current function, which is automatically destroyed, when
> the current function terminates. Your problem of above would then be
> written:
>
>   int len = ::GetTextLen();
>   char *str = (char *)alloca( len+1 );
>   ::GetText(str);
>   DoSomeThingWith( str );
>   return;
>   /* str not valid any more! */

> As for an object oriented drop-in replacement: Is it possible to overwrite
> the new operator for a class and to not use new or malloc on the heap but
> alloca on the stack?

This really is not possible. If you tried it, alloca() would allocate
space on the top of the stack, with operator new still active. Returning
from operator new would also free the memory just allocated, so the
pointer would point into unallocated stack space.

      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

> But getting to the point, would there not be some benefit to
> programming the basic_string so that it allocates from the same memory as
> basic_string a small buffer of its own as part of the class?  Basically,
> just adding char abuffer[16]; or something like it to the class.
>
> This is sort of a half baked idea but thought I would put it out there and
> see what other say.  Perhaps there are better answers.

The idea is good, but to avoid reinventing the wheel, you might want to take
a look at:
http://www.cuj.com/experts/1906/alexandr.htm?topic=experts&topic=experts
http://www.johnpanzer.com/tsc_cuj/ToolboxOfStrings.html.

--
Peter van Merkerk
merkerk(at)turnkiek.nl
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

"Greg Brewer" <nospam.greg@brewer.net> wrote in message news:<amsho8$k6$1@news.hal-pc.org>...
> We have a situation where a program runs for several days and terminates due
> to "memory problems".  I have not been consulted on this and have only heard
> about it from hallway conversations.  From what I hear, they are absolutely
> sure there isn't a memory leak; there is plenty of memory available but
> memory requests are coming back null.  In my experience, there is one cause
> of this -- memory fragmentation.

Fragmentation will get you only if the program occasionally needs
large blocks of memory.

>
> Looking at the people's code, I see lots of the following
>    String effDate = dly.effective_date;
>    startTs.SetValues(0, 0,
>          GetContractHour(dly.rollup_calendar_id),
>          effDate.SubString(9,2).ToInt(), // day
>          effDate.SubString(6,2).ToInt(), // month
>          effDate.SubString(1,4).ToInt()); // year
>
> This is Borland code and I assume String is very similar to string in the
> standard template library; I confess to not being too familiar with either.
> Looking at that code, I see no less than 4 calls that briefly allocate a few
> bytes of memory.  Code of this nature is used extensively throughout the
> code and the code is executed frequently.
>
> To me, this is bad programming for something that is expected to execute
> continuously for months at a time.
> I would not have done it this way;
> however, my way takes longer and is more error prone.  I am unable to
> explain these things to the boss though; especially since it isn't a problem
> under ordinary circumstances -- ie lots of memory, i/o bound, and limited
> execution time.
>
> Does anyone else think that is bad programming?

Yes, but not for the reason you think.

It's bad because the data structures and function interfaces
are poorly designed.  This code cries out for a 'date' class,
but there is none.

dly.effective_date should be a date, not a string.

GetContractHour should be modified to accept a date, not
separate year/month/day arguments.

The code should read simply:

   startTs.setValues(0, 0,
                     GetContractHour(dly.rollup_calendar_id,
                                     dly.effective_date));


> If so, does not the
> existance of these string types lead to bad programming practices?

No, and I doubt that the program is crashing due to fragmentation.
I expect that they do have a memory leak.

>
> Also, what is the alternative.  I frequently see code similar to the
> following
>    int len = ::GetTextLen();
>    char *str = new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    delete str;

No delete[]?  This code is horrible too.  Why not use std::string
or the String class shown above?  The interaction between
::GetTextLen() and ::GetText(char*) is also very poor.  It's is quite
easy for a call to ::GetText to result in a buffer overflow.
::GetText should be modified to accept a string argument by reference:
   void GetText(String& text);

Anyway, if this is a typical pattern in their code, I would be
amazed if they didn't have memory leaks.  It's all too easy
to write:

   char* str = new char[len+1];
   ...
   if (cond) {
      ...
      return;
   }

   delete[] str;

>
> I will code the same thing as follows
>    int len = ::GetTextLen();
>    char buffer[64], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;

No, this is worse by far.  You are locally implementing a 'small
string'
optimization.  Low-level code like this should always be encapsulated
in some class so it doesn't have to be cut-and-pasted ad nauseum.

It would be good if you could bring in an outside expert to give
you all a class in C++ programming.  Happy bug hunting.
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

"Greg Brewer" <nospam.greg@brewer.net> wrote:
> Looking at the people's code, I see lots of the following
>    String effDate = dly.effective_date;
>    startTs.SetValues(0, 0,
>          GetContractHour(dly.rollup_calendar_id),
>          effDate.SubString(9,2).ToInt(), // day
>          effDate.SubString(6,2).ToInt(), // month
>          effDate.SubString(1,4).ToInt()); // year

> Looking at that code, I see no less than 4 calls that briefly allocate a few
> bytes of memory.  Code of this nature is used extensively throughout the
> code and the code is executed frequently.

I don't see any call which necessarily allocates any memory! First of all, the
strings might share their representation. Whether this is a feasible approach
depends on the interface of the complete string (it is, for example, not a
feasible approach for the standard string interface). However, the strings are
all small and it is very well possible that a "small string optimization" is
used, where each string object has a small array of characters used in case
the string is small (of course, this would be a union and the array would be
used for other maintainance data if the string is bigger and stored outside
the string). AFAIK, the Dinkumware implementation shipping with VC++7.* uses
this approach.

> To me, this is bad programming for something that is expected to execute
> continuously for months at a time.

It looks pretty safe and it does not cause memory leaks - a reasonable goal
for the a long running program. The only danger is possible fragmentation of
the memory pool: There is enough memory available but it is fragmented into
small unused portions between a few use portions. If the memory allocator
does not merge adjacent portions of unused memory into bigger portions, this
can indeed be a substantial problem!

> I would not have done it this way;

I would *definitely* use a different approach than the one you propose below!
(see comments below for details)

> Also, what is the alternative.  I frequently see code similar to the
> following
>    int len = ::GetTextLen();
>    char *str = new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    delete str;

This is a pretty bad idea for two reasons:
1. The above code invariable travels into "undefined behavior"-land (unless
   the variable 'str' is passed to one of the functions by reference and this
   function releases the character array and replaces 'str' by a suitable
   alternate value): The problem is that the approaches to allocation and
   deallocation do not match! If you use 'new T[n]' to allocate an array
   object, you have to use 'delete[] ptr' (not the brackets) to release the
   array object (anybody who wants to discuss this, should please provides the
   relevant quotes indicating why 5.3.5, expr.delete, does not apply...).
2. The memory is not savely released because one of the functions might
   throw an exception or because the deleting expression is easy to foget.

Apart from these two problems, this code does not address the memory
fragmentation and, even worse, it does not provide an easy possibility to
address this by changing a specific class.

> I will code the same thing as follows
>    int len = ::GetTextLen();
>    char buffer[64], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;

This code shared the problems of the previous code segment with the additional
drawback that the code has become even more complex! Basically, what you are
doing here manually is something which can be done in the string class at a
local place.

> This is sort of a half baked idea but thought I would put it out there and
> see what other say.  Perhaps there are better answers.

Indeed there are: Basically, your idea is taken and encapsulated in some
string implementations. Another idea which is used is allocation of memory
from pools where the memory portions in a pool all have the same size. This
addresses certain overhead issues and, at least to some degree, the
fragmentation problem. I'd probably favor a string class with a small string
optimization (it somewhat depends on the operations done with the strings,
though).

> I would be open to
> just having a 2nd string type that is similar to the old Basic string type
> (ie {public: short len; char buffer[256];};) so that just change all strings
> to new type.

Maximum-sized strings are typically a bad idea: Yes, they will work in the
envisioned uses but they may stop work silently. Just use a string with a
small buffer which allocates from the heap when necessary. As mentioned
already, some 'std::basic_string<>()' implementations do it this way already.
--
<mailto:dietmar_kuehl@yahoo.com> <http://www.dietmar-kuehl.de/>
Phaidros eaSE - Easy Software Engineering: <http://www.phaidros.com/>
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

"Kevin Cline" <kcline17@hotmail.com> wrote in message
news:ba162549.0209260645.4865be4d@posting.google.com...
> "Greg Brewer" <nospam.greg@brewer.net> wrote in message
news:<amsho8$k6$1@news.hal-pc.org>...
> > Looking at the people's code, I see lots of the following
> >    String effDate = dly.effective_date;
> >    startTs.SetValues(0, 0,
> >          GetContractHour(dly.rollup_calendar_id),
> >          effDate.SubString(9,2).ToInt(), // day
> >          effDate.SubString(6,2).ToInt(), // month
> >          effDate.SubString(1,4).ToInt()); // year
> > Does anyone else think that is bad programming?
> Yes, but not for the reason you think.
>
> It's bad because the data structures and function interfaces
> are poorly designed.  This code cries out for a 'date' class,
> but there is none.

I couple of people pointed this out.  startTs is a date and time class.  I
would have done that code as follows

    startTs.Set(dly.effective_date);
    startTs.Set(DateTime::Hour(GetContractHour(dly.rollup_calendar_id)));

I have no idea why they didn't.  The DateTime class is something I wrote 9
years ago.  It is pretty good about figuring out where the year, month and
day are.  Most of the time (this time for example) it is feed a 4 digit
year.  It expects the year to be first (for sorting) or last (usual us
format).  It assumes the later for 2 digit year for then 10+ years (01-12
could be month or 2 digit year).  It will also process a 3 digit year (102
is 2002).

I didn't explain all of this because it was a long post already and my time
is limited.

Greg




      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

kanze@gabi-soft.de (James Kanze) writes:

[snip]
> In my experience, I've never seen excessive memory consumation due to
> memory fragmentation.  I know that it is possible in theory, but it is
> rarely a problem in practice.

My experience is:

    (a) On machines without virtual memory (game consoles and embeded
        systems are 2 examples) memory fragmentation causes enough
        problems that few significant programs can be written without
        both carefully designing to avoid it, and also looking for it
        whenever memory problems arise.

    (b) On machines with virtual memory, memory fragmentation seldom
        if ever becomes an issue.

    As to why:

    (a) On systems without virtual memory, a fragmented heap
        reduces the maximum allocation size (often by 1/2, or much
        more), unless one writes code to defragment the heap by moving
        and merge blocks. Note that moving and merging allocated
        blocks requires client code to make religious use of handles
        (double indirection, foo** or equivalent wrapped in a smart
        pointer) and associated tricks. Merging only freed blocks
        requires one to free everything (or knowing which allocated
        blocks cause the fragmentation, etc) in order to defragment
        the heap.

    (b) On systems with virtual memory, it's usually posible to simply
        allocate more memory, so a fragmented heap does not prevent
        one from allocating large blocks; it simply means a program
        will have a larger memory footprint. The smaller fragments can
        then be used for other objects, so the footprint may not
        increase by much. (In the non-vm systems I have
        used, the large objects that could not be allocated due to
        fragmentation were usually so critical there was no sense
        continuing without them.)
[snip]

      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

Greg Brewer wrote:
> Does anyone else think that is bad programming?

Sorry, I cannot judge on this topic. But I have some suggestions.

> In the previous code, len could be over 64 but never is in practice.  I
> would like to be able to
> do it as follows
>    int len = ::GetTextLen();
>    int bufferlen = len < 64 ? len+1 : 1;
>    char buffer[bufferlen], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;
> so that a minimum amount is allocated from the stack when the heap will be
> used.  But that requires variable length arrays which won't come soon.

Why not use alloca() for the memory allocation? It allocates memory on the
stack frame of the current function, which is automatically destroyed, when
the current function terminates. Your problem of above would then be
written:

  int len = ::GetTextLen();
  char *str = (char *)alloca( len+1 );
  ::GetText(str);
  DoSomeThingWith( str );
  return;
  /* str not valid any more! */

During DoSomeThingWith() the stack frame in which str lies is still valid.

> may be necessary.  I personally would like something easy to drop in.

IMHO this seems to fit your needs and avoids memory fragmentation. If you
have enough memory there should always be space on the stack.

As for an object oriented drop-in replacement: Is it possible to overwrite
the new operator for a class and to not use new or malloc on the heap but
alloca on the stack? If the programmer gives away references to such
objects (eg as a return value), how could you recognize this
_at_compile_time_?


Hope this helps,
Matthias.
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

In article <amsho8$k6$1@news.hal-pc.org>, Greg Brewer
<nospam.greg@brewer.net> writes
>We have a situation where a program runs for several days and terminates due
>to "memory problems".  I have not been consulted on this and have only heard
>about it from hallway conversations.  From what I hear, they are absolutely
>sure there isn't a memory leak; there is plenty of memory available but
>memory requests are coming back null.  In my experience, there is one cause
>of this -- memory fragmentation.

You are likely to be right. The solutions to this are such things as

1)  providing a custom allocator (there are various forms, such as using
a pool, if the objects are of fixed size, retiring the memory for
destructed objects to a linked list where it can be re-used -- perhaps
creating such a list to use a pre-allocated pool etc.)

2) For variable size objects such as string where the majority of
instances are below a certain size and stack space is not at a premium,
use internal buffers (as do most of the recent implementations of
string) so that all instances below a certain size are completely on the
stack.


Note, I hope that all the usages of new are encapsulated in ctors. I.e.
the code uses RAII. The code you quoted did not seem to be that way and
is poor code in modern exception handling contexts.




--
Francis Glassborow      ACCU
64 Southfield Rd
Oxford OX4 1PA          +44(0)1865 246490
All opinions are mine and do not represent those of any organisation
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

llewelly <llewelly.@@xmission.dot.com> wrote in message
news:<86y99m4ro1.fsf@Zorthluthik.foo>...
> kanze@gabi-soft.de (James Kanze) writes:

> [snip]
> > In my experience, I've never seen excessive memory consumation due
> > to memory fragmentation. I know that it is possible in theory, but
> > it is rarely a problem in practice.

> My experience is:

>     (a) On machines without virtual memory (game consoles and embeded
>         systems are 2 examples) memory fragmentation causes enough
>         problems that few significant programs can be written without
>         both carefully designing to avoid it, and also looking for it
>         whenever memory problems arise.

It's interesting to hear from someone who has experience in domains I
haven't.  In my experience with embedded systems (and limited memory),
we generally avoided using dynamic memory entirely, so fragmentation
wasn't a problem, but I can't quite see this as a solution for something
like a game console (and since each game must have more features than
any other game, no matter how much memory you usually have, it ends up
being tight).

>     (b) On machines with virtual memory, memory fragmentation seldom
>         if ever becomes an issue.

>     As to why:

>     (a) On systems without virtual memory, a fragmented heap reduces
>         the maximum allocation size (often by 1/2, or much more),
>         unless one writes code to defragment the heap by moving and
>         merge blocks. Note that moving and merging allocated blocks
>         requires client code to make religious use of handles (double
>         indirection, foo** or equivalent wrapped in a smart pointer)
>         and associated tricks. Merging only freed blocks requires one
>         to free everything (or knowing which allocated blocks cause
>         the fragmentation, etc) in order to defragment the heap.

There's no doubt that fragmentation increases memory requirements.  My
point was more along the lines that the increase isn't infinite (your
estimation of doubling sounds about right), and that most systems can
afford it.  (But it obviously depends on what you are doing, and how
much memory you have to do it in.)

>     (b) On systems with virtual memory, it's usually posible to simply
>         allocate more memory, so a fragmented heap does not prevent
>         one from allocating large blocks; it simply means a program
>         will have a larger memory footprint. The smaller fragments can
>         then be used for other objects, so the footprint may not
>         increase by much. (In the non-vm systems I have used, the
>         large objects that could not be allocated due to fragmentation
>         were usually so critical there was no sense continuing without
>         them.)

Actually, I suspect that the issue is more one of a shared system, as
opposed to a system dedicated to just one job.  Virtual memory isn't
infinite, either.  But on a shared system, there's a good chance that
when you need the memory, someone else doesn't.

In fact, virtual memory introduces another aspect to fragmentation:
fragmentation may not cause you to run out of memory, but it will reduce
locality, which can lead to more page faults, and slower running.

I'll still stand by my original evaluation, however.  Fragmentation is
not a problem for most programs, and probably shouldn't be addressed
until it is shown to be a problem.

--
James Kanze                           mailto:jkanze@caicheuvreux.com
Conseils en informatique orient   e objet/
                    Beratung in objektorientierter Datenverarbeitung
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

Matthias D   rfel wrote:
>
> Why not use alloca() for the memory allocation? It allocates memory on the
> stack frame of the current function, which is automatically destroyed, when
> the current function terminates. Your problem of above would then be
> written:
>
>   int len = ::GetTextLen();
>   char *str = (char *)alloca( len+1 );
>   ::GetText(str);
>   DoSomeThingWith( str );
>   return;
>   /* str not valid any more! */
>
> During DoSomeThingWith() the stack frame in which str lies is still valid.

alloca() has never been standard and I hear its usage is discouraged.
( http://www.eskimo.com/~scs/C-faq/q7.32.html )

--
Sungbom Kim <musiphil@bawi.org>

      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

Francis Glassborow <francis.glassborow@ntlworld.com> wrote in message
news:<SoYxecCT9sk9EwMV@robinton.demon.co.uk>...

> Note, I hope that all the usages of new are encapsulated in
> ctors. I.e.  the code uses RAII. The code you quoted did not seem to
> be that way and is poor code in modern exception handling contexts.

You mean, of course, that all new'ed memory rapidly ends up in the hands
of a class which will delete it in the destructor.  It's quite frequent
to allocate the memory outside of the constructor; both std::auto_ptr
and booch::shared_ptr, for example, require the user to allocate the
memory outside of their constructors.

(Of course, there was no problem with this in the posted code, where all
of the dynamic allocations -- supposing that there were any -- were
encapsulated in the String class.)

--
James Kanze                           mailto:jkanze@caicheuvreux.com
Conseils en informatique orient   e objet/
                    Beratung in objektorientierter Datenverarbeitung

      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

We have a situation where a program runs for several days and terminates due
to "memory problems".  I have not been consulted on this and have only heard
about it from hallway conversations.  From what I hear, they are absolutely
sure there isn't a memory leak; there is plenty of memory available but
memory requests are coming back null.  In my experience, there is one cause
of this -- memory fragmentation.

Looking at the people's code, I see lots of the following
   String effDate = dly.effective_date;
   startTs.SetValues(0, 0,
         GetContractHour(dly.rollup_calendar_id),
         effDate.SubString(9,2).ToInt(), // day
         effDate.SubString(6,2).ToInt(), // month
         effDate.SubString(1,4).ToInt()); // year

This is Borland code and I assume String is very similar to string in the
standard template library; I confess to not being too familiar with either.
Looking at that code, I see no less than 4 calls that briefly allocate a few
bytes of memory.  Code of this nature is used extensively throughout the
code and the code is executed frequently.

To me, this is bad programming for something that is expected to execute
continuously for months at a time.  I would not have done it this way;
however, my way takes longer and is more error prone.  I am unable to
explain these things to the boss though; especially since it isn't a problem
under ordinary circumstances -- ie lots of memory, i/o bound, and limited
execution time.

Does anyone else think that is bad programming?  If so, does not the
existance of these string types lead to bad programming practices?

Also, what is the alternative.  I frequently see code similar to the
following
   int len = ::GetTextLen();
   char *str = new char[len+1];
   ::GetText(str);
   DoSomethingWith(str);
   delete str;

I will code the same thing as follows
   int len = ::GetTextLen();
   char buffer[64], *str = len < 64 ? buffer : new char[len+1];
   ::GetText(str);
   DoSomethingWith(str);
   if (str != buffer)
      delete str;

In the previous code, len could be over 64 but never is in practice.  I
would like to be able to
do it as follows
   int len = ::GetTextLen();
   int bufferlen = len < 64 ? len+1 : 1;
   char buffer[bufferlen], *str = len < 64 ? buffer : new char[len+1];
   ::GetText(str);
   DoSomethingWith(str);
   if (str != buffer)
      delete str;
so that a minimum amount is allocated from the stack when the heap will be
used.  But that requires variable length arrays which won't come soon.  And
considering Stroustrup's opposition to that, may never happen.  I set the
bufferlen to 1 because I know I can allocate that much; I'm not sure about 0
bytes.  But getting to the point, would there not be some benefit to
programming the basic_string so that it allocates from the same memory as
basic_string a small buffer of its own as part of the class?  Basically,
just adding char abuffer[16]; or something like it to the class.

This is sort of a half baked idea but thought I would put it out there and
see what other say.  Perhaps there are better answers.  I suspect that many
will say that the problem is really a memory management problem; however, if
the os/memorymanager doesn't handle fragmentation well then a work-around
may be necessary.  I personally would like something easy to drop in.
Providing a local buffer would do that with no changes.  I would be open to
just having a 2nd string type that is similar to the old Basic string type
(ie {public: short len; char buffer[256];};) so that just change all strings
to new type.

Greg Brewer




      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

Greg Brewer wrote:
 > In my experience, there is one cause of this -- memory fragmentation.

Really? In my experience, there is also one cause of this --
somewhere in the code, an errant pointer or array access is
clobbering memory that doesn't belong to it, and that screws
up the memory allocator.

If there is "plenty of memory available" that means that
the program has not attempted to allocate all of memory.
If fragmentation was the problem, the program would be
requesting more and more memory from the operating system,
and it would look like all memory was being used up.

I think fragmentation is a red herring here, and the program
has a plain old bug. Have they tried using Purify?


      [ Send an empty e-mail to c++-help@netlab.cs.rpi.edu for info ]
      [ about comp.lang.c++.moderated. First time posters: do this! ]

[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

"Greg Brewer" <nospam.greg@brewer.net> wrote...
> We have a situation where a program runs for several days and terminates
due
> to "memory problems".  I have not been consulted on this and have only
heard
> about it from hallway conversations.  From what I hear, they are
absolutely
> sure there isn't a memory leak; there is plenty of memory available but
> memory requests are coming back null.  In my experience, there is one
cause
> of this -- memory fragmentation.

Usually the answer to this is a better memory manager than the one
that comes with your compiler.  Look up MicroQuill (a company).

> Looking at the people's code, I see lots of the following
>    String effDate = dly.effective_date;
>    startTs.SetValues(0, 0,
>          GetContractHour(dly.rollup_calendar_id),
>          effDate.SubString(9,2).ToInt(), // day
>          effDate.SubString(6,2).ToInt(), // month
>          effDate.SubString(1,4).ToInt()); // year
>
> This is Borland code and I assume String is very similar to string in the
> standard template library; I confess to not being too familiar with
either.
> Looking at that code, I see no less than 4 calls that briefly allocate a
few
> bytes of memory.  Code of this nature is used extensively throughout the
> code and the code is executed frequently.
>
> To me, this is bad programming for something that is expected to execute
> continuously for months at a time.  I would not have done it this way;

Which way _would_ you have done this?

> however, my way takes longer and is more error prone.  I am unable to
> explain these things to the boss though; especially since it isn't a
problem
> under ordinary circumstances -- ie lots of memory, i/o bound, and limited
> execution time.

It could be an indication that your boss' priorities are different
from yours.  You can (a) change your priorities or (b) change your
boss.

> Does anyone else think that is bad programming?

Probably.  I don't, if it matters.

>  If so, does not the
> existance of these string types lead to bad programming practices?

Absolutely not.  Existence of handguns does NOT lead to murders,
no matter how much gun control lobbyists would like you to believe
that.

>
> Also, what is the alternative.  I frequently see code similar to the
> following
>    int len = ::GetTextLen();
>    char *str = new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    delete str;

That's really bad.  It should be

     delete[] str;

>
> I will code the same thing as follows
>    int len = ::GetTextLen();
>    char buffer[64], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;

Again, this should be

        delete[] str;

However, that's probably not the point.  There is no guarantee that
in your code automatic array 'buffer' is not allocated from the same
heap as the dynamic array.  It could just as well lead to memory
fragmentation.

>
> In the previous code, len could be over 64 but never is in practice.  I
> would like to be able to
> do it as follows
>    int len = ::GetTextLen();
>    int bufferlen = len < 64 ? len+1 : 1;
>    char buffer[bufferlen], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;

    delete[] str;

> so that a minimum amount is allocated from the stack when the heap will be
> used.

"Stack" and "heap" are not necessarily different things.

>  But that requires variable length arrays which won't come soon.  And
> considering Stroustrup's opposition to that, may never happen.

That's not the problem.  Run-time-sized arrays can be easily implemented
(and look at G++ extension, they already do that if you so desperately
need them), but they can be implemented as dynamic arrays (i.e. allocated
from the heap) with an automatic storage duration.

>  I set the
> bufferlen to 1 because I know I can allocate that much; I'm not sure about
0
> bytes.  But getting to the point, would there not be some benefit to
> programming the basic_string so that it allocates from the same memory as
> basic_string a small buffer of its own as part of the class?  Basically,
> just adding char abuffer[16]; or something like it to the class.

There might be.  Nobody prevents you from supplying your own
Allocator to any 'basic_string<>' object to take advantage of the
use of one big chunk of heap for small strings.

> This is sort of a half baked idea but thought I would put it out there and
> see what other say.

That's one approach.

>  Perhaps there are better answers.  I suspect that many
> will say that the problem is really a memory management problem; however,
if
> the os/memorymanager doesn't handle fragmentation well then a work-around
> may be necessary.

Sure.  And there are work-arounds.  I hope I have presented a couple.

>  I personally would like something easy to drop in.
> Providing a local buffer would do that with no changes.  I would be open
to
> just having a 2nd string type that is similar to the old Basic string type
> (ie {public: short len; char buffer[256];};) so that just change all
strings
> to new type.

You have an infinite number of string types at your disposal.
'basic_string' is a TEMPLATE.  Give it your own allocator, give
it your own traits, base it on a different than 'char' type, and
you will open a very broad field of possibilities.  Standard
Library is not there to constrain you, but to provide you with
an opportunity.  And it does.  It seems, judging from your own
admission that you aren't very familiar with 'std::basic_string',
that you just need some education.  Get yourself a copy of Nicolai
Josuttis' "The C++ Standard Library" and Scott Meyers' "Effective
STL".  Mighty helpful reading.

Victor
--
Please remove capital A's from my address when replying by mail
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

"Greg Brewer" <nospam.greg@brewer.net> wrote...
[snip]
> Looking at the people's code, I see lots of the following
>    String effDate = dly.effective_date;
>    startTs.SetValues(0, 0,
>          GetContractHour(dly.rollup_calendar_id),
>          effDate.SubString(9,2).ToInt(), // day
>          effDate.SubString(6,2).ToInt(), // month
>          effDate.SubString(1,4).ToInt()); // year
>
> This is Borland code and I assume String is very similar to string in the
> standard template library; I confess to not being too familiar with
either.
> Looking at that code, I see no less than 4 calls that briefly allocate a
few
> bytes of memory.  Code of this nature is used extensively throughout the
> code and the code is executed frequently.
>
> To me, this is bad programming for something that is expected to execute
> continuously for months at a time.  I would not have done it this way;
> however, my way takes longer and is more error prone.  I am unable to
> explain these things to the boss though; especially since it isn't a
problem
> under ordinary circumstances -- ie lots of memory, i/o bound, and limited
> execution time.
>
> Does anyone else think that is bad programming?  If so, does not the
> existance of these string types lead to bad programming practices?

This doesn't look like bad programming but poor design.  The conversion from
one data type to another (string to date) should be encapsulated in a
separate class (Date) that takes a String as a ctor argument.  Otherwise you
have a very fragile design - say if you decide to add hours & minutes.

A separate Date class would be much more robust, probably more efficient (by
not using the heap) and simpler to use and maintain:

struct Date
{
  int year, month, day;
  Date( const char* date)  // Error checking omitted
    { sscanf( date, "%d/%d/%d", &year, &month, &day); }
};

   Date date( dly.effective_date);
   startTs.SetValues(0, 0,
         GetContractHour(dly.rollup_calendar_id),
         date.day, // day
         date.month, // month
         date.year); // year

[snip]
> This is sort of a half baked idea but thought I would put it out there and
> see what other say.  Perhaps there are better answers.  I suspect that
many
> will say that the problem is really a memory management problem; however,
if
> the os/memorymanager doesn't handle fragmentation well then a work-around
> may be necessary.

Fragmentation is a serious issue in any system that must run for long
periods.  A common solution is to partition memory into zones that serve
small, medium and large requests.  This would require a user written
implementation of operator new, either globally or on a class by class
basis.  Even zoning is not guaranteed to solve bad fragmentation.  The only
sure fire solution is to pre-allocate all memory at start up :-(  If
de-allocations can be made in the reverse order to allocations then
fragmentation will be minimal.

-- Lawrence Rust, Software Systems, www.softsystem.co.uk
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]

"Greg Brewer" <nospam.greg@brewer.net> wrote in message
news:<amsho8$k6$1@news.hal-pc.org>...
> We have a situation where a program runs for several days and
> terminates due to "memory problems". I have not been consulted on this
> and have only heard about it from hallway conversations. From what I
> hear, they are absolutely sure there isn't a memory leak; there is
> plenty of memory available but memory requests are coming back null.

Sure.  And I'm absolutely sure that I never make a programming mistake
either:-).

What does Purify say?  (If Purify says no memory leaks, it doesn't mean
very much -- only that you haven't exercised the correct branches, or
that you are holding the memory in a map, or some other structure.  But
if it does say something, that's always one less bug that goes out the
door.)

> In my experience, there is one cause of this -- memory fragmentation.

In my experience, I've never seen excessive memory consumation due to
memory fragmentation.  I know that it is possible in theory, but it is
rarely a problem in practice.

> Looking at the people's code, I see lots of the following
>    String effDate = dly.effective_date;
>    startTs.SetValues(0, 0,
>          GetContractHour(dly.rollup_calendar_id),
>          effDate.SubString(9,2).ToInt(), // day
>          effDate.SubString(6,2).ToInt(), // month
>          effDate.SubString(1,4).ToInt()); // year

> This is Borland code and I assume String is very similar to string in
> the standard template library; I confess to not being too familiar
> with either.  Looking at that code, I see no less than 4 calls that
> briefly allocate a few bytes of memory.

That depends on the implementation of String.  I know of several
implementations (the majority today?) where these SubString
implementations won't allocate any dynamic memory, or will only allocate
from a separate pool of fixed length blocks.

> Code of this nature is used extensively throughout the code and the
> code is executed frequently.

Sounds reasonable to me.  That's the whole purpose of having a String
class.

> To me, this is bad programming for something that is expected to
> execute continuously for months at a time.

It's something I've done without hesitation in code that really runs for
years at a time.

> I would not have done it this way; however, my way takes longer and is
> more error prone.

Which means that it is less appropriate for code that should run without
stopping.

> I am unable to explain these things to the boss though; especially
> since it isn't a problem under ordinary circumstances -- ie lots of
> memory, i/o bound, and limited execution time.

> Does anyone else think that is bad programming?  If so, does not the
> existance of these string types lead to bad programming practices?

Obviously.  Forget the abstraction.  We should all be programming in
assembler.

> Also, what is the alternative.  I frequently see code similar to the
> following
>    int len = ::GetTextLen();
>    char *str = new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    delete str;

> I will code the same thing as follows
>    int len = ::GetTextLen();
>    char buffer[64], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;

I've used this trick once or twice.  Not because of problems with memory
fragmentation, but because the profiler showed the memory allocation to
be a bottle neck.  (I've even got a class, Buffer, which encapsulates
it.)

Most modern implementations of std::string, from what I understand, use
it.

> In the previous code, len could be over 64 but never is in practice.
> I would like to be able to do it as follows
>    int len = ::GetTextLen();
>    int bufferlen = len < 64 ? len+1 : 1;
>    char buffer[bufferlen], *str = len < 64 ? buffer : new char[len+1];
>    ::GetText(str);
>    DoSomethingWith(str);
>    if (str != buffer)
>       delete str;

> so that a minimum amount is allocated from the stack when the heap
> will be used.  But that requires variable length arrays which won't
> come soon.

Well, you seem to want to return to C.  C already has variable length
arrays.

If portability isn't a concern, of course, there is always (or rather,
there is sometimes) alloca.

> And considering Stroustrup's opposition to that, may never happen.

I don't rememver ever having seen Stroustrup express an opinion one way
or the other.  I do know that he has recently encouraged some efforts
for C compatibility, so presumably, he would favor variable length
arrays, at least in the contexts where they are legal in C.

> I set the bufferlen to 1 because I know I can allocate that much; I'm
> not sure about 0 bytes.  But getting to the point, would there not be
> some benefit to programming the basic_string so that it allocates from
> the same memory as basic_string a small buffer of its own as part of
> the class?  Basically, just adding char abuffer[16]; or something like
> it to the class.

That's what most implementations today do.

And that is where the correct solution lies.  Not in breaking
encapsulation, and writing the code in an unnatural way.

Of course, the standard C++ idiom for this would be to use something
like: toInt( effDate.begin() + 9, effDate.begin() + 11 ), etc.  With
iterators.  This also avoids any unnecessary allocations.  But the real
argument in favor of it is that it is idiomatic C++ -- real C++
programmers feel at home with it (and others find it completely wierd).

--
James Kanze                           mailto:jkanze@caicheuvreux.com
Conseils en informatique orient   e objet/
                    Beratung in objektorientierter Datenverarbeitung
---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html                       ]