James Kanze wrote:
> Greg Herlihy wrote:
> > James Kanze wrote:
> > > Ben Hutchings wrote:
> > > >                                                              I would
> > > > like to see mktemp() go the same way but as yet there is no alternative
> > > > in the standard library.
>
> > > Just a nit, but there is no function mktemp() in standard C++,
> > > standard C or in Posix.  The only place I could find it was in
> > > the Open Systems standard, where it is marked as legacy (which
> > > if I understand correctly, is the same thing as deprecated).
>
> > The function's name is "mkstemp" and can be found in the POSIX 2001
> > standard.
>
> Not in my copy.  It's in the unified Unix standard, but it is
> clearly marked as an Open Group extension.  In sum, the same as
> mktemp, except that it isn't marked as "legacy".  And I don't
> think it was the function Ben was referring to.

In 2001 Edition of POSIX 1003.1, mkstemp() was included as an XSI
extension. In the 2004 Edition (ISO/IEC 9945-2003), mkstemp() is no
longer an extension, but a member of the Base POSIX interface.

Since the latest draft of the C++ Standard includes ISO/IEC 9945-2003
(the POSIX operating system interface) as a normative reference, we can
answer the original question in the affirmative: there will be an
atomic create/open file function (incorporated by reference) in the
next C++ Standard.

Greg

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Greg Herlihy wrote:
> In 2001 Edition of POSIX 1003.1, mkstemp() was included as an XSI
> extension. In the 2004 Edition (ISO/IEC 9945-2003), mkstemp() is no
> longer an extension, but a member of the Base POSIX interface.
>
> Since the latest draft of the C++ Standard includes ISO/IEC 9945-2003
> (the POSIX operating system interface) as a normative reference, we can
> answer the original question in the affirmative: there will be an
> atomic create/open file function (incorporated by reference) in the
> next C++ Standard.

I have another (nonportable) suggestion for the OP that entirely
sidesteps the issue - Unices have an 'mcookie' program that generates a
random string. Create a file with that name - it's practically
impossible that you'll have a clash. When you're done with it, you can
rename it. I'm not sure whether renaming is atomic.

Andrei

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Greg Herlihy wrote:
> James Kanze wrote:
> > Greg Herlihy wrote:
> > > James Kanze wrote:
> > > > Ben Hutchings wrote:

> > > > > like to see mktemp() go the same way but as yet there
> > > > > is no alternative in the standard library.

> > > > Just a nit, but there is no function mktemp() in standard C++,
> > > > standard C or in Posix.  The only place I could find it was in
> > > > the Open Systems standard, where it is marked as legacy (which
> > > > if I understand correctly, is the same thing as deprecated).

> > > The function's name is "mkstemp" and can be found in the POSIX 2001
> > > standard.

> > Not in my copy.  It's in the unified Unix standard, but it is
> > clearly marked as an Open Group extension.  In sum, the same as
> > mktemp, except that it isn't marked as "legacy".  And I don't
> > think it was the function Ben was referring to.

> In 2001 Edition of POSIX 1003.1, mkstemp() was included as an XSI
> extension. In the 2004 Edition (ISO/IEC 9945-2003), mkstemp() is no
> longer an extension, but a member of the Base POSIX interface.

In the on line edition of the standard, the function is still
clearly marked as an Open Group extension.  I don't think you're
the only person to have claimed otherwise here, but the official
documentation doesn't seem to agree with you.

> Since the latest draft of the C++ Standard includes ISO/IEC 9945-2003
> (the POSIX operating system interface) as a normative reference, we can
> answer the original question in the affirmative: there will be an
> atomic create/open file function (incorporated by reference) in the
> next C++ Standard.

The latest draft of the C++ Standard officially refers to the
Posix standard.  It does NOT include the entire Posix standard
by reference, and unless there is explicit text in the body of
the standard saying that such and such a paragraph, function or
whatever is explicitly included by reference, the functionality
is not guaranteed by the C++ standard.

In particular, neither the official version nor the latest draft
of the C++ standard require an atomic create functionality.
(The only normative references to Posix which I could find in
the draft were with regards to extended regular expressions,
which are as defined in Posix.)

--
James Kanze (Gabi Software)            email: james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

"Andrei Alexandrescu See Website For Email wrote:
> Greg Herlihy wrote:
> > In 2001 Edition of POSIX 1003.1, mkstemp() was included as an XSI
> > extension. In the 2004 Edition (ISO/IEC 9945-2003), mkstemp() is no
> > longer an extension, but a member of the Base POSIX interface.

> > Since the latest draft of the C++ Standard includes ISO/IEC 9945-2003
> > (the POSIX operating system interface) as a normative reference, we can
> > answer the original question in the affirmative: there will be an
> > atomic create/open file function (incorporated by reference) in the
> > next C++ Standard.

> I have another (nonportable) suggestion for the OP that entirely
> sidesteps the issue - Unices have an 'mcookie' program that generates a
> random string.

Do they?  Neither Posix nor the Open Group seem to know about
it.  (I think it's part of X.  Which means that it won't be
present if X isn't installed.)

> Create a file with that name - it's practically
> impossible that you'll have a clash.

Again, not standard, but present on the Unices I'm familiar with
(Solaris and Linux): /dev/random.  (I've started using it when
generating my header guards:-).)  But the problem isn't so much
avoiding an accidental class---cobbling a name together using
the machine name, the pid, the uid, and the current timestamp is
largely sufficient for that.  (In fact, in practice, just using
the pid has proved sufficient.)  The issue here is security;
someone intentionally trying to mess with your stuff.

> When you're done with it, you can
> rename it. I'm not sure whether renaming is atomic.

It's not.  If the two names are on the same filesystem, however,
the effect is just as good: you atomically create a hard link
with the first name, then atomically remove the original name.

But of course, if you're willing to use Unix specific features,
open with the flags O_CREAT | O_EXCL is the simplest solution.

--
James Kanze (Gabi Software)            email: james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

James Kanze wrote:

> Greg Herlihy wrote:
>> James Kanze wrote:
>> > Ben Hutchings wrote:
>> > >                                                              I would
>> > > like to see mktemp() go the same way but as yet there is no
>> > > alternative in the standard library.
>
>> > Just a nit, but there is no function mktemp() in standard C++,
>> > standard C or in Posix.  The only place I could find it was in
>> > the Open Systems standard, where it is marked as legacy (which
>> > if I understand correctly, is the same thing as deprecated).
>
>> The function's name is "mkstemp" and can be found in the POSIX 2001
>> standard.
>
> Not in my copy.  It's in the unified Unix standard, but it is
> clearly marked as an Open Group extension.  In sum, the same as
> mktemp, except that it isn't marked as "legacy".  And I don't
> think it was the function Ben was referring to.

You are looking at the 1996 edition?  It was moved into Posix with IEEE Std
1003.1-2001.  See http://ieeexplore.ieee.org/iel5/7686/21000/00974400.pdf

>
>> > The C standard (and the C++ standard, by inclusion) has two
>> > functions involving temporary files: tmpfile() (which presumably
>> > is atomic, if the OS permits it, and can be implemented to be
>> > perfectly secure, even if this isn't normally the case), and
>> > tmpnam(), which is what I use, but which does require some
>> > system specific code behind it to be safe.
>
>> Neither tmpfile() nor tempnam() are considered to be safe nor to be
>> free of race conditions.
>
> The safety of tmpfile depends on the implementation.  There's
> nothing in the standard which prevents it from being safe, even
> if most implementations don't do anything particularly
> intelligent.

Right, tmpfile() ought to be fine.  The Linux man pages for tmpnam() and
tempnam() include big warnings never to use them.

>
> I don't know of a function named tempnam().  If you meant
> tmpnam(), the safety depends on what you do with the generated
> name.  It's true that there are race conditions, but these can
> be detected and handled by the user.

tempnam() is another one of those pesky functions that was added to POSIX
after 1996;)

IIRC, the big problem with races in tempnam and tmpnam is that you cannot
check that a pathname is not a symlink and open it in a single atomic
operation.  I don't fully understand the implications of this though, or
how that would be exploited.

>
>> Furhermore, neither routine offers the ability
>> to set permissions on the created file, so the effective security
>> provided by either is close to none.
>
> More correctly, the user cannot choose the level of security he
> wants.  An implementation can set the permissions as restrictive
> as it wants; a quality implementation will set them as
> restrictive as possible.  (Under Unix, for example, an
> implementation should use open with the flags O_CREAT | O_EXCL,
> and mode 0.)
>
>> For these reasons, mkstemp() was
>> added to POSIX as a more secure routine to create temporary files
>> securely and free of any race condition.
>
> About the only differences I can see between mkstemp (which is
> not Posix, but only Open Group) are 1) mkstemp doesn't delete
> the file when it is finally closed, and 2) mkstemp allows you do
> define part of the name.  Now I agree that there are times when
> you might want one or the other of these features, but neither
> are necessary for security, nor have anything to do with race
> conditions.

You are allowed to immediately unlink() the file returned by mkstemp(), so
your point 1) is moot.

The whole point of mkstemp was to avoid races in tmpnam.  I used to be under
the impression it was basically impossible to fix without using mkstemp(),
but after reading a bit more, I must admit it is no longer clear to me
whether mkstemp() is essentially equivalent to tmpnam() followed by an
open() with O_EXCL.

>
> On a system more intelligent than Unix, tmpfile() could actually
> create a file that couldn't even be seen, much less named or
> opened by anyone else.  With an intelligent implementation
> under Unix, it could only be seen for a very, very short time;
> presumably, the intent is that the implementation will delete
> the directory entry as soon as it has the file descriptor.
> Ideal for security, although not necessarily the best thing for
> debugging.

I've never understood why UNIX lets to open a file then immediately unlink
it, but doesn't let you construct a file that is anonymous from the start.
Debugging isn't an issue, at least on linux, as it will always be
accessible as /proc/$PID/fd/$N

>
>>  (Generally speaking,
>> > the best solution is to use the results of tmpnam() to create a
>> > directory, with access rights positionned so that it is not
>> > accessible to other users.  The lack of any standard support for
>> > directories and access rights is probably something that should
>> > be addressed.)
>
>> It was addressed five years ago.
>
> Where?  It's been part of Posix for much, much longer than that,
> but there's still nothing in C++ nor C which even admits that
> directories exist, or that files have access rights.
>
> Currently, my code only has to work under Solaris and Linux; for
> temporary files, I generally use tmpnam() and mkdir() once, at
> the beginning, and create all of my temporary files in the
> directory thus created.  Generally, I'll explicitly invoke
> `system( "rm -r " + tmpdirname )' at normal program
> termination, with an option which suppresses this (and displays
> the tmpdirname on cerr) for debugging purposes.  If I really
> want to be sure that the directory is deleted, regardless of
> what happens later, I'll set up a pipe to `sort >/dev/null ; rm
> -fr tmpdirname' as soon as I've created the directory.

This is probably fine in context, but that would be dangerous for an suid
program to do this, unless you can be sure that tmpdirname is not
accessible to users.

Cheers,
Ian

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Ian McCulloch wrote:
> James Kanze wrote:
> > Greg Herlihy wrote:
> >> James Kanze wrote:
> >> > Ben Hutchings wrote:
> >> > >                                                              I would
> >> > > like to see mktemp() go the same way but as yet there is no
> >> > > alternative in the standard library.

> >> > Just a nit, but there is no function mktemp() in standard C++,
> >> > standard C or in Posix.  The only place I could find it was in
> >> > the Open Systems standard, where it is marked as legacy (which
> >> > if I understand correctly, is the same thing as deprecated).

> >> The function's name is "mkstemp" and can be found in the POSIX 2001
> >> standard.

> > Not in my copy.  It's in the unified Unix standard, but it is
> > clearly marked as an Open Group extension.  In sum, the same as
> > mktemp, except that it isn't marked as "legacy".  And I don't
> > think it was the function Ben was referring to.

> You are looking at the 1996 edition?  It was moved into Posix with IEEE Std
> 1003.1-2001.  See http://ieeexplore.ieee.org/iel5/7686/21000/00974400.pdf

I don't think so.  I'm looking at a copy I downloaded from
http://www.unix.org/ a year or two ago, and the top of the page
says "IEEE Std 1003.1, 2003 Edition".  I can't access the site
you mention because I don't have an active IEEE Web account.  If
there are two different versions of the same standard floating
around, both from official organizations, it's a very worrisome
situation.

> >> > The C standard (and the C++ standard, by inclusion) has two
> >> > functions involving temporary files: tmpfile() (which presumably
> >> > is atomic, if the OS permits it, and can be implemented to be
> >> > perfectly secure, even if this isn't normally the case), and
> >> > tmpnam(), which is what I use, but which does require some
> >> > system specific code behind it to be safe.

> >> Neither tmpfile() nor tempnam() are considered to be safe nor to be
> >> free of race conditions.

> > The safety of tmpfile depends on the implementation.  There's
> > nothing in the standard which prevents it from being safe, even
> > if most implementations don't do anything particularly
> > intelligent.

> Right, tmpfile() ought to be fine.  The Linux man pages for
> tmpnam() and tempnam() include big warnings never to use them.

Which is silly.

> > I don't know of a function named tempnam().  If you meant
> > tmpnam(), the safety depends on what you do with the generated
> > name.  It's true that there are race conditions, but these can
> > be detected and handled by the user.

> tempnam() is another one of those pesky functions that was added to POSIX
> after 1996;)

> IIRC, the big problem with races in tempnam and tmpnam is that you cannot
> check that a pathname is not a symlink and open it in a single atomic
> operation.  I don't fully understand the implications of this though, or
> how that would be exploited.

If you use the name returned by tmpnam() to create a normal
file, I can see a lot of ways it could be a problem.  Using it
to create a directory is safe, however (at least as far as I can
tell).  Using it on a dedicated machine, where the only programs
running are those that you explicitly run, can be safe as well,
and that covers a lot of machines I've had to target.

In the case of tempnam, of course, you specify the directory; if
you specify a directory where you are the only person who has
access rights, I think you should be safe as well.

> >> Furhermore, neither routine offers the ability
> >> to set permissions on the created file, so the effective security
> >> provided by either is close to none.

> > More correctly, the user cannot choose the level of security he
> > wants.  An implementation can set the permissions as restrictive
> > as it wants; a quality implementation will set them as
> > restrictive as possible.  (Under Unix, for example, an
> > implementation should use open with the flags O_CREAT | O_EXCL,
> > and mode 0.)

> >> For these reasons, mkstemp() was
> >> added to POSIX as a more secure routine to create temporary files
> >> securely and free of any race condition.

> > About the only differences I can see between mkstemp (which is
> > not Posix, but only Open Group) are 1) mkstemp doesn't delete
> > the file when it is finally closed, and 2) mkstemp allows you do
> > define part of the name.  Now I agree that there are times when
> > you might want one or the other of these features, but neither
> > are necessary for security, nor have anything to do with race
> > conditions.

> You are allowed to immediately unlink() the file returned by mkstemp(), so
> your point 1) is moot.

Good point.  Under Unix, anyway.  Under a more intelligent OS
(and I don't mean Windows), you wouldn't even have this small
window where the file didn't get deleted.  Most of the systems
I've worked with in the past has a system request to create a
temporary file, the system ensured that it would be deleted when
the returned file descriptor (or whatever the system used) was
closed, for whatever reason, and that the name or the existance
of the file wasn't visible from outside your process, and the
file itself wasn't nameable.  Just what the doctor ordered for
tmpfile(), in sum.

> The whole point of mkstemp was to avoid races in tmpnam.  I used to be under
> the impression it was basically impossible to fix without using mkstemp(),
> but after reading a bit more, I must admit it is no longer clear to me
> whether mkstemp() is essentially equivalent to tmpnam() followed by an
> open() with O_EXCL.

Under Unix, I imagine that that's about it.  The difference is
that mkstemp() manages the race, and not you; as far as the user
is concerned, there is no race.  (Of course, the standard makes
no guarantees about how long the call to mkstemp will last.
Statistically, however, I'd be surprised if you ever needed more
than two or three tries.)

> > On a system more intelligent than Unix, tmpfile() could actually
> > create a file that couldn't even be seen, much less named or
> > opened by anyone else.  With an intelligent implementation
> > under Unix, it could only be seen for a very, very short time;
> > presumably, the intent is that the implementation will delete
> > the directory entry as soon as it has the file descriptor.
> > Ideal for security, although not necessarily the best thing for
> > debugging.

> I've never understood why UNIX lets to open a file then immediately unlink
> it, but doesn't let you construct a file that is anonymous from the start.

History, probably.  That's the usual excuse.  (FWIW: all of the
systems I'd encountered before Unix, except CP/M, which arguably
isn't an OS anyway, offered this functionality.  The fact that
Unix allowed you to delete the file immediately was originally
more a side effect of other aspects of how the file system was
organized, and not something created explicitly with the idea of
ensuring the deletion of temporary files.)

> Debugging isn't an issue, at least on linux, as it will always be
> accessible as /proc/$PID/fd/$N

Most of the time, for debugging, I want the file to still be
there (and accessible) after the core dump:-).

> >>  (Generally speaking,
> >> > the best solution is to use the results of tmpnam() to create a
> >> > directory, with access rights positionned so that it is not
> >> > accessible to other users.  The lack of any standard support for
> >> > directories and access rights is probably something that should
> >> > be addressed.)

> >> It was addressed five years ago.

> > Where?  It's been part of Posix for much, much longer than that,
> > but there's still nothing in C++ nor C which even admits that
> > directories exist, or that files have access rights.

> > Currently, my code only has to work under Solaris and Linux; for
> > temporary files, I generally use tmpnam() and mkdir() once, at
> > the beginning, and create all of my temporary files in the
> > directory thus created.  Generally, I'll explicitly invoke
> > `system( "rm -r " + tmpdirname )' at normal program
> > termination, with an option which suppresses this (and displays
> > the tmpdirname on cerr) for debugging purposes.  If I really
> > want to be sure that the directory is deleted, regardless of
> > what happens later, I'll set up a pipe to `sort >/dev/null ; rm
> > -fr tmpdirname' as soon as I've created the directory.

> This is probably fine in context, but that would be dangerous for an suid
> program to do this, unless you can be sure that tmpdirname is not
> accessible to users.

When creating a directory, you specify the access rights; for
the temporary file directory, these are always 0700 in my code.

--
James Kanze (GABI Software)             email:james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

On 2006-12-04, Greg Herlihy <greghe@pacbell.net> wrote:
> Fred Long wrote:
>> Are there any intentions in the upcoming revision to the C++ standard to
>> solve the problem that makes it impossible in C++ to *atomically* open
>> a file for writing that does not already exist?
>
> I don't think it would be wise for C++ to start providing security
> guarantees that are already (and more appropriately) handled at the OS
> level.

Many OS facilities are, thankfully, exposed in a standard way in the
C and C++ libraries.  What makes you think that atomic file creation
- which is not only a security issue but often necessary for avoiding
the risk of data loss - should not (or cannot) be a standard feature?

> Even in the absence of OS support, the C++ Standard should still not
> enter the business of providing secure, OS-level operations. After all,
> a C++ implementor is a language and library expert and so is unlikely
> to have expertise in OS security as well. But it is exactly that kind
> of OS-level expertise that would be needed to implement an atomic file
> open and create operation in a way that would not be
> worse-than-useless.

I find that almost insulting to the C++ implementers here.  Even if it
is quite difficult, it's also widely useful.  How do you think they
implement tempfile() anyway?

> So not everything that could conceivably be added
> to the C++ Standard - necessarily should be added. Some features -
> particular those that involve security - are best implemented elsewhere
> and implemented by individuals who best know what they are doing.

IT security cannot be achieved though the use of secure "features" by
some small group of experts.  (SSL and firewalls didn't fix "the"
Internet security problem.)  Security concerns are unfortunately
pervasive in applications that aren't part of closed trusted systems,
and there are few of them.

There are however, features that are insecure in their very design.
Except in very restricted cases, gets() is an invitation to code
injection, and I hope everyone now knows to avoid using it.  I would
like to see mktemp() go the same way but as yet there is no alternative
in the standard library.

Ben.

--
Ben Hutchings
I haven't lost my mind; it's backed up on tape somewhere.

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Ben Hutchings wrote:

>                                                              I would
> like to see mktemp() go the same way but as yet there is no alternative
> in the standard library.

Just a nit, but there is no function mktemp() in standard C++,
standard C or in Posix.  The only place I could find it was in
the Open Systems standard, where it is marked as legacy (which
if I understand correctly, is the same thing as deprecated).

The C standard (and the C++ standard, by inclusion) has two
functions involving temporary files: tmpfile() (which presumably
is atomic, if the OS permits it, and can be implemented to be
perfectly secure, even if this isn't normally the case), and
tmpnam(), which is what I use, but which does require some
system specific code behind it to be safe.  (Generally speaking,
the best solution is to use the results of tmpnam() to create a
directory, with access rights positionned so that it is not
accessible to other users.  The lack of any standard support for
directories and access rights is probably something that should
be addressed.)

--
James Kanze (GABI Software)             email:james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Ben Hutchings wrote:
> On 2006-12-04, Greg Herlihy <greghe@pacbell.net> wrote:
> > Fred Long wrote:
> >> Are there any intentions in the upcoming revision to the C++ standard to
> >> solve the problem that makes it impossible in C++ to *atomically* open
> >> a file for writing that does not already exist?
> >
> > I don't think it would be wise for C++ to start providing security
> > guarantees that are already (and more appropriately) handled at the OS
> > level.
>
> Many OS facilities are, thankfully, exposed in a standard way in the
> C and C++ libraries.  What makes you think that atomic file creation
> - which is not only a security issue but often necessary for avoiding
> the risk of data loss - should not (or cannot) be a standard feature?

Atomic file creation should be (and already is) a standard feature -
but that standard happens to be POSIX 2001. I don't see much practical
benefit for C++ to offer an alternative to - or an imitation of -
mkstemp().

> > Even in the absence of OS support, the C++ Standard should still not
> > enter the business of providing secure, OS-level operations. After all,
> > a C++ implementor is a language and library expert and so is unlikely
> > to have expertise in OS security as well. But it is exactly that kind
> > of OS-level expertise that would be needed to implement an atomic file
> > open and create operation in a way that would not be
> > worse-than-useless.
>
> I find that almost insulting to the C++ implementers here.  Even if it
> is quite difficult, it's also widely useful.  How do you think they
> implement tempfile() anyway?

There are many people in this world (myself included) who are not
computer security experts - and most of them are still able to lead
fulfilling and productive lives nonetheless. So I don't see how anyone
who is not a security expert would be insulted by not being called a
security expert. In fact, all that I stated was that its likely that
security experts constitute a minority of C++ implementors. And if that
is not the case - if it turns out that most, if not all - C++
implementors are computer security experts than my conjecture would
have been wrong. But the underlying point is that: the greater the
number of disciplines that a C++ implementor must master - the more
likely that the resources of an implementor will be stretched thin.

> > So not everything that could conceivably be added
> > to the C++ Standard - necessarily should be added. Some features -
> > particular those that involve security - are best implemented elsewhere
> > and implemented by individuals who best know what they are doing.
>
> IT security cannot be achieved though the use of secure "features" by
> some small group of experts.  (SSL and firewalls didn't fix "the"
> Internet security problem.)  Security concerns are unfortunately
> pervasive in applications that aren't part of closed trusted systems,
> and there are few of them.

It makes much more sense to have one routine per secure operation, than
to offer a wide variety of functions to choose from. The latter is
simply harder to assess - and the more code, the greater the likelihood
that a vulnerability exists therein.

> There are however, features that are insecure in their very design.
> Except in very restricted cases, gets() is an invitation to code
> injection, and I hope everyone now knows to avoid using it.  I would
> like to see mktemp() go the same way but as yet there is no alternative
> in the standard library.

The safer replacement for tmpnam(), tempfile() and related routines are
the mkstemp(), mkstemps(), mkdtemp() set of functions.

Greg

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

James Kanze wrote:
> Ben Hutchings wrote:
>
> >                                                              I would
> > like to see mktemp() go the same way but as yet there is no alternative
> > in the standard library.
>
> Just a nit, but there is no function mktemp() in standard C++,
> standard C or in Posix.  The only place I could find it was in
> the Open Systems standard, where it is marked as legacy (which
> if I understand correctly, is the same thing as deprecated).

The function's name is "mkstemp" and can be found in the POSIX 2001
standard.

> The C standard (and the C++ standard, by inclusion) has two
> functions involving temporary files: tmpfile() (which presumably
> is atomic, if the OS permits it, and can be implemented to be
> perfectly secure, even if this isn't normally the case), and
> tmpnam(), which is what I use, but which does require some
> system specific code behind it to be safe.

Neither tmpfile() nor tempnam() are considered to be safe nor to be
free of race conditions. Furhermore, neither routine offers the ability
to set permissions on the created file, so the effective security
provided by either is close to none. For these reasons, mkstemp() was
added to POSIX as a more secure routine to create temporary files
securely and free of any race condition.

 (Generally speaking,
> the best solution is to use the results of tmpnam() to create a
> directory, with access rights positionned so that it is not
> accessible to other users.  The lack of any standard support for
> directories and access rights is probably something that should
> be addressed.)

It was addressed five years ago.

Greg

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

On 2006-12-06, Greg Herlihy <greghe@pacbell.net> wrote:
> Ben Hutchings wrote:
>> On 2006-12-04, Greg Herlihy <greghe@pacbell.net> wrote:
>> > Fred Long wrote:
>> >> Are there any intentions in the upcoming revision to the C++ standard to
>> >> solve the problem that makes it impossible in C++ to *atomically* open
>> >> a file for writing that does not already exist?
>> >
>> > I don't think it would be wise for C++ to start providing security
>> > guarantees that are already (and more appropriately) handled at the OS
>> > level.
>>
>> Many OS facilities are, thankfully, exposed in a standard way in the
>> C and C++ libraries.  What makes you think that atomic file creation
>> - which is not only a security issue but often necessary for avoiding
>> the risk of data loss - should not (or cannot) be a standard feature?
>
> Atomic file creation should be (and already is) a standard feature -
> but that standard happens to be POSIX 2001.

It cannot have escaped your notice that there are many popular operating
systems that are not based on Unix, and implementations of POSIX on other
systems can represent a vast amount of work.  There is room for some
subset, which presumably is why the C and C++ standard libraries include
remove(), rename(), and some other OS facilities that were originally
Unix features.

> I don't see much practical benefit for C++ to offer an alternative to
> - or an imitation of - mkstemp().

A C or C++ standard version would of course yield a streambuf or FILE *,
not a file descriptor, and on POSIX systems would probably be
implemented using mkstemp().

The benefit, I hope, would be that new C++ programmers could be taught
to use standard library facilities without the need for either
encouraging insecure programming techniques or for introducing platform
specifics.

> >> > Even in the absence of OS support, the C++ Standard should still not
>> > enter the business of providing secure, OS-level operations. After all,
>> > a C++ implementor is a language and library expert and so is unlikely
>> > to have expertise in OS security as well. But it is exactly that kind
>> > of OS-level expertise that would be needed to implement an atomic file
>> > open and create operation in a way that would not be
>> > worse-than-useless.
>>
>> I find that almost insulting to the C++ implementers here.  Even if it
>> is quite difficult, it's also widely useful.  How do you think they
>> implement tempfile() anyway?
>
> There are many people in this world (myself included) who are not
> computer security experts - and most of them are still able to lead
> fulfilling and productive lives nonetheless. So I don't see how anyone
> who is not a security expert would be insulted by not being called a
> security expert.

You referred to "expertise in OS security" and "OS-level expertise".  I
believe most library implementers have experts in the OSs they support
- this seems to me to be essential - and are aware of the potential for
introducing or mitigating security risks in the programs that use them.
I understand that most library implementers would probably not be ready
to implement, say, sandboxing or cryptography in the standard library.
However, the suggestion that they may be so security-ignorant that they
could not even arrange to create a file atomically seems like an insult
to their competence.

There might be some platforms that currently support C++ and on which it
is simply impossible to create a file atomically.  This could be a good
reason to oppose standardisation of such a feature.

<snip>
>> > So not everything that could conceivably be added
>> > to the C++ Standard - necessarily should be added. Some features -
>> > particular those that involve security - are best implemented elsewhere
>> > and implemented by individuals who best know what they are doing.
>>
>> IT security cannot be achieved though the use of secure "features" by
>> some small group of experts.  (SSL and firewalls didn't fix "the"
>> Internet security problem.)  Security concerns are unfortunately
>> pervasive in applications that aren't part of closed trusted systems,
>> and there are few of them.
>
> It makes much more sense to have one routine per secure operation, than
> to offer a wide variety of functions to choose from.  The latter is
> simply harder to assess - and the more code, the greater the likelihood
> that a vulnerability exists therein.
<snip>

A few wrapper functions in a few standard library implementation are a
*lot* easier to audit than thousands of applications that try to do the
same.

Ben.

--
Ben Hutchings
Never attribute to conspiracy what can adequately be explained by stupidity.

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

On 2006-12-06, James Kanze <james.kanze@gmail.com> wrote:
> Ben Hutchings wrote:
>
>>                                                              I would
>> like to see mktemp() go the same way but as yet there is no alternative
>> in the standard library.
>
> Just a nit, but there is no function mktemp() in standard C++,
> standard C or in Posix.

I remembered that later.  I should have said tmpnam().

<snip>
> The C standard (and the C++ standard, by inclusion) has two
> functions involving temporary files: tmpfile() (which presumably
> is atomic, if the OS permits it, and can be implemented to be
> perfectly secure, even if this isn't normally the case),

tmpfile() doesn't represent any inherent security problem, but the
lack of a name is a problem.  You can only keep as many anonymous
temporary files as you can have open files.  The same doesn't apply
to named temporary files.

> and tmpnam(), which is what I use, but which does require some
> system specific code behind it to be safe.

Which is why I don't even bother with tmpnam().  My latest temporary
directory class (the previous two are locked up in previous employers)
uses mkdtemp() for directory creation.  If and when I need greater
portability I'll have to change that, of course.  At least glib, which
is fairly widely ported, provides one of the missing pieces: the name
of the preferred directory for temporary files.

> (Generally speaking,
> the best solution is to use the results of tmpnam() to create a
> directory, with access rights positionned so that it is not
> accessible to other users.  The lack of any standard support for
> directories and access rights is probably something that should
> be addressed.)

I believe there's some work on standardising the Boost.Filesystem
library, which provides directories and hierarchical paths.

Ben.

--
Ben Hutchings
Never attribute to conspiracy what can adequately be explained by stupidity.

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Greg Herlihy wrote:
> James Kanze wrote:
> > Ben Hutchings wrote:
> > >                                                              I would
> > > like to see mktemp() go the same way but as yet there is no alternative
> > > in the standard library.

> > Just a nit, but there is no function mktemp() in standard C++,
> > standard C or in Posix.  The only place I could find it was in
> > the Open Systems standard, where it is marked as legacy (which
> > if I understand correctly, is the same thing as deprecated).

> The function's name is "mkstemp" and can be found in the POSIX 2001
> standard.

Not in my copy.  It's in the unified Unix standard, but it is
clearly marked as an Open Group extension.  In sum, the same as
mktemp, except that it isn't marked as "legacy".  And I don't
think it was the function Ben was referring to.

> > The C standard (and the C++ standard, by inclusion) has two
> > functions involving temporary files: tmpfile() (which presumably
> > is atomic, if the OS permits it, and can be implemented to be
> > perfectly secure, even if this isn't normally the case), and
> > tmpnam(), which is what I use, but which does require some
> > system specific code behind it to be safe.

> Neither tmpfile() nor tempnam() are considered to be safe nor to be
> free of race conditions.

The safety of tmpfile depends on the implementation.  There's
nothing in the standard which prevents it from being safe, even
if most implementations don't do anything particularly
intelligent.

I don't know of a function named tempnam().  If you meant
tmpnam(), the safety depends on what you do with the generated
name.  It's true that there are race conditions, but these can
be detected and handled by the user.

> Furhermore, neither routine offers the ability
> to set permissions on the created file, so the effective security
> provided by either is close to none.

More correctly, the user cannot choose the level of security he
wants.  An implementation can set the permissions as restrictive
as it wants; a quality implementation will set them as
restrictive as possible.  (Under Unix, for example, an
implementation should use open with the flags O_CREAT | O_EXCL,
and mode 0.)

> For these reasons, mkstemp() was
> added to POSIX as a more secure routine to create temporary files
> securely and free of any race condition.

About the only differences I can see between mkstemp (which is
not Posix, but only Open Group) are 1) mkstemp doesn't delete
the file when it is finally closed, and 2) mkstemp allows you do
define part of the name.  Now I agree that there are times when
you might want one or the other of these features, but neither
are necessary for security, nor have anything to do with race
conditions.

On a system more intelligent than Unix, tmpfile() could actually
create a file that couldn't even be seen, much less named or
opened by anyone else.  With an intelligent implementation
under Unix, it could only be seen for a very, very short time;
presumably, the intent is that the implementation will delete
the directory entry as soon as it has the file descriptor.
Ideal for security, although not necessarily the best thing for
debugging.

>  (Generally speaking,
> > the best solution is to use the results of tmpnam() to create a
> > directory, with access rights positionned so that it is not
> > accessible to other users.  The lack of any standard support for
> > directories and access rights is probably something that should
> > be addressed.)

> It was addressed five years ago.

Where?  It's been part of Posix for much, much longer than that,
but there's still nothing in C++ nor C which even admits that
directories exist, or that files have access rights.

Currently, my code only has to work under Solaris and Linux; for
temporary files, I generally use tmpnam() and mkdir() once, at
the beginning, and create all of my temporary files in the
directory thus created.  Generally, I'll explicitly invoke
`system( "rm -r " + tmpdirname )' at normal program
termination, with an option which suppresses this (and displays
the tmpdirname on cerr) for debugging purposes.  If I really
want to be sure that the directory is deleted, regardless of
what happens later, I'll set up a pipe to `sort >/dev/null ; rm
-fr tmpdirname' as soon as I've created the directory.

--
James Kanze (GABI Software)             email:james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Ben Hutchings wrote:
> On 2006-12-06, James Kanze <james.kanze@gmail.com> wrote:
> > Ben Hutchings wrote:

> >>                                                              I would
> >> like to see mktemp() go the same way but as yet there is no alternative
> >> in the standard library.

> > Just a nit, but there is no function mktemp() in standard C++,
> > standard C or in Posix.

> I remembered that later.  I should have said tmpnam().

> <snip>
> > The C standard (and the C++ standard, by inclusion) has two
> > functions involving temporary files: tmpfile() (which presumably
> > is atomic, if the OS permits it, and can be implemented to be
> > perfectly secure, even if this isn't normally the case),

> tmpfile() doesn't represent any inherent security problem, but the
> lack of a name is a problem.  You can only keep as many anonymous
> temporary files as you can have open files.  The same doesn't apply
> to named temporary files.

But all of the ways to get named temporary files have race
conditions.  And there's no way to get guaranteed automatic
deletion with a named temporary file; if the program terminates
automatically, you may be stuck with it.

In practice, of course, it depends on the application.
Sometimes you need the name for other things, and sometimes, you
don't want the file to automatically disappear, no matter what.
I generally find an option to not remove the temporary files
useful for debugging, for example (but only, obviously, if you
know the names).

> > and tmpnam(), which is what I use, but which does require some
> > system specific code behind it to be safe.

> Which is why I don't even bother with tmpnam().

Well, I need the other system specific code anyway.  And in
practice, where I currently work, portability is restricted to
Solaris and Linux, do I have no problem with using mkdir.

> My latest temporary
> directory class (the previous two are locked up in previous employers)
> uses mkdtemp() for directory creation.

What's that one?  I can't even find it in the Open Group (Unix)
standards.

> If and when I need greater
> portability I'll have to change that, of course.  At least glib, which
> is fairly widely ported, provides one of the missing pieces: the name
> of the preferred directory for temporary files.

Weren't you the person who told me that Posix says that
functions like tmpnam and tmpfile should use $TMPDIR?

I also occasionally use my own class, but mainly to allow
specifying a prefix and/or a suffix.  Being able to specify a
prefix (to the directory) allows getting rid of the temporary
files from several runs with a single rm, provided the prefix is
well chosen, and doesn't conflict with other uses, and the
suffix is useful if e.g. I'm generating C++ code, which I'll
later feed to the compiler.

> > (Generally speaking,
> > the best solution is to use the results of tmpnam() to create a
> > directory, with access rights positionned so that it is not
> > accessible to other users.  The lack of any standard support for
> > directories and access rights is probably something that should
> > be addressed.)

> I believe there's some work on standardising the Boost.Filesystem
> library, which provides directories and hierarchical paths.

I'll have to have a look at it.  I'm gradually trying to get
Boost into use here---a difficult job, as some of the developers
are still not using std::string and std::vector, because they
are too advanced:-(.

--
James Kanze (GABI Software)             email:james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Greg Herlihy wrote:
    [...]
> > There are however, features that are insecure in their very design.
> > Except in very restricted cases, gets() is an invitation to code
> > injection, and I hope everyone now knows to avoid using it.  I would
> > like to see mktemp() go the same way but as yet there is no alternative
> > in the standard library.

> The safer replacement for tmpnam(), tempfile() and related routines are
> the mkstemp(), mkstemps(), mkdtemp() set of functions.

None of which are standard.  Not even if you consider Posix.
The first is part of the Open Group common Unix specification.
The other two, I've never even heard of---they aren't Posix,
they aren't common Unix, and they aren't present on Solaris, or
at least, if they are present, they are well hidden, since
there's no man page for them, and the symbols aren't present in
any file in /usr/include.

In my own application code, portability to Solaris and Linux is
all that counts (and some applications are only Solaris or only
Linux), so I can freely use mkstemp (but not the other two) if
it corresponds to my needs.  Not all of my collegues have this
liberty; the client stations are Windows on PC's.  And I suspect
that we're not the only company in the world using PC's under
Windows.

--
James Kanze (GABI Software)             email:james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Are there any intentions in the upcoming revision to the C++ standard to
solve the problem that makes it impossible in C++ to *atomically* open
a file for writing that does not already exist?

(I understand that an early draft of the C++ standard had a mode
noreplace, but that did not make it into the standard.)

Attempts to first check for existence of the file and then open it
suffer from a "Time of Check, Time of Use" (or TOCTOU) vulnerability.

For example:

   ifstream fi("foo.txt");// attempt to open as input file
   if (!fi)
   {
     // file doesn't exist; so it's safe to create it and write to it
     ofstream fo("foo.txt");
     // can now write to fo, e.g., fo << "Some output" << endl;
     ...
   }
   else // file exists; close and handle error
   {
     fi.close();
     // handle error
   }

On a shared multitasking system there is a window of opportunity between
the first call of open and the second call for a malicious attacker to,
for example, create a link with the given filename to an existing file,
so that the existing file is overwritten by the second call of open and
the subsequent writing to the file.

See:

https://www.securecoding.cert.org/confluence/display/cplusplus/FIO01-A.+Do+not+make+assumptions+about+open%28%29+and+file+creation

for more information and possible solutions that are not standard C++.

Thanks,
     Fred Long
     CERT    Coordination Center
     Software Engineering Institute

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Fred Long wrote:
> Are there any intentions in the upcoming revision to the C++
> standard to solve the problem that makes it impossible in C++
> to *atomically* open a file for writing that does not already
> exist?

> (I understand that an early draft of the C++ standard had a mode
> noreplace, but that did not make it into the standard.)

It would be useful, but I think the reason that it didn't make
it into the C standard is that there are systems where it can't
be implemented.  Older variants of Unix, for example.

Of course, it could be specified along the lines of the app
flag, where it is clear that the intent is atomicity if the
system supports it, but the standard doesn't formally require
it.

And finally, of course, there are a lot more options which would
also be usable.  I would find this one useful, but the one I
really need is to require reads and writes to be fully
synchronized, so that data integrity is ensured.  In practice,
on a server, about the only thing you can use filebuf for is
logging; for all other uses, you need to use system level reads
and writes to ensure data integrity.

--
James Kanze (Gabi Software)            email: james.kanze@gmail.com
Conseils en informatique orient   e objet/
                   Beratung in objektorientierter Datenverarbeitung
9 place S   mard, 78210 St.-Cyr-l'   cole, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Fred Long wrote:
> Are there any intentions in the upcoming revision to the C++ standard to
> solve the problem that makes it impossible in C++ to *atomically* open
> a file for writing that does not already exist?
.

I'm not sure about C99, but in C++98 I think the only feature that
acknolodges
the possibility that memory (primary or secondary) can be changed while
a program is executing (by something other than the program itself)
is the "volatile" keyword.  So we could speculate that there is a
feeling
that the "multiple simultaneous excution contexts" issue should not be
addressed piece by piece, but in some more comprehensive way.

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]

Fred Long wrote:
> Are there any intentions in the upcoming revision to the C++ standard to
> solve the problem that makes it impossible in C++ to *atomically* open
> a file for writing that does not already exist?

I don't think it would be wise for C++ to start providing security
guarantees that are already (and more appropriately) handled at the OS
level. For one, there is the problem of overlapping language and OS
standards. For example, the POSIX routine, mkstemp(), both creates and
opens a file in a single call, thereby eliminating any potential race
condition.

Even in the absence of OS support, the C++ Standard should still not
enter the business of providing secure, OS-level operations. After all,
a C++ implementor is a language and library expert and so is unlikely
to have expertise in OS security as well. But it is exactly that kind
of OS-level expertise that would be needed to implement an atomic file
open and create operation in a way that would not be
worse-than-useless. So not everything that could conceivably be added
to the C++ Standard - necessarily should be added. Some features -
particular those that involve security - are best implemented elsewhere
and implemented by individuals who best know what they are doing.

Greg

---
[ comp.std.c++ is moderated.  To submit articles, try just posting with ]
[ your news-reader.  If that fails, use mailto:std-c++@ncar.ucar.edu    ]
[              --- Please see the FAQ before posting. ---               ]
[ FAQ: http://www.comeaucomputing.com/csc/faq.html                      ]