Should Emacs provide a uuid function?

Should Emacs provide a uuid function?

[Leo]

Hello all,

I wonder if Emacs can provide a make-uuid function to generate RFC4122
(http://www.ietf.org/rfc/rfc4122.txt) Universally Unique IDentifiers.
Many elisp packages have already come up with their own either by
calling external tool uuidgen or implementing a dumbed-down version.

Leo

Re: Should Emacs provide a uuid function?

[Chong Yidong]

Leo <sdl.web@gmail.com> writes:

> I wonder if Emacs can provide a make-uuid function to generate RFC4122
> (http://www.ietf.org/rfc/rfc4122.txt) Universally Unique IDentifiers.
> Many elisp packages have already come up with their own either by
> calling external tool uuidgen or implementing a dumbed-down version.

I'd rather not clutter up Emacs with such odds and ends, unless there's
a demonstrable need; OTOH, anyone who cares can just use uuidgen.  Do
you have any specific application in mind?

Re: Should Emacs provide a uuid function?

[Leo]

On 2011-04-25 04:43 +0800, Chong Yidong wrote:
> I'd rather not clutter up Emacs with such odds and ends, unless there's
> a demonstrable need; OTOH, anyone who cares can just use uuidgen.  Do
> you have any specific application in mind?

Calendar, bbdb, org, message etc can make use of this without inventing
their own unique id generator.

Leo

Re: Should Emacs provide a uuid function?

[Stephen J. Turnbull]

Leo writes:
 > On 2011-04-25 04:43 +0800, Chong Yidong wrote:
 > > I'd rather not clutter up Emacs with such odds and ends, unless there's
 > > a demonstrable need; OTOH, anyone who cares can just use uuidgen.  Do
 > > you have any specific application in mind?
 > 
 > Calendar, bbdb, org, message

Leave mail apps out.  uuidgen doesn't produce a syntactically legal
Message-ID by itself (see RFC 5322, Section 3.6.4), and in practice
it's unreasonably long for a component.  It's often convenient to have
a Message-ID of reasonable length that is unique to the host, but not
necessarily universally (until the @domain part is added).

I have to agree with Yidong; there's no good reason not to inline

    (shell-command-to-string "uuidgen")

in any of the applications you've mentioned if you want an
uuidgen-format UUID.

Re: Should Emacs provide a uuid function?

[Ken Raeburn]

On Apr 25, 2011, at 00:37, Stephen J. Turnbull wrote:
> I have to agree with Yidong; there's no good reason not to inline
> 
>    (shell-command-to-string "uuidgen")
> 
> in any of the applications you've mentioned if you want an
> uuidgen-format UUID.

How widespread is the uuidgen program these days?  Several systems I just checked have it, but it's not a program I'd been aware of.  Though, at least one version I ran across generates (by default?) the information-revealing "version 1" format that encodes the machine's MAC address, and while some versions have options to control which form is generated, not all do.

Using the version-4 random-number-based form described in wikipedia, it would probably be faster to just suck in 16 bytes from /dev/urandom or other good pseudo-random number source (which is a different portability question) and format the string directly.  I wouldn't consider the elisp 'random' function to be remotely good enough though.

Ken

Re: Should Emacs provide a uuid function?

[Stephen J. Turnbull]

Ken Raeburn writes:
 > On Apr 25, 2011, at 00:37, Stephen J. Turnbull wrote:
 > > I have to agree with Yidong; there's no good reason not to inline
 > > 
 > >    (shell-command-to-string "uuidgen")
 > > 
 > > in any of the applications you've mentioned if you want an
 > > uuidgen-format UUID.
 > 
 > How widespread is the uuidgen program these days?

Don't know.  The only system I'd worry about is Windows, though.

I wonder if the POP (mail protocol) standard says anything about
this.  ISTR that some kind of unique identifier is part of it.  If so,
I'd imagine something equivalent to uuidgen is available at a hard
drive near you.

 > Though, at least one version I ran across generates (by default?) 
 > the information-revealing "version 1" format that encodes the
 > machine's MAC address, and while some versions have options to
 > control which form is generated, not all do.

Application where you'd care, please.  If any of the data for the
applications mentioned by Leo leaked enough that somebody got hold of
one of the UUIDs, I'd be worried about everything *except* my MAC
address.

 > Using the version-4 random-number-based form described in
 > wikipedia, it would probably be faster

"Speed kills."<wink>  Or should I haul out the old Don Knuth chestnut?

Again, what applications of UUIDs are you thinking of where you'd care?

If somebody wants to put a uuidgen.el into ELPA, that would probably
be cool though.

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Mon, 25 Apr 2011 17:45:24 +0900
> Cc: Emacs Dev <emacs-devel@gnu.org>
> 
> Ken Raeburn writes:
>  > On Apr 25, 2011, at 00:37, Stephen J. Turnbull wrote:
>  > > I have to agree with Yidong; there's no good reason not to inline
>  > > 
>  > >    (shell-command-to-string "uuidgen")
>  > > 
>  > > in any of the applications you've mentioned if you want an
>  > > uuidgen-format UUID.
>  > 
>  > How widespread is the uuidgen program these days?
> 
> Don't know.  The only system I'd worry about is Windows, though.

I wouldn't:

   http://msdn.microsoft.com/en-us/library/aa373930(v=vs.85).aspx
   http://www.microsoft.com/downloads/en/details.aspx?familyid=94551f58-484f-4a8c-bb39-adb270833afc&displaylang=en

Re: Should Emacs provide a uuid function?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > >  > How widespread is the uuidgen program these days?
 > > 
 > > Don't know.  The only system I'd worry about is Windows, though.
 > 
 > I wouldn't:
 > 
 >    http://msdn.microsoft.com/en-us/library/aa373930(v=vs.85).aspx
 >    http://www.microsoft.com/downloads/en/details.aspx?familyid=94551f58-484f-4a8c-bb39-adb270833afc&displaylang=en

Hrm, I don't think I'll even bother to click, just looking at the link
is good enough for me!

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: raeburn@raeburn.org,
>     emacs-devel@gnu.org
> Date: Mon, 25 Apr 2011 19:33:42 +0900
> 
> Eli Zaretskii writes:
> 
>  > >  > How widespread is the uuidgen program these days?
>  > > 
>  > > Don't know.  The only system I'd worry about is Windows, though.
>  > 
>  > I wouldn't:
>  > 
>  >    http://msdn.microsoft.com/en-us/library/aa373930(v=vs.85).aspx
>  >    http://www.microsoft.com/downloads/en/details.aspx?familyid=94551f58-484f-4a8c-bb39-adb270833afc&displaylang=en
> 
> Hrm, I don't think I'll even bother to click

I don't blame you ;-)

> just looking at the link is good enough for me!

The gist of it is that it's now part of the SDK, so no worries there.

Re: Should Emacs provide a uuid function?

[Chong Yidong]

Ken Raeburn <raeburn@raeburn.org> writes:

> I wouldn't consider the elisp 'random' function to be remotely good
> enough though.

You mean, because of the way it's seeded?  I guess we can (should?)
improve it to use /dev/random where that's available.

Re: Should Emacs provide a uuid function?

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

Hello,

When responding to a question about the availability of the uuidgen tool on
Windows Eli Zaretskii provided links to the Windows SDK uuidgen tool and the
Microsoft Exchange Server GUID Generator tool.  The problem is that we
cannot depend on either of these tools.

To begin with, according to the Man page of the GNU/Linux uuidgen tool, the
generated UUID is written "to the standard output."  The Windows SDK uuidgen
tool does not write to standard output; it instead creates a template IDL
file.  The Microsoft Exchange Server GUID Generator is a GUI application
that allows GUIDs to be generated in any of several formats and copied to
the clipboard.  Neither of these programs write output to stdout or support
the -r and -t command line arguments supported by the GNU/Linux uuidgen
tool, therefore they cannot be used as direct replacements of the GNU/Linux
uuidgen tool.

Even if they were functionally identical to the GNU/Linux uuidgen tool we
should not depend on them because both the Windows SDK uuidgen tool and the
Microsoft Exchange Server GUID Generator are developer tools that will not
be installed by default.

Thus the proposal to simply use the following line of code cannot be
expected to work on Windows unless the binary distribution of Emacs for
Windows provides its own implementation of uuidgen.
  (shell-command-to-string "uuidgen")

I have no opinion on whether or not Emacs should provide a uuid function but
I feel strongly that we should not be advising our users to depend on an
external tool that is generally not available on Microsoft Windows.

[-- Attachment #2: Type: text/html, Size: 1695 bytes --]

Re: Should Emacs provide a uuid function?

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

Eli Zaretskii writes:

> The gist of it is that it's now part of the SDK, so no worries
> there.

I have two comments.  First, we should *NEVER* expect our users to have to
install an enormous, developer only, package like the Microsoft SDK just to
use Emacs.  It may be reasonable to expect them to install it to
*BUILD*Emacs but not to
*USE* Emacs.

Second, as I documented in an earlier message, the Microsoft SDK uuidgen
tool is *NOT* a drop in replacement for the GNU/Linux uuidgen tool.

For these two reasons, we *CANNOT* depend on the Microsoft SDK uuidgen tool.

[-- Attachment #2: Type: text/html, Size: 708 bytes --]

Re: Should Emacs provide a uuid function?

[Chong Yidong]

Ben Key <bkey76@gmail.com> writes:

> First, we should NEVER expect our users to have to install an
> enormous, developer only, package like the Microsoft SDK just to use
> Emacs.
>
> Second, as I documented in an earlier message, the Microsoft SDK
> uuidgen tool is NOT a drop in replacement for the GNU/Linux uuidgen
> tool.

Fair enough.  So we come back to the question of how important uuid
generation is.  Suppose someone writes a Lisp implementation.  Whether
or not it's acceptable for Emacs core depends on whether there are real
uses (otherwise we could always add it to elpa.gnu.org, of course).

Leo mentioned calendar, bbdb, and org, three parts of Emacs that I don't
use heavily.  Could anyone elaborate?  Would a uuid function prove
immediately useful for these applications, or is it a case of "it might
or might not be useful"?

Re: Should Emacs provide a uuid function?

[Leo]

On 2011-04-27 06:51 +0800, Chong Yidong wrote:
> Fair enough.  So we come back to the question of how important uuid
> generation is.  Suppose someone writes a Lisp implementation.  Whether
> or not it's acceptable for Emacs core depends on whether there are real
> uses (otherwise we could always add it to elpa.gnu.org, of course).

At least two elisp (partial) implementations of uuid¹². They might not
be fully compliant with RFC4122.

> Leo mentioned calendar, bbdb, and org, three parts of Emacs that I
>don't use heavily. Could anyone elaborate? Would a uuid function prove
>immediately useful for these applications, or is it a case of "it might
>or might not be useful"?

`unique id' is useful as we have seen in many places in the elisp files
in Emacs. SASL.el also has its own generator. For me now, they are
useful in identifying information such as contacts, calendar events
which I want to sync with external apps such as google's
contacts/calendars.

Footnotes: 
¹  http://www.emacswiki.org/emacs/uuid.el
²  http://cvs.savannah.gnu.org/viewvc/?root=addressbook

Leo

Re: Should Emacs provide a uuid function?

[Ken Raeburn]

On Apr 25, 2011, at 04:45, Stephen J. Turnbull wrote:
>> Though, at least one version I ran across generates (by default?) 
>> the information-revealing "version 1" format that encodes the
>> machine's MAC address, and while some versions have options to
>> control which form is generated, not all do.
> 
> Application where you'd care, please.

I don't know; I'm not familiar with many uses of UUIDs, either.  I'd expect some messaging protocols might be able to use such a thing, but an argument has been made against email/news message-ids using them.  Just pointing it out, in case anyone is assuming uuidgen will always give something suitable for exposing to the net at large associated with one's identity.

>  If any of the data for the
> applications mentioned by Leo leaked enough that somebody got hold of
> one of the UUIDs, I'd be worried about everything *except* my MAC
> address.

Are calendar or org files not suitable for being made public?  (That's a serious question; I don't use either one.)  Should they include information that helps track what computers I use?

Ken

Re: Should Emacs provide a uuid function?

[Ken Raeburn]

On Apr 25, 2011, at 15:52, Chong Yidong wrote:

> Ken Raeburn <raeburn@raeburn.org> writes:
> 
>> I wouldn't consider the elisp 'random' function to be remotely good
>> enough though.
> 
> You mean, because of the way it's seeded?  I guess we can (should?)
> improve it to use /dev/random where that's available.

How it's seeded currently, yes, but also the fact that it can (depending on the system) use some fairly weak PRNG algorithms, with patterns in their output and relatively small amounts of internal state.

Ken

Re: Should Emacs provide a uuid function?

[Stephen J. Turnbull]

Ken Raeburn writes:
 > On Apr 25, 2011, at 04:45, Stephen J. Turnbull wrote:
 > >> Though, at least one version I ran across generates (by default?) 
 > >> the information-revealing "version 1" format that encodes the
 > >> machine's MAC address, and while some versions have options to
 > >> control which form is generated, not all do.
 > > 
 > > Application where you'd care, please.
 > 
 > I don't know; I'm not familiar with many uses of UUIDs, either.
 > I'd expect some messaging protocols might be able to use such a
 > thing, but an argument has been made against email/news message-ids
 > using them.

Well, I've recently seen them used in MSDN archive URLs :-) and I
believe Apple plists use them to tag nodes.

 > > If any of the data for the applications mentioned by Leo leaked
 > > enough that somebody got hold of one of the UUIDs, I'd be worried
 > > about everything *except* my MAC address.
 > 
 > Are calendar or org files not suitable for being made public?
 > (That's a serious question; I don't use either one.)  Should they
 > include information that helps track what computers I use?

Uh, if I were young, slim, and sexy I really wouldn't want my stalkers
to know when I plan to walk alone in Central Park, or when I'm going
anywhere for that matter.  And I would certainly give that concern
priority over them knowing whether I made a particular appointment via
my iPhone or Barrack Obama's PC.  Oops, you're not supposed to know I
have access to that, I guess it would be scary if that leaked.  Yeah,
right.

No, they shouldn't include that information if it can easily be
avoided.  But get real.  When Emacs net apps all route via TOR only by
default, then I'll concede you have a point.

Re: Should Emacs provide a uuid function?

[Ken Raeburn]

On Apr 28, 2011, at 23:47, Stephen J. Turnbull wrote:
>> Are calendar or org files not suitable for being made public?
>> (That's a serious question; I don't use either one.)  Should they
>> include information that helps track what computers I use?
> 
> Uh, if I were young, slim, and sexy I really wouldn't want my stalkers
> to know when I plan to walk alone in Central Park, or when I'm going
> anywhere for that matter.

Like I said, I don't know these particular tools.  With some calendar software, it's perfectly reasonable to use it to publish a calendar with certain public or semi-public information -- theater performance schedules, sports events, conference schedules, stuff like that.  If the Emacs calendar and org software aren't in that category, which I guess I should take the tone of your reply to imply, then fine, it's not an issue.

>  And I would certainly give that concern
> priority over them knowing whether I made a particular appointment via
> my iPhone or Barrack Obama's PC.  Oops, you're not supposed to know I
> have access to that, I guess it would be scary if that leaked.  Yeah,
> right.

IF I were publishing a calendar file with the schedule of talks at an upcoming conference, for myself and other attendees to use, I wouldn't necessarily want to include with it information that could be used to identify the computer I used or where I live.  Not directly, but perhaps when correlated other information that may be out there -- e.g., MACs in UUIDs show that I and Suzy Q both frequently use a certain computer, and Suzy also posts a lot on Facebook about life in a certain city and how she frequents a certain internet cafe; one might conclude that I live around there and may use that cafe too.  (Why, yes, I *have* spent a lot of years thinking about some aspects of computer and network security.)

But it's not an issue if these Emacs apps are not intended to be used for that sort of thing, and the files containing the UUIDs not to be shared.

> No, they shouldn't include that information if it can easily be
> avoided.  But get real.  When Emacs net apps all route via TOR only by
> default, then I'll concede you have a point.

Different sorts of exposures lead to different kinds of opportunities for attacks.  Just because one hasn't been closed off doesn't mean it's not worth looking at others.  E.g., it's possible I could get mugged on my way home from work some night; that doesn't make it pointless for me to lock my doors at night.

Ken

Re: Should Emacs provide a uuid function?

[Richard Stallman]

Could someone give me a brief explanation of what a uuid is?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> Date: Sat, 30 Apr 2011 09:32:49 -0400
> From: Richard Stallman <rms@gnu.org>
> 
> Could someone give me a brief explanation of what a uuid is?

Universally Unique Identifier, see

   http://en.wikipedia.org/wiki/Universally_unique_identifier

Re: Should Emacs provide a uuid function?

[Stephen J. Turnbull]

Ken Raeburn writes:

 > (Why, yes, I *have* spent a lot of years thinking about some
 > aspects of computer and network security.)

Sure, I have too.  Enough so that I have one computer that doesn't
have a network connection at all.  (Not coincidentally, it's the only
Windows-based computer I use regularly.)

On the other hand, correlating my location with Suzy's is a waste of
the hacker's time, because it's easy enough to figure out where I am
several days a month from the online university course catalog and
from webserver logs (IP addresses for the rather balkanized campus
network often allows determining probable -- assuming no deliberate
obfuscation -- location within 50m).

 > Different sorts of exposures lead to different kinds of
 > opportunities for attacks.  Just because one hasn't been closed off
 > doesn't mean it's not worth looking at others.

Certainly.  I don't think this one justifies an addition to core
because

(1) the attacks it *might* foreclose for *some* people with insecure
    versions of uuidgen are minor, even compared to the minor costs of
    writing, documenting, and maintaining an internal uuidgen;

(2) at least some of the use cases proposed so far would likely not
    want to use uuidgen-style identifiers, and IMO it is rather likely
    that adding an internal uuidgen to core is unlikely to much reduce
    the number of different implementations, most of which will be
    more or less insecure; and

(3) I think the whole idea is currently only half baked, especially
    with respect to UUID formats, and it would not hurt to have one or
    more implementations in ELPA, which would allow experience to
    determine best practice before putting in core.

Re: Should Emacs provide a uuid function?

[Ken Raeburn]

On Apr 30, 2011, at 14:39, Stephen J. Turnbull wrote:
> On the other hand, correlating my location with Suzy's is a waste of
> the hacker's time, because it's easy enough to figure out where I am
> several days a month from the online university course catalog and
> from webserver logs (IP addresses for the rather balkanized campus
> network often allows determining probable -- assuming no deliberate
> obfuscation -- location within 50m).

Just because some of us don't go to such efforts to keep our information private doesn't mean others don't, or don't want to.  One could use Tor (as you suggested), or a VPN configured to tunnel even traffic destined for the outside world.  Add use of a pseudonym instead of one's real name, and it gets trickier to trace a location.

>> Different sorts of exposures lead to different kinds of
>> opportunities for attacks.  Just because one hasn't been closed off
>> doesn't mean it's not worth looking at others.
> 
> Certainly.  I don't think this one justifies an addition to core
> because

On its own, I'm not sure it does, either.  But if any of the use cases might want stronger privacy guarantees than some of the uuidgen implementations provide, or there are portability issues, then "just use uuidgen in each package" looks like a poor solution.

> (3) I think the whole idea is currently only half baked, especially
>    with respect to UUID formats, and it would not hurt to have one or
>    more implementations in ELPA, which would allow experience to
>    determine best practice before putting in core.

I'm inclined to agree, though if multiple packages in the main distribution are (and should be) using UUIDs, eventually having a common function included to generate them doesn't seem like a bad thing.

Ken

Re: Should Emacs provide a uuid function?

[Stephen J. Turnbull]

Ken Raeburn writes:

 > On its own, I'm not sure it does, either.  But if any of the use
 > cases might want stronger privacy guarantees than some of the
 > uuidgen implementations provide, or there are portability issues,
 > then "just use uuidgen in each package" looks like a poor
 > solution.

Let's not oversell this, please.  If we want to give anything like a
"privacy guarantee" or "solution," we have a truckload of code to
audit....  That might be a good thing to do, but I doubt there's much
appetite for it in the community at this point in time.  "Minor
improvement" is more like it.

Re: Should Emacs provide a uuid function?

[Leo]

On 2011-04-26 03:52 +0800, Chong Yidong wrote:
> Ken Raeburn <raeburn@raeburn.org> writes:
>
>> I wouldn't consider the elisp 'random' function to be remotely good
>> enough though.
>
> You mean, because of the way it's seeded?  I guess we can (should?)
> improve it to use /dev/random where that's available.

I think this is a good proposal. It seems everything can be built on top
of the ability to get some bytes from /dev/random or /dev/urandom.

For example:

(defun secure-random-bytes (n)
  (let ((file (cond
               ((file-exists-p "/dev/random")  "/dev/random")
               ((file-exists-p "/dev/urandom") "/dev/urandom")
               (t (error "Secure random device not available")))))
    (with-temp-buffer
      (set-buffer-multibyte nil)
      (insert-file-contents file nil 0 n)
      (buffer-string))))

After patching insert-file-contents to support CHAR DEV.

diff --git a/src/fileio.c b/src/fileio.c
index c6e93ceb..c8ab7ef6 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -3275,7 +3275,7 @@ variable `last-coding-system-used' to the coding system actually used.  */)
   /* This code will need to be changed in order to work on named
      pipes, and it's probably just not worth it.  So we should at
      least signal an error.  */
-  if (!S_ISREG (st.st_mode))
+  if (!S_ISREG (st.st_mode) && !S_ISCHR (st.st_mode))
     {
       not_regular = 1;
 
Leo

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Leo <sdl.web@gmail.com>
> Date: Mon, 09 May 2011 14:29:20 +0800
> Cc: Ken Raeburn <raeburn@raeburn.org>, Emacs Dev <emacs-devel@gnu.org>
> 
> (defun secure-random-bytes (n)
>   (let ((file (cond
>                ((file-exists-p "/dev/random")  "/dev/random")
>                ((file-exists-p "/dev/urandom") "/dev/urandom")
>                (t (error "Secure random device not available")))))

Feh.  Using platform-dependent interfaces in Lisp is not the Emacs
way, in my book.  How about a primitive which would read from
/dev/random or /dev/urandom if they are available, or use other
methods if they are not?

> -  if (!S_ISREG (st.st_mode))
> +  if (!S_ISREG (st.st_mode) && !S_ISCHR (st.st_mode))
>      {
>        not_regular = 1;

This is wrong, I think.  We will now treat character devices as
regular files, and will trust their st_size, seek them for `coding'
cookies, allow to visit them, etc. -- search the function for all the
uses of not_regular, and you will see.

This all will be avoided if we instead provide a primitive that will
both hide system-dependent stuff and avoid the need to teach
insert-file-contents about this special device.

Re: Should Emacs provide a uuid function?

[Ted Zlatanov]

On Mon, 09 May 2011 10:23:06 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> Feh.  Using platform-dependent interfaces in Lisp is not the Emacs
EZ> way, in my book.  How about a primitive which would read from
EZ> /dev/random or /dev/urandom if they are available, or use other
EZ> methods if they are not?
...
EZ> This all will be avoided if we instead provide a primitive that will
EZ> both hide system-dependent stuff and avoid the need to teach
EZ> insert-file-contents about this special device.

GnuTLS has facilities for this, using /dev/*random and the Yarrow
algorithm from libnettle.  Look at lib/nettle/rnd.c
http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=tree;f=lib/nettle;hb=HEAD
for the `do_trivia_source' and `do_device_source' functions.

Since GnuTLS is enabled by default in Emacs 24 maybe we can rely on that
functionality instead of writing it anew?

Ted

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 09 May 2011 05:51:50 -0500
> 
> On Mon, 09 May 2011 10:23:06 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
> 
> EZ> Feh.  Using platform-dependent interfaces in Lisp is not the Emacs
> EZ> way, in my book.  How about a primitive which would read from
> EZ> /dev/random or /dev/urandom if they are available, or use other
> EZ> methods if they are not?
> ...
> EZ> This all will be avoided if we instead provide a primitive that will
> EZ> both hide system-dependent stuff and avoid the need to teach
> EZ> insert-file-contents about this special device.
> 
> GnuTLS has facilities for this, using /dev/*random and the Yarrow
> algorithm from libnettle.  Look at lib/nettle/rnd.c
> http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=tree;f=lib/nettle;hb=HEAD
> for the `do_trivia_source' and `do_device_source' functions.

Yes, I meant to use on Windows the 2 APIs that rnd.c uses.

> Since GnuTLS is enabled by default in Emacs 24 maybe we can rely on that
> functionality instead of writing it anew?

It is enabled, but it isn't guaranteed to be present.  It isn't a good
idea, IMO, to have features unrelated to gnutls depend on it being
available.

Re: Should Emacs provide a uuid function?

[Leo]

On 2011-05-09 18:51 +0800, Ted Zlatanov wrote:
> GnuTLS has facilities for this, using /dev/*random and the Yarrow
> algorithm from libnettle.  Look at lib/nettle/rnd.c
> http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=tree;f=lib/nettle;hb=HEAD
> for the `do_trivia_source' and `do_device_source' functions.
>
> Since GnuTLS is enabled by default in Emacs 24 maybe we can rely on that
> functionality instead of writing it anew?

This could work.

Leo

Re: Should Emacs provide a uuid function?

[Stefan Monnier]

>> (defun secure-random-bytes (n)
>> (let ((file (cond
>> ((file-exists-p "/dev/random")  "/dev/random")
>> ((file-exists-p "/dev/urandom") "/dev/urandom")
>> (t (error "Secure random device not available")))))

> Feh.  Using platform-dependent interfaces in Lisp is not the Emacs
> way, in my book.  How about a primitive which would read from
> /dev/random or /dev/urandom if they are available, or use other
> methods if they are not?

FWIW, I don't think there's anything wrong with writing the
platform-dependent code in Elisp rather than in C.  I generally prefer
code to be written in Elisp when possible and when it's not introducing
a performance problem.


        Stefan

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Leo <sdl.web@gmail.com>,  cyd@stupidchicken.com,  raeburn@raeburn.org,  emacs-devel@gnu.org
> Date: Mon, 09 May 2011 11:38:08 -0300
> 
> >> (defun secure-random-bytes (n)
> >> (let ((file (cond
> >> ((file-exists-p "/dev/random")  "/dev/random")
> >> ((file-exists-p "/dev/urandom") "/dev/urandom")
> >> (t (error "Secure random device not available")))))
> 
> > Feh.  Using platform-dependent interfaces in Lisp is not the Emacs
> > way, in my book.  How about a primitive which would read from
> > /dev/random or /dev/urandom if they are available, or use other
> > methods if they are not?
> 
> FWIW, I don't think there's anything wrong with writing the
> platform-dependent code in Elisp rather than in C.  I generally prefer
> code to be written in Elisp when possible and when it's not introducing
> a performance problem.

Suit yourself, but then there will be immediately a test of
system-type and a call to a primitive on w32 (because there's no
similar device there).  IMO, it's ugly, but that's me.

Re: Should Emacs provide a uuid function?

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> (defun secure-random-bytes (n)
>>> (let ((file (cond
>>> ((file-exists-p "/dev/random")  "/dev/random")
>>> ((file-exists-p "/dev/urandom") "/dev/urandom")
>>> (t (error "Secure random device not available")))))
>
>> Feh.  Using platform-dependent interfaces in Lisp is not the Emacs
>> way, in my book.  How about a primitive which would read from
>> /dev/random or /dev/urandom if they are available, or use other
>> methods if they are not?
>
> FWIW, I don't think there's anything wrong with writing the
> platform-dependent code in Elisp rather than in C.  I generally prefer
> code to be written in Elisp when possible and when it's not introducing
> a performance problem.

I'm not sure we should introduce a separate function, though.  It makes
more sense to improve the existing `random' primitive, making it return
a random number (via /dev/random or /dev/urandom or GnuTLS) rather than
a pseudo-random number when possible.

We could add an optional arg to `random' that says that if no secure
random source is available, signal an error instead of falling back on
the old method.

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Leo <sdl.web@gmail.com>, raeburn@raeburn.org,
>         emacs-devel@gnu.org
> Date: Mon, 09 May 2011 11:09:24 -0400
> 
> I'm not sure we should introduce a separate function, though.  It makes
> more sense to improve the existing `random' primitive, making it return
> a random number (via /dev/random or /dev/urandom or GnuTLS) rather than
> a pseudo-random number when possible.
> 
> We could add an optional arg to `random' that says that if no secure
> random source is available, signal an error instead of falling back on
> the old method.

That'd be fine with me, thanks.

Re: Should Emacs provide a uuid function?

[Ted Zlatanov]

On Mon, 09 May 2011 11:09:24 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> I'm not sure we should introduce a separate function, though.  It makes
CY> more sense to improve the existing `random' primitive, making it return
CY> a random number (via /dev/random or /dev/urandom or GnuTLS) rather than
CY> a pseudo-random number when possible.

On Mon, 09 May 2011 14:35:40 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> GnuTLS has facilities for this, using /dev/*random and the Yarrow
>> algorithm from libnettle.  Look at lib/nettle/rnd.c
>> http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=tree;f=lib/nettle;hb=HEAD
>> for the `do_trivia_source' and `do_device_source' functions.

EZ> Yes, I meant to use on Windows the 2 APIs that rnd.c uses.

>> Since GnuTLS is enabled by default in Emacs 24 maybe we can rely on that
>> functionality instead of writing it anew?

EZ> It is enabled, but it isn't guaranteed to be present.  It isn't a good
EZ> idea, IMO, to have features unrelated to gnutls depend on it being
EZ> available.

Random number generation is implemented very well in GnuTLS, though
(cryptographers care about randomness more than almost anyone else).

Maybe we should make GnuTLS a source of random numbers together with a
slower ELisp version when GnuTLS is not available?  That would work for
everyone, I think.

Ted

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 09 May 2011 10:27:53 -0500
> 
> Random number generation is implemented very well in GnuTLS, though
> (cryptographers care about randomness more than almost anyone else).
> 
> Maybe we should make GnuTLS a source of random numbers together with a
> slower ELisp version when GnuTLS is not available?  That would work for
> everyone, I think.

I like Ching's proposal better.  There's nothing to prevent us from
improving the existing `random' to support cryptographic strength
random numbers, is there?

Re: Should Emacs provide a uuid function?

[Stefan Monnier]

> Suit yourself, but then there will be immediately a test of
> system-type and a call to a primitive on w32 (because there's no
> similar device there).  IMO, it's ugly, but that's me.

I was only making a general remark about your assertion that "using
platform-dependent interfaces in Lisp is not the Emacs way, in my book".

In this particular case, if some half gets written in C and the other in
Elisp, the benefit of using Elisp might not be worth the trouble, indeed.


        Stefan

Re: Should Emacs provide a uuid function?

[Ted Zlatanov]

On Mon, 09 May 2011 18:36:19 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Mon, 09 May 2011 10:27:53 -0500
>> 
>> Random number generation is implemented very well in GnuTLS, though
>> (cryptographers care about randomness more than almost anyone else).
>> 
>> Maybe we should make GnuTLS a source of random numbers together with a
>> slower ELisp version when GnuTLS is not available?  That would work for
>> everyone, I think.

EZ> I like Ching's proposal better.  There's nothing to prevent us from
EZ> improving the existing `random' to support cryptographic strength
EZ> random numbers, is there?

With Chong's proposal GnuTLS would be used, so I think it's fine :)

Ted

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: cyd@stupidchicken.com,  raeburn@raeburn.org,  sdl.web@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 09 May 2011 12:41:04 -0300
> 
> I was only making a general remark about your assertion that "using
> platform-dependent interfaces in Lisp is not the Emacs way, in my book".

Fair enough.  My wording was probably too general, sorry.  I have that
bad habit ;-)

> In this particular case, if some half gets written in C and the other in
> Elisp, the benefit of using Elisp might not be worth the trouble, indeed.

I was particularly trying to avoid proliferating the ugliness of the
kind we have in battery.el:

  (defcustom battery-status-function
    (cond ((and (eq system-type 'gnu/linux)
		(file-readable-p "/proc/apm"))
	   'battery-linux-proc-apm)
	  ((and (eq system-type 'gnu/linux)
		(file-directory-p "/proc/acpi/battery"))
	   'battery-linux-proc-acpi)
	  ((and (eq system-type 'gnu/linux)
		(file-directory-p "/sys/class/power_supply/")
		(directory-files "/sys/class/power_supply/" nil "BAT[0-9]$"))
	   'battery-linux-sysfs)
	  ((and (eq system-type 'darwin)
		(condition-case nil
		    (with-temp-buffer
		      (and (eq (call-process "pmset" nil t nil "-g" "ps") 0)
			   (> (buffer-size) 0)))
		  (error nil)))
	   'battery-pmset)
	  ((eq system-type 'windows-nt)
	   'w32-battery-status))

I think it would be much cleaner to have 5 different implementations
in sysdep.c of the same primitive, than have the above followed by 4
different Lisp functions plus one primitive.

Re: Should Emacs provide a uuid function?

[Stefan Monnier]

> I'm not sure we should introduce a separate function, though.  It makes
> more sense to improve the existing `random' primitive, making it return
> a random number (via /dev/random or /dev/urandom or GnuTLS) rather than
> a pseudo-random number when possible.

`random' is supposed to be a PRNG, not an RNG.
I don't see any reason why we should make it significantly better than
libc's own PRNG (i.e. any bug report on a weakness of our `random'
should be forwarded to the corresponding libc's maintainers).

OTOH we should provide a way to seed the PRNG with an RN as well as to
re-seed it with the same seed.


        Stefan

Re: Should Emacs provide a uuid function?

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> With Chong's proposal GnuTLS would be used, so I think it's fine :)

Well, applications should never use /dev/random, since it will block,
and depleting it will make both Emacs and other programs suffer.

Using /dev/urandom is fine, though.

But I think /dev/urandom should only be used as a seed, really.  Nobody
needs that much randomness in Emacs, I think.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Should Emacs provide a uuid function?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> OTOH we should provide a way to seed the PRNG with an RN as well as to
> re-seed it with the same seed.

As long as you don't re-seed with /dev/random.  LispWorks 5.0 made that
mistake, which meant that you could only start one instance per second
(on machines without a high entropy source).

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Should Emacs provide a uuid function?

[Stefan Monnier]

> I was particularly trying to avoid proliferating the ugliness of the
> kind we have in battery.el:

>   (defcustom battery-status-function
>     (cond ((and (eq system-type 'gnu/linux)
> 		(file-readable-p "/proc/apm"))
> 	   'battery-linux-proc-apm)
> 	  ((and (eq system-type 'gnu/linux)
> 		(file-directory-p "/proc/acpi/battery"))
> 	   'battery-linux-proc-acpi)
> 	  ((and (eq system-type 'gnu/linux)
> 		(file-directory-p "/sys/class/power_supply/")
> 		(directory-files "/sys/class/power_supply/" nil "BAT[0-9]$"))
> 	   'battery-linux-sysfs)
> 	  ((and (eq system-type 'darwin)
> 		(condition-case nil
> 		    (with-temp-buffer
> 		      (and (eq (call-process "pmset" nil t nil "-g" "ps") 0)
> 			   (> (buffer-size) 0)))
> 		  (error nil)))
> 	   'battery-pmset)
> 	  ((eq system-type 'windows-nt)
> 	   'w32-battery-status))

I don't mind this kind of dispatch table, personally.

> I think it would be much cleaner to have 5 different implementations
> in sysdep.c of the same primitive, than have the above followed by 4
> different Lisp functions plus one primitive.

I don't find the dispatch table worse than the sysdep.c
code, personally.


        Stefan

Re: Should Emacs provide a uuid function?

[Stefan Monnier]

>> OTOH we should provide a way to seed the PRNG with an RN as well as to
>> re-seed it with the same seed.
> As long as you don't re-seed with /dev/random.

I do mean "a number coming from /dev/random" when I say "RN".
But I did not say to do it all the time: only to provide a way to do it.


        Stefan

Re: Should Emacs provide a uuid function?

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm not sure we should introduce a separate function, though.  It makes
>> more sense to improve the existing `random' primitive, making it return
>> a random number (via /dev/random or /dev/urandom or GnuTLS) rather than
>> a pseudo-random number when possible.
>
> `random' is supposed to be a PRNG, not an RNG.
> I don't see any reason why we should make it significantly better than
> libc's own PRNG (i.e. any bug report on a weakness of our `random'
> should be forwarded to the corresponding libc's maintainers).
>
> OTOH we should provide a way to seed the PRNG with an RN as well as to
> re-seed it with the same seed.
>
>
>         Stefan

Is this thread relevant?

http://comments.gmane.org/gmane.emacs.devel/137319

(My interest is rather esoteric so I haven't done it yet for lack of
time. This is my usecase https://github.com/jave/el-tarot)



-- 
Joakim Verona

Re: Should Emacs provide a uuid function?

[Lars Magne Ingebrigtsen]

joakim@verona.se writes:

> Is this thread relevant?

Not really.  :-)

But getting the Emacs RNG seeded would be nice, because it's now 100%
predictable.  Which is annoying when I say `M-x jukebox-play-at-random',
which always plays songs in exactly the same order (after an Emacs
restart).

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Should Emacs provide a uuid function?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: cyd@stupidchicken.com,  raeburn@raeburn.org,  sdl.web@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 09 May 2011 14:03:44 -0300
> 
> >   (defcustom battery-status-function
> >     (cond ((and (eq system-type 'gnu/linux)
> > 		(file-readable-p "/proc/apm"))
> > 	   'battery-linux-proc-apm)
> > 	  ((and (eq system-type 'gnu/linux)
> > 		(file-directory-p "/proc/acpi/battery"))
> > 	   'battery-linux-proc-acpi)
> > 	  ((and (eq system-type 'gnu/linux)
> > 		(file-directory-p "/sys/class/power_supply/")
> > 		(directory-files "/sys/class/power_supply/" nil "BAT[0-9]$"))
> > 	   'battery-linux-sysfs)
> > 	  ((and (eq system-type 'darwin)
> > 		(condition-case nil
> > 		    (with-temp-buffer
> > 		      (and (eq (call-process "pmset" nil t nil "-g" "ps") 0)
> > 			   (> (buffer-size) 0)))
> > 		  (error nil)))
> > 	   'battery-pmset)
> > 	  ((eq system-type 'windows-nt)
> > 	   'w32-battery-status))
> 
> I don't mind this kind of dispatch table, personally.
> 
> > I think it would be much cleaner to have 5 different implementations
> > in sysdep.c of the same primitive, than have the above followed by 4
> > different Lisp functions plus one primitive.
> 
> I don't find the dispatch table worse than the sysdep.c
> code, personally.

FWIW, I'm very surprised to hear this from you.

I understand the motivation to move to Lisp everything that can be
moved, especially if the code deals with Lisp data types, but there
should be limits, I think.  Lisp is not suited well to reading raw
data from files and device drivers.  insert-file-contents is a much
higher abstraction than `read' and `fscanf'.

It's not a coincidence that battery.el's implementation of these 4
functions take at least twice as much code as a C version would.
Worse, a programmer who wants to change battery.el now needs to
understand how the OS stores the data on those special files, which is
something most Lisp programmers don't and shouldn't care about.

Such code is okay in one's ~/.emacs, but not in a bundled package.

It is much better, IMO, to have the low-level stuff in a language that
was designed for that, and present to Lisp lispy data types and
structures that are as platform independent as possible.

Re: Should Emacs provide a uuid function?

[Jason Rumney]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But getting the Emacs RNG seeded would be nice, because it's now 100%
> predictable.  Which is annoying when I say `M-x jukebox-play-at-random',
> which always plays songs in exactly the same order (after an Emacs
> restart).

I had a sense of deja vu reading your post, and looked it up.  Sure
enough, this was implemented already. Try adding (random t) to your
.emacs.

Re: Should Emacs provide a uuid function?

[Lars Magne Ingebrigtsen]

Jason Rumney <jasonr@gnu.org> writes:

> I had a sense of deja vu reading your post, and looked it up.  Sure
> enough, this was implemented already. Try adding (random t) to your
> .emacs.

Nice; thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/