Network security manager

Network security manager

[Lars Magne Ingebrigtsen]

I plan on starting to implement the Emacs network security manager soon
(hopefully this evening), but I was wondering whether to do it on a
public feature branch or on the trunk.

Doing it on a branch will make it less likely that I'll be disrupting
much, but (by experience) this also means that there will be no testing
or feedback by anybody else but me until I merge the entire thing.

If I'm implementing this directly on the trunk, it will be disabled by
default, and the people who want to test it will have to set a variable
called something like `network-security-policy' to something like
`enabled'.  The default will be nil, so (in theory) nobody else should
be bothered by it while we're working out the details.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Mon, 17 Nov 2014 13:46:59 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I plan on starting to implement the Emacs network security manager soon
LMI> (hopefully this evening), but I was wondering whether to do it on a
LMI> public feature branch or on the trunk.

LMI> Doing it on a branch will make it less likely that I'll be disrupting
LMI> much, but (by experience) this also means that there will be no testing
LMI> or feedback by anybody else but me until I merge the entire thing.

LMI> If I'm implementing this directly on the trunk, it will be disabled by
LMI> default, and the people who want to test it will have to set a variable
LMI> called something like `network-security-policy' to something like
LMI> `enabled'.  The default will be nil, so (in theory) nobody else should
LMI> be bothered by it while we're working out the details.

Feature branch or emacs-24 please.  It should be applicable to
emacs-24, and the maintainers have requested we work against emacs-24 in
those cases, forward-porting the changes into master.

I'll be testing it with you.

Ted

Re: Network security manager

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Feature branch or emacs-24 please.  It should be applicable to
> emacs-24, and the maintainers have requested we work against emacs-24 in
> those cases, forward-porting the changes into master.

I don't think emacs-24 is supposed to receive new features, only bug
fixes.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Network security manager

[Stefan Monnier]

> Doing it on a branch will make it less likely that I'll be disrupting
> much, but (by experience) this also means that there will be no testing
> or feedback by anybody else but me until I merge the entire thing.

I suggest you do it on a personal branch until the point where it
becomes testable, at which point you can move it to trunk (aka "master").


        Stefan

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Feature branch or emacs-24 please.  It should be applicable to
> emacs-24, and the maintainers have requested we work against emacs-24 in
> those cases, forward-porting the changes into master.

Yeah, I didn't consider emacs-24.  Yes, that's a good idea.  If we push
a new version of emacs-24, the security manager should definitely be in
there...

> I'll be testing it with you.

Great.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@suse.de> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> Feature branch or emacs-24 please.  It should be applicable to
>> emacs-24, and the maintainers have requested we work against emacs-24 in
>> those cases, forward-porting the changes into master.
>
> I don't think emacs-24 is supposed to receive new features, only bug
> fixes.

True, but at present there isn't any TLS security in Emacs 24, and
that's kinda a big bug.

(This is where the Professional Security Professionals will chime in and
say IT"S A RELEASE BLOCKER!!!  DO NOTHING UNTIL THIS IS FIXED!  TALK TO
NOONE!)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Stefan Monnier]

>> Feature branch or emacs-24 please.  It should be applicable to
>> emacs-24, and the maintainers have requested we work against emacs-24 in
>> those cases, forward-porting the changes into master.
> I don't think emacs-24 is supposed to receive new features, only bug
> fixes.

Indeed.  I could consider including it in a 24.5 release because it's
a somewhat important issue, but it would have to be "obviously safe"
(in the sense of "won't break anything").  That sounds fairly unlikely.


        Stefan

Re: Network security manager

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Indeed.  I could consider including it in a 24.5 release because it's
> a somewhat important issue, but it would have to be "obviously safe"
> (in the sense of "won't break anything").  That sounds fairly unlikely.

True.  It will probably disrupt some workflows.  It'll ask the user some
questions where none was asked before, and in non-interactive use, some
network connections will be refused by default.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Mon, 17 Nov 2014 09:17:19 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>> Feature branch or emacs-24 please.  It should be applicable to
>>> emacs-24, and the maintainers have requested we work against emacs-24 in
>>> those cases, forward-porting the changes into master.
>> I don't think emacs-24 is supposed to receive new features, only bug
>> fixes.

SM> Indeed.  I could consider including it in a 24.5 release because it's
SM> a somewhat important issue, but it would have to be "obviously safe"
SM> (in the sense of "won't break anything").  That sounds fairly unlikely.

I don't know how complicated it will be internally, but I don't think it
will endanger any existing functionality (except TLS connections, of
course). The only reason for it in 24.x is to add reasonable certificate
handling so we can turn on certificate verification by default. I don't
think it can be done otherwise without seriously damaging the user
experience.

Would you rather not make those changes in 24.x?

Ted

Re: Network security manager

[Ted Zlatanov]

On Mon, 17 Nov 2014 10:00:58 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Mon, 17 Nov 2014 09:17:19 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>>>> Feature branch or emacs-24 please.  It should be applicable to
>>>> emacs-24, and the maintainers have requested we work against emacs-24 in
>>>> those cases, forward-porting the changes into master.
>>> I don't think emacs-24 is supposed to receive new features, only bug
>>> fixes.

SM> Indeed.  I could consider including it in a 24.5 release because it's
SM> a somewhat important issue, but it would have to be "obviously safe"
SM> (in the sense of "won't break anything").  That sounds fairly unlikely.

TZ> I don't know how complicated it will be internally, but I don't think it
TZ> will endanger any existing functionality (except TLS connections, of
TZ> course). The only reason for it in 24.x is to add reasonable certificate
TZ> handling so we can turn on certificate verification by default. I don't
TZ> think it can be done otherwise without seriously damaging the user
TZ> experience.

TZ> Would you rather not make those changes in 24.x?

BTW, I proposed using emacs-24 3 weeks ago in the thread "removing SSLv3
support by default from the Emacs GnuTLS integration (was: Bug#766395:
emacs/gnus: Uses s_client to for SSL.)" you can find here
https://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00936.html

There were no comments or objections. Sorry if I didn't ping people back
then. I assumed it was a reasonable proposal and everyone was nodding
agreement into their beards...

Ted

Re: Network security manager

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Doing it on a branch will make it less likely that I'll be disrupting
>> much, but (by experience) this also means that there will be no testing
>> or feedback by anybody else but me until I merge the entire thing.
>
> I suggest you do it on a personal branch until the point where it
> becomes testable, at which point you can move it to trunk (aka
> "master").

If you create the personal branch from emacs-24, you can delay the
decision about merging to emacs-24 or master until later.

-- 
-- Stephe

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I don't know how complicated it will be internally, but I don't think it
> will endanger any existing functionality (except TLS connections, of
> course).

Let's say you fetch mail from pop3 from a server that has a self-signed
certificate as a batch job.  The network security manager will say "The
server uses a self-signed certificate, so Emacs can't verify the
authenticity of the server.  Connect anyway? (no, this session only,
always)" (or something like that).

But since it's a batch job, we can't ask the user, and the connection
will fail.  (Unless we decide to have the batch default be the
opposite -- always answer "this session only".)

So perhaps it's better for Emacs 25.1?

Especially if we can release 25.1 in a timely manner.  >"? 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> If you create the personal branch from emacs-24, you can delay the
> decision about merging to emacs-24 or master until later.

That sounds reasonable.  Hm...  Er...  do you have a recipe for how to
start a new branch off of emacs-24?  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Kelvin White]

[-- Attachment #1: Type: text/plain, Size: 305 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> That sounds reasonable.  Hm...  Er...  do you have a recipe for how to
> start a new branch off of emacs-24?  >"?

git checkout -b NEW_BRANCH
git commit -m "first commit"
git push -u origin NEW_BRANCH

substitue the name of the new branch for NEW_BRANCH

[-- Attachment #2: Type: text/html, Size: 723 bytes --]

Re: Network security manager

[Kelvin White]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]

Assuming you are already on emacs-24 branch, of course

[-- Attachment #2: Type: text/html, Size: 76 bytes --]

Re: Network security manager

[Ted Zlatanov]

On Mon, 17 Nov 2014 16:22:57 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I don't know how complicated it will be internally, but I don't think it
>> will endanger any existing functionality (except TLS connections, of
>> course).

LMI> Let's say you fetch mail from pop3 from a server that has a self-signed
LMI> certificate as a batch job.  The network security manager will say "The
LMI> server uses a self-signed certificate, so Emacs can't verify the
LMI> authenticity of the server.  Connect anyway? (no, this session only,
LMI> always)" (or something like that).

How common is this scenario and how strongly do you feel we should
support it?

Generally we could distinguish between POP3 and SMTP and IMAP and such,
where self-signed certificates are common, and HTTP/S and generic
connections, where they aren't. Does that seem reasonable?

I would personally prefer forcing the user to run interactively at least
once and accept the certificate.  Too much magic is sure to complicate
everyone's life.

LMI> But since it's a batch job, we can't ask the user, and the connection
LMI> will fail.  (Unless we decide to have the batch default be the
LMI> opposite -- always answer "this session only".)

I'd add a CLI option --insecure/-k (same as curl) to override the
default, but no more than that, and without special --batch behavior.

LMI> So perhaps it's better for Emacs 25.1?

LMI> Especially if we can release 25.1 in a timely manner.  >"? 

I really would prefer that we treat this as a bug.  It's unfortunate
that resolving it is complicated, but we've delayed the fix for a while.

Can you please work against emacs-24? It's easy enough to apply the
changes to master if that's the final decision and I don't think master
has anything you need. Except maybe the read-only text property thing
you added.

Ted

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 17 Nov 2014 13:46:59 +0100
> 
> I plan on starting to implement the Emacs network security manager soon
> (hopefully this evening), but I was wondering whether to do it on a
> public feature branch or on the trunk.

What is an "Emacs network security manager"?  What will it manage?

Whether to develop on a separate branch depends on how invasive are
the changes in parts that are not turned off/bypassed when that
variable of yours is off.

Re: Network security manager

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Date: Mon, 17 Nov 2014 14:59:48 +0100
> 
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
> > Feature branch or emacs-24 please.  It should be applicable to
> > emacs-24, and the maintainers have requested we work against emacs-24 in
> > those cases, forward-porting the changes into master.
> 
> I don't think emacs-24 is supposed to receive new features, only bug
> fixes.

Indeed.  So my vote is against adding this to emacs-24.

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 17 Nov 2014 15:00:59 +0100
> MailScanner-NULL-Check: 1416837661.64938@HmWvdhlpOP65nwA+f1Irtg
> 
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
> > Feature branch or emacs-24 please.  It should be applicable to
> > emacs-24, and the maintainers have requested we work against emacs-24 in
> > those cases, forward-porting the changes into master.
> 
> Yeah, I didn't consider emacs-24.  Yes, that's a good idea.  If we push
> a new version of emacs-24, the security manager should definitely be in
> there...

That new version should be rock-stable, otherwise it's just 25.1 in
disguise.

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 17 Nov 2014 15:04:10 +0100
> MailScanner-NULL-Check: 1416837850.98866@HQFf/VIPRnTVab8MOv4K1A
> Cc: emacs-devel@gnu.org
> 
> Andreas Schwab <schwab@suse.de> writes:
> 
> > Ted Zlatanov <tzz@lifelogs.com> writes:
> >
> >> Feature branch or emacs-24 please.  It should be applicable to
> >> emacs-24, and the maintainers have requested we work against emacs-24 in
> >> those cases, forward-porting the changes into master.
> >
> > I don't think emacs-24 is supposed to receive new features, only bug
> > fixes.
> 
> True, but at present there isn't any TLS security in Emacs 24, and
> that's kinda a big bug.

No, it's a missing feature.

> (This is where the Professional Security Professionals will chime in and
> say IT"S A RELEASE BLOCKER!!!  DO NOTHING UNTIL THIS IS FIXED!  TALK TO
> NOONE!)

Ignore him.

Re: Network security manager

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 17 Nov 2014 09:17:19 -0500
> Cc: emacs-devel@gnu.org
> 
> >> Feature branch or emacs-24 please.  It should be applicable to
> >> emacs-24, and the maintainers have requested we work against emacs-24 in
> >> those cases, forward-porting the changes into master.
> > I don't think emacs-24 is supposed to receive new features, only bug
> > fixes.
> 
> Indeed.  I could consider including it in a 24.5 release because it's
> a somewhat important issue, but it would have to be "obviously safe"
> (in the sense of "won't break anything").  That sounds fairly unlikely.

"Fairly unlikely" is an understatement of the month.

Re: Network security manager

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 17 Nov 2014 10:00:58 -0500
> 
> On Mon, 17 Nov 2014 09:17:19 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> 
> SM> Indeed.  I could consider including it in a 24.5 release because it's
> SM> a somewhat important issue, but it would have to be "obviously safe"
> SM> (in the sense of "won't break anything").  That sounds fairly unlikely.
> 
> I don't know how complicated it will be internally, but I don't think it
> will endanger any existing functionality (except TLS connections, of
> course).

Please consider possible bugs as well, not just the effect on
workflows.

Re: Network security manager

[Romain Francoise]

On Mon, Nov 17, 2014 at 08:59:51AM -0500, Stefan Monnier wrote:
> I suggest you do it on a personal branch until the point where it
> becomes testable, at which point you can move it to trunk (aka
> "master").

And should personal branches be hosted in the official repository? If
so, shouldn't we agree on a namespace (e.g. `feature/larsi/secmanager')?
Or should developers create separate forked repositories for their
personal branches, ala Github/Gitorious?

Re: Network security manager

[Stefan Monnier]

TZ> I don't know how complicated it will be internally, but I don't think it
TZ> will endanger any existing functionality (except TLS connections, of
TZ> course).  The only reason for it in 24.x is to add reasonable certificate
TZ> handling so we can turn on certificate verification by default.  I don't
TZ> think it can be done otherwise without seriously damaging the user
TZ> experience.

The issue is that if we have a 24.5 release, I want a very short pretest
phase, so such changes need to be "obviously safe".

One way to do that can be to make the changes conditional on some config
var, which stays disabled by default.  So random users will use the old
code and those who care about security can enable it at the risk of
helping us fix bugs.

> BTW, I proposed using emacs-24 3 weeks ago in the thread "removing SSLv3
> support by default from the Emacs GnuTLS integration (was: Bug#766395:
> emacs/gnus: Uses s_client to for SSL.)" you can find here
> https://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00936.html

I don't know the underlying issues well enough.  But it doesn't sound
"obviously safe" either.  I'd rather just follow gnutls's own defaults.


        Stefan

Re: Network security manager

[Ted Zlatanov]

On Mon, 17 Nov 2014 12:31:35 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

TZ> I don't know how complicated it will be internally, but I don't think it
TZ> will endanger any existing functionality (except TLS connections, of
TZ> course).  The only reason for it in 24.x is to add reasonable certificate
TZ> handling so we can turn on certificate verification by default.  I don't
TZ> think it can be done otherwise without seriously damaging the user
TZ> experience.

SM> The issue is that if we have a 24.5 release, I want a very short pretest
SM> phase, so such changes need to be "obviously safe".

SM> One way to do that can be to make the changes conditional on some config
SM> var, which stays disabled by default.  So random users will use the old
SM> code and those who care about security can enable it at the risk of
SM> helping us fix bugs.

I'd rather not ship with security disabled by default. That's exactly
the situation we have now, just swept into a different corner.

If fixing it is too risky then we put out an insecure 24.5 and 25.1 will
be the first release to manage certificates and verify them by default.
This is no worse than the current 24.x situation. It seems this is
acceptable to everyone so far.

I would have preferred to avoid that situation but the fault is mostly
mine for leaving this unfinished for so long.

>> BTW, I proposed using emacs-24 3 weeks ago in the thread "removing SSLv3
>> support by default from the Emacs GnuTLS integration (was: Bug#766395:
>> emacs/gnus: Uses s_client to for SSL.)" you can find here
>> https://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00936.html

SM> I don't know the underlying issues well enough.  But it doesn't sound
SM> "obviously safe" either.  I'd rather just follow gnutls's own defaults.

We are.

Ted

Re: Network security manager

[Stefan Monnier]

>> I suggest you do it on a personal branch until the point where it
>> becomes testable, at which point you can move it to trunk (aka
>> "master").
> And should personal branches be hosted in the official repository?

No, I meant a branch that's elsewhere, possibly not hosted anywhere at all.


        Stefan

Re: Network security manager

[Lars Magne Ingebrigtsen]

Kelvin White <kwhite@gnu.org> writes:

> git checkout -b NEW_BRANCH
> git commit -m "first commit"
> git push -u origin NEW_BRANCH
>
> substitue the name of the new branch for NEW_BRANCH

Thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Generally we could distinguish between POP3 and SMTP and IMAP and such,
> where self-signed certificates are common, and HTTP/S and generic
> connections, where they aren't. Does that seem reasonable?

The default things we warn about may differ per protocol.  For instance,
I don't really think that anybody expects (or cares) whether their SMTP
connections are encrypted, or whether that encryption is based on a
self-signed or expired certificate.  While they certainly do care with
HTTPS, and probably with POP3, I think.

So there will be a range of security actions we can take here, and in
addition, the user should be allowed to have `low', `medium', `high' and
`professional-security-professional', I mean `paranoid', settings.

> I'd add a CLI option --insecure/-k (same as curl) to override the
> default, but no more than that, and without special --batch behavior.

Yes, that might be nice.

> Can you please work against emacs-24? It's easy enough to apply the
> changes to master if that's the final decision and I don't think master
> has anything you need. Except maybe the read-only text property thing
> you added.

This won't need that, and, yes, I'm doing this based on the emacs-24
tree.  I mean, if I said the right thing to git just now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> What is an "Emacs network security manager"?  What will it manage?

The, er, network security.  >"?  Whether to connect to services that
have invalid TLS certificates and the like.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Rob Browning]

Kelvin White <kwhite@gnu.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>
>> That sounds reasonable.  Hm...  Er...  do you have a recipe for how to
>> start a new branch off of emacs-24?  >"?
>
> git checkout -b NEW_BRANCH
> git commit -m "first commit"
> git push -u origin NEW_BRANCH
>
> substitue the name of the new branch for NEW_BRANCH

And minor note -- this assumes your current tree (working directory) is
already at the commit that should be the branch point.  If not, then
you'll need to "git checkout COMMIT" first (i.e. "git checkout
emacs-24").

Or you can use this syntax to create and switch to a new branch based on
any other COMMIT:

  git checkout -b NEW_BRANCH COMMIT

So for example, assuming you didn't change any of the defaults when you
originally cloned or branched, the following commands should work.

Create local NEW_BRANCH from the tip of your local emacs-24 branch:

  git checkout -b NEW_BRANCH emacs-24

or if you want to start from the tip of your current upstream emacs-24
(from the most recently fetched tip):

  git checkout -b NEW_BRANCH origin/emacs-24

or from a commit you find some other way:

  git checkout -b NEW_BRANCH SHA1

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 17 Nov 2014 19:58:08 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What is an "Emacs network security manager"?  What will it manage?
> 
> The, er, network security.  >"?  Whether to connect to services that
> have invalid TLS certificates and the like.  

So a user will have to arrange for certificate files on her machine,
for this to work "properly"?  If so, how much will Emacs help in this?

Re: Network security manager

[Óscar Fuentes]

Rob Browning <rlb@defaultvalue.org> writes:

> or if you want to start from the tip of your current upstream emacs-24
> (from the most recently fetched tip):
>
>   git checkout -b NEW_BRANCH origin/emacs-24

Please note that this sets origin/emacs-24 as the upstream branch of
NEW_BRANCH, which is something you don't want to do when creating a
feature branch.

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> The, er, network security.  >"?  Whether to connect to services that
>> have invalid TLS certificates and the like.  
>
> So a user will have to arrange for certificate files on her machine,
> for this to work "properly"?  If so, how much will Emacs help in this?

No no.  It's about confirming the validity of the peer you're talking
to.  Haven't you ever been presented with the "This certificate is
invalid" screen in Firefox?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> The, er, network security.  >"?  Whether to connect to services that
>>> have invalid TLS certificates and the like.  
>>
>> So a user will have to arrange for certificate files on her machine,
>> for this to work "properly"?  If so, how much will Emacs help in this?
>
> No no.  It's about confirming the validity of the peer you're talking
> to.  Haven't you ever been presented with the "This certificate is
> invalid" screen in Firefox?

Will this "security manager" be effective against this type of things?

https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks

(just asking)

Re: Network security manager

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Will this "security manager" be effective against this type of things?
>
> https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks

Yes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> Will this "security manager" be effective against this type of things?
>>
>> https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks
>
> Yes.

Great feature, then.

Re: Network security manager

[Lars Magne Ingebrigtsen]

Kelvin White <kwhite@gnu.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>
>> That sounds reasonable. Hm... Er... do you have a recipe for how to
>> start a new branch off of emacs-24? >"?
>
> git checkout -b NEW_BRANCH
> git commit -m "first commit"
> git push -u origin NEW_BRANCH
>
> substitue the name of the new branch for NEW_BRANCH

Stefan requested that I didn't push this to the public repository, but
I'm not going to finish it tonight, and I need some feedback.

So I did anyway.  The new branch is called "nsm".

This is my first test case, which is nice to use because it has a
self-signed certificate:

(setq process
      (open-network-stream
       "nntpd" (get-buffer-create "*nntp*") "news.gmane.org" "nntp"
       :end-of-command "^\\([2345]\\|[.]\\).*\n"
       :capability-command "HELP\r\n"
       :success "^3"
       :starttls-function
       (lambda (capabilities)
	 (if (not (string-match "STARTTLS" capabilities))
	     nil
	   "STARTTLS\r\n"))))

;; This new function returns a certificate hash and what's wrong with it.

(gnutls-peer-status process)

;; Here's the security manager interface:

(nsm-verify-connection process "news.gmane.org" "nntp")

;; And please don't leave a gazillion connections open to my server.  >"?

(delete-process process)

Give it a whirl if you want.  It's not finished, but it does some basic
stuff, like keeping track of your responses.

But here's the feedback I need:

1) What's the proper mapping for these error messages?

  if (verification & GNUTLS_CERT_INVALID)
    warnings = Fcons (list2 (intern (":invalid"),
  if (verification & GNUTLS_CERT_REVOKED)
    warnings = Fcons (list2 (intern (":revoked"),
  if (verification & GNUTLS_CERT_SIGNER_NOT_FOUND)
    warnings = Fcons (list2 (intern (":signer-not-found"),
  if (verification & GNUTLS_CERT_SIGNER_NOT_CA)
    warnings = Fcons (list2 (intern (":self-signed"),
  if (verification & GNUTLS_CERT_INSECURE_ALGORITHM)
    warnings = Fcons (list2 (intern (":insecure"),
  if (verification & GNUTLS_CERT_NOT_ACTIVATED)
    warnings = Fcons (list2 (intern (":not-activated"),
  if (verification & GNUTLS_CERT_EXPIRED)
    warnings = Fcons (list2 (intern (":expired"),

Which one is the real "self-signed" message?  It's an important
distinction between a self-signed certificate and a forged
certificate...

2) What's the best way to ask longer questions these days?  I just did a
`read-char' on a 8-line message, but perhaps people don't like that...
Put up a help message instead?  Is there an easy-to-use
pop-up-long-help-message-buffer-while-prompting-for-three-different-chars
thing?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

I've been looking for the list of things I wrote that the security
manager had to do (in all the different failure/warning cases), but I
can't find it.  >"?  I think it was about a year ago, so I was one year
smarter back then, and feel the need to consult it...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

There's one slight privacy leak in the security manager.  To keep track
of STARTTLS man-in-the-middle downgrades, nsm needs to store data on all
STARTTLS connections you've made.  A wily hacker (I mean, the NSA) could
use this file to determine what servers you've been talking to.

The ~/.emacs.d/network-security.data will have things like

(:id "sha1:ac7feb949147490ee549b5b6c3ae7edd929ea335" :fingerprint "sha1:c0:ec:2f:01:6c:ff:4a:43:c1:a7:c7:83:4b:48:0b:3a:c5:4e:90:f9")

it it, where the :id is the sha1 of "host:port", and the latter is the
fingerprint of the certificate.

The wily hacker (I mean, the NSA) wouldn't find it easy to get a list of
the servers (because they would have to check all servers/port names in
existence), but they could use it to check for specific servers.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> 2) What's the best way to ask longer questions these days?  I just did a
> `read-char' on a 8-line message, but perhaps people don't like that...
> Put up a help message instead?  Is there an easy-to-use
> pop-up-long-help-message-buffer-while-prompting-for-three-different-chars
> thing?

We should probably also present all the pertinent data from the
certificate, but gnutls doesn't seem to have a function that just
returns all the info?

I'm looking at print_cert in output.c in the gnutls ui stuff, and it's
pretty long, but should be easy enough to adapt to Emacs.  But it's
going to be quite a few lines of C code.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Stephen Leake]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> I suggest you do it on a personal branch until the point where it
>>> becomes testable, at which point you can move it to trunk (aka
>>> "master").
>> And should personal branches be hosted in the official repository?
>
> No, I meant a branch that's elsewhere, possibly not hosted anywhere at
> all.

We will also need branches for features that are being worked on by
several developers, but are not yet in master. emacs-dynamic-module, for
one.

So some sort of naming convention for those in the official repository
would be useful.

Or some designated alternate repository.

-- 
-- Stephe

Re: Network security manager

[Sebastien Vauban]

Óscar Fuentes wrote:
> Rob Browning <rlb-A9c2TQsEEmz2Vlu8Lxc9IQ@public.gmane.org> writes:
>
>> or if you want to start from the tip of your current upstream emacs-24
>> (from the most recently fetched tip):
>>
>>   git checkout -b NEW_BRANCH origin/emacs-24
>
> Please note that this sets origin/emacs-24 as the upstream branch of
> NEW_BRANCH, which is something you don't want to do when creating a
> feature branch.

This isn't fully clear to me yet.  Can you add information about the
pitfalls of doing so?

What would advice doing, then?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But here's the feedback I need:

Haven't tested the code, but feel like I can weigh in on some of this:

>   if (verification & GNUTLS_CERT_INVALID)
>     warnings = Fcons (list2 (intern (":invalid"),

As far as I can tell from the GnuTLS example code, this is a flag that
GnuTLS sets when a cert is not trusted, rather than when it's malformed
(as I would have guessed from the name)? I.e. it doesn't ever appear on
its own?

>   if (verification & GNUTLS_CERT_REVOKED)
>     warnings = Fcons (list2 (intern (":revoked"),

This should probably be treated as fairly suspicious; since if the cert
has been explicitly revoked, there's probably a reason (not sure how
GnuTLS determines this second one; does it do OCSP revocation checks?).
SO carrying on would probably be... ill-advised. Perhaps by default fail
this completely (rather than ask), and optionally have a variable option
to override it?

>   if (verification & GNUTLS_CERT_SIGNER_NOT_FOUND)
>     warnings = Fcons (list2 (intern (":signer-not-found"),
>   if (verification & GNUTLS_CERT_SIGNER_NOT_CA)
>     warnings = Fcons (list2 (intern (":self-signed"),

Not sure which of these would indicate the common self-signed case?
Could probably be both...


>   if (verification & GNUTLS_CERT_INSECURE_ALGORITHM)
>     warnings = Fcons (list2 (intern (":insecure"),

I'd default to failing here as well; incidentally, does Emacs check the
cipher mode of the connection itself (I'm assuming this warning pertains
to the certificate itself, not the connection encryption). I have (setq
gnutls-algorithm-priority "PFS") in my .emacs, but AFAIK that is not the
default (and it does fail in some cases). For instance, in light of
POODLE, turning off SSLv3 completely would probably be a good idea, at
least as a default?

>   if (verification & GNUTLS_CERT_NOT_ACTIVATED)
>     warnings = Fcons (list2 (intern (":not-activated"),

This would probably be an issue with the clock?

>   if (verification & GNUTLS_CERT_EXPIRED)
>     warnings = Fcons (list2 (intern (":expired"),

I would expect this to be mostly benign (someone forgot to replace a
cert), but can also indicate someone stole an old cert and is using it
to MITM...

> Which one is the real "self-signed" message? It's an important
> distinction between a self-signed certificate and a forged
> certificate...

An important distinction, yes, but not one that can be made in general.
The main indicator of a forged certificate is if the presented
certificate does not match the one that is stored for the connection.
If it does, it's a possible forgery, if not, it's (probably) fine.
In the presence of rogue CAs, there's not really a better distinction to
be made, in the worst case.

However, in terms of UI we might be able to do a bit better. I'd advise
taking a look at the Certificate Patrol firefox extension
(http://patrol.psyced.org/), which does some heuristics to determine if
a changed certificate is benign or not. The main thing it does is to
look at the expiration date of the stored certificate; if that is
expired (or close to being), and the new certificate has the same CA as
the old one, it pops up a notice and continues. Otherwise, it interrupts
the connection and pops up a warning dialog with the changes highlighted
(including certificate fingerprint, CA chain etc). The common case
should be that an expired certificate is simply renewed with the same
CA, and this probably shouldn't be cause for alarm. The trouble is that
some popular sites use multiple certificates simultaneously
(corresponding to different endpoints in a server farm, I assume), which
can give some spurious popups from this algorithm.

Distinguishing these types of errors requires storing more than just the
certificate fingerprint, of course, so don't know if it's worth it. If
not, I'd treat any deviation from the stored value as suspicious.

There's also the issue of ports and addresses: If I connect to a mail
server on port 993 and get a certificate, there's a chance the same
certificate is also used for submitting mail on port 587. If so, warning
again could be avoided. On the other hand, folding the stored
certificate into just being stored per hostname would fail if it is
*not* the same certificate being used. So maybe treating ports as
completely separate (as I think you're doing now?) is best.

Finally, doing DANE verification (and trusting that more than the CA)
would be nice; but not sure how viably it is presently.


Sorry if that got a bit long; there seems to be quite a lot of cases to
consider here.

Will give the code a spin when I have chance :)

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm looking at print_cert in output.c in the gnutls ui stuff, and it's
> pretty long, but should be easy enough to adapt to Emacs.  But it's
> going to be quite a few lines of C code.

This is now implemented.

I've switched the NSM on in the nsm branch by default, and I'm now
properly warned about all the invalidly encrypted servers I'm talking
to.

So it kinda seems to work.  >"?

The only thing remaining is to have the queries be prettier.  They're
kinda messy at the moment, with too much unstructured information.

And bugs, of course.  But give the nsm branch a whirl and see whether it
works...

The related thing I was also going to implement is the "shouldn't this
connection be encrypted?" thing previously discussed.  That is, if
you're talking to an IMAP server, you most likely want that connection
to be encrypted, and if not, Emacs should tell you that it isn't.

This is trivial to implement in the NSM, but what should the defaults
be?

IMAP, POP3: I think most users would want to be warned here
SMTP, IRC: I don't think anybody cares
NNTP: They might care if they're sending a password

Uhm...  is that all the protocols?  I feel I'm forgetting one...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Óscar Fuentes]

Sebastien Vauban <sva-news@mygooglest.com>
writes:

> Óscar Fuentes wrote:
>> Rob Browning <rlb@defaultvalue.org> writes:
>>
>>> or if you want to start from the tip of your current upstream emacs-24
>>> (from the most recently fetched tip):
>>>
>>>   git checkout -b NEW_BRANCH origin/emacs-24
>>
>> Please note that this sets origin/emacs-24 as the upstream branch of
>> NEW_BRANCH, which is something you don't want to do when creating a
>> feature branch.
>
> This isn't fully clear to me yet.  Can you add information about the
> pitfalls of doing so?

If a local branch has an associated upstream branch, that's used as the
default when pushing changes. On this case this is not what the OP
wants, because he intends to push changes to a new upstream branch (that
is created the first time he pushes to "origin"), not to
origin/emacs-24.

> What would advice doing, then?

Using a git front end that implements sane defaults and shows you the
context you are working on (such as Magit) ;-)

Re: Network security manager

[Rasmus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> This is trivial to implement in the NSM, but what should the defaults
> be?

IMO: Yes.

> IMAP, POP3: I think most users would want to be warned here
> SMTP, IRC: I don't think anybody cares
> NNTP: They might care if they're sending a password

Excuse my potential ignorance:

I think I sent a (username, password) tuple to my SMTP server when
sending mails.  Why should I not care if it's encrypted?  If someone
snatched the password they'd be able to get to my IMAP, no?

In fact, in my setup, offlineimap talks to IMAP server, but Emacs talks
directly sents via the SMPT server.

-- 
Summon the Mothership!

Re: Network security manager

[Lars Magne Ingebrigtsen]

Rasmus <rasmus@gmx.us> writes:

>> IMAP, POP3: I think most users would want to be warned here
>> SMTP, IRC: I don't think anybody cares
>> NNTP: They might care if they're sending a password
>
> Excuse my potential ignorance:
>
> I think I sent a (username, password) tuple to my SMTP server when
> sending mails.  Why should I not care if it's encrypted?

Yes, if you send a password via SMTP, you should be warned.  It's fairly
rare, though.

But let's move SMTP to the same category as NNTP in that list, then.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The related thing I was also going to implement is the "shouldn't this
> connection be encrypted?" thing previously discussed.  That is, if
> you're talking to an IMAP server, you most likely want that connection
> to be encrypted, and if not, Emacs should tell you that it isn't.
>
> This is trivial to implement in the NSM, but what should the defaults
> be?
>
> IMAP, POP3: I think most users would want to be warned here
> SMTP, IRC: I don't think anybody cares
> NNTP: They might care if they're sending a password

Why do you think that sending passwords unencrypted with SMTP is ok but
with NNTP it's not ok?

So IMHO, I would always expect a warning.  For all those protocols
there's usually an encrypted version (possibly on another port), and in
general everybody should use that.  But of course it's possible that,
say, irc.foobar.org doesn't support encrypted connections, and if so,
I'd prefer to get a warning only the first time I connect.  Maybe some
query like with file-local variables and eval forms would be good where
you can say "No (don't connect)", "Yes (only this time)", "Yes (only
this emacs session)", and "Yes (always)".

> Uhm...  is that all the protocols?  I feel I'm forgetting one...

FTP maybe?

Bye,
Tassilo

Re: Network security manager

[Lars Magne Ingebrigtsen]

Tassilo Horn <tsdh@gnu.org> writes:

> I'd prefer to get a warning only the first time I connect.  Maybe some
> query like with file-local variables and eval forms would be good where
> you can say "No (don't connect)", "Yes (only this time)", "Yes (only
> this emacs session)", and "Yes (always)".

That's what the network security manager does.  (But not with file-local
variables.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 11:12:32 +0100 Toke Høiland-Jørgensen <toke@toke.dk> wrote: 

TH> incidentally, does Emacs check the cipher mode of the connection
TH> itself (I'm assuming this warning pertains to the certificate
TH> itself, not the connection encryption).

No, after establishing the connection we don't check its properties.  In
many cases, depending on the priority string, it could be very different
from what we expected IIUC, so this is neither simple nor very useful.

TH> I have (setq gnutls-algorithm-priority "PFS") in my .emacs, but
TH> AFAIK that is not the default (and it does fail in some cases). For
TH> instance, in light of POODLE, turning off SSLv3 completely would
TH> probably be a good idea, at least as a default?

This was discussed recently here and in the GnuTLS mailing list.  With
the default settings in Emacs, it's not vulnerable to POODLE.

TH> Finally, doing DANE verification (and trusting that more than the CA)
TH> would be nice; but not sure how viably it is presently.

Can you clarify?  What are the requirements and benefits in your opinion?

Also, would you like to integrate your TOFU patch with the new nsm branch?

Thanks
Ted

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 15:41:50 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> The related thing I was also going to implement is the "shouldn't this
LMI> connection be encrypted?" thing previously discussed.  That is, if
LMI> you're talking to an IMAP server, you most likely want that connection
LMI> to be encrypted, and if not, Emacs should tell you that it isn't.

LMI> This is trivial to implement in the NSM, but what should the defaults
LMI> be?

Definitely yes.  Unencrypted should be the exception nowadays.

LMI> IMAP, POP3: I think most users would want to be warned here
LMI> SMTP, IRC: I don't think anybody cares
LMI> NNTP: They might care if they're sending a password

IRC should be upgraded if possible.  Freenode at least supports both
modes.

SMTP is tricky.  I would care if sending to an external server but not
internally, and there's no easy way to distinguish them.  Also if the
message itself is encrypted, I wouldn't care.

LMI> Uhm...  is that all the protocols?  I feel I'm forgetting one...

HTTP?  It's certainly recommended to encrypt it nowadays.

Ted

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 00:26:17 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> There's one slight privacy leak in the security manager.  To keep track
LMI> of STARTTLS man-in-the-middle downgrades, nsm needs to store data on all
LMI> STARTTLS connections you've made.  A wily hacker (I mean, the NSA) could
LMI> use this file to determine what servers you've been talking to.

LMI> The ~/.emacs.d/network-security.data will have things like

LMI> (:id "sha1:ac7feb949147490ee549b5b6c3ae7edd929ea335" :fingerprint "sha1:c0:ec:2f:01:6c:ff:4a:43:c1:a7:c7:83:4b:48:0b:3a:c5:4e:90:f9")

LMI> it it, where the :id is the sha1 of "host:port", and the latter is the
LMI> fingerprint of the certificate.

LMI> The wily hacker (I mean, the NSA) wouldn't find it easy to get a list of
LMI> the servers (because they would have to check all servers/port names in
LMI> existence), but they could use it to check for specific servers.

You could name the file `~/.emacs.d/network-security.gpg' by default :)

Ted

Re: Network security manager

[Ted Zlatanov]

I'm testing and using the NSM.  The number one thing it needs is a
`tabulated-list-mode' interface to review all the entries.  See also my
note about the GPG key management functionality, which I think naturally
fits in the NSM.

(And I'm starting to wonder if managing auth-source tokens doesn't make
sense in the NSM as well.)

Ted

Re: Network security manager

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> I'd prefer to get a warning only the first time I connect.  Maybe
>> some query like with file-local variables and eval forms would be
>> good where you can say "No (don't connect)", "Yes (only this time)",
>> "Yes (only this emacs session)", and "Yes (always)".
>
> That's what the network security manager does.

Ah, cool.

> (But not with file-local variables.)

No, of course.  I just wanted to mention an example where something
similar is done.  ("Similar" in the sense that the query is similar and
the answer might or might not to be persisted.)

Bye,
Tassilo

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> TH> incidentally, does Emacs check the cipher mode of the connection
> TH> itself (I'm assuming this warning pertains to the certificate
> TH> itself, not the connection encryption).
>
> No, after establishing the connection we don't check its properties.  In
> many cases, depending on the priority string, it could be very different
> from what we expected IIUC, so this is neither simple nor very useful.

Well, yes, that's exactly what we do.  Open the connection, and then
check the properties.  >"?

> Also, would you like to integrate your TOFU patch with the new nsm branch?

The NSM does TOFU.  No patch necessary.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> Uhm...  is that all the protocols?  I feel I'm forgetting one...
>
> HTTP?  It's certainly recommended to encrypt it nowadays.

Heh.  That's the one.  >"?

I wonder whether I can just adjust url-http to use
`open-network-stream'...  that should give us the NSM action "for
free".  Will investigate.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I'm testing and using the NSM.  The number one thing it needs is a
> `tabulated-list-mode' interface to review all the entries.  See also my
> note about the GPG key management functionality, which I think naturally
> fits in the NSM.

Sure...  but since there's almost nothing human-readable (or something a
machine can transform into something human-readable), I'm not quite sure
what it should display...

I mean, I can see a user wanting to say to Emacs "delete everything you
know about me contacting news.gmane.org", but there's no real way to
match that up to the entries in the file unless you also know the port
number/service name used.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> LMI> Uhm...  is that all the protocols?  I feel I'm forgetting one...
>>
>> HTTP?  It's certainly recommended to encrypt it nowadays.
>
> Heh.  That's the one.  >"?
>
> I wonder whether I can just adjust url-http to use
> `open-network-stream'...  that should give us the NSM action "for
> free".  Will investigate.

It already does.  Does anybody know of https sites with interesting
types of un-verifiable certificates so that I can test?  Self-signed or
expired certificates would be nice...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It already does.  Does anybody know of https sites with interesting
> types of un-verifiable certificates so that I can test?  Self-signed or
> expired certificates would be nice...

Found one.  And it just works:

-------
Issuer: O=Cybertrust Inc,CN=Cybertrust Public SureServer SV CA
Host informasjon: C=US,ST=MA,L=Cambridge,O=Akamai Technologies\, Inc.,CN=a248.e.akamai.net
Public key: RSA, signature: RSA-SHA1, security level: Low
Valid from: 2014-06-12, valid to: 2015-06-12

The TLS connection to tv.eurosport.com:443 is insecure
for the following reason:

certificate host does not match hostname

Continue connecting? (No, Session only, Always)
------

The issuer and host information has to be formatted prettier, though.
And is it at all interesting to display the "Public key: RSA, signature:
RSA-SHA1" bits?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Stefan Monnier]

> We will also need branches for features that are being worked on by
> several developers, but are not yet in master.  emacs-dynamic-module,
> for one.

We've used "plain" names in the past for such branches.  I'd be happy to
see a `dynamic-module' branch added to the Git repository.


        Stefan

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 16:29:30 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
TH> incidentally, does Emacs check the cipher mode of the connection
TH> itself (I'm assuming this warning pertains to the certificate
TH> itself, not the connection encryption).
>> 
>> No, after establishing the connection we don't check its properties.  In
>> many cases, depending on the priority string, it could be very different
>> from what we expected IIUC, so this is neither simple nor very useful.

LMI> Well, yes, that's exactly what we do.  Open the connection, and then
LMI> check the properties.  >"?

You're checking the certificate, but not the cipher mode or anything
else that was negotiated.  I think that's what Toke meant.

>> Also, would you like to integrate your TOFU patch with the new nsm branch?

LMI> The NSM does TOFU.  No patch necessary.

Cool.

>> I'm testing and using the NSM.  The number one thing it needs is a
>> `tabulated-list-mode' interface to review all the entries.  See also my
>> note about the GPG key management functionality, which I think naturally
>> fits in the NSM.

LMI> Sure...  but since there's almost nothing human-readable (or something a
LMI> machine can transform into something human-readable), I'm not quite sure
LMI> what it should display...

The list of explicitly saved security exceptions.

LMI> I mean, I can see a user wanting to say to Emacs "delete everything you
LMI> know about me contacting news.gmane.org", but there's no real way to
LMI> match that up to the entries in the file unless you also know the port
LMI> number/service name used.

True, but I really don't see the harm in saving those in cleartext. Like
I said, I would use a .gpg file if I was worried about leaking that
data. With the current approach I think you'll see two problems:

1) cruft will accumulate, since you don't know what's what

2) when servers change names or ports, you don't know what to remove

HTH
Ted

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 10:49:48 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> We will also need branches for features that are being worked on by
>> several developers, but are not yet in master.  emacs-dynamic-module,
>> for one.

SM> We've used "plain" names in the past for such branches.  I'd be happy to
SM> see a `dynamic-module' branch added to the Git repository.

Aurelien, would you like to redo your branch against the new Emacs Git
repo and put it there as a feature branch? I can do it too, if you
prefer.  It would make it very easy to test and share your code.

Ted

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 16:45:31 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> -------
LMI> Issuer: O=Cybertrust Inc,CN=Cybertrust Public SureServer SV CA
LMI> Host informasjon: C=US,ST=MA,L=Cambridge,O=Akamai Technologies\, Inc.,CN=a248.e.akamai.net
LMI> Public key: RSA, signature: RSA-SHA1, security level: Low
LMI> Valid from: 2014-06-12, valid to: 2015-06-12

LMI> The TLS connection to tv.eurosport.com:443 is insecure
LMI> for the following reason:

LMI> certificate host does not match hostname

LMI> Continue connecting? (No, Session only, Always)
LMI> ------

LMI> The issuer and host information has to be formatted prettier, though.

Details :)

LMI> And is it at all interesting to display the "Public key: RSA, signature:
LMI> RSA-SHA1" bits?

I think so.

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> Sure...  but since there's almost nothing human-readable (or something a
> LMI> machine can transform into something human-readable), I'm not quite sure
> LMI> what it should display...
>
> The list of explicitly saved security exceptions.

But they are per sha1, so it's not really feasible to do anything about
it for a human.

> LMI> I mean, I can see a user wanting to say to Emacs "delete everything you
> LMI> know about me contacting news.gmane.org", but there's no real way to
> LMI> match that up to the entries in the file unless you also know the port
> LMI> number/service name used.
>
> True, but I really don't see the harm in saving those in cleartext.

I don't like the information leakage.

> Like I said, I would use a .gpg file if I was worried about leaking
> that data. With the current approach I think you'll see two problems:

GPG isn't feasible because nobody wants to type passwords.

> 1) cruft will accumulate, since you don't know what's what

Does it matter?

> 2) when servers change names or ports, you don't know what to remove

You don't have to remove anything.  No manual administration should be
necessary.  Unless you want to revoke a security exception.  And then
you might as well just delete the entire file.  It's not like it's a lot
of bother hitting the `a' key a couple times the next time you start
up...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

I parsed this bits a bit more, and ended up with

Certificate issued by Cybertrust Public SureServer SV CA
Issued to Akamai Technologies, Inc.
Certificate host name: a248.e.akamai.net
Public key: RSA, signature: RSA-SHA1, security level: Low
Valid from: 2014-06-12, valid to: 2015-06-12

The TLS connection to tv.eurosport.com:443 is insecure
for the following reason:

certificate host does not match hostname

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Hey, that went a lot faster than expected.  I can't remember anything
more to implement in this area...

Barring bugs, Emacs now has a more complete network security handling
thingamabob than...  most things.

Time to merge with the trunk, or should it percolate in the nsm branch a
bit more?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Oh, I should probably write a manual entry on this?  Or should I?  I
mean, how interested are the users in all the ins and outs of network
security?  I think "not very".

Perhaps it's sufficient to mention `nsm-security-level' and leave it at
that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

I just had an interesting user interface experience.

eww loads images asynchronously, and one of the images that it tried to
fetch had an invalid certificate, so the NSM queried me about this
certificate.  Which was kinda surprising, since I was in the middle of
typing something else...

I see at least two ways of dealing with this:

1) Drop certificate checking for images in shr.  I mean, do we care?

2) If being run from the async context (how do we check for that?),
refuse to handle insecure TLS connections silently.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Glenn Morris]

Lars Magne Ingebrigtsen wrote:

> Stefan requested that I didn't push this to the public repository, but
> I'm not going to finish it tonight, and I need some feedback.
>
> So I did anyway. 

Isn't that somewhat impolite? (And precedent-setting?)

Re: Network security manager

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

On 11/18/2014 09:03 AM, Glenn Morris wrote:
> Lars Magne Ingebrigtsen wrote:
> 
>> Stefan requested that I didn't push this to the public repository, but
>> I'm not going to finish it tonight, and I need some feedback.
>>
>> So I did anyway. 
> 
> Isn't that somewhat impolite? (And precedent-setting?)

Pushing to a development branch is somewhat less rude than pushing to emacs-24 proper, right?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 18:00:15 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> I just had an interesting user interface experience.
LMI> eww loads images asynchronously, and one of the images that it tried to
LMI> fetch had an invalid certificate, so the NSM queried me about this
LMI> certificate.  Which was kinda surprising, since I was in the middle of
LMI> typing something else...

LMI> I see at least two ways of dealing with this:

LMI> 1) Drop certificate checking for images in shr.  I mean, do we care?

I think we care.

LMI> 2) If being run from the async context (how do we check for that?),
LMI> refuse to handle insecure TLS connections silently.

Works for me, as long as the errors are reviewable in the NSM.  I should
be able to go somewhere and hit a button "allow this cert from now on".

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> 1) Drop certificate checking for images in shr.  I mean, do we care?
>
> I think we care.

What are the security implications of inserting an image from a source
we can't validate?  99% of the images aren't over TLS, anyway, and
aren't validated...

> LMI> 2) If being run from the async context (how do we check for that?),
> LMI> refuse to handle insecure TLS connections silently.
>
> Works for me, as long as the errors are reviewable in the NSM.  I should
> be able to go somewhere and hit a button "allow this cert from now on".

shr should really insert "broken image" markers into the buffers (and
"loading images"), and then the user could just hit RET on one of the
broken images and then get queried about the certificate
interactively...

Which reminds me: We need a way to determine that Emacs is running
non-interactively as well as being run from an async context.  What's
the way to do that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 17:15:09 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> Sure...  but since there's almost nothing human-readable (or something a
LMI> machine can transform into something human-readable), I'm not quite sure
LMI> what it should display...
>> 
>> The list of explicitly saved security exceptions.

LMI> But they are per sha1, so it's not really feasible to do anything about
LMI> it for a human.

That's how you implemented it.  It's not necessarily how it should be.

>> True, but I really don't see the harm in saving those in cleartext.

LMI> I don't like the information leakage.

Then the NSM is really a blob storage manager.

>> Like I said, I would use a .gpg file if I was worried about leaking
>> that data. With the current approach I think you'll see two problems:

LMI> GPG isn't feasible because nobody wants to type passwords.

Whuhh?

>> 1) cruft will accumulate, since you don't know what's what

LMI> Does it matter?

Yes!  Regular reviews are essential to actually managing security
exceptions.

>> 2) when servers change names or ports, you don't know what to remove

LMI> You don't have to remove anything.  No manual administration should be
LMI> necessary.  Unless you want to revoke a security exception.  And then
LMI> you might as well just delete the entire file.  It's not like it's a lot
LMI> of bother hitting the `a' key a couple times the next time you start
LMI> up...

Yes, it's a bother.  We're talking about potentially dozens or hundreds
of exceptions in a large enterprise.  But let's assume the `a' key is
large and easy to hit.

Scenario 1: you allow a compromised server accidentally.  You now can't
review the exception list to remove that compromise.

Scenario 2: someone allows a compromised server on purpose in a few
seconds.  You have no idea it happened.

I'm sure there are other scenarios, but please don't make this a
write-only data store.

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Jochen Hein <jochen@jochen.org> writes:

> [Quoted text removed due to X-No-Archive]

Yeah, listing the dates might be useful...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> GPG isn't feasible because nobody wants to type passwords.
>
> Whuhh?

Yeah?

> Yes, it's a bother.  We're talking about potentially dozens or hundreds
> of exceptions in a large enterprise.  But let's assume the `a' key is
> large and easy to hit.
>
> Scenario 1: you allow a compromised server accidentally.  You now can't
> review the exception list to remove that compromise.
>
> Scenario 2: someone allows a compromised server on purpose in a few
> seconds.  You have no idea it happened.
>
> I'm sure there are other scenarios, but please don't make this a
> write-only data store.

Well, we could have a setting that says that the NSM should re-query
security exceptions...

On the other hand, we could store the server names in plain text when we
store security exceptions to make reviews easier.  That is, keep the
hash-only thing for STARTTLS man-in-the-middle tracking and the like,
but if the user registers an exception, then we'd stash the server name
in there, too.

This would avoid leaving a complete list of STARTTLS servers in that
file, but still allow easy removal of specific exceptions.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 18:28:26 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> 1) Drop certificate checking for images in shr.  I mean, do we care?
>> 
>> I think we care.

LMI> What are the security implications of inserting an image from a source
LMI> we can't validate?

Malicious binary payloads in images are quite common.  There are also
attacks/exploits/hacks that load Javascript from images.  Regardless,
you'd be lowering the security level of the data exchange.

LMI> 99% of the images aren't over TLS, anyway, and aren't validated...

OK, but that's not relevant to the above :)

LMI> 2) If being run from the async context (how do we check for that?),
LMI> refuse to handle insecure TLS connections silently.
>> 
>> Works for me, as long as the errors are reviewable in the NSM.  I should
>> be able to go somewhere and hit a button "allow this cert from now on".

LMI> shr should really insert "broken image" markers into the buffers (and
LMI> "loading images"), and then the user could just hit RET on one of the
LMI> broken images and then get queried about the certificate
LMI> interactively...

OK with me, that's a good solution for this particular case.  But there
will be others where you can't see the things that went wrong in the
background.  I suggested a modeline indicator previously... it's better
than silent failure, right?

LMI> Which reminds me: We need a way to determine that Emacs is running
LMI> non-interactively as well as being run from an async context.  What's
LMI> the way to do that?

I know in non-interactive mode the minibuffer reads from stdio, so
there's definitely some distinction for batch mode. But I don't know how
to distinguish it in ELisp or check the async mode, sorry.

Ted

Re: Network security manager

[Eli Zaretskii]

> Date: Tue, 18 Nov 2014 09:17:05 -0800
> From: Daniel Colascione <dancol@dancol.org>
> Cc: Kelvin White <kwhite@gnu.org>,
> 	Stephen Leake <stephen_leake@stephe-leake.org>,
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> >> Stefan requested that I didn't push this to the public repository, but
> >> I'm not going to finish it tonight, and I need some feedback.
> >>
> >> So I did anyway. 
> > 
> > Isn't that somewhat impolite? (And precedent-setting?)
> 
> Pushing to a development branch is somewhat less rude than pushing to emacs-24 proper, right?

I can think of a few things that are even more rude than that.  So?

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 18 Nov 2014 18:28:26 +0100
> 
> shr should really insert "broken image" markers into the buffers (and
> "loading images"), and then the user could just hit RET on one of the
> broken images and then get queried about the certificate
> interactively...

Do that, and you've added one more reason for me to disable the whole
darn thing right away.

I mean, how much annoyance should one endure to be "secure"? is there
any limit?

> Which reminds me: We need a way to determine that Emacs is running
> non-interactively as well as being run from an async context.  What's
> the way to do that?

It's in the manual, I'd say.

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 18:36:25 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> GPG isn't feasible because nobody wants to type passwords.
>> 
>> Whuhh?

LMI> Yeah?

Let me rephrase: I don't think that's accurate :)

>> Yes, it's a bother.  We're talking about potentially dozens or hundreds
>> of exceptions in a large enterprise.  But let's assume the `a' key is
>> large and easy to hit.
>> 
>> Scenario 1: you allow a compromised server accidentally.  You now can't
>> review the exception list to remove that compromise.
>> 
>> Scenario 2: someone allows a compromised server on purpose in a few
>> seconds.  You have no idea it happened.
>> 
>> I'm sure there are other scenarios, but please don't make this a
>> write-only data store.

LMI> On the other hand, we could store the server names in plain text when we
LMI> store security exceptions to make reviews easier.  That is, keep the
LMI> hash-only thing for STARTTLS man-in-the-middle tracking and the like,
LMI> but if the user registers an exception, then we'd stash the server name
LMI> in there, too.

LMI> This would avoid leaving a complete list of STARTTLS servers in that
LMI> file, but still allow easy removal of specific exceptions.

Works for me, as long as I can customize it to always store the server
name and port for all stored data.

Ted

Re: Network security manager

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Tue, 18 Nov 2014 12:40:33 -0500
> 
> On Tue, 18 Nov 2014 18:28:26 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
> 
> LMI> What are the security implications of inserting an image from a source
> LMI> we can't validate?
> 
> Malicious binary payloads in images are quite common.  There are also
> attacks/exploits/hacks that load Javascript from images.  Regardless,
> you'd be lowering the security level of the data exchange.

Let the user opt-in for this, please!  Do not force it on them.

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 18 Nov 2014 18:28:26 +0100
>> 
>> shr should really insert "broken image" markers into the buffers (and
>> "loading images"), and then the user could just hit RET on one of the
>> broken images and then get queried about the certificate
>> interactively...
>
> Do that, and you've added one more reason for me to disable the whole
> darn thing right away.

I meant that generally for all images it can't download, for one reason
or other.  

> I mean, how much annoyance should one endure to be "secure"? is there
> any limit?

That was my question.  Do we care about the verifiability (is that a
word?) of the servers where images come from?  I certainly don't, so my
vote would be to disable these checks in shr for images.

>> Which reminds me: We need a way to determine that Emacs is running
>> non-interactively as well as being run from an async context.  What's
>> the way to do that?
>
> It's in the manual, I'd say.

Do you know where?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> What are the security implications of inserting an image from a source
> LMI> we can't validate?
>
> Malicious binary payloads in images are quite common.  There are also
> attacks/exploits/hacks that load Javascript from images.

I really hope we don't have any exploitable bugs in the image handling
code.  

> Regardless, you'd be lowering the security level of the data exchange.

I don't think we care...

> LMI> 99% of the images aren't over TLS, anyway, and aren't validated...
>
> OK, but that's not relevant to the above :)

Sure it it.  >"?

> OK with me, that's a good solution for this particular case.  But there
> will be others where you can't see the things that went wrong in the
> background.  I suggested a modeline indicator previously... it's better
> than silent failure, right?

Well...  No, annoying the user with things the user doesn't care about
is worse than silent failure.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 18:54:44 +0100
> 
> >> Which reminds me: We need a way to determine that Emacs is running
> >> non-interactively as well as being run from an async context.  What's
> >> the way to do that?
> >
> > It's in the manual, I'd say.
> 
> Do you know where?

Not by heart.  But it should be easy to find.

Re: Network security manager

[Lars Magne Ingebrigtsen]

Thinking about it a bit more, I see one security implication when
downloading images in eww without verifying the certificates.

Let's say you've logged in to https://example.com so you have a login
cookie.  Somebody could man-in-the-middle you between when you've loaded
the HTML and when you're loading the images from https://example.com,
and then you will be sending your login cookie to that man who sits
there in the middle.

This is the sort of scenario that Professional Security Professionals
love.

So...  failing and leaving a "broken image" icon in the buffer is
probably the safe thing to do.  (It's what all other browsers do.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 18 Nov 2014 18:54:44 +0100
>> 
>> >> Which reminds me: We need a way to determine that Emacs is running
>> >> non-interactively as well as being run from an async context.  What's
>> >> the way to do that?
>> >
>> > It's in the manual, I'd say.
>> 
>> Do you know where?
>
> Not by heart.  But it should be easy to find.

I should, but I grepped through the lispref/*.texi for "batch", and
couldn't find anything likely.  Then I grepped src/*.c.  Then I asked.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I should, but I grepped through the lispref/*.texi for "batch", and
> couldn't find anything likely.  Then I grepped src/*.c.  Then I asked.  >"?

Found it!  `noninteractive'.

Now I just need to know how to determine if a function is running from
an async callback...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 18:57:15 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:

LMI> What are the security implications of inserting an image from a source
LMI> we can't validate?
>> 
>> Malicious binary payloads in images are quite common.  There are also
>> attacks/exploits/hacks that load Javascript from images.

LMI> I really hope we don't have any exploitable bugs in the image handling
LMI> code.

On many platforms (NS comes to mind) image handling happens before Emacs
knows about it.  So this is not necessarily an Emacs issue.

Here's a list of libpng (just picking one library out of many that Emacs
uses) CVEs: http://www.cvedetails.com/vulnerability-list/vendor_id-7294/Libpng.html

Do we care? I do, others may not. Regardless, I don't think Emacs should
choose to sometimes disregard the HTTP/S channel's security checks.  If
it does, it would be a rather unique web browser.

>> OK with me, that's a good solution for this particular case.  But there
>> will be others where you can't see the things that went wrong in the
>> background.  I suggested a modeline indicator previously... it's better
>> than silent failure, right?

LMI> Well...  No, annoying the user with things the user doesn't care about
LMI> is worse than silent failure.  >"?

I don't think a passive indicator e.g. in the modeline is annoying. If
you make the list of failures accessible somehow, the rest can be done
by add-ons, so we don't need to figure it out now.

Ted

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 19:13:53 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> >> Cc: emacs-devel@gnu.org
> >> Date: Tue, 18 Nov 2014 18:54:44 +0100
> >> 
> >> >> Which reminds me: We need a way to determine that Emacs is running
> >> >> non-interactively as well as being run from an async context.  What's
> >> >> the way to do that?
> >> >
> >> > It's in the manual, I'd say.
> >> 
> >> Do you know where?
> >
> > Not by heart.  But it should be easy to find.
> 
> I should, but I grepped through the lispref/*.texi for "batch", and
> couldn't find anything likely.  Then I grepped src/*.c.  Then I asked.  >"?

Grepping is not the most efficient way of finding stuff in the Emacs
manuals (or any Info manuals).  Try

  i noninteractive RET

(I don't understand what you mean by "from an async context", so I
cannot help you there.)

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Now I just need to know how to determine if a function is running from
> an async callback...

There's a C variable called running_asynch_code, but it's not exposed to
Lisp.  Unless there's something else in there that can be used, should
we just export that to Lisp, like the `noninteractive' variable?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 19:18:35 +0100
> 
> Now I just need to know how to determine if a function is running from
> an async callback...

What is an "async callback"?

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 18 Nov 2014 19:29:02 +0100
> Cc: emacs-devel@gnu.org
> 
> There's a C variable called running_asynch_code, but it's not exposed to
> Lisp.  Unless there's something else in there that can be used, should
> we just export that to Lisp, like the `noninteractive' variable?

Why do you need that exposed?  If you want to know when some Lisp is
run by a process filter, then make the filter function pass an
argument to that Lisp saying it's run by a filter.  Will that do?

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 18 Nov 2014 19:18:35 +0100
>> 
>> Now I just need to know how to determine if a function is running from
>> an async callback...
>
> What is an "async callback"?

Process sentinels, filters, timers and stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 19:41:04 +0100
> 
> >> Now I just need to know how to determine if a function is running from
> >> an async callback...
> >
> > What is an "async callback"?
> 
> Process sentinels, filters, timers and stuff.

But these always know that they are run asynchronously, so what's the
problem you are trying to solve?

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 18 Nov 2014 19:29:02 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> There's a C variable called running_asynch_code, but it's not exposed to
>> Lisp.  Unless there's something else in there that can be used, should
>> we just export that to Lisp, like the `noninteractive' variable?
>
> Why do you need that exposed?  If you want to know when some Lisp is
> run by a process filter, then make the filter function pass an
> argument to that Lisp saying it's run by a filter.  Will that do?

Passing an argument to this function, which is way way way down in the
call stack, isn't practical.  And it would be irrelevant to most of the
url.el code, anyway.

Being able to discern whether you're running code asynchronously seems
like a generally useful thing.  Our goal is to make Emacs more
asynchronous, so we need a way for functions to know what context they
are being run in, so they know what they're allowed to do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 20:19:58 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> >> Date: Tue, 18 Nov 2014 19:29:02 +0100
> >> Cc: emacs-devel@gnu.org
> >> 
> >> There's a C variable called running_asynch_code, but it's not exposed to
> >> Lisp.  Unless there's something else in there that can be used, should
> >> we just export that to Lisp, like the `noninteractive' variable?
> >
> > Why do you need that exposed?  If you want to know when some Lisp is
> > run by a process filter, then make the filter function pass an
> > argument to that Lisp saying it's run by a filter.  Will that do?
> 
> Passing an argument to this function, which is way way way down in the
> call stack, isn't practical.  And it would be irrelevant to most of the
> url.el code, anyway.

Then bind a variable to a special value, and test it down below.  Will
that fit the bill?

Re: Network security manager

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On 11/18/2014 11:19 AM, Lars Magne Ingebrigtsen wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
>>> Date: Tue, 18 Nov 2014 19:29:02 +0100
>>> Cc: emacs-devel@gnu.org
>>>
>>> There's a C variable called running_asynch_code, but it's not exposed to
>>> Lisp.  Unless there's something else in there that can be used, should
>>> we just export that to Lisp, like the `noninteractive' variable?
>>
>> Why do you need that exposed?  If you want to know when some Lisp is
>> run by a process filter, then make the filter function pass an
>> argument to that Lisp saying it's run by a filter.  Will that do?
> 
> Passing an argument to this function, which is way way way down in the
> call stack, isn't practical.  And it would be irrelevant to most of the
> url.el code, anyway.
> 
> Being able to discern whether you're running code asynchronously seems
> like a generally useful thing.  Our goal is to make Emacs more
> asynchronous, so we need a way for functions to know what context they
> are being run in, so they know what they're allowed to do

You can also look at that as making it harder to debug code that magically behaves differently when invoked through the repl or the debugger. Why does this code need to know whether it's being run asynchronously?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Then bind a variable to a special value, and test it down below.  Will
> that fit the bill?

It might, but this seems like a generally useful thing to have.  So I've
implemented it and pushed it to the nsm branch.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Achim Gratz]

Lars Magne Ingebrigtsen writes:
>> I think I sent a (username, password) tuple to my SMTP server when
>> sending mails.  Why should I not care if it's encrypted?
>
> Yes, if you send a password via SMTP, you should be warned.  It's fairly
> rare, though.

Huh?  No ISP I know of lets you use their SMTP servers without
authentication if you're on dialup or anything else with dynamic IP
anymore.  POP before SMTP whitelisting has been falling out of favor…


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> But here's the feedback I need:
>
> Haven't tested the code, but feel like I can weigh in on some of this:
>
>>   if (verification & GNUTLS_CERT_INVALID)
>>     warnings = Fcons (list2 (intern (":invalid"),
>
> As far as I can tell from the GnuTLS example code, this is a flag that
> GnuTLS sets when a cert is not trusted, rather than when it's malformed
> (as I would have guessed from the name)? I.e. it doesn't ever appear on
> its own?

Ah, right, so it's a general catch-all that's set in addition to other
flags?

>>   if (verification & GNUTLS_CERT_REVOKED)
>>     warnings = Fcons (list2 (intern (":revoked"),
>
> This should probably be treated as fairly suspicious; since if the cert
> has been explicitly revoked, there's probably a reason (not sure how
> GnuTLS determines this second one; does it do OCSP revocation checks?).
> SO carrying on would probably be... ill-advised. Perhaps by default fail
> this completely (rather than ask), and optionally have a variable option
> to override it?

I don't see why we shouldn't ask.  The user should be able to decide
without setting variables.

>>   if (verification & GNUTLS_CERT_SIGNER_NOT_FOUND)
>>     warnings = Fcons (list2 (intern (":signer-not-found"),
>>   if (verification & GNUTLS_CERT_SIGNER_NOT_CA)
>>     warnings = Fcons (list2 (intern (":self-signed"),
>
> Not sure which of these would indicate the common self-signed case?
> Could probably be both...

Yeah, that's what I'm mainly wondering about.

>>   if (verification & GNUTLS_CERT_NOT_ACTIVATED)
>>     warnings = Fcons (list2 (intern (":not-activated"),
>
> This would probably be an issue with the clock?
>
>>   if (verification & GNUTLS_CERT_EXPIRED)
>>     warnings = Fcons (list2 (intern (":expired"),
>
> I would expect this to be mostly benign (someone forgot to replace a
> cert), but can also indicate someone stole an old cert and is using it
> to MITM...

Yup.

> However, in terms of UI we might be able to do a bit better. I'd advise
> taking a look at the Certificate Patrol firefox extension
> (http://patrol.psyced.org/), which does some heuristics to determine if
> a changed certificate is benign or not. The main thing it does is to
> look at the expiration date of the stored certificate; if that is
> expired (or close to being), and the new certificate has the same CA as
> the old one, it pops up a notice and continues.

Interesting.  It does this even if the new certificate is valid?  To
mitigate against rogue CAs?

The NSM will also warn about new certificates if the user has switched
to `paranoid', but it doesn't compare old and new CAs and stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Achim Gratz <Stromeko@nexgo.de> writes:

> Huh?  No ISP I know of lets you use their SMTP servers without
> authentication if you're on dialup or anything else with dynamic IP
> anymore.  POP before SMTP whitelisting has been falling out of favor…

It's common for the ISPs here to use an SMTP submit port setup (with
TLS) that only accepts messages from within their network.  And without
authentication.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Achim Gratz]

Lars Magne Ingebrigtsen writes:
> It already does.  Does anybody know of https sites with interesting
> types of un-verifiable certificates so that I can test?  Self-signed or
> expired certificates would be nice...

https://revoked.grc.com/


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Network security manager

[Lars Magne Ingebrigtsen]

Achim Gratz <Stromeko@nexgo.de> writes:

> https://revoked.grc.com/

Thanks.

----

Security Certificate
Revocation Awareness Test
If you can see this (and apparently you can), you
are using a revocation UNaware web browser!

----

I guess gnutls doesn't do revocation on its own.  Not that I see how it
can do that.  Requires network chatter and stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I guess gnutls doesn't do revocation on its own.  Not that I see how it
> can do that.  Requires network chatter and stuff.

Heh:

====
* Google's Chrome browser is the least certificate-secure browser on the
  Internet. It puts speed before security, so it is the only browser on the
  Internet to disable certificate checking by default. 
====

NOT TRUE!!!1!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 20:26:37 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then bind a variable to a special value, and test it down below.  Will
> > that fit the bill?
> 
> It might, but this seems like a generally useful thing to have.  So I've
> implemented it and pushed it to the nsm branch.

Sigh...

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ah, right, so it's a general catch-all that's set in addition to other
> flags?

Yeah, seems to be: http://www.gnutls.org/manual/html_node/Verifying-a-certificate.html

> I don't see why we shouldn't ask. The user should be able to decide
> without setting variables.

Sure. I just mean that it would probably be nice to have some sort of
distinction for severeness (perhaps coupled to the configured paranoia
level), so that, for instance, an expired certificate that is replaced
with a new one raises less of a warning than a revoked certificate, or
one that doesn't match the trust store.

>> Not sure which of these would indicate the common self-signed case?
>> Could probably be both...
>
> Yeah, that's what I'm mainly wondering about.

Well, according to the documentation:
http://www.gnutls.org/manual/html_node/Verifying-X_002e509-certificate-paths.html

GNUTLS_CERT_SIGNER_NOT_CA means:

    "The certificate’s signer was not a CA. This may happen if this was
    a version 1 certificate, which is common with some CAs, or a version
    3 certificate without the basic constrains extension."

Whereas GNUTLS_CERT_SIGNER_NOT_FOUND is the common "we don't trust
whoever signed this". So I'd think that GNUTLS_CERT_SIGNER_NOT_FOUND
would be returned for all self-signed certificates, and possibly
GNUTLS_CERT_SIGNER_NOT_CA in addition. If GNUTLS_CERT_SIGNER_NOT_CA is
returned for a legitimately signed certificate (from the trust store),
the CA is probably doing something wrong.

> Interesting. It does this even if the new certificate is valid? To
> mitigate against rogue CAs?

Yep, that's basically the whole reason for the extension.

> The NSM will also warn about new certificates if the user has switched
> to `paranoid', but it doesn't compare old and new CAs and stuff.

Right, cool. Will give it a spin and see if I can break it as soon as
I've compiled your branch :)

-Toke

Re: Network security manager

[Stefan Monnier]

>> Now I just need to know how to determine if a function is running from
>> an async callback...
> There's a C variable called running_asynch_code, but it's not exposed to
> Lisp.  Unless there's something else in there that can be used, should
> we just export that to Lisp, like the `noninteractive' variable?

I guess we could export it (read-only) to Lisp, yes.

Tho, maybe it would be worth it to have a separate var for it,
writeable, which we could arrange to consider (some?) process filters to
be synchronous when run during a sync call to accept-process-output.


        Stefan

Re: Network security manager

[N. Jackson]

At 11:40 -0400 on Tuesday 2014-11-18, Lars Magne Ingebrigtsen wrote:

> Does anybody know of https sites with interesting types of
> un-verifiable certificates so that I can test? Self-signed or expired
> certificates would be nice...

The Parabola GNU/Linux-libre site at https://www.parabola.nu/

Re: Network security manager

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Tue, 18 Nov 2014 15:40:37 -0500
> 
> >> Now I just need to know how to determine if a function is running from
> >> an async callback...
> > There's a C variable called running_asynch_code, but it's not exposed to
> > Lisp.  Unless there's something else in there that can be used, should
> > we just export that to Lisp, like the `noninteractive' variable?
> 
> I guess we could export it (read-only) to Lisp, yes.

That variable is an internal implementation detail designed for a
specific purpose that has nothing to do with running asynchronously.
Exposing it to Lisp is IMO extremely unclean.  Especially since this
is not really needed anyway, because Lars's use case has a much
simpler solution that doesn't require any such kludges.

So I'm violently opposed to this.  But since Lars blatantly ignored
even your requests, who am I to complain?

Re: Network security manager

[Toke Høiland-Jørgensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> This was discussed recently here and in the GnuTLS mailing list. With
> the default settings in Emacs, it's not vulnerable to POODLE.

Well it could also be something like warning when Perfect Forward
Security is not available (for instance). However, as long as
gnutls-algorithm-priority keeps working I can live with that. :)

> TH> Finally, doing DANE verification (and trusting that more than the CA)
> TH> would be nice; but not sure how viably it is presently.
>
> Can you clarify?  What are the requirements and benefits in your
> opinion?

Well, DANE allows for storing certificate info in DNS and verifying its
integrity with DNSSEC. This has the nice property that no CA is needed,
and can give as good or stronger guarantees on cert integrity as
verifying against a CA can.

The downside is that it's not terribly widely deployed yet, and also
that it requires working DNSSEC support to work.

> True, but I really don't see the harm in saving those in cleartext.
> Like I said, I would use a .gpg file if I was worried about leaking
> that data. With the current approach I think you'll see two problems:

Tangentially related, one thing I would like to be able to have, is to
have multiple fingerprints stored for the same host,post tuple *at the
same time*. I run into this problem with servers that do round-robin to
different servers with different certs for the same hostname. I'd like
to be able to store all of them at once (by, for instance, connecting a
bunch of times and trusting the certificates one by one, and then know
that after that a mismatch should be considered suspicious).

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> Now I just need to know how to determine if a function is running from
>>> an async callback...
>> There's a C variable called running_asynch_code, but it's not exposed to
>> Lisp.  Unless there's something else in there that can be used, should
>> we just export that to Lisp, like the `noninteractive' variable?
>
> I guess we could export it (read-only) to Lisp, yes.

How does one mark variables as read-only?

> Tho, maybe it would be worth it to have a separate var for it,
> writeable, which we could arrange to consider (some?) process filters to
> be synchronous when run during a sync call to accept-process-output.

Oh, I see.  Yes, that would make sense, yes.

I also did some further testing with my patch, and apparently timers do
not set the running_asynch_code variable?  They probably should set
that separate variable, too.  I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Tangentially related, one thing I would like to be able to have, is to
> have multiple fingerprints stored for the same host,post tuple *at the
> same time*. I run into this problem with servers that do round-robin to
> different servers with different certs for the same hostname. I'd like
> to be able to store all of them at once (by, for instance, connecting a
> bunch of times and trusting the certificates one by one, and then know
> that after that a mismatch should be considered suspicious).

That should be easy to implement -- we can just allow the :fingerprint
slot to be a list and check against that.

But what would the user interface say?  Today it says

The fingerprint for the connection to %s:%s has changed from\n%s to\n%s
Connect anyway?  (no, session only, always)

So...  erm...  

Connect anyway?  (no, session only, always, add new fingerprint)

No, that's two "a"'s...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

nljlistbox2@gmail.com (N. Jackson) writes:

> At 11:40 -0400 on Tuesday 2014-11-18, Lars Magne Ingebrigtsen wrote:
>
>> Does anybody know of https sites with interesting types of
>> un-verifiable certificates so that I can test? Self-signed or expired
>> certificates would be nice...
>
> The Parabola GNU/Linux-libre site at https://www.parabola.nu/

Looks like a valid certificate?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Connect anyway?  (no, session only, always, add new fingerprint)
>
> No, that's two "a"'s...

One 'a' and one 'n'?

I just tried running the thing; it does ask for verification when
connecting to news.gwene.org, but I can't get it to ask to trust a
fingerprint when connecting to my mail server (which has a cert that
otherwise verifies)?

-Toke

Re: Network security manager

[David Engster]

Lars Magne Ingebrigtsen writes:
> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> https://revoked.grc.com/
>
> Thanks.

Expired:

https://testssl-expire.disig.sk/index.en.html

-David

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I parsed this bits a bit more, and ended up with
>
> Certificate issued by Cybertrust Public SureServer SV CA
> Issued to Akamai Technologies, Inc.
> Certificate host name: a248.e.akamai.net
> Public key: RSA, signature: RSA-SHA1, security level: Low
> Valid from: 2014-06-12, valid to: 2015-06-12

It would be nice if this part could be tabulated, I think; to make it
easier to separate out the data from the description. Something like:

Certificate info:
Issued by:         Cybertrust Public SureServer SV CA
Issued to:         Akamai Technologies, Inc.
Hostname:          a248.e.akamai.net
Public key:        RSA, signature: RSA-SHA1
Security level:    Low
Valid:             From 2014-06-12 to 2015-06-12



For the list of reasons, I think printing the 'certificate could not be
verified' part is redundant; that comes with a reason -- for gmane.org,
for instance, I get:

The TLS connection to news.gmane.org:nntp is insecure
for the following reasons:

certificate signer was not found
certificate could not be verified

-Toke

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 21:50:36 +0100 Toke Høiland-Jørgensen <toke@toke.dk> wrote: 

TH> Well, DANE allows for storing certificate info in DNS and verifying its
TH> integrity with DNSSEC. This has the nice property that no CA is needed,
TH> and can give as good or stronger guarantees on cert integrity as
TH> verifying against a CA can.

TH> The downside is that it's not terribly widely deployed yet, and also
TH> that it requires working DNSSEC support to work.

I think it makes sense as a feature for later, definitely.

Ted

Re: Network security manager

[David Engster]

David Engster writes:
> Expired:
>
> https://testssl-expire.disig.sk/index.en.html

Another test: Connect to

https://www.randomsample.de

Accept the self-signed certificate. Then go to

https://foo.www.randomsample.de

This should fail since certificate is not valid for that name.

-David

Re: Network security manager

[N. Jackson]

At 17:07 -0400 on Tuesday 2014-11-18, Lars Magne Ingebrigtsen wrote:

> nljlistbox2@gmail.com (N. Jackson) writes:
>
>> At 11:40 -0400 on Tuesday 2014-11-18, Lars Magne Ingebrigtsen wrote:
>>
>>> Does anybody know of https sites with interesting types of
>>> un-verifiable certificates so that I can test? Self-signed or expired
>>> certificates would be nice...
>>
>> The Parabola GNU/Linux-libre site at https://www.parabola.nu/
>
> Looks like a valid certificate?

Yes, it is. It's only interesting because Firefox, Chrome, and IE are
all reluctant to allow one to view that site because the certificate is
not from a well-known authority. Sounds like nsm correctly allows it. :)

Re: Network security manager

[David Engster]

N. Jackson writes:
> At 17:07 -0400 on Tuesday 2014-11-18, Lars Magne Ingebrigtsen wrote:
>
>> nljlistbox2@gmail.com (N. Jackson) writes:
>>
>>> At 11:40 -0400 on Tuesday 2014-11-18, Lars Magne Ingebrigtsen wrote:
>>>
>>>> Does anybody know of https sites with interesting types of
>>>> un-verifiable certificates so that I can test? Self-signed or expired
>>>> certificates would be nice...
>>>
>>> The Parabola GNU/Linux-libre site at https://www.parabola.nu/
>>
>> Looks like a valid certificate?
>
> Yes, it is. It's only interesting because Firefox, Chrome, and IE are
> all reluctant to allow one to view that site because the certificate is
> not from a well-known authority. Sounds like nsm correctly allows it. :)

CaCert's root CA is present in Debian stable, but was removed in
testing. Ubuntu already removed it.

-David

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The NSM will also warn about new certificates if the user has switched
> to `paranoid',

Yeah, couldn't get this to work. Also, there doesn't seem to be a
difference between 'medium' and 'high' security levels?

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> I just tried running the thing; it does ask for verification when
> connecting to news.gwene.org, but I can't get it to ask to trust a
> fingerprint when connecting to my mail server (which has a cert that
> otherwise verifies)?

If the certificate is valid, then nothing is queried.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> David Engster writes:
>> Expired:
>>
>> https://testssl-expire.disig.sk/index.en.html
>
> Another test: Connect to
>
> https://www.randomsample.de
>
> Accept the self-signed certificate. Then go to
>
> https://foo.www.randomsample.de
>
> This should fail since certificate is not valid for that name.

Thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> CaCert's root CA is present in Debian stable, but was removed in
> testing. Ubuntu already removed it.

Oh, I didn't know that.  What was the reason behind that decision?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> I just tried running the thing; it does ask for verification when
>> connecting to news.gwene.org, but I can't get it to ask to trust a
>> fingerprint when connecting to my mail server (which has a cert that
>> otherwise verifies)?
>
> If the certificate is valid, then nothing is queried.

Well I'd like to request that feature, please. This is the idea behind
TOFU: Only connect if the cert is in the database, whether it is
otherwise valid or not...

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> The NSM will also warn about new certificates if the user has switched
>> to `paranoid',
>
> Yeah, couldn't get this to work.

It should warn about the certificate changing, but it won't say anything
the first time it sees a certificate for the site.

> Also, there doesn't seem to be a difference between 'medium' and
> 'high' security levels?

No, I'm not sure what the difference should be.  Perhaps I should get
rid of the `paranoid' setting and just make that `high'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[David Engster]

Lars Magne Ingebrigtsen schreibt:
> David Engster <deng@randomsample.de> writes:
>
>> CaCert's root CA is present in Debian stable, but was removed in
>> testing. Ubuntu already removed it.
>
> Oh, I didn't know that.  What was the reason behind that decision?

I'm afraid I'm unable to summarize that discussion. If you have a few
hours to kill, see

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=718434

-David

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> Yeah, couldn't get this to work.
>
> It should warn about the certificate changing, but it won't say
> anything the first time it sees a certificate for the site.

Ah, right. Okay, that works for me; didn't catch that it stored the
cert. Though perhaps giving a notice when the certificate is first
stored would be nice?

> No, I'm not sure what the difference should be. Perhaps I should get
> rid of the `paranoid' setting and just make that `high'.

I'd get behind that. Also, perhaps document the meaning of the setting
somewhere? ;)

-Toke

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> On the other hand, we could store the server names in plain text when
> we store security exceptions to make reviews easier. That is, keep the
> hash-only thing for STARTTLS man-in-the-middle tracking and the like,
> but if the user registers an exception, then we'd stash the server
> name in there, too.

Would it make sense to have a hostname-based setting for credentials
storage? I.e. similar to how gnutls-verify-error is currently a hostname
match, I might want to set nsm-security-level per hostname. For
instance, I'd like to have 'paranoid' security for the services I
provide credentials to (most notably my mail server), but would probably
not mind keeping random TLS servers I may happen to download an image
from out of my certificate list file.

> This would avoid leaving a complete list of STARTTLS servers in that
> file, but still allow easy removal of specific exceptions.

OpenSSH has the 'HashKnownHosts' configuration parameter which
determines whether hostnames should be hashed in the trust store
(similar to what you are doing). I tend to turn it off to be able to
find things...

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>>> I just tried running the thing; it does ask for verification when
>>> connecting to news.gwene.org, but I can't get it to ask to trust a
>>> fingerprint when connecting to my mail server (which has a cert that
>>> otherwise verifies)?
>>
>> If the certificate is valid, then nothing is queried.
>
> Well I'd like to request that feature, please. This is the idea behind
> TOFU: Only connect if the cert is in the database, whether it is
> otherwise valid or not...

Then I misunderstood TOFU -- I thought it was about certificate
pinning.  The first time you connect, you don't have much to compare it
against, so it seemed superfluous to query the user about it.

And that's going to be a *lot* of querying if you're using Emacs to
browse the web.

But I can move the present pinning code down to `high', and then add
"query on first usage" on `paranoid'?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

>> No, I'm not sure what the difference should be. Perhaps I should get
>> rid of the `paranoid' setting and just make that `high'.
>
> I'd get behind that. Also, perhaps document the meaning of the setting
> somewhere? ;)

Sure.  Once we decide what it is.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> And that's going to be a *lot* of querying if you're using Emacs to
> browse the web.

Yeah, true. Hence also my suggestion from somewhere else in the thread
about making the security level match on hostnames?

> But I can move the present pinning code down to `high', and then add
> "query on first usage" on `paranoid'?

That would be a meaningful distinction, I think. And probably also make
'paranoid' live up to its name ;)

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> It would be nice if this part could be tabulated, I think; to make it
> easier to separate out the data from the description. Something like:
>
> Certificate info:
> Issued by:         Cybertrust Public SureServer SV CA
> Issued to:         Akamai Technologies, Inc.
> Hostname:          a248.e.akamai.net
> Public key:        RSA, signature: RSA-SHA1
> Security level:    Low
> Valid:             From 2014-06-12 to 2015-06-12

Yeah, that looks better.  I've now pushed a change that makes it look
like this.

> For the list of reasons, I think printing the 'certificate could not be
> verified' part is redundant; that comes with a reason -- for gmane.org,
> for instance, I get:
>
> The TLS connection to news.gmane.org:nntp is insecure
> for the following reasons:
>
> certificate signer was not found
> certificate could not be verified

Yeah, I left it in there for now because I wanted to see whether it
really (in practice) always showed up along with one of the other ones
in all cases.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Toke Høiland-Jørgensen]

>Yeah, that looks better.  I've now pushed a change that makes it look
>like this.

Awesome!

>Yeah, I left it in there for now because I wanted to see whether it
>really (in practice) always showed up along with one of the other ones
>in all cases.

Ah, right. Cool.

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Well, according to the documentation:
> http://www.gnutls.org/manual/html_node/Verifying-X_002e509-certificate-paths.html
>
> GNUTLS_CERT_SIGNER_NOT_CA means:
>
>     "The certificate’s signer was not a CA. This may happen if this was
>     a version 1 certificate, which is common with some CAs, or a version
>     3 certificate without the basic constrains extension."
>
> Whereas GNUTLS_CERT_SIGNER_NOT_FOUND is the common "we don't trust
> whoever signed this". So I'd think that GNUTLS_CERT_SIGNER_NOT_FOUND
> would be returned for all self-signed certificates, and possibly
> GNUTLS_CERT_SIGNER_NOT_CA in addition. If GNUTLS_CERT_SIGNER_NOT_CA is
> returned for a legitimately signed certificate (from the trust store),
> the CA is probably doing something wrong.

Right.  I've now tweaked the values returned so that we get the
:self-signed error for SIGNER_NOT_FOUND, which should make more sense to
the user.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> And that's going to be a *lot* of querying if you're using Emacs to
>> browse the web.
>
> Yeah, true. Hence also my suggestion from somewhere else in the thread
> about making the security level match on hostnames?

Things that require extensive customisations almost never get used, so
I'm not sure it's worth it.

>> But I can move the present pinning code down to `high', and then add
>> "query on first usage" on `paranoid'?
>
> That would be a meaningful distinction, I think. And probably also make
> 'paranoid' live up to its name ;)

Pushed now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That variable is an internal implementation detail designed for a
> specific purpose that has nothing to do with running asynchronously.
> Exposing it to Lisp is IMO extremely unclean.  Especially since this
> is not really needed anyway, because Lars's use case has a much
> simpler solution that doesn't require any such kludges.

Yeah, it turned out to not work in my use case either.  I had forgotten
that eww sometimes renders stuff asynchronously anyway (when url.el gets
a redirection, the entire thing is then run off of a process filter
instead in the main thread, if I follow the logic here (and I may well
not)).

So the user interface problem remains.  We don't want to suddenly start
asking people stuff while they're doing other stuff, but here we kinda
need to ask them stuff...

In any case, exporting that variable is not the answer.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 00:02:58 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> So the user interface problem remains.  We don't want to suddenly start
LMI> asking people stuff while they're doing other stuff, but here we kinda
LMI> need to ask them stuff...

https://en.wikipedia.org/wiki/Infobar is the UI element most web
browsers use today.  We don't have it in Emacs AFAIK.

http://www.w3.org/TR/wsc-ui/#indicators and
http://www.w3.org/TR/wsc-ui/#Robustness are the W3C recommendations on
this topic.  To summarize (but please please please read the document,
it's quite good):

* show identity signal in a consistent visual position where web content
  can't obscure it

* make the identity signal human-readable

* let the user access the extra information like site owner and source
  of trust; optionally expose even more

* have a distinct TLS indicator

* warning, caution, and danger messages should interrupt the user's
  task (if in the foreground)

As I said, there is much more in the document. Of course, Emacs is not
just a web browser, so we must adapt rather than blindly adopt these
guidelines, but I hope we don't ignore them. Should I make a list of
concrete recommendations for EWW and Emacs in general based on that
document?

Ted

Re: Network security manager

[Stephen J. Turnbull]

Lars Magne Ingebrigtsen writes:
 > David Engster <deng@randomsample.de> writes:
 > 
 > > CaCert's root CA is present in Debian stable, but was removed in
 > > testing. Ubuntu already removed it.
 > 
 > Oh, I didn't know that.  What was the reason behind that decision?

https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1258286
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=718434

Re: Network security manager

[Stefan Monnier]

> I also did some further testing with my patch, and apparently timers do
> not set the running_asynch_code variable?  They probably should set
> that separate variable, too.  I think.

BTW, another existing variable that you could use is `inhibit-quit'.
Usually, async code is run with inhibit-quit bound to a non-nil value,
because we don't want the user's C-g to interrupt the async code (which
the user didn't even know was running) when she intended to just get out
of the minibuffer.


        Stefan

Re: Network security manager

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Tue, 18 Nov 2014 21:09:04 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> > I also did some further testing with my patch, and apparently timers do
> > not set the running_asynch_code variable?  They probably should set
> > that separate variable, too.  I think.
> 
> BTW, another existing variable that you could use is `inhibit-quit'.

That's dangerous, IMO.  Again, it uses a variable not designed for
this functionality, IOW this could break without notice.

Re: Network security manager

[Ted Zlatanov]

On Tue, 18 Nov 2014 16:29:30 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Also, would you like to integrate your TOFU patch with the new nsm branch?

LMI> The NSM does TOFU.  No patch necessary.

What do you think about the verification and TOFU implementation in
gnutls-cli? Please see
https://gitorious.org/gnutls/gnutls/raw/master:src/cli.c inside
cert_verify_callback() for the details.

* uses SSH-style gnutls_store_pubkey() and gnutls_verify_stored_pubkey()
  to DTRT and pins the public key rather than the certificate
  fingerprint. The pub keys are stored by default in a way that lets the
  user look them up by hostname, but we can customize that. And it's
  mostly handled by GnuTLS internals as far as pubkey extraction and
  verification.

* does DANE auth (although I don't know the details on DANE, the
  client implementation looks reasonable and Toke suggested it)

* checks OCSP for revocations using cert_verify_ocsp() in the same cli.c

Ted

Re: Network security manager

[Toke Høiland-Jørgensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> * uses SSH-style gnutls_store_pubkey() and gnutls_verify_stored_pubkey()
>   to DTRT and pins the public key rather than the certificate
>   fingerprint. The pub keys are stored by default in a way that lets the
>   user look them up by hostname, but we can customize that. And it's
>   mostly handled by GnuTLS internals as far as pubkey extraction and
>   verification.

AFAICT this is functionally equivalent to what is currently in NSM;
except it stores the public key rather than the fingerprint. I am not
sure if there area any security implications to storing just the
fingerprint...

> * does DANE auth (although I don't know the details on DANE, the
>   client implementation looks reasonable and Toke suggested it)

I think the right thing to do would probably be to check DANE and use
that as an additional input to the NSM dialog. I'd suggest the
following:

- Supply the DANE status as part of the 'certificate information' blurb
  when popping up a prompt. For many (most?) setups this will be
  'unknown' either because no DANE info is published in DNS or DNSSEC
  validation fails (or both).

- If valid DANE info is available *and* this doesn't match the shown
  certificate, treat it as a reason to consider the certificate
  insecure.

I.e. treat a positive DANE verification as information to present to the
user, and a verified failure as a cause for alarm. This corresponds to
the current DANE RFC recommendations AFAICT...

> * checks OCSP for revocations using cert_verify_ocsp() in the same
> cli.c

This would probably be a good idea to implement in any case.

-Toke

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Things that require extensive customisations almost never get used, so
> I'm not sure it's worth it.

Well it would default to something sensible, of course. I'd use it ;)

> Pushed now.

Okay, so the initial prompt on paranoid level works. Would be nice if
the initial prompt popped up the same certificate information as the
other confirmation prompts, to make it easier to verify that it's the
right certificate. That goes for when the fingerprint changes as well, I
suppose...

Once the fingerprint is stored, though, it fails in weird ways. I tried
manually modifying the fingerprint in the network-security.data file (to
make verification fail). This elicits this behaviour:

- On security levels high and paranoid, verification just fails silently
  (open-network-stream returns nil), with no option to update the stored
  fingerprint.

- On security levels low and medium, verification *succeeds*, even
  though a fingerprint is stored that does not match the certificate.

I would consider especially the second point to be a big no-no; even if
the security level is subsequently lowered, having a stored fingerprint
should take precedence and fail the verification. Maybe the "continue
anyway" could cause the stored fingerprint to be removed, but just
continuing regardless is bad IMO.


Finally, GnuTLS has the ability to generate ASCII art of the certificate
public key, like this:

	Public key's random art:
		+--[ RSA 4096]----+
		|           ..o  .|
		|            ooo.o|
		|            .o..o|
		|       .    o + .|
		|      . S    = E |
		|     o . o  .    |
		|      = o .  o   |
		|       B .. .... |
		|     .+ oo..o++  |
		+-----------------+

Supposedly, this should make it possible to verify a certificate at a
glance (relying on human visual memory being superior to our ability to
recognise long strings of alphanumericals). Might be worthwhile to
include this in (some of) the popups? Can't really figure out if I think
it's just a gimmick, or what, but I thought I'd suggest it. Gnutls-cli
uses it... The function is gnutls_random_art().

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So the user interface problem remains.  We don't want to suddenly start
> asking people stuff while they're doing other stuff, but here we kinda
> need to ask them stuff...
>
> In any case, exporting that variable is not the answer.

Sleeping on this, the only feasible solution I currently see here is
that each thing that wants to do a HTTP request has to say that the
request should be "non-interactive".

Since url.el is asynchronous, just binding a special variable to be
picked up later isn't feasible -- url.el will (on redirects) run the
redirected request in a different context.

So this will require some slight surgery of the url.el code...  which is
not my favourite thing in the world to do.  *sigh*

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 19 Nov 2014 00:02:58 +0100 Lars Magne Ingebrigtsen
> <larsi@gnus.org> wrote:
>
> LMI> So the user interface problem remains.  We don't want to suddenly start
> LMI> asking people stuff while they're doing other stuff, but here we kinda
> LMI> need to ask them stuff...
>
> https://en.wikipedia.org/wiki/Infobar is the UI element most web
> browsers use today.  We don't have it in Emacs AFAIK.
>
> http://www.w3.org/TR/wsc-ui/#indicators and
> http://www.w3.org/TR/wsc-ui/#Robustness are the W3C recommendations on
> this topic.  To summarize (but please please please read the document,
> it's quite good):
>
> * show identity signal in a consistent visual position where web content
>   can't obscure it

Sure, eww should display TLS markers and stuff, but that's kinda
orthogonal to the issue I was discussing, which is how (and whether) to
query the user when running in an asynchronous context.   >"?

> As I said, there is much more in the document. Of course, Emacs is not
> just a web browser, so we must adapt rather than blindly adopt these
> guidelines, but I hope we don't ignore them. Should I make a list of
> concrete recommendations for EWW and Emacs in general based on that
> document?

Yeah, that would be nice.  File it as an enhancement bug report so that
we don't forget.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> AFAICT this is functionally equivalent to what is currently in NSM;
> except it stores the public key rather than the fingerprint. I am not
> sure if there area any security implications to storing just the
> fingerprint...

You'd hope not.  If there is, that's not a good fingerprint.  >"?

>> * does DANE auth (although I don't know the details on DANE, the
>>   client implementation looks reasonable and Toke suggested it)
>
> I think the right thing to do would probably be to check DANE and use
> that as an additional input to the NSM dialog. I'd suggest the
> following:
>
> - Supply the DANE status as part of the 'certificate information' blurb
>   when popping up a prompt. For many (most?) setups this will be
>   'unknown' either because no DANE info is published in DNS or DNSSEC
>   validation fails (or both).
>
> - If valid DANE info is available *and* this doesn't match the shown
>   certificate, treat it as a reason to consider the certificate
>   insecure.
>
> I.e. treat a positive DANE verification as information to present to the
> user, and a verified failure as a cause for alarm. This corresponds to
> the current DANE RFC recommendations AFAICT...
>
>> * checks OCSP for revocations using cert_verify_ocsp() in the same
>> cli.c

DANE and especially revocation checking is kinda slow though?  Which is
why Chrome doesn't do it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> What do you think about the verification and TOFU implementation in
> gnutls-cli? Please see
> https://gitorious.org/gnutls/gnutls/raw/master:src/cli.c inside
> cert_verify_callback() for the details.
>
> * uses SSH-style gnutls_store_pubkey() and gnutls_verify_stored_pubkey()
>   to DTRT and pins the public key rather than the certificate
>   fingerprint. The pub keys are stored by default in a way that lets the
>   user look them up by hostname, but we can customize that. And it's
>   mostly handled by GnuTLS internals as far as pubkey extraction and
>   verification.
>
> * does DANE auth (although I don't know the details on DANE, the
>   client implementation looks reasonable and Toke suggested it)
>
> * checks OCSP for revocations using cert_verify_ocsp() in the same cli.c

So gnutls proper doesn't do this?  We'd have to implement it ourselves
if we want it...  (I mean, copy chunks of their code.  >"?)

Can we do DANE and OCSP from Emacs Lisp?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Once the fingerprint is stored, though, it fails in weird ways. I tried
> manually modifying the fingerprint in the network-security.data file (to
> make verification fail). This elicits this behaviour:
>
> - On security levels high and paranoid, verification just fails silently
>   (open-network-stream returns nil), with no option to update the stored
>   fingerprint.
>
> - On security levels low and medium, verification *succeeds*, even
>   though a fingerprint is stored that does not match the certificate.

Sounds like a bug.  >"?  I'll have a look at it tonight.

> Finally, GnuTLS has the ability to generate ASCII art of the certificate
> public key, like this:
>
> 	Public key's random art:
> 		+--[ RSA 4096]----+
> 		|           ..o  .|
> 		|            ooo.o|
> 		|            .o..o|
> 		|       .    o + .|
> 		|      . S    = E |
> 		|     o . o  .    |
> 		|      = o .  o   |
> 		|       B .. .... |
> 		|     .+ oo..o++  |
> 		+-----------------+

Unfortunately, this seems to have been introduced in a later version of
the library than what I have on my development machine, so I haven't
been able to test.

> Supposedly, this should make it possible to verify a certificate at a
> glance (relying on human visual memory being superior to our ability to
> recognise long strings of alphanumericals). Might be worthwhile to
> include this in (some of) the popups? Can't really figure out if I think
> it's just a gimmick, or what, but I thought I'd suggest it. Gnutls-cli
> uses it... The function is gnutls_random_art().

Yeah, I don't know either whether it's useful.  Does anybody else have
an opinion?  Anybody ever found the "random art" handy?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 06:43:39 +0100 Toke Høiland-Jørgensen <toke@toke.dk> wrote: 

TH> Ted Zlatanov <tzz@lifelogs.com> writes:
>> * uses SSH-style gnutls_store_pubkey() and gnutls_verify_stored_pubkey()
>> to DTRT and pins the public key rather than the certificate
>> fingerprint. The pub keys are stored by default in a way that lets the
>> user look them up by hostname, but we can customize that. And it's
>> mostly handled by GnuTLS internals as far as pubkey extraction and
>> verification.

TH> AFAICT this is functionally equivalent to what is currently in NSM;
TH> except it stores the public key rather than the fingerprint. I am not
TH> sure if there area any security implications to storing just the
TH> fingerprint...

See
https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning#What_Should_Be_Pinned.3F

Briefly, they say pinning the public key is better in most cases but is
not subject to expiration like a certificate.

Furthermore, see https://www.imperialviolet.org/2011/05/04/pinning.html
section "Why public key hashes, not certificate hashes?" on Google's
implementation and a stronger opinion.

Excerpt:

"In general, hashing certificates is the obvious solution, but the wrong
one. The problem is that CA certificates are often reissued: there are
multiple certificates with the same public key, subject name etc but
different extensions or expiry dates. Browsers build certificates chains
from a pool of certificates, bottom up, and an alternative version of a
certificate might be substituted for the one that you expect.

...

Conversely, public key hashes must be correct:

Browsers assume that the leaf certificate is fixed: it's always the
starting point of the chain. The leaf certificate contains a signature
which must be a valid signature, from its parent, for that certificate.
That implies that the public key of the parent is fixed by the leaf
certificate. So, inductively, the chain of public keys is fixed, modulo
truncation.

The only sharp edge is that you mustn't pin to a cross-certifying root.
For example, GoDaddy's root is signed by Valicert so that older clients,
which don't recognise GoDaddy as a root, still trust those certificates.
However, you wouldn't want to pin to Valicert because newer clients will
stop their chain at GoDaddy.

Also, we're hashing the SubjectPublicKeyInfo not the public key bit
string. The SPKI includes the type of the public key and some parameters
along with the public key itself. This is important because just hashing
the public key leaves one open to misinterpretation attacks. Consider a
Diffie-Hellman public key: if one only hashes the public key, not the
full SPKI, then an attacker can use the same public key but make the
client interpret it in a different group. Likewise one could force an
RSA key to be interpreted as a DSA key etc."

I am not a cryptographer so I hope some of those step in and suggest
what's best. To me from what I know and based on the cited references,
it seems it could be a choice but pinning the public key is better for
most people. They won't have to accept again every time the certificate
is reissued.

>> * does DANE auth (although I don't know the details on DANE, the
>> client implementation looks reasonable and Toke suggested it)

TH> I think the right thing to do would probably be to check DANE and use
TH> that as an additional input to the NSM dialog. I'd suggest the
TH> following:

TH> - Supply the DANE status as part of the 'certificate information' blurb
TH>   when popping up a prompt. For many (most?) setups this will be
TH>   'unknown' either because no DANE info is published in DNS or DNSSEC
TH>   validation fails (or both).

TH> - If valid DANE info is available *and* this doesn't match the shown
TH>   certificate, treat it as a reason to consider the certificate
TH>   insecure.

TH> I.e. treat a positive DANE verification as information to present to the
TH> user, and a verified failure as a cause for alarm. This corresponds to
TH> the current DANE RFC recommendations AFAICT...

Works for me. If we implement it in ELisp as Lars suggested it might
even be easy. Could you please open the feature request in the bug
tracker with your plan of action so we can keep it in mind?

>> * checks OCSP for revocations using cert_verify_ocsp() in the same
>> cli.c

TH> This would probably be a good idea to implement in any case.

I think Lars agrees, also with the "in ELisp if possible" caveat.  Can
you create a separate feature in the bug tracker?

On Wed, 19 Nov 2014 09:44:49 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> DANE and especially revocation checking is kinda slow though?  Which is
LMI> why Chrome doesn't do it.

It's definitely in the high-to-paranoid level, but if the level can be
enabled per site or per subnet, it would be ideal.

On Wed, 19 Nov 2014 09:55:00 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Unfortunately, this seems to have been introduced in a later version of
LMI> the library than what I have on my development machine, so I haven't
LMI> been able to test.

We can make such GnuTLS features optional or explicitly require the
latest if the feature is very appealing.  This one isn't to me :)  I
hate the "random art" and always disable it with SSH.

Ted

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 09:37:08 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> http://www.w3.org/TR/wsc-ui/#indicators and
>> http://www.w3.org/TR/wsc-ui/#Robustness are the W3C recommendations on
>> this topic.  To summarize (but please please please read the document,
>> it's quite good):
>> 
>> * show identity signal in a consistent visual position where web content
>> can't obscure it

LMI> Sure, eww should display TLS markers and stuff, but that's kinda
LMI> orthogonal to the issue I was discussing, which is how (and whether) to
LMI> query the user when running in an asynchronous context.   >"?

I mean that EWW's visual indicators of identity and trust should be
global. Then you don't interrupt the user (they get cranky!) but show
them a visual indicator that something requires their attention. I can't
think of a better place that works in graphical and text modes and has
the precedent of embedded infobar-style buttons than the modeline.

Furthermore, I think it would make sense to use the same indicators for
GnuTLS connections in general (whatever NSM handles), not just EWW. I
couldn't find UI recommendations for mail clients, but from experience
with a few they treat encryption problems as a high-priority dialog and
interrupt the user experience until you say "OK, trust XYZ." Which is,
again, not ideal for Emacs so we should find a nicer way to indicate
problems without interrupting.

I also mean that those indicators should not be solely implied by "the
image looks broken" because that's displaying the indicator in the
content where it can be obscured and the location varies.

>> As I said, there is much more in the document. Of course, Emacs is not
>> just a web browser, so we must adapt rather than blindly adopt these
>> guidelines, but I hope we don't ignore them. Should I make a list of
>> concrete recommendations for EWW and Emacs in general based on that
>> document?

LMI> Yeah, that would be nice.  File it as an enhancement bug report so that
LMI> we don't forget.

All right.

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I am not a cryptographer so I hope some of those step in and suggest
> what's best. To me from what I know and based on the cited references,
> it seems it could be a choice but pinning the public key is better for
> most people. They won't have to accept again every time the certificate
> is reissued.

Hm...  might one not want to track the certificate, though?  If it's
changed, then there might be shenanigans.

But if the attacker can generate traffic with the trusted public key,
the site would have larger problems than with the certificate, so
perhaps it doesn't add anything much security-wise...

> Also, we're hashing the SubjectPublicKeyInfo not the public key bit
> string. The SPKI includes the type of the public key and some parameters
> along with the public key itself.

Does gnutls have a function to fingerprint that info?  Or access it in
raw form?  I guess we could just sha1 it ourselves.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I mean that EWW's visual indicators of identity and trust should be
> global. Then you don't interrupt the user (they get cranky!) but show
> them a visual indicator that something requires their attention. I can't
> think of a better place that works in graphical and text modes and has
> the precedent of embedded infobar-style buttons than the modeline.

Oh, I see.  I misunderstood you completely; sorry.  >"?

That does sound attractive...  eww would put an icon into the mode line,
and then the user could click it to answer the query.  A kind of "an
action is needed" thing.

> Furthermore, I think it would make sense to use the same indicators for
> GnuTLS connections in general (whatever NSM handles), not just EWW. I
> couldn't find UI recommendations for mail clients, but from experience
> with a few they treat encryption problems as a high-priority dialog and
> interrupt the user experience until you say "OK, trust XYZ." Which is,
> again, not ideal for Emacs so we should find a nicer way to indicate
> problems without interrupting.

The other connections things are usually done interactively, so just
asking the user directly is more user-friendly.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 12:19:46 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> cited:

>> "Also, we're hashing the SubjectPublicKeyInfo not the public key bit
>> string. The SPKI includes the type of the public key and some parameters
>> along with the public key itself."

(I didn't write the above originally, just cited it)

LMI> Does gnutls have a function to fingerprint that info?  Or access it in
LMI> raw form?  I guess we could just sha1 it ourselves.

http://gnutls.org/manual/gnutls.html#X_002e509-public-and-private-keys

You want gnutls_x509_crt_get_key_id() I think.  The key itself can be
exported with gnutls_pubkey_export2() into PEM or DER formats, but
that's not very useful in the NSM context

Ted

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 12:23:19 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I mean that EWW's visual indicators of identity and trust should be
>> global. Then you don't interrupt the user (they get cranky!) but show
>> them a visual indicator that something requires their attention. I can't
>> think of a better place that works in graphical and text modes and has
>> the precedent of embedded infobar-style buttons than the modeline.

LMI> Oh, I see.  I misunderstood you completely; sorry.  >"?

LMI> That does sound attractive...  eww would put an icon into the mode line,
LMI> and then the user could click it to answer the query.  A kind of "an
LMI> action is needed" thing.

Yes!  I don't know how it could work well in text mode, though.

>> Furthermore, I think it would make sense to use the same indicators for
>> GnuTLS connections in general (whatever NSM handles), not just EWW. I
>> couldn't find UI recommendations for mail clients, but from experience
>> with a few they treat encryption problems as a high-priority dialog and
>> interrupt the user experience until you say "OK, trust XYZ." Which is,
>> again, not ideal for Emacs so we should find a nicer way to indicate
>> problems without interrupting.

LMI> The other connections things are usually done interactively, so just
LMI> asking the user directly is more user-friendly.

I'm kind of 50-50 on this; I see your point but also think it looks
weird as a user experience, especially that you can keep typing and
accidentally answer "always".  Long-term we know we'll have all the
connections on background threads eventualy, so it's probably good to
handle that case consistently across all usage modes.

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> Ted Zlatanov <tzz@lifelogs.com> cited:
>
>>> "Also, we're hashing the SubjectPublicKeyInfo not the public key bit
>>> string. The SPKI includes the type of the public key and some parameters
>>> along with the public key itself."
>
> (I didn't write the above originally, just cited it)
>
> LMI> Does gnutls have a function to fingerprint that info?  Or access it in
> LMI> raw form?  I guess we could just sha1 it ourselves.
>
> http://gnutls.org/manual/gnutls.html#X_002e509-public-and-private-keys
>
> You want gnutls_x509_crt_get_key_id() I think.  The key itself can be
> exported with gnutls_pubkey_export2() into PEM or DER formats, but
> that's not very useful in the NSM context

Is that the SPKI or just a hash of the public key bit string?  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Garreau, Alexandre]

[-- Attachment #1: Type: text/plain, Size: 2025 bytes --]

On 2014-11-19 at 09:55, Lars Magne Ingebrigtsen wrote:
> Toke Høiland-Jørgensen <toke@toke.dk> writes:
>> Finally, GnuTLS has the ability to generate ASCII art of the certificate
>> public key, like this:
>>
>> 	Public key's random art:
>> 		+--[ RSA 4096]----+
>> 		|           ..o  .|
>> 		|            ooo.o|
>> 		|            .o..o|
>> 		|       .    o + .|
>> 		|      . S    = E |
>> 		|     o . o  .    |
>> 		|      = o .  o   |
>> 		|       B .. .... |
>> 		|     .+ oo..o++  |
>> 		+-----------------+
>
> Unfortunately, this seems to have been introduced in a later version of
> the library than what I have on my development machine, so I haven't
> been able to test.
>
>> Supposedly, this should make it possible to verify a certificate at a
>> glance (relying on human visual memory being superior to our ability to
>> recognise long strings of alphanumericals). Might be worthwhile to
>> include this in (some of) the popups? Can't really figure out if I think
>> it's just a gimmick, or what, but I thought I'd suggest it. Gnutls-cli
>> uses it... The function is gnutls_random_art().
>
> Yeah, I don't know either whether it's useful.  Does anybody else have
> an opinion?  Anybody ever found the "random art" handy?

Hexadecimal fingerprint are hard to check. At least if someone want to
take less time she just check some last or first characters, and that
can be easily faked, and is not secure. There’s no way with an
hexadecimal string to do a “global approximative check”, what could
really accord security with a fingerpint.

ASCII art, and images in general, are really easily checkable, it takes
only 2s, and done. It also exists in graphic, it’s named vizhash: it
just compute simple colored (shaped or not) forms (triangles, circles…)
according output of long hash functions. It’s *really* efficient to
check things.

I’d love to see Emacs being the pioneer of introducing this nice feature
to the end user, GNUnet should be the next.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 12:50:00 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
LMI> Does gnutls have a function to fingerprint that info?  Or access it in
LMI> raw form?  I guess we could just sha1 it ourselves.
>> 
>> http://gnutls.org/manual/gnutls.html#X_002e509-public-and-private-keys
>> 
>> You want gnutls_x509_crt_get_key_id() I think.  The key itself can be
>> exported with gnutls_pubkey_export2() into PEM or DER formats, but
>> that's not very useful in the NSM context

LMI> Is that the SPKI or just a hash of the public key bit string?  >"?

Read the docs :)

(It's the whole thing, not just the key itself.)

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

"Garreau, Alexandre" <galex-713@galex-713.eu> writes:

> ASCII art, and images in general, are really easily checkable, it takes
> only 2s, and done. It also exists in graphic, it’s named vizhash: it
> just compute simple colored (shaped or not) forms (triangles, circles…)
> according output of long hash functions. It’s *really* efficient to
> check things.

Is vizhash a C library?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Garreau, Alexandre]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

Le 19/11/2014 à 13h17, Lars Magne Ingebrigtsen a écrit :
> "Garreau, Alexandre" <galex-713@galex-713.eu> writes:
>
>> ASCII art, and images in general, are really easily checkable, it takes
>> only 2s, and done. It also exists in graphic, it’s named vizhash: it
>> just compute simple colored (shaped or not) forms (triangles, circles…)
>> according output of long hash functions. It’s *really* efficient to
>> check things.
>
> Is vizhash a C library?

Unfortunately no, but there are several implementations, mainly in
javascript, PHP, and, err, Java, as far as I know. But it’s quite simple
and should be easily possible in any language where you can draw simple
figures.

The three I know:
https://github.com/sebsauvage/VizHash/
https://github.com/sametmax/VizHash.js
https://github.com/inouire/VizHash4j

I’d be really glad if someone found a way to do it with elisp… maybe
using an external program? imagemagick?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

Re: Network security manager

[Lars Magne Ingebrigtsen]

"Garreau, Alexandre" <galex-713@galex-713.eu> writes:

> Unfortunately no, but there are several implementations, mainly in
> javascript, PHP, and, err, Java, as far as I know. But it’s quite simple
> and should be easily possible in any language where you can draw simple
> figures.
>
> The three I know:
> https://github.com/sebsauvage/VizHash/
> https://github.com/sametmax/VizHash.js
> https://github.com/inouire/VizHash4j
>
> I’d be really glad if someone found a way to do it with elisp… maybe
> using an external program? imagemagick?

If somebody creates an SVG structure from the hash, we can display that
as an image in Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Stefan Monnier]

>> > I also did some further testing with my patch, and apparently timers do
>> > not set the running_asynch_code variable?  They probably should set
>> > that separate variable, too.  I think.
>> BTW, another existing variable that you could use is `inhibit-quit'.
> That's dangerous, IMO.  Again, it uses a variable not designed for
> this functionality, IOW this could break without notice.

It's definitely not dangerous.  IIUC he needs this info to decide
whether his code can prompt the user or not.  Maybe it won't do the
right thing in 100% of the cases, but it's clear that if inhibit-quit is
non-nil, we're in a context where we shouldn't prompt the user.


        Stefan

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 08:40:54 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> > I also did some further testing with my patch, and apparently timers do
>>> > not set the running_asynch_code variable?  They probably should set
>>> > that separate variable, too.  I think.
>>> BTW, another existing variable that you could use is `inhibit-quit'.
>> That's dangerous, IMO.  Again, it uses a variable not designed for
>> this functionality, IOW this could break without notice.

SM> It's definitely not dangerous.  IIUC he needs this info to decide
SM> whether his code can prompt the user or not.  Maybe it won't do the
SM> right thing in 100% of the cases, but it's clear that if inhibit-quit is
SM> non-nil, we're in a context where we shouldn't prompt the user.

Would it work to use the logic of "the buffer that initiated the
connection is in the foreground"? In that case, we could store the
buffer name as an optional record in the process info structure--
`open-network-stream' could figure it out mostly automatically?

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> Is that the SPKI or just a hash of the public key bit string?  >"?
>
> Read the docs :)

Delegate delegate delegate!  >"?

> (It's the whole thing, not just the key itself.)

I've now switched NSM to using the public key hash.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> Once the fingerprint is stored, though, it fails in weird ways. I tried
> manually modifying the fingerprint in the network-security.data file (to
> make verification fail). This elicits this behaviour:
>
> - On security levels high and paranoid, verification just fails silently
>   (open-network-stream returns nil), with no option to update the stored
>   fingerprint.

I edited a fingerprint, set the level to `high', and then reconnected.
It notified me that it had changed, and then returned the process.  So I
seem to be unable to reproduce this.

This is my test case:

(setq process
      (open-network-stream
       "nntpd" (get-buffer-create "*nntp*") "google.com" "https"
       :type 'tls))

> - On security levels low and medium, verification *succeeds*, even
>   though a fingerprint is stored that does not match the certificate.
>
> I would consider especially the second point to be a big no-no; even if
> the security level is subsequently lowered, having a stored fingerprint
> should take precedence and fail the verification. Maybe the "continue
> anyway" could cause the stored fingerprint to be removed, but just
> continuing regardless is bad IMO.

No I think that's the correct behaviour.  If you want `medium' security,
you only care about whether the certificate is valid or not.  And the
google.com certificate is valid, even though it changed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Would it work to use the logic of "the buffer that initiated the
> connection is in the foreground"? In that case, we could store the
> buffer name as an optional record in the process info structure--
> `open-network-stream' could figure it out mostly automatically?

Hm...  so shr would tell `url-retrieve' that the buffer that the "user
buffer" for the request is "*eww*", and then if that's the buffer that's
active when url.el finally has decided which server to connect to, and
the NSM decides to query the user -- then NSM would only query the user
if the user's active buffer is the same buffer?

I think that sounds practically doable, and I think it would solve the
main problem, unless there are scenarios I'm not considering...  hm...

Yes!  I though of one.  >"?

If you `M-x eww RET http://google.com RET', then we don't create the
*eww* buffer until we have downloaded the HTML.  (Which will actually be
from https://www.google.com, since there's a redirect.)  Meanwhile the
user may well have left the buffer she typed `M-x eww' in, but that
(probably) shouldn't stop NSM from querying about whether the user
really wants to visit the version of https://www.google.com that seems
to be signed by an invalid Chinese CA for some strange reason or
other...

I think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

The "prompting for TLS credentials for images in eww" problem has now
been fixed.  That is, invalidly encrypted images simply won't be shown
if you have the NSM switched on.

I did this by introducing yet another variable in URL.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 15:45:52 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Would it work to use the logic of "the buffer that initiated the
>> connection is in the foreground"? In that case, we could store the
>> buffer name as an optional record in the process info structure--
>> `open-network-stream' could figure it out mostly automatically?

LMI> Hm...  so shr would tell `url-retrieve' that the buffer that the "user
LMI> buffer" for the request is "*eww*", and then if that's the buffer that's
LMI> active when url.el finally has decided which server to connect to, and
LMI> the NSM decides to query the user -- then NSM would only query the user
LMI> if the user's active buffer is the same buffer?

Yes!

LMI> If you `M-x eww RET http://google.com RET', then we don't create the
LMI> *eww* buffer until we have downloaded the HTML.  (Which will actually be
LMI> from https://www.google.com, since there's a redirect.)  Meanwhile the
LMI> user may well have left the buffer she typed `M-x eww' in, but that
LMI> (probably) shouldn't stop NSM from querying about whether the user
LMI> really wants to visit the version of https://www.google.com that seems
LMI> to be signed by an invalid Chinese CA for some strange reason or
LMI> other...

You could create the *eww* buffer immediately?  Or just look for that
buffer name (if you store just the name in the process)?

At medium or lower `nsm-security-level', I wouldn't expect to be queried
in the case you describe. But at high or above, I would.

Ted

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> You could create the *eww* buffer immediately?

We could, but it would look kinda odd.  Or perhaps not?  Pop up the
empty buffer saying "Loading..."  It's possible.

> Or just look for that buffer name (if you store just the name in the
> process)?

When the connection is made (and the certificate is examined), the *eww*
buffer is still not created.

> At medium or lower `nsm-security-level', I wouldn't expect to be queried
> in the case you describe. But at high or above, I would.

If the certificate is invalid when we're visiting www.google.com, I
think the default (i.e., `medium') setting should be to query the user.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Ted Zlatanov]

On Wed, 19 Nov 2014 16:47:43 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> You could create the *eww* buffer immediately?

LMI> We could, but it would look kinda odd.  Or perhaps not?  Pop up the
LMI> empty buffer saying "Loading..."  It's possible.

It's an opportunity for a spinning 3-D EWW logo, so I wouldn't turn it down.

>> At medium or lower `nsm-security-level', I wouldn't expect to be queried
>> in the case you describe. But at high or above, I would.

LMI> If the certificate is invalid when we're visiting www.google.com, I
LMI> think the default (i.e., `medium') setting should be to query the user.

Right, sorry.  By "queried" I meant "queried even if you're in a different buffer."

Ted

Re: Network security manager

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Wed, 19 Nov 2014 08:40:54 -0500
> 
> >> > I also did some further testing with my patch, and apparently timers do
> >> > not set the running_asynch_code variable?  They probably should set
> >> > that separate variable, too.  I think.
> >> BTW, another existing variable that you could use is `inhibit-quit'.
> > That's dangerous, IMO.  Again, it uses a variable not designed for
> > this functionality, IOW this could break without notice.
> 
> It's definitely not dangerous.  IIUC he needs this info to decide
> whether his code can prompt the user or not.  Maybe it won't do the
> right thing in 100% of the cases, but it's clear that if inhibit-quit is
> non-nil, we're in a context where we shouldn't prompt the user.

It's not clear to me at all.  Some code could set the variable for
reasons that have no relation to prompts.

EWW buffers

[Ivan Shmakov]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:

 >> You could create the *eww* buffer immediately?

 > We could, but it would look kinda odd.  Or perhaps not?  Pop up the
 > empty buffer saying "Loading..."  It's possible.

	It’s also perfectly possible to create that buffer but do /not/
	switch to it until it’s ready.  It won’t help in the scenario
	being discussed, but to be honest, EWW already pops its buffers
	way to often to my taste.

	Consider, for instance, invoking eww-reload in a handful of
	buffers in a row, – EWW will switch to each of these buffers as
	soon as one’s done, which could very well happen in the middle
	of user interaction with some specific buffer.

	Granted, it’s possible to switch to a EWW buffer, invoke
	eww-reload, wait for it to complete, and only /then/ go to some
	other buffer (whether EWW or not), but that kind of spoils the
	benefits of asynchronous url-retrieve operation, doesn’t it?

[…]

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: Network security manager

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It's an opportunity for a spinning 3-D EWW logo, so I wouldn't turn it down.

Heh heh heh.

>>> At medium or lower `nsm-security-level', I wouldn't expect to be queried
>>> in the case you describe. But at high or above, I would.
>
> LMI> If the certificate is invalid when we're visiting www.google.com, I
> LMI> think the default (i.e., `medium') setting should be to query the user.
>
> Right, sorry.  By "queried" I meant "queried even if you're in a
> different buffer."

Yeah, if we already have the *eww* buffer and you've moved out of it, I
don't think we would want to query...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: EWW buffers

[Lars Magne Ingebrigtsen]

Ivan Shmakov <ivan@siamics.net> writes:

> It’s also perfectly possible to create that buffer but do /not/
> switch to it until it’s ready.  It won’t help in the scenario
> being discussed, but to be honest, EWW already pops its buffers
> way to often to my taste.
>
> Consider, for instance, invoking eww-reload in a handful of
> buffers in a row, – EWW will switch to each of these buffers as
> soon as one’s done, which could very well happen in the middle
> of user interaction with some specific buffer.

I think this is a case of "don't do that, then".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I edited a fingerprint, set the level to `high', and then reconnected.
> It notified me that it had changed, and then returned the process.  So I
> seem to be unable to reproduce this.
>
> This is my test case:
>
> (setq process
>       (open-network-stream
>        "nntpd" (get-buffer-create "*nntp*") "google.com" "https"
>        :type 'tls))

It works for google, but when I connect to my mail server
(mail2.tohojo.dk on port 'imaps' or 993) it fails the second time
around. Perhaps because some of the certificate data is missing (no
"issued to" value for StartSSL certificates)?

-Toke

Re: Network security manager

[Lars Magne Ingebrigtsen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> It works for google, but when I connect to my mail server
> (mail2.tohojo.dk on port 'imaps' or 993) it fails the second time
> around. Perhaps because some of the certificate data is missing (no
> "issued to" value for StartSSL certificates)?

Oh, yeah, it could be a bug in the code that displays the certificate
information.  If that happens, the connection is closed.  And I think I
see the problem.

Please check out the fix in a few minutes...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19109: eww-setup-buffer: use set-buffer instead of switch-to-buffer

[Ivan Shmakov]

[-- Attachment #1: Type: text/plain, Size: 1705 bytes --]

Package:  emacs
Severity: wishlist
X-Debbugs-Cc: emacs-devel@gnu.org

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ivan Shmakov <ivan@siamics.net> writes:

 >> It’s also perfectly possible to create that buffer but do /not/
 >> switch to it until it’s ready.  It won’t help in the scenario being
 >> discussed, but to be honest, EWW already pops its buffers way to
 >> often to my taste.

 >> Consider, for instance, invoking eww-reload in a handful of buffers
 >> in a row, – EWW will switch to each of these buffers as soon as
 >> one’s done, which could very well happen in the middle of user
 >> interaction with some specific buffer.

 > I think this is a case of "don't do that, then".

	Yes.  And that means that eww-reload is essentially synchronous,
	– you can’t really invoke it and switch to doing some other
	thing; you have to wait until it completes.

 > Granted, it’s possible to switch to a EWW buffer, invoke eww-reload,
 > wait for it to complete, and only /then/ go to some other buffer
 > (whether EWW or not), but that kind of spoils the benefits of
 > asynchronous url-retrieve operation, doesn’t it?

	Personally, I just use the patch MIMEd, which makes EWW forget
	about its indiscreet habit of interrupting my activity.

	FWIW, ERC provides support for several possible behaviors when a
	new buffer gets created (see erc-join-buffer), and perhaps EWW
	should follow the suit.  OTOH, I fail to readily recall an Emacs
	package which would use switch-to-buffer on a priorly existing
	buffer as part of some background task.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/diff, Size: 256 bytes --]

--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -419,7 +419,7 @@
   (goto-char (point-min)))
 
 (defun eww-setup-buffer (&optional buffer)
-  (switch-to-buffer
+  (set-buffer
    (if (buffer-live-p buffer)
        buffer
      (get-buffer-create "*eww*")))

Re: Network security manager

[Toke Høiland-Jørgensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Oh, yeah, it could be a bug in the code that displays the certificate
> information.  If that happens, the connection is closed.  And I think I
> see the problem.
>
> Please check out the fix in a few minutes...

Yep, seems to work now. Cool! :)

-Toke

Re: Network security manager

[Toke Høiland-Jørgensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I mean that EWW's visual indicators of identity and trust should be
> global. Then you don't interrupt the user (they get cranky!) but show
> them a visual indicator that something requires their attention. I
> can't think of a better place that works in graphical and text modes
> and has the precedent of embedded infobar-style buttons than the
> modeline.

I think this may be something that could be addressed more generally.
Personally I use the Sauron package (https://github.com/djcb/sauron)
which has an interface to log notifications; I enhanced this with a
modeline indicator to show when a notification is pending (inspired by
the ERC modeline indicator stuff). This works fairly well, but an
"official" notification mechanism in Emacs would be pretty cool :)

-Toke

Re: Network security manager

[Stefan Monnier]

>> It's definitely not dangerous.  IIUC he needs this info to decide
>> whether his code can prompt the user or not.  Maybe it won't do the
>> right thing in 100% of the cases, but it's clear that if inhibit-quit is
>> non-nil, we're in a context where we shouldn't prompt the user.
> It's not clear to me at all.  Some code could set the variable for
> reasons that have no relation to prompts.

Can you give a scenario where inhibit-quit is non-nil and yet prompting
the user would be OK?


        Stefan

Re: EWW buffers

[Stefan Monnier]

> 	It’s also perfectly possible to create that buffer but do /not/
> 	switch to it until it’s ready.

Actually changing the displayed buffer from a process filter
(i.e. asynchronously) is a bad idea, just like prompting the user.
So yes, the *eww* buffer should be created and displayed right away
(empty at first).
Iceweasel does the same, FWIW, so I don't think it's a problem.


        Stefan

Re: EWW buffers

[Ivan Shmakov]

>>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

 >> It’s also perfectly possible to create that buffer but do /not/
 >> switch to it until it’s ready.

 > Actually changing the displayed buffer from a process filter
 > (i. e. asynchronously) is a bad idea, just like prompting the user.

	Yet, this is exactly what EWW currently does.  Specifically,
	eww-setup-buffer is called asynchronously when the download
	completes (from the eww-display- functions, which in turn get
	called from eww-render, which is used as the ‘url-retrieve’
	callback by EWW), /and/ eww-setup-buffer uses switch-to-buffer.

	I have filed a bug (#19109) for the issue, which was promptly
	tagged ‘notabug’ and closed.  Could someone please check if it
	really is a proper resolution?

	Personally, I’ve replaced switch-to-buffer there with
	set-buffer, and use that for I guess around a year now, –
	without any apparent issues.

 > So yes, the *eww* buffer should be created and displayed right away
 > (empty at first).  Iceweasel does the same, FWIW, so I don't think
 > it's a problem.

	(Neither do I.)

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: Network security manager

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Wed, 19 Nov 2014 17:23:39 -0500
> 
> >> It's definitely not dangerous.  IIUC he needs this info to decide
> >> whether his code can prompt the user or not.  Maybe it won't do the
> >> right thing in 100% of the cases, but it's clear that if inhibit-quit is
> >> non-nil, we're in a context where we shouldn't prompt the user.
> > It's not clear to me at all.  Some code could set the variable for
> > reasons that have no relation to prompts.
> 
> Can you give a scenario where inhibit-quit is non-nil and yet prompting
> the user would be OK?

Some hypothetical Lisp program that forces users to answer a question?

Perhaps also the "emergency exit" feature?

In any case, nothing stops some Lisp from doing the above.

Re: Network security manager

[Stefan Monnier]

>> Can you give a scenario where inhibit-quit is non-nil and yet prompting
>> the user would be OK?
> Some hypothetical Lisp program that forces users to answer a question?
> Perhaps also the "emergency exit" feature?

I can come up with hypothetical scenarios of course, but they're all
rather contrived and don't apply to Lars's situation where the prompt
can't be considered an emergency or that something that deserves to be
"forced".


        Stefan

Re: Network security manager

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Thu, 20 Nov 2014 18:34:14 -0500
> 
> >> Can you give a scenario where inhibit-quit is non-nil and yet prompting
> >> the user would be OK?
> > Some hypothetical Lisp program that forces users to answer a question?
> > Perhaps also the "emergency exit" feature?
> 
> I can come up with hypothetical scenarios of course, but they're all
> rather contrived and don't apply to Lars's situation where the prompt
> can't be considered an emergency or that something that deserves to be
> "forced".

But you were suggesting that as a general principle, not as solution
to that particular problem alone?  Your question about "a scenario"
also sounded as something rather general.  And that is how I
understood it and replied.

As for Lars's situation, there is a much simpler solution to that,
which I already pointed out earlier in this thread.  It is also much
cleaner, IMO.

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But you were suggesting that as a general principle, not as solution
> to that particular problem alone?  Your question about "a scenario"
> also sounded as something rather general.  And that is how I
> understood it and replied.

I'm not sure, but I suspect that `inhibit-quit' is not a complete
solution to the general problem of determining when we're allowed to
prompt people from asynchronous code.  I think Ted outlined a mechanism
that would work well, but would need extension to other
non-process-based forms of asynchronous code, like timers.

(Executive summary: The asynchronous code should be allowed to prompt
users in buffers "where it belongs", so if the user has moved on to a
different buffer, then it should not prompt.)

> As for Lars's situation, there is a much simpler solution to that,
> which I already pointed out earlier in this thread.  It is also much
> cleaner, IMO.

The solution you outlined ("just bind a variable") would not work for
the specific HTTPS problem that started the discussion, since url.el
works very asynchronously -- the actual code is run outside of the
dynamic extent of the binding.

But like I said about a hundred messages earlier in this thread, and
which is understandable if you and Stefan didn't catch, I've solved this
specific problem by introducing new functionality to url.el (see the
`url-request-noninteractive' parts in the nsm branch).

So you can stop discussing this specific problem.  :-)  The general
problem remains, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  emacs-devel@gnu.org
> Date: Fri, 21 Nov 2014 10:24:29 +0100
> 
> The solution you outlined ("just bind a variable") would not work for
> the specific HTTPS problem that started the discussion, since url.el
> works very asynchronously -- the actual code is run outside of the
> dynamic extent of the binding.

Sorry, I don't follow: doesn't the code run from a process filter?  If
not, what do you mean by "asynchronously" here?

Re: Network security manager

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm not sure, but I suspect that `inhibit-quit' is not a complete
> solution to the general problem of determining when we're allowed to
> prompt people from asynchronous code.

Prompting from asynchronous code is always bad.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Network security manager

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry, I don't follow: doesn't the code run from a process filter?  If
> not, what do you mean by "asynchronously" here?

It depends on what you mean by "the code".  This is a call to
`url-retrieve':

(url-retrieve "https://google.com" (lambda (&rest) (application-code)))

It's the same code used "interactively" (i.e., saying `M-x eww') as when
eww is inserting images into the buffer after it's been generated (this
is the really asynchronous bits where we don't want any prompting in
eww, because it's just too annoying).

The prompting we're talking about here is done from
`nsm-verify-connection'.  This is the stack trace for `M-x eww RET
https://google.com', and it's synchronous, and prompting is a-ok:

Debugger entered--entering a function:
* nsm-verify-connection(#<process google.com> "google.com" 443)
  network-stream-open-tls("google.com" #<buffer  *url-http-temp*> "google.com" 443 (:type tls :nowait t))
  open-network-stream("google.com" #<buffer  *url-http-temp*> "google.com" 443 :type tls :nowait t)

[...]

  url-retrieve("https://google.com/" eww-render ("https://google.com/"))
  eww("https://google.com")
  call-interactively(eww record nil)
  command-execute(eww record)
  execute-extended-command(nil "eww")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

But then, since we get a redirect to www.google.com, we get a new
`open-network-stream':

Debugger entered--entering a function:
* nsm-verify-connection(#<process www.google.no> "www.google.no" 443)
  network-stream-open-tls("www.google.no" #<buffer  *url-http-temp*> "www.google.no" 443 (:type tls :nowait t))
  open-network-stream("www.google.no" #<buffer  *url-http-temp*> "www.google.no" 443 :type tls :nowait t)

[...]

  url-http-content-length-after-change-function(221 479 258)
  url-http-wait-for-headers-change-function(1 487 486)
  url-http-generic-filter()

And that one is asynchronous, but should (in this instance) also be
allowed to prompt, since a redirect is an implementation detail the user
shouldn't know or care about.

But now we're asynchronous mode, and the user may have decided to do
something else where prompting would be ungood.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: EWW buffers

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> Actually changing the displayed buffer from a process filter
> (i.e. asynchronously) is a bad idea, just like prompting the user.
> So yes, the *eww* buffer should be created and displayed right away
> (empty at first).
> Iceweasel does the same, FWIW, so I don't think it's a problem.

True.  I'll change eww to make the buffer immediately.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 547 bytes --]

On 11/21/2014 02:36 AM, Andreas Schwab wrote:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
>> I'm not sure, but I suspect that `inhibit-quit' is not a complete
>> solution to the general problem of determining when we're allowed to
>> prompt people from asynchronous code.
> 
> Prompting from asynchronous code is always bad.

Agreed. Prompting from asynchronous code always races with C-g: users can type C-g at arbitrary times for arbitrary reasons, and that C-g can accidentally dismiss an asynchronously-displayed prompt.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Network security manager

[Stefan Monnier]

> But you were suggesting that as a general principle, not as solution
> to that particular problem alone?

No, I was thinking about a solution to Lars's situation.

> As for Lars's situation, there is a much simpler solution to that,
> which I already pointed out earlier in this thread.  It is also much
> cleaner, IMO.

I must have missed it.


        Stefan

Re: Network security manager

[Stefan Monnier]

> I'm not sure, but I suspect that `inhibit-quit' is not a complete
> solution to the general problem of determining when we're allowed to
> prompt people from asynchronous code.

No, indeed, it's probably not complete.  But maybe it's close enough for
cases like the one that started this thread.

> (Executive summary: The asynchronous code should be allowed to prompt
> users in buffers "where it belongs",

I don't think that's true in general.  It can be just as annoying for
the user to be interrupted while doing something "in the same buffer".


        Stefan

emacs-dynamic-module in Emacs Git?

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 18 Nov 2014 10:49:48 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
>>> We will also need branches for features that are being worked on by
>>> several developers, but are not yet in master.  emacs-dynamic-module,
>>> for one.
>
> SM> We've used "plain" names in the past for such branches.  I'd be happy to
> SM> see a `dynamic-module' branch added to the Git repository.
>
> Aurelien, would you like to redo your branch against the new Emacs Git
> repo and put it there as a feature branch? I can do it too, if you
> prefer.  It would make it very easy to test and share your code.

I'd like to start using this.

I looked into applying the patches myself, but it's beyond me at the
moment; I'm not familiar enough with the C code. I'll keep working on
it, both to practice with git and to learn the code. 

-- 
-- Stephe

Re: Network security manager

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Stefan requested that I didn't push this to the public repository, but
> I'm not going to finish it tonight, and I need some feedback.

> So I did anyway.  The new branch is called "nsm".

One thing you could have done is to create a new and empty repository
emacs-feature-branches on github, and then
 git cd ~/emacs/emacs-24/
 git remote add github https://github.com/larsi/emacs-feature-branches
 git push -u github nsm

Others that wanted to look at it would need to do:
 git cd ~/emacs/emacs-24/
 git remote add larsi-emacs-feature-branches https://github.com/larsi/emacs-feature-branches
 git fetch larsi-emacs-feature-branches
 git checkout  larsi-emacs-feature-branches/nsm

After this, the nsm branch will be present in their working directory,
without "polluting" upstream.

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

>> Aurelien, would you like to redo your branch against the new Emacs Git
>> repo and put it there as a feature branch? I can do it too, if you
>> prefer.  It would make it very easy to test and share your code.

BTW, this feature is expected in master "sooner rather than later", so
it's not necessary to wait for the feature to be complete before
installing it into trunk: as soon as a part is stable, it can be merged
into trunk (probably wrapped in #ifdef for now).


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Óscar Fuentes]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> Aurelien, would you like to redo your branch against the new Emacs Git
>>> repo and put it there as a feature branch? I can do it too, if you
>>> prefer.  It would make it very easy to test and share your code.
>
> BTW, this feature is expected in master "sooner rather than later", so
> it's not necessary to wait for the feature to be complete before
> installing it into trunk: as soon as a part is stable, it can be merged
> into trunk (probably wrapped in #ifdef for now).

This seems to contradict your answer to my recent question:

http://permalink.gmane.org/gmane.emacs.devel/176862

So now emacs-dynamic-module will be merged into master as soon as it
stabilizes?

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Sat, 22 Nov 2014 10:49:10 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> Aurelien, would you like to redo your branch against the new Emacs Git
>>> repo and put it there as a feature branch? I can do it too, if you
>>> prefer.  It would make it very easy to test and share your code.

SM> BTW, this feature is expected in master "sooner rather than later", so
SM> it's not necessary to wait for the feature to be complete before
SM> installing it into trunk: as soon as a part is stable, it can be merged
SM> into trunk (probably wrapped in #ifdef for now).

As I said, I'm happy to do this. I am waiting for Aurelien as a courtesy
(I CC-ed him directly on my previous e-mail) and was planning to go ahead
next week if we don't hear back.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

Hi!

On Sun, Nov 23, 2014 at 12:28 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Sat, 22 Nov 2014 10:49:10 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
>
>>>> Aurelien, would you like to redo your branch against the new Emacs Git
>>>> repo and put it there as a feature branch? I can do it too, if you
>>>> prefer.  It would make it very easy to test and share your code.
>
> SM> BTW, this feature is expected in master "sooner rather than later", so
> SM> it's not necessary to wait for the feature to be complete before
> SM> installing it into trunk: as soon as a part is stable, it can be merged
> SM> into trunk (probably wrapped in #ifdef for now).

That's very good news!

I was also trying to move my code to a proper branch of the git repo
but it's taking me some time because:

a) I initialized my repo (the one on github) with a simple tarball of
emacs sources. I ended up using git format-patch to export my commits
as patch files.
b) Applying my commits as a series of patch on top of master (in a new
branch) already resulted in several merge conflicts which need some
attention to resolve.

I'll try to have everything ready by wednesday.

> (I CC-ed him directly on my previous e-mail) and was planning to go ahead
> next week if we don't hear back.

Yeah sorry, everything sent to my +emacs address is moved to an
"emacs" folder which I have not looked at in the last few days.

mailing control@, but requesting that no replies be sent there

[Ivan Shmakov]

>>>>> Glenn Morris <rgm@gnu.org> writes:

	[Please drop Cc: 19109@debbugs.gnu.org when replying, unless the
	reply is specific to that bug report.]

 > I suggest not setting Reply-to <bug-address> when you mail the
 > control server, else the bug tracker tries to mail itself.  Such
 > messages are auto-discarded by Mailman, but it still seems like a bad
 > idea to me.

	Agreed.

	Curiously, what’s the proper way to prevent (wide) replies to my
	messages from having Cc: control@ by default?  (Other than
	requesting just that in plain English at the top of the
	message’s body, that is.)

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: Network security manager

[Lars Magne Ingebrigtsen]

"Garreau, Alexandre" <galex-713@galex-713.eu> writes:

> Unfortunately no, but there are several implementations, mainly in
> javascript, PHP, and, err, Java, as far as I know. But it’s quite simple
> and should be easily possible in any language where you can draw simple
> figures.
>
> The three I know:
> https://github.com/sebsauvage/VizHash/
> https://github.com/sametmax/VizHash.js
> https://github.com/inouire/VizHash4j
>
> I’d be really glad if someone found a way to do it with elisp… maybe
> using an external program?

It seems really easy to implement in Emacs Lisp + svg, so that's no
problem.  I've started implementing an SVG creation library.

However, I'm now looking at the algorithm this uses, and I notice:

 var hash = hex_sha1(text) + hex_md5(text);

I think the common reaction to seeing md5 being used for anything these
days is "err".  Although it's probably OK here, I wonder what's the
chance of this algorithm getting much uptake?  Has anybody started using
this?  Is there an RFC?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Although it's probably OK here, I wonder what's the
> chance of this algorithm getting much uptake?  Has anybody started using
> this?  Is there an RFC?

And the gnutls library exports a sha1 hash of the pubkey, so I'm not
quite sure how to get the md5 of it as well...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Garreau, Alexandre]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

Le 23/11/2014 à 20h53, Lars Magne Ingebrigtsen a écrit :
> "Garreau, Alexandre" <galex-713@galex-713.eu> writes:
>> Unfortunately no, but there are several implementations, mainly in
>> javascript, PHP, and, err, Java, as far as I know. But it’s quite simple
>> and should be easily possible in any language where you can draw simple
>> figures.
>>
>> The three I know:
>> https://github.com/sebsauvage/VizHash/
>> https://github.com/sametmax/VizHash.js
>> https://github.com/inouire/VizHash4j
>>
>> I’d be really glad if someone found a way to do it with elisp… maybe
>> using an external program?
>
> It seems really easy to implement in Emacs Lisp + svg, so that's no
> problem.  I've started implementing an SVG creation library.
>
> However, I'm now looking at the algorithm this uses, and I notice:
>
>  var hash = hex_sha1(text) + hex_md5(text);
>
> I think the common reaction to seeing md5 being used for anything these
> days is "err".  Although it's probably OK here, I wonder what's the
> chance of this algorithm getting much uptake?  Has anybody started using
> this?

Yeah, I’ve been surprised by that too. I were thinking that if I had to
make an implementation some day I’d use SHA512 instead.

> Is there an RFC?

No, the developers had the idea and gave some examples of usages
(background change within firefox according domain name’s vizhash to
prevent unicode-phishing for instance, or password verification, or
things like that) without taking care of spreading the idea (which I
think could have a real success).

Le 23/11/2014 à 20h59, Lars Magne Ingebrigtsen a écrit :
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Although it's probably OK here, I wonder what's the
>> chance of this algorithm getting much uptake?  Has anybody started using
>> this?  Is there an RFC?
>
> And the gnutls library exports a sha1 hash of the pubkey, so I'm not
> quite sure how to get the md5 of it as well...

Oh, I thought gnutls could give an md5 of pubkey since certtool --info
give the md5sum just before the sha1… Anyway if it’s to gnutls to
calculate it it means it’ll be less secure and more likely to find
collisions… :/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

Re: Network security manager

[Lars Magne Ingebrigtsen]

"Garreau, Alexandre" <galex-713@galex-713.eu> writes:

> No, the developers had the idea and gave some examples of usages
> (background change within firefox according domain name’s vizhash to
> prevent unicode-phishing for instance, or password verification, or
> things like that) without taking care of spreading the idea (which I
> think could have a real success).

Right.

> Oh, I thought gnutls could give an md5 of pubkey since certtool --info
> give the md5sum just before the sha1… Anyway if it’s to gnutls to
> calculate it it means it’ll be less secure and more likely to find
> collisions… :/

If I remember correctly, it gives both md5 and sha1 of the certificate
ID, but not the public key ID.

The gnutls function for getting the public key ID is
gnutls_x509_crt_get_key_id, which does not take the hashing function as
an input -- it just outputs the sha1.

My take on the situation is that I think stuff like this:

function hashString(text) {
  var hash = hex_sha1(text) + hex_md5(text);
  return hash + hash.split('').reverse().join('');
}

(i.e., sha1+md5, and then add a reversed version of that to get plenty
of values to make drawings out of) is unlikely to get much uptake as a
visualisation method throughout the industry.

I like the idea: Showing a (somewhat) memorable image (and it's
certainly a lot more memorable than the ssh "random art").  But if this
doesn't get any uptake outside of Emacs, is it worth doing in Emacs?

Of course, the images we show in Emacs could be Emacs-"proprietary".
But then we could just disregard the vizhash implementation completely
and do our own algorithm based on better hashes.

I think.

Anyway, to implement the algorithm as is, we'd have to replicate most of
gnutls_x509_crt_get_key_id to get at the md5.  That's not a major issue,
but...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Anyway, to implement the algorithm as is, we'd have to replicate most of
> gnutls_x509_crt_get_key_id to get at the md5.  That's not a major issue,
> but...

Actually, looking at that code, it's longwinded, but seems to boils down
to just this:

  result = asn1_der_coding (crt->cert, "tbsCertificate.subjectPublicKeyInfo",
                            pubkey.data, &pubkey.size, NULL);
  result = gnutls_fingerprint (GNUTLS_DIG_SHA1, &pubkey,
                               output_data, output_data_size);

So re-implementing this to get both the MD5 and the SHA1 is actually
pretty easy.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[Lars Magne Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

I think the graphical representation of certificates may be a research
project for a cryptography person.  Even if we start from a good hash,
are we sure that the graphical representation can't been manipulated so
that to different hashes get very similar representations?

Etc.

So I don't know about vizhash.

Anybody got an opinion?

Meanwhile, I think the svg.el library is good to go.  :-)

(setq svg (svg-create 256 256 :stroke "orange" :stroke-width 5))
(svg-gradient svg "gradient" 'linear '(0 . "red") '(100 . "blue"))
(svg-rectangle svg 100 100 150 150 :gradient "gradient")
(svg-circle svg 150 200 20)
(svg-ellipse svg 100 100 50 70 :stroke "red")
(svg-line svg 100 100 50 70)
(svg-polyline svg '((100 . 100) (200 . 150) (150 . 90)) :stroke "green")
(insert-image (svg-image svg))

Gives this pretty image:


[-- Attachment #2: svg.png --]
[-- Type: image/png, Size: 7118 bytes --]

[-- Attachment #3: Type: text/plain, Size: 216 bytes --]


Might be more of an ELPA candidate than an Emacs candidate, though,
unless we decide to do vizhash.el anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Network security manager

[joakim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I think the graphical representation of certificates may be a research
> project for a cryptography person.  Even if we start from a good hash,
> are we sure that the graphical representation can't been manipulated so
> that to different hashes get very similar representations?
>
> Etc.
>
> So I don't know about vizhash.
>
> Anybody got an opinion?
>
> Meanwhile, I think the svg.el library is good to go.  :-)
>
> (setq svg (svg-create 256 256 :stroke "orange" :stroke-width 5))
> (svg-gradient svg "gradient" 'linear '(0 . "red") '(100 . "blue"))
> (svg-rectangle svg 100 100 150 150 :gradient "gradient")
> (svg-circle svg 150 200 20)
> (svg-ellipse svg 100 100 50 70 :stroke "red")
> (svg-line svg 100 100 50 70)
> (svg-polyline svg '((100 . 100) (200 . 150) (150 . 90)) :stroke "green")
> (insert-image (svg-image svg))
>
> Gives this pretty image:
>
>
>
> Might be more of an ELPA candidate than an Emacs candidate, though,
> unless we decide to do vizhash.el anyway.

Very nice!

I have some hackish code to do printable expense forms in svg with
emacs. This looks better than my xsltproc hacks.

-- 
Joakim Verona

Re: bug#19109: mailing control@, but requesting that no replies be sent there

[Glenn Morris]

Ivan Shmakov wrote:

> 	Curiously, what's the proper way to prevent (wide) replies to my
> 	messages from having Cc: control@ by default? (Other than
> 	requesting just that in plain English at the top of the
> 	message's body, that is.)

Use bcc when you want to talk to control.

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

On Sun, Nov 23, 2014 at 11:38 AM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> I was also trying to move my code to a proper branch of the git repo
> ...
> I'll try to have everything ready by wednesday.

I ended up doing it today after all. So I have a local
"dynamic_modules" branch on my repo. What is the next step?

bug#19109: mailing control@, but requesting that no replies be sent there

[Stephen J. Turnbull]

Ivan Shmakov writes:

 > 	Curiously, what’s the proper way to prevent (wide) replies to my
 > 	messages from having Cc: control@ by default?  (Other than
 > 	requesting just that in plain English at the top of the
 > 	message’s body, that is.)

You might be able to Bcc control, which is probably best.  However
control (or the MTA in front of it) may insist that it be in the
explicit addressees.

Otherwise, it's not possible for you to control this.

Re: mailing control@, but requesting that no replies be sent there

[Ivan Shmakov]

>>>>> Glenn Morris <rgm@gnu.org> writes:
>>>>> Ivan Shmakov wrote:

 >> Curiously, what's the proper way to prevent (wide) replies to my
 >> messages from having Cc: control@ by default?  (Other than
 >> requesting just that in plain English at the top of the message's
 >> body, that is.)

 > Use bcc when you want to talk to control.

	Won’t Mail-Followup-To: suffice?  Granted, the support for that
	isn’t universal, yet still widespread enough.  Then, I hope
	control@ itself, not being a MUA, would just ignore one.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: mailing control@, but requesting that no replies be sent there

[Stephen J. Turnbull]

Ivan Shmakov writes:

 > 	Won’t Mail-Followup-To: suffice?  Granted, the support for that
 > 	isn’t universal, yet still widespread enough.

Depends on your definition of "widespread enough".  I don't think it
is, but YMMV.

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Mon, 24 Nov 2014 02:19:50 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Sun, Nov 23, 2014 at 11:38 AM, Aurélien Aptel
AA> <aurelien.aptel+emacs@gmail.com> wrote:
>> I was also trying to move my code to a proper branch of the git repo
>> ...
>> I'll try to have everything ready by wednesday.

AA> I ended up doing it today after all. So I have a local
AA> "dynamic_modules" branch on my repo. What is the next step?

Push it to your Github repo
https://github.com/aaptel/emacs-dynamic-module so it can be reviewed.  We
will try to get to it quickly and merge it with the necessary ifdefs, if
you don't have them.

Thanks
Ted

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

On Tue, Nov 25, 2014 at 11:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Push it to your Github repo
> https://github.com/aaptel/emacs-dynamic-module so it can be reviewed.  We
> will try to get to it quickly and merge it with the necessary ifdefs, if
> you don't have them.

I've force-pushed on my repo. The branch is "dynamic-modules". It
should already be properly ifdef'd and turned off by default. You can
turn it on via the configure script with --with-ltdl.

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Wed, 26 Nov 2014 18:05:55 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Tue, Nov 25, 2014 at 11:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Push it to your Github repo
>> https://github.com/aaptel/emacs-dynamic-module so it can be reviewed.  We
>> will try to get to it quickly and merge it with the necessary ifdefs, if
>> you don't have them.

AA> I've force-pushed on my repo. The branch is "dynamic-modules". It
AA> should already be properly ifdef'd and turned off by default. You can
AA> turn it on via the configure script with --with-ltdl.

I've pushed your dynamic-modules branch (rebased against today's master
branch) to the dynamic-modules branch on the Emacs repo for review and
testing. Please use that in your repo as well.

There are just a few issues, none major:

* two XXX comments

* the ChangeLog entries need rewriting (I would squash to a single
  commit anyway, if that's OK with you, which would make the above
  rebasing note irrelevant)

* no documentation (NEWS, ELisp manual, etc.)

For me only the first one is a blocker. The documentation is also
important--at least mention how to use this new facility in a text file
please. We can try to write it if you're unable, but it would be good to
give us a first draft at least.

Thanks
Ted

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

On Thu, Nov 27, 2014 at 3:10 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Wed, 26 Nov 2014 18:05:55 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote:
>
> AA> On Tue, Nov 25, 2014 at 11:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> Push it to your Github repo
>>> https://github.com/aaptel/emacs-dynamic-module so it can be reviewed.  We
>>> will try to get to it quickly and merge it with the necessary ifdefs, if
>>> you don't have them.
>
> AA> I've force-pushed on my repo. The branch is "dynamic-modules". It
> AA> should already be properly ifdef'd and turned off by default. You can
> AA> turn it on via the configure script with --with-ltdl.
>
> I've pushed your dynamic-modules branch (rebased against today's master
> branch) to the dynamic-modules branch on the Emacs repo for review and
> testing. Please use that in your repo as well.

Will do.

> There are just a few issues, none major:
>
> * two XXX comments

Well, it's still experimental code, what's bothering you?

> * the ChangeLog entries need rewriting (I would squash to a single
>   commit anyway, if that's OK with you, which would make the above
>   rebasing note irrelevant)

I didn't write any changelog entries. Are you talking about commit
messages? Not everything in there is relevant to the ChangeLog file
and some might be obsolete.

> * no documentation (NEWS, ELisp manual, etc.)
>
> For me only the first one is a blocker. The documentation is also
> important--at least mention how to use this new facility in a text file
> please. We can try to write it if you're unable, but it would be good to
> give us a first draft at least.

Sure. Where should I write this documentation? I can reuse what I
wrote in my announcement [1].

1: https://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00292.html

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Thu, 27 Nov 2014 16:38:48 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Thu, Nov 27, 2014 at 3:10 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:

>> There are just a few issues, none major:
>> 
>> * two XXX comments

AA> Well, it's still experimental code, what's bothering you?

They seemed like blockers.  If you say they're OK, all right :)

>> * the ChangeLog entries need rewriting (I would squash to a single
>> commit anyway, if that's OK with you, which would make the above
>> rebasing note irrelevant)

AA> I didn't write any changelog entries. Are you talking about commit
AA> messages? Not everything in there is relevant to the ChangeLog file
AA> and some might be obsolete.

Yes, sorry, I meant the commit messages need rewriting.

>> * no documentation (NEWS, ELisp manual, etc.)
>> 
>> For me only the first one is a blocker. The documentation is also
>> important--at least mention how to use this new facility in a text file
>> please. We can try to write it if you're unable, but it would be good to
>> give us a first draft at least.

AA> Sure. Where should I write this documentation? I can reuse what I
AA> wrote in my announcement [1].

AA> 1: https://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00292.html

I'm not sure, honestly. There's a "hacking on Emacs" thing but I'm not
sure this belongs there. Maybe a text file is best while we figure it
out.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 26 Nov 2014 21:10:31 -0500
> 
> I've pushed your dynamic-modules branch (rebased against today's master
> branch) to the dynamic-modules branch on the Emacs repo for review and
> testing.

Thanks (to both of you).

> There are just a few issues, none major:

I have a few more, specific to the Windows build:

  . As with other optional libraries, using libltdl should detect its
    availability at run time, and load it dynamically, instead of
    passing -ltdl switch to the linker when Emacs is built.  See
    examples of how to do that in image.c and gnutls.c.

  . The Makefile's use literal .so extension for the dynamic libraries
    being built -- this is non-portable and should be determined at
    configure time.

  . It seems to me that the modules call functions implemented by
    Emacs, like make_number and Fmember, on the assumption that
    calling any Emacs function will "just work".  This is false for
    Windows (the link command that produces the shared object will
    fail), unless we mark such exportable functions and build an
    import library that will be passed to the linker when the module's
    shared object is built.  Likewise with global variables defined by
    Emacs.

    (In general, I question why "modules" that require such tight
    integration with Emacs internals are a good idea: why not simply
    add them to the Emacs core and be done with that?  What do we gain
    by having them as separate .so shared objects?)

  . I don't understand why load-module-suffixes includes extensions
    from different platforms, nor why is this variable needed at all.
    AFAIK, libltdl is perfectly capable of finding shared objects on
    each platform without us feeding it the extension to use.  It
    knows what extensions are used by the underlying platform for
    shared libraries.  Moreover, trying to load .dylib or .dll on
    GNU/Linux will hardly produce good results, so why risk that by
    exposing the platform-specific extensions at all?

Please note that the above is based solely on examining the
source-level diffs; I didn't yet try to build this branch or use it.

Thanks again for working on this.

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> Date: Sat, 29 Nov 2014 19:05:06 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
>   . It seems to me that the modules call functions implemented by
>     Emacs, like make_number and Fmember, on the assumption that
>     calling any Emacs function will "just work".  This is false for
>     Windows (the link command that produces the shared object will
>     fail), unless we mark such exportable functions and build an
>     import library that will be passed to the linker when the module's
>     shared object is built.  Likewise with global variables defined by
>     Emacs.

Alternatively, I think libltdl itself supports dynamic linking with
functions in the main module (Emacs itself), but that would require an
explicit lt_dlsym call for every Emacs function a module needs to
call.

In any case, just calling a function is not going to cut it.

Re: Network security manager

[Stefan Monnier]

> Might be more of an ELPA candidate than an Emacs candidate, though,
> unless we decide to do vizhash.el anyway.

Yes, please add it to elpa.git.  As a general rule, people should feel
free to add packages to elpa.git, as long as the package is of
general use.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

>     fail), unless we mark such exportable functions and build an
>     import library that will be passed to the linker when the module's
>     shared object is built.  Likewise with global variables defined by
>     Emacs.

Indeed, we'll have to define clearly what can be called.  Basically,
design an API for modules.

>     (In general, I question why "modules" that require such tight
>     integration with Emacs internals are a good idea: why not simply
>     add them to the Emacs core and be done with that?  What do we gain
>     by having them as separate .so shared objects?)

Because we want to provide the possibility to distribute dynamic
moduless via GNU ELPA, MELPA, Marmalade, ...


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>   emacs-devel@gnu.org
> Date: Sun, 30 Nov 2014 09:08:42 -0500
> 
> >     (In general, I question why "modules" that require such tight
> >     integration with Emacs internals are a good idea: why not simply
> >     add them to the Emacs core and be done with that?  What do we gain
> >     by having them as separate .so shared objects?)
> 
> Because we want to provide the possibility to distribute dynamic
> moduless via GNU ELPA, MELPA, Marmalade, ...

Like I said: I question whether this is a good idea.  Such modules
will break all the time due to internal changes, and at best will only
work with one particular version of Emacs internals.  These archives
are not a good way to impose the kind of discipline it takes to keep
backward compatibility.

But I guess all this was already brought up.  I just saw the slippery
slope the few modules in that branch start on and couldn't resist.

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> Like I said: I question whether this is a good idea.  Such modules
> will break all the time due to internal changes, and at best will only
> work with one particular version of Emacs internals.

The compiled module will probably be very closely tied to a particular
Emacs version, indeed.  The source code will probably also depend on
some narrow version range.  The way I view it, the issue is simply that
it will take time to stabilize the API.


        Stefan

Re: Network security manager

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Yes, please add it to elpa.git.  As a general rule, people should feel
> free to add packages to elpa.git, as long as the package is of
> general use.

Okidoke; I've now added svg.el to ELPA.  It seemed too easy, so I've
probably forgotten to do something I should have.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Sun, 30 Nov 2014 13:09:38 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Like I said: I question whether this is a good idea.  Such modules
>> will break all the time due to internal changes, and at best will only
>> work with one particular version of Emacs internals.

SM> The compiled module will probably be very closely tied to a particular
SM> Emacs version, indeed.  The source code will probably also depend on
SM> some narrow version range.  The way I view it, the issue is simply that
SM> it will take time to stabilize the API.

What do you think about the current modules?

Ted

Re: Network security manager

[Stefan Monnier]

> Okidoke; I've now added svg.el to ELPA.

Thanks.

> It seemed too easy,

It's designed to be "as easy as possible".

> so I've probably forgotten to do something I should have.  :-)

We'll see tomorrow.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> What do you think about the current modules?

Haven't really looked at them yet.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Sun, 30 Nov 2014 22:41:28 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> What do you think about the current modules?
SM> Haven't really looked at them yet.

It would be good if everyone did before it's merged. It's probably the
most important new architectural feature we'll see in the next release.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

>>> What do you think about the current modules?
SM> Haven't really looked at them yet.
> It would be good if everyone did before it's merged. It's probably the
> most important new architectural feature we'll see in the next release.

Then I must have misunderstood.  We were talking about APIs, and AFAIK
the new code doesn't really define an API (yet).  So "current modules"
above seemed to refer to actual modules, not about the new Emacs-side
code that hosts/loads those modules.

Please clarify,


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

On Sat, Nov 29, 2014 at 6:05 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> I have a few more, specific to the Windows build:
>
>   . As with other optional libraries, using libltdl should detect its
>     availability at run time, and load it dynamically, instead of
>     passing -ltdl switch to the linker when Emacs is built.  See
>     examples of how to do that in image.c and gnutls.c.

What's the point of doing it this way? Honest question.

>   . The Makefile's use literal .so extension for the dynamic libraries
>     being built -- this is non-portable and should be determined at
>     configure time.

The makefile is temporary. It can be replaced by a more portable
script or elisp afterwards.

>   . It seems to me that the modules call functions implemented by
>     Emacs, like make_number and Fmember, on the assumption that
>     calling any Emacs function will "just work".  This is false for

I had to add a linker flag to expose every symbol of Emacs. See the
relevant commit:

http://git.savannah.gnu.org/cgit/emacs.git/commit/configure.ac?h=dynamic-modules&id=5c710fba15e0a3a2ae5d831e5cdb555332238752

>     Windows (the link command that produces the shared object will
>     fail), unless we mark such exportable functions and build an
>     import library that will be passed to the linker when the module's
>     shared object is built.  Likewise with global variables defined by
>     Emacs.

I've never used Windows much for developement. Isn't there a similar
flag we can use on mingw?

>     (In general, I question why "modules" that require such tight
>     integration with Emacs internals are a good idea: why not simply
>     add them to the Emacs core and be done with that?  What do we gain
>     by having them as separate .so shared objects?)

We could directly add them to Emacs I guess... It's just that time has
shown no-one actually agrees to do it.

>   . I don't understand why load-module-suffixes includes extensions
>     from different platforms, nor why is this variable needed at all.

`load' has complex logic based on suffixes and knowing the type of
files based on them. I tried to re-use what I could without risking
breaking something.

>     AFAIK, libltdl is perfectly capable of finding shared objects on
>     each platform without us feeding it the extension to use.  It
>     knows what extensions are used by the underlying platform for
>     shared libraries.  Moreover, trying to load .dylib or .dll on
>     GNU/Linux will hardly produce good results, so why risk that by
>     exposing the platform-specific extensions at all?

I've never had other shared libraries than my system native ones,
especially in an emacs related dir. I don't think it's a problem given
you can load a module using the full path to it.

>
> Please note that the above is based solely on examining the
> source-level diffs; I didn't yet try to build this branch or use it.
>
> Thanks again for working on this.

Thanks for the feedback :-)

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

On Mon, Dec 1, 2014 at 2:45 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> Then I must have misunderstood.  We were talking about APIs, and AFAIK
> the new code doesn't really define an API (yet).  So "current modules"
> above seemed to refer to actual modules, not about the new Emacs-side
> code that hosts/loads those modules.

The biggest architectural change will probably be the new Lisp_Object
type dedicated to opaque module type we were talking about in a
previous thread (still not implemented).

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Mon, 01 Dec 2014 08:45:53 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>>> What do you think about the current modules?
SM> Haven't really looked at them yet.
>> It would be good if everyone did before it's merged. It's probably the
>> most important new architectural feature we'll see in the next release.

SM> Then I must have misunderstood.  We were talking about APIs, and AFAIK
SM> the new code doesn't really define an API (yet).  So "current modules"
SM> above seemed to refer to actual modules, not about the new Emacs-side
SM> code that hosts/loads those modules.

The branch defines both actual modules and the means to load and call
them. Currently they can call the Emacs core directly, so there's no
API. Aurélien was already asking how to define a global variable from
inside a module. Also modules can have a license and other metadata that
should be easy to obtain before loading them. So I hope the API is
settled to some degree before we merge.

IMHO the first API should be call-only and very simple:

* load: makes the functions available

* describe: list the functions, the module metadata (version, license,
  etc), and the loaded/not loaded state. Functions are passed as a
  struct with a name and 1-7 parameters; each parameter has a name and
  type (text, 32-bit int, or 48-bit float).

* funcall: call a specific function (this would be hidden from the ELisp
  code, which just calls the function by name)

Ted

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> The branch defines both actual modules and the means to load and call
> them.  Currently they can call the Emacs core directly, so there's no
> API.  Aurélien was already asking how to define a global variable from
> inside a module.

Exactly.

By "API" I meant something like a .h file which modules can include to
define the functions and datatypes that they can use.  I.e. by API
I mean those things that the module code can do, rather than the things
that Emacs code can do with modules.

Currently you can use anything, which is clearly incompatible with
providing some amount of API stability.

> Also modules can have a license and other metadata that should be easy
> to obtain before loading them.  So I hope the API is settled to some
> degree before we merge.

The code to check modules's license and to load them should be settled,
indeed.  I don't see any big problems on this side.

OTOH the API above probably will take a long time to settle, much
longer than "before we merge" and probably even longer than "before the
next release".

Another part of the modules's infrastructure will be to provide support
for building/installing modules via something like ELPA.

So, IIUC currently we have the first part (load a module) mostly working.
The second part (define an API) will shape up as experience is gained by
writing modules.
The third part (ELPA support) is still in the future.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Mon, 01 Dec 2014 10:04:34 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> By "API" I meant something like a .h file which modules can include to
SM> define the functions and datatypes that they can use.  I.e. by API
SM> I mean those things that the module code can do, rather than the things
SM> that Emacs code can do with modules.

By API I meant both directions, the module API for registration and
metadata, and the Emacs API that modules can use. So I still think a
call-only API (only in the direction of calling the module) is best for
now, so that .h file is unnecessary.

I agree with the rest of your comments, except that it's not clear when
you'll feel that the module loading is settled enough to merge into the
master branch.  I think the module metadata should be formalized, at
least, so it can be obtained before loading the module.  Aurélien, is
that possible?  And do you have any comments?

Ted

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 01 Dec 2014 05:31:06 -0500
> 
> On Sun, 30 Nov 2014 22:41:28 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
> 
> >> What do you think about the current modules?
> SM> Haven't really looked at them yet.
> 
> It would be good if everyone did before it's merged. It's probably the
> most important new architectural feature we'll see in the next release.

What exactly would I like us to review there?  I briefly scanned
through them, and all I saw was rather simple and straightforward code
to implement simple Emacs primitives.  What aspects of that need more
serious review?

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

Stefan's 3 part plan is sound.

I agree that using any Emacs function in modules is too fragile and we
need to define a subset or a new layer to expose Emacs features to
modules.

On Mon, Dec 1, 2014 at 4:36 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Mon, 01 Dec 2014 10:04:34 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
>
> SM> By "API" I meant something like a .h file which modules can include to
> SM> define the functions and datatypes that they can use.  I.e. by API
> SM> I mean those things that the module code can do, rather than the things
> SM> that Emacs code can do with modules.
>
> By API I meant both directions, the module API for registration and
> metadata, and the Emacs API that modules can use. So I still think a
> call-only API (only in the direction of calling the module) is best for
> now, so that .h file is unnecessary.

In order to do anything useful (even in a "call-only" API) you need
access to Emacs facilities. At the very least access to Elisp data
structures (symbols, numbers, strings and buffers). The point of
compiled modules is also efficiency so the access has to be low level
enough (no big conversion or copying needed).

> I agree with the rest of your comments, except that it's not clear when
> you'll feel that the module loading is settled enough to merge into the
> master branch.  I think the module metadata should be formalized, at
> least, so it can be obtained before loading the module.  Aurélien, is
> that possible?  And do you have any comments?

Apart from the the new module/opaque type I'd like to add, the way I
see it the branch can already be merged and used as a very
experimental new feature. Writing the yaml module was a good exercise.
We need people to try writing modules and see what they end up
using/needing in order to choose what we should keep in the API.

For me a module package will be composed of a metadata file, a bunch
of C files implementing small and simple primitives and a bunch of
elisp files for the logic.

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> Date: Mon, 1 Dec 2014 14:59:00 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> On Sat, Nov 29, 2014 at 6:05 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > I have a few more, specific to the Windows build:
> >
> >   . As with other optional libraries, using libltdl should detect its
> >     availability at run time, and load it dynamically, instead of
> >     passing -ltdl switch to the linker when Emacs is built.  See
> >     examples of how to do that in image.c and gnutls.c.
> 
> What's the point of doing it this way? Honest question.

Honest answer: to allow users to download and install Emacs binaries
and run Emacs without also installing the optional libraries, which
are available as separate DLLs.  If we link Emacs against those DLLs,
it will refuse to start when those DLLs are not present on the
end-user's machine.

> >   . It seems to me that the modules call functions implemented by
> >     Emacs, like make_number and Fmember, on the assumption that
> >     calling any Emacs function will "just work".  This is false for
> 
> I had to add a linker flag to expose every symbol of Emacs. See the
> relevant commit:
> 
> http://git.savannah.gnu.org/cgit/emacs.git/commit/configure.ac?h=dynamic-modules&id=5c710fba15e0a3a2ae5d831e5cdb555332238752

I don't think this is correct: we don't really want to export all the
symbols.

> >     Windows (the link command that produces the shared object will
> >     fail), unless we mark such exportable functions and build an
> >     import library that will be passed to the linker when the module's
> >     shared object is built.  Likewise with global variables defined by
> >     Emacs.
> 
> I've never used Windows much for developement. Isn't there a similar
> flag we can use on mingw?

The same flag will also work with MinGW, but it is not the problem I
was talking about.  The problem is that on Windows, when you link the
modules themselves into shared libraries, the linker must be presented
with information about symbols exported by Emacs.  You could use the
Emacs binary, of course, but that's cumbersome.  The usual technique
is to generate a special import library as side effect of building
Emacs, and then submit that import library to the linker when you
create the module's shared library.

The import library is by convention called libFOO-NN.dll.a (where
"FOO" in this case will probably be "emacs", and NN will probably be
the Emacs version).  It is automatically created by the linker if you
pass the "-Wl,--out-implib=libFOO-NN.dll.a" switch to the compiler on
the link command line.

All this is pretty standard practice on Windows and works well, the
only issue for us is to decide which functions to export (unless we
really want to export all of them, in which case the --export-dynamic
linker switch is all that's needed).

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Mon, 1 Dec 2014 17:28:01 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> In order to do anything useful (even in a "call-only" API) you need
AA> access to Emacs facilities. At the very least access to Elisp data
AA> structures (symbols, numbers, strings and buffers). 

I think it's less efficient but also safer to return data as text in
sexp or JSON format.  Exposing the Emacs internals just to return data
seems risky.  Some modules may need that level of access, but most
don't.

AA> The point of compiled modules is also efficiency so the access has
AA> to be low level enough (no big conversion or copying needed).

Of course, but the difference is not that big and it can be optimized
later.

AA> Apart from the the new module/opaque type I'd like to add, the way I
AA> see it the branch can already be merged and used as a very
AA> experimental new feature. Writing the yaml module was a good exercise.
AA> We need people to try writing modules and see what they end up
AA> using/needing in order to choose what we should keep in the API.

Yup.  I will write one for libnettle/libhogweed as soon as I am able.

AA> For me a module package will be composed of a metadata file, a bunch
AA> of C files implementing small and simple primitives and a bunch of
AA> elisp files for the logic.

Great.  But can Emacs find out a module's metadata before loading it?
Do you think it should just parse the metadata file?  That doesn't
guarantee compliance like calling a "get metadata" function in the
library itself, but perhaps that's all right.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> Date: Mon, 1 Dec 2014 17:28:01 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> 
> Stefan's 3 part plan is sound.
> 
> I agree that using any Emacs function in modules is too fragile and we
> need to define a subset or a new layer to expose Emacs features to
> modules.

Note that there are several possible ways of doing this, and IMO we
should decide which one we would like to use.

One possibility is not to create Lisp objects in the module, but
instead ask Emacs to create an object and hand it to the module.

Another possibility is to expose an array of function pointers through
which modules will call Emacs functions.  (This alternative avoids the
problems on Windows that require an import library.)

In any case, freeing memory should always be done on the same side of
the divide as its allocation.

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> By API I meant both directions, the module API for registration and
> metadata, and the Emacs API that modules can use. So I still think a
> call-only API (only in the direction of calling the module) is best for
> now, so that .h file is unnecessary.

The exported functions will need to take Lisp_Object arguments and
return Lisp_Object values, so they need to be able to test&create
Lisp_Objects, hence they need to make calls to Emacs's C code even for
the most trivial module imaginable.

> I agree with the rest of your comments, except that it's not clear when
> you'll feel that the module loading is settled enough to merge into the
> master branch.

The criteria for me is not whether the feature is ready for general use,
but whether the code that needs to be merged is sufficiently clean
and stable ("stable" in the sense that it probably won't need to be
completely replaced by a different implementation, so future changes
should be "incremental improvements").
The current code can pretty much be merged as it is.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> Another possibility is to expose an array of function pointers through
> which modules will call Emacs functions.  (This alternative avoids the
> problems on Windows that require an import library.)

I think creating an import library is the way to go on Windows.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Mon, 01 Dec 2014 14:12:08 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> By API I meant both directions, the module API for registration and
>> metadata, and the Emacs API that modules can use. So I still think a
>> call-only API (only in the direction of calling the module) is best for
>> now, so that .h file is unnecessary.

SM> The exported functions will need to take Lisp_Object arguments and
SM> return Lisp_Object values, so they need to be able to test&create
SM> Lisp_Objects, hence they need to make calls to Emacs's C code even for
SM> the most trivial module imaginable.

I don't think that's the only way.  Eli and I listed some alternatives.

>> I agree with the rest of your comments, except that it's not clear when
>> you'll feel that the module loading is settled enough to merge into the
>> master branch.

SM> The criteria for me is not whether the feature is ready for general use,
SM> but whether the code that needs to be merged is sufficiently clean
SM> and stable ("stable" in the sense that it probably won't need to be
SM> completely replaced by a different implementation, so future changes
SM> should be "incremental improvements").
SM> The current code can pretty much be merged as it is.

OK, I'll do it in a few hours unless someone objects.  Thanks for your review.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 01 Dec 2014 10:04:34 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
> SM> By "API" I meant something like a .h file which modules can include to
> SM> define the functions and datatypes that they can use.  I.e. by API
> SM> I mean those things that the module code can do, rather than the things
> SM> that Emacs code can do with modules.
>
> By API I meant both directions, the module API for registration and
> metadata, and the Emacs API that modules can use. So I still think a
> call-only API (only in the direction of calling the module) is best for
> now, so that .h file is unnecessary.

In my particular case, this will not be useful; I need to call some
ada-mode elisp functions from a compiled parser.

These elisp functions will not be in any .h file, but I will need the
appropriate "funcall" in that .h file.

> I agree with the rest of your comments, except that it's not clear when
> you'll feel that the module loading is settled enough to merge into the
> master branch.  

I suggest waiting until a couple more people write actual modules.


-- 
-- Stephe

Re: emacs-dynamic-module in Emacs Git?

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 1 Dec 2014 17:28:01 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 
>
> AA> In order to do anything useful (even in a "call-only" API) you need
> AA> access to Emacs facilities. At the very least access to Elisp data
> AA> structures (symbols, numbers, strings and buffers). 
>
> I think it's less efficient but also safer to return data as text in
> sexp or JSON format.  Exposing the Emacs internals just to return data
> seems risky.  Some modules may need that level of access, but most
> don't.

My module will be setting text properties in buffers, based on the
results of parsing the code; it's replacing elisp code that does that
now, which is too slow.

> AA> The point of compiled modules is also efficiency so the access has
> AA> to be low level enough (no big conversion or copying needed).
>
> Of course, but the difference is not that big and it can be optimized
> later.

I have moved the parsing code into a separate executable communicating
by pipes. The communication and subsequent text processing takes a
significant time; too long for my requirements. Which is why I'm looking
for a more direct approach.

> AA> For me a module package will be composed of a metadata file, a bunch
> AA> of C files implementing small and simple primitives and a bunch of
> AA> elisp files for the logic.
>
> Great.  But can Emacs find out a module's metadata before loading it?

It can if it's an ELPA package.

-- 
-- Stephe

Re: emacs-dynamic-module in Emacs Git?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>
>> >   . It seems to me that the modules call functions implemented by
>> >     Emacs, like make_number and Fmember, on the assumption that
>> >     calling any Emacs function will "just work".  This is false for
>> 
>> I had to add a linker flag to expose every symbol of Emacs. See the
>> relevant commit:
>> 
>> http://git.savannah.gnu.org/cgit/emacs.git/commit/configure.ac?h=dynamic-modules&id=5c710fba15e0a3a2ae5d831e5cdb555332238752
>
> I don't think this is correct: we don't really want to export all the
> symbols.

Why not?

If we were writing this code to be included in Emacs core, we'd have
access to all of the symbols.

_if_ the module author wants to be somewhat isolated from Emacs changes,
and/or support more than one Emacs version, then they will want to stick
to some stable subset. But I don't think we can define that subset ahead
of time, and it will certainly be a different subset for each module.

We don't have a single .el file that defines the "Emacs core elisp API
for packages"; I don't think we can define a .h file for modules either.

> All this is pretty standard practice on Windows and works well, the
> only issue for us is to decide which functions to export (unless we
> really want to export all of them, in which case the --export-dynamic
> linker switch is all that's needed).

Let's make it simple; export all of them.

However, previous posts have identified some C functions that should
_not_ be called by modules, so perhaps we need an "module_anti_api.h"
that #defines those to throw a compilation error.

-- 
-- Stephe

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Mon, 01 Dec 2014 16:42:07 -0600 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 

SL> Ted Zlatanov <tzz@lifelogs.com> writes:

>> By API I meant both directions, the module API for registration and
>> metadata, and the Emacs API that modules can use. So I still think a
>> call-only API (only in the direction of calling the module) is best for
>> now, so that .h file is unnecessary.

SL> In my particular case, this will not be useful; I need to call some
SL> ada-mode elisp functions from a compiled parser.

SL> These elisp functions will not be in any .h file, but I will need the
SL> appropriate "funcall" in that .h file.
...
SL> My module will be setting text properties in buffers, based on the
SL> results of parsing the code; it's replacing elisp code that does that
SL> now, which is too slow.

Got it, thanks for explaining.  Yes, that kind of parser will need
tighter integration.

SL> _if_ the module author wants to be somewhat isolated from Emacs changes,
SL> and/or support more than one Emacs version, then they will want to stick
SL> to some stable subset. But I don't think we can define that subset ahead
SL> of time, and it will certainly be a different subset for each module.

From the Emacs core side, it's not fun to have to support internal
details for years because modules are using them. For instance, that
kind of deep integration is very likely to be a blocker for concurrent
threads of execution in the core.

SL> However, previous posts have identified some C functions that should
SL> _not_ be called by modules, so perhaps we need an "module_anti_api.h"
SL> that #defines those to throw a compilation error.

I don't think maintaining such a list is viable, but I could be wrong.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> From the Emacs core side, it's not fun to have to support internal
> details for years because modules are using them.

Indeed.  At first, I'll expect this API will get broken on a
regular basis.  Then it will stabilize such that old modules can still
be used in slightly more recent Emacsen.  But I don't expect to ever aim
for the same kind of backward compatibility we expect for Elisp packages.
IOW, "backward compatibility with existing dynamic modules" will be
desirable but not "top priority".

> For instance, that kind of deep integration is very likely to be
> a blocker for concurrent threads of execution in the core.

The alternative (which currently looks more reasonable) is that the
introduction of concurrency will simply break many/most existing
dynamic modules.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> for packages"; I don't think we can define a .h file for modules either.

We will need a .h file just to compile those modules (we want to be able
to compile those modules from a compiled version of Emacs, i.e. in the
absence of the complete C source of Emacs).


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Mon, 01 Dec 2014 16:58:21 -0600
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >
> >> >   . It seems to me that the modules call functions implemented by
> >> >     Emacs, like make_number and Fmember, on the assumption that
> >> >     calling any Emacs function will "just work".  This is false for
> >> 
> >> I had to add a linker flag to expose every symbol of Emacs. See the
> >> relevant commit:
> >> 
> >> http://git.savannah.gnu.org/cgit/emacs.git/commit/configure.ac?h=dynamic-modules&id=5c710fba15e0a3a2ae5d831e5cdb555332238752
> >
> > I don't think this is correct: we don't really want to export all the
> > symbols.
> 
> Why not?

Security: you don't want to expose all of the Emacs bowels to any
external program out there.

Or ask yourself why the latest GCC and Binutils default to not export
anything, contrary to what they did in older versions.

> If we were writing this code to be included in Emacs core, we'd have
> access to all of the symbols.

You can have access to symbols via specific protocols without
exporting everything.  And I'm not sure you really do need access to
everything.

> _if_ the module author wants to be somewhat isolated from Emacs changes,
> and/or support more than one Emacs version, then they will want to stick
> to some stable subset. But I don't think we can define that subset ahead
> of time, and it will certainly be a different subset for each module.
> 
> We don't have a single .el file that defines the "Emacs core elisp API
> for packages"; I don't think we can define a .h file for modules either.

You are just re-iterating my doubts about usability and wisdom in this
feature.

> > All this is pretty standard practice on Windows and works well, the
> > only issue for us is to decide which functions to export (unless we
> > really want to export all of them, in which case the --export-dynamic
> > linker switch is all that's needed).
> 
> Let's make it simple; export all of them.

The other alternative is also simple; see GNU Make for an example.

Re: emacs-dynamic-module in Emacs Git?

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Or ask yourself why the latest GCC and Binutils default to not export
> anything, contrary to what they did in older versions.

Is this through for ELF as well?

Back when I was messing around with dynamic linking, ELF didn't have a
mechanism for hiding symbols (which could be a major pain if you had the
same symbols provided by two separate DLLs... especially when the same
symbols were different things, as eg. the expat XML parser compiled with
UTF-8 API vs. expat compiled UTF-16 API).

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Tue, 02 Dec 2014 18:58:56 +0100
> 
> >>>>> Eli Zaretskii <eliz@gnu.org>:
> 
> > Or ask yourself why the latest GCC and Binutils default to not export
> > anything, contrary to what they did in older versions.
> 
> Is this through for ELF as well?

I have a very limited knowledge and experience about ELF, but here's
what the ld manual says in a section that is not specific to
MS-Windows:

  `-E'
  `--export-dynamic'
       When creating a dynamically linked executable, add all symbols to
       the dynamic symbol table.  The dynamic symbol table is the set of
       symbols which are visible from dynamic objects at run time.

       If you do not use this option, the dynamic symbol table will
       normally contain only those symbols which are referenced by some
       dynamic object mentioned in the link.

       If you use `dlopen' to load a dynamic object which needs to refer
       back to the symbols defined by the program, rather than some other
       dynamic object, then you will probably need to use this option when
       linking the program itself.

This does sound a general feature.

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

OK, dynamic-modules was rebased, squashed to a single commit, and pushed
to branch "dynamic-modules-rc1".  I added ChangeLogs and renamed
Makefile to Makefile.in in all the modules, but otherwise this is pretty
much what Aurélien wrote.

I got the error "Empty change log entry" from the commit-msg hook and
couldn't figure out why.  I disabled it to make the commit and would
appreciate a note if you (Paul?) figure it out.

If it's OK, go ahead and merge and delete the branches "dynamic-modules"
and "dynamic-modules-rc1".  Or I can do it...

Thanks
Ted

Re: emacs-dynamic-module in Emacs Git?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> for packages"; I don't think we can define a .h file for modules either.
>
> We will need a .h file just to compile those modules (we want to be able
> to compile those modules from a compiled version of Emacs, i.e. in the
> absence of the complete C source of Emacs).

That's not a requirement for me as a package developer, nor if the
modules are distributed as binary.

But if we want to distribute the modules as source, to be compiled on
each user's machine (as elisp code in ELPA packages is now), then we
need that .h file, yes.

I guess we'll find out if that works when someone tries it.

-- 
-- Stephe

Re: emacs-dynamic-module in Emacs Git?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Mon, 01 Dec 2014 16:58:21 -0600
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >
>> >> >   . It seems to me that the modules call functions implemented by
>> >> >     Emacs, like make_number and Fmember, on the assumption that
>> >> >     calling any Emacs function will "just work".  This is false for
>> >> 
>> >> I had to add a linker flag to expose every symbol of Emacs. See the
>> >> relevant commit:
>> >> 
>> >> http://git.savannah.gnu.org/cgit/emacs.git/commit/configure.ac?h=dynamic-modules&id=5c710fba15e0a3a2ae5d831e5cdb555332238752
>> >
>> > I don't think this is correct: we don't really want to export all the
>> > symbols.
>> 
>> Why not?
>
> Security: you don't want to expose all of the Emacs bowels to any
> external program out there.

There are many other aspects to security; I doubt this particular
strategy will really help.

There are better ways to prevent bad code getting into Emacs; code
reviewed signed modules is probably the best way. That's essentially how
we currently prevent bad code in Emacs core.

Hiding a function that a module turns out to need just inhibits
functionality; it does not gain security.

I'm advocating allowing any code that could be in Emacs core to instead
by in a dynamic module, to allow separate development subject to the
same restrictions as Emacs core code - FSF copyright, code review.

Obviously, the ability to load dynamic modules allows users to choose
other modules that do not meet those criteria. But that should not
restrict what we can do in a dynamic module. We are _not_ building a
sandbox, but a powerful development environment; people must be allowed
to shoot themselves in the foot.

> Or ask yourself why the latest GCC and Binutils default to not export
> anything, contrary to what they did in older versions.

I would guess that has more to do with namespace control, but I'd have
to read the rationale to be sure.

Since we are talking about code that is intended to be tightly
integrated with Emacs, we _want_ the Emacs namespace to be visible.

>> If we were writing this code to be included in Emacs core, we'd have
>> access to all of the symbols.
>
> You can have access to symbols via specific protocols without
> exporting everything.  And I'm not sure you really do need access to
> everything.

Protocols tend to get in the way. If a pipe interface was viable, I'd
use it. But it's not; I need direct, tight integration in order to be
fast enough.

I am very sure I don't need access to absolutely every symbol in the
Emacs namespace. But it's not worth the time to try to figure out ahead
of time which ones I might need. I can certainly provide a list of what
was used after I've got a first version working.

Stefan's approach makes sense; try to define an API, but assume it will
change/evolve. I'm simply arguing that it will not be worth the effort.
The only way to find out is to try it.

Stefan's point about maintaining code that was intended to be internal
to some core package because some other package happens to use it is
also valid. That's why I use Ada instead of C - it's much easier to
enforce such visibility rules. 

>> _if_ the module author wants to be somewhat isolated from Emacs changes,
>> and/or support more than one Emacs version, then they will want to stick
>> to some stable subset. But I don't think we can define that subset ahead
>> of time, and it will certainly be a different subset for each module.
>> 
>> We don't have a single .el file that defines the "Emacs core elisp API
>> for packages"; I don't think we can define a .h file for modules either.
>
> You are just re-iterating my doubts about usability and wisdom in this
> feature.

I don't understand.

You say it would be ok to add this code to core Emacs; all of the
statements above would apply to that choice as well.

We are talking about a dynamically linked module in a separate source
repository, as compared to a staticly linked one in the core repository.
Why should that choice affect the choice of the namespace that is
visible to the module?

>> > All this is pretty standard practice on Windows and works well, the
>> > only issue for us is to decide which functions to export (unless we
>> > really want to export all of them, in which case the --export-dynamic
>> > linker switch is all that's needed).
>> 
>> Let's make it simple; export all of them.
>
> The other alternative is also simple; see GNU Make for an example.

By "the other alternative" I assume you mean "define an Emacs module
API"; that's _not_ simple. Proof: no one has done it yet, but "export
all of them" has been done. QED.


Note that there are actually two namespaces we are talking about here;
the compile time namespace, determined by .h files, and the link time
namespace, determined by --export-dynamic and/or link libraries.

The future maintenance issue is best addressed via .h files; don't put
functions you don't want to support in future versions in any .h file,
or have a naming convention that clearly indicates which .h files will
be supported.

I don't see any reason to restrict the link time namespace.

-- 
-- Stephe

Re: emacs-dynamic-module in Emacs Git?

[David Kastrup]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>>> Date: Mon, 01 Dec 2014 16:58:21 -0600
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> >
>>> >> >   . It seems to me that the modules call functions implemented by
>>> >> >     Emacs, like make_number and Fmember, on the assumption that
>>> >> >     calling any Emacs function will "just work".  This is false for
>>> >> 
>>> >> I had to add a linker flag to expose every symbol of Emacs. See the
>>> >> relevant commit:
>>> >> 
>>> >> http://git.savannah.gnu.org/cgit/emacs.git/commit/configure.ac?h=dynamic-modules&id=5c710fba15e0a3a2ae5d831e5cdb555332238752
>>> >
>>> > I don't think this is correct: we don't really want to export all the
>>> > symbols.
>>> 
>>> Why not?
>>
>> Security: you don't want to expose all of the Emacs bowels to any
>> external program out there.
>
> There are many other aspects to security; I doubt this particular
> strategy will really help.
>
> There are better ways to prevent bad code getting into Emacs; code
> reviewed signed modules is probably the best way.

That does not help against things like buffer overrun exploits, and when
some malicious code has all the symbols available, it can be made to
work on a larger variety of binaries.

-- 
David Kastrup

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> I guess we'll find out if that works when someone tries it.

We also need this .h file in order to document the API; both for module
writers (so they know what they can use) and for Emacs maintainers (so
they know what they should try to preserve when making changes).


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Wed, 03 Dec 2014 03:27:41 -0600
> 
> > We will need a .h file just to compile those modules (we want to be able
> > to compile those modules from a compiled version of Emacs, i.e. in the
> > absence of the complete C source of Emacs).
> 
> That's not a requirement for me as a package developer, nor if the
> modules are distributed as binary.
> 
> But if we want to distribute the modules as source, to be compiled on
> each user's machine (as elisp code in ELPA packages is now), then we
> need that .h file, yes.

We want to allow developers of modules to compile their modules
without full access to the Emacs sources, and that requires a header
file.

Re: emacs-dynamic-module in Emacs Git?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Wed, 03 Dec 2014 04:04:40 -0600
> 
> >> > I don't think this is correct: we don't really want to export all the
> >> > symbols.
> >> 
> >> Why not?
> >
> > Security: you don't want to expose all of the Emacs bowels to any
> > external program out there.
> 
> There are many other aspects to security; I doubt this particular
> strategy will really help.
> 
> There are better ways to prevent bad code getting into Emacs; code
> reviewed signed modules is probably the best way.

See David's response, with which I fully agree.  I'm sure you had your
share of vulnerabilities exploited by bugs if not by malicious
software, and you know very well the dangers of such over-exposure.

> That's essentially how we currently prevent bad code in Emacs core.

How can we code-review modules that are not bundled?  We have no
control on those whatsoever.

> I'm advocating allowing any code that could be in Emacs core to instead
> by in a dynamic module, to allow separate development subject to the
> same restrictions as Emacs core code - FSF copyright, code review.
>
> Obviously, the ability to load dynamic modules allows users to choose
> other modules that do not meet those criteria. But that should not
> restrict what we can do in a dynamic module. We are _not_ building a
> sandbox, but a powerful development environment; people must be allowed
> to shoot themselves in the foot.

I'm okay with allowing people to shoot themselves, but I'm not okay
with letting them shoot Emacs.

What you suggest is a very slippery slope.  If we agree to such an
unlimited exposure of internals, I'm quite sure that before long we'll
have modules all over the place depending on those internals, and
their authors will apply pressure not to change the internals on which
they happen to depend.  I don't think we want Emacs development to
become hostage to every package out there.  No other project I know
of, not even libraries (whose proclaimed raison d'être is to expose
APIs) do that, and for very good reasons.  I see no reason why Emacs,
of all the packages, should do what no other one does.

> > Or ask yourself why the latest GCC and Binutils default to not export
> > anything, contrary to what they did in older versions.
> 
> I would guess that has more to do with namespace control, but I'd have
> to read the rationale to be sure.

Yes, that too.  Won't there be problems in that department as well,
e.g., if some library function replaced by Emacs clashes with its
namesake in the module?  With toy modules, this is not a problem, but
what about those that will use large libraries, where it's not so easy
to rename a function?  Again, very slippery slope.

> Since we are talking about code that is intended to be tightly
> integrated with Emacs, we _want_ the Emacs namespace to be visible.

I'm not sure who is "we" here.  I don't think I've heard Stefan, or
anyone else expressing such far-reaching desires.

> >> If we were writing this code to be included in Emacs core, we'd have
> >> access to all of the symbols.
> >
> > You can have access to symbols via specific protocols without
> > exporting everything.  And I'm not sure you really do need access to
> > everything.
> 
> Protocols tend to get in the way. If a pipe interface was viable, I'd
> use it. But it's not; I need direct, tight integration in order to be
> fast enough.

A software API is much faster than a pipe, so I don't see how this
comparison is useful, or even valid.

> I am very sure I don't need access to absolutely every symbol in the
> Emacs namespace. But it's not worth the time to try to figure out ahead
> of time which ones I might need. I can certainly provide a list of what
> was used after I've got a first version working.
> 
> Stefan's approach makes sense; try to define an API, but assume it will
> change/evolve. I'm simply arguing that it will not be worth the effort.
> The only way to find out is to try it.

Trying is fine, but it's a two-way street: expect the maintainers to
resist adding some of the symbols to your list.  There will be
negotiations, and at least I will object to granting access to
everything, which I consider insane in the long run.  Every interface
and every bit of internals we expose should be scrutinized to
determine whether exposing it is a good idea, how necessary that is,
what are its chances to change without notice, etc.

> >> _if_ the module author wants to be somewhat isolated from Emacs changes,
> >> and/or support more than one Emacs version, then they will want to stick
> >> to some stable subset. But I don't think we can define that subset ahead
> >> of time, and it will certainly be a different subset for each module.
> >> 
> >> We don't have a single .el file that defines the "Emacs core elisp API
> >> for packages"; I don't think we can define a .h file for modules either.
> >
> > You are just re-iterating my doubts about usability and wisdom in this
> > feature.
> 
> I don't understand.
> 
> You say it would be ok to add this code to core Emacs; all of the
> statements above would apply to that choice as well.

When code is added to the core, it is automatically updated when the
internals change.  That's the huge difference you are overlooking.

> We are talking about a dynamically linked module in a separate source
> repository, as compared to a staticly linked one in the core repository.
> Why should that choice affect the choice of the namespace that is
> visible to the module?

I hope by now you understand my answer to that question.

> >> Let's make it simple; export all of them.
> >
> > The other alternative is also simple; see GNU Make for an example.
> 
> By "the other alternative" I assume you mean "define an Emacs module
> API"

Not necessarily.  It should be enough to make a list of interfaces to
which we allow external linkage.

> that's _not_ simple. Proof: no one has done it yet, but "export
> all of them" has been done. QED.

:-)

Do take a look at GNU Make, though.  It might be useful to put our
case in context and in its true proportions.

> Note that there are actually two namespaces we are talking about here;
> the compile time namespace, determined by .h files, and the link time
> namespace, determined by --export-dynamic and/or link libraries.
> 
> The future maintenance issue is best addressed via .h files; don't put
> functions you don't want to support in future versions in any .h file,
> or have a naming convention that clearly indicates which .h files will
> be supported.

If we make the effort to create the header file, restricting the
linkage only to those interfaces is a trivial job.

> I don't see any reason to restrict the link time namespace.

It makes no sense whatsoever to allow linkage to interfaces that are
not declared in the header file.  Do you expect package authors to
reverse-engineer or disassemble Emacs in order to find that info?

Re: emacs-dynamic-module in Emacs Git?

[Stefan Monnier]

> What you suggest is a very slippery slope.  If we agree to such an
> unlimited exposure of internals, I'm quite sure that before long we'll

Luckily we don't agree with such unlimited exposure.  It's not up
for debate.  The only thing still up for debate is what should be
included in the API.


        Stefan

Re: emacs-dynamic-module in Emacs Git?

[Stephen Leake]

David Kastrup <dak@gnu.org> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> > I don't think this is correct: we don't really want to export all the
>>>> > symbols.
>>>> 
>>>> Why not?
>>>
>>> Security: you don't want to expose all of the Emacs bowels to any
>>> external program out there.
>>
>> There are many other aspects to security; I doubt this particular
>> strategy will really help.
>>
>> There are better ways to prevent bad code getting into Emacs; code
>> reviewed signed modules is probably the best way.
>
> That does not help against things like buffer overrun exploits, 

If someone can get a buffer overrun exploit past an Emacs developer code
review, then they can get it in Emacs core, so we are already vulnerable
to that.

Code reviewed dynamically linked modules do not change that risk.

-- 
-- Stephe

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Tue, Dec 2, 2014 at 10:22 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> OK, dynamic-modules was rebased, squashed to a single commit, and pushed
> to branch "dynamic-modules-rc1". I added ChangeLogs and renamed
> Makefile to Makefile.in in all the modules, but otherwise this is pretty
> much what Aurélien wrote.

You forgot to include the opaque module in SUBDIR_MAKEFILES.

I'm currently on a different computer than usual and although loading
works, calling any external symbol from a lib linked against the module
crashes emacs (similar error with yaml).

./src/emacs: symbol lookup error:
/home/aaptel/dev/emacs/modules/curl/curl.so: undefined symbol:
curl_global_init

Strangely enough the module file doesn't show any curl dependency with ldd,
although it was compiled with -lcurl.

$ ldd curl.so
    linux-gate.so.1 =>  (0xb7757000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb757f000)
    /lib/ld-linux.so.2 (0xb7758000)

I don't know what's wrong (probably comes from my system).

[-- Attachment #2: Type: text/html, Size: 1266 bytes --]

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Thu, 4 Dec 2014 21:40:34 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Tue, Dec 2, 2014 at 10:22 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> OK, dynamic-modules was rebased, squashed to a single commit, and pushed
>> to branch "dynamic-modules-rc1". I added ChangeLogs and renamed
>> Makefile to Makefile.in in all the modules, but otherwise this is pretty
>> much what Aurélien wrote.

AA> You forgot to include the opaque module in SUBDIR_MAKEFILES.

I pushed with that minor change to dynamic-modules-rc2

AA> I'm currently on a different computer than usual and although loading
AA> works, calling any external symbol from a lib linked against the module
AA> crashes emacs (similar error with yaml).

AA> ./src/emacs: symbol lookup error:
AA> /home/aaptel/dev/emacs/modules/curl/curl.so: undefined symbol:
AA> curl_global_init

AA> Strangely enough the module file doesn't show any curl dependency with ldd,
AA> although it was compiled with -lcurl.

AA> $ ldd curl.so
AA>     linux-gate.so.1 =>  (0xb7757000)
AA>     libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb757f000)
AA>     /lib/ld-linux.so.2 (0xb7758000)

AA> I don't know what's wrong (probably comes from my system).

Weird.  Anyone else tested this?

I'll hold off merging for 1 more day...

Ted

Re: emacs-dynamic-module in Emacs Git?

[Ivan Andrus]

On Dec 4, 2014, at 6:02 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
On Thu, 4 Dec 2014 21:40:34 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 
> 
> AA> On Tue, Dec 2, 2014 at 10:22 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> OK, dynamic-modules was rebased, squashed to a single commit, and pushed
>>> to branch "dynamic-modules-rc1". I added ChangeLogs and renamed
>>> Makefile to Makefile.in in all the modules, but otherwise this is pretty
>>> much what Aurélien wrote.
> 
> AA> You forgot to include the opaque module in SUBDIR_MAKEFILES.
> 
> I pushed with that minor change to dynamic-modules-rc2
> 
> AA> I'm currently on a different computer than usual and although loading
> AA> works, calling any external symbol from a lib linked against the module
> AA> crashes emacs (similar error with yaml).
> 
> AA> ./src/emacs: symbol lookup error:
> AA> /home/aaptel/dev/emacs/modules/curl/curl.so: undefined symbol:
> AA> curl_global_init
> 
> AA> Strangely enough the module file doesn't show any curl dependency with ldd,
> AA> although it was compiled with -lcurl.
> 
> AA> $ ldd curl.so
> AA>     linux-gate.so.1 =>  (0xb7757000)
> AA>     libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb757f000)
> AA>     /lib/ld-linux.so.2 (0xb7758000)
> 
> AA> I don't know what's wrong (probably comes from my system).
> 
> Weird.  Anyone else tested this?
> 
> I'll hold off merging for 1 more day…

I wanted to, but couldn’t get it to compile on my Mac.  Though I must admit I didn’t
try very hard.  I should also say that “gcc” is actually clang:

$ make
gcc -ggdb3 -Wall -I../../src -I../../lib -fPIC -c elisp.c
gcc -shared -o elisp.so elisp.o
Undefined symbols for architecture x86_64:
  "_Fprovide", referenced from:
      _init in elisp.o
  "_Qnil", referenced from:
      _init in elisp.o
  "_call3", referenced from:
      _Felisp_test in elisp.o
  "_defsubr", referenced from:
      _init in elisp.o
  "_intern_c_string", referenced from:
      _init in elisp.o
  "_make_string", referenced from:
      _Felisp_test in elisp.o
  "_staticpro", referenced from:
      _init in elisp.o
ld: symbol(s) not found for architecture x86_64

-Ivan

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Thu, 04 Dec 2014 20:02:02 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I pushed with that minor change to dynamic-modules-rc2

Strangely, I get this error:

Symbol's function definition is void: defun

which seems like a "make maintainer-clean; make bootstrap" would fix it, but it doesn't.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Ted Zlatanov]

On Thu, 04 Dec 2014 20:02:02 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I pushed with that minor change to dynamic-modules-rc2
...
TZ> I'll hold off merging for 1 more day...

I am still getting an error on that branch so I'm not merging until
Aurélien or someone else takes a look.  Sorry for the delay, I couldn't
find the issue.

Ted

Re: emacs-dynamic-module in Emacs Git?

[Aurélien Aptel]

On Wed, Dec 10, 2014 at 1:53 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Thu, 04 Dec 2014 20:02:02 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote:
>
> TZ> I pushed with that minor change to dynamic-modules-rc2
>
> Strangely, I get this error:
>
> Symbol's function definition is void: defun
>
> which seems like a "make maintainer-clean; make bootstrap" would fix it, but it doesn't.

Thanks for all this testing and attention!
Unfortunately I don't have a lot of free time to work on this right
now, I'll take a look when I can.