bug#7487: 24.0.50; Gnus nnimap broken

bug#7487: 24.0.50; Gnus nnimap broken

[Jason Rumney]

I updated from bzr for the first time in a few months today, and
immediately ran into problems with Gnus.

My Gnus related variables are:

 '(gnus-select-method (quote (nnimap "localhost")))
 '(gnus-sieve-crosspost nil)
 '(gnus-sieve-file "~/.sieve.d/Gnus.sieve")

everything else is set to defaults.

Upon starting Gnus, some of my groups are showing unread articles in the
summary buffer, including groups which I only manually move articles
into, and which I have not touched for months.  After entering these
groups, I see some old articles that I have ticked, and some lines
saying that an article has not been downloaded (I have fixed the problem
now, and I did not have the foresight to note down the exact text).  It
seems gnus-agent (which I never explicitly enabled, but have had problems
with in past upgrades after it was enabled by default) is out of sync
with the server, and deleting the ~/News/agent directory fixed the
problem.

Now when I start Gnus, I get the message 

nnimap (localhost) open error: 'Unable to contact localhost:imaps via
ssl'.  Continue?

I answer 'y', expecting Gnus to continue using imap, as I configured it,
but get the following messages:

Reading /home/jasonr/.newsrc.eld...
Opening nnimap server on localhost...
Denied server nnimap+localhost
Opening nnimap server on localhost...failed: Unable to contact
localhost:imaps via ssl

This is a terrible user experience.  Trying SSL first is a good feature,
but if it fails, Gnus should fall back to using plain imap as the user
has configured.  It isn't even obvious to me what variable I have to set
to access plain imap anymore.

Following this, I configured Gnus to use imaps (which I have configured on a
non-standard port to get through certain firewalls but never bothered to
use before on localhost), and now run into a third problem.  Upon
logging in, I am prompted to add my login credentials to
~/.authinfo.gpg.  This brings up a confusing buffer asking me to mark a
public key, which I have discovered by trial and error only works if I
select no public keys so it falls back on symetric encryption.  After
entering an encryption password twice, I was able to read my mail at
last.

But my problems aren't over yet. When I quit Gnus and start it again,
my encryption password for authinfo.gpg doesn't appear to work.
Actually, it is working, because if I enter a different password, it
fails immediately, whereas if I enter the correct password it loops
forever asking for my password (maybe not forever, but my patience for
finding out is limited).

Using C-x C-f to find ~/.authinfo.gpg works after entering the password
once, the the problem seems to be particular to the way Gnus is opening
the file, not a general problem with file decryption.

The messages from this failure are below.




In GNU Emacs 24.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.22.0)
 of 2010-11-25 on wanchan
Windowing system distributor `The X.Org Foundation', version 11.0.10900000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_NZ.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  show-paren-mode: t
  display-time-mode: t
  cua-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <menu-bar> <help-menu> <send-emacs-bug
-report>

Recent messages:
/home/jasonr/.authin: 100% (96/96)
/home/jasonr/.authin: 0% (0/96)
/home/jasonr/.authin: 100% (96/96)
/home/jasonr/.authin: 0% (0/96)
/home/jasonr/.authin: 100% (96/96)
/home/jasonr/.authin: 0% (0/96)
Unable to open server nnimap+localhost due to: Opening input file: Can't decrypt, ((exit) (quit))
nnimap (localhost) open error: ''.  Continue?  n
Couldn't open server on localhost
Warning: Unable to open server nnimap+localhost due to: Opening input file: Can't decrypt, ((exit) (quit))

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug epa-file epa derived epg auth-source
nnimap parse-time tls utf7 netrc gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 epg-config mm-view
smime password-cache dig mailcap nntp gnus-cache gnus-sum nnoo
gnus-group gnus-undo nnmail mail-source format-spec gnus-start gnus-spec
gnus-int gnus-range message sendmail rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums
mailabbrev gmm-utils mailheader gnus-win jka-compr org-mouse org-w3m
org-jsinfo org-infojs org-html org-exp ob-exp org-exp-blocks org-agenda
org-info org-id org-gnus org-bbdb paren gnus gnus-ems nnheader gnus-util
mail-utils mm-util mail-prsvr wid-edit time cua-base cus-start cus-load
remember org-remember org-datetree org byte-opt warnings bytecomp
byte-compile advice help-fns advice-preload ob-emacs-lisp ob-tangle
ob-ref ob-lob ob-table org-footnote org-src ob-comint comint ring
ob-keys ob org-list org-faces org-compat org-entities org-macs time-date
noutline outline easy-mmode regexp-opt cal-menu easymenu calendar
cal-loaddefs server tooltip ediff-hook vc-hooks lisp-float-type mwheel
x-win x-dnd tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Jason Rumney <jasonr@gnu.org> writes:

> Upon starting Gnus, some of my groups are showing unread articles in the
> summary buffer, including groups which I only manually move articles
> into, and which I have not touched for months. 

[...]

> It seems gnus-agent (which I never explicitly enabled, but have had
> problems with in past upgrades after it was enabled by default) is out
> of sync with the server, and deleting the ~/News/agent directory fixed
> the problem.

Gnus has stopped enabling the Agent by default now, but older
installations can still get in the way.  I've Cc'd this to the Gnus
mailing list -- do any of you know why the Agent got so wildly out of
what with nnimap in the past?

> This is a terrible user experience.  Trying SSL first is a good feature,
> but if it fails, Gnus should fall back to using plain imap as the user
> has configured.  It isn't even obvious to me what variable I have to set
> to access plain imap anymore.

We've discussed the connection methodology on the list in the past, but
it seems I punted on actually implementing this.  I've now changed the
(default) to try imaps first, and if it fails, imap.

If you wish to force a specific protocol, then `nnimap-stream' is the
backend slot to customize.

> Following this, I configured Gnus to use imaps (which I have configured on a
> non-standard port to get through certain firewalls but never bothered to
> use before on localhost), and now run into a third problem.

I don't follow you here.  How did you configure Gnus to use imaps?  And
since that's the default, isn't that what failed?

> Upon logging in, I am prompted to add my login credentials to
> ~/.authinfo.gpg.  This brings up a confusing buffer asking me to mark
> a public key, which I have discovered by trial and error only works if
> I select no public keys so it falls back on symetric encryption.

Yes, that interface is so confusing it's ridiculous.

Ted is rewriting the way Emacs asks for and stores user credentials.
It'll support using the Secrets API in addition to the standard Emacs
.gpg file handling, but the .gpg interface is so wildly user hostile
that I'm tempted to go back to just storing this data in the plain-text
~/.authinfo file until all this has been worked out.

> After entering an encryption password twice, I was able to read my
> mail at last.

Yay!  :-)

> But my problems aren't over yet. When I quit Gnus and start it again,
> my encryption password for authinfo.gpg doesn't appear to work.
> Actually, it is working, because if I enter a different password, it
> fails immediately, whereas if I enter the correct password it loops
> forever asking for my password (maybe not forever, but my patience for
> finding out is limited).

Weird.  Could you

(setq debug-on-quit t)

and then `C-g' when it starts asking you again and again?  Post the
backtrace here.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

This report was handled in a different forum, and debbugs wasn't Cc'd,
apparently.  So I'm just closing this report.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Ted Zlatanov]

On Sun, 28 Nov 2010 03:56:21 +0100 Lars Magne Ingebrigtsen <lmi@gnus.org> wrote: 

LMI> Jason Rumney <jasonr@gnu.org> writes:
>> Upon logging in, I am prompted to add my login credentials to
>> ~/.authinfo.gpg.  This brings up a confusing buffer asking me to mark
>> a public key, which I have discovered by trial and error only works if
>> I select no public keys so it falls back on symetric encryption.

LMI> Yes, that interface is so confusing it's ridiculous.

I also had trouble with it the first time.  Perhaps there should be more
information on that screen.

LMI> Ted is rewriting the way Emacs asks for and stores user credentials.
LMI> It'll support using the Secrets API in addition to the standard Emacs
LMI> .gpg file handling, but the .gpg interface is so wildly user hostile
LMI> that I'm tempted to go back to just storing this data in the plain-text
LMI> ~/.authinfo file until all this has been worked out.

Yeah, OTOH once it's set up, the .gpg file Just Works.  So for me the
pain has faded away.

On Sun, 05 Dec 2010 17:49:52 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> This report was handled in a different forum, and debbugs wasn't Cc'd,
LMI> apparently.  So I'm just closing this report.

Hey, where do I go to track it?

Thanks
Ted

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

>>> Upon logging in, I am prompted to add my login credentials to
>>> ~/.authinfo.gpg.  This brings up a confusing buffer asking me to mark
>>> a public key, which I have discovered by trial and error only works if
>>> I select no public keys so it falls back on symetric encryption.
>
> LMI> Yes, that interface is so confusing it's ridiculous.
>
> I also had trouble with it the first time.  Perhaps there should be more
> information on that screen.

This is what the user is faced with:

----
Select recipents for encryption.
If no one is selected, symmetric encryption will be performed.  
- `m' to mark a key on the line
- `u' to unmark a key on the line
[Cancel][OK]

e <person I somehow have a certificate for>
----

And this is for storing a ~/.authinfo.gpg file.  If there's one thing
the user don't want to do, it's "select recipient for encryption".
And how many users know what "symmetric encryption" is anyway?

When writing the ~/.authinfo.gpg file, the user should be queried one
thing: "Password for ~/.authinfo.gpg: ****".  And that's it, in my
extremely humble opinion.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

bug#7487: 24.0.50; Gnus nnimap broken

[Jason Rumney]

Lars Magne Ingebrigtsen <lmi@gnus.org> writes:

>> It seems gnus-agent (which I never explicitly enabled, but have had
>> problems with in past upgrades after it was enabled by default) is out
>> of sync with the server, and deleting the ~/News/agent directory fixed
>> the problem.
>
> Gnus has stopped enabling the Agent by default now, but older
> installations can still get in the way.  I've Cc'd this to the Gnus
> mailing list -- do any of you know why the Agent got so wildly out of
> what with nnimap in the past?

In earlier versions, Agent did not work well when other mail clients
were also reading and moving mail around on the same imap account.  I
think that has not been a problem since Emacs 23. This time, I think the
sync problems happened in mail groups that had not been entered by Gnus
on this machine recently, but had had new mail read on other machines.
My guess is that Agent only updates its indexes when you enter the
group, so it can get out of sync in cases where mail is read elsewhere.

>> Following this, I configured Gnus to use imaps (which I have configured on a
>> non-standard port to get through certain firewalls but never bothered to
>> use before on localhost), and now run into a third problem.
>
> I don't follow you here.  How did you configure Gnus to use imaps?  And
> since that's the default, isn't that what failed?

I mean imaps as a protocol (SSL enabled imap), not as a port name. As I
mentioned, I am running this on a non-standard port in order to get it
through a particular firewall.

>> But my problems aren't over yet. When I quit Gnus and start it again,
>> my encryption password for authinfo.gpg doesn't appear to work.
>> Actually, it is working, because if I enter a different password, it
>> fails immediately, whereas if I enter the correct password it loops
>> forever asking for my password (maybe not forever, but my patience for
>> finding out is limited).
>
> Weird.  Could you
>
> (setq debug-on-quit t)
>
> and then `C-g' when it starts asking you again and again?  Post the
> backtrace here.

Will do. I suspect this problem may also be related to the non-standard
port.

bug#7487: 24.0.50; Gnus nnimap broken

[Jason Rumney]

Lars Magne Ingebrigtsen <lmi@gnus.org> writes:

>> But my problems aren't over yet. When I quit Gnus and start it again,
>> my encryption password for authinfo.gpg doesn't appear to work.
>> Actually, it is working, because if I enter a different password, it
>> fails immediately, whereas if I enter the correct password it loops
>> forever asking for my password (maybe not forever, but my patience for
>> finding out is limited).
>
> Weird.  Could you
>
> (setq debug-on-quit t)
>
> and then `C-g' when it starts asking you again and again?  Post the
> backtrace here.

This seems to be fixed now.

bug#7487: 24.0.50; Gnus nnimap broken

[Stefan Monnier]

>> It seems gnus-agent (which I never explicitly enabled, but have had
>> problems with in past upgrades after it was enabled by default) is out
>> of sync with the server, and deleting the ~/News/agent directory fixed
>> the problem.

> Gnus has stopped enabling the Agent by default now, but older
> installations can still get in the way.  I've Cc'd this to the Gnus
> mailing list -- do any of you know why the Agent got so wildly out of
> what with nnimap in the past?

I need the Agent with nnimap.  And I think Gnus should enable the Agent
everywhere by default nowadays.  Most/all other MUAs do.

> that I'm tempted to go back to just storing this data in the plain-text
> ~/.authinfo file until all this has been worked out.

No!!!! Or only after prompting the user five times for
(different) confirmation.

> When writing the ~/.authinfo.gpg file, the user should be queried one
> thing: "Password for ~/.authinfo.gpg: ****".  And that's it, in my
> extremely humble opinion.

I partly agree, though some users won't have a key-pair setup, others
will have several, so the right thing to do may be either to use
symmetric encryption, or to guess which key-pair to use, and since it's
a guess there needs to be a way for the user to override the guess.


        Stefan

bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> that I'm tempted to go back to just storing this data in the plain-text
>> ~/.authinfo file until all this has been worked out.
>
> No!!!! Or only after prompting the user five times for
> (different) confirmation.

If you look at other widely used software packages, like Firefox, they
default to just storing the passwords in an (obfuscated) non-encrypted
file.  I don't think that's such a bad default.

>> When writing the ~/.authinfo.gpg file, the user should be queried one
>> thing: "Password for ~/.authinfo.gpg: ****".  And that's it, in my
>> extremely humble opinion.
>
> I partly agree, though some users won't have a key-pair setup, others
> will have several, so the right thing to do may be either to use
> symmetric encryption, or to guess which key-pair to use, and since it's
> a guess there needs to be a way for the user to override the guess.

Again, if you look at what the user experience is with, say, Firefox --
if you have password encryption turned on, then Firefox will prompt you
for the password to unlock your credential storage.  This is intuitive
and works well.

If you want a more complicated credential storage setup, then that
should be a user option, not a default.  At present, the ~/.authinfo.gpg
credential storage is not something you can present to a normal user and
expect them to understand at all.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Jason Rumney <jasonr@gnu.org> writes:

> My guess is that Agent only updates its indexes when you enter the
> group, so it can get out of sync in cases where mail is read elsewhere.

The Agent should ideally just "extend" the article storage, but
otherwise be invisible to the user.  I think that, at the very least,
the Agent storage for a particular group should be nuked if the
UIDVALIDITY of the IMAP mailbox changes, but I'm not sure that's
sufficient.  And it's not been easy to find out what the failure modes
actually are, because when people get problems in this area, they ask
for advice, and are told "rm -r", and things just work.  But it doesn't
help debugging any.  :-/

>>> Following this, I configured Gnus to use imaps (which I have configured on a
>>> non-standard port to get through certain firewalls but never bothered to
>>> use before on localhost), and now run into a third problem.
>>
>> I don't follow you here.  How did you configure Gnus to use imaps?  And
>> since that's the default, isn't that what failed?
>
> I mean imaps as a protocol (SSL enabled imap), not as a port name. As I
> mentioned, I am running this on a non-standard port in order to get it
> through a particular firewall.

Ah, right.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

bug#7487: 24.0.50; Gnus nnimap broken

[Stefan Monnier]

> The Agent should ideally just "extend" the article storage, but
> otherwise be invisible to the user.  I think that, at the very least,
> the Agent storage for a particular group should be nuked if the
> UIDVALIDITY of the IMAP mailbox changes, but I'm not sure that's
> sufficient.  And it's not been easy to find out what the failure modes
> actually are, because when people get problems in this area, they ask
> for advice, and are told "rm -r", and things just work.  But it doesn't
> help debugging any.  :-/

I'd be happy to help debug it: I use the Agent and I access my imap
account from multiple machines.  And since the nnimap rewrite I feel
like I keep reading the same email messages whenever I move from one
machine to another.


        Stefan

bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I'd be happy to help debug it: I use the Agent and I access my imap
> account from multiple machines.  And since the nnimap rewrite I feel
> like I keep reading the same email messages whenever I move from one
> machine to another.

If you could try to see what the pattern is, that would be helpful.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Ted Zlatanov]

On Thu, 09 Dec 2010 22:10:35 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> that I'm tempted to go back to just storing this data in the plain-text
>>> ~/.authinfo file until all this has been worked out.
>> 
>> No!!!! Or only after prompting the user five times for
>> (different) confirmation.

LMI> If you look at other widely used software packages, like Firefox, they
LMI> default to just storing the passwords in an (obfuscated) non-encrypted
LMI> file.  I don't think that's such a bad default.

It's a terrible default IMO.  But you knew I'd say that :)

LMI> If you want a more complicated credential storage setup, then that
LMI> should be a user option, not a default.  At present, the ~/.authinfo.gpg
LMI> credential storage is not something you can present to a normal user and
LMI> expect them to understand at all.

How about a .sgpg or .spg extension that signals EPA/EPG that only
symmetric encryption is desired?

Ted

bug#7487: 24.0.50; Gnus nnimap broken

[Stefan Monnier]

>>>> that I'm tempted to go back to just storing this data in the plain-text
>>>> ~/.authinfo file until all this has been worked out.
>>> No!!!! Or only after prompting the user five times for
>>> (different) confirmation.
LMI> If you look at other widely used software packages, like Firefox, they
LMI> default to just storing the passwords in an (obfuscated) non-encrypted
LMI> file.  I don't think that's such a bad default.
> It's a terrible default IMO.  But you knew I'd say that :)

I also find it terrible.  Tho it is at least protected by a 3-way prompt
(tho only 1 rather than 5).

LMI> If you want a more complicated credential storage setup, then that
LMI> should be a user option, not a default.  At present, the ~/.authinfo.gpg
LMI> credential storage is not something you can present to a normal user and
LMI> expect them to understand at all.
> How about a .sgpg or .spg extension that signals EPA/EPG that only
> symmetric encryption is desired?

I think that will only push the problem elsewhere, which is "which file
name to use: .authinfo.gpg or .authinfo.spg".  It seems simpler to just
let the user configure the behavior she wants.  By default just use
symmetric encryption.


        Stefan

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think that will only push the problem elsewhere, which is "which file
> name to use: .authinfo.gpg or .authinfo.spg".  It seems simpler to just
> let the user configure the behavior she wants.  By default just use
> symmetric encryption.

I think that sounds like a good solution.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Ted Zlatanov]

On Mon, 13 Dec 2010 04:19:01 +0100 Lars Magne Ingebrigtsen <lmi@gnus.org> wrote: 

LMI> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I think that will only push the problem elsewhere, which is "which file
>> name to use: .authinfo.gpg or .authinfo.spg".  It seems simpler to just
>> let the user configure the behavior she wants.  By default just use
>> symmetric encryption.

LMI> I think that sounds like a good solution.

Hmm, I haven't heard from Daiki Ueno, and he knows EPA/EPG best.  So I'd
like his opinion.  Maybe he agrees the UI is complicated and is planning
something already.

As far as auth-source is concerned, it could have backend initialization
functionality that, when a backend (e.g. a .gpg file) doesn't exist,
will run specific functions, including creating a .gpg file with
symmetric encryption.  But right now it simply lets EPA handle everything.

Ted

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> As far as auth-source is concerned, it could have backend initialization
> functionality that, when a backend (e.g. a .gpg file) doesn't exist,
> will run specific functions, including creating a .gpg file with
> symmetric encryption.  But right now it simply lets EPA handle everything.

It might make sense to have auth-source control that, but I think it
would be better if this was controlled by the user "centrally" in Emacs
somewhere.

And default to symmetric encryption, unless there's are some obvious
hints somewhere that the user doesn't want that, like if a gpg agent is
present and can be used.  (I don't really know how that works, so I'm
not sure whether that's feasible.)

I've spent the evening testing out starting Gnus as a blank slate, with
just

(setq gnus-select-method '(nnimap "imap.gmail.com"))

in .emacs, and absolutely nothing else.  I've cleaned up some
weirdnesses in that area, but now the .gpg thing is really the only
major "er, what?" thing a new user should encounter in this scenario.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

bug#7487: 24.0.50; Gnus nnimap broken

[Daiki Ueno]

Lars Magne Ingebrigtsen <lmi@gnus.org> writes:

>> As far as auth-source is concerned, it could have backend initialization
>> functionality that, when a backend (e.g. a .gpg file) doesn't exist,
>> will run specific functions, including creating a .gpg file with
>> symmetric encryption.  But right now it simply lets EPA handle everything.
>
> It might make sense to have auth-source control that, but I think it
> would be better if this was controlled by the user "centrally" in Emacs
> somewhere.

I have not yet caught up the discussion, but If you want to simply skip
the key selection, I'd suggest to do:

(make-local-variable 'epa-file-encrypt-to)
(setq epa-file-encrypt-to nil)

in the "authinfo.gpg" buffer (see epa-file-write-region).

Probably you may want to add an option to auth-source.el like:

(defcustom auth-source-gpg-encrypt-to t
  "List of recipient keys that `authinfo.gpg' encrypted to.
If the value is not a list, symmetric encryption will be used."
  ...)

and in auth-source-create(), expand the function body of netrc-store-data()
and put:

(with-temp-buffer
  (if auth-source-gpg-encrypt-to
    (make-local-variable 'epa-file-encrypt-to))
  (if (listp auth-source-gpg-encrypt-to)
    (setq epa-file-encrypt-to auth-source-gpg-encrypt-to))
  ...
  (write-region ...))

Regards,
-- 
Daiki Ueno

bug#7487: 24.0.50; Gnus nnimap broken

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

Daiki Ueno <ueno@unixuser.org> writes:

> If you want to simply skip the key selection, I'd suggest to do:
>
> (make-local-variable 'epa-file-encrypt-to)
> (setq epa-file-encrypt-to nil)
>
> in the "authinfo.gpg" buffer (see epa-file-write-region).

Find below a proposed patch to auth-source.el.

BTW, I forgot to mention in the previous response, I think it overkill
to make epa-file to encrypt files with symmetric encryption by default,
because as long as a file is visited in a buffer, epa-file remembers the
last used encryption method (in epa-file-encrypt-to local variable),
which will be used on the next save-buffer.  So a user should see the
key selection UI only the first time she saves the buffer.

Default symmetric might be useful when Emacs does visit/save/kill-buffer
repeatedly, but I think it is a rare case.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: auth-source-gpg-encrypt-to.diff --]
[-- Type: text/x-patch, Size: 2158 bytes --]

=== modified file 'lisp/gnus/auth-source.el'
--- lisp/gnus/auth-source.el	2010-10-10 22:48:40 +0000
+++ lisp/gnus/auth-source.el	2010-12-15 01:32:00 +0000
@@ -159,6 +159,15 @@
                                                      (const :tag "Any" t)
                                                      (string :tag "Specific user name"))))))))
 
+(defcustom auth-source-gpg-encrypt-to t
+  "List of recipient keys that `authinfo.gpg' encrypted to.
+If the value is not a list, symmetric encryption will be used."
+  :group 'auth-source
+  :version "23.2" ;; No Gnus
+  :type '(choice (const :tag "Symmetric encryption" t)
+		 (repeat :tag "Recipient public keys"
+			 (string :tag "Recipient public key"))))
+
 ;; temp for debugging
 ;; (unintern 'auth-source-protocols)
 ;; (unintern 'auth-sources)
@@ -352,9 +361,28 @@
       ;; netrc interface.
       (when (y-or-n-p (format "Do you want to save this password in %s? "
                               source))
-        (netrc-store-data source host prot
-                          (or user (cdr (assoc "login" result)))
-                          (cdr (assoc "password" result))))))
+	;; the code below is almost same as `netrc-store-data' except
+	;; the `epa-file-encrypt-to' hack (see bug#7487).
+	(with-temp-buffer
+	  (when (file-exists-p source)
+	    (insert-file-contents source))
+	  (when auth-source-gpg-encrypt-to
+	    ;; making `epa-file-encrypt-to' local to this buffer lets
+	    ;; epa-file skip the key selection query (see the
+	    ;; `local-variable-p' check in `epa-file-write-region').
+	    (unless (local-variable-p 'epa-file-encrypt-to)
+	      (make-local-variable 'epa-file-encrypt-to))
+	    (if (listp auth-source-gpg-encrypt-to)
+		(setq epa-file-encrypt-to auth-source-gpg-encrypt-to)))
+	  (goto-char (point-max))
+	  (unless (bolp)
+	    (insert "\n"))
+	  (insert (format "machine %s login %s password %s port %s\n"
+			  host
+			  (or user (cdr (assoc "login" result)))
+			  (cdr (assoc "password" result))
+			  prot))
+	  (write-region (point-min) (point-max) source nil 'silent)))))
     (if (consp mode)
         (mapcar #'cdr result)
       (cdar result))))


[-- Attachment #3: Type: text/plain, Size: 25 bytes --]


Regards,
-- 
Daiki Ueno

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Ted Zlatanov]

On Wed, 15 Dec 2010 11:06:42 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> BTW, I forgot to mention in the previous response, I think it overkill
DU> to make epa-file to encrypt files with symmetric encryption by default,
DU> because as long as a file is visited in a buffer, epa-file remembers the
DU> last used encryption method (in epa-file-encrypt-to local variable),
DU> which will be used on the next save-buffer.  So a user should see the
DU> key selection UI only the first time she saves the buffer.

DU> Default symmetric might be useful when Emacs does visit/save/kill-buffer
DU> repeatedly, but I think it is a rare case.

I agree.  The user may not even know the implications of symmetric
encryption.  So IMHO this should be an external option (maybe requested
by setting `epa-file-encrypt-to' to 'ask if it's nil) which would change
the current key selection UI as follows (this is a refinement of my
previous UI proposal):

1) show prompt "Do you want to encrypt %s with a passphrase (symmetric)
or with a specific key?  (p/k/P/K/h)"

2) on `h', show some help, mentioning for instance that this question is
asked only once per file and link to the EPA manual

3) on `p', proceed without the key selection UI

4) on `k', proceed with the current key selection UI

5) on `P' or `K', save the choice as a new customized default for
`epa-file-encrypt-to'

All of this shouldn't involve auth-source.el beyond setting
`epa-file-encrypt-to' to 'ask if it's nil.  That way other packages can
use this same approach.

So this inconveniences users at least once, but gives them a chance to
understand what's going on, and with `P' will not be asked again.  What
do you think?

Ted

bug#7487: 24.0.50; Gnus nnimap broken

[Daiki Ueno]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I agree.  The user may not even know the implications of symmetric
> encryption.  So IMHO this should be an external option (maybe requested
> by setting `epa-file-encrypt-to' to 'ask if it's nil) which would change
> the current key selection UI as follows (this is a refinement of my
> previous UI proposal):

Sorry, I don't want to extend `epa-file-encrypt-to' to a global variable
to accept the value such as 'ask.  It purely intends to be used in
"Local variables:" section of the file.

Perhaps you may think my previous patch is too large (and as you usually
complains there is a code duplication between netrc.el and
auth-source.el)?  I don't think so since there is only a few packages
(auth-source.el and anything else?) will benefit from the simplicity.

> So this inconveniences users at least once, but gives them a chance to
> understand what's going on, and with `P' will not be asked again.  What
> do you think?

Personally I don't like this kind of multiple-candidate question in the
mini-buffer :-) Maybe we could reuse the prefix argument (as `C-u M-x
epa-sign-region') to control whether the key selection UI is shown, or
simplify the key selection dialog by hiding the public key list by
default, with say `visibility' widget.

Regards,
-- 
Daiki Ueno

bug#7487: 24.0.50; Gnus nnimap broken

[Daiki Ueno]

Daiki Ueno <ueno@unixuser.org> writes:

> Personally I don't like this kind of multiple-candidate question in the
> mini-buffer :-) Maybe we could reuse the prefix argument (as `C-u M-x
> epa-sign-region') to control whether the key selection UI is shown, or
> simplify the key selection dialog by hiding the public key list by
> default, with say `visibility' widget.

Hmm, I recalled that why we currently don't use the prefix argument
there: epa-file-write-region() may be implicitly called from other
interactive commands like save-buffer, write-file, etc. where the prefix
argument has different meanings.

So my current suggestion is:

1. add epa-file-encrypt-to hack to auth-source.el, and
2. extend epa-file-select-keys (not epa-file-encrypt-to) so that it can
   have 'ask or 'always or 'quiet, etc., and
3. simplify the key selection UI by hiding public key list

where 1 may look unnecessary once 2 is done, but it would be better to
do both 1 and 2, as long as Gnus supports older versions of Emacs.

What do you think?  Other ideas are welcome of course :-)

Regards,
-- 
Daiki Ueno

bug#7487: 24.0.50; Gnus nnimap broken

[Daiki Ueno]

> 1. add epa-file-encrypt-to hack to auth-source.el, and
> 2. extend epa-file-select-keys (not epa-file-encrypt-to) so that it can
>    have 'ask or 'always or 'quiet, etc., and
> 3. simplify the key selection UI by hiding public key list

I did 2.  Try:

(setq epa-file-select-keys 'silent)

After playing with this option for a couple of hours, I realized that
this behavior is more natural, so I changed the default to 'silent, IOW
now epa-file does symmetric encryption by default :-)

Regards,
-- 
Daiki Ueno

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Ted Zlatanov]

On Thu, 16 Dec 2010 11:45:30 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

>> 1. add epa-file-encrypt-to hack to auth-source.el, and
>> 2. extend epa-file-select-keys (not epa-file-encrypt-to) so that it can
>> have 'ask or 'always or 'quiet, etc., and
>> 3. simplify the key selection UI by hiding public key list

DU> I did 2.  Try:

DU> (setq epa-file-select-keys 'silent)

DU> After playing with this option for a couple of hours, I realized that
DU> this behavior is more natural, so I changed the default to 'silent, IOW
DU> now epa-file does symmetric encryption by default :-)

That's awesome, but do I still need to set `epa-file-encrypt-to' locally
in auth-source.el?  IIUC I don't, because the default is to never show
the key selection UI.

Ted

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Ted Zlatanov]

On Thu, 16 Dec 2010 10:00:20 -0600 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Thu, 16 Dec 2010 11:45:30 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 
>>> 1. add epa-file-encrypt-to hack to auth-source.el, and
>>> 2. extend epa-file-select-keys (not epa-file-encrypt-to) so that it can
>>> have 'ask or 'always or 'quiet, etc., and
>>> 3. simplify the key selection UI by hiding public key list

DU> I did 2.  Try:

DU> (setq epa-file-select-keys 'silent)

DU> After playing with this option for a couple of hours, I realized that
DU> this behavior is more natural, so I changed the default to 'silent, IOW
DU> now epa-file does symmetric encryption by default :-)

TZ> That's awesome, but do I still need to set `epa-file-encrypt-to' locally
TZ> in auth-source.el?  IIUC I don't, because the default is to never show
TZ> the key selection UI.

OK, I see now that it's still necessary for older Emacs versions (I
missed that in your earlier mail).  I'll do it.

Thanks
Ted

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Daiki Ueno <ueno@unixuser.org> writes:

> After playing with this option for a couple of hours, I realized that
> this behavior is more natural, so I changed the default to 'silent, IOW
> now epa-file does symmetric encryption by default :-)

Great!  I think that's what most users would expect (unless they've set
up symmetric encryption already). 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug#7487: 24.0.50; Gnus nnimap broken

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <lmi@gnus.org> writes:

> Great!  I think that's what most users would expect (unless they've set
> up symmetric encryption already). 

I mean, asymmetric...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen