bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 2790 bytes --]

A .netrc or .authinfo file typically contains the credentials for a
specific machine on a single line, but according to netrc(5)
<http://linux.die.net/man/5/netrc> it's allowed to use newlines between
the tokens as well.  Using (netrc-parse) leaves such items out which is
clearly an error in the implementation.



In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.6)
 of 2015-09-10 on foutrelis
Windowing system distributor `The X.Org Foundation', version 11.0.11704000
System Description: Arch Linux

Configured using:
 `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils time-date tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 70945 6811)
 (symbols 48 17553 0)
 (miscs 40 35 138)
 (strings 32 9020 4341)
 (string-bytes 1 247698)
 (vectors 16 8908)
 (vector-slots 8 383094 17840)
 (floats 8 63 74)
 (intervals 56 177 0)
 (buffers 960 11)
 (heap 1024 49544 969))

[-- Attachment #2: Type: text/html, Size: 3153 bytes --]

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Ted Zlatanov]

On Thu, 3 Dec 2015 11:20:29 +0100 Vasilij Schneidermann <v.schneidermann@gmail.com> wrote: 

VS> A .netrc or .authinfo file typically contains the credentials for a
VS> specific machine on a single line, but according to netrc(5)
VS> <http://linux.die.net/man/5/netrc> it's allowed to use newlines between
VS> the tokens as well.  Using (netrc-parse) leaves such items out which is
VS> clearly an error in the implementation.

The Emacs implementation in auth-source.el `auth-source-netrc-parse',
which is used by default for netrc files today, is not a
reimplementation of that semi-standard:

* there is no RFC for the netrc format as far as I know

* Emacs supports both single and double quotes around strings (note the
  man page doesn't indicate how tokens are quoted!)

* macdef and default are not explicitly supported

* arbitrary keywords are supported

* the "machine" keyword is not required for an entry

The last two, especially, make it hard to support multi-line entries.
Since there is no RFC to my knowledge and no functional need to spread
entries across multiple lines, I'd like to close this as a WONTFIX. Any
opinions?

(The netrc.el `netrc-parse' code is not actively used nowadays in Emacs,
as far as I know, but hasn't been removed due to my laziness. I see a
single reference to it in gnus/nntp.el. It has some deficiencies that
were improved in `auth-source-netrc-parse', for instance supporting
single-quoted strings. Any objections to removing it? CC to Lars since
he wrote it originally.)

Ted

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Glenn Morris]

Ted Zlatanov wrote:

> (The netrc.el `netrc-parse' code is not actively used nowadays in Emacs,

Just to point out that there are even more functions for netrc parsing
in Emacs, as pointed out three years ago in http://debbugs.gnu.org/12511
(of which this report is a duplicate).
At the time it seemed to me that ange-ftp-parse-netrc looked
like it might handle newlines.

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Andy Moreton]

On Thu 03 Dec 2015, Ted Zlatanov wrote:

> On Thu, 3 Dec 2015 11:20:29 +0100 Vasilij Schneidermann <v.schneidermann@gmail.com> wrote: 
>
> VS> A .netrc or .authinfo file typically contains the credentials for a
> VS> specific machine on a single line, but according to netrc(5)
> VS> <http://linux.die.net/man/5/netrc> it's allowed to use newlines between
> VS> the tokens as well.  Using (netrc-parse) leaves such items out which is
> VS> clearly an error in the implementation.
>
> The Emacs implementation in auth-source.el `auth-source-netrc-parse',
> which is used by default for netrc files today, is not a
> reimplementation of that semi-standard:
>
> * there is no RFC for the netrc format as far as I know
>
> * Emacs supports both single and double quotes around strings (note the
>   man page doesn't indicate how tokens are quoted!)
>
> * macdef and default are not explicitly supported
>
> * arbitrary keywords are supported
>
> * the "machine" keyword is not required for an entry
>
> The last two, especially, make it hard to support multi-line entries.
> Since there is no RFC to my knowledge and no functional need to spread
> entries across multiple lines, I'd like to close this as a WONTFIX. Any
> opinions?

RFCs describe wire-protocols and interchange data formats, so I would
not expect an RFC for this. The .netrc file format is from BSD UNIX.

Support for arbitrary keywords, and omission of the machine token mean
that emacs will parse a .netrc file that other programs reject as
ill-formed. That is unhelpful: either be compatible with other programs
that read .netrc, or define  a new file format with a different name.

    AndyM

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Ted Zlatanov]

On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 

AM> RFCs describe wire-protocols and interchange data formats, so I would
AM> not expect an RFC for this. The .netrc file format is from BSD UNIX.

For instance, https://tools.ietf.org/html/rfc1035 defines the format of
master zone files, and https://tools.ietf.org/html/rfc4180 defines CSV
files. The closest RFC in spirit is https://tools.ietf.org/html/rfc4155
which defines the mbox format but says

    "The mbox database format is not documented in an
    authoritative specification, but instead exists as a well-known output
    format that is anecdotally documented, or which is only authoritatively
    documented for a specific platform or tool."

Sounds familiar? :) I'm sure we can argue for a while about the nuances
of RFCs, but I hope this is enough to explain why I looked for an RFC.

My more important point was that there's no standard (or library) for
parsing netrc files. In particular, quoting is not specified as I
mentioned. The BSD spec is ambiguous and allowing multi-line entries
would not serve any practical purpose.

I did a quick survey. I first remember using netrc files with ncftp. The
documentation at http://cng.seas.rochester.edu/CNG/docs/man/ncftp.html
shows examples of the "macdef" keyword that make it clear that it just
doesn't make sense across multiple programs:

       machine wuarchive.wustl.edu
                macdef init
                      cd /graphics/gif
                      ls -lt
                (mandatory blank line to end the macro)

Note that quoting is not shown, so it's up to the user to figure it out.

The most used implementation is probably
http://curl.haxx.se/docs/manual.html but they don't specify the syntax
and from looking at their netrc.c don't support "default" or "macdef"
keywords either.

wget just says to consult "your system manuals" but does support
"default" and "macdef" in its netrc.c so it's a bit more compliant.

I hope this was useful to illustrate my original statements.

AM> Support for arbitrary keywords, and omission of the machine token mean
AM> that emacs will parse a .netrc file that other programs reject as
AM> ill-formed. That is unhelpful: either be compatible with other programs
AM> that read .netrc, or define  a new file format with a different name.

Do you have an actual example? Can you be more specific about what broke
and how? Otherwise I think it's OK to leave things as they are.

Ted

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Andy Moreton]

On Sat 05 Dec 2015, Ted Zlatanov wrote:

> On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 
> My more important point was that there's no standard (or library) for
> parsing netrc files. In particular, quoting is not specified as I
> mentioned. The BSD spec is ambiguous and allowing multi-line entries
> would not serve any practical purpose.

The spec clearly defines that tokens are separated with SPACE TAB or
NEWLINE, which seems unambiguous to me. Multi-line entries are an
inherent part of the file format, and are useful for readable layout.
For an example, see:
    http://www.catb.org/esr/writings/taoup/html/ch10s03.html

It's perfectly ok to parse an extended version of the file format which
allows for quoting and escaping mechanisms, but not if it is done is a way
which is incompatible with reasonable and well formed existing files.

Disallowing NEWLINE as a token separator means you are not parsing an
extension of the existing format, but something incompatible with it.
That's fine as it offers useful features, but it should use a different
file name to make it clear that it is not the same format.

    AndyM

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Ted Zlatanov]

On Mon, 07 Dec 2015 14:14:13 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 

AM> On Sat 05 Dec 2015, Ted Zlatanov wrote:
>> On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 
>> My more important point was that there's no standard (or library) for
>> parsing netrc files. In particular, quoting is not specified as I
>> mentioned. The BSD spec is ambiguous and allowing multi-line entries
>> would not serve any practical purpose.

AM> The spec clearly defines that tokens are separated with SPACE TAB or
AM> NEWLINE, which seems unambiguous to me.

Without a quoting standard, it's not clear whether strings inside quotes
can be multiline and whether the newline itself can be escaped.

AM> Disallowing NEWLINE as a token separator means you are not parsing an
AM> extension of the existing format, but something incompatible with it.
AM> That's fine as it offers useful features, but it should use a different
AM> file name to make it clear that it is not the same format.

Fortunately `auth-sources' defaults to ~/.authinfo and ~/.authinfo.gpg
before ~/.netrc right now.

Ted

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Andy Moreton]

On Mon 07 Dec 2015, Ted Zlatanov wrote:

> On Mon, 07 Dec 2015 14:14:13 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 
>
> AM> On Sat 05 Dec 2015, Ted Zlatanov wrote:
>>> On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 
>>> My more important point was that there's no standard (or library) for
>>> parsing netrc files. In particular, quoting is not specified as I
>>> mentioned. The BSD spec is ambiguous and allowing multi-line entries
>>> would not serve any practical purpose.
>
> AM> The spec clearly defines that tokens are separated with SPACE TAB or
> AM> NEWLINE, which seems unambiguous to me.
>
> Without a quoting standard, it's not clear whether strings inside quotes
> can be multiline and whether the newline itself can be escaped.

The original spec is archaic - there is no quoting standard, only tokens
separated by whitespace. As such, a quote is simply another character.

> AM> Disallowing NEWLINE as a token separator means you are not parsing an
> AM> extension of the existing format, but something incompatible with it.
> AM> That's fine as it offers useful features, but it should use a different
> AM> file name to make it clear that it is not the same format.
>
> Fortunately `auth-sources' defaults to ~/.authinfo and ~/.authinfo.gpg
> before ~/.netrc right now.

Indeed.

    AndyM

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Ted Zlatanov]

On Tue, 08 Dec 2015 09:35:22 +0000 Andy Moreton <andrewjmoreton@gmail.com> wrote: 

AM> On Mon 07 Dec 2015, Ted Zlatanov wrote:

>> Without a quoting standard, it's not clear whether strings inside quotes
>> can be multiline and whether the newline itself can be escaped.

AM> The original spec is archaic - there is no quoting standard, only tokens
AM> separated by whitespace. As such, a quote is simply another character.

Right, and therefore we can't support the original spec as long as we
also support the more important single-quoting and double-quoting of strings.

I'm closing this without prejudice because I think we've stated our
positions clearly and it's not a bug, but if you or anyone else want to
propose a patch that can support multiline or the other keywords without
breaking the current functionality, we can reopen.

I would also welcome patches to remove the netrc.el usage.

Thanks
Ted

bug#22081: 24.5; netrc.el fails parsing authinfo items spread over multiple lines

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> (The netrc.el `netrc-parse' code is not actively used nowadays in Emacs,
> as far as I know, but hasn't been removed due to my laziness. I see a
> single reference to it in gnus/nntp.el. It has some deficiencies that
> were improved in `auth-source-netrc-parse', for instance supporting
> single-quoted strings. Any objections to removing it? CC to Lars since
> he wrote it originally.)

Let's see...

Geez, that's some convoluted code.  I think what it's meant to do is to
allow the user to define `nntp-authinfo-file' and allow that to override
the other Emacs authinfo choices.

I think it would be nice to not break people's setups, but I think it
would be clearer just to do something like

(let ((auth-sources (cons nntp-authinfo-file auth-sources)))
  ...)

before calling auth-source, and delete the explicit calls to netrc
completely.  And then you could move netrc.el to obsolete/.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no