mail-extract-address-components extract modified full name

mail-extract-address-components extract modified full name

[Yoichi NAKAYAMA]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

Description of `mail-extract-address-components' says
 "extract full name and canonical address"
But it actually returns modified (canonicalized) full name like:

 (mail-extract-address-components "\"Nakayama, Y\" <yoichi@geiin.org>")
 => ("Y. Nakayama" "yoichi@geiin.org")
 (mail-extract-address-components "Y Nakayama <yoichi@geiin.org>")
 => ("Y. Nakayama" "yoichi@geiin.org")

So the description would be "extract canonical full name...".

By the way, I think it would be better that canonicalization of full
name is done outside and the extracting function just return full name
as it is.  It is because full-name canonicalization code strongly
depends on language, culture and custom to describe names(*1), and
dividing those two features will allow more general use of
`mail-extract-address-components'.

(*1) An example can be found at the next part of this message.
Picture version with note is at http://yoichi.geiin.org/tmp/mail-extr.png

Best regards,
-- 
Yoichi NAKAYAMA

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

;; Note that
;; (mail-extract-address-components "手塚 治 <foo@bar.com>")
;; => ("手塚 治." "foo@bar.com")
;; seems funny.

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: mail-extract-address-components extract modified full name

[Richard Stallman]

      It is because full-name canonicalization code strongly
    depends on language, culture and custom to describe names(*1), and
    dividing those two features will allow more general use of
    `mail-extract-address-components'.

I don't think we should change what the existing function does, but we
could provide two subroutines to do just the extraction and just the
canonicalization.  I don't have time to write this myself, and I tend
to think it is not urgently needed.

    (*1) An example can be found at the next part of this message.
    Picture version with note is at http://yoichi.geiin.org/tmp/mail-extr.png

I can't access that, but if you email it to me, I can view it that way.
(I don't have fonts for the characters, so I can't see it that way.)
Please email it to me alone, not the list.

Although it is true that different languages and countries put a
person's names in different orders, a comma clearly indicates that
they have been swapped.  If you write the family name first because
family names always come first in your country, you won't use a comma.
Don't you agree?

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

>>>>> In <E1BouIW-0001E5-5D@fencepost.gnu.org>
>>>>>	Richard Stallman <rms@gnu.org> wrote:

> Although it is true that different languages and countries put a
> person's names in different orders, a comma clearly indicates that
> they have been swapped.  If you write the family name first because
> family names always come first in your country, you won't use a comma.
> Don't you agree?

The problem of m-e-a-c is not only for the name order.  Here's
an example:

(gnus-extract-address-components "王ヶ頭ホテル <ougatou@example.com>")
 => ("王. ヶ. 頭. ホテル" "ougatou@example.com")

That conversion is completely nonsense because "王ヶ頭ホテル" is
a proper noun.  So, I put the following line in my .emacs file:

(eval-after-load "mail-extr" '(defalias 'mail-extr-voodoo 'ignore))

I haven't checked it thoroughly whether it may cause another
problem, though.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

>>>>> In <b9yekmzbjuv.fsf@namazu.org> Katsumi Yamaoka wrote:

> The problem of m-e-a-c is not only for the name order.  Here's
> an example:

> (gnus-extract-address-components "王ヶ頭ホテル <ougatou@example.com>")
>  => ("王. ヶ. 頭. ホテル" "ougatou@example.com")

I've copied it from a wrong place, sorry.  The mail-, not gnus-
function behaves as shown above.

Gnus uses gnus-extract-address-components in some places because
m-e-a-c didn't support non-ASCII characters until Emacs 21.3.50.
However, the importance of using it increased in the other sense.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

>>>>> In <b9yk6wrmpm4.fsf@namazu.org> Katsumi Yamaoka wrote:

>> (gnus-extract-address-components ...

I must apologize to have written Japanese characters.  The
picture for that is here:

ftp://ftp.jpl.org/pub/tmp/ougatou.png
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: mail-extract-address-components extract modified full name

[Miles Bader]

Katsumi Yamaoka <yamaoka@jpl.org> writes:
> The problem of m-e-a-c is not only for the name order.  Here's
> an example:
>
> (gnus-extract-address-components "王ヶ頭ホテル <ougatou@example.com>")
>  => ("王. ヶ. 頭. ホテル" "ougatou@example.com")

FWIW, I think it's perfectly fine _not_ to add the dot even for roman
characters, e.g., my name is "M Bader".  Doing that might simplify
things.

-Miles
-- 
`The suburb is an obsolete and contradictory form of human settlement'

Re: mail-extract-address-components extract modified full name

[Lars Magne Ingebrigtsen]

Yoichi NAKAYAMA <yoichi@geiin.org> writes:

> By the way, I think it would be better that canonicalization of full
> name is done outside and the extracting function just return full name
> as it is.  It is because full-name canonicalization code strongly
> depends on language, culture and custom to describe names(*1), and
> dividing those two features will allow more general use of
> `mail-extract-address-components'.

I think in most instances using `mail-extract-address-components' is
probably the wrong thing.  It doesn't really do RFC(2)822 parsing --
it just takes something that looks pretty much like a mail address
and a name and tries to do something pleasing with it, even if it
isn't in the least valid as a RFC2822 address.

"手塚 治 <foo@bar.com>", for instance, isn't a valid address.  (It
has to be encoded first.)

So I'm not sure that this function should be altered.

On the other hand, if what you want is RFC2822 address parsing, we
have `mail-header-parse-address', which does strict parsing, and
nothing else.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

>>>>> In <m3u0vvuudd.fsf@quimbies.gnus.org>
>>>>>	Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> I think in most instances using `mail-extract-address-components' is
> probably the wrong thing.

I agree with you.  m-e-a-c is being used in not a few places in
Gnus, and it sometimes causes funny result, although it is not
serious so far as far as I know.  For instance, when I reply to
the address "手塚 治 <foo@bar.com>", the buffer named "*reply to
手塚 治.*" is created.

> It doesn't really do RFC(2)822 parsing --
> it just takes something that looks pretty much like a mail address
> and a name and tries to do something pleasing with it, even if it
> isn't in the least valid as a RFC2822 address.

> "手塚 治 <foo@bar.com>", for instance, isn't a valid address.  (It
> has to be encoded first.)

> So I'm not sure that this function should be altered.

> On the other hand, if what you want is RFC2822 address parsing, we
> have `mail-header-parse-address', which does strict parsing, and
> nothing else.

mail-header-parse-address doesn't support non-ASCII characters,
so a function like gnus-extract-address-components is also
needed.

P.S. In last message, I mixed up mail-extract-address-components
and mail-header-parse-address about non-ASCII characters, sorry.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

>>>>> In <b9yy8l76n4c.fsf@namazu.org> Katsumi Yamaoka wrote:

> mail-header-parse-address doesn't support non-ASCII characters,
> so a function like gnus-extract-address-components is also
> needed.

In my opinion, we need a much simpler function to replace the
present mail-extract-address-components.  The required feature
is only to parse the following four patterns:

ADRESS
<ADRESS>
ADRESS (NAME)
NAME <ADDRESS>

They may be combined separated with commas of course.  The NAME
portion may contain non-ASCII characters.  The NAME and ADDRESS
portions should never be modified.  Otherwise, it may be
sufficient to separate functions concerned with customs or
languages from the present mail-extract-address-components.  For
the later case, here's a way to explain what I'd like to say,
for example:

(defvar mail-extr-enable-voodoo nil "*docstring.")

(defun mail-extract-address-components (address &optional all)
  [...]
	  (when mail-extr-enable-voodoo
	    (mail-extr-voodoo ...))

I noticed mail-extract-address-components is a too complicated
function recently, in spite of the name looks like a standard
Emacs function.  Especially, it often breaks Japanese personal
names.  The attached picture is a collection of how m-e-a-c
works bad:


[-- Attachment #2: bad-meac.png --]
[-- Type: image/png, Size: 2568 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: mail-extract-address-components extract modified full name

[Simon Josefsson]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <b9yy8l76n4c.fsf@namazu.org> Katsumi Yamaoka wrote:
>
>> mail-header-parse-address doesn't support non-ASCII characters,
>> so a function like gnus-extract-address-components is also
>> needed.
>
> In my opinion, we need a much simpler function to replace the
> present mail-extract-address-components.

Good idea.  I have disliked mail-extr* for a long time.  It is both
complicated and not standards compliant.  If we are not going for
standards compliance, it should be possible to do something simple,
like the approach you propose.  XEmacs users have reported even
Latin-2 problems with the current implementation (Emacs do not have
those problems, though, but it suggest the implementation could be
improved).

> The required feature is only to parse the following four patterns:
>
> ADRESS
> <ADRESS>
> ADRESS (NAME)
> NAME <ADDRESS>
>
> They may be combined separated with commas of course.  The NAME
> portion may contain non-ASCII characters.  The NAME and ADDRESS
> portions should never be modified.

Quotation is what makes things complicated.  Consider:

"foo \"baz bar" <foo@bar>
"foo <foo@bar> bar" <foo@bar>
"foo '"<foo@bar>'" bar" <foo@bar>

The standard permit even more weirder things, though, but we probably
don't have to support those.

Read me right, this is not critique of your idea, just something to
keep in mind.

If you, or someone else, would like to implement the above idea, I
will try to assist and write a self test suite of it.  Then we can
detect regression problems in the future.  It is a big problem with
mail-extr* that you don't know how a small change might affect
practical use.

Btw, I assume you are familiar with g-e-a-c.  It is rather simple,
perhaps too simple.

(defun gnus-extract-address-components (from)
  "Extract address components from a From header.
Given an RFC-822 address FROM, extract full name and canonical address.
Returns a list of the form (FULL-NAME CANONICAL-ADDRESS).  Much more simple
solution than `mail-extract-address-components', which works much better, but
is slower."
  (let (name address)
    ;; First find the address - the thing with the @ in it.  This may
    ;; not be accurate in mail addresses, but does the trick most of
    ;; the time in news messages.
    (when (string-match "\\b[^@ \t<>]+[!@][^@ \t<>]+\\b" from)
      (setq address (substring from (match-beginning 0) (match-end 0))))
    ;; Then we check whether the "name <address>" format is used.
    (and address
	 ;; Linear white space is not required.
	 (string-match (concat "[ \t]*<" (regexp-quote address) ">") from)
	 (and (setq name (substring from 0 (match-beginning 0)))
	      ;; Strip any quotes from the name.
	      (string-match "^\".*\"$" name)
	      (setq name (substring name 1 (1- (match-end 0))))))
    ;; If not, then "address (name)" is used.
    (or name
	(and (string-match "(.+)" from)
	     (setq name (substring from (1+ (match-beginning 0))
				   (1- (match-end 0)))))
	(and (string-match "()" from)
	     (setq name address))
	;; XOVER might not support folded From headers.
	(and (string-match "(.*" from)
	     (setq name (substring from (1+ (match-beginning 0))
				   (match-end 0)))))
    (list (if (string= name "") nil name) (or address from))))

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

>>>>> In <ilu7jspu7bp.fsf@latte.josefsson.org>
>>>>>	Simon Josefsson <jas@extundo.com> wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> In my opinion, we need a much simpler function to replace the
>> present mail-extract-address-components.

> Good idea.  I have disliked mail-extr* for a long time.  It is both
> complicated and not standards compliant.  If we are not going for
> standards compliance, it should be possible to do something simple,
> like the approach you propose.

I was encouraged reading your message.

[...]

> Quotation is what makes things complicated.  Consider:

> "foo \"baz bar" <foo@bar>
> "foo <foo@bar> bar" <foo@bar>
> "foo '"<foo@bar>'" bar" <foo@bar>

Wow!  I saw myself trying them on the rfc2047 encoder. :-p

> The standard permit even more weirder things, though, but we probably
> don't have to support those.

> Read me right, this is not critique of your idea, just something to
> keep in mind.

I see.  In fact, I have many opportunities to encounter such
things.  So, we will probably be unable to ignore them, I think.

> If you, or someone else, would like to implement the above idea, I
> will try to assist and write a self test suite of it.  Then we can
> detect regression problems in the future.  It is a big problem with
> mail-extr* that you don't know how a small change might affect
> practical use.

Thanks a lot.  I'm interested in writing the new code, although
for the moment I'm not sure whether I have time for it...

> Btw, I assume you are familiar with g-e-a-c.  It is rather simple,
> perhaps too simple.

> (defun gnus-extract-address-components (from)
>   "Extract address components from a From header.

Yes, it's too simple to parse two or more addresses in the
argument.  However, it can be an exemplar for writing a simple
and fast code.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: mail-extract-address-components extract modified full name

[Stefan Monnier]

> like the approach you propose.  XEmacs users have reported even
> Latin-2 problems with the current implementation (Emacs do not have
> those problems, though, but it suggest the implementation could be
> improved).

[ The below is all "IIRC". ]

The function is supposed to receive ASCII input, so it's no wonder it might
break in other circumstances.  Why ASCII input?

Because the way things are defined in the RFCs, you should split the address
before doing the un-quoting of base64 and QP thingies.
I.e. after unquoting, the string might not be parsable any more (because
one of the QP chars could be a ", a \, a <, or something like that).

So the usual answer is that if you call the function with non-ASCII input,
you're not using it properly.  But of course, it's not that simple since you
might want to call that function e.g. on an email message that is being
written and that hasn't been QP-encoded yet.


        Stefan

Re: mail-extract-address-components extract modified full name

[Simon Josefsson]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> like the approach you propose.  XEmacs users have reported even
>> Latin-2 problems with the current implementation (Emacs do not have
>> those problems, though, but it suggest the implementation could be
>> improved).
>
> [ The below is all "IIRC". ]
>
> The function is supposed to receive ASCII input, so it's no wonder it might
> break in other circumstances.  Why ASCII input?
>
> Because the way things are defined in the RFCs, you should split the address
> before doing the un-quoting of base64 and QP thingies.
> I.e. after unquoting, the string might not be parsable any more (because
> one of the QP chars could be a ", a \, a <, or something like that).
>
> So the usual answer is that if you call the function with non-ASCII input,
> you're not using it properly.  But of course, it's not that simple since you
> might want to call that function e.g. on an email message that is being
> written and that hasn't been QP-encoded yet.

I agree, and have been arguing the same thing when people complain
that mail-extr* cannot handle their weird input.

Unfortunately, it is a losing discussion, since I can't claim that
mail-extr* is only intended for use with all-ASCII valid RFC 822
input, since that isn't what it implement.  It is just a big hack, and
could be massaged into behaving (badly) for any purpose.

One example is that BBDB reportedly uses mail-extr* to split the
e-mail addresses it store locally, in ~/.bbdb, which naturally aren't
QP encoded.  This probably illustrate a class of applications, that
deal with mail addresses, but aren't proper mail reader or writer, so
it wouldn't make sense for them to use QP.

IMHO, there should be two packages:

1) Proper RFC (2)822 parser.  There is rfc822.el but it is
   insufficient, and I'm not sure it is correct -- it uses regexp's a
   lot, but I recall that the "correct" 2822 grammar, expressed as
   regexp's, is much more complex than what rfc822.el does.
   Naturally, it should only accept valid RFC 822 input, which is
   ASCII only.

   (Incidentally, the QP encoder/decoder need to use this package,
   since QP must only be applied to certain RFC 2822 grammatical
   terminals, not all text, and I believe the current QP
   encoder/decoder doesn't do this properly.)

2) Ad-hoc approach that split real world textual e-mail address,
   including non-ASCII, into its components.  Might use the proper
   parser, at least partially.  Perhaps similar to what Katsumi
   Yamaoka proposed.

When these two packages exist, each current uses of mail-extr* should
be investigated to find out what is really intended there.

At some point in time, I counted the number of functions in Emacs that
implement something similar than the mail-extr* functions do
(e.g. take a textual e-mail address and split it up) and found ~5-10
versions, all with their own problems.

Sadly, I keep writing rants about the situation instead of working on
solving it...  Perhaps partly that is because it is not straight
forward to solve this; you will probably have to implement one API
first, tinker with it to get experience with it, and then rewrite it
slightly, and so on.  Sounds like real work.  Perhaps someone else has
a clearer vision on how to implement it, and time to try it out.

Thanks.

Re: mail-extract-address-components extract modified full name

[Katsumi Yamaoka]

>>>>> In <ilubri1ifdy.fsf@latte.josefsson.org>
>>>>>	Simon Josefsson <jas@extundo.com> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The function is supposed to receive ASCII input, so it's no wonder it might
>> break in other circumstances.  Why ASCII input?

I didn't know the function is only for ASCII characters until
quite recently.  It is because the function looks like a generic
function and works with non-ASCII characters after a fashion.
In addition, I cannot imagine who needs the voodoo function.  It
seems very much personalized for (by?) someone.  Why isn't it
sufficient to extract and separate name and address portions
simply without any modifications?  It can be done later if
needed, though.

[...]

> IMHO, there should be two packages:

> 1) Proper RFC (2)822 parser.  There is rfc822.el but it is
>    insufficient, and I'm not sure it is correct -- it uses regexp's a
>    lot, but I recall that the "correct" 2822 grammar, expressed as
>    regexp's, is much more complex than what rfc822.el does.
>    Naturally, it should only accept valid RFC 822 input, which is
>    ASCII only.

>    (Incidentally, the QP encoder/decoder need to use this package,
>    since QP must only be applied to certain RFC 2822 grammatical
>    terminals, not all text, and I believe the current QP
>    encoder/decoder doesn't do this properly.)

> 2) Ad-hoc approach that split real world textual e-mail address,
>    including non-ASCII, into its components.  Might use the proper
>    parser, at least partially.  Perhaps similar to what Katsumi
>    Yamaoka proposed.

3) Disable the voodoo function by default.  Since using the
   following form seems helpful not to break Japanese names.

   (eval-after-load "mail-extr" '(defalias 'mail-extr-voodoo 'ignore))

   Furthermore, I think the voodoo function should be moved to
   the language or the international subdirectory.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: mail-extract-address-components extract modified full name

[Yoichi NAKAYAMA]

At Tue, 27 Jul 2004 16:11:36 +0900,
Katsumi Yamaoka wrote:
> In my opinion, we need a much simpler function to replace the
> present mail-extract-address-components.

I agree.
Although, for compatibility, as others say, I think the new really
generic function would better to have another name. (Even though
m-e-a-c seems to be a too generic name for the current specific
feature, it have been exist for a long period).
-- 
Yoichi NAKAYAMA