[PATCH] EUDC email addresses via completion-at-point in message-mode

[PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

Hello,

when switching to a new in-buffer completion user interface package
which is solely based on completion-at-point, I observed that email
address completion in message-mode happens in the minibuffer, regardless
of what UI on top of completion-at-point I use. Also, EUDC's ability to
aggregate search results from several servers, which was recently added
with commit 0470a4a939772c4bd25123b15f5eadab41f8bee5, was not kicking
in, i.e. I was presented results from a single EUDC server only.

Browsing through message.el quickly brought me to the sixth of the 21
FIXMEs in that file, which reads on line 3183:

  ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
  ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
  (add-hook 'completion-at-point-functions #'message-completion-function nil t)

Thus, the most straightforward solution to my problem seemed to make a
new EUDC function that can be added to completion-at-point-functions,
and to update message-mode to actually add that function to
completion-at-point-functions. This is what the below attached patch
does.

As a result, both of my issues are fixed: the new completion-at-point
user interface presents the email addresses, and I get results from all
configured EUDC servers.

For EUDC, the added value is that it gains a function for
completion-at-point-functions, which it didn't have before. The new
function also makes use of EUDC's new "try all servers" feature,
merging the search results from several sources.

For message-mode, the added value is that the FIXME regarding the
merging of email address completion tables is addressed, as this is now
done by EUDC. Further sources for email addresses can be added to EUDC
by writing EUDC back-ends for them, which is not difficult in my
experience. Any new email address sources will thus become available in
message-mode without any further changes in message-mode itself (neither
code nor config).


Hoping to have helped, and looking forward to your thoughts,

  --alexander



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-EUDC-email-addresses-via-completion-at-point-in-mess.patch --]
[-- Type: text/x-patch, Size: 10197 bytes --]

From 0fe373c62f0a5186f9609bb33b0890cce2fb8dcc Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Fri, 8 Apr 2022 22:28:45 +0200
Subject: [PATCH] EUDC email addresses via completion-at-point in message-mode

* lisp/net/eudc-capf.el (new file): Add new 'eudc-capf-complete'
function.
* lisp/gnus/message.el (message-mode): Add 'eudc-capf-complete' to
'completion-at-point-functions' when a 'message-mode' buffer is
created, removing the FIXME.
* doc/misc/eudc.texi (Inline Query Expansion): Add a new subsection,
describing the new 'completion-at-point' mechanism in 'message-mode'.
* etc/NEWS (EUDC): Describe the new 'completion-at-point' method.
---
 doc/misc/eudc.texi    |  23 ++++++++
 etc/NEWS              |   6 ++
 lisp/gnus/message.el  |   4 +-
 lisp/net/eudc-capf.el | 130 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 161 insertions(+), 2 deletions(-)
 create mode 100644 lisp/net/eudc-capf.el

diff --git a/doc/misc/eudc.texi b/doc/misc/eudc.texi
index 71e3e6b9ed..cc98b0606c 100644
--- a/doc/misc/eudc.texi
+++ b/doc/misc/eudc.texi
@@ -711,6 +711,7 @@ be passed to the program.
 
 @node Inline Query Expansion
 @section Inline Query Expansion
+@subsection Inline Query Expansion Using a Key Binding
 
 Inline query expansion is a powerful method to get completion from
 your directory servers.  The most common usage is for expanding names
@@ -883,6 +884,28 @@ An error is signaled.  The expansion aborts.
 Default is @code{select}
 @end defvar
 
+@subsection Inline Query Expansion Using completion-at-point
+
+In addition to providing a dedicated EUDC function for binding to a
+key shortcut (@pxref{Inline Query Expansion Using a Key Binding}),
+EUDC also provides a function to contribute search results to the
+Emacs in-buffer completion system available via the function
+@code{completion-at-point} (@pxref{Identifier
+Inquiries,,,maintaining}) in @code{message-mode} buffers
+(@pxref{Message}).  When using this mechanism, queries are made in the
+multi-server query mode of operation (@pxref{Multi-server Queries}).
+
+When a buffer in @code{message-mode} is created, EUDC's inline
+expansion function is automatically added to the variable
+@code{completion-at-point-functions}.  As a result, whenever
+@code{completion-at-point} is invoked in a @code{message-mode} buffer,
+EUDC will be queried for email addresses matching the words before
+point.  Since this will be useful only when editing specific message
+header fields that require specifying one or more email addresses, an
+additional check is performed whether point is actually in one of
+those header fields.  Thus, any matching email addresses will be
+offered for completion in suitable message header fields only, and not
+in other places, like for example the body of the message.
 
 
 @node The Server Hotlist
diff --git a/etc/NEWS b/etc/NEWS
index 2fac893cc5..d6790424a7 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -773,6 +773,12 @@ is called, and the returned values are used to populate the phrase and
 comment parts (see RFC 5322 for definitions).  In both cases, the
 phrase part will be automatically quoted if necessary.
 
++++
+*** New function 'eudc-capf-complete' with message-mode integration
+EUDC can now contribute email addresses to 'completion-at-point' by
+adding the new function 'eudc-capf-complete' to
+'completion-at-point-functions' in message-mode.
+
 ** eww/shr
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index 30734b8f1a..9800167147 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -51,6 +51,7 @@
 (require 'yank-media)
 (require 'mailcap)
 (require 'sendmail)
+(require 'eudc-capf)
 
 (autoload 'mailclient-send-it "mailclient")
 
@@ -3180,8 +3181,7 @@ Like `text-mode', but with these additional commands:
     (mail-abbrevs-setup))
    ((message-mail-alias-type-p 'ecomplete)
     (ecomplete-setup)))
-  ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
-  ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
+  (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
   (add-hook 'completion-at-point-functions #'message-completion-function nil t)
   (unless buffer-file-name
     (message-set-auto-save-file-name))
diff --git a/lisp/net/eudc-capf.el b/lisp/net/eudc-capf.el
new file mode 100644
index 0000000000..673fbb440f
--- /dev/null
+++ b/lisp/net/eudc-capf.el
@@ -0,0 +1,130 @@
+;;; eudc-capf.el --- EUDC - completion-at-point bindings  -*- lexical-binding:t -*-
+
+;; Copyright (C) 2022 Free Software Foundation, Inc.
+;;
+;; Author: Alexander Adolf
+;;
+;; This file is part of GNU Emacs.
+;;
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;;    This library provides functions to deliver email addresses from
+;;    EUDC search results to `completion-at-point'.
+;;
+;;    Email address completion will likely be desirable only in
+;;    situations where designating email recipients plays a role, such
+;;    as when composing or replying to email messages, or when posting
+;;    to newsgroups, possibly with copies of the post being emailed.
+;;    Hence, modes relevant in such contexts, such as for example
+;;    `message-mode' and `mail-mode', often at least to some extent
+;;    provide infrastructure for different functions to be called when
+;;    completing in certain message header fields, or in the body of
+;;    the message.  In other modes for editing email messages or
+;;    newsgroup posts, which do not provide such infrastructure, any
+;;    completion function providing email addresses will need to check
+;;    whether the completion attempt occurs in an appropriate context
+;;    (that is, in a relevant message header field) before providing
+;;    completion candidates.  Two mechanisms are thus provided by this
+;;    library.
+;;
+;;    The first mechanism is intended for use by the modes listed in
+;;    `eudc-capf-modes', and relies on these modes adding
+;;    `eudc-capf-complete' to `completion-at-point-functions', as
+;;    would be usually done for any general-purpose completion
+;;    function.  In this mode of operation, and in order to offer
+;;    email addresses only in contexts where the user would expect
+;;    them, a check is performed whether point is on a line that is a
+;;    message header field suitable for email addresses, such as for
+;;    example "To:", "Cc:", etc.
+;;
+;;    The second mechanism is intended for when the user modifies
+;;    `message-completion-alist' to replace `message-expand-name' with
+;;    the function `eudc-capf-message-expand-name'.  As a result,
+;;    minibuffer completion (`completing-read') for email addresses
+;;    would no longer enabled in `message-mode', but
+;;    `completion-at-point' (in-buffer completion) only.
+
+;;; Usage:
+
+;;    (require 'eudc-capf)
+;;    (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
+
+;;; Code:
+
+(require 'eudc)
+
+(defvar message-email-recipient-header-regexp)
+(defvar mail-abbrev-mode-regexp)
+(declare-function mail-abbrev-in-expansion-header-p "mailabbrev" ())
+
+(defconst eudc-capf-modes '(message-mode) "List of modes in which email \
+address completion is to be attempted.")
+
+;; completion functions
+
+;;;###autoload
+(defun eudc-capf-complete ()
+  "Email address completion function for `completion-at-point-functions'.
+
+This function checks whether the current major mode is one of the
+modes listed in `eudc-capf-modes', and whether point is on a line
+with a message header listing email recipients, that is, a line
+whose beginning matches `message-email-recipient-header-regexp',
+and, if the check succeeds, searches for records matching the
+words before point.
+
+The return value is either nil when no match is found, or a
+completion table as required for functions listed in
+`completion-at-point-functions'.  When a completion table is
+returned, it is marked as non-exclusive."
+  (if (and (seq-some #'derived-mode-p eudc-capf-modes)
+           (let ((mail-abbrev-mode-regexp message-email-recipient-header-regexp))
+             (mail-abbrev-in-expansion-header-p)))
+      (eudc-capf-message-expand-name)
+    nil))
+
+;;;###autoload
+(defun eudc-capf-message-expand-name ()
+  "Email address completion function for `message-completion-alist'.
+
+When this function is added to `message-completion-alist',
+replacing any existing entry for `message-expand-name' there,
+with an appropriate regular expression such as for example
+`message-email-recipient-header-regexp', then EUDC will be
+queried for email addresses, and the results delivered to
+`completion-at-point'."
+  (if (or (and (boundp 'eudc-server) eudc-server)
+          (and (boundp 'eudc-server-hotlist) eudc-server-hotlist))
+      (progn
+        (setq-local completion-styles '(substring partial-completion)
+                    completion-ignore-case t)
+        (let* ((beg (save-excursion
+                      (if (re-search-backward "\\([:,]\\|^\\)[ \t]*"
+                                              (point-at-bol) 'move)
+                          (goto-char (match-end 0)))
+                      (point)))
+               (end (point))
+               (prefix (save-excursion (buffer-substring-no-properties beg end))))
+          (list beg end
+                (completion-table-with-cache
+                 (lambda (_)
+                   (eudc-query-with-words (split-string prefix "[ \t]+") t))
+                 t)
+                :exclusive 'no)))
+    nil))
+
+(provide 'eudc-capf)
+;;; eudc-capf.el ends here
-- 
2.35.1

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Alexander,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> when switching to a new in-buffer completion user interface package
> which is solely based on completion-at-point, I observed that email
> address completion in message-mode happens in the minibuffer, regardless
> of what UI on top of completion-at-point I use. Also, EUDC's ability to
> aggregate search results from several servers, which was recently added
> with commit 0470a4a939772c4bd25123b15f5eadab41f8bee5, was not kicking
> in, i.e. I was presented results from a single EUDC server only.

I tested this patch with my current setup, and it doesn't break
anything, however I'm not using completion-at-point yet, see below.

Is there a way to implement eudc-capf-complete such that it doesn't need
eudc-capf-modes?  I don't like the fact that eudc-capf-modes would need
to be updated every time a new mode wants to use eudc-capf-complete.
What would go wrong if you just omitted the mode check?  (Maybe I'm
missing something about completion-at-point's design here.)

Can you add an example in message-mode's manual, for how to make use of
completion-at-point, even if it means mentioning a specific
completion-at-point UI package?

Currently I use:

     (with-eval-after-load "message"
       (define-key message-mode-map
        [(control ?c) (tab)]
        'eudc-expand-try-all))

as per the EUDC manual.  I shouldn't need that anymore with this patch,
correct?  What completion-at-point UI package should I install in order
to test this?

Thanks,
Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello Thomas,

Many thanks for your swift response.

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> [...]
> I tested this patch with my current setup, and it doesn't break
> anything,

I'm glad to hear this.

> [...]
> Is there a way to implement eudc-capf-complete such that it doesn't need
> eudc-capf-modes?  I don't like the fact that eudc-capf-modes would need
> to be updated every time a new mode wants to use eudc-capf-complete.

Yes, it may at first seem onerous having to add each new mode to
eudc-capf-modes. But the integration works both ways: eudc-capf-complete
also needs to be able to work out whether point is on a line where email
addresses are wanted/useful. Hence, some basic assumptions
eudc-capf-complete makes about the mode in which it is used, need to be
confirmed first. For the modes listed in eudc-capf-modes, these
assumptions have been confirmed.

> What would go wrong if you just omitted the mode check? (Maybe I'm
> missing something about completion-at-point's design here.)

I am trying to limit the cases where eudc-capf-complete "kicks in" to
those where I think it can be assumed that its results will quite
certainly desirable. The reason is that completion-at-point works along
the functions in completion-at-point-functions, and stops when the first
function returns a completion table. Thus, by calling eudc-capf-complete
too "aggressively", more useful completion results may be prevented from
being offered to the user.

The check to establish whether point is in a suitable header line, calls
mail-abbrev-in-expansion-header-p (from mailabbrev.el). I haven't tested
what effects calling this in any other mode than mail-mode or
message-mode has. It may well work ok in the majority of cases, but then
the list of "other modes" to test could potentially be quite long...

Also, an EUDC query may take a second or two (remote servers and stuff),
so triggering it when not useful could hamper the speed of editing
without user-visible benefit.

> Can you add an example in message-mode's manual, for how to make use of
> completion-at-point, even if it means mentioning a specific
> completion-at-point UI package?

Um, you just call it? That said, I'd of course be happy to add a couple
of sentences along the lines of what I explain below.

> Currently I use:
>
>      (with-eval-after-load "message"
>        (define-key message-mode-map
>         [(control ?c) (tab)]
>         'eudc-expand-try-all))
>
> as per the EUDC manual.  I shouldn't need that anymore with this patch,
> correct?  What completion-at-point UI package should I install in order
> to test this?

You don't need any additional package to test it. Just start composing a
new message in message-mode (e.g. "C-x m"), put point in the "To:" line,
type a substring you know there will be a match for, and then invoke
completion-at-point (e.g. "M-: (completion-at-point)"). The default UI
is completing-read (i.e. you'll see candidates presented in the
minibuffer).

By default, message-mode binds TAB to message-tab, which in turn calls
completion-at-point. Depending on your keymap setup it may thus be
sufficient to type TAB in the "To:" line, or to change your setup to
replace eudc-expand-try-all with completion-at-point.

The only info documentation about in-buffer completion using
completion-at-point I was able to find, is the info node "Completion for
Symbol Names", which is rather short IMO. Whereas minibuffer completion
enjoys a quite extensive info documentation (see info node
"Completion").


Cheers, and looking forward to your thoughts,

  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Alexander,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Hello Thomas,
>
> Many thanks for your swift response.
>
> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> [...]
>> I tested this patch with my current setup, and it doesn't break
>> anything,
>
> I'm glad to hear this.
>
>> [...]
>> Is there a way to implement eudc-capf-complete such that it doesn't need
>> eudc-capf-modes?  I don't like the fact that eudc-capf-modes would need
>> to be updated every time a new mode wants to use eudc-capf-complete.
>
> Yes, it may at first seem onerous having to add each new mode to
> eudc-capf-modes. But the integration works both ways: eudc-capf-complete
> also needs to be able to work out whether point is on a line where email
> addresses are wanted/useful. Hence, some basic assumptions
> eudc-capf-complete makes about the mode in which it is used, need to be
> confirmed first. For the modes listed in eudc-capf-modes, these
> assumptions have been confirmed.

OK.

>> What would go wrong if you just omitted the mode check? (Maybe I'm
>> missing something about completion-at-point's design here.)
>
> I am trying to limit the cases where eudc-capf-complete "kicks in" to
> those where I think it can be assumed that its results will quite
> certainly desirable. The reason is that completion-at-point works along
> the functions in completion-at-point-functions, and stops when the first
> function returns a completion table. Thus, by calling eudc-capf-complete
> too "aggressively", more useful completion results may be prevented from
> being offered to the user.
>
> The check to establish whether point is in a suitable header line, calls
> mail-abbrev-in-expansion-header-p (from mailabbrev.el). I haven't tested
> what effects calling this in any other mode than mail-mode or
> message-mode has. It may well work ok in the majority of cases, but then
> the list of "other modes" to test could potentially be quite long...
>
> Also, an EUDC query may take a second or two (remote servers and stuff),
> so triggering it when not useful could hamper the speed of editing
> without user-visible benefit.

OK, I see.

>> Can you add an example in message-mode's manual, for how to make use of
>> completion-at-point, even if it means mentioning a specific
>> completion-at-point UI package?
>
> Um, you just call it? That said, I'd of course be happy to add a couple
> of sentences along the lines of what I explain below.
>
>> Currently I use:
>>
>>      (with-eval-after-load "message"
>>        (define-key message-mode-map
>>         [(control ?c) (tab)]
>>         'eudc-expand-try-all))
>>
>> as per the EUDC manual.  I shouldn't need that anymore with this patch,
>> correct?  What completion-at-point UI package should I install in order
>> to test this?
>
> You don't need any additional package to test it. Just start composing a
> new message in message-mode (e.g. "C-x m"), put point in the "To:" line,
> type a substring you know there will be a match for, and then invoke
> completion-at-point (e.g. "M-: (completion-at-point)"). The default UI
> is completing-read (i.e. you'll see candidates presented in the
> minibuffer).
>
> By default, message-mode binds TAB to message-tab, which in turn calls
> completion-at-point. Depending on your keymap setup it may thus be
> sufficient to type TAB in the "To:" line, or to change your setup to
> replace eudc-expand-try-all with completion-at-point.

It looks like the default binding for TAB in message-mode-map is
message-tab; so I tested by rebinding TAB to message-tab.  It worked,
except I was expecting case sensitive results.  Can you please change
`completion-ignore-case' to nil?  And figure out what to do with
`completion-table-with-cache''s IGNORE-CASE argument accordingly, if
necessary?

I had to run (message-tab) repeatedly to get the final result if
multiple results were available, whereas with eudc-expand-try-all bound
to TAB, I get a "Multiple matches found; choose one:" prompt.  I guess
that's just a different UI style for completion-at-point?  FWIW, I think
I prefer eudc-expand-try-all's behaviour.

I have one bbdb entry and one LDAP server in the hotlist.  It seems like
the LDAP operation happens every time I call (message-tab).  Is that
expected?  Is the completion table caching meant to prevent this?  I
don't like the idea that hitting TAB several times per email address
will result in several calls to the LDAP server.  Can
completion-at-point be made to have the same behaviour as
eudc-expand-try-all, which only results in each backend being queried
once?

> The only info documentation about in-buffer completion using
> completion-at-point I was able to find, is the info node "Completion for
> Symbol Names", which is rather short IMO. Whereas minibuffer completion
> enjoys a quite extensive info documentation (see info node
> "Completion").

Depending on the outcome of the above we might want to describe the
alternative completion-at-point and eudc-expand-try-all behaviours in
the EUDC manual.

Thanks,
Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric Abrahamsen]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Hello,
>
> when switching to a new in-buffer completion user interface package
> which is solely based on completion-at-point, I observed that email
> address completion in message-mode happens in the minibuffer, regardless
> of what UI on top of completion-at-point I use. Also, EUDC's ability to
> aggregate search results from several servers, which was recently added
> with commit 0470a4a939772c4bd25123b15f5eadab41f8bee5, was not kicking
> in, i.e. I was presented results from a single EUDC server only.
>
> Browsing through message.el quickly brought me to the sixth of the 21
> FIXMEs in that file, which reads on line 3183:
>
>   ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
>   ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
>   (add-hook 'completion-at-point-functions #'message-completion-function nil t)
>
> Thus, the most straightforward solution to my problem seemed to make a
> new EUDC function that can be added to completion-at-point-functions,
> and to update message-mode to actually add that function to
> completion-at-point-functions. This is what the below attached patch
> does.

Thank you for working on this! I think everyone can agree that the
current setup is tricky to work within. As the author of (yet another)
contact management package, I do what I assume anyone else would do:
clobber the binding of TAB altogether.

I understand that EUDC has the capability of combining results from
multiple backends, but completion-at-point also has those capabilities,
and personally I would much prefer to see message-mode make use of one
of completion-table combining functions. Better to be wrapped inside one
burrito than two!

The `message-tab' would still need to check where it's been called and
make a decision before calling `completion-at-point', but it seems much
cleaner for each package to provide a function that's suitable for use
in a combined capf completion table, rather than writing an adapter for
EUDC.

What do you think?

Eric

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric S Fraga]

Dear Alexander, 

I am excited about the potential of what you are proposing.

On Wednesday, 13 Apr 2022 at 16:44, Alexander Adolf wrote:
> By default, message-mode binds TAB to message-tab, which in turn calls
> completion-at-point. Depending on your keymap setup it may thus be
> sufficient to type TAB in the "To:" line, or to change your setup to
> replace eudc-expand-try-all with completion-at-point.

This does not seem to work for me.  I do have other capf functions that
I want to use within message mode (for completion in the actual body of
the email) so I have

,----[ C-h v completion-at-point-functions RET ]
| completion-at-point-functions is a variable defined in ‘minibuffer.el’.
| 
| Its value is (eudc-capf-complete cape-ispell cape-dabbrev cape-dict)
| Local in buffer *unsent followup to Alexander Adolf on gmane.emacs.devel*; global value is 
| (tags-completion-at-point-function)
`----

In a To: field, say, hitting TAB gives me dabbrev choices, none of which
are emails.  I've tried with eudc-capf-complete at the end as well.

M-x eudc-expand-try-all RET

does work if I invoke it manually although it seems (using local bbdb
database).  By the way, how can I change the server used by eudc?
(maybe I need to look around)

In an ideal world, i.e. my Holy Grail, I would like to have completion
on the union of mail abbrevs, bbdb, and ecomplete.  Is this possible
somehow?  message-tab gives me none of these, annoyingly.

What am I missing?

Thank you,
eric

-- 
Eric S Fraga with org 9.5.2 in Emacs 29.0.50 on Debian 11.3

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric S Fraga]

On Wednesday, 13 Apr 2022 at 18:44, Eric Abrahamsen wrote:
> I do what I assume anyone else would do: clobber the binding of TAB
> altogether.

But then you use the functionality provided by message-tab when
composing the actual body of the email?  This is more important to me
than address completion, if I had to choose... but obviously would like
to have TAB complete appropriately wherever I am in the massage buffer!

-- 
Eric S Fraga with org 9.5.2 in Emacs 29.0.50 on Debian 11.3

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Eric,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

[...]

> M-x eudc-expand-try-all RET
>
> does work if I invoke it manually although it seems (using local bbdb
> database).  By the way, how can I change the server used by eudc?
> (maybe I need to look around)

To use multiple backends, Customize `eudc-server-hotlist'.  The command
you tried above will give you results from the first entry in the
hotlist.

Additionally, this:

C-u M-x eudc-expand-try-all RET

will give you a list to choose from, of the completions from all servers
in the hotlist.

(There is also M-x eudc-set-server which sets the `eudc-server' and
`eudc-protocol' variables, but you should probably just ignore all them,
and focus on getting `eudc-server-hotlist' configured to your liking.

This is an area I want to reorganize.  Alexander has suggested having
the `eudc-server' and `eudc-protocol' variables "point" into
`eudc-server-hotlist'.  Basically, I'd like the hotlist to be the only
place the user needs to configure backends.  We'll have to be careful to
retain backward compatibility here though.)

> In an ideal world, i.e. my Holy Grail, I would like to have completion
> on the union of mail abbrevs, bbdb, and ecomplete.

Agreed, but also LDAP and macOS contacts, other sources of email
addresses.  And ideally without the user needing to configure
message-mode hooks to change keybindings, by default.  And ideally
introduce all this in a backward compatible way.

> Is this possible somehow?  message-tab gives me none of these,
> annoyingly.

One approach I see working for what you want, would be to add new EUDC
backends for each of mail abbrevs and ecomplete, then in message-mode,
make `eudc-expand-try-all' the default owner of the TAB keybinding when
point is in a To:, CC: or other field that wants an email address.

Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric S Fraga]

Hi Thomas,

thank you for the very helpful response.

On Thursday, 14 Apr 2022 at 09:27, Thomas Fitzsimmons wrote:
> Basically, I'd like the hotlist to be the only place the user needs to
> configure backends.  

This would be good.

> Agreed, but also LDAP and macOS contacts, other sources of email
> addresses.  

Yes, definitely.  I only listed those *I* need/use at the moment ;-).
LDAP would potentially be useful for me as well (although I try to
minimise network dependencies as much as possible) and other systems
will of course be useful to others.

> One approach I see working for what you want, would be to add new EUDC
> backends for each of mail abbrevs and ecomplete, then in message-mode,
> make `eudc-expand-try-all' the default owner of the TAB keybinding when
> point is in a To:, CC: or other field that wants an email address.

Ummm yes, having servers for ecomplete and mail abbrevs would make
sense.

As a first step towards using eudc-expand-try-all as the basis for
completion, I tried changing message-completion-alist (which is what
message-tab uses) to invoke eudc-expand-try-all but this didn't work.
Not sure why.  Will explore and report back as this might be the easiest
approach.

-- 
Eric S Fraga with org 9.5.2 in Emacs 29.0.50 on Debian 11.3

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Stefan Monnier]

> I am trying to limit the cases where eudc-capf-complete "kicks in" to
> those where I think it can be assumed that its results will quite
> certainly desirable. The reason is that completion-at-point works along
> the functions in completion-at-point-functions, and stops when the first
> function returns a completion table. Thus, by calling eudc-capf-complete
> too "aggressively", more useful completion results may be prevented from
> being offered to the user.

Maybe EUDC shouldn't be in charge of providing a CAPF function.
The CAPF function needs to understand the current document's syntax to
know where an email address is expected, but that part is completely
unrelated to EUDC itself (and related to the major mode instead).

E.g. `ecomplete.el` doesn't provide a CAPF function, instead it only
provide a completion table and relies on relevant major modes to provide
a CAPF function that internally uses `ecomplete-completion-table`.

This is used in `message.el` by calling this function from
`message--name-table` which is the function that tries to combine the
various possible backends.

The decision of when/where to use `message--name-table` is made by
`message-completion-alist`.

Side note: `message-expand-name` may call currently (depending on
circumstances) things like `eudc-expand-inline`, `bbdb-complete-name`,
or `expand-abbrev` but this is deprecated and we should get rid of this
part of its code (i.e. move the parts that may still be missing to
`message--name-table`).

We may also want to change `message--expand-name` so that backends
can be added to it without changing its code.


        Stefan

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Wednesday, 13 Apr 2022 at 18:44, Eric Abrahamsen wrote:
>> I do what I assume anyone else would do: clobber the binding of TAB
>> altogether.
>
> But then you use the functionality provided by message-tab when
> composing the actual body of the email?  This is more important to me
> than address completion, if I had to choose... but obviously would like
> to have TAB complete appropriately wherever I am in the massage buffer!

I misremembered! I went and looked at the code, and EBDB does clobber,
but "inside" the `message-tab' function: it pushes a value onto
`message-completion-alist' which matches all the mail-message type
headers, so that `message-expand-name' never gets called.

The whole `message-tab' machinery is awfully complicated for something
that, in the end, just forces you to choose between BBDB and EUDC.

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Stefan Monnier]

> The whole `message-tab' machinery is awfully complicated for something
> that, in the end, just forces you to choose between BBDB and EUDC.

There's a bit more choice than that.  I know of:
- completion of newsgroup names on "Followup-to".
- completion of email addresses (can come from EUDC, BBDD, ecomplete,
  EBDB, and probably more).
- I have a pending patch for Debbugs commands completion.


        Stefan

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The whole `message-tab' machinery is awfully complicated for something
>> that, in the end, just forces you to choose between BBDB and EUDC.
>
> There's a bit more choice than that.  I know of:
> - completion of newsgroup names on "Followup-to".
> - completion of email addresses (can come from EUDC, BBDD, ecomplete,
>   EBDB, and probably more).
> - I have a pending patch for Debbugs commands completion.

What I meant was that I don't see any way of inserting EBDB completion
results into an arbitrary list of other results coming from other user
configuration. If we set `message-expand-name-standard-ui' non-nil then
we get to use the `message--name-table' function, but that function only
consults BBDB, EUDC, and ecomplete. 

I guess I was expecting some sort of setup that uses
`completion-table-merge' on an option holding a bunch of functions. I'd
just provide a EBDB function to be added to that option, and we're done.

All this code makes my eyes cross, though, so I wouldn't be surprised if
I'm missing the right point of injection.

Thanks,
Eric

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Stefan Monnier]

> What I meant was that I don't see any way of inserting EBDB completion
> results into an arbitrary list of other results coming from other user
> configuration. If we set `message-expand-name-standard-ui' non-nil then
> we get to use the `message--name-table' function, but that function only
> consults BBDB, EUDC, and ecomplete. 

Indeed, that's the function that needs to be improved so it takes a list
of "provider functions" and merges the results.

> I guess I was expecting some sort of setup that uses
> `completion-table-merge' on an option holding a bunch of functions. I'd
> just provide a EBDB function to be added to that option, and we're done.

Sounds about right.


        Stefan

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello Thomas,

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> [...]
> It looks like the default binding for TAB in message-mode-map is
> message-tab; so I tested by rebinding TAB to message-tab.  It worked,
> except I was expecting case sensitive results.  Can you please change
> `completion-ignore-case' to nil?  And figure out what to do with
> `completion-table-with-cache''s IGNORE-CASE argument accordingly, if
> necessary?

Ah, well spotted! Thanks for noticing; consider it fixed. "Thou shalt
not impose your will on users more than necessary." ;-)

I have simply removed completion-ignore-case from the setq-local call,
as it shouldn't have been there (probably a left-over from the first
attempts when I was still trying to force things).

The IGNORE-CASE argument to completion-table-with-cache should stay at
t, as it signals whether the function being called
(eudc-query-with-words) is case sensitive (it is not).

> I had to run (message-tab) repeatedly to get the final result if
> multiple results were available, whereas with eudc-expand-try-all bound
> to TAB, I get a "Multiple matches found; choose one:" prompt.  I guess
> that's just a different UI style for completion-at-point?  FWIW, I think
> I prefer eudc-expand-try-all's behaviour.

Does changing the value of the variable message-expand-name-standard-ui
change anything for you? There are a couple of completion-specific
variables changing the behaviour [1], and 3rd party packages might have
an effect, too.

[1] https://www.gnu.org/software/emacs/manual/html_node/emacs/Completion-Options.html

That said, the patch leaves it entirely to the user to configure the
user interface and its behaviour.

> I have one bbdb entry and one LDAP server in the hotlist.  It seems like
> the LDAP operation happens every time I call (message-tab).  Is that
> expected? 

Yes.

> Is the completion table caching meant to prevent this? 

Yes.

> I don't like the idea that hitting TAB several times per email address
> will result in several calls to the LDAP server. Can
> completion-at-point be made to have the same behaviour as
> eudc-expand-try-all, which only results in each backend being queried
> once?
> [...]

That's what I had hoped would be the result of using the function
completion-table-with-cache. From the docstring: "...saves the last
argument-result pair from FUN, so that several lookups with the same
argument (or with an argument that starts with the first one) only need
to call FUN once." The thing is, I cannot control how many times the
function completion-at-point calls functions on
completion-at-point-functions, nor how often completion-at-point is
called itself, which is determined by the user-interface on top of
completion-at-point.

If anyone gas further insight, any hints are highly appreciated.


  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello Eric,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> [...]
> I am excited about the potential of what you are proposing.

Cool!

> On Wednesday, 13 Apr 2022 at 16:44, Alexander Adolf wrote:
>> By default, message-mode binds TAB to message-tab, which in turn calls
>> completion-at-point. Depending on your keymap setup it may thus be
>> sufficient to type TAB in the "To:" line, or to change your setup to
>> replace eudc-expand-try-all with completion-at-point.
>
> This does not seem to work for me.  I do have other capf functions that
> I want to use within message mode (for completion in the actual body of
> the email) so I have
> 
> ,----[ C-h v completion-at-point-functions RET ]
> | completion-at-point-functions is a variable defined in ‘minibuffer.el’.
> | 
> | Its value is (eudc-capf-complete cape-ispell cape-dabbrev cape-dict)
> | Local in buffer *unsent followup to Alexander Adolf on gmane.emacs.devel*; global value is 
> | (tags-completion-at-point-function)
> `----
> [...]
>
> What am I missing?

message-mode has some - well, let's call them specifics around
completion, which might perhaps deserve a separate discussion (IMHO
anyway).

As you are using the cape package, are you by any chance also using the
corfu package? If yes, with the value of completion-at-point-functions
you have, you should get email addresses in the To:, and From: lines via
corfu.

In any case, if you move point to the To: line, type a substring for
which you know there will be a match in EUDC, and then do
"M-: (completion-at-point)", what happens?


  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> [...]
>> One approach I see working for what you want, would be to add new EUDC
>> backends for each of mail abbrevs and ecomplete, then in message-mode,
>> make `eudc-expand-try-all' the default owner of the TAB keybinding when
>> point is in a To:, CC: or other field that wants an email address.
>
> Ummm yes, having servers for ecomplete and mail abbrevs would make
> sense.
> [...]

Once this patch will have been pushed, I'll give it a try to write EUDC
back-ends for these. Shouldn't be hard.


  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> [...]
> Maybe EUDC shouldn't be in charge of providing a CAPF function.

I'd tend to disagree.

> The CAPF function needs to understand the current document's syntax to
> know where an email address is expected, but that part is completely
> unrelated to EUDC itself (and related to the major mode instead).
> [...]

Fully agree.

I think there may be a middle ground. I appreciate you seem to share the
idea of following "separation of concerns", some also call it
"one-thing-well mantra" (doe one thing only, do it well), as it
typically helps keeping code simpler.

A couple of thoughts on completion in message mode:

---------------------------- Begin Quote -----------------------------
Variable: completion-at-point-functions

     The value of this abnormal hook should be a list of functions,
     which are used to compute a completion table (see Basic Completion)
     for completing the text at point. It can be used by major modes to
     provide mode-specific completion tables (see Major Mode
     Conventions).
----------------------------- End Quote ------------------------------

But I think there may be a "catch 22" here, though. If the major mode
provides the completion table(s) itself, fine. But what about completion
tables contributed by 3rd party packages? I don't think a major mode
could, or should have an, inevitably incomplete, list of 3rd party
packages to query for completion tables. IMO it would seem much more
sensible to reverse responsibilities: any package that is capable of
providing additional completion tables for certain major modes, should
register itself with those modes (e.g. by adding itself to
completion-at-point-functions).

In message-mode (there may be others, too), the in-buffer completion
will additionally need to figure out the context where completion is
attempted, i.e. some function will need to be called to determine the
semantic context of the stuff around point in order to choose one or
more suitable functions for delivering completion candidates. For
instance in a programming language buffer, do I need candidates for a
function, or for a variable? For instance in a message buffer, do I need
candidates for an email address, or text snippets for the message body?

In the current architecture for completion-at-point, such checks for
context have to be performed by the functions registered in
completion-at-point-functions. This seems wasteful, as it means a lot of
very similar code all over (possibly with subtly different bugs in each
copy), and an unnecessary runtime burden as completion-at-point may be
called often (for every keystroke in some situations).

mail-mode and message-mode are sort of half way there by providing
mail-complete-alist and message-completion-alist, respectively. Both
variables allow for registering a _single_ function to be called for
buffer lines matching a regex. This already takes the burden of figuring
out the completion context away from the function being called, and
together with the try-all-servers feature recently added to EUDC it
would already suffice to aggregate email address candidates from more
than one source in message-mode.

A more generic mechanism could look like this:

1) Each 3rd party package that wants to provide general purpose
   completion tables for a given major mode, adds a function to the
   respective *-mode-hook that adds that package's completion function
   to completion-at-point-functions (this is current good practice).

2) Each major mode that wants to allow users to have different
   completion tables based on the context where the completion happens,
   offers a *-mode-context-completion-functions-alist variable. The keys
   in this alist should be symbols describing the completion context or
   purpose (for example 'email, 'newsgroup, etc.). The values are
   symbols representing abnormal hook variables. (This is where
   mail-mode and message-mode are half way there.)

3) Users (or init code of 3rd party packages) call add-hook to add
   completion functions to the values of entries in
   *-mode-context-completion-functions-alist variable(s) as they deem
   fit. These functions must be "cooperative" in the sense that they
   should add ":exclusive 'no" to the completion table they return,
   unless there is a particular reason for claiming exclusive completion
   "rights".

4) Each major mode that wants to allow users to have different
   completion tables based on the context where the completion happens,
   does the following:

    4.1) After all mode hooks have been run, sample the current value of
	 completion-at-point-functions, and store it in a variable.

    4.2) Next, set completion-at-point-functions to a single function,
	 which is the mode's completion function.

    4.3) When completion-at-point is called, and hence in the mode's
	 completion function:

	4.3.1) Check whether point is in any of specific completion
	       context.

	4.3.2) If point is in a specific context, consult the
	       *-mode-context-completion-functions-alist variable to
	       determine the list of functions to call. If that is not
	       the case, use the previously sampled value of
	       completion-at-point-functions.

	4.3.3) Do a let-binding setting completion-at-point-functions to
	       the value determined in the previous step, and then call
	       completion-at-point (recursive) within the let context,
	       and return whatever completion-at-point returned. (Can
	       completion-at-point be called recursively? I haven't
	       tried.)

5) Job done.


Apologies for the lengthy explanation...


  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello Eric,

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> [...]
> Thank you for working on this!

It seems several people have already been scratching their heads about
this. I just happen to be the first one to stick it out the window...
;-)

> I think everyone can agree that the current setup is tricky to work
> within. 

Very true.

> As the author of (yet another) contact management package, 

Curious; which one?

> I do what I assume anyone else would do: clobber the binding of TAB
> altogether.

Agree.

> I understand that EUDC has the capability of combining results from
> multiple backends,

Yep.

> but completion-at-point also has those capabilities,

I had hoped that ":exclusive 'no" would do this, but haven't been able
to meet my expectations.

Any hints how to go about this?

As an aside, in minibuffer.el, in function completion--capf-wrapper, the
comment on line 2559 states a caveat on ":exclusive 'no":
---------------------------- Begin Quote -----------------------------
;; FIXME: Here we'd need to decide whether there are
;; valid completions against the current text.  But this depends
;; on the actual completion UI (e.g. with the default completion
;; it depends on completion-style) ;-(
;; We approximate this result by checking whether prefix
;; completion might work, which means that non-prefix completion
;; will not work (or not right) for completion functions that
;; are non-exclusive.
----------------------------- End Quote ------------------------------

> and personally I would much prefer to see message-mode make use of one
> of completion-table combining functions. Better to be wrapped inside
> one burrito than two!
>
> The `message-tab' would still need to check where it's been called and
> make a decision before calling `completion-at-point', but it seems much
> cleaner for each package to provide a function that's suitable for use
> in a combined capf completion table, rather than writing an adapter for
> EUDC.
>
> What do you think?
> [...]

I basically agree with your assessment, but have come to a slightly
different conclusion with respect to what to do about it. See my
first response to Stefan Monnier on this thread for more details.


Cheers,

  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> [...]
> What I meant was that I don't see any way of inserting EBDB completion
> results into an arbitrary list of other results coming from other user
> configuration. If we set `message-expand-name-standard-ui' non-nil then
> we get to use the `message--name-table' function, but that function only
> consults BBDB, EUDC, and ecomplete. 
>
> I guess I was expecting some sort of setup that uses
> `completion-table-merge' on an option holding a bunch of functions. I'd
> just provide a EBDB function to be added to that option, and we're done.
> [...]

If it's for email addresses, EUDC's result merging does the job:

notmuch-address  \          
	   BBDB  |          
	   LDAP   > --> EUDC --+
 macOS Contacts  |             |
	    ...  /             V
			       |
      +----------<-------------+
      |   
      V                               /  completing-read
      |                              |   corfu          
      + --> completion-at-point --> <    company        
				     |   ...            
				      \


Provide an EUDC back-end for what you need, and the patch I proposed
provides your results to message-mode.

For a more general discussion, see
id:dc51ed8dcb5b1d44bf51859758d120ba@condition-alpha.com on this very
same thread.


Looking forward to your thoughts,

  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric Abrahamsen]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Hello Stefan,
>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> [...]
>> Maybe EUDC shouldn't be in charge of providing a CAPF function.
>
> I'd tend to disagree.
>
>> The CAPF function needs to understand the current document's syntax to
>> know where an email address is expected, but that part is completely
>> unrelated to EUDC itself (and related to the major mode instead).
>> [...]
>
> Fully agree.
>
> I think there may be a middle ground. I appreciate you seem to share the
> idea of following "separation of concerns", some also call it
> "one-thing-well mantra" (doe one thing only, do it well), as it
> typically helps keeping code simpler.
>
> A couple of thoughts on completion in message mode:
>
> ---------------------------- Begin Quote -----------------------------
> Variable: completion-at-point-functions
>
>      The value of this abnormal hook should be a list of functions,
>      which are used to compute a completion table (see Basic Completion)
>      for completing the text at point. It can be used by major modes to
>      provide mode-specific completion tables (see Major Mode
>      Conventions).
> ----------------------------- End Quote ------------------------------
>
> But I think there may be a "catch 22" here, though. If the major mode
> provides the completion table(s) itself, fine. But what about completion
> tables contributed by 3rd party packages? I don't think a major mode
> could, or should have an, inevitably incomplete, list of 3rd party
> packages to query for completion tables. IMO it would seem much more
> sensible to reverse responsibilities: any package that is capable of
> providing additional completion tables for certain major modes, should
> register itself with those modes (e.g. by adding itself to
> completion-at-point-functions).
>
> In message-mode (there may be others, too), the in-buffer completion
> will additionally need to figure out the context where completion is
> attempted, i.e. some function will need to be called to determine the
> semantic context of the stuff around point in order to choose one or
> more suitable functions for delivering completion candidates. For
> instance in a programming language buffer, do I need candidates for a
> function, or for a variable? For instance in a message buffer, do I need
> candidates for an email address, or text snippets for the message body?
>
> In the current architecture for completion-at-point, such checks for
> context have to be performed by the functions registered in
> completion-at-point-functions. This seems wasteful, as it means a lot of
> very similar code all over (possibly with subtly different bugs in each
> copy), and an unnecessary runtime burden as completion-at-point may be
> called often (for every keystroke in some situations).
>
> mail-mode and message-mode are sort of half way there by providing
> mail-complete-alist and message-completion-alist, respectively. Both
> variables allow for registering a _single_ function to be called for
> buffer lines matching a regex. This already takes the burden of figuring
> out the completion context away from the function being called, and
> together with the try-all-servers feature recently added to EUDC it
> would already suffice to aggregate email address candidates from more
> than one source in message-mode.
>
> A more generic mechanism could look like this:
>
> 1) Each 3rd party package that wants to provide general purpose
>    completion tables for a given major mode, adds a function to the
>    respective *-mode-hook that adds that package's completion function
>    to completion-at-point-functions (this is current good practice).
>
> 2) Each major mode that wants to allow users to have different
>    completion tables based on the context where the completion happens,
>    offers a *-mode-context-completion-functions-alist variable. The keys
>    in this alist should be symbols describing the completion context or
>    purpose (for example 'email, 'newsgroup, etc.). The values are
>    symbols representing abnormal hook variables. (This is where
>    mail-mode and message-mode are half way there.)
>
> 3) Users (or init code of 3rd party packages) call add-hook to add
>    completion functions to the values of entries in
>    *-mode-context-completion-functions-alist variable(s) as they deem
>    fit. These functions must be "cooperative" in the sense that they
>    should add ":exclusive 'no" to the completion table they return,
>    unless there is a particular reason for claiming exclusive completion
>    "rights".
>
> 4) Each major mode that wants to allow users to have different
>    completion tables based on the context where the completion happens,
>    does the following:
>
>     4.1) After all mode hooks have been run, sample the current value of
> 	 completion-at-point-functions, and store it in a variable.
>
>     4.2) Next, set completion-at-point-functions to a single function,
> 	 which is the mode's completion function.
>
>     4.3) When completion-at-point is called, and hence in the mode's
> 	 completion function:
>
> 	4.3.1) Check whether point is in any of specific completion
> 	       context.
>
> 	4.3.2) If point is in a specific context, consult the
> 	       *-mode-context-completion-functions-alist variable to
> 	       determine the list of functions to call. If that is not
> 	       the case, use the previously sampled value of
> 	       completion-at-point-functions.
>
> 	4.3.3) Do a let-binding setting completion-at-point-functions to
> 	       the value determined in the previous step, and then call
> 	       completion-at-point (recursive) within the let context,
> 	       and return whatever completion-at-point returned. (Can
> 	       completion-at-point be called recursively? I haven't
> 	       tried.)
>
> 5) Job done.
>
>
> Apologies for the lengthy explanation...

Hi Alexander,

I'll respond here, though I've read your other replies in other
subthreads.

First of all, my apologies for an attempted hijacking of the thread! But
you're right this is something that has irked me for a long time, I just
haven't gotten around to writing code, and it's easier to yell about it
on this thread than go write my own patch :)

While I don't object to EUDC at all (and have a half-written patch
somewhere to adapt EBDB to EUDC, as well), I don't think it's necessary
to funnel all email completion through it. The mechanism already exists
in the completion-at-point code, and if that doesn't work the way we
want it to, I think it should be tweaked until it does, because this is
a common use case.

We already have `message-expand-name-databases', and it is merely a bit
of historical perversity that only two values are allowed in there. It
should accept any appropriate function.

`message-tab' will always be necessary for checking where in the message
buffer we are, and feeding the correct string to the completion tables.

`completion-table-merge' is the only missing piece. It collections all
potential completions from all tables and, as the name suggests, merges
them.

I think that's all we need. Users should certainly be free to only use
EUDC if they want, but then all they have to do is funnel their various
functions into EUDC, and set `message-expand-name-databases' to '(eudc).
I don't see any need to enforce the use of EUDC, and think that
message.el should not (and should not need to) make any mention of
specific contact-management packages at all.

That's my point of view!

Thanks,
Eric

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Stefan Monnier]

> I had hoped that ":exclusive 'no" would do this, but haven't been able
> to meet my expectations.

Better stay far away from ":exclusive 'no".
It's an ugly hack that's fundamentally broken (not just in its implementation).

If you want to merge several completion tables (or several CAPFs) then
you should do so explicitly (for email completion, I think the best
option is probably to do it at the level of the completion table by providing
a new table which calls `all-completions` on all the other tables and
appends the results).


        Stefan

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric S Fraga]

On Friday, 15 Apr 2022 at 23:39, Alexander Adolf wrote:
> Once this patch will have been pushed, I'll give it a try to write EUDC
> back-ends for these. Shouldn't be hard.

Looking forward to these!  Thank you.

-- 
Eric S Fraga with org 9.5.2 in Emacs 29.0.50 on Debian 11.3

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric S Fraga]

On Friday, 15 Apr 2022 at 23:35, Alexander Adolf wrote:
> In any case, if you move point to the To: line, type a substring for
> which you know there will be a match in EUDC, and then do
> "M-: (completion-at-point)", what happens?

I get completion using one of the other sources, not eudc.

In any case, I have now changed the behaviour of tab to always use eudc
(if in an appropriate header line), with abbrev and ecomplete invoked
with other keys (SPC or comma and M-n respectively).  This works
effectively for me and will continue to work if/when eudc includes the
other sources of addresses.

Thank you,
eric

-- 
Eric S Fraga with org 9.5.2 in Emacs 29.0.50 on Debian 11.3

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Eric,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 15 Apr 2022 at 23:35, Alexander Adolf wrote:
>> In any case, if you move point to the To: line, type a substring for
>> which you know there will be a match in EUDC, and then do
>> "M-: (completion-at-point)", what happens?
>
> I get completion using one of the other sources, not eudc.
>
> In any case, I have now changed the behaviour of tab to always use eudc
> (if in an appropriate header line), with abbrev and ecomplete invoked
> with other keys (SPC or comma and M-n respectively).  This works
> effectively for me and will continue to work if/when eudc includes the
> other sources of addresses.

Glad to hear you got a satisfactory setup.

Interesting idea of multiple keybindings for different completions.

If EUDC includes the other sources, do you foresee using just EUDC?  Or
do you think keeping the extra keys will give better usability?

What do you think the "out-of-the-box" behavior should be?  Should the
following:

    "emacs -Q"
    C-x m
    TAB

say "no completions found; to start adding contacts, do <something>".

Should the following:

    "emacs -Q"
    (in *scratch*) emacs-devel@gnu.org
    C-x m
    TAB

expand to emacs-devel@gnu.org via dabbrev-expand?

Pressing TAB in both those examples on Emacs 25.1 does nothing, and on
master inserts a literal TAB.

Thanks,
Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Eric S Fraga]

On Sunday, 17 Apr 2022 at 09:58, Thomas Fitzsimmons wrote:
> Interesting idea of multiple keybindings for different completions.

Yes.  The combination works because (a) abbrev is for those very
commonly used addresses so having them expand on , or SPC means no
thought at all, (b) ecomplete shows possibilities as I type so I will
hit M-n if/when I see what I want, and (c) eudc (TAB) works any time to
complete by typing incrementally.

> If EUDC includes the other sources, do you foresee using just EUDC?  Or
> do you think keeping the extra keys will give better usability?

Not yet sure; my initial view had been that TAB completion (i.e. via
EUDC) would be what I needed for all cases but I'm actually quite
comfortable with what I've ended up with.  However, given that I use tab
completion everywhere else, it probably will be better to have
everything as an option via eudc.

> What do you think the "out-of-the-box" behavior should be?  Should the
> following:
>
>     "emacs -Q"
>     C-x m
>     TAB
>
> say "no completions found; to start adding contacts, do <something>".

No, I would expect it to provide all my possible email addresses: union
of bbdb, abbrevs, and ecomplete, just as M-x does for me right now with
selectrum+consult, with most recently used or most popular first.

>     (in *scratch*) emacs-devel@gnu.org
>     C-x m
>     TAB
>
> expand to emacs-devel@gnu.org via dabbrev-expand?

Ummm, interesting.  No idea what I would expect!

-- 
Eric S Fraga with org 9.5.2 in Emacs 29.0.50 on Debian 11.3

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

Hello,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> [...]
> Browsing through message.el quickly brought me to the sixth of the 21
> FIXMEs in that file, which reads on line 3183:
>
>   ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
>   ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
>   (add-hook 'completion-at-point-functions #'message-completion-function nil t)
> [...]

Thanks to all who cared to share their views, and their suggestions on
my proposed patch. I am attaching an updated version that includes the
improvements that were suggested:

 - remove override, and thus respect user's setting of
   completion-ignore-case (Thomas Fitzsimmons's comment)

 - do not use ":exclusive 'no" in the completion table (Stefan Monnier's
   comment)

Thanks for pointing these out, and for helping to improve the proposed
patch!

Then there was a wider discussion on potential, architectural
improvements of completion handling in message.el.

Before diving into this wider discussion, it would seem worthwhile to me
trying to get to a conclusion on the proposed patch.

Do we have a consensus, that the proposed, updated patch does not solve
all potential issues around email address completion in message mode
once and for all, but that it still provides a useful improvement for
some users without disturbing others, thus making it seem worthwhile to
include the patch?


Many thanks and looking forward to your thoughts,

  --alexander


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-EUDC-email-addresses-via-completion-at-point-in-mess.patch --]
[-- Type: text/x-patch, Size: 10050 bytes --]

From 0954a5be3e01283eeb0e56059c6210f0c22037ff Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Tue, 26 Apr 2022 15:38:12 +0200
Subject: [PATCH] EUDC email addresses via completion-at-point in message-mode

* lisp/net/eudc-capf.el (new file): Add new 'eudc-capf-complete'
function.
* lisp/gnus/message.el (message-mode): Add 'eudc-capf-complete' to
'completion-at-point-functions' when a 'message-mode' buffer is
created, removing the FIXME.
* doc/misc/eudc.texi (Inline Query Expansion): Add a new subsection,
describing the new 'completion-at-point' mechanism in 'message-mode'.
* etc/NEWS (EUDC): Describe the new 'completion-at-point' method.
---
 doc/misc/eudc.texi    |  23 ++++++++
 etc/NEWS              |   6 ++
 lisp/gnus/message.el  |   4 +-
 lisp/net/eudc-capf.el | 127 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 158 insertions(+), 2 deletions(-)
 create mode 100644 lisp/net/eudc-capf.el

diff --git a/doc/misc/eudc.texi b/doc/misc/eudc.texi
index d2850282fe..d2a1efeed0 100644
--- a/doc/misc/eudc.texi
+++ b/doc/misc/eudc.texi
@@ -713,6 +713,7 @@ be passed to the program.
 
 @node Inline Query Expansion
 @section Inline Query Expansion
+@subsection Inline Query Expansion Using a Key Binding
 
 Inline query expansion is a powerful method to get completion from
 your directory servers.  The most common usage is for expanding names
@@ -885,6 +886,28 @@ An error is signaled.  The expansion aborts.
 Default is @code{select}
 @end defvar
 
+@subsection Inline Query Expansion Using completion-at-point
+
+In addition to providing a dedicated EUDC function for binding to a
+key shortcut (@pxref{Inline Query Expansion Using a Key Binding}),
+EUDC also provides a function to contribute search results to the
+Emacs in-buffer completion system available via the function
+@code{completion-at-point} (@pxref{Identifier
+Inquiries,,,maintaining}) in @code{message-mode} buffers
+(@pxref{Message}).  When using this mechanism, queries are made in the
+multi-server query mode of operation (@pxref{Multi-server Queries}).
+
+When a buffer in @code{message-mode} is created, EUDC's inline
+expansion function is automatically added to the variable
+@code{completion-at-point-functions}.  As a result, whenever
+@code{completion-at-point} is invoked in a @code{message-mode} buffer,
+EUDC will be queried for email addresses matching the words before
+point.  Since this will be useful only when editing specific message
+header fields that require specifying one or more email addresses, an
+additional check is performed whether point is actually in one of
+those header fields.  Thus, any matching email addresses will be
+offered for completion in suitable message header fields only, and not
+in other places, like for example the body of the message.
 
 
 @node The Server Hotlist
diff --git a/etc/NEWS b/etc/NEWS
index dc2e7c616a..1ebbaa76e8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -882,6 +882,12 @@ is called, and the returned values are used to populate the phrase and
 comment parts (see RFC 5322 for definitions).  In both cases, the
 phrase part will be automatically quoted if necessary.
 
++++
+*** New function 'eudc-capf-complete' with message-mode integration
+EUDC can now contribute email addresses to 'completion-at-point' by
+adding the new function 'eudc-capf-complete' to
+'completion-at-point-functions' in message-mode.
+
 ** eww/shr
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index cc994d3ba5..b59ab2c31d 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -51,6 +51,7 @@
 (require 'yank-media)
 (require 'mailcap)
 (require 'sendmail)
+(require 'eudc-capf)
 
 (autoload 'mailclient-send-it "mailclient")
 
@@ -3180,8 +3181,7 @@ Like `text-mode', but with these additional commands:
     (mail-abbrevs-setup))
    ((message-mail-alias-type-p 'ecomplete)
     (ecomplete-setup)))
-  ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
-  ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
+  (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
   (add-hook 'completion-at-point-functions #'message-completion-function nil t)
   (unless buffer-file-name
     (message-set-auto-save-file-name))
diff --git a/lisp/net/eudc-capf.el b/lisp/net/eudc-capf.el
new file mode 100644
index 0000000000..b11ef8776b
--- /dev/null
+++ b/lisp/net/eudc-capf.el
@@ -0,0 +1,127 @@
+;;; eudc-capf.el --- EUDC - completion-at-point bindings  -*- lexical-binding:t -*-
+
+;; Copyright (C) 2022 Free Software Foundation, Inc.
+;;
+;; Author: Alexander Adolf
+;;
+;; This file is part of GNU Emacs.
+;;
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;;    This library provides functions to deliver email addresses from
+;;    EUDC search results to `completion-at-point'.
+;;
+;;    Email address completion will likely be desirable only in
+;;    situations where designating email recipients plays a role, such
+;;    as when composing or replying to email messages, or when posting
+;;    to newsgroups, possibly with copies of the post being emailed.
+;;    Hence, modes relevant in such contexts, such as for example
+;;    `message-mode' and `mail-mode', often at least to some extent
+;;    provide infrastructure for different functions to be called when
+;;    completing in certain message header fields, or in the body of
+;;    the message.  In other modes for editing email messages or
+;;    newsgroup posts, which do not provide such infrastructure, any
+;;    completion function providing email addresses will need to check
+;;    whether the completion attempt occurs in an appropriate context
+;;    (that is, in a relevant message header field) before providing
+;;    completion candidates.  Two mechanisms are thus provided by this
+;;    library.
+;;
+;;    The first mechanism is intended for use by the modes listed in
+;;    `eudc-capf-modes', and relies on these modes adding
+;;    `eudc-capf-complete' to `completion-at-point-functions', as
+;;    would be usually done for any general-purpose completion
+;;    function.  In this mode of operation, and in order to offer
+;;    email addresses only in contexts where the user would expect
+;;    them, a check is performed whether point is on a line that is a
+;;    message header field suitable for email addresses, such as for
+;;    example "To:", "Cc:", etc.
+;;
+;;    The second mechanism is intended for when the user modifies
+;;    `message-completion-alist' to replace `message-expand-name' with
+;;    the function `eudc-capf-message-expand-name'.  As a result,
+;;    minibuffer completion (`completing-read') for email addresses
+;;    would no longer enabled in `message-mode', but
+;;    `completion-at-point' (in-buffer completion) only.
+
+;;; Usage:
+
+;;    (require 'eudc-capf)
+;;    (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
+
+;;; Code:
+
+(require 'eudc)
+
+(defvar message-email-recipient-header-regexp)
+(defvar mail-abbrev-mode-regexp)
+(declare-function mail-abbrev-in-expansion-header-p "mailabbrev" ())
+
+(defconst eudc-capf-modes '(message-mode) "List of modes in which email \
+address completion is to be attempted.")
+
+;; completion functions
+
+;;;###autoload
+(defun eudc-capf-complete ()
+  "Email address completion function for `completion-at-point-functions'.
+
+This function checks whether the current major mode is one of the
+modes listed in `eudc-capf-modes', and whether point is on a line
+with a message header listing email recipients, that is, a line
+whose beginning matches `message-email-recipient-header-regexp',
+and, if the check succeeds, searches for records matching the
+words before point.
+
+The return value is either nil when no match is found, or a
+completion table as required for functions listed in
+`completion-at-point-functions'."
+  (if (and (seq-some #'derived-mode-p eudc-capf-modes)
+           (let ((mail-abbrev-mode-regexp message-email-recipient-header-regexp))
+             (mail-abbrev-in-expansion-header-p)))
+      (eudc-capf-message-expand-name)
+    nil))
+
+;;;###autoload
+(defun eudc-capf-message-expand-name ()
+  "Email address completion function for `message-completion-alist'.
+
+When this function is added to `message-completion-alist',
+replacing any existing entry for `message-expand-name' there,
+with an appropriate regular expression such as for example
+`message-email-recipient-header-regexp', then EUDC will be
+queried for email addresses, and the results delivered to
+`completion-at-point'."
+  (if (or (and (boundp 'eudc-server) eudc-server)
+          (and (boundp 'eudc-server-hotlist) eudc-server-hotlist))
+      (progn
+        (setq-local completion-styles '(substring partial-completion))
+        (let* ((beg (save-excursion
+                      (if (re-search-backward "\\([:,]\\|^\\)[ \t]*"
+                                              (point-at-bol) 'move)
+                          (goto-char (match-end 0)))
+                      (point)))
+               (end (point))
+               (prefix (save-excursion (buffer-substring-no-properties beg end))))
+          (list beg end
+                (completion-table-with-cache
+                 (lambda (_)
+                   (eudc-query-with-words (split-string prefix "[ \t]+") t))
+                 t))))
+    nil))
+
+(provide 'eudc-capf)
+;;; eudc-capf.el ends here
-- 
2.36.0

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Filipp Gunbin]

I've only skimmed through previous discussion, but will point out a few
things in the patch.

Thanks.

On 26/04/2022 16:39 +0200, Alexander Adolf wrote:

> +;;; Usage:
> +
> +;;    (require 'eudc-capf)
> +;;    (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)

As you're showing add-hook with LOCAL t, perhaps you should also show
some context, to avoid users pasting this directly into their .emacs.

> +(defconst eudc-capf-modes '(message-mode) "List of modes in which email \
> +address completion is to be attempted.")

Put docstring on its own line.

> +  (if (and (seq-some #'derived-mode-p eudc-capf-modes)
> +           (let ((mail-abbrev-mode-regexp message-email-recipient-header-regexp))
> +             (mail-abbrev-in-expansion-header-p)))
> +      (eudc-capf-message-expand-name)

You don't need to specify else-branch for an if.  Just omit it.  (same
below)

> +`completion-at-point'."
> +  (if (or (and (boundp 'eudc-server) eudc-server)
> +          (and (boundp 'eudc-server-hotlist) eudc-server-hotlist))

You're requiring eudc, why should these be unbound?

> +      (progn
> +        (setq-local completion-styles '(substring partial-completion))
> +        (let* ((beg (save-excursion
> +                      (if (re-search-backward "\\([:,]\\|^\\)[ \t]*"
> +                                              (point-at-bol) 'move)

t instead of 'move?

I also don't like the side-effect of setting completion-styles for the
user, although only locally.  Can it be done in some other way?

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 3590 bytes --]

Hello Filipp,

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> I've only skimmed through previous discussion, but will point out a few
> things in the patch.

Thanks for your helpful review! Updated patch at the very bottom of this
message.

> [...]
>> +;;; Usage:
>> +
>> +;;    (require 'eudc-capf)
>> +;;    (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
>
> As you're showing add-hook with LOCAL t, perhaps you should also show
> some context, to avoid users pasting this directly into their .emacs.

Good point. I've added a few more words of comment.

>> +(defconst eudc-capf-modes '(message-mode) "List of modes in which email \
>> +address completion is to be attempted.")
>
> Put docstring on its own line.

Done.

>> +  (if (and (seq-some #'derived-mode-p eudc-capf-modes)
>> +           (let ((mail-abbrev-mode-regexp message-email-recipient-header-regexp))
>> +             (mail-abbrev-in-expansion-header-p)))
>> +      (eudc-capf-message-expand-name)
>
> You don't need to specify else-branch for an if.  Just omit it.  (same
> below)

Ah, well spotted. Removed.

>> +`completion-at-point'."
>> +  (if (or (and (boundp 'eudc-server) eudc-server)
>> +          (and (boundp 'eudc-server-hotlist) eudc-server-hotlist))
>
> You're requiring eudc, why should these be unbound?

Another well spotted one. I borrowed this fragment from some other code
I wrote, where eudc is not being required. Boundp removed.

>> +      (progn
>> +        (setq-local completion-styles '(substring partial-completion))
>> +        (let* ((beg (save-excursion
>> +                      (if (re-search-backward "\\([:,]\\|^\\)[ \t]*"
>> +                                              (point-at-bol) 'move)
>
> t instead of 'move?

Hm. Quoting from the docstring (re-search-forward in this case):
---------------------------- Begin Quote -----------------------------
The optional third argument NOERROR indicates how errors are handled
  when the search fails. If it is nil or omitted, emit an error; if it
  is t, simply return nil and do nothing; if it is neither nil nor t,
  move to the limit of search and return nil.
----------------------------- End Quote ------------------------------

This reads as if t ("do nothing") vs. 'move ("move to the limit of
search") should make a difference? A few quick experiments seem to
indicate that in practice both seem to behave the same though. In this
light, I'm fine with changing 'move to t.

> I also don't like the side-effect of setting completion-styles for the
> user, although only locally. 

Me neither.

> Can it be done in some other way?

Short answer: I don't think so.

Longer answer:

The motivation for putting this is that completion-at-point filters the
completion tables based on completion-styles. Thus, if completion-styles
is set too restrictive, useful results may not be offered to the user.
The default completion-styles in message-mode is '(basic
partial-completion emacs22). I wanted 'substring to be the primary as it
matches the search term anywhere within the candidate, as opposed to
'basic which matches it at the beginning only. In this respect, you
might view 'substring as a super-set of 'basic.

That said, it would probably be desirable for message mode to have
different values for both, completion-at-point-functions and
completion-styles, depending on where point is (email header, newsgroup
header, message body, etc.). But this seems like a wider discussion
about the architecture of message.el rather than this patch.


Many thanks and looking forward to your thoughts,

  --alexander



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-EUDC-email-addresses-via-completion-at-point-in-mess.patch --]
[-- Type: text/x-patch, Size: 10468 bytes --]

From ad3199033f0e317d91098a80dbc5ee7583d83443 Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Thu, 28 Apr 2022 19:12:10 +0200
Subject: [PATCH] EUDC email addresses via completion-at-point in message-mode

* lisp/net/eudc-capf.el (new file): Add new 'eudc-capf-complete'
function.
* lisp/gnus/message.el (message-mode): Add 'eudc-capf-complete' to
'completion-at-point-functions' when a 'message-mode' buffer is
created, removing the FIXME.
* doc/misc/eudc.texi (Inline Query Expansion): Add a new subsection,
describing the new 'completion-at-point' mechanism in 'message-mode'.
* etc/NEWS (EUDC): Describe the new 'completion-at-point' method.
---
 doc/misc/eudc.texi    |  23 +++++++
 etc/NEWS              |   6 ++
 lisp/gnus/message.el  |   4 +-
 lisp/net/eudc-capf.el | 136 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 167 insertions(+), 2 deletions(-)
 create mode 100644 lisp/net/eudc-capf.el

diff --git a/doc/misc/eudc.texi b/doc/misc/eudc.texi
index d2850282fe..d2a1efeed0 100644
--- a/doc/misc/eudc.texi
+++ b/doc/misc/eudc.texi
@@ -713,6 +713,7 @@ be passed to the program.
 
 @node Inline Query Expansion
 @section Inline Query Expansion
+@subsection Inline Query Expansion Using a Key Binding
 
 Inline query expansion is a powerful method to get completion from
 your directory servers.  The most common usage is for expanding names
@@ -885,6 +886,28 @@ An error is signaled.  The expansion aborts.
 Default is @code{select}
 @end defvar
 
+@subsection Inline Query Expansion Using completion-at-point
+
+In addition to providing a dedicated EUDC function for binding to a
+key shortcut (@pxref{Inline Query Expansion Using a Key Binding}),
+EUDC also provides a function to contribute search results to the
+Emacs in-buffer completion system available via the function
+@code{completion-at-point} (@pxref{Identifier
+Inquiries,,,maintaining}) in @code{message-mode} buffers
+(@pxref{Message}).  When using this mechanism, queries are made in the
+multi-server query mode of operation (@pxref{Multi-server Queries}).
+
+When a buffer in @code{message-mode} is created, EUDC's inline
+expansion function is automatically added to the variable
+@code{completion-at-point-functions}.  As a result, whenever
+@code{completion-at-point} is invoked in a @code{message-mode} buffer,
+EUDC will be queried for email addresses matching the words before
+point.  Since this will be useful only when editing specific message
+header fields that require specifying one or more email addresses, an
+additional check is performed whether point is actually in one of
+those header fields.  Thus, any matching email addresses will be
+offered for completion in suitable message header fields only, and not
+in other places, like for example the body of the message.
 
 
 @node The Server Hotlist
diff --git a/etc/NEWS b/etc/NEWS
index 70087f2629..1b3b70a46f 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -889,6 +889,12 @@ is called, and the returned values are used to populate the phrase and
 comment parts (see RFC 5322 for definitions).  In both cases, the
 phrase part will be automatically quoted if necessary.
 
++++
+*** New function 'eudc-capf-complete' with message-mode integration
+EUDC can now contribute email addresses to 'completion-at-point' by
+adding the new function 'eudc-capf-complete' to
+'completion-at-point-functions' in message-mode.
+
 ** eww/shr
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index cc994d3ba5..b59ab2c31d 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -51,6 +51,7 @@
 (require 'yank-media)
 (require 'mailcap)
 (require 'sendmail)
+(require 'eudc-capf)
 
 (autoload 'mailclient-send-it "mailclient")
 
@@ -3180,8 +3181,7 @@ Like `text-mode', but with these additional commands:
     (mail-abbrevs-setup))
    ((message-mail-alias-type-p 'ecomplete)
     (ecomplete-setup)))
-  ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
-  ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
+  (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
   (add-hook 'completion-at-point-functions #'message-completion-function nil t)
   (unless buffer-file-name
     (message-set-auto-save-file-name))
diff --git a/lisp/net/eudc-capf.el b/lisp/net/eudc-capf.el
new file mode 100644
index 0000000000..0c4449fd46
--- /dev/null
+++ b/lisp/net/eudc-capf.el
@@ -0,0 +1,136 @@
+;;; eudc-capf.el --- EUDC - completion-at-point bindings  -*- lexical-binding:t -*-
+
+;; Copyright (C) 2022 Free Software Foundation, Inc.
+;;
+;; Author: Alexander Adolf
+;;
+;; This file is part of GNU Emacs.
+;;
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;;    This library provides functions to deliver email addresses from
+;;    EUDC search results to `completion-at-point'.
+;;
+;;    Email address completion will likely be desirable only in
+;;    situations where designating email recipients plays a role, such
+;;    as when composing or replying to email messages, or when posting
+;;    to newsgroups, possibly with copies of the post being emailed.
+;;    Hence, modes relevant in such contexts, such as for example
+;;    `message-mode' and `mail-mode', often at least to some extent
+;;    provide infrastructure for different functions to be called when
+;;    completing in certain message header fields, or in the body of
+;;    the message.  In other modes for editing email messages or
+;;    newsgroup posts, which do not provide such infrastructure, any
+;;    completion function providing email addresses will need to check
+;;    whether the completion attempt occurs in an appropriate context
+;;    (that is, in a relevant message header field) before providing
+;;    completion candidates.  Two mechanisms are thus provided by this
+;;    library.
+;;
+;;    The first mechanism is intended for use by the modes listed in
+;;    `eudc-capf-modes', and relies on these modes adding
+;;    `eudc-capf-complete' to `completion-at-point-functions', as
+;;    would be usually done for any general-purpose completion
+;;    function.  In this mode of operation, and in order to offer
+;;    email addresses only in contexts where the user would expect
+;;    them, a check is performed whether point is on a line that is a
+;;    message header field suitable for email addresses, such as for
+;;    example "To:", "Cc:", etc.
+;;
+;;    The second mechanism is intended for when the user modifies
+;;    `message-completion-alist' to replace `message-expand-name' with
+;;    the function `eudc-capf-message-expand-name'.  As a result,
+;;    minibuffer completion (`completing-read') for email addresses
+;;    would no longer enabled in `message-mode', but
+;;    `completion-at-point' (in-buffer completion) only.
+
+;;; Usage:
+
+;;    In a major mode, or context where you want email address
+;;    completion, you would do something along the lines of:
+;;
+;;    (require 'eudc-capf)
+;;    (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
+;;
+;;    The minus one argument puts it at the front of the list so it is
+;;    called first, and the t value for the LOCAL parameter causes the
+;;    setting to be buffer local, so as to avoid modifying any global
+;;    setting.
+;;
+;;    The value of the variable `eudc-capf-modes' indicates which
+;;    major modes do such a setup as part of their initialisation
+;;    code.
+
+;;; Code:
+
+(require 'eudc)
+
+(defvar message-email-recipient-header-regexp)
+(defvar mail-abbrev-mode-regexp)
+(declare-function mail-abbrev-in-expansion-header-p "mailabbrev" ())
+
+(defconst eudc-capf-modes '(message-mode)
+  "List of modes in which email address completion is to be attempted.")
+
+;; completion functions
+
+;;;###autoload
+(defun eudc-capf-complete ()
+  "Email address completion function for `completion-at-point-functions'.
+
+This function checks whether the current major mode is one of the
+modes listed in `eudc-capf-modes', and whether point is on a line
+with a message header listing email recipients, that is, a line
+whose beginning matches `message-email-recipient-header-regexp',
+and, if the check succeeds, searches for records matching the
+words before point.
+
+The return value is either nil when no match is found, or a
+completion table as required for functions listed in
+`completion-at-point-functions'."
+  (if (and (seq-some #'derived-mode-p eudc-capf-modes)
+           (let ((mail-abbrev-mode-regexp message-email-recipient-header-regexp))
+             (mail-abbrev-in-expansion-header-p)))
+      (eudc-capf-message-expand-name)))
+
+;;;###autoload
+(defun eudc-capf-message-expand-name ()
+  "Email address completion function for `message-completion-alist'.
+
+When this function is added to `message-completion-alist',
+replacing any existing entry for `message-expand-name' there,
+with an appropriate regular expression such as for example
+`message-email-recipient-header-regexp', then EUDC will be
+queried for email addresses, and the results delivered to
+`completion-at-point'."
+  (if (or eudc-server eudc-server-hotlist)
+      (progn
+        (setq-local completion-styles '(substring partial-completion))
+        (let* ((beg (save-excursion
+                      (if (re-search-backward "\\([:,]\\|^\\)[ \t]*"
+                                              (point-at-bol) t)
+                          (goto-char (match-end 0)))
+                      (point)))
+               (end (point))
+               (prefix (save-excursion (buffer-substring-no-properties beg end))))
+          (list beg end
+                (completion-table-with-cache
+                 (lambda (_)
+                   (eudc-query-with-words (split-string prefix "[ \t]+") t))
+                 t))))))
+
+(provide 'eudc-capf)
+;;; eudc-capf.el ends here
-- 
2.36.0

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Alexander,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

[...]

> Many thanks and looking forward to your thoughts,

Thanks for the updated patch.  One open issue I had when I tested the
prior patch was the fact that repeated completion attempts would query
the LDAP search on each attempt, even if the completion table could
already have been populated with all relevant LDAP entries.

Can you test that behaviour, perhaps with the macOS contact EUDC backend
that you use, to see what I mean?  Is there a way the default
completion-at-point implementation can do the same as
eudc-expand-try-all, where the final completion selection is done from
the collected list of options, rather than requerying all the backends
each time TAB is pressed?

Thanks,
Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Filipp Gunbin]

Hello Alexander,

On 28/04/2022 19:15 +0200, Alexander Adolf wrote:

>>> +      (progn
>>> +        (setq-local completion-styles '(substring partial-completion))
>>> +        (let* ((beg (save-excursion
>>> +                      (if (re-search-backward "\\([:,]\\|^\\)[ \t]*"
>>> +                                              (point-at-bol) 'move)
>>
>> t instead of 'move?
>
> Hm. Quoting from the docstring (re-search-forward in this case):
> ---------------------------- Begin Quote -----------------------------
> The optional third argument NOERROR indicates how errors are handled
>   when the search fails. If it is nil or omitted, emit an error; if it
>   is t, simply return nil and do nothing; if it is neither nil nor t,
>   move to the limit of search and return nil.
> ----------------------------- End Quote ------------------------------
>
> This reads as if t ("do nothing") vs. 'move ("move to the limit of
> search") should make a difference? A few quick experiments seem to
> indicate that in practice both seem to behave the same though. In this
> light, I'm fine with changing 'move to t.

Well honestly I was fooled by save-excursion, and didn't notice the
return value of (point), with which the value we're talking about _may_
matter.  However, your regexp will always match, due to \\|^ branch
which will find bol, so you can just do:

(save-excursion
  (re-search-backward "\\([:,]\\|^\\)[ \t]*")
  (match-end 0))

>> Can it be done in some other way?
>
> Short answer: I don't think so.
>
> Longer answer:
>
> The motivation for putting this is that completion-at-point filters the
> completion tables based on completion-styles. Thus, if completion-styles
> is set too restrictive, useful results may not be offered to the user.
> The default completion-styles in message-mode is '(basic
> partial-completion emacs22). I wanted 'substring to be the primary as it
> matches the search term anywhere within the candidate, as opposed to
> 'basic which matches it at the beginning only. In this respect, you
> might view 'substring as a super-set of 'basic.
>
> That said, it would probably be desirable for message mode to have
> different values for both, completion-at-point-functions and
> completion-styles, depending on where point is (email header, newsgroup
> header, message body, etc.). But this seems like a wider discussion
> about the architecture of message.el rather than this patch.

Yes, sounds like another use case for completion-category-overrides.

Thanks.

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

Hello Thomas,

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> [...]
> One open issue I had when I tested the prior patch was the fact that
> repeated completion attempts would query the LDAP search on each
> attempt, even if the completion table could already have been
> populated with all relevant LDAP entries.
> [...]

Fully agree that that's inefficient and impairs the user experience. It
shouldn't happen, but seems to depend on the user interface.

I do get a completions buffer popped up when calling either of the
functions completion-at-point or message-tab (see attached).

What is your value of the variable completion-auto-help?

Dies changing the value of the variable message-expand-name-standard-ui
change anything for you?

Which function gets called when you press TAB? message-tab?


Cheers,

  --alexander



[-- Attachment #2: Screenshot 2022-05-02 at 17.58.22.png --]
[-- Type: image/png, Size: 289531 bytes --]

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

Hello Filipp,

Filipp Gunbin <fgunbin@fastmail.fm> writes:

>> [...]
>> This reads as if t ("do nothing") vs. 'move ("move to the limit of
>> search") should make a difference? A few quick experiments seem to
>> indicate that in practice both seem to behave the same though. In this
>> light, I'm fine with changing 'move to t.
>
> Well honestly I was fooled by save-excursion, and didn't notice the
> return value of (point), with which the value we're talking about _may_
> matter.  However, your regexp will always match, due to \\|^ branch
> which will find bol, so you can just do:
>
> (save-excursion
>   (re-search-backward "\\([:,]\\|^\\)[ \t]*")
>   (match-end 0))

I tried without the surrounding save-excursion, and even then I didn't
notice any difference in behaviour between t and 'move for NOERROR.

Your suggested simplification works just as well, of course. Thanks for
pointing out! Updated patch below.

>> [...]
>> That said, it would probably be desirable for message mode to have
>> different values for both, completion-at-point-functions and
>> completion-styles, depending on where point is (email header, newsgroup
>> header, message body, etc.). But this seems like a wider discussion
>> about the architecture of message.el rather than this patch.
>
> Yes, sounds like another use case for completion-category-overrides.

Hadn't come across that one yet; thanks for the pointer.

Message.el does this:

(add-to-list 'completion-category-defaults '(email (styles substring)))

It seems to me you suggest amending that line in message.el, rather than
having the setq-local in my capf function? I have included this in the
below patch, too.


Many thanks and looking forward to your thoughts,

  --alexander



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-EUDC-email-addresses-via-completion-at-point-in-mess.patch --]
[-- Type: text/x-patch, Size: 10692 bytes --]

From 7da38d56d7dddf4f60f4e8055634a64ef6709870 Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Mon, 2 May 2022 23:01:11 +0200
Subject: [PATCH] EUDC email addresses via completion-at-point in message-mode

* lisp/net/eudc-capf.el (new file): Add new 'eudc-capf-complete'
function.
* lisp/gnus/message.el (message-mode): Add 'eudc-capf-complete' to
'completion-at-point-functions' when a 'message-mode' buffer is
created, removing the FIXME.
* doc/misc/eudc.texi (Inline Query Expansion): Add a new subsection,
describing the new 'completion-at-point' mechanism in 'message-mode'.
* etc/NEWS (EUDC): Describe the new 'completion-at-point' method.
---
 doc/misc/eudc.texi    |  23 ++++++++
 etc/NEWS              |   6 ++
 lisp/gnus/message.el  |   7 ++-
 lisp/net/eudc-capf.el | 133 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 166 insertions(+), 3 deletions(-)
 create mode 100644 lisp/net/eudc-capf.el

diff --git a/doc/misc/eudc.texi b/doc/misc/eudc.texi
index d2850282fe..d2a1efeed0 100644
--- a/doc/misc/eudc.texi
+++ b/doc/misc/eudc.texi
@@ -713,6 +713,7 @@ be passed to the program.
 
 @node Inline Query Expansion
 @section Inline Query Expansion
+@subsection Inline Query Expansion Using a Key Binding
 
 Inline query expansion is a powerful method to get completion from
 your directory servers.  The most common usage is for expanding names
@@ -885,6 +886,28 @@ An error is signaled.  The expansion aborts.
 Default is @code{select}
 @end defvar
 
+@subsection Inline Query Expansion Using completion-at-point
+
+In addition to providing a dedicated EUDC function for binding to a
+key shortcut (@pxref{Inline Query Expansion Using a Key Binding}),
+EUDC also provides a function to contribute search results to the
+Emacs in-buffer completion system available via the function
+@code{completion-at-point} (@pxref{Identifier
+Inquiries,,,maintaining}) in @code{message-mode} buffers
+(@pxref{Message}).  When using this mechanism, queries are made in the
+multi-server query mode of operation (@pxref{Multi-server Queries}).
+
+When a buffer in @code{message-mode} is created, EUDC's inline
+expansion function is automatically added to the variable
+@code{completion-at-point-functions}.  As a result, whenever
+@code{completion-at-point} is invoked in a @code{message-mode} buffer,
+EUDC will be queried for email addresses matching the words before
+point.  Since this will be useful only when editing specific message
+header fields that require specifying one or more email addresses, an
+additional check is performed whether point is actually in one of
+those header fields.  Thus, any matching email addresses will be
+offered for completion in suitable message header fields only, and not
+in other places, like for example the body of the message.
 
 
 @node The Server Hotlist
diff --git a/etc/NEWS b/etc/NEWS
index f897158afd..9df23ee326 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -951,6 +951,12 @@ is called, and the returned values are used to populate the phrase and
 comment parts (see RFC 5322 for definitions).  In both cases, the
 phrase part will be automatically quoted if necessary.
 
++++
+*** New function 'eudc-capf-complete' with message-mode integration
+EUDC can now contribute email addresses to 'completion-at-point' by
+adding the new function 'eudc-capf-complete' to
+'completion-at-point-functions' in message-mode.
+
 ** eww/shr
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index e7dc089a3c..3cef247522 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -51,6 +51,7 @@
 (require 'yank-media)
 (require 'mailcap)
 (require 'sendmail)
+(require 'eudc-capf)
 
 (autoload 'mailclient-send-it "mailclient")
 
@@ -3180,8 +3181,7 @@ Like `text-mode', but with these additional commands:
     (mail-abbrevs-setup))
    ((message-mail-alias-type-p 'ecomplete)
     (ecomplete-setup)))
-  ;; FIXME: merge the completion tables from ecomplete/bbdb/...?
-  ;;(add-hook 'completion-at-point-functions #'message-ecomplete-capf nil t)
+  (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
   (add-hook 'completion-at-point-functions #'message-completion-function nil t)
   (unless buffer-file-name
     (message-set-auto-save-file-name))
@@ -8364,7 +8364,8 @@ set to nil."
 	(t
 	 (expand-abbrev))))
 
-(add-to-list 'completion-category-defaults '(email (styles substring)))
+(add-to-list 'completion-category-defaults '(email (styles substring
+                                                           partial-completion)))
 
 (defun message--bbdb-query-with-words (words)
   ;; FIXME: This (or something like this) should live on the BBDB side.
diff --git a/lisp/net/eudc-capf.el b/lisp/net/eudc-capf.el
new file mode 100644
index 0000000000..68cbfd93ff
--- /dev/null
+++ b/lisp/net/eudc-capf.el
@@ -0,0 +1,133 @@
+;;; eudc-capf.el --- EUDC - completion-at-point bindings  -*- lexical-binding:t -*-
+
+;; Copyright (C) 2022 Free Software Foundation, Inc.
+;;
+;; Author: Alexander Adolf
+;;
+;; This file is part of GNU Emacs.
+;;
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;;    This library provides functions to deliver email addresses from
+;;    EUDC search results to `completion-at-point'.
+;;
+;;    Email address completion will likely be desirable only in
+;;    situations where designating email recipients plays a role, such
+;;    as when composing or replying to email messages, or when posting
+;;    to newsgroups, possibly with copies of the post being emailed.
+;;    Hence, modes relevant in such contexts, such as for example
+;;    `message-mode' and `mail-mode', often at least to some extent
+;;    provide infrastructure for different functions to be called when
+;;    completing in certain message header fields, or in the body of
+;;    the message.  In other modes for editing email messages or
+;;    newsgroup posts, which do not provide such infrastructure, any
+;;    completion function providing email addresses will need to check
+;;    whether the completion attempt occurs in an appropriate context
+;;    (that is, in a relevant message header field) before providing
+;;    completion candidates.  Two mechanisms are thus provided by this
+;;    library.
+;;
+;;    The first mechanism is intended for use by the modes listed in
+;;    `eudc-capf-modes', and relies on these modes adding
+;;    `eudc-capf-complete' to `completion-at-point-functions', as
+;;    would be usually done for any general-purpose completion
+;;    function.  In this mode of operation, and in order to offer
+;;    email addresses only in contexts where the user would expect
+;;    them, a check is performed whether point is on a line that is a
+;;    message header field suitable for email addresses, such as for
+;;    example "To:", "Cc:", etc.
+;;
+;;    The second mechanism is intended for when the user modifies
+;;    `message-completion-alist' to replace `message-expand-name' with
+;;    the function `eudc-capf-message-expand-name'.  As a result,
+;;    minibuffer completion (`completing-read') for email addresses
+;;    would no longer enabled in `message-mode', but
+;;    `completion-at-point' (in-buffer completion) only.
+
+;;; Usage:
+
+;;    In a major mode, or context where you want email address
+;;    completion, you would do something along the lines of:
+;;
+;;    (require 'eudc-capf)
+;;    (add-hook 'completion-at-point-functions #'eudc-capf-complete -1 t)
+;;
+;;    The minus one argument puts it at the front of the list so it is
+;;    called first, and the t value for the LOCAL parameter causes the
+;;    setting to be buffer local, so as to avoid modifying any global
+;;    setting.
+;;
+;;    The value of the variable `eudc-capf-modes' indicates which
+;;    major modes do such a setup as part of their initialisation
+;;    code.
+
+;;; Code:
+
+(require 'eudc)
+
+(defvar message-email-recipient-header-regexp)
+(defvar mail-abbrev-mode-regexp)
+(declare-function mail-abbrev-in-expansion-header-p "mailabbrev" ())
+
+(defconst eudc-capf-modes '(message-mode)
+  "List of modes in which email address completion is to be attempted.")
+
+;; completion functions
+
+;;;###autoload
+(defun eudc-capf-complete ()
+  "Email address completion function for `completion-at-point-functions'.
+
+This function checks whether the current major mode is one of the
+modes listed in `eudc-capf-modes', and whether point is on a line
+with a message header listing email recipients, that is, a line
+whose beginning matches `message-email-recipient-header-regexp',
+and, if the check succeeds, searches for records matching the
+words before point.
+
+The return value is either nil when no match is found, or a
+completion table as required for functions listed in
+`completion-at-point-functions'."
+  (if (and (seq-some #'derived-mode-p eudc-capf-modes)
+           (let ((mail-abbrev-mode-regexp message-email-recipient-header-regexp))
+             (mail-abbrev-in-expansion-header-p)))
+      (eudc-capf-message-expand-name)))
+
+;;;###autoload
+(defun eudc-capf-message-expand-name ()
+  "Email address completion function for `message-completion-alist'.
+
+When this function is added to `message-completion-alist',
+replacing any existing entry for `message-expand-name' there,
+with an appropriate regular expression such as for example
+`message-email-recipient-header-regexp', then EUDC will be
+queried for email addresses, and the results delivered to
+`completion-at-point'."
+  (if (or eudc-server eudc-server-hotlist)
+      (progn
+        (let* ((beg (save-excursion
+                      (re-search-backward "\\([:,]\\|^\\)[ \t]*")
+                      (match-end 0)))
+               (end (point))
+               (prefix (save-excursion (buffer-substring-no-properties beg end))))
+          (list beg end
+                (completion-table-with-cache
+                 (lambda (_)
+                   (eudc-query-with-words (split-string prefix "[ \t]+") t))
+                 t))))))
+
+(provide 'eudc-capf)
+;;; eudc-capf.el ends here
-- 
2.36.0

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Filipp Gunbin]

On 02/05/2022 23:38 +0200, Alexander Adolf wrote:

>> Yes, sounds like another use case for completion-category-overrides.
>
> Hadn't come across that one yet; thanks for the pointer.
>
> Message.el does this:
>
> (add-to-list 'completion-category-defaults '(email (styles substring)))
>
> It seems to me you suggest amending that line in message.el, rather than
> having the setq-local in my capf function? I have included this in the
> below patch, too.
>

I haven't dug any deep into using
completion-category-overrides/defaults, I just knew they exist.  If that
does what you want, then great!

Filipp

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> [...]
>> Message.el does this:
>>
>> (add-to-list 'completion-category-defaults '(email (styles substring)))
>>
>> It seems to me you suggest amending that line in message.el, rather than
>> having the setq-local in my capf function? I have included this in the
>> below patch, too.
>
> I haven't dug any deep into using
> completion-category-overrides/defaults, I just knew they exist.  If that
> does what you want, then great!

Yes, it does what I wanted to achieve. Thanks for your time!


  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> [...]
> Thanks for pointing these out, and for helping to improve the proposed
> patch!
>
> Then there was a wider discussion on potential, architectural
> improvements of completion handling in message.el.
>
> Before diving into this wider discussion, it would seem worthwhile to me
> trying to get to a conclusion on the proposed patch.
>
> Do we have a consensus, that the proposed, updated patch does not solve
> all potential issues around email address completion in message mode
> once and for all, but that it still provides a useful improvement for
> some users without disturbing others, thus making it seem worthwhile to
> include the patch?
> [...]

After some further help from Filipp Gunbin (setq-local removed,
re-serach-backward simplified), for which "Thanks, Filipp!", it seems
I'm back at the above question ("Do we have a consensus...") with
respect to my proposed patch.

Do we have consensus?


Many thanks and looking forward to your thoughts,

  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Alexander,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> [...]
>> One open issue I had when I tested the prior patch was the fact that
>> repeated completion attempts would query the LDAP search on each
>> attempt, even if the completion table could already have been
>> populated with all relevant LDAP entries.
>> [...]
>
> Fully agree that that's inefficient and impairs the user experience. It
> shouldn't happen, but seems to depend on the user interface.
>
> I do get a completions buffer popped up when calling either of the
> functions completion-at-point or message-tab (see attached).
>
> What is your value of the variable completion-auto-help?

It is t.

> Dies changing the value of the variable message-expand-name-standard-ui
> change anything for you?

It is currently nil.

> Which function gets called when you press TAB? message-tab?

The behaviour I described was when I was testing with TAB bound to
message-tab, yes.

Can you reproduce the scenario in your screenshot, then subsequently
type '-' then TAB, to complete on "emacs-" instead of "emacs"; I think
the completion backends will be queried again.  That's what I want to
avoid (if that doesn't happen for you, then I'll have to recreate my
testing environment and test more carefully with all the combinations of
the above variables, I guess).

Did you confirm completion-table-with-cache is doing something?  If so,
do you mind adding a comment within eudc-capf-message-expand-name's
body, describing what the caching is doing, when the cache is
invalidated, etc.?

Thanks,
Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello Thomas,

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> [...]
>> What is your value of the variable completion-auto-help?
>
> It is t.

Ok, so that's not the reason for you having to TAB through the
alternatives, instead of being offered the "*Completions*" buffer.

>> Dies changing the value of the variable message-expand-name-standard-ui
>> change anything for you?
>
> It is currently nil.

Did you try with a non-nil value?

> [...]
> Can you reproduce the scenario in your screenshot, then subsequently
> type '-' then TAB, to complete on "emacs-" instead of "emacs"; I think
> the completion backends will be queried again.  That's what I want to
> avoid (if that doesn't happen for you, then I'll have to recreate my
> testing environment and test more carefully with all the combinations of
> the above variables, I guess).
>
> Did you confirm completion-table-with-cache is doing something?  If so,
> do you mind adding a comment within eudc-capf-message-expand-name's
> body, describing what the caching is doing, when the cache is
> invalidated, etc.?

I have done a few experiments with this version of
completion-table-with-cache(), sprinkled with some message printing:

---------------------------- Begin Quote -----------------------------
    (defun completion-table-with-cache (fun &optional ignore-case)
      (let* (last-arg last-result
             (new-fun
              (lambda (arg)
                (if (and last-arg (string-prefix-p last-arg arg ignore-case))
                    (prog1
                        last-result
                      (message "[cache HIT] arg = [%s]" arg))
                  (prog1
                      (setq last-result (funcall fun arg))
                    (message "[cache miss] arg = [%s]" arg)
                    (setq last-arg arg))))))
        (completion-table-dynamic new-fun)))
----------------------------- End Quote ------------------------------


Experiment #1: 3rd Party UI Package
===================================

[1] https://github.com/minad/corfu

Using the corfu package [1], I get the following output in the messages
buffer for your use-case:

--------------------------- Begin Transcript -------------------------
        <<< typing "emacs"

    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name
    eudc-query-with-words
    [cache miss] arg = []
    [cache HIT] arg = [emacs]

        <<< typing "-"

    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name
    [cache HIT] arg = []
    [cache HIT] arg = [emacs-]
---------------------------- End Transcript --------------------------

On the first message "eudc-capf-complete", I type "emacs", and on the
second, I type "-" as you describe.

I.e. eudc-query-with-words gets called once only, whereas the capf
function gets called every time. The caching works as expected for me.


Experiment #2: message-tab
==========================

Similar workflow as before, but instead of the 3rd party package,
invoking message-tab:

--------------------------- Begin Transcript -------------------------
        <<< typing "emacs"
        <<< invoking message-tab

    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name
    eudc-query-with-words
    [cache miss] arg = []
    [cache HIT] arg = [emacs]
    Making completion list...
    [cache HIT] arg = []
    nil
    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name

        <<< typing "-"

    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name

        <<< invoking message-tab

    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name
    eudc-query-with-words
    [cache miss] arg = []
    [cache HIT] arg = [emacs-]
    Making completion list...
    [cache HIT] arg = []
    nil
    eudc-capf-complete --------------------------------------
    eudc-capf-message-expand-name
---------------------------- End Transcript --------------------------

As you observed, too, eudc-query-with-words is invoked every time.


Experiment #3: completion-at-point
==================================

Same as #2 above, but instead of message-tab, this time invoking
completion-at-point.

The result is almost identical to #2, too, with the difference that
instead of the nil near the end of the end of the bigger message blocks,
it now shows a t. Not sure where this nil/t output comes from anyway?


Conclusions
===========

How completion-at-point is wrapped makes a significant difference. With
the vanilla front-end(s) in message mode, the CAPF function gets invoked
many more times than with the 3rd party package. This also seems to
affect to what extent the cache can be taken advantage of by
completion-at-point.

Given that the behaviour appears linked to message.el, what could seem a
useful way forward? Perhaps this is something for the maintainer of
message.el to look into? A view from someone familiar with the details
of completion-at-point might also be helpful?


Many thanks and looking forward to your thoughts,

  --alexander

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Alexander,

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:
>
>> [...]
>>> What is your value of the variable completion-auto-help?
>>
>> It is t.
>
> Ok, so that's not the reason for you having to TAB through the
> alternatives, instead of being offered the "*Completions*" buffer.
>
>>> Dies changing the value of the variable message-expand-name-standard-ui
>>> change anything for you?
>>
>> It is currently nil.
>
> Did you try with a non-nil value?
>
>> [...]
>> Can you reproduce the scenario in your screenshot, then subsequently
>> type '-' then TAB, to complete on "emacs-" instead of "emacs"; I think
>> the completion backends will be queried again.  That's what I want to
>> avoid (if that doesn't happen for you, then I'll have to recreate my
>> testing environment and test more carefully with all the combinations of
>> the above variables, I guess).
>>
>> Did you confirm completion-table-with-cache is doing something?  If so,
>> do you mind adding a comment within eudc-capf-message-expand-name's
>> body, describing what the caching is doing, when the cache is
>> invalidated, etc.?
>
> I have done a few experiments with this version of
> completion-table-with-cache(), sprinkled with some message printing:
>
> ---------------------------- Begin Quote -----------------------------
>     (defun completion-table-with-cache (fun &optional ignore-case)
>       (let* (last-arg last-result
>              (new-fun
>               (lambda (arg)
>                 (if (and last-arg (string-prefix-p last-arg arg ignore-case))
>                     (prog1
>                         last-result
>                       (message "[cache HIT] arg = [%s]" arg))
>                   (prog1
>                       (setq last-result (funcall fun arg))
>                     (message "[cache miss] arg = [%s]" arg)
>                     (setq last-arg arg))))))
>         (completion-table-dynamic new-fun)))
> ----------------------------- End Quote ------------------------------
>
>
> Experiment #1: 3rd Party UI Package
> ===================================
>
> [1] https://github.com/minad/corfu
>
> Using the corfu package [1], I get the following output in the messages
> buffer for your use-case:
>
> --------------------------- Begin Transcript -------------------------
>         <<< typing "emacs"
>
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
>     eudc-query-with-words
>     [cache miss] arg = []
>     [cache HIT] arg = [emacs]
>
>         <<< typing "-"
>
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
>     [cache HIT] arg = []
>     [cache HIT] arg = [emacs-]
> ---------------------------- End Transcript --------------------------
>
> On the first message "eudc-capf-complete", I type "emacs", and on the
> second, I type "-" as you describe.
>
> I.e. eudc-query-with-words gets called once only, whereas the capf
> function gets called every time. The caching works as expected for me.
>
>
> Experiment #2: message-tab
> ==========================
>
> Similar workflow as before, but instead of the 3rd party package,
> invoking message-tab:
>
> --------------------------- Begin Transcript -------------------------
>         <<< typing "emacs"
>         <<< invoking message-tab
>
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
>     eudc-query-with-words
>     [cache miss] arg = []
>     [cache HIT] arg = [emacs]
>     Making completion list...
>     [cache HIT] arg = []
>     nil
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
>
>         <<< typing "-"
>
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
>
>         <<< invoking message-tab
>
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
>     eudc-query-with-words
>     [cache miss] arg = []
>     [cache HIT] arg = [emacs-]
>     Making completion list...
>     [cache HIT] arg = []
>     nil
>     eudc-capf-complete --------------------------------------
>     eudc-capf-message-expand-name
> ---------------------------- End Transcript --------------------------
>
> As you observed, too, eudc-query-with-words is invoked every time.
>
>
> Experiment #3: completion-at-point
> ==================================
>
> Same as #2 above, but instead of message-tab, this time invoking
> completion-at-point.
>
> The result is almost identical to #2, too, with the difference that
> instead of the nil near the end of the end of the bigger message blocks,
> it now shows a t. Not sure where this nil/t output comes from anyway?
>
>
> Conclusions
> ===========
>
> How completion-at-point is wrapped makes a significant difference. With
> the vanilla front-end(s) in message mode, the CAPF function gets invoked
> many more times than with the 3rd party package. This also seems to
> affect to what extent the cache can be taken advantage of by
> completion-at-point.

This is good analysis, thanks for writing it up.  Interesting that corfu
seems to be more efficient than the built-in front-ends.  For this use
case, it seems to me that the built-in front-ends should continue
narrowing down from what's in the cache, since no new completions will
come from the backends if the search string only gets more specific.

> Given that the behaviour appears linked to message.el, what could seem a
> useful way forward? Perhaps this is something for the maintainer of
> message.el to look into? A view from someone familiar with the details
> of completion-at-point might also be helpful?

Yes; in particular, it would be nice to have an overview node in the
manual about how everything CAPF-related is supposed to fit together,
for mode authors who want to "complete this type of thing", for
completion UI authors who want to show a list of choices in some way,
and for completion backend authors who want to provide lists to
requesters.

In any case, I think your patch is an improvement to the default
message.el behaviour, and I don't necessarily want to hold it up waiting
for built-in CAPF front-end redesigns.  Let's give it another day and if
we haven't figured out this design/optimization issue by then, I'll push
what you've provided so far, as a work-in-progress.

Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Thomas Fitzsimmons]

Hi Alexander,

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

[...]

> In any case, I think your patch is an improvement to the default
> message.el behaviour, and I don't necessarily want to hold it up waiting
> for built-in CAPF front-end redesigns.  Let's give it another day and if
> we haven't figured out this design/optimization issue by then, I'll push
> what you've provided so far, as a work-in-progress.

I pushed your patch.  I made some cosmetic changes to the log message,
making the first line shorter and quoting function names correctly.  I
also fixed the following texinfo warnings:

eudc.texi:892: @pxref reference to nonexistent node
 `Inline Query Expansion Using a Key Binding'
eudc.texi:897: @pxref reference to nonexistent node
 `Message'

Thanks,
Thomas

Re: [PATCH] EUDC email addresses via completion-at-point in message-mode

[Alexander Adolf]

Hello THomas,

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> [...]
> I pushed your patch.  I made some cosmetic changes to the log message,
> making the first line shorter and quoting function names correctly.  I
> also fixed the following texinfo warnings:
> [...]

MANY thanks for your kind support!

  --alexander