[ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

Hi.  I wrote a new package, 'mailaprop', that provides popup-style autofill for email addresses in recipient headers of message composition buffers.  

For those who read and write email in Emacs, I hope this will save some keystrokes.  I've been using it for a while, and it certainly saves me many keystrokes.

Code, documentation, and a demo video are available at:

  https://github.com/kfogel/mailaprop

Also, as the README.md points out:

  "You don't have to use any proprietary Javascript to interact with
  this project.  You can use plain git to clone the repository from
  GitHub at https://github.com/kfogel/mailaprop.git, and I'll happily
  take bug reports and patches by email instead of via the GitHub issue
  tracker: kfogel {AT} red-bean.com."

I chose GitHub because it is the most reliable zero-cost git hosting service out there; browser interaction with GitHub is strictly optional for this project.

Best regards,
-Karl

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[John Wiegley]

>>>>> "KF" == Karl Fogel <kfogel@red-bean.com> writes:

KF> Hi. I wrote a new package, 'mailaprop', that provides popup-style autofill
KF> for email addresses in recipient headers of message composition buffers.

If you want a nice source of candidates, I wrote this unpublished package:

https://github.com/jwiegley/dot-emacs/blob/master/lisp/gnus-harvest/gnus-harvest.el

It watches who you write to, and who you read e-mail from (if you hook it into
Gnus), and stores them all in an SQLite database with timestamps. It then lets
you query for a candidate list. I use it via message-x.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

"John Wiegley" <johnw@gnu.org> writes:
>KF> Hi. I wrote a new package, 'mailaprop', that provides popup-style autofill
>KF> for email addresses in recipient headers of message composition buffers.
>
>If you want a nice source of candidates, I wrote this unpublished package:
>
>https://github.com/jwiegley/dot-emacs/blob/master/lisp/gnus-harvest/gnus-harvest.el
>
>It watches who you write to, and who you read e-mail from (if you hook it into
>Gnus), and stores them all in an SQLite database with timestamps. It then lets
>you query for a candidate list. I use it via message-x.

Thanks, John.  That looks handy.

To be useable by mailaprop, it would need to remember the datestamp of the message the address was harvested from (or at least remember the date of the most recent message that address was harvested from), and keep separate counts for how often the address had been sent to and how often it had been received from.

This is because the order in which addresses are offered in autofill matters.  The most likely candidate should be at the top, the next-most-likely underneath it, and so on.  Right now, the inputs to the likelihood-calculating algorithm are the last-seen date, the total sent-to count, and total received-from count.  I could make things fancier later, but for now those are the inputs.

Would those pieces of information be useful for message-x and other integratees of gnus-harvest too?  Obviously, that information would be pretty easy to harvest.  If mailaprop and message-x could share inputs, that'd be nice.

By the way, mailaprop's current solution to the problem of gathering input data is a standalone Python script.  It crawls all your mbox files (other formats to be supported in the future, I hope) and builds the autofill database from that.  I've just added a paragraph to the README.md explaining that it could take input from other sources too, and linking to this discussion.

Best regards,
-Karl

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[joakim]

Thanks, this looks useful.

As an aside, I have at least three different broken emacs addressbooks
lying around. Does mailaprop integrate with any of the existing emacs addressbooks?

Karl Fogel <kfogel@red-bean.com> writes:

> Hi.  I wrote a new package, 'mailaprop', that provides popup-style autofill for email addresses in recipient headers of message composition buffers.  
>
> For those who read and write email in Emacs, I hope this will save some keystrokes.  I've been using it for a while, and it certainly saves me many keystrokes.
>
> Code, documentation, and a demo video are available at:
>
>   https://github.com/kfogel/mailaprop
>
> Also, as the README.md points out:
>
>   "You don't have to use any proprietary Javascript to interact with
>   this project.  You can use plain git to clone the repository from
>   GitHub at https://github.com/kfogel/mailaprop.git, and I'll happily
>   take bug reports and patches by email instead of via the GitHub issue
>   tracker: kfogel {AT} red-bean.com."
>
> I chose GitHub because it is the most reliable zero-cost git hosting service out there; browser interaction with GitHub is strictly optional for this project.
>
> Best regards,
> -Karl
>
-- 
Joakim Verona
joakim@verona.se
+46705459454

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

joakim@verona.se writes:
>Thanks, this looks useful.
>
>As an aside, I have at least three different broken emacs addressbooks
>lying around. Does mailaprop integrate with any of the existing emacs
>addressbooks?

I've never used any of the Emacs address books, so I don't know.  

However, it should just be a matter of converting the information in the address book to a format that mailaprop can understand.  In practice, that means either the format of the `mailaprop-address-file' or the format of the in-memory list `mailaprop-addresses', both of which are documented in mailaprop.el.

Summary: There's no integration right now, but my guess is integration wouldn't be difficult.

Best regards,
-Karl

>Karl Fogel <kfogel@red-bean.com> writes:
>
>> Hi.  I wrote a new package, 'mailaprop', that provides popup-style
>> autofill for email addresses in recipient headers of message
>> composition buffers.
>>
>> For those who read and write email in Emacs, I hope this will save
>> some keystrokes.  I've been using it for a while, and it certainly
>> saves me many keystrokes.
>>
>> Code, documentation, and a demo video are available at:
>>
>>   https://github.com/kfogel/mailaprop
>>
>> Also, as the README.md points out:
>>
>>   "You don't have to use any proprietary Javascript to interact with
>>   this project.  You can use plain git to clone the repository from
>>   GitHub at https://github.com/kfogel/mailaprop.git, and I'll happily
>>   take bug reports and patches by email instead of via the GitHub issue
>>   tracker: kfogel {AT} red-bean.com."
>>
>> I chose GitHub because it is the most reliable zero-cost git hosting
>> service out there; browser interaction with GitHub is strictly
>> optional for this project.
>>
>> Best regards,
>> -Karl
>>

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> Code, documentation, and a demo video are available at:
>
>   https://github.com/kfogel/mailaprop

Looks very nice...  but the main difference between this and ecomplete
seems to be that ecomplete puts the completions in the echo area and
mailaprop uses a tooltip?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>Looks very nice...  but the main difference between this and ecomplete
>seems to be that ecomplete puts the completions in the echo area and
>mailaprop uses a tooltip?

Hmm, I can't tell.  As of commit 9f22b7d2, ecomplete.el has no function doc strings nor any usage-informing variable doc strings, and the format of ~/.ecompleterc is not documented.  From looking at the elisp file, I wasn't immediately sure what ecomplete does :-).  I grepped around and found this in doc/misc/message.texi:

  If you're using @code{ecomplete}, all addresses from @code{To} and
  @code{Cc} headers will automatically be put into the
  @file{~/.ecompleterc} file.  When you enter text in the @code{To} and
  @code{Cc} headers, @code{ecomplete} will check out the values stored
  there and ``electrically'' say what completions are possible.  To
  choose one of these completions, use the @kbd{M-n} command to move
  down to the list.  Use @kbd{M-n} and @kbd{M-p} to move down and up the
  list, and @kbd{RET} to choose a completion.

It seems like ecomplete does something at least similar to what mailaprop does.  The main questions I have are:

1) Is there a way to load in existing email history?  If I knew the format of the .ecompletrc file, maybe I could make mailaprop.py generate it.  Personally, with a million email messages' worth of legacy data, it seems a pity to start collecting addresses from scratch now :-).

2) Does it prioritize the completion list according to some scoring algorithm, so that the most likely candidates are keystrokewise closest?  The "(loop for (key count time text) ...)" in `ecomplete-get-matches' gave me hope, but further reading led me to think maybe it's not doing that after all.  I'm not sure, though.

Best regards,
-Karl

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> It seems like ecomplete does something at least similar to what
> mailaprop does.  The main questions I have are:
>
> 1) Is there a way to load in existing email history?  If I knew the
> format of the .ecompletrc file, maybe I could make mailaprop.py
> generate it.  Personally, with a million email messages' worth of
> legacy data, it seems a pity to start collecting addresses from
> scratch now :-).

The format is like this:

((mail
 ("larsi@gnus.org" 38154 1516109510 "Lars Ingebrigtsen <larsi@gnus.org>")
 ("kfogel@red-bean.com" 10 1516065455 "Karl Fogel <kfogel@red-bean.com>")
 ...
 ))

Which is email address/times used/last time used/full name.

> 2) Does it prioritize the completion list according to some scoring
> algorithm, so that the most likely candidates are keystrokewise
> closest?

It only prioritises by the number of times the address has been used...
Hm...  I think I had meant to also make it prioritise based on recency,
but I apparently didn't.  But that can be tweaked, of course.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>The format is like this:
>
>((mail
> ("larsi@gnus.org" 38154 1516109510 "Lars Ingebrigtsen <larsi@gnus.org>")
> ("kfogel@red-bean.com" 10 1516065455 "Karl Fogel <kfogel@red-bean.com>")
> ...
> ))
>
>Which is email address/times used/last time used/full name.

Thanks.  How does it handle cases where the same email address appears with different full names?  Is there a separate top-level entry for each of those (i.e., the same car is repeated)?

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>The format is like this:
>>
>>((mail
>> ("larsi@gnus.org" 38154 1516109510 "Lars Ingebrigtsen <larsi@gnus.org>")
>> ("kfogel@red-bean.com" 10 1516065455 "Karl Fogel <kfogel@red-bean.com>")
>> ...
>> ))
>>
>>Which is email address/times used/last time used/full name.
>
> Thanks.  How does it handle cases where the same email address appears
> with different full names?  Is there a separate top-level entry for
> each of those (i.e., the same car is repeated)?

The first elem is the key, so the full name appears only once per key.
That is, it'll be updated.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>Karl Fogel <kfogel@red-bean.com> writes:
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>>The format is like this:
>>>
>>>((mail
>>> ("larsi@gnus.org" 38154 1516109510 "Lars Ingebrigtsen <larsi@gnus.org>")
>>> ("kfogel@red-bean.com" 10 1516065455 "Karl Fogel <kfogel@red-bean.com>")
>>> ...
>>> ))
>>>
>>>Which is email address/times used/last time used/full name.
>>
>> Thanks.  How does it handle cases where the same email address appears
>> with different full names?  Is there a separate top-level entry for
>> each of those (i.e., the same car is repeated)?
>
>The first elem is the key, so the full name appears only once per key.
>That is, it'll be updated.

Thanks, Lars.

Mailaprop stores a superset of the information ecomplete stores.  Namely, mailaprop tracks the sent-to count separately from the received-from count, and mailaprop tracks each real-name form of a given email address separately.  I'd like to discuss the possibility of having ecomplete store that information too, so that mailaprop and ecomplete can work from the same data.

Should I start that thread on Emacs Devel?  I think so, but wanted to get your opinion first.

Best regards,
-Karl

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> Mailaprop stores a superset of the information ecomplete stores.
> Namely, mailaprop tracks the sent-to count separately from the
> received-from count, and mailaprop tracks each real-name form of a
> given email address separately.  I'd like to discuss the possibility
> of having ecomplete store that information too, so that mailaprop and
> ecomplete can work from the same data.
>
> Should I start that thread on Emacs Devel?  I think so, but wanted to
> get your opinion first.

Yes, sounds like a good idea.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>Karl Fogel <kfogel@red-bean.com> writes:
>
>> Mailaprop stores a superset of the information ecomplete stores.
>> Namely, mailaprop tracks the sent-to count separately from the
>> received-from count, and mailaprop tracks each real-name form of a
>> given email address separately.  I'd like to discuss the possibility
>> of having ecomplete store that information too, so that mailaprop and
>> ecomplete can work from the same data.
>>
>> Should I start that thread on Emacs Devel?  I think so, but wanted to
>> get your opinion first.
>
>Yes, sounds like a good idea.

Done:

https://lists.gnu.org/archive/html/emacs-devel/2018-02/msg00144.html

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

I wrote:
>Lars Ingebrigtsen <larsi@gnus.org> writes:
>>Looks very nice...  but the main difference between this and ecomplete
>>seems to be that ecomplete puts the completions in the echo area and
>>mailaprop uses a tooltip?
>
>Hmm, I can't tell.  As of commit 9f22b7d2, ecomplete.el has no function doc >strings nor any usage-informing variable doc strings, and the format of >~/.ecompleterc is not documented.  From looking at the elisp file, I wasn't >immediately sure what ecomplete does :-).  I grepped around and found this in >doc/misc/message.texi: [...]

Sorry, Lars.  I think my tone of voice was a little off there!  ecomplete looks great -- I was glad you pointed me to it, and mostly wished that I had found it earlier (say, before I wrote mailaprop, ahem).

I confess I got a little frustrated when I looked at the ecomplete.el source code, and that frustration is what came out in my original reply.  The code itself was very clean & readable.  But I was hungry to know the format and semantics of ~/.ecompleterc, and to know the exact completion behavior, because because then I would know whether mailaprop should continue on its own, or whether I should focus on using (and maybe extending) ecomplete.

Best regards,
-Karl

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> Sorry, Lars.  I think my tone of voice was a little off there!
> ecomplete looks great -- I was glad you pointed me to it, and mostly
> wished that I had found it earlier (say, before I wrote mailaprop,
> ahem).

No problem.  :-)  There definitely should be a comment in ecomplete.el
somewhere that documents the format.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

And I've now added doc strings and comments to ecomplete.el on master.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

But this reminded me that I wanted to have the matches "decay" when
they're not being used, so I've not implemented that and pushed to
master (with customisation support).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Karl Fogel]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>But this reminded me that I wanted to have the matches "decay" when
>they're not being used, so I've not implemented that and pushed to
>master (with customisation support).

At first I assumed "not" was a typo for "now" there.  But I don't see such a commit on master -- I only see the documentation commit 743323e5.

So I guess you really do mean that you have not implemented decay (yet), and that later, when you do, you plan to push it to master with customization support?

Best regards,
-Karl

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>But this reminded me that I wanted to have the matches "decay" when
>>they're not being used, so I've not implemented that and pushed to
>>master (with customisation support).
>
> At first I assumed "not" was a typo for "now" there.  But I don't see
> such a commit on master -- I only see the documentation commit
> 743323e5.

No, it's a typo for "now".  :-)  It this commit:

commit f121b5d7f3aef080d0019eb39a57de51015ceb39
Author: Lars Ingebrigtsen <larsi@gnus.org>
Date:   Tue Jan 16 15:22:11 2018 +0100

    Introduce a variable to control ecomplete sorting
    
    * lisp/ecomplete.el (ecomplete-sort-predicate): New variable.
    (ecomplete-get-matches): Use it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 16 Jan 2018 14:54:11 +0100
> Cc: Emacs Tangents <emacs-tangents@gnu.org>
> 
> And I've now added doc strings and comments to ecomplete.el on master.

Thanks, but as with any documentation fixes, this should have been
done on the release branch.

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 16 Jan 2018 14:54:11 +0100
>> Cc: Emacs Tangents <emacs-tangents@gnu.org>
>> 
>> And I've now added doc strings and comments to ecomplete.el on master.
>
> Thanks, but as with any documentation fixes, this should have been
> done on the release branch.

I assumed that the release branch was too close to release for
non-regression patches; even doc string additions.

I'll cherry-pick the patch and push it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [ANNOUNCE] mailaprop: modern popup-style autofill for email addresses

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: kfogel@red-bean.com,  emacs-tangents@gnu.org
> Date: Wed, 17 Jan 2018 11:53:23 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks, but as with any documentation fixes, this should have been
> > done on the release branch.
> 
> I assumed that the release branch was too close to release for
> non-regression patches; even doc string additions.

No, we are not there yet.

> I'll cherry-pick the patch and push it.

Thanks.