all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#40355: [PATCH] Implement caching for libravatar lookup
@ 2020-03-31 18:03 Philip K
  2020-03-31 19:58 ` Robert Pluim
  2020-08-08 13:03 ` Lars Ingebrigtsen
  0 siblings, 2 replies; 9+ messages in thread
From: Philip K @ 2020-03-31 18:03 UTC (permalink / raw)
  To: 40355

Checking if a domain has an avatar server associated with it requires
at most two (synchronous) DNS queries, at least to ultimately default
to "libravatar.org". Caching the results of domain lookups reduced the
evaluation time (on my machine) from ~0.3s to an instantaneous
evaluation.

* lisp/image/gravatar.el (gravatar--service-libravatar): Check if
  domain has already been resolved before staring DNS queries
(gravatar-libravatar-cache): New variable.
---
 lisp/image/gravatar.el | 40 +++++++++++++++++++++++++++++++---------
 1 file changed, 31 insertions(+), 9 deletions(-)

diff --git a/lisp/image/gravatar.el b/lisp/image/gravatar.el
index a9cd540aa4..2572f9136f 100644
--- a/lisp/image/gravatar.el
+++ b/lisp/image/gravatar.el
@@ -138,14 +138,24 @@ gravatar-service
   :link '(url-link "https://gravatar.com/")
   :group 'gravatar)
 
-(defun gravatar--service-libravatar (addr)
-  "Find domain that hosts avatars for email address ADDR."
+(defvar gravatar-libravatar-cache (make-hash-table :test 'equal)
+  "Cache for `gravatar--service-libravatar'.")
+
+(defun gravatar--service-libravatar (addr &optional cache)
+  "Find domain that hosts avatars for email address ADDR.
+The optional argument CACHE must either be a hash table to
+memorise avatar server resolutions in, or nil, in which case it
+will default to `gravatar-libravatar-cache'."
   ;; implements https://wiki.libravatar.org/api/
   (save-match-data
     (if (not (string-match ".+@\\(.+\\)" addr))
         "https://seccdn.libravatar.org/avatar"
-      (let ((domain (match-string 1 addr)))
+      (let ((domain (downcase (match-string 1 addr))))
         (catch 'found
+          (setq cache (or cache gravatar-libravatar-cache))
+          (let ((cache (gethash domain cache)))
+            (when (and cache (time-less-p (current-time) (cdr cache)))
+              (throw 'found (car cache))))
           (dolist (record '(("_avatars-sec" . "https")
                             ("_avatars" . "http")))
             (let* ((query (concat (car record) "._tcp." domain))
@@ -173,12 +183,24 @@ gravatar--service-libravatar
                                (<= 1 (dns-get 'port rec) 65535)
                                (string-match-p "\\`[-.0-9A-Za-z]+\\'"
                                                (dns-get 'target rec)))
-                      (throw 'found (format "%s://%s:%s/avatar"
-                                            (cdr record)
-                                            (dns-get 'target rec)
-                                            (dns-get 'port rec))))
-                    (setq sum (- sum (dns-get 'weight rec)))))))
-            "https://seccdn.libravatar.org/avatar"))))))
+                      (let ((url (format "%s://%s:%s/avatar"
+                                         (cdr record)
+                                         (dns-get 'target rec)
+                                         (dns-get 'port rec)))
+                            (timeout (if (time-less-p
+                                          (seconds-to-time (dns-get 'ttl rec))
+                                          (days-to-time 1))
+                                         (days-to-time 1)
+                                       (seconds-to-time (dns-get 'ttl rec)))))
+                        (puthash domain
+                                 (cons url (time-add (current-time) timeout))
+                                 cache)
+                        (throw 'found url)))
+                    (setq sum (- sum (dns-get 'weight rec))))))))
+          (car (puthash domain
+                        (cons "https://seccdn.libravatar.org/avatar"
+                              (time-add (current-time) (days-to-time 30)))
+                        cache)))))))
 
 (defun gravatar-hash (mail-address)
   "Return the Gravatar hash for MAIL-ADDRESS."
-- 
2.20.1






^ permalink raw reply related	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-03-31 18:03 bug#40355: [PATCH] Implement caching for libravatar lookup Philip K
@ 2020-03-31 19:58 ` Robert Pluim
  2020-03-31 21:30   ` Philip K.
  2020-08-08 13:03 ` Lars Ingebrigtsen
  1 sibling, 1 reply; 9+ messages in thread
From: Robert Pluim @ 2020-03-31 19:58 UTC (permalink / raw)
  To: Philip K; +Cc: 40355

>>>>> On Tue, 31 Mar 2020 20:03:36 +0200, Philip K <philip@warpmail.net> said:

    Philip> Checking if a domain has an avatar server associated with it requires
    Philip> at most two (synchronous) DNS queries, at least to ultimately default
    Philip> to "libravatar.org". Caching the results of domain lookups reduced the
    Philip> evaluation time (on my machine) from ~0.3s to an instantaneous
    Philip> evaluation.

I doubt you need the 'cache' arg, you can keep the cache purely
internal.

How big is this cache likely to get? Perhaps use a weak hash table?

Robert





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-03-31 19:58 ` Robert Pluim
@ 2020-03-31 21:30   ` Philip K.
  2020-04-01  7:27     ` Robert Pluim
  0 siblings, 1 reply; 9+ messages in thread
From: Philip K. @ 2020-03-31 21:30 UTC (permalink / raw)
  To: Robert Pluim; +Cc: 40355

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 31 Mar 2020 20:03:36 +0200, Philip K <philip@warpmail.net> said:
>
> I doubt you need the 'cache' arg, you can keep the cache purely
> internal.

I added it because I was considering defining gravatar-libravatar-cache
as

    (let ((cache (make-hash-table :test 'equal)))
      (dolist (domain '("aol.com" "att.net" "comcast.net" "facebook.com"
                        "gmail.com" "gmx.com" "googlemail.com" "google.com"
                        "hotmail.com" "hotmail.co.uk" "mac.com" "me.com"
                        "mail.com" "msn.com" "live.com" "sbcglobal.net"
                        ;; ...
                        ))
        (gravatar--service-libravatar (concat "ignored@" domain) cache))
      cache)

with a few popular domains. The only issue is that evaluating it takes
forever.

> How big is this cache likely to get? Perhaps use a weak hash table?

I haven't managed to gather good real-world data yet, but I would assume
something along the lines of 50-200 domains?

-- 
	Philip K.





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-03-31 21:30   ` Philip K.
@ 2020-04-01  7:27     ` Robert Pluim
  2020-04-01 10:34       ` Philip K.
  0 siblings, 1 reply; 9+ messages in thread
From: Robert Pluim @ 2020-04-01  7:27 UTC (permalink / raw)
  To: Philip K.; +Cc: 40355

>>>>> On Tue, 31 Mar 2020 23:30:23 +0200, philip@warpmail.net (Philip K.) said:

    Philip> Robert Pluim <rpluim@gmail.com> writes:
    >>>>>>> On Tue, 31 Mar 2020 20:03:36 +0200, Philip K <philip@warpmail.net> said:
    >> 
    >> I doubt you need the 'cache' arg, you can keep the cache purely
    >> internal.

    Philip> I added it because I was considering defining gravatar-libravatar-cache
    Philip> as

    Philip>     (let ((cache (make-hash-table :test 'equal)))
    Philip>       (dolist (domain '("aol.com" "att.net" "comcast.net" "facebook.com"
    Philip>                         "gmail.com" "gmx.com" "googlemail.com" "google.com"
    Philip>                         "hotmail.com" "hotmail.co.uk" "mac.com" "me.com"
    Philip>                         "mail.com" "msn.com" "live.com" "sbcglobal.net"
    Philip>                         ;; ...
    Philip>                         ))
    Philip>         (gravatar--service-libravatar (concat "ignored@" domain) cache))
    Philip>       cache)

    Philip> with a few popular domains. The only issue is that evaluating it takes
    Philip> forever.

Perhaps thatʼs a sign that pre-population is not needed :-)

    >> How big is this cache likely to get? Perhaps use a weak hash table?

    Philip> I haven't managed to gather good real-world data yet, but I would assume
    Philip> something along the lines of 50-200 domains?

OK, I donʼt think thatʼs onerous.

Robert





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-04-01  7:27     ` Robert Pluim
@ 2020-04-01 10:34       ` Philip K.
  0 siblings, 0 replies; 9+ messages in thread
From: Philip K. @ 2020-04-01 10:34 UTC (permalink / raw)
  To: Robert Pluim; +Cc: 40355

Robert Pluim <rpluim@gmail.com> writes:

>     Philip> I added it because I was considering defining gravatar-libravatar-cache
>     Philip> as
>
>     Philip>     (let ((cache (make-hash-table :test 'equal)))
>     Philip>       (dolist (domain '("aol.com" "att.net" "comcast.net" "facebook.com"
>     Philip>                         "gmail.com" "gmx.com" "googlemail.com" "google.com"
>     Philip>                         "hotmail.com" "hotmail.co.uk" "mac.com" "me.com"
>     Philip>                         "mail.com" "msn.com" "live.com" "sbcglobal.net"
>     Philip>                         ;; ...
>     Philip>                         ))
>     Philip>         (gravatar--service-libravatar (concat "ignored@" domain) cache))
>     Philip>       cache)
>
>     Philip> with a few popular domains. The only issue is that evaluating it takes
>     Philip> forever.
>
> Perhaps thatʼs a sign that pre-population is not needed :-)

It's not needed, but as soon as it is pre-populated the user experience
becomes a lot better. But unless there's a sensible way to fix it, I'll
just resubmit the patch without the optional argument :/

-- 
	Philip K.





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-03-31 18:03 bug#40355: [PATCH] Implement caching for libravatar lookup Philip K
  2020-03-31 19:58 ` Robert Pluim
@ 2020-08-08 13:03 ` Lars Ingebrigtsen
  2020-08-18 14:46   ` Lars Ingebrigtsen
  1 sibling, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2020-08-08 13:03 UTC (permalink / raw)
  To: Philip K; +Cc: 40355

Philip K <philip@warpmail.net> writes:

> Checking if a domain has an avatar server associated with it requires
> at most two (synchronous) DNS queries, at least to ultimately default
> to "libravatar.org". Caching the results of domain lookups reduced the
> evaluation time (on my machine) from ~0.3s to an instantaneous
> evaluation.
>
> * lisp/image/gravatar.el (gravatar--service-libravatar): Check if
>   domain has already been resolved before staring DNS queries
> (gravatar-libravatar-cache): New variable.

The code has changed a bit (and become asynchronous), but adding caching
would be nice.

Can you re-spin the patch on top of the current implementation?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-08-08 13:03 ` Lars Ingebrigtsen
@ 2020-08-18 14:46   ` Lars Ingebrigtsen
  2020-08-24 14:09     ` Robert Pluim
  0 siblings, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2020-08-18 14:46 UTC (permalink / raw)
  To: Philip K; +Cc: 40355

Lars Ingebrigtsen <larsi@gnus.org> writes:

> The code has changed a bit (and become asynchronous), but adding caching
> would be nice.
>
> Can you re-spin the patch on top of the current implementation?

I've now implemented this in a very different way -- with a general
in-memory cache for all types of gravatars.  They're pretty small
images, anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-08-18 14:46   ` Lars Ingebrigtsen
@ 2020-08-24 14:09     ` Robert Pluim
  2020-08-24 14:13       ` Lars Ingebrigtsen
  0 siblings, 1 reply; 9+ messages in thread
From: Robert Pluim @ 2020-08-24 14:09 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: 40355, Philip K

>>>>> On Tue, 18 Aug 2020 16:46:09 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Lars Ingebrigtsen <larsi@gnus.org> writes:
    >> The code has changed a bit (and become asynchronous), but adding caching
    >> would be nice.
    >> 
    >> Can you re-spin the patch on top of the current implementation?

    Lars> I've now implemented this in a very different way -- with a general
    Lars> in-memory cache for all types of gravatars.  They're pretty small
    Lars> images, anyway.

Doesnʼt url-retrieve cache on-disk anyway?

Robert





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#40355: [PATCH] Implement caching for libravatar lookup
  2020-08-24 14:09     ` Robert Pluim
@ 2020-08-24 14:13       ` Lars Ingebrigtsen
  0 siblings, 0 replies; 9+ messages in thread
From: Lars Ingebrigtsen @ 2020-08-24 14:13 UTC (permalink / raw)
  To: Robert Pluim; +Cc: 40355, Philip K

Robert Pluim <rpluim@gmail.com> writes:

> Doesnʼt url-retrieve cache on-disk anyway?

It does if you ask it to, but I thought it was simpler to just do a
general cache -- that way we didn't need two levels of caches.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-08-24 14:13 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-03-31 18:03 bug#40355: [PATCH] Implement caching for libravatar lookup Philip K
2020-03-31 19:58 ` Robert Pluim
2020-03-31 21:30   ` Philip K.
2020-04-01  7:27     ` Robert Pluim
2020-04-01 10:34       ` Philip K.
2020-08-08 13:03 ` Lars Ingebrigtsen
2020-08-18 14:46   ` Lars Ingebrigtsen
2020-08-24 14:09     ` Robert Pluim
2020-08-24 14:13       ` Lars Ingebrigtsen

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.