From: Oleh Krehel <ohwoeowho@gmail.com>
To: emacs-devel@gnu.org
Subject: What to do for faster `remove-duplicates'?
Date: Wed, 06 May 2015 09:33:02 +0200 [thread overview]
Message-ID: <87383atb2p.fsf@gmail.com> (raw)
Hi all,
Should `cl-remove-duplicates' switch to using a hash table if the
candidates list is large?
I see here a speedup factor of >500 for removing the duplicates of
`load-library' candidates. Helm has a function for this, which looks
very simple (much more simple that the current `cl-remove-duplicates'):
(cl-defun helm-fast-remove-dups (seq &key (test 'eq))
"Remove duplicates elements in list SEQ.
This is same as `remove-duplicates' but with memoisation.
It is much faster, especially in large lists.
A test function can be provided with TEST argument key.
Default is `eq'."
(cl-loop with cont = (make-hash-table :test test)
for elm in seq
unless (gethash elm cont)
do (puthash elm elm cont)
finally return
(cl-loop for i being the hash-values in cont collect i)))
And here are the benchmark tests:
(setq cands (locate-file-completion-table
load-path (get-load-suffixes) "" nil t))
(length cands)
5357
(length (cl-remove-duplicates cands :test 'equal))
2481
(benchmark-run (cl-remove-duplicates cands :test 'equal))
(0.67873101 0 0.0)
(benchmark-run (helm-fast-remove-dups cands :test 'equal))
(0.001350054 0 0.0)
(benchmark-run (seq-uniq cands 'equal))
(5.270219822 27 2.396615401000002)
This is also a request for seq.el to revise its optimization strategy.
So should `cl-remove-duplicates' be a 2-in-1 (lists for small input,
hash table for large input), or should there be
`cl-remove-duplicates-hash', or an optional argument to
`cl-remove-duplicates' to use a hash?
Oleh
next reply other threads:[~2015-05-06 7:33 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-06 7:33 Oleh Krehel [this message]
2015-05-06 12:59 ` What to do for faster `remove-duplicates'? Stefan Monnier
2015-05-06 13:30 ` Oleh Krehel
2015-05-06 13:50 ` Stefan Monnier
2015-05-06 13:51 ` Oleh Krehel
2015-05-06 17:43 ` Thierry Volpiatto
2015-05-06 17:54 ` Oleh Krehel
2015-05-06 18:31 ` Artur Malabarba
2015-05-06 18:33 ` Artur Malabarba
2015-05-06 18:41 ` Oleh Krehel
2015-05-06 19:24 ` Artur Malabarba
2015-05-06 19:32 ` Oleh Krehel
2015-05-06 21:22 ` Artur Malabarba
2015-05-06 18:48 ` Thierry Volpiatto
2015-05-06 20:54 ` Stefan Monnier
2015-05-06 20:54 ` Stefan Monnier
2015-05-06 21:18 ` Artur Malabarba
2015-05-06 14:04 ` Artur Malabarba
2015-05-06 14:13 ` Oleh Krehel
2015-05-06 14:49 ` Artur Malabarba
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87383atb2p.fsf@gmail.com \
--to=ohwoeowho@gmail.com \
--cc=emacs-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.