unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Streams: add stream-delay and streams of directory files
@ 2016-02-11 13:32 Michael Heerdegen
  2016-02-11 13:36 ` Michael Heerdegen
                   ` (2 more replies)
  0 siblings, 3 replies; 24+ messages in thread
From: Michael Heerdegen @ 2016-02-11 13:32 UTC (permalink / raw)
  To: Emacs Development; +Cc: Nicolas Petton

[-- Attachment #1: Type: text/plain, Size: 3944 bytes --]

Hello,

attached (at the end) is a patch for stream.el adding two things: (1) a
new macro `stream-delay' and (2) a function `stream-of-directory-files'
returning streams of directory files, optionally recursively.
Especially in the recursive case, a stream of directory files is better
to handle than a list: since it's delayed, it just encapsulates a
directory search to do.  No huge list needs to be accumulated; you can
stop when you have found what you have searched for, resume searching
later, etc. - just as with any stream.


Some notes about semantics and implementation:

(1) It turned out that the template of the implementation of `seq-copy'
for streams is useful in other scenarios, too.  I made the template a
macro `stream-delay', fixed the case of an empty stream that the old
`stream-copy' didn't handle correctly, and got that:

--8<---------------cut here---------------start------------->8---
(defmacro stream-delay (expr)
  "Return a new stream to be obtained by evaluating EXPR.
EXPR will be evaluated once when an element of the resulting
stream is requested for the first time, and must return a stream.
EXPR will be evaluated in the lexical environment present when
calling this function."
  (let ((stream (make-symbol "stream")))
    `(stream-make (let ((,stream ,expr))
                    (if (stream-empty-p ,stream)
                        nil
                      (cons (stream-first ,stream)
                            (stream-rest ,stream)))))))
--8<---------------cut here---------------end--------------->8---


So, now

--8<---------------cut here---------------start------------->8---
(cl-defmethod seq-copy ((stream stream))
  "Return a shallow copy of STREAM."
  (stream-delay stream))
--8<---------------cut here---------------end--------------->8---

As a function, it is different from `stream-delay': the argument is
evaluated.  OTOH `stream-delay' accepts an arbitrary expression that is
evaluated delayed (i.e. when an element is requested the firs ttime) and
must return a stream.  I need this for (2), see there.  When the EXPR is
evaluated later, this is done in the lexical environment present when
`stream-delay' was called (this happens automatically, because
`thunk-delay' creates a closure).


An example of what `stream-delay' does, and what not:

--8<---------------cut here---------------start------------->8---
ELISP> (let* ((a 0))
         (setq f (lambda (x) (setq a x)))
         (setq s (stream-delay (stream-range a 10))))
(--stream--..21..)

ELISP> s
(--stream--..21..)

ELISP> (funcall f 5)
5 (#o5, #x5, ?\C-e)

ELISP> (seq-into s 'list)
(5 6 7 8 9)

ELISP> (funcall f 1)
1 (#o1, #x1, ?\C-a)

ELISP> (seq-into s 'list)
(5 6 7 8 9)
--8<---------------cut here---------------end--------------->8---


(2) I chose the signature

 stream-of-directory-files (directory &optional full nosort recurse follow-links filter)

where FULL and NOSORT are directly passed to `directory-files', RECURSE
can be boolean or a predicate (of a dir name) to decide into which dirs
to recurse.  FILTER can be a predicate to limit the resulting stream to
files fulfilling it.  It replaces the MATCH argument of
`directory-files'.

A typical call to `stream-of-directory-files' looks like this:

(setq s (stream-of-directory-files
         "~/gnu-emacs" t nil
         (lambda (dir) (not (string= ".git" (file-name-nondirectory dir))))
         nil
         #'file-directory-p))

This sets `s' to a stream of files under "~/gnu-emacs" that will recurse
into all directories not named ".git".  The last (the FILTER) argument
specifies that only directories will be included.

The function call terminates immediately without looking at the file
system (as we expect for streams).  This is exactly the reason for why I
needed something like `stream-delay' - see the implementation.

For testing: to list the elements of s, one can do e.g.:

(seq-doseq (file s) (message "%s" file))


Finally the patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: stream.patch --]
[-- Type: text/x-diff, Size: 3701 bytes --]

diff --git a/stream.el b/stream.el
index 567a9e3..4d61cf1 100644
--- a/stream.el
+++ b/stream.el
@@ -152,7 +152,7 @@ range is infinite."
        (eq (car stream) stream--identifier)))
 
 (defun stream-empty ()
-  "Return an empty stream."
+  "Return a new empty stream."
   (list stream--identifier (thunk-delay nil)))
 
 (defun stream-empty-p (stream)
@@ -317,10 +317,69 @@ kind of nonlocal exit."
        (cons (stream-first stream)
              (seq-filter pred (stream-rest stream)))))))
 
+(defmacro stream-delay (expr)
+  "Return a new stream to be obtained by evaluating EXPR.
+EXPR will be evaluated once when an element of the resulting
+stream is requested for the first time, and must return a stream.
+EXPR will be evaluated in the lexical environment present when
+calling this function."
+  (let ((stream (make-symbol "stream")))
+    `(stream-make (let ((,stream ,expr))
+                    (if (stream-empty-p ,stream)
+                        nil
+                      (cons (stream-first ,stream)
+                            (stream-rest ,stream)))))))
+
 (cl-defmethod seq-copy ((stream stream))
   "Return a shallow copy of STREAM."
-  (stream-cons (stream-first stream)
-               (stream-rest stream)))
+  (stream-delay stream))
+
+(defun stream-of-directory-files-1 (directory &optional nosort recurse follow-links)
+  "Helper for `stream-of-directory-files'."
+  (stream-delay
+   (if (file-accessible-directory-p directory)
+       (let (files dirs (reverse-fun (if nosort #'identity #'nreverse)))
+         (dolist (file (directory-files directory t nil nosort))
+           (let ((is-dir (file-directory-p file)))
+             (unless (and is-dir
+                          (member (file-name-nondirectory (directory-file-name file))
+                                  '("." "..")))
+               (push file files)
+               (when (and is-dir
+                          (or follow-links (not (file-symlink-p file)))
+                          (if (functionp recurse) (funcall recurse file) recurse))
+                 (push file dirs)))))
+         (apply #'stream-append
+                (stream (funcall reverse-fun files))
+                (mapcar
+                 (lambda (dir) (stream-of-directory-files-1 dir nosort recurse follow-links))
+                 (funcall reverse-fun dirs))))
+     (stream-empty))))
+
+(defun stream-of-directory-files (directory &optional full nosort recurse follow-links filter)
+  "Return a stream of names of files in DIRECTORY.
+Call `directory-files' to list file names in DIRECTORY and make
+the result a stream.  Don't include files named \".\" or \"..\".
+The arguments FULL and NOSORT are directly passed to
+`directory-files'.
+
+Third optional argument RECURSE non-nil means recurse on
+subdirectories.  If RECURSE is a function, it should be a
+predicate accepting one argument, an absolute file name of a
+directory, and return non-nil when the returned stream should
+recurse into that directory.  Any other non-nil value means
+recurse into every readable subdirectory.
+
+Even with recurse non-nil, don't descent into directories by
+following symlinks unless FOLLOW-LINKS is non-nil.
+
+If FILTER is non-nil, it should be a predicate accepting one
+argument, an absolute file name.  It is used to limit the
+resulting stream to the files fulfilling this predicate."
+  (let* ((stream (stream-of-directory-files-1 directory nosort recurse follow-links))
+         (filtered-stream (if filter (seq-filter filter stream) stream)))
+    (if full filtered-stream
+      (seq-map (lambda (file) (file-relative-name file directory)) filtered-stream))))
 
 (provide 'stream)
 ;;; stream.el ends here
-- 
2.7.0


^ permalink raw reply related	[flat|nested] 24+ messages in thread
end of thread, other threads:[~2016-03-04 14:21 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-02-11 13:32 Streams: add stream-delay and streams of directory files Michael Heerdegen
2016-02-11 13:36 ` Michael Heerdegen
2016-02-11 20:33 ` Nicolas Petton
2016-02-12 14:18   ` Michael Heerdegen
2016-02-13 19:25     ` Nicolas Petton
2016-02-14 12:03       ` Michael Heerdegen
2016-02-14 12:07         ` Nicolas Petton
2016-02-14 13:55         ` Nicolas Petton
2016-02-14 14:24           ` Michael Heerdegen
2016-02-24 16:38   ` Michael Heerdegen
2016-02-24 17:01     ` Nicolas Petton
2016-02-29 15:05     ` Nicolas Petton
2016-03-03 14:42     ` Nicolas Petton
2016-03-03 15:26       ` Michael Heerdegen
2016-03-03 15:29         ` Nicolas Petton
2016-03-03 22:05           ` Artur Malabarba
2016-03-04  9:02             ` Nicolas Petton
2016-03-04 12:15             ` Michael Heerdegen
2016-03-04 14:21               ` Artur Malabarba
2016-02-11 20:48 ` Eli Zaretskii
2016-02-11 21:10   ` Nicolas Petton
2016-02-11 21:22     ` Eli Zaretskii
2016-02-12 14:14       ` Michael Heerdegen
2016-02-12 15:22         ` Eli Zaretskii

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).