Re: map-file-lines

[Thierry Volpiatto <thierry.volpiatto@gmail.com>]

Hi Ted!
Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 02 Feb 2009 11:20:07 -0600 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>
> TZ> Emacs Lisp lacks a good way to iterate over all the lines of a file,
> TZ> especially for a large file.  The following code tries to provide a
> TZ> solution, concentrating on reading a block of data in one shot and then
> TZ> processing it line by line.  It may be more efficient to write this in
> TZ> C.  Also, it does not deal with cases where the first line read is
> TZ> bigger than the buffer size, and may have other bugs, but it works for
> TZ> me so I thought I'd post it for comments and criticism.
>
> Updated:
>
> - line count 0-based now, logic is cleaner
> - buffer size 128K by default
> - accept start line and count
> - abort when the lambda func returns nil
> - renamed endline to line-end for clarity
>
> Thanks
> Ted

Can you try `tve-flines-iterator' that work a little like python
iterators work.
I plan to use it in futures versions of traverselisp if it is faster than
actual code (traverselisp don't use this code actually).

,----[ C-h f tve-flines-iterator RET ]
| tve-flines-iterator is a Lisp function in `traverselisp.el'.
| 
| (tve-flines-iterator file &optional nlines startpos bufsize)
| 
| Return an iterator on `nlines' lines of file.
| `startpos' and `bufsize' are the byte options to give to
| `insert-file-contents'.
| 
| [back]
| 
| ===*===*===*===*===*===*===*===*===*===*===
| Example:
| ,----
| | ;; create an elisp-iterator object that
| | ;; record the first 1024 bytes of my .emacs
| | (setq A (tve-flines-iterator "~/.emacs.el" nil 0 1024))
| | 
| | ;; eval as many times as needed or launch it in a loop
| | (tve-next A)
| `----
`----

You can get it with hg here:
hg clone http://freehg.org/u/thiedlecques/traverselisp/

> (defun map-file-lines (file func &optional startline count bufsize)
>   (let ((filepos 0)
> 	(linenum 0)
> 	(bufsize (or bufsize (* 128 1024))))
>     (with-temp-buffer
>       (while
> 	  (let*
> 	      ((inserted (insert-file-contents
> 			  file nil
> 			  filepos (+ filepos bufsize) 
> 			  t))
> 	       (numlines (count-lines (point-min) (point-max)))
> 	       (read (nth 1 inserted))
> 	       (done (< 1 read))
> 	       result line-end)
> 	    (dotimes (n (count-lines (point-min) (point-max)))
> 		(goto-char (point-min))
> 		(setq line-end (line-end-position)
> 		      result (if (and startline (< linenum startline))
> 				 ()
> 			       (if (and 
> 				    count
> 				    (>= (- linenum startline) count))
> 				   (return)
> 				 (funcall func 
> 					  (buffer-substring 
> 					   (line-beginning-position)
> 					   line-end)
> 					  linenum)))
> 		      done (and done result))
> 		(incf filepos line-end)
> 		(forward-line)
> 		(incf linenum))
> 	    done)))
>     linenum))
>
> ;;(map-file-lines "/tmp/test" (lambda (line num) (message "%d: %s" num line)))
> ;;(map-file-lines "/tmp/test" (lambda (line num) (message "%d: %s" num line)) 100)
> ;;(map-file-lines "/tmp/test" (lambda (line num) (message "%d: %s" num line)) 100 10)
>

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France