Re: map-file-lines

[Ted Zlatanov <tzz@lifelogs.com>]

On Mon, 02 Feb 2009 17:42:31 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> How would you use map-file-lines to implement the "large file viewing
SM> and searching"?
...
SM> I know about the large-file problem, obviously, but I wonder what kind
SM> of UI you expect to provide, in order for it to be able to work just one
SM> line at a time.
...
SM> I don't se how such a "line-at-a-time" or "chunk-at-a-time" processing
SM> (i.e. stream processing) will enable Emacs to let you conveniently edit
SM> a large binary file.
...
SM> OK, that's indeed how I imagine it as well, but I fail to see how this
SM> relates to map-file-lines.  All you need for that is to use the BEG and
SM> END args of insert-file-contents (and maybe also to extend those args
SM> so they can be floats, in case Emacs's ints are too limited).

On Tue, 03 Feb 2009 08:27:01 +0100 joakim@verona.se wrote: 

j> That being said, how would the insert-file-contents solution work in
j> practice? Has something been done along these lines already? Would it be
j> possible to make some kind of generic solution that would make for
j> instance hexl mode work on large files?

I tried to condense the various messages into one reply.

map-file-lines as it stands is just a stream processor, and not useful
to *view* a large file.  Unfortunately Emacs does almost everything in
the buffer, so true large-file view and edit requires core-level work to
make the buffer able to address the whole file.  In addition, almost
every Emacs package assumes that the buffer is small and can be scanned
quickly; imagine how slow it will be to open and edit a 20GB gzipped
file.

map-file-lines was much easier to implement, and (just like
`grep' and many other utilities) can be useful on its own despite the
single-line limitations.

So to answer your and Joakim's questions, based on what I know so far,
the best approach is to have a special narrowing mode to view large
files.  It won't use map-file-lines, but like it, it will fetch the next
chunk only when needed.  It needs to be special because the normal
narrow/widen calls should not widen to the whole file.  It's best, in
fact, if the decision to move the "window" into the file back and forth
is only left to the user and to special motion commands, and normal
packages can not move that window.

Writing inserted text, in particular, is very slow with large files.  A
binary editor is not so hard to implement, but inserting text may need
special permission (like the motion commands above) to make the user
experience bearable.

Ted