Reading huge files

Reading huge files

[Leonid Grinberg]

Hello,

I need to read a file, which contains approximately 1.2GB of data.
Obviously, this is well above the maximum buffer limit. Is there any
way for me to read it without

1) breaking it up into smaller files and
2) using a different editor

I am using GNU Emacs 21.4.1 on Ubuntu Edgy Eft on a Dell Latitude C600
with 256MB of memory.

Thanks in advance!

--
Leonid Grinberg

Re: Reading huge files

[Kevin Rodgers]

Leonid Grinberg wrote:
> I need to read a file, which contains approximately 1.2GB of data.
> Obviously, this is well above the maximum buffer limit. Is there any
> way for me to read it without
> 
> 1) breaking it up into smaller files and
> 2) using a different editor

Not that I'm aware of.  You could cook up something like:

(defun view-large-file-contents (file beg end)
   "View FILE contents from bytes BEG through END, in View mode."
   (interactive "fView file: \nnFrom byte: \nnTo byte: ")
   (switch-to-buffer
    (generate-new-buffer (format "%s[%d,%d]"
				(file-name-nondirectory file) beg end)))
   (insert-file-contents file nil beg end)
   (view-mode 1))

> I am using GNU Emacs 21.4.1 on Ubuntu Edgy Eft on a Dell Latitude C600
> with 256MB of memory.

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: Reading huge files

[Eli Zaretskii]

> From: Kevin Rodgers <kevin.d.rodgers@gmail.com>
> Date: Tue, 09 Jan 2007 08:24:52 -0700
> 
> (defun view-large-file-contents (file beg end)
>    "View FILE contents from bytes BEG through END, in View mode."
>    (interactive "fView file: \nnFrom byte: \nnTo byte: ")
>    (switch-to-buffer
>     (generate-new-buffer (format "%s[%d,%d]"
> 				(file-name-nondirectory file) beg end)))
>    (insert-file-contents file nil beg end)
>    (view-mode 1))

Does this really work?  I think it won't: the underlying problem is
that for large files, END overflows the limits of the ELisp integer
type, and your suggestion doesn't resolve this fundamental problem.
So insert-file-contents will still barf with large files.

Or am I missing something?

Re: Reading huge files

[Kevin Rodgers]

Eli Zaretskii wrote:
>> From: Kevin Rodgers <kevin.d.rodgers@gmail.com>
>> Date: Tue, 09 Jan 2007 08:24:52 -0700
>>
>> (defun view-large-file-contents (file beg end)
>>    "View FILE contents from bytes BEG through END, in View mode."
>>    (interactive "fView file: \nnFrom byte: \nnTo byte: ")
>>    (switch-to-buffer
>>     (generate-new-buffer (format "%s[%d,%d]"
>> 				(file-name-nondirectory file) beg end)))
>>    (insert-file-contents file nil beg end)
>>    (view-mode 1))
> 
> Does this really work?  I think it won't: the underlying problem is
> that for large files, END overflows the limits of the ELisp integer
> type, and your suggestion doesn't resolve this fundamental problem.
> So insert-file-contents will still barf with large files.
> 
> Or am I missing something?

No, you are far wiser in the ways of the buddha.

Why does insert-file-contents signal an error if you pass floating point
values for BEG and END?

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: Reading huge files

[Stefan Monnier]

> Why does insert-file-contents signal an error if you pass floating point
> values for BEG and END?

Because it doesn't know better.  Patches welcome (after the release of
Emacs-22).


        Stefan

Re: Reading huge files

[Mathias Dahl]

Eli Zaretskii <eliz@gnu.org> writes:

> Does this really work?  I think it won't: the underlying problem is
> that for large files, END overflows the limits of the ELisp integer
> type, and your suggestion doesn't resolve this fundamental problem.
> So insert-file-contents will still barf with large files.
>
> Or am I missing something?

I have only tested the code below with a 700 MB large movie file, and
that is probably within the bounds you talk about.  But even if it
would fail (when does it fail?), being able to browse files hundreds
of MB in size seems quite nice. Hence my new hack below.

Enjoy!

/Mathias

;;; vlf.el --- View Large Files

;; Copyright (C) 2006  Mathias Dahl

;; Version: 0.1
;; Keywords: files, helpers, utilities
;; Author: Mathias Dahl <mathias.rem0veth1s.dahl@gmail.com>
;; Maintainer: Mathias Dahl
;; URL: http://www.emacswiki.org/cgi-bin/wiki/VLF

;; This file is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 2, or (at your option)
;; any later version.

;; This file is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to
;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
;; Boston, MA 02111-1307, USA.

;;; Commentary:
;; 
;; After reading the Nth post on Gnu Emacs Help about Viewing Large
;; Files in Emacs, it itched so much that I decided to make a try.  It
;; helped quite a lot when Kevin Rodgers posted a snippet on how to
;; use `insert-file-contents' to extract part of a file.  At first I
;; made a try using head and tail and that worked too, but using
;; internal Emacs commands is nicer.  Here is the code to extract data
;; using head and tail in case someone wanna try that out in the
;; future:

;; (defun vlf-extract-part-of-file (file from to)
;;   "Returns bytes in FILE from FROM to TO."
;;   (let ((size (vlf-file-size file)))
;;     (if (or (> from size)
;;             (> to size))
;;         (error "From or to is larger that the file size"))
;;     (with-temp-buffer
;;       (shell-command
;;        (format "head --bytes %d %s | tail --bytes %d"
;; 	       to file (+ (- to from) 1)) t)
;;       (buffer-substring (point-min) (point-max)))))

;;; History:
;;
;; - Wed Jan 10 00:13:45 2007
;;
;;    First version created and released into the wild.

;;; Bugs
;;
;; Probably some. Feel free to fix them :)

;;; Code:

(defgroup vlf nil
  "Browse large files in Emacs"
  :prefix "vlf-"
  :group 'files)

(defcustom vlf-batch-size 1000
  "Defines how large each batch of file data is."
  :type 'integer
  :group 'vlf)

(defvar vlf-current-start-pos 1
  "Keeps track of file position.")

(defvar vlf-current-batch-size nil
  "Keeps track of current batch size.")

(defvar vlf-current-file nil
  "File that is currently viewed.")

(defvar vlf-mode-map (make-sparse-keymap)
  "Keymap for `vlf-mode'.")

(defun vlf-define-keymap ()
  "Define keymap for `vlf-mode'."
  (define-key vlf-mode-map [next] 'vlf-next)
  (define-key vlf-mode-map [prior] 'vlf-prev)
  (define-key vlf-mode-map "q" 'vlf-quit))

(define-derived-mode vlf-mode
  fundamental-mode "vlf-mode"
  "Mode to browse large files in.
See `vlf' for details."
  (vlf-define-keymap)
  (toggle-read-only 1)
  (message "vlf-mode enabled"))

(defun vlf-file-size (file)
  "Get size of FILE."
  (nth 7 (file-attributes file)))

(defun vlf-quit ()
  "Quit vlf."
  (interactive)
  (kill-buffer (current-buffer)))

(defun vlf-insert-batch ()
  "Insert current batch of data."
  (let* ((beg (1- vlf-current-start-pos))
        (end (+ beg vlf-current-batch-size)))
  (insert-file-contents
   vlf-current-file nil
   beg end)))

(defun vlf-next ()
  "Display the next batch of file data."
  (interactive)
  (let ((inhibit-read-only t)
        left next-start-pos
        (size (vlf-file-size vlf-current-file)))
    (setq next-start-pos (+ vlf-current-start-pos
                            vlf-batch-size))
    (if (> next-start-pos size)
        (message "End of file")
      (setq vlf-current-batch-size
            vlf-batch-size
            vlf-current-start-pos next-start-pos
            left (1+ (- size vlf-current-start-pos)))     
      (if (< left vlf-current-batch-size)
          (setq vlf-current-batch-size left))
      (erase-buffer)
      (vlf-insert-batch)
      (rename-buffer
       (format "%s[%d,%d]"
               (file-name-nondirectory vlf-current-file)
               vlf-current-start-pos
               (1- (+ vlf-current-start-pos
                      vlf-current-batch-size)))))))

(defun vlf-prev ()
  "Display the previous batch of file data."
  (interactive)
  (if (= 1 vlf-current-start-pos)
      (message "At beginning of file")
    (let ((inhibit-read-only t))
      (erase-buffer)
      (setq vlf-current-start-pos (- vlf-current-start-pos
                                     vlf-batch-size)
            vlf-current-batch-size vlf-batch-size)
      (vlf-insert-batch)
      (rename-buffer
       (format "%s[%d,%d]"
               (file-name-nondirectory vlf-current-file)
               vlf-current-start-pos
               (1- (+ vlf-current-start-pos
                      vlf-current-batch-size)))))))

(defun vlf (file)
  "View a large file in Emacs FILE is the file to open.
Batches of the file data from FILE will be displayed in a
read-only buffer.  You can customize the amount of bytes to
display by customizing `vlf-batch-size'."
  (interactive "fFile to open: ")
  (setq vlf-current-file file
        vlf-current-start-pos 1
        vlf-current-batch-size
        (1- (+ vlf-current-start-pos
               vlf-batch-size)))
  (switch-to-buffer
   (generate-new-buffer (format "%s[%d,%d]"
				(file-name-nondirectory file)
                                vlf-current-start-pos
                                (1- (+ vlf-current-start-pos
                                       vlf-current-batch-size)))))
  (erase-buffer)
  (vlf-insert-batch)
  (vlf-mode))

;; (setq vlf-current-file "/home/mathias/movies/devil_in_a_blue_dress.avi")
;; (setq vlf-current-file "/home/mathias/extracttext")
;; (setq vlf-batch-size 3)
;; (setq vlf-current-start-pos 1)

(provide 'vlf)

;;; vlf.el ends here

Re: Reading huge files

[Leonid Grinberg]

Nice, thank you! That was very helpful.

It is still annoying that reading it normally doesn't work, though!

Re: Reading huge files

[Mathias Dahl]

"Leonid Grinberg" <lgrinberg@gmail.com> writes:

> Nice, thank you! That was very helpful.

My pleasure.

> It is still annoying that reading it normally doesn't work, though!

Define `normal' in Emacs :)

Seriously, if you wanted to you could hook this onto find-file in some
way so that this would be used for files over a certain limit, maybe
with a question. Hack away!

Re: Reading huge files

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@gmail.com>
> Date: Wed, 10 Jan 2007 00:29:15 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Does this really work?  I think it won't: the underlying problem is
> > that for large files, END overflows the limits of the ELisp integer
> > type, and your suggestion doesn't resolve this fundamental problem.
> > So insert-file-contents will still barf with large files.
> >
> > Or am I missing something?
> 
> I have only tested the code below with a 700 MB large movie file, and
> that is probably within the bounds you talk about.  But even if it
> would fail (when does it fail?), being able to browse files hundreds
> of MB in size seems quite nice. Hence my new hack below.

If this is a 32-bit machine, then the Lisp integer overflows at
256MB.  If that's a 64-bit machine, then you don't need any tricks to
edit 700MB files.

Re: Reading huge files

[Mathias Dahl]

Eli Zaretskii <eliz@gnu.org> writes:

> If this is a 32-bit machine, then the Lisp integer overflows at
> 256MB.  If that's a 64-bit machine, then you don't need any tricks
> to edit 700MB files.

Okay. Then I probably never scrolled that far when I tested :) I guess
the head + tail trick would work though, I suspect those tools does
not have this limitation.

Re: Reading huge files

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@gmail.com>
> Date: Wed, 10 Jan 2007 08:39:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If this is a 32-bit machine, then the Lisp integer overflows at
> > 256MB.  If that's a 64-bit machine, then you don't need any tricks
> > to edit 700MB files.
> 
> Okay. Then I probably never scrolled that far when I tested :)

If you never scrolled past 256MB, then you indeed wouldn't notice.

> I guess
> the head + tail trick would work though, I suspect those tools does
> not have this limitation.

Yes, that should work.

Re: Reading huge files

[Giorgos Keramidas]

On Wed, 10 Jan 2007 22:27:57 +0200, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Mathias Dahl <mathias.dahl@gmail.com>
>> Date: Wed, 10 Jan 2007 08:39:16 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > If this is a 32-bit machine, then the Lisp integer overflows at
>> > 256MB.  If that's a 64-bit machine, then you don't need any tricks
>> > to edit 700MB files.
>> 
>> Okay. Then I probably never scrolled that far when I tested :)
>
> If you never scrolled past 256MB, then you indeed wouldn't notice.
>
>> I guess
>> the head + tail trick would work though, I suspect those tools does
>> not have this limitation.
>
> Yes, that should work.

FWIW, I've also uses split(1) with great success, when I had to
process *huge* log files and tcpdump output, ranging from 500 MB
to almost 1 GB of text.

Handling multi-hundred-megabyte files works fine, for instance,
with something like:

    freebsd % split -b 50m huge-file

or with:

    linux % split -b 50000000 huge-file

Re: Reading huge files

[Stefan Monnier]

> ;;; vlf.el --- View Large Files

I like that.

> ;; (defun vlf-extract-part-of-file (file from to)
> ;;   "Returns bytes in FILE from FROM to TO."
> ;;   (let ((size (vlf-file-size file)))
> ;;     (if (or (> from size)
> ;;             (> to size))
> ;;         (error "From or to is larger that the file size"))
> ;;     (with-temp-buffer
> ;;       (shell-command
> ;;        (format "head --bytes %d %s | tail --bytes %d"
> ;; 	       to file (+ (- to from) 1)) t)
> ;;       (buffer-substring (point-min) (point-max)))))

You'll indeed need something like that.  I'd recommend to use `dd' rather
than `split' or `head&tail'.  But note that the problem is bigger than you
think: the above will need to work with floating-point arguments (since
fixnums suffer from the 2^28 limit) and the %d format also suffers from the
2^28 limit (or 2^31 in Emacs-CVS).  That's another good reason to use `dd'
since you can use a 1048576 block size and thus divide your large floats by
1048576 before turning them into fixnums.

> (defvar vlf-mode-map (make-sparse-keymap)
>   "Keymap for `vlf-mode'.")

> (defun vlf-define-keymap ()
>   "Define keymap for `vlf-mode'."
>   (define-key vlf-mode-map [next] 'vlf-next)
>   (define-key vlf-mode-map [prior] 'vlf-prev)
>   (define-key vlf-mode-map "q" 'vlf-quit))

Please just use

  (defvar vlf-mode-map
    (let ((map (make-sparse-keymap)))
      (define-key map [next] 'vlf-next)
      (define-key map [prior] 'vlf-prev)
      (define-key map "q" 'vlf-quit)
      map)
    "Keymap for `vlf-mode'.")
 
Also, wrt features, it'd probably be good to "slide more smoothly":
e.g. always keep 2 or 3 blocks in the buffer at the same time so you can
comfortably look at the text at the boundaries between blocks.

Next step: use window-scroll-functions or jit-lock to detect when reaching
one of the ends so that we can automatically load in the next
consecutive block.


        Stefan

Re: Reading huge files

[Mathias Dahl]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> You'll indeed need something like that.  I'd recommend to use `dd'
> rather than `split' or `head&tail'.

Thanks for the hint, I will take a look at that.

> But note that the problem is bigger than you think: the above will
> need to work with floating-point arguments (since fixnums suffer
> from the 2^28 limit) and the %d format also suffers from the 2^28
> limit (or 2^31 in Emacs-CVS).

After I read Eli's comment about the integer problem I added back the
head-tail version into the code (configurable) and then I started to
notice problemd with integers in various places, so I started to look
into using float, which solved some problems (I am using %.0f instead
of %d now in the call to `format', for example). There still seems to
be some problems though.

>  That's another good reason to use
> `dd' since you can use a 1048576 block size and thus divide your
> large floats by 1048576 before turning them into fixnums.

Umm, okay... (I did not understand half of that, but I probably will
after looking into it.. :)

> Please just use
>
>   (defvar vlf-mode-map
>     (let ((map (make-sparse-keymap)))
>       (define-key map [next] 'vlf-next)
>       (define-key map [prior] 'vlf-prev)
>       (define-key map "q" 'vlf-quit)
>       map)
>     "Keymap for `vlf-mode'.")

Ah, of course! :) The reason I do that is that when testing and adding
key bindings, I cannot simply reevaluate that defvar (because it won't
"bite"), so I use a function to set the map instead and then I can
reevaluate a call to this function.
  
> Also, wrt features, it'd probably be good to "slide more smoothly":
> e.g. always keep 2 or 3 blocks in the buffer at the same time so you
> can comfortably look at the text at the boundaries between blocks.

> Next step: use window-scroll-functions or jit-lock to detect when
> reaching one of the ends so that we can automatically load in the
> next consecutive block.

Aaargh, all these feature-requests make me crazy! :)

Re: Reading huge files

[Markus Triska]

Mathias Dahl <brakjoller@gmail.com> wrote:

> Ah, of course! :) The reason I do that is that when testing and adding
> key bindings, I cannot simply reevaluate that defvar (because it won't
> "bite"),

Use C-M-x for eval-defun:

  If the current defun is actually a call to `defvar' or `defcustom',
  evaluating it this way resets the variable using its initial value
  expression even if the variable already has some other value.

All the best,
Markus Triska

Re: Reading huge files

[Mathias Dahl]

Markus Triska <markus.triska@gmx.at> writes:

> Use C-M-x for eval-defun:
>
>   If the current defun is actually a call to `defvar' or
>   `defcustom', evaluating it this way resets the variable using its
>   initial value expression even if the variable already has some
>   other value.

Aha! I always use C-x C-e apart from when I am instrumenting a
function for edebug, when I use C-u C-M-x. Cool!

Re: Reading huge files

[Andreas Roehler]

Kevin Rodgers wrote:

> Leonid Grinberg wrote:
>> I need to read a file, which contains approximately 1.2GB of
>> data. Obviously, this is well above the maximum buffer limit.
>> Is there any way for me to read it without
>> 
>> 1) breaking it up into smaller files and
>> 2) using a different editor
> 
> Not that I'm aware of.  You could cook up something like:
> 
> (defun view-large-file-contents (file beg end)
>    "View FILE contents from bytes BEG through END, in View
>    mode." (interactive "fView file: \nnFrom byte: \nnTo byte:
>    ") (switch-to-buffer
>     (generate-new-buffer (format "%s[%d,%d]"
> (file-name-nondirectory file) beg end)))
>    (insert-file-contents file nil beg end)
>    (view-mode 1))
> 
>> I am using GNU Emacs 21.4.1 on Ubuntu Edgy Eft on a Dell
>> Latitude C600 with 256MB of memory.
> 

Think it should be possible to have an ed-mode.

Until then, opening a file into `ed' from an emacs-shell
seems much more convenient than calling it from bash or
so.

__
Andreas Roehler

Re: Reading huge files

[Andreas Roehler]

Kevin Rodgers wrote:

> Leonid Grinberg wrote:
>> I need to read a file, which contains approximately 1.2GB of
>> data. Obviously, this is well above the maximum buffer limit.
>> Is there any way for me to read it without
>> 
>> 1) breaking it up into smaller files and
>> 2) using a different editor
> 
> Not that I'm aware of.  You could cook up something like:
> 
> (defun view-large-file-contents (file beg end)
>    "View FILE contents from bytes BEG through END, in View
>    mode." (interactive "fView file: \nnFrom byte: \nnTo byte:
>    ") (switch-to-buffer
>     (generate-new-buffer (format "%s[%d,%d]"
> (file-name-nondirectory file) beg end)))
>    (insert-file-contents file nil beg end)
>    (view-mode 1))
> 
>> I am using GNU Emacs 21.4.1 on Ubuntu Edgy Eft on a Dell
>> Latitude C600 with 256MB of memory.
> 

Think it should be possible to have an ed-mode.

Until then, opening a file into `ed' from an emacs-shell
seems much more convenient than calling it from bash or
so.

__
Andreas Roehler