feature request: view part of file

feature request: view part of file

[Sam Steingold]

Sometimes I want to __VIEW__ a huge file and I get

File ZZZ is large (1645MB), really open? (y or n)  n
abort-if-file-too-large: Aborted

What I want is another option, in addition to y/n:
open a part of the file.
E.g., the first+the last N lines or M characters, whichever comes first.
(something like head+tail).

Is this already available or do I need to put something up myself?

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://openvotingconsortium.org http://www.memritv.org
http://jihadwatch.org http://mideasttruth.com http://dhimmi.com
He who laughs last did not get the joke.

Re: feature request: view part of file

[Stefan Monnier]

> What I want is another option, in addition to y/n:
> open a part of the file.

> Is this already available or do I need to put something up myself?

There's vlf.el.  I wish someone would take this on and develop it
further.  E.g. With the isearch hooks we have it should be possible to
automatically search through the whole file, with "manual paging".


        Stefan

Re: feature request: view part of file

[Sam Steingold]

> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2012-06-13 19:16:59 -0400]:
>
>> What I want is another option, in addition to y/n:
>> open a part of the file.
>
>> Is this already available or do I need to put something up myself?
>
> There's vlf.el.  I wish someone would take this on and develop it
> further.

Is it okay if I clean it up and put into the emacs tree, hooking up into
the "really open this huge file" dialog?

Do we have the papers for Mathias Dahl, the vlf author?

Also, does anyone know what xemacs uses for viewing huge files?

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://www.memritv.org http://truepeace.org
http://ffii.org http://pmw.org.il http://mideasttruth.com http://camera.org
char*a="char*a=%c%s%c;main(){printf(a,34,a,34);}";main(){printf(a,34,a,34);}

Re: feature request: view part of file

[Mathias Dahl]

[-- Attachment #1.1: Type: text/plain, Size: 1030 bytes --]

>
> > There's vlf.el.  I wish someone would take this on and develop it
> > further.
>
> Is it okay if I clean it up and put into the emacs tree, hooking up into
> the "really open this huge file" dialog?
>

Be my guest, do what you like with it. As you can read in the comment
section of the file I created vlf.el because I got kind of tired about
hearing that same question over and over again. I had no real *personal*
itch. Hence I got tired of it and kind of abandoned it after some
experiments.


> Do we have the papers for Mathias Dahl, the vlf author?
>

There should be papers for me from the time when Tumme/image-dired was
included in Emacs.

Attached you can find my latest local version of vlf.el (working state
unknown). The code should be quite clean. I get only one warning when byte
compiling and only two warnings with checkdoc.

It would feel nice if something "real" finally came out of it. The basic
mechanisms is quite simple. What Stefan mentions about hooking into isearch
sounds wicked cool :)

/Mathias

[-- Attachment #1.2: Type: text/html, Size: 1513 bytes --]

[-- Attachment #2: vlf.el --]
[-- Type: application/octet-stream, Size: 6715 bytes --]

;;; vlf.el --- View Large Files

;; Copyright (C) 2006  Mathias Dahl

;; Version: 0.1.2
;; Keywords: files, helpers, utilities
;; Author: Mathias Dahl <mathias.rem0veth1s.dahl@gmail.com>
;; Maintainer: Mathias Dahl
;; URL: http://www.emacswiki.org/cgi-bin/wiki/VLF

;; This file is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 2, or (at your option)
;; any later version.

;; This file is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to
;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
;; Boston, MA 02111-1307, USA.

;;; Commentary:
;; 
;; After reading the Nth post on Gnu Emacs Help about Viewing Large
;; Files in Emacs, it itched so much that I decided to make a try.  It
;; helped quite a lot when Kevin Rodgers posted a snippet on how to
;; use `insert-file-contents' to extract part of a file.  At first I
;; made a try using head and tail and that worked too, but using
;; internal Emacs commands is nicer.  Here is the code to extract data
;; using head and tail in case someone wanna try that out in the
;; future:

;; (defun vlf-extract-part-of-file (file from to)
;;   "Returns bytes in FILE from FROM to TO."
;;   (let ((size (vlf-file-size file)))
;;     (if (or (> from size)
;;             (> to size))
;;         (error "From or to is larger that the file size"))
;;     (with-temp-buffer
;;       (shell-command
;;        (format "head --bytes %d %s | tail --bytes %d"
;; 	       to file (+ (- to from) 1)) t)
;;       (buffer-substring (point-min) (point-max)))))

;;; History:
;;
;; - Wed Jan 10 00:13:45 2007
;;
;;    First version created and released into the wild.
;;
;; - Wed Jan 10 18:58:47 2007
;;
;;    0.1.2
;;
;;    Added option to use external tools (head and tail) for
;;    extracting the data from the file.
;;
;;    Refactored buffer name format code into a new function.
;;
;;    Started to fiddle with float/integer conversions.
;;

;;; Bugs
;;
;; Probably some. Feel free to fix them :)

;;; Code:

(defgroup vlf nil
  "Browse large files in Emacs"
  :prefix "vlf-"
  :group 'files)

(defcustom vlf-batch-size 1000
  "Defines how large each batch of file data is."
  :type 'integer
  :group 'vlf)

(defcustom vlf-external-extraction nil
  "How to extract the data from a file.
`nil' means to use internal extraction, using
`insert-file-contents'. `t' means to use external `head' and
`tail' tools."
  :type 'boolean
  :group 'vlf)

(defvar vlf-current-start-pos 1
  "Keeps track of file position.")

(defvar vlf-current-batch-size nil
  "Keeps track of current batch size.")

(defvar vlf-current-file nil
  "File that is currently viewed.")

(defvar vlf-current-file-size 0
  "Size of current file.")

(defvar vlf-mode-map (make-sparse-keymap)
  "Keymap for `vlf-mode'.")

(defun vlf-define-keymap ()
  "Define keymap for `vlf-mode'."
  (define-key vlf-mode-map [next] 'vlf-next)
  (define-key vlf-mode-map [prior] 'vlf-prev)
  (define-key vlf-mode-map "q" 'vlf-quit))

(define-derived-mode vlf-mode
  fundamental-mode "vlf-mode"
  "Mode to browse large files in.
See `vlf' for details."
  (vlf-define-keymap)
  (toggle-read-only 1)
  (message "vlf-mode enabled"))

(defun vlf-file-size (file)
  "Get size of FILE."
  (nth 7 (file-attributes file)))

(defun vlf-quit ()
  "Quit vlf."
  (interactive)
  (kill-buffer (current-buffer)))

(defun vlf-extract-with-head-and-tail (file from to)
  "Returns bytes in FILE from FROM to TO."
  (let ((size (vlf-file-size file)))
    (if (or (> from size)
            (> to size))
        (error "From or to is larger that the file size"))
    (with-temp-buffer
      (shell-command
       (format "head --bytes %.0f \"%s\" | tail --bytes %.0f"
	       (float to) (expand-file-name file)
               (float (+ (- to from) 1))) t)
      (buffer-substring (point-min) (point-max)))))

(defun vlf-insert-batch ()
  "Insert current batch of data."
  (let* ((beg (1- vlf-current-start-pos))
        (end (+ beg vlf-current-batch-size)))
    (if vlf-external-extraction
        (insert
         (vlf-extract-with-head-and-tail
          vlf-current-file (1+ beg) end))
      (insert-file-contents
       vlf-current-file nil
       (floor beg) (floor end)))))

(defun vlf-format-buffer-name ()
  "Return format for vlf buffer name."
  (format "%s[%.0f,%.0f(%.0f)]"
          (file-name-nondirectory vlf-current-file)
          vlf-current-start-pos
          (1- (+ vlf-current-start-pos
                 vlf-current-batch-size))
          vlf-current-file-size))

(defun vlf-next ()
  "Display the next batch of file data."
  (interactive)
  (let ((inhibit-read-only t)
        left next-start-pos
        (size (vlf-file-size vlf-current-file)))
    (setq next-start-pos (float (+ vlf-current-start-pos
                                   vlf-batch-size)))
    (if (> next-start-pos size)
        (message "End of file")
      (setq vlf-current-batch-size
            vlf-batch-size
            vlf-current-start-pos next-start-pos
            left (1+ (- size vlf-current-start-pos)))     
      (if (< left vlf-current-batch-size)
          (setq vlf-current-batch-size left))
      (erase-buffer)
      (vlf-insert-batch)
      (rename-buffer
       (vlf-format-buffer-name)))))

(defun vlf-prev ()
  "Display the previous batch of file data."
  (interactive)
  (if (= 1 vlf-current-start-pos)
      (message "At beginning of file")
    (let ((inhibit-read-only t))
      (erase-buffer)
      (setq vlf-current-start-pos (- vlf-current-start-pos
                                     vlf-batch-size)
            vlf-current-batch-size vlf-batch-size)
      (vlf-insert-batch)
      (rename-buffer
       (vlf-format-buffer-name)))))

(defun vlf (file)
  "View a large file in Emacs FILE is the file to open.
Batches of the file data from FILE will be displayed in a
read-only buffer.  You can customize the amount of bytes to
display by customizing `vlf-batch-size'."
  (interactive "fFile to open: ")
  (setq vlf-current-file file
        vlf-current-start-pos 1
        vlf-current-file-size (vlf-file-size file)
        vlf-current-batch-size
        (1- (+ vlf-current-start-pos
               vlf-batch-size)))
  (switch-to-buffer
   (generate-new-buffer (vlf-format-buffer-name)))
  (erase-buffer)
  (vlf-insert-batch)
  (vlf-mode))

(provide 'vlf)

;;; vlf.el ends here

Re: feature request: view part of file

[Stephen J. Turnbull]

Sam Steingold writes:

 > Also, does anyone know what xemacs uses for viewing huge files?

Big buffers.  XEmacs topped out at 1GB buffers (on 32-bit platforms)
in 1998 or so, and since then there's been no serious demand for an
increase in supported file size -- we never even bothered with large
file support on 32-bit platforms.  (There was pushback against
suggestions that the 31-bit type be characters to directly support the
ISO 10646 character model, so some users did care about files bigger
than 512MB.)

I imagine some people have personal hacks, but as far as I know there
is no huge file viewer in our distribution.

Re: feature request: view part of file

[Paul Eggert]

Following up to the thread at
<http://lists.gnu.org/archive/html/emacs-devel/2012-06/msg00225.html>,
on 06/14/2012 10:32 AM, Stephen J. Turnbull wrote:

> some users did care about files bigger than 512MB

Sam Steingold was talking about the large-file-warning-threshold limit,
which is soft and which defaults to 10 MB, whereas GNU Emacs's hard
512 MiB limit (on 32-bit hosts) is imposed by how fixnums are implemented.

We could increase GNU Emacs's hard limit from 512 MiB to 2 GiB
by making --with-wide-int the default.  This is Emacs bug 8794.
Stefan asked in <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8794#52>
for measurements about the performance consequences of this change.
I haven't gotten around to that, but will try to bump the priority.

I'll CC: this to 8794@debbugs.gnu.org and ask for replies to
be sent there too, as this topic is a somewhat-separate issue.

Re: feature request: view part of file

[Sam Steingold]

> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2012-06-14 14:58:35 -0400]:
>
> BTW, we could easily extend insert-file-contents to accept floats for
> BEG and END, which should make it possible to have vlf work without
> any external tool.

please do!

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://thereligionofpeace.com
http://mideasttruth.com http://ffii.org http://americancensorship.org
There is an exception to every rule, including this one.

Re: feature request: view part of file

[Sam Steingold]

> * Sam Steingold <fqf@tah.bet> [2012-06-13 16:52:35 -0400]:
>
> Sometimes I want to __VIEW__ a huge file and I get
>
> File ZZZ is large (1645MB), really open? (y or n)  n
> abort-if-file-too-large: Aborted
>
> What I want is another option, in addition to y/n:
> open a part of the file.

I put vlf.el by Mathias into elpa.
I rewrote it almost completely (bzr still has the original version at
revno 233).

it works for me.
I wish I knew how to hook it into files.el, but I don't...

vlf.el has 2 issues:

1. insert-file-contents hangs emacs (see `vlf-next-batch'):

    ;; replacing `erase-buffer' with replace arg to `insert-file-contents'
    ;; hangs emacs
    (unless append (erase-buffer))
    (insert-file-contents buffer-file-name nil vlf-end-pos end)

2. I have this:

    (define-key map [C-+] 'vlf-change-batch-size)

When I do C-+ (by pressing Ctrl, Shift, and the "+=" key to the left of
backspace), I get "C-+ is undefined".
when I do M-x vlf-change-batch-size RET, I get
"You can run the command `vlf-change-batch-size' with <C-+>"
The "+" in both messages is the same character (43, #o53, #x2b).


-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://camera.org
http://openvotingconsortium.org http://honestreporting.com http://memri.org
If you will not bring your husband coffee in bed, his day may start with a beer.

Re: feature request: view part of file

[Štěpán Němec]

On Thu, 14 Jun 2012 23:29:58 +0200
Sam Steingold wrote:

> 2. I have this:
>
>     (define-key map [C-+] 'vlf-change-batch-size)
>
> When I do C-+ (by pressing Ctrl, Shift, and the "+=" key to the left of
> backspace), I get "C-+ is undefined".

(kbd "C-+") ; => [67108907]

[?\C-+] is the way to put it more readably in vector notation (i.e., the
issue here is that [C-+] tries to bind a symbol `C-+' ("<C-+>" in `kbd'
speak), whereas you want the character, hence the ?\ prefix; it's
documented somewhere in the elisp manual I think).

-- 
Štěpán

Re: feature request: view part of file

[Samuel Bronson]

On Jun 18, 2012, at 4:34 PM, Štěpán Němec wrote:

> (kbd "C-+") ; => [67108907]
>
> [?\C-+] is the way to put it more readably in vector notation (i.e.,  
> the
> issue here is that [C-+] tries to bind a symbol `C-+' ("<C-+>" in  
> `kbd'
> speak), whereas you want the character, hence the ?\ prefix; it's
> documented somewhere in the elisp manual I think).

You know, this would be easier with a character type...

Re: feature request: view part of file

[Stephen J. Turnbull]

Samuel Bronson writes:
 > On Jun 18, 2012, at 4:34 PM, Štěpán Němec wrote:
 > 
 > > (kbd "C-+") ; => [67108907]
 > >
 > > [?\C-+] is the way to put it more readably in vector notation (i.e.,  
 > > the
 > > issue here is that [C-+] tries to bind a symbol `C-+' ("<C-+>" in  
 > > `kbd'
 > > speak), whereas you want the character, hence the ?\ prefix; it's
 > > documented somewhere in the elisp manual I think).
 > 
 > You know, this would be easier with a character type...

Not really.  XEmacs *has* a character type, but that doesn't help us
with this kind of thing, because this is about syntax for describing
keystrokes in Lisp, not about managing characters.  And it's not just
backward compatibility.

Our preferred approach is to use the array of chords of keysyms.  (Eg
the above would be expressed `[(control +)]'.

Re: feature request: view part of file

[Andrey Kotlarski]

Sam Steingold <sds@gnu.org> writes:

> I put vlf.el by Mathias into elpa.

Hello,

thanks for putting this in gnu elpa, it's quite useful.

> I rewrote it almost completely (bzr still has the original version at
> revno 233).
>
> it works for me.
> I wish I knew how to hook it into files.el, but I don't...
>
> vlf.el has 2 issues:
>
> 1. insert-file-contents hangs emacs (see `vlf-next-batch'):
>
>     ;; replacing `erase-buffer' with replace arg to `insert-file-contents'
>     ;; hangs emacs
>     (unless append (erase-buffer))
>     (insert-file-contents buffer-file-name nil vlf-end-pos end)

I guess you've been using rather large file.  My attempts showed that
non nil REPLACE argument of insert-file-contents doesn't honour BEG END
and always inserts the whole file.

Anyway, I've made a few additions that I think are worth sharing.

* Option to start viewing from the end of file
* Ability to jump/insert given number of batches at once
* Ability to view newly added content if the file has grown meanwhile

Here is a changelog entry if necessary and a diff:

2012-08-04  Andrey Kotlarski  <m00naticus@gmail.com>

	* vlf.el (vlf-mode-map): Add decrease batch key binding.
	(vlf-change-batch-size): Fix doc string.
	(vlf-format-buffer-name): Fix indentation.
	(vlf-next-batch): Optionally jump/insert several batches based on
	prefix argument. If requested batch is past the end of file, get
	current file length and view as much as possible.
	(vlf-prev-batch): Optionally jump/insert several batches based on
	prefix argument.
	(vlf): Start viewing from end of file if prefix argument is
	given. Add autoload cookie.


diff -u vlf.el\~ vlf.el
--- vlf.el~	2012-07-23 17:34:54.000000000 +0300
+++ vlf.el	2012-07-27 23:44:43.696003339 +0300
@@ -2,7 +2,7 @@
 
 ;; Copyright (C) 2006, 2012  Free Software Foundation, Inc.
 
-;; Version: 0.2
+;; Version: 0.3
 ;; Keywords: large files, utilities
 ;; Authors: 2006 Mathias Dahl <mathias.dahl@gmail.com>
 ;;          2012 Sam Steingold <sds@gnu.org>
@@ -57,6 +57,10 @@
     (define-key map [M-next] 'vlf-next-batch)
     (define-key map [M-prior] 'vlf-prev-batch)
     (define-key map (kbd "C-+") 'vlf-change-batch-size)
+    (define-key map (kbd "C--")
+      (lambda () "Decrease vlf batch size by factor of 2."
+	(interactive)
+	(vlf-change-batch-size t)))
     map)
   "Keymap for `vlf-mode'.")
 
@@ -71,22 +75,22 @@
 (defun vlf-change-batch-size (decrease)
   "Change the buffer-local value of `vlf-batch-size'.
 Normally, the value is doubled;
-with the prefix argument it is halved."
+with the prefix argument DECREASE it is halved."
   (interactive "P")
   (or (assq 'vlf-batch-size (buffer-local-variables))
       (error "%s is not local in this buffer" 'vlf-batch-size))
   (setq vlf-batch-size
-        (if decrease
-            (/ vlf-batch-size 2)
-            (* vlf-batch-size 2)))
+	(if decrease
+	    (/ vlf-batch-size 2)
+	  (* vlf-batch-size 2)))
   (vlf-update-buffer-name))
 
 (defun vlf-format-buffer-name ()
   "Return format for vlf buffer name."
   (format "%s(%s)[%d,%d](%d)"
-          (file-name-nondirectory buffer-file-name)
-          (file-size-human-readable vlf-file-size)
-          vlf-start-pos vlf-end-pos vlf-batch-size))
+	  (file-name-nondirectory buffer-file-name)
+	  (file-size-human-readable vlf-file-size)
+	  vlf-start-pos vlf-end-pos vlf-batch-size))
 
 (defun vlf-update-buffer-name ()
   "Update the current buffer name."
@@ -94,56 +98,80 @@
 
 (defun vlf-next-batch (append)
   "Display the next batch of file data.
-Append to the existing buffer when the prefix argument is supplied."
-  (interactive "P")
-  (when (= vlf-end-pos vlf-file-size)
-    (error "Already at EOF"))
-  (let ((inhibit-read-only t)
-        (end (min vlf-file-size (+ vlf-end-pos vlf-batch-size))))
-    (goto-char (point-max))
-    ;; replacing `erase-buffer' with replace arg to `insert-file-contents'
-    ;; hangs emacs
-    (unless append (erase-buffer))
-    (insert-file-contents buffer-file-name nil vlf-end-pos end)
-    (unless append
-      (setq vlf-start-pos vlf-end-pos))
-    (setq vlf-end-pos end)
-    (set-buffer-modified-p nil)
-    (vlf-update-buffer-name)))
+When prefix argument is supplied and positive
+ jump over APPEND number of batches.
+When prefix argument is negative
+ append next APPEND number of batches to the existing buffer."
+  (interactive "p")
+  (let ((end (+ vlf-end-pos (* vlf-batch-size
+			       (abs append)))))
+    (when (< vlf-file-size end)		; re-check file size
+      (setq vlf-file-size (nth 7 (file-attributes buffer-file-name)))
+      (cond ((= vlf-end-pos vlf-file-size)
+	     (error "Already at EOF"))
+	    ((< vlf-file-size end)
+	     (setq end vlf-file-size))))
+    (let ((inhibit-read-only t)
+	  (do-append (< append 0)))
+      (if do-append
+	  (goto-char (point-max))
+	(setq vlf-start-pos (- end vlf-batch-size))
+	(erase-buffer))
+      (insert-file-contents buffer-file-name nil
+			    (if do-append
+				vlf-end-pos
+			      vlf-start-pos)
+			    end))
+    (setq vlf-end-pos end))
+  (set-buffer-modified-p nil)
+  (vlf-update-buffer-name))
 
 (defun vlf-prev-batch (prepend)
   "Display the previous batch of file data.
-Prepend to the existing buffer when the prefix argument is supplied."
-  (interactive "P")
-  (when (= vlf-start-pos 0)
-    (error "Already at BOF"))
+When prefix argument is supplied and positive
+ jump over PREPEND number of batches.
+When prefix argument is negative
+ append previous PREPEND number of batches to the existing buffer."
+  (interactive "p")
+  (if (zerop vlf-start-pos)
+      (error "Already at BOF"))
   (let ((inhibit-read-only t)
-        (start (max 0 (- vlf-start-pos vlf-batch-size))))
-    (goto-char (point-min))
-    (unless prepend (erase-buffer))
-    (insert-file-contents buffer-file-name nil start vlf-start-pos)
-    (unless prepend
-      (setq vlf-end-pos vlf-start-pos))
-    (setq vlf-start-pos start)
-    (set-buffer-modified-p nil)
-    (vlf-update-buffer-name)))
+	(start (max 0 (- vlf-start-pos (* vlf-batch-size
+					  (abs prepend)))))
+	(do-prepend (< prepend 0)))
+    (if do-prepend
+	(goto-char (point-min))
+      (setq vlf-end-pos (+ start vlf-batch-size))
+      (erase-buffer))
+    (insert-file-contents buffer-file-name nil start
+			  (if do-prepend
+			      vlf-start-pos
+			    vlf-end-pos))
+    (setq vlf-start-pos start))
+  (set-buffer-modified-p nil)
+  (vlf-update-buffer-name))
 
-(defun vlf (file)
+;;;###autoload
+(defun vlf (from-end file)
   "View a Large File in Emacs.
+With FROM-END prefix, view from the back.
 FILE is the file to open.
 Batches of the file data from FILE will be displayed in a
  read-only buffer.
 You can customize the number of bytes to
  display by customizing `vlf-batch-size'."
-  (interactive "fFile to open: ")
+  (interactive "P\nfFile to open: ")
   (with-current-buffer (generate-new-buffer "*vlf*")
     (setq buffer-file-name file
-          vlf-start-pos 0
-          vlf-end-pos vlf-batch-size
-          vlf-file-size (nth 7 (file-attributes file)))
+	  vlf-file-size (nth 7 (file-attributes file)))
+    (if from-end
+	(setq vlf-start-pos (max 0 (- vlf-file-size vlf-batch-size))
+	      vlf-end-pos vlf-file-size)
+      (setq vlf-start-pos 0
+	    vlf-end-pos (min vlf-batch-size vlf-file-size)))
     (vlf-update-buffer-name)
     (insert-file-contents buffer-file-name nil
-                          vlf-start-pos vlf-end-pos nil)
+			  vlf-start-pos vlf-end-pos)
     (vlf-mode)
     (display-buffer (current-buffer))))

Re: feature request: view part of file

[Vitalie Spinu]

  >> Sam Steingold <sds@gnu.org>
  >> on Thu, 14 Jun 2012 17:29:58 -0400 wrote:


  > I put vlf.el by Mathias into elpa. I rewrote it almost completely
  > (bzr still has the original version at revno 233). it works for me.

It doesn't work for me. vlf-next-batch pops:
   
   Debugger entered--Lisp error: (wrong-type-argument file-offset 2048.0)
     insert-file-contents("/home/vitoshka/test/foo.csv" nil 1024 2048.0)
     vlf-next-batch(nil)
     call-interactively(vlf-next-batch nil nil)


Would be nice to actually write the batch into a local file and put the
buffer into the appropriate mode. So if the file is csv I view it in
csv-mode. Then get a special command to write the contents into the
original file.

The dired integration could be improved by making it pop into vlf mode
whenever the file exceeds some user specified size.

    Vitalie


  > I wish I knew how to hook it into files.el, but I don't...

  > vlf.el has 2 issues:

  > 1. insert-file-contents hangs emacs (see `vlf-next-batch'):

  >     ;; replacing `erase-buffer' with replace arg to `insert-file-contents'
  >     ;; hangs emacs
  >     (unless append (erase-buffer))
  >     (insert-file-contents buffer-file-name nil vlf-end-pos end)

  > 2. I have this:

  >     (define-key map [C-+] 'vlf-change-batch-size)

  > When I do C-+ (by pressing Ctrl, Shift, and the "+=" key to the left of
  > backspace), I get "C-+ is undefined".
  > when I do M-x vlf-change-batch-size RET, I get
  > "You can run the command `vlf-change-batch-size' with <C-+>"
  > The "+" in both messages is the same character (43, #o53, #x2b).

Re: feature request: view part of file

[Vitalie Spinu]

  >> Vitalie Spinu <spinuvit@gmail.com>
  >> on Sat, 19 Jan 2013 00:30:38 +0100 wrote:

  >> Sam Steingold <sds@gnu.org>
  >> on Thu, 14 Jun 2012 17:29:58 -0400 wrote:

  >> I put vlf.el by Mathias into elpa. I rewrote it almost completely
  >> (bzr still has the original version at revno 233). it works for me.

  VS> It doesn't work for me. vlf-next-batch pops:

  VS>    Debugger entered--Lisp error: (wrong-type-argument file-offset 2048.0)
  VS>      insert-file-contents("/home/vitoshka/test/foo.csv" nil 1024 2048.0)
  VS>      vlf-next-batch(nil)
  VS>      call-interactively(vlf-next-batch nil nil)

So it is really a bug in emacs, insert-file-contents accepts integer
values only, but for big files the size and positions are floats. In my
case 802200060.0. 

It happens because vlf-file-size if float:

╭──────── #143 ─ /home/vitoshka/Dropbox/ELPA/vlf-0.2/vlf.el ──
│           vlf-file-size ((nth 7 (file-attributes file)))
╰──────── #143 ─

and 

╭──────── #102 ─ /home/vitoshka/Dropbox/ELPA/vlf-0.2/vlf.el ──
│         (end (min vlf-file-size (+ vlf-end-pos vlf-batch-size))))
╰──────── #102 ─

returns float which is passed down to insert-file-contents.


So I am stuck here, I just cannot technically access the last piece of
my (700MB) file.

    Thanks, 
    Vitalie


  VS> Would be nice to actually write the batch into a local file and put the
  VS> buffer into the appropriate mode. So if the file is csv I view it in
  VS> csv-mode. Then get a special command to write the contents into the
  VS> original file.

  VS> The dired integration could be improved by making it pop into vlf mode
  VS> whenever the file exceeds some user specified size.

  VS>     Vitalie

  >> I wish I knew how to hook it into files.el, but I don't...

  >> vlf.el has 2 issues:

  >> 1. insert-file-contents hangs emacs (see `vlf-next-batch'):

  >> ;; replacing `erase-buffer' with replace arg to `insert-file-contents'
  >> ;; hangs emacs
  >> (unless append (erase-buffer))
  >> (insert-file-contents buffer-file-name nil vlf-end-pos end)

  >> 2. I have this:

  >> (define-key map [C-+] 'vlf-change-batch-size)

  >> When I do C-+ (by pressing Ctrl, Shift, and the "+=" key to the left of
  >> backspace), I get "C-+ is undefined".
  >> when I do M-x vlf-change-batch-size RET, I get
  >> "You can run the command `vlf-change-batch-size' with <C-+>"
  >> The "+" in both messages is the same character (43, #o53, #x2b).

Re: feature request: view part of file

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Date: Sat, 19 Jan 2013 00:52:50 +0100
> Cc: emacs-devel@gnu.org
> 
> So it is really a bug in emacs, insert-file-contents accepts integer
> values only, but for big files the size and positions are floats. In my
> case 802200060.0. 

It's not a bug, it's a fundamental limitation of how Emacs accesses
buffers.  Any valid buffer position _must_ fit into the width of an
Emacs integer.  The only way to lift that limitation is to compile
Emacs with a wider EMACS_INT type.

Re: feature request: view part of file

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On 01/18/2013 10:54 PM, Eli Zaretskii wrote:
> It's not a bug, it's a fundamental limitation of how Emacs accesses
> buffers.  Any valid buffer position _must_ fit into the width of an
> Emacs integer.

Sure, but Vitalie's not asking for large buffer positions,
just for large file offsets.  Emac's a bit squirrelly in this
area, as it represents large file offsets by using floating
point numbers, but if it's going to be squirrelly it should
be *consistently* squirrelly, so I installed the attached patch
into the trunk as bzr 111554.

If 64-bit EMACS_INT were the default Vitalie wouldn't have had this
problem.  We have a simple fix (change configure so that --with-wide-int
is the default, see Bug#8794) but this most likely would slow Emacs a bit.
Stefan was skeptical and asked for benchmarks, but maybe
I should boost the priority of this.  Or, more drastically,
maybe we should bite the bullet and add bignums (Bug#8611);
it can't be *that* hard.

[-- Attachment #2: float-offset.txt --]
[-- Type: text/plain, Size: 5678 bytes --]

=== modified file 'doc/lispref/ChangeLog'
--- doc/lispref/ChangeLog	2013-01-10 03:43:02 +0000
+++ doc/lispref/ChangeLog	2013-01-19 09:57:59 +0000
@@ -1,3 +1,9 @@
+2013-01-19  Paul Eggert  <eggert@cs.ucla.edu>
+
+	Allow floating-point file offsets.
+	* files.texi (Reading from Files, Writing to Files):
+	Say that file offsets can be numbers, not just integers.
+
 2013-01-09  Glenn Morris  <rgm@gnu.org>
 
 	* commands.texi (Interactive Codes):

=== modified file 'doc/lispref/files.texi'
--- doc/lispref/files.texi	2013-01-04 02:42:08 +0000
+++ doc/lispref/files.texi	2013-01-19 09:57:59 +0000
@@ -533,9 +533,9 @@
 file name and its last save file modtime.  This feature is used by
 @code{find-file-noselect} and you probably should not use it yourself.
 
-If @var{beg} and @var{end} are non-@code{nil}, they should be integers
-specifying the portion of the file to insert.  In this case, @var{visit}
-must be @code{nil}.  For example,
+If @var{beg} and @var{end} are non-@code{nil}, they should be numbers
+that are byte offsets specifying the portion of the file to insert.
+In this case, @var{visit} must be @code{nil}.  For example,
 
 @example
 (insert-file-contents filename nil 0 500)
@@ -605,8 +605,8 @@
 this case.
 
 If @var{append} is non-@code{nil}, then the specified text is appended
-to the existing file contents (if any).  If @var{append} is an
-integer, @code{write-region} seeks to that byte offset from the start
+to the existing file contents (if any).  If @var{append} is a
+number, @code{write-region} seeks to that byte offset from the start
 of the file and writes the data from there.
 
 If @var{mustbenew} is non-@code{nil}, then @code{write-region} asks

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2013-01-19 08:49:17 +0000
+++ src/ChangeLog	2013-01-19 09:57:59 +0000
@@ -1,3 +1,12 @@
+2013-01-19  Paul Eggert  <eggert@cs.ucla.edu>
+
+	Allow floating-point file offsets.
+	Problem reported by Vitalie Spinu in
+	<http://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00411.html>.
+	* fileio.c (emacs_lseek): Remove.
+	(file_offset): New function.
+	(Finsert_file_contents, Fwrite_region): Use it.
+
 2013-01-19  Chong Yidong  <cyd@gnu.org>
 
 	* emacs.c (Fkill_emacs): Set waiting_for_input to 0 to avoid

=== modified file 'src/fileio.c'
--- src/fileio.c	2013-01-19 04:44:34 +0000
+++ src/fileio.c	2013-01-19 09:57:59 +0000
@@ -3443,19 +3443,25 @@
   return Qnil;
 }
 
-/* Reposition FD to OFFSET, based on WHENCE.  This acts like lseek
-   except that it also tests for OFFSET being out of lseek's range.  */
+/* Return the file offset that VAL represents, checking for type
+   errors and overflow.  */
 static off_t
-emacs_lseek (int fd, EMACS_INT offset, int whence)
+file_offset (Lisp_Object val)
 {
-  /* Use "&" rather than "&&" to suppress a bogus GCC warning; see
-     <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43772>.  */
-  if (! ((offset >= TYPE_MINIMUM (off_t)) & (offset <= TYPE_MAXIMUM (off_t))))
+  if (RANGED_INTEGERP (0, val, TYPE_MAXIMUM (off_t)))
+    return XINT (val);
+
+  if (FLOATP (val))
     {
-      errno = EINVAL;
-      return -1;
+      double v = XFLOAT_DATA (val);
+      if (0 <= v
+	  && (sizeof (off_t) < sizeof v
+	      ? v <= TYPE_MAXIMUM (off_t)
+	      : v < TYPE_MAXIMUM (off_t)))
+	return v;
     }
-  return lseek (fd, offset, whence);
+
+  wrong_type_argument (intern ("file-offset"), val);
 }
 
 /* Return a special time value indicating the error number ERRNUM.  */
@@ -3606,20 +3612,12 @@
     }
 
   if (!NILP (beg))
-    {
-      if (! RANGED_INTEGERP (0, beg, TYPE_MAXIMUM (off_t)))
-	wrong_type_argument (intern ("file-offset"), beg);
-      beg_offset = XFASTINT (beg);
-    }
+    beg_offset = file_offset (beg);
   else
     beg_offset = 0;
 
   if (!NILP (end))
-    {
-      if (! RANGED_INTEGERP (0, end, TYPE_MAXIMUM (off_t)))
-	wrong_type_argument (intern ("file-offset"), end);
-      end_offset = XFASTINT (end);
-    }
+    end_offset = file_offset (end);
   else
     {
       if (not_regular)
@@ -4714,7 +4712,7 @@
 instead of any buffer contents; END is ignored.
 
 Optional fourth argument APPEND if non-nil means
-  append to existing file contents (if any).  If it is an integer,
+  append to existing file contents (if any).  If it is a number,
   seek to that offset in the file before writing.
 Optional fifth argument VISIT, if t or a string, means
   set the last-save-file-modtime of buffer to this file's modtime
@@ -4743,6 +4741,7 @@
   (Lisp_Object start, Lisp_Object end, Lisp_Object filename, Lisp_Object append, Lisp_Object visit, Lisp_Object lockname, Lisp_Object mustbenew)
 {
   int desc;
+  off_t offset;
   bool ok;
   int save_errno = 0;
   const char *fn;
@@ -4864,13 +4863,14 @@
   encoded_filename = ENCODE_FILE (filename);
 
   fn = SSDATA (encoded_filename);
+  offset = 0;
   desc = -1;
   if (!NILP (append))
-#ifdef DOS_NT
-    desc = emacs_open (fn, O_WRONLY | O_BINARY, 0);
-#else  /* not DOS_NT */
-    desc = emacs_open (fn, O_WRONLY, 0);
-#endif /* not DOS_NT */
+    {
+      if (NUMBERP (append))
+	offset = file_offset (append);
+      desc = emacs_open (fn, O_WRONLY | O_BINARY, 0);
+    }
 
   if (desc < 0 && (NILP (append) || errno == ENOENT))
 #ifdef DOS_NT
@@ -4897,14 +4897,9 @@
 
   record_unwind_protect (close_file_unwind, make_number (desc));
 
-  if (!NILP (append) && !NILP (Ffile_regular_p (filename)))
+  if (!NILP (append))
     {
-      off_t ret;
-
-      if (NUMBERP (append))
-	ret = emacs_lseek (desc, XINT (append), SEEK_CUR);
-      else
-	ret = lseek (desc, 0, SEEK_END);
+      off_t ret = lseek (desc, offset, NUMBERP (append) ? SEEK_SET : SEEK_END);
       if (ret < 0)
 	{
 #ifdef CLASH_DETECTION

Re: feature request: view part of file

[Eli Zaretskii]

> Date: Sat, 19 Jan 2013 02:18:56 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: sds@gnu.org, Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
> 
> Sure, but Vitalie's not asking for large buffer positions,
> just for large file offsets.  Emac's a bit squirrelly in this
> area, as it represents large file offsets by using floating
> point numbers, but if it's going to be squirrelly it should
> be *consistently* squirrelly, so I installed the attached patch
> into the trunk as bzr 111554.

That patch calls lseek inside write-region on non-regular files as
well, where previously we didn't.  Why is that a good idea?  (It's an
unrelated change anyway.)  This log entry

  1998-06-29  Andrew Innes  <andrewi@harlequin.co.uk>

	  * fileio.c (Ffile_regular_p) [WINDOWSNT]: Bind
	  `w32-get-true-file-attributes' to t while calling stat.
	  (Finsert_file_contents) [WINDOWSNT]: Likewise.
	  (Fwrite_region): Don't try seeking to end of FILENAME for
	  appending if not a regular file.

seems to indicate that this condition was there for a reason.

Re: feature request: view part of file

[Paul Eggert]

On 01/19/2013 02:51 AM, Eli Zaretskii wrote:
> That patch calls lseek inside write-region on non-regular files as
> well, where previously we didn't.  Why is that a good idea?  (It's an
> unrelated change anyway.) 

Yes, it's unrelated; sorry, it was related in an earlier version
of the patch, but I forgot to undo that part.

It's a good idea for a couple of reasons.  First, the old code
mishandled things if the file was renamed by some other process
between the time that Emacs opened it and the time that
file-regular-p invoked stat on it.  Second, the typical GNU style
is that the behavior of a program should not depend on the type of
output device.

I looked at the old change you mentioned, and it appears that
there was a problem on DOS_NT with invoking lseek (fd, 0, SEEK_END) when
FD is not a regular file.  Here's a patch that should avoid this problem:

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2013-01-19 12:29:10 +0000
+++ src/ChangeLog	2013-01-19 12:37:41 +0000
@@ -1,5 +1,16 @@
 2013-01-19  Paul Eggert  <eggert@cs.ucla.edu>
 
+	* fileio.c: Use O_APPEND to append.
+	This corresponds better to the natural interpretation of "append",
+	and avoids the need to open the output file twice, or to invoke
+	lseek when APPEND is neither nil nor a number.
+	This relies on POSIX 1003.1-1988 or later, which is OK nowadays.
+	(Fwrite_region): Simplify.  Use O_APPEND instead of opening the
+	file possibly twice, and lseeking to its end; this avoids the
+	need to lseek on non-regular files.  Do not use O_EXCL and O_TRUNC
+	at the same time: the combination is never needed and apparently
+	it doesn't work with DOS_NT.
+
 	Fix size bug on DOS_NT introduced by CIFS workaround (Bug#13149).
 	* fileio.c (Fwrite_region): Use O_BINARY in checking code, too.
 

=== modified file 'src/fileio.c'
--- src/fileio.c	2013-01-19 12:29:10 +0000
+++ src/fileio.c	2013-01-19 12:37:41 +0000
@@ -4741,7 +4741,9 @@
   (Lisp_Object start, Lisp_Object end, Lisp_Object filename, Lisp_Object append, Lisp_Object visit, Lisp_Object lockname, Lisp_Object mustbenew)
 {
   int desc;
-  off_t offset;
+  int open_flags;
+  int mode;
+  off_t offset IF_LINT (= 0);
   bool ok;
   int save_errno = 0;
   const char *fn;
@@ -4863,26 +4865,19 @@
   encoded_filename = ENCODE_FILE (filename);
 
   fn = SSDATA (encoded_filename);
-  offset = 0;
-  desc = -1;
-  if (!NILP (append))
-    {
-      if (NUMBERP (append))
-	offset = file_offset (append);
-      desc = emacs_open (fn, O_WRONLY | O_BINARY, 0);
-    }
-
-  if (desc < 0 && (NILP (append) || errno == ENOENT))
+  open_flags = O_WRONLY | O_BINARY | O_CREAT;
+  open_flags |= EQ (mustbenew, Qexcl) ? O_EXCL : O_TRUNC;
+  if (NUMBERP (append))
+    offset = file_offset (append);
+  else if (!NILP (append))
+    open_flags |= O_APPEND;
 #ifdef DOS_NT
-  desc = emacs_open (fn,
-		     O_WRONLY | O_CREAT | O_BINARY
-		     | (EQ (mustbenew, Qexcl) ? O_EXCL : O_TRUNC),
-		     S_IREAD | S_IWRITE);
-#else  /* not DOS_NT */
-  desc = emacs_open (fn, O_WRONLY | O_TRUNC | O_CREAT
-		     | (EQ (mustbenew, Qexcl) ? O_EXCL : 0),
-		     auto_saving ? auto_save_mode_bits : 0666);
-#endif /* not DOS_NT */
+  mode = S_IREAD | S_IWRITE;
+#else
+  mode = auto_saving ? auto_save_mode_bits : 0666;
+#endif
+
+  desc = emacs_open (fn, open_flags, mode);
 
   if (desc < 0)
     {
@@ -4897,9 +4892,9 @@
 
   record_unwind_protect (close_file_unwind, make_number (desc));
 
-  if (!NILP (append))
+  if (NUMBERP (append))
     {
-      off_t ret = lseek (desc, offset, NUMBERP (append) ? SEEK_SET : SEEK_END);
+      off_t ret = lseek (desc, offset, SEEK_SET);
       if (ret < 0)
 	{
 #ifdef CLASH_DETECTION

Re: feature request: view part of file

[Eli Zaretskii]

> Date: Sat, 19 Jan 2013 04:47:58 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: sds@gnu.org, spinuvit@gmail.com, emacs-devel@gnu.org
> 
> It's a good idea for a couple of reasons.  First, the old code
> mishandled things if the file was renamed by some other process
> between the time that Emacs opened it and the time that
> file-regular-p invoked stat on it.  Second, the typical GNU style
> is that the behavior of a program should not depend on the type of
> output device.
> 
> I looked at the old change you mentioned, and it appears that
> there was a problem on DOS_NT with invoking lseek (fd, 0, SEEK_END) when
> FD is not a regular file.  Here's a patch that should avoid this problem:

Thanks.  I think this should go in, with one caveat:

> +  open_flags = O_WRONLY | O_BINARY | O_CREAT;
> +  open_flags |= EQ (mustbenew, Qexcl) ? O_EXCL : O_TRUNC;
> +  if (NUMBERP (append))
> +    offset = file_offset (append);
> +  else if (!NILP (append))
> +    open_flags |= O_APPEND;

We cannot use O_TRUNC if the APPEND argument is non-nil, can we?  Or
did I misunderstand something?

Re: feature request: view part of file

[Paul Eggert]

On 01/19/2013 05:47 AM, Eli Zaretskii wrote:
> We cannot use O_TRUNC if the APPEND argument is non-nil, can we?

Right you are; thanks for catching that.  I installed it as
trunk bzr 111559, with the following line instead:

  open_flags |= EQ (mustbenew, Qexcl) ? O_EXCL : !NILP (append) ? 0 : O_TRUNC;