help editing huge files in emacs

help editing huge files in emacs

[Jesús Velazquez]

Hi guys:


How can I edit a selected number of lines in a 4GB file in Emacs
without open the whole file? Is it possible?


Thanks.

Re: help editing huge files in emacs

[Stefan Monnier]

> How can I edit a selected number of lines in a 4GB file in Emacs
> without open the whole file? Is it possible?

`insert-file-contents' takes two integer parameters BEG and END to allow
getting only part of a file into a buffer.  The corresponding `write-region'
takes an APPEND parameter which can be an integer indicating at which
position in the file the text should be written.

With those two elisp functions, you can read&write parts of files, so you
can do what you want.  But it requires some coding on top of that to make it
really usable (probably some kind of special major or minor mode, or maybe
a file-name-handler) and I don't know of any package that does that.


        Stefan

Re: help editing huge files in emacs

[Kevin Rodgers]

Jesús Velazquez wrote:
> How can I edit a selected number of lines in a 4GB file in Emacs
> without open the whole file? Is it possible?

beg=12345
end=67890

sed -n "${beg},${end}p" big_file > excerpt
emacs excerpt
sed "${beg},${end}d;$((beg - 1)r excerpt" big_file > new_big_file

-- 
Kevin Rodgers

Re: help editing huge files in emacs

[Mathias Dahl]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> `insert-file-contents' takes two integer parameters BEG and END to allow
> getting only part of a file into a buffer.  The corresponding `write-region'
> takes an APPEND parameter which can be an integer indicating at which
> position in the file the text should be written.
>
> With those two elisp functions, you can read&write parts of files, so you
> can do what you want.  But it requires some coding on top of that to make it
> really usable (probably some kind of special major or minor mode, or maybe
> a file-name-handler) and I don't know of any package that does that.

Interesting idea! But it would only work in the situations where you
wanted the edited text (the text you inserted into emacs and edited
there) to be written at position X in the original file and could not
be used (easily) to overwrite the data that was originally extracted
with `insert-file-contents'. Right?

The above might be confusing, so I'll give an example. Say I have this
really huge file :)

-- huge file --
line 1
line 2
line 3
line 4
line 5
line 6
-- end --

We extract line 3 to 5 into Emacs:

-- emacs buffer --
line 3
line 4
line 5
-- end --

We might change some stuff and add stuff:

-- emacs buffer --
line 3
line 4 is changed
line 4.1 was added
line 5
-- end --

The result, if I use `write-region' and the `append' with the original
start position (X) as argument, would be:

-- huge file --
line 1
line 2
line 3
line 4 is changed
line 4.1 was added
line 5
line 3
line 4
line 5
line 6
-- end --

It would be cool if it could instead become:

-- huge file --
line 1
line 2
line 3
line 4 is changed
line 4.1 was added
line 5
line 6
-- end --

Possible?

Note: I have no need of doing this but I find it interesting
nevertheless.

/Mathias

Re: help editing huge files in emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 15 Mar 2006 01:09:11 -0500
> 
> > How can I edit a selected number of lines in a 4GB file in Emacs
> > without open the whole file? Is it possible?
> 
> `insert-file-contents' takes two integer parameters BEG and END to allow
> getting only part of a file into a buffer.  The corresponding `write-region'
> takes an APPEND parameter which can be an integer indicating at which
> position in the file the text should be written.

??? But aren't those parameters limited to EMACS_INT max values,
i.e. to the same number of less-than-32 bits that limits the maximum
size of an Emacs buffer?  Or am I missing something?

If I'm right, then editing of a 4GB file using your suggestions is
possible only in a 64-bit build of Emacs.

Re: help editing huge files in emacs

[Stefan Monnier]

>> > How can I edit a selected number of lines in a 4GB file in Emacs
>> > without open the whole file? Is it possible?
>> 
>> `insert-file-contents' takes two integer parameters BEG and END to allow
>> getting only part of a file into a buffer.  The corresponding `write-region'
>> takes an APPEND parameter which can be an integer indicating at which
>> position in the file the text should be written.

> ??? But aren't those parameters limited to EMACS_INT max values,

Yes.

> i.e. to the same number of less-than-32 bits that limits the maximum
> size of an Emacs buffer?  Or am I missing something?

He didn't say why he didn't want to open the whole file.  And he didn't say
that he was using a 32bit system.  In my exprience the 128MB file size limit
(aka 256MB in Emacs-CVS, 512MB with my local hack, and 1GB with XEmacs) is
not the only reason why I wouldn't want to open directly such a large file.
Manipulating large buffers isn't Emacs's forte.

> If I'm right, then editing of a 4GB file using your suggestions is
> possible only in a 64-bit build of Emacs.

Indeed.  Although if there's enough interest it'd be easy to extend
insert-file-contents and write-region to accept floats as well so as to push
this limit a good bit further even on 32bit systems.


        Stefan