Memory-efficient way of opening large files in Emacs read-only

Memory-efficient way of opening large files in Emacs read-only

[Mikhail Pomaznoy]

Greetings!

In my daily workflows I commonly stumble upon a task of peeking into a 
header of a large (sometimes compressed) file. Those files can commonly 
be gigabytes in compressed form so can't fit into memory. I was 
wondering how can this be done in Emacs to leverage its search and 
navigate capabilities?

In a terminal this task is nicely accomplished with `less`, which loads 
file on demand while you scroll (memory-efficiently) and automatically 
detects at least gzip compression. However I am struggling to do 
something similar in Emacs for quite a long time.  I tried VLF (requires 
manually switching chunks loaded, encountered problems with 
compression), view-file (loads everything in memory), running less in 
Emacs terminals (I prefer outside-Emacs terminals) but for various 
reasons all this solutions are suboptimal and I usually end up with 
using `less`.

Is there something in Emacs I am missing? Is something like this makes 
sense to develop (I consider to write something for Emacs) ? I am using 
Emacs 28 and maybe in Emacs 29 there are novelties related to this issue?

Kind regards,

Mikhail

Re: Memory-efficient way of opening large files in Emacs read-only

[Filipp Gunbin]

On 08/12/2022 09:49 +0700, Mikhail Pomaznoy wrote:

> In my daily workflows I commonly stumble upon a task of peeking into a
> header of a large (sometimes compressed) file. Those files can
> commonly be gigabytes in compressed form so can't fit into memory. I
> was wondering how can this be done in Emacs to leverage its search and
> navigate capabilities?
>
> In a terminal this task is nicely accomplished with `less`, which
> loads file on demand while you scroll (memory-efficiently) and
> automatically detects at least gzip compression. However I am
> struggling to do something similar in Emacs for quite a long time.  I
> tried VLF (requires manually switching chunks loaded, encountered
> problems with compression), view-file (loads everything in memory),
> running less in Emacs terminals (I prefer outside-Emacs terminals) but
> for various reasons all this solutions are suboptimal and I usually
> end up with using `less`.
>
> Is there something in Emacs I am missing? Is something like this makes
> sense to develop (I consider to write something for Emacs) ? I am
> using Emacs 28 and maybe in Emacs 29 there are novelties related to
> this issue?

Perhaps the best way is to report vlf bugs and fix them.

Filipp

Re: Memory-efficient way of opening large files in Emacs read-only

[Marcin Borkowski]

On 2022-12-08, at 03:49, Mikhail Pomaznoy <mikpom@mikpom.ru> wrote:

> Greetings!
>
> In my daily workflows I commonly stumble upon a task of peeking into
> a header of a large (sometimes compressed) file. Those files can
> commonly be gigabytes in compressed form so can't fit into
> memory. I was wondering how can this be done in Emacs to leverage its
> search and navigate capabilities?
>
> In a terminal this task is nicely accomplished with `less`, which
> loads file on demand while you scroll (memory-efficiently) and
> automatically detects at least gzip compression. However I am
> struggling to do something similar in Emacs for quite a long time.
> I tried VLF (requires manually switching chunks loaded, encountered
> problems with compression), view-file (loads everything in memory),
> running less in Emacs terminals (I prefer outside-Emacs terminals) but
> for various reasons all this solutions are suboptimal and I usually
> end up with using `less`.
>
> Is there something in Emacs I am missing? Is something like this makes
> sense to develop (I consider to write something for Emacs) ? I am
> using Emacs 28 and maybe in Emacs 29 there are novelties related to
> this issue?

I have no idea if Emacs has anything like this (I doubt), but the way
I'd go about it would be to either leverage `less' inside one of the
Emacs terminals or write some Elisp, colling to things like `gunzip' and
`head' and then putting the result into some buffer.  (I think you could
even make it so that scrolling down would load the next portion in the
background, à la modern Web.  While I /hate/ this behavior of websites,
here it could actually make sense.)

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Memory-efficient way of opening large files in Emacs read-only

[Mikhail Pomaznoy]

Thanks for your replies. Seamless less-like viewing in Emacs (not 
terminals of any kind) is my ultimate goal and I will look into that per 
Marcin'c suggestion. If it will turn out not fun (too difficult), 
further investment into VLF makes total sense.

Thanks,

-Mikhail

On 12/8/22 21:27, Marcin Borkowski wrote:
> On 2022-12-08, at 03:49, Mikhail Pomaznoy <mikpom@mikpom.ru> wrote:
>
>> Greetings!
>>
>> In my daily workflows I commonly stumble upon a task of peeking into
>> a header of a large (sometimes compressed) file. Those files can
>> commonly be gigabytes in compressed form so can't fit into
>> memory. I was wondering how can this be done in Emacs to leverage its
>> search and navigate capabilities?
>>
>> In a terminal this task is nicely accomplished with `less`, which
>> loads file on demand while you scroll (memory-efficiently) and
>> automatically detects at least gzip compression. However I am
>> struggling to do something similar in Emacs for quite a long time.
>> I tried VLF (requires manually switching chunks loaded, encountered
>> problems with compression), view-file (loads everything in memory),
>> running less in Emacs terminals (I prefer outside-Emacs terminals) but
>> for various reasons all this solutions are suboptimal and I usually
>> end up with using `less`.
>>
>> Is there something in Emacs I am missing? Is something like this makes
>> sense to develop (I consider to write something for Emacs) ? I am
>> using Emacs 28 and maybe in Emacs 29 there are novelties related to
>> this issue?
> I have no idea if Emacs has anything like this (I doubt), but the way
> I'd go about it would be to either leverage `less' inside one of the
> Emacs terminals or write some Elisp, colling to things like `gunzip' and
> `head' and then putting the result into some buffer.  (I think you could
> even make it so that scrolling down would load the next portion in the
> background, à la modern Web.  While I /hate/ this behavior of websites,
> here it could actually make sense.)
>
> Best,
>

Re: Memory-efficient way of opening large files in Emacs read-only

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On Fri, Dec 09, 2022 at 10:03:15AM +0700, Mikhail Pomaznoy wrote:
> Thanks for your replies. Seamless less-like viewing in Emacs (not terminals
> of any kind) is my ultimate goal and I will look into that per Marcin'c
> suggestion. If it will turn out not fun (too difficult), further investment
> into VLF makes total sense.

The elephant in the room is file coding. Emacs tries to get that right,
so usually it has to transform the whole file into its internal coding
system (which is utf8-ish but can cope with non-UTF8 stuff as well).

Less just punts on the problem and leaves it to your terminal to
cope with the display of funny stuff (apart from filtering things
it thinks might break your display, cf. less's options -r, -R and
friends).

So I guess a good starting point for you would be to play with the
function `insert-file-contents-literally' or its interactive sister
`insert-file-literally', which bypass this.

The interesting part would be how Emacs displays potentially strange
stuff in this "raw" mode, I think.

Good luck and keep us posted :)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]