Lisp Questions - reading a file and processes stalling

Lisp Questions - reading a file and processes stalling

[Jeff Clough]

I've got a few questions I'm hoping someone here can answer.

First, what is the lightest weight way of getting the contents of a file
into a buffer?  Stepping through insert-file-contents and
format-insert-file (with the requisite nil) seems to involve huge
numbers of calls and a lot of winding/unwinding the stack.  I have a
function that wants to open thousands of files, examine them then put
them away.  This is obviously slow and I'm wondering if I made some
boneheaded error in using these functions in the first place.  To be
fair, my code is slow for reasons other than this, but I'm taking one
lump of molasses at a time.

Second, I have Emacs running an external program as a process.  When
some other lengthy operation is happening elsewhere in Emacs (like Gnus
is trying to display the headers for a group*), that process stalls,
then picks up where it left off once the operation is done.  Is there
any way to make Emacs not steal the resources from this process, or am I
doing something hopelessly wrong?

* - Yes, I'm running Gnus now.  Those who remember my earlier rantings
    can feel free to slap me around.  I still hate it, but it's proven
    to be better than Mew.

Jeff

Re: Lisp Questions - reading a file and processes stalling

[Pascal J. Bourguignon]

Jeff Clough <jeff@chaosphere.com> writes:

> I've got a few questions I'm hoping someone here can answer.
>
> First, what is the lightest weight way of getting the contents of a file
> into a buffer?  Stepping through insert-file-contents and
> format-insert-file (with the requisite nil) seems to involve huge
> numbers of calls and a lot of winding/unwinding the stack. 

insert-file-contents is really the lightest weight of getting the
content of a file into a buffer.  Why do you doubt it?


Otherwise you may like find-file-literally, but it does more work.




> I have a
> function that wants to open thousands of files, examine them then put
> them away.  This is obviously slow and I'm wondering if I made some
> boneheaded error in using these functions in the first place.  To be
> fair, my code is slow for reasons other than this, but I'm taking one
> lump of molasses at a time.

Openning thousands of files will be slow anyways.

The standard solution is to build an index with the data you need, so
that you can just open one index file.


> Second, I have Emacs running an external program as a process.  When
> some other lengthy operation is happening elsewhere in Emacs (like Gnus
> is trying to display the headers for a group*), that process stalls,
> then picks up where it left off once the operation is done.  Is there
> any way to make Emacs not steal the resources from this process, or am I
> doing something hopelessly wrong?

No, there's no way.  GNU emacs is not multi-threaded.

My solution is to run ERC and GNUS each in its own instance of emacs.


-- 
__Pascal Bourguignon__

Re: Lisp Questions - reading a file and processes stalling

[Jeff Clough]

pjb@informatimago.com (Pascal J. Bourguignon) writes:

> insert-file-contents is really the lightest weight of getting the
> content of a file into a buffer.  Why do you doubt it?

When I step through my function with the debugger I see
insert-file-contents doing a lot of work.  I haven't counted, but
there's at least a dozen calls going on and that overhead
seems...excessive.  For instance, I can see it trying to match file
extensions in several batches, I assume as an attempt to determine the
correct mode for the buffer.

I can't seem to find a way of telling Emacs to put the contents of the
file in a buffer then move along.

> Otherwise you may like find-file-literally, but it does more work.

Interesting, and I see the related function
insert-file-contents-literally.  I'll take a look at it, but the
statement "it does more work" probably means it's not going to be much
different.

> Openning thousands of files will be slow anyways.
>
> The standard solution is to build an index with the data you need, so
> that you can just open one index file.

Yeah, that's what I'm doing now.  Realistically, it's unlikely that I'll
ever have to process the entire set of files more than once (to generate
the index), and can just process additions as they come.

I'm just poking at the code, trying to figure out both what's going on
under the hood and how to do things better.

>> Second, I have Emacs running an external program as a process.  When
>> some other lengthy operation is happening elsewhere in Emacs (like Gnus
>> is trying to display the headers for a group*), that process stalls,
>> then picks up where it left off once the operation is done.  Is there
>> any way to make Emacs not steal the resources from this process, or am I
>> doing something hopelessly wrong?
>
> No, there's no way.  GNU emacs is not multi-threaded.
>
> My solution is to run ERC and GNUS each in its own instance of emacs.

Yeah, I didn't think so but it was worth asking the gurus.  I'll
probably just deal with it since I'm generally happier with just one
instance of emacs.  For now, at least.

Thanks!

Jeff

Re: Lisp Questions - reading a file and processes stalling

[Stefan Monnier]

>> insert-file-contents is really the lightest weight of getting the
>> content of a file into a buffer.  Why do you doubt it?
[...]
> I can't seem to find a way of telling Emacs to put the contents of the
> file in a buffer then move along.

insert-file-contents *is* the "way of telling Emacs to put the contents
of the file in a buffer then move along".  Yes, it does a lot of work
because it needs to find the coding-system to use to read the file, and
other such things, but that's what Emacs provides.  If it's too slow,
you'll probably have to do some of the work in another process (e.g. as
is done by M-x grep).

> Yeah, I didn't think so but it was worth asking the gurus.  I'll
> probably just deal with it since I'm generally happier with just one
> instance of emacs.  For now, at least.

There's a few people who started working on adding concurrency to Emacs.
It's going to be a long road, but we hope to include a first
experimental version in Emacs-24.  IIUC their current code is able to
run Gnus in its own thread.


        Stefan