Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

(Cross posted to comp.emacs.help, before I realized this may really be of a
more specialized topic).

I have started to rewrite gud from the ground up.

For the process buffer I have 3 choices.

1. I can stick with comint.el. It seems the most creaky.

2. term.el is pretty cool, but it doesn't provide a hook to run when
output is produced and this is something I need. I use it both in
comint.el and eshell.el in "shell tracker" (think pdb-track) mode. So
this leads to the last choice ...

3. eshell.el. It also seems pretty cool too. However its focus seems
to be more as a command shell rather than a process buffer manager
interacting via elisp to a debugger front-end. This mismatch in goals
manifests itself in little things like the ability to set the name of
the buffer initially, starting the shell with a specific debugger
invocation, customizing the banner shown on entry, avoiding the myriad
of key bindings that are not applicable here. Although little things
each easly addressed, I think they are manifestations of the larger
issue may keep cropping up if I go down this path.

So right now, my take is to add a output filter hook to term.el. But
I'd be interested and grateful in thoughts and suggestions.

[-- Attachment #2: Type: text/html, Size: 1375 bytes --]

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or

[Stefan Monnier]

> I have started to rewrite gud from the ground up.

That's interesting.  I hope you will start with the assumption that
there are several active GUD sessions (each using potentially different
debuggers for different languages).  It's a kind of change that's
difficult to make after the fact, but that's fairly easy to build-in
from the beginning.

> 1. I can stick with comint.el. It seems the most creaky.
> 2. term.el is pretty cool, but it doesn't provide a hook to run when
> output is produced and this is something I need. I use it both in
> comint.el and eshell.el in "shell tracker" (think pdb-track) mode. So
> this leads to the last choice ...

In order to make a choice, I think we need to know what problems we're
trying to address.


        Stefan

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 5054 bytes --]

On Fri, Oct 30, 2009 at 3:19 PM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

> > I have started to rewrite gud from the ground up.
>
> That's interesting.  I hope you will start with the assumption that
> there are several active GUD sessions (each using potentially different
> debuggers for different languages).  It's a kind of change that's
> difficult to make after the fact, but that's fairly easy to build-in
> from the beginning.
>

begin digression/rant:

Yep.  It has been annoying to me as well. And what makes this deja vu is
that some of the same problems in gud, you typically find in debuggers in
general: they too assume that there is just one debugger, install one trace
hook without bothering to care about others that might be set. (In some
systems, profiling and debugging share the same trace hook mechanism). And
too often the debuggers are also monolithic like gud, poorly understood, and
badly maintained.

As a small small example of how both miss the mark in about the same way
from opposite ends, debuggers often refer to a breakpoint location in terms
of a line in a file. This is inadequate given that several statements can
span a line (especially in functional programs). An in the presence of
optimization, statements may get distributed in the code and moved around
with respect to the order that they appear in the source. For the VM-based
programming language debugger I am working on, I am experimenting with
making breakpoint be a particular VM instruction; you then have other tables
that tell you how to map that from/to a source language position. A position
might include ranges of lines with column offset(s) and how to map the
instruction to a "file" which might not be a file in a filesystem at all but
could be a string variable which got eval'ed or a member of some sort of
code package.

As for gud, it uses this global variable "*gud-last-frame*'  -- actually
pair of variables if you include *gud-last-last-frame*. And what's in that
lofty sounding "frame" thing? A cons node of a file name and a line number.
But emacs has had since day one this wonderful object called a "marker". And
as one edits text,  markers move around. From the marker you can get the
buffer, and file name (if any) and its line number, or better, a column for
those systems like LLVM which support better position information. And from
day one LISP has had *lists*, so why the global two variable thing? Right
now I use this other data structure in emacs called a *ring* to store a
sequence of locations. So when I am stopped I can just go back in history to
the last *n* positions that I've stopped on.

In general I'm going to try to remove as many of the global variables as
possible in favor of buffer-local variables. And I'll assume Emacs 23 which
has structs, hash tables, rings, common lisp features and whatever newer
greater functions and packages that are commonly distributed in Emacs 23.

But here are things in particular to address the problem you mention which
stems partly from globalness. There will be process buffer(s) which
interacts with some sort of debugger(s) outside of Emacs.  These will have
buffer-local variables which contain some of the state information that in
gud is stored globally. However the source buffers that are referred to by
the debugger(s), need a buffer-local object which refers back to all of the
debugger process buffers that the source buffer currently "belongs" to.

In an earlier work that Anders Lindgren and I did (but really it was mostly
Anders), there were temporary buffers that contained collections: call stack
information, breakpoint information, local variable information. I may
eventually get to that stage, but for the first cut I think I will try to
work with just the debugger process buffer(s) and a little bit of
information in the "source" buffers which the debuggers refer to.


> > 1. I can stick with comint.el. It seems the most creaky.
> > 2. term.el is pretty cool, but it doesn't provide a hook to run when
> > output is produced and this is something I need. I use it both in
> > comint.el and eshell.el in "shell tracker" (think pdb-track) mode. So
> > this leads to the last choice ...
>
> In order to make a choice, I think we need to know what problems we're
> trying to address.
>

Perhaps some of this is addressed above.

I think of this as the same situation as comes up in any Emacs package that
interacts with some sort of process or remote process. For example TeXing a
buffer, perhaps compiling a file, or getting a directory listing. Where I
guess it is a little bit different is those things one largely thinks of as
batch jobs: you run something, get some output and that's it or maybe
iterate some more. Perhaps dired is a little bit different here.

Here there is more interaction between the back-end and front-end process.
But surely there must be other packages that have this back-and-forth
interactive feel to it too.

If this doesn't answer the question, please ask another more detailed
question.


>
>
>        Stefan
>

[-- Attachment #2: Type: text/html, Size: 5981 bytes --]

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or

[Stefan Monnier]

>> > I have started to rewrite gud from the ground up.
>> That's interesting.  I hope you will start with the assumption that
>> there are several active GUD sessions (each using potentially different
>> debuggers for different languages).  It's a kind of change that's
>> difficult to make after the fact, but that's fairly easy to build-in
>> from the beginning.

> Yep.

Good, thank you.

> [..digression/rant..]

Glad to here it.

>> > 1. I can stick with comint.el. It seems the most creaky.
>> > 2. term.el is pretty cool, but it doesn't provide a hook to run when
>> > output is produced and this is something I need. I use it both in
>> > comint.el and eshell.el in "shell tracker" (think pdb-track) mode. So
>> > this leads to the last choice ...
>> 
>> In order to make a choice, I think we need to know what problems we're
>> trying to address.
>> 

> Perhaps some of this is addressed above.

Not at all.  The apparently relevant issues have to do with whether or
not you need a terminal emumlation and things like that.
My recommendation: stick to comint.


        Stefan

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Miles Bader]

Rocky Bernstein <rocky@gnu.org> writes:
> 1. I can stick with comint.el. It seems the most creaky.
> 2. term.el is pretty cool, but it doesn't provide a hook to run when
> 3. eshell.el. It also seems pretty cool too. However its focus seems

One thing is that people are used to using comint, and are used to using
its features (both interactive and customization) with both shells and
sub-modes such as gud-mode.  While both term.el and eshell are
well-intentioned, they both started development a long time ago when
comint was less capable, and have in general not picked up new features
added to comint.  So... you can't really think of them as being strict
functional supersets of comint, they're more like... different branches
in the same tree.

E.g., one thing I've worked on a lot in comint has been prompt handling,
and as a result, comint handles prompts nicely (with highlighting and
"prompt avoidance" etc), whereas term.el doesn't (and eshell kinda
does, but not so well, and only in some cases).

Eshell seems a very different thing than either comint or term.el.  The
latter two essentially talk to an I/O stream attached to a subprocess,
and work by interpreting the output.  Eshell, on the other hand, issues
individual commands separately, and so can maintain far finer control
_between_ subcommands, but as a result tends to be not so good as long
as the subprocess itself still has control.  [As an example, start
eshell, and then type "/bin/sh" to start a subshell.  Note that although
you get shell prompts etc, everything suddenly seems "dumb" -- whereas
if you start a subshell in a comint shell, it basically seems exactly
the same as the parent shell.]

I think that eshell probably isn't appropriate for gdb for that reason.

-Miles

-- 
Infancy, n. The period of our lives when, according to Wordsworth, 'Heaven
lies about us.' The world begins lying about us pretty soon afterward.

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 2926 bytes --]

Thanks for the information.

I should have mentioned that although in the long term there are some who
feel that one will never switch to this buffer and work with the debugger in
command-line mode, in the short term that will have to be the case.

So in the long term having highlighting prompts is probably not as important
as having a good API to interface to from Emacs Lisp. But that said, term.el
doesn't have an output filter function while comint.el does.

I originally started an output filter function for term.el by calling
set-process-filter on the term buffer, but based in this info have switched
to comint instead.

Down the line if it is really important to support term.el, I think that can
be done. For example the tracker mode currently works for programs run from
eshell and from shell (but not term for the reason mentioned).

Again, thanks for the info.

On Sat, Oct 31, 2009 at 1:51 AM, Miles Bader <miles@gnu.org> wrote:

> Rocky Bernstein <rocky@gnu.org> writes:
> > 1. I can stick with comint.el. It seems the most creaky.
> > 2. term.el is pretty cool, but it doesn't provide a hook to run when
> > 3. eshell.el. It also seems pretty cool too. However its focus seems
>
> One thing is that people are used to using comint, and are used to using
> its features (both interactive and customization) with both shells and
> sub-modes such as gud-mode.  While both term.el and eshell are
> well-intentioned, they both started development a long time ago when
> comint was less capable, and have in general not picked up new features
> added to comint.  So... you can't really think of them as being strict
> functional supersets of comint, they're more like... different branches
> in the same tree.
>
> E.g., one thing I've worked on a lot in comint has been prompt handling,
> and as a result, comint handles prompts nicely (with highlighting and
> "prompt avoidance" etc), whereas term.el doesn't (and eshell kinda
> does, but not so well, and only in some cases).
>
> Eshell seems a very different thing than either comint or term.el.  The
> latter two essentially talk to an I/O stream attached to a subprocess,
> and work by interpreting the output.  Eshell, on the other hand, issues
> individual commands separately, and so can maintain far finer control
> _between_ subcommands, but as a result tends to be not so good as long
> as the subprocess itself still has control.  [As an example, start
> eshell, and then type "/bin/sh" to start a subshell.  Note that although
> you get shell prompts etc, everything suddenly seems "dumb" -- whereas
> if you start a subshell in a comint shell, it basically seems exactly
> the same as the parent shell.]
>
> I think that eshell probably isn't appropriate for gdb for that reason.
>
> -Miles
>
> --
> Infancy, n. The period of our lives when, according to Wordsworth, 'Heaven
> lies about us.' The world begins lying about us pretty soon afterward.
>

[-- Attachment #2: Type: text/html, Size: 3522 bytes --]

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Dan Nicolaescu]

Rocky Bernstein <rocky@gnu.org> writes:

  > I have started to rewrite gud from the ground up.

Another thing you might want to consider in the design is to allow
debugger interaction with simple key bindings from file buffers.  

Now to send commands to the debugger you'd have to use the *gud* buffer,
or use complicated key bindings.  
Compare to how `edebug' works, the buffer is made read-only, and simple
bindings like: n c d  can be used.

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

Check out what Anders Lindgren did with rdebug-shortkey-mode for
rdebug<http://ruby-debug.rubyforge.org/svn/trunk/emacs/rdebug-shortkey.el>

On Fri, Nov 6, 2009 at 11:57 AM, Dan Nicolaescu <dann@ics.uci.edu> wrote:

> Rocky Bernstein <rocky@gnu.org> writes:
>
>   > I have started to rewrite gud from the ground up.
>
> Another thing you might want to consider in the design is to allow
> debugger interaction with simple key bindings from file buffers.
>
> Now to send commands to the debugger you'd have to use the *gud* buffer,
> or use complicated key bindings.
> Compare to how `edebug' works, the buffer is made read-only, and simple
> bindings like: n c d  can be used.
>

[-- Attachment #2: Type: text/html, Size: 1080 bytes --]

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Nick Roberts]

 > Another thing you might want to consider in the design is to allow
 > debugger interaction with simple key bindings from file buffers.  
 > 
 > Now to send commands to the debugger you'd have to use the *gud* buffer,
 > or use complicated key bindings.  
 > Compare to how `edebug' works, the buffer is made read-only, and simple
 > bindings like: n c d  can be used.

This is not such a good idea.  The user migh actuallyt want to edit his source
code during a debug session.

-- 
Nick                                           http://users.snap.net.nz/~nickrob

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

On Fri, Nov 6, 2009 at 4:49 PM, Nick Roberts <nickrob@snap.net.nz> wrote:

>  > Another thing you might want to consider in the design is to allow
>  > debugger interaction with simple key bindings from file buffers.
>  >
>  > Now to send commands to the debugger you'd have to use the *gud* buffer,
>  > or use complicated key bindings.
>  > Compare to how `edebug' works, the buffer is made read-only, and simple
>  > bindings like: n c d  can be used.
>
> This is not such a good idea.  The user migh actuallyt want to edit his
> source
> code during a debug session.
>

Anders allows for toggling that mode. Again see the code sited for details.

Personally I found myself toggling it all the time to turn it off. That's
the point at which I realized, that the underneath support was really
lacking. So in my opinion it was pointless to pursue this until the
fundamentals were fixed up.

Previously in this thread I mentioned that gud uses a cons node (filename .
line-number) to store a position.  I have now use a struct which contains a
marker plus the source location position reported. That way I  can know not
only that the file has changed (and this is not to be the only such way),
but also by checking all markers of a buffer for a given debugger session
against their source location information, I can get a rough idea of how
other lines may have changed relative to what the source location reported
in the location structs.

Admittedly to do all of this is pretty heavy coding even if it is language
and debugger independent. But at least with these fundamentals doing this is
conceivable while with gud, it isn't.

>
> --
> Nick
> http://users.snap.net.nz/~nickrob <http://users.snap.net.nz/%7Enickrob>
>

[-- Attachment #2: Type: text/html, Size: 2377 bytes --]

Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[rocky]

I have started to rewrite gud from the ground up.

For the process buffer I have 3 choices.

1. I can stick with comint.el. It seems the most creaky.

2. term.el is pretty cool, but it doesn't provide a hook to run when
output is produced and this is something I need. I use it both in
comint.el and eshell.el in "shell tracker" (think pdb-track) mode. So
this leads to the last choice ...

3. eshell.el. It also seems pretty cool too. However its focus seems
to be more as a command shell rather than a process buffer manager
interacting via elisp to a debugger front-end. This mismatch in goals
manifests itself in little things like the ability to set the name of
the buffer initially, starting the shell with a specific debugger
invocation, customizing the banner shown on entry, avoiding the myriad
of key bindings that are not applicable here. Although little things
each easly addressed, I think they are manifestations of the larger
issue may keep cropping up if I go down this path.

So right now, my take is to add a output filter hook to term.el. But
I'd be interested and grateful in thoughts and suggestions.

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Dave Love]

rocky <rocky@gnu.org> writes:

> I have started to rewrite gud from the ground up.

Why, exactly?

> For the process buffer I have 3 choices.
>
> 1. I can stick with comint.el. It seems the most creaky.

`Creaky' how?  The whole idea of comint is to be a general way to run
sub-processes like debuggers, although there should be another layer to
abstract things like inferior programming interpreters.  What's wrong
with it for that?

> I use it both in
> comint.el and eshell.el in "shell tracker" (think pdb-track) mode. 

I implemented a prototype gud-minor-mode long ago for that sort of
thing, as alluded to in python.el if I recall correctly.  It's
straightforward but can't be a simple add-on to GUD, so wasn't going to
be useful.  The pdbtrack stuff is a mistake, apparently from a
mono-lingual programming culture, and it's unfortunate it's been
installed in Emacs, making it harder to do the right thing.

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[rocky]

On Nov 1, 4:30 pm, Dave Love <f...@domain.invalid> wrote:
> rocky <ro...@gnu.org> writes:
> > I have started to rewrite gud from the ground up.
>
> Why, exactly?
>
> > For the process buffer I have 3 choices.
>
> > 1. I can stick with comint.el. It seems the most creaky.
>
> `Creaky' how?  The whole idea of comint is to be a general way to run
> sub-processes like debuggers, although there should be another layer to
> abstract things like inferior programming interpreters.  What's wrong
> with it for that?

That comment was based on writing both a output filter hook for comint
and then one for eshell. The comint hook is passed a string with the
text to do the insertion, while the eshell hook you pass nothing. Why
nothing? Experience I guess seems to show that it's easier just to
work with the process buffer assuming the there have been marks set.
eshell and term have such marks {term,eshell}-last-input-{start,end}
while  comint seems to have only gotten this recently (Mar '08). I
guess the last time I looked this wasn't there.


>
> > I use it both in
> > comint.el and eshell.el in "shell tracker" (think pdb-track) mode.
>
> I implemented a prototype gud-minor-mode long ago for that sort of
> thing, as alluded to in python.el if I recall correctly.  It's
> straightforward but can't be a simple add-on to GUD, so wasn't going to
> be useful.  The pdbtrack stuff is a mistake,

If by "a mistake" you mean something other than it is language
specific, I'd be interested to learn in what ways you think it a
mistake.

> apparently from a
> mono-lingual programming culture, and it's unfortunate it's been
> installed in Emacs, making it harder to do the right thing.

A multi-lingual track mode is working in http://github.com/rocky/emacs-dbgr
and
it has been configured for bashdb, kshdb, zshdb, rdebug, rbdbgr, pydb,
pydbgr and remake which is enough for my needs.

Re: Grand Unified Debugger Rewrite's process buffer: comint, eterm or eshell?

[Tom Tromey]

>>>>> "Dave" == Dave Love <fx@domain.invalid> writes:

Dave> I implemented a prototype gud-minor-mode long ago for that sort of
Dave> thing, as alluded to in python.el if I recall correctly.

This sounds like an excellent idea.

Tom