auto-refresh TAGS file on ChangeLog mod?

auto-refresh TAGS file on ChangeLog mod?

[Ted Zlatanov]

I'm curious if there's a way to auto-run etags in a directory when I run
`find-tag' if the ChangeLog file (or any other file I specify) is newer
than the TAGS file.  In Makefile format, this is simply

TAGS: ChangeLog my-files-here
	etags file-list-here

or, for full interdependency

TAGS: ChangeLog file-list-here
	etags file-list-here

I didn't see anything relevant in the docs, so instead of writing it
myself I thought I'd check if anyone has done it before.

Thanks
Ted

Re: auto-refresh TAGS file on ChangeLog mod?

[Ted Zlatanov]

On Mon, 18 Aug 2008 09:28:59 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I'm curious if there's a way to auto-run etags in a directory when I run
TZ> `find-tag' if the ChangeLog file (or any other file I specify) is newer
TZ> than the TAGS file.  In Makefile format, this is simply

TZ> TAGS: ChangeLog my-files-here
TZ> 	etags file-list-here

TZ> or, for full interdependency

TZ> TAGS: ChangeLog file-list-here
TZ> 	etags file-list-here

TZ> I didn't see anything relevant in the docs, so instead of writing it
TZ> myself I thought I'd check if anyone has done it before.

I got no answers on gnu.emacs.help.  Maybe someone on emacs-devel will
have an idea.

Ted

Re: auto-refresh TAGS file on ChangeLog mod?

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> On Mon, 18 Aug 2008 09:28:59 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 
TZ> I'm curious if there's a way to auto-run etags in a directory when I run
TZ> `find-tag' if the ChangeLog file (or any other file I specify) is newer
TZ> than the TAGS file.  In Makefile format, this is simply

Ted> I got no answers on gnu.emacs.help.  Maybe someone on emacs-devel will
Ted> have an idea.

I tried to solve this problem a couple times.  Actually, I tried to
solve the more general problem of auto-updating TAGS in response to
any change, not just a change to some sentinel file.

Once I wrote a big patch to put the etags command-line arguments into
the TAGS file, so you could reliably re-run etags when a file was
saved.  This never went in; in the end I think it is not the best
approach, since it doesn't handle changes occurring outside of Emacs,
and it doesn't have a way to handle new files.

My next approach was a "retags" script.  This uses inotify to watch
for file changes, then re-runs etags.  It uses a simple ".retags" file
to decide how to invoke etags -- this solves the "new file" problem.

I can send retags to you if you want.  I posted it to emacs-sources
once.  It is pretty slow, especially the first time it starts.  That
is the main reason I don't use it.

I've wanted to roll the retags idea into etags itself.  It could read
a ".retags" file, daemonize, and use inotify to watch for changes in a
directory tree.  I never found the time to do it though.

Tom

Re: auto-refresh TAGS file on ChangeLog mod?

[Ted Zlatanov]

On Thu, 21 Aug 2008 10:31:26 -0600 Tom Tromey <tromey@redhat.com> wrote: 

Tom> I tried to solve this problem a couple times.  Actually, I tried to
Tom> solve the more general problem of auto-updating TAGS in response to
Tom> any change, not just a change to some sentinel file.

Tom> Once I wrote a big patch to put the etags command-line arguments into
Tom> the TAGS file, so you could reliably re-run etags when a file was
Tom> saved.  This never went in; in the end I think it is not the best
Tom> approach, since it doesn't handle changes occurring outside of Emacs,
Tom> and it doesn't have a way to handle new files.

Tom> My next approach was a "retags" script.  This uses inotify to watch
Tom> for file changes, then re-runs etags.  It uses a simple ".retags" file
Tom> to decide how to invoke etags -- this solves the "new file" problem.

Tom> I can send retags to you if you want.  I posted it to emacs-sources
Tom> once.  It is pretty slow, especially the first time it starts.  That
Tom> is the main reason I don't use it.

Tom> I've wanted to roll the retags idea into etags itself.  It could read
Tom> a ".retags" file, daemonize, and use inotify to watch for changes in a
Tom> directory tree.  I never found the time to do it though.

I'd like the solution to be implemented in Emacs, not externally.  Emacs
doesn't have inotify facilities AFAIK, but (like D-BUS) such system
support should be possible.  The protocol, according to
http://lwn.net/Articles/142751/, is not too complicated.  Is that a
possibility?  If so, Emacs could watch for changes anywhere in the
directory where the TAGS file was found and, when changes occur, it
could redo the indexing.  I'm sure other packages could use inotify
support as well.

Until inotify support, the list of files can be built when TAGS is
loaded (it has the file names), and then rescanned when needed.  For
most cases that will perform OK.

New files of interest can probably be inferred by building a list of
extensions in TAGS; if you have .c files already then you'll want new.c
as well.  I think anything more than this should require manual
reindexing or some user configuration of "I want to tag all *.c files."

Remembering the original list of files is not needed, right?  Can you
just run "etags --append new_or_updated_files"?  I think it's OK to have
outdated (deleted) entries in TAGS until a full rescan is done by the
user; I can't think of any other problems with this.

Ted

Re: auto-refresh TAGS file on ChangeLog mod?

[Tom Tromey]

Tom> I've wanted to roll the retags idea into etags itself.  It could read
Tom> a ".retags" file, daemonize, and use inotify to watch for changes in a
Tom> directory tree.  I never found the time to do it though.

Ted> I'd like the solution to be implemented in Emacs, not externally.

Why is that?

My first attempt was done that way.  But, given that etags is already
external to emacs, I reasoned that it doesn't buy much to go this
route.

Ted> Emacs doesn't have inotify facilities AFAIK

Nope.  FWIW, retags actually uses the inotify-tools package.  This is
just a couple of little inotify wrappers that are handy for shell
scripts; Emacs can use them just as easily.  But, as you say, adding
inotify support to Emacs would also not be difficult.

Ted> New files of interest can probably be inferred by building a list of
Ted> extensions in TAGS; if you have .c files already then you'll want new.c
Ted> as well.  I think anything more than this should require manual
Ted> reindexing or some user configuration of "I want to tag all *.c files."

Right, this is where the .retags file comes in.  It tells retags what
to do.  The particular use case that I needed here was "don't index
anything in testsuite/".  I definitely do not want new .c files in
testsuite showing up...

This project is easy if you have a simple source tree, or a small one.
I'm more interested in the weird cases, because the programs I work on
are large and all have odd cases -- files I don't want to index,
unknown file types that require regex tags, etc.

In this spirit of overdoing it, I made retags notice when .retags is
modified, and then apply any changes :)

Ted> Remembering the original list of files is not needed, right?  Can you
Ted> just run "etags --append new_or_updated_files"?  I think it's OK to have
Ted> outdated (deleted) entries in TAGS until a full rescan is done by the
Ted> user; I can't think of any other problems with this.

I have a patch somewhere that makes 'etags -u' work for emacs-style
tags files.  I can send that if you want.

Tom

Re: auto-refresh TAGS file on ChangeLog mod?

[Ted Zlatanov]

On Fri, 22 Aug 2008 09:43:14 -0600 Tom Tromey <tromey@redhat.com> wrote: 

Tom> I've wanted to roll the retags idea into etags itself.  It could read
Tom> a ".retags" file, daemonize, and use inotify to watch for changes in a
Tom> directory tree.  I never found the time to do it though.

Ted> I'd like the solution to be implemented in Emacs, not externally.

Tom> Why is that?

I like to avoid external dependencies.

Tom> My first attempt was done that way.  But, given that etags is already
Tom> external to emacs, I reasoned that it doesn't buy much to go this
Tom> route.

I understand and see your side as well.  I was hoping to avoid another
external tool.  I'm not against it, just exploring the options.

Ted> Emacs doesn't have inotify facilities AFAIK

Tom> Nope.  FWIW, retags actually uses the inotify-tools package.  This is
Tom> just a couple of little inotify wrappers that are handy for shell
Tom> scripts; Emacs can use them just as easily.  But, as you say, adding
Tom> inotify support to Emacs would also not be difficult.

I don't know if asynchronous notices will be possible (I have to look at
the libraries), or if it should be implemented as a process filter to
inotify-tools.  The latter seems a much easier target.  I'm leaving this
despite (see below) thinking it's not necessary for retagging, because
inotify support for Emacs is probably going to be useful for other
packages.

Ted> New files of interest can probably be inferred by building a list of
Ted> extensions in TAGS; if you have .c files already then you'll want new.c
Ted> as well.  I think anything more than this should require manual
Ted> reindexing or some user configuration of "I want to tag all *.c files."

Tom> Right, this is where the .retags file comes in.  It tells retags what
Tom> to do.  The particular use case that I needed here was "don't index
Tom> anything in testsuite/".  I definitely do not want new .c files in
Tom> testsuite showing up...

Thanks for clarifying.  That's something I did not consider.  It's
probably easier to just have a Makefile that will track those
dependencies automatically.  Actually, what I think may be appropriate
is just a way to say "make tags" before running `find-tag'.  All the
extra intelligence is just duplicating what the user already has in the
Makefile, and what Make provides.  WDYT?

Ted

Re: auto-refresh TAGS file on ChangeLog mod?

[Lennart Borgman (gmail)]

Ted Zlatanov wrote:
> On Fri, 22 Aug 2008 09:43:14 -0600 Tom Tromey <tromey@redhat.com> wrote: 
> 
> Tom> I've wanted to roll the retags idea into etags itself.  It could read
> Tom> a ".retags" file, daemonize, and use inotify to watch for changes in a
> Tom> directory tree.  I never found the time to do it though.
> 
> Ted> I'd like the solution to be implemented in Emacs, not externally.
> 
> Tom> Why is that?
> 
> I like to avoid external dependencies.


It might be a good idea also since inotify is not cross-platform.
(Implementing it in a way that is not cross-platform will tie up
developers resources.)

inotify support (was: auto-refresh TAGS file on ChangeLog mod?)

[Ted Zlatanov]

On Mon, 25 Aug 2008 16:53:55 +0200 "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> wrote: 

LB(> Ted Zlatanov wrote:
>> On Fri, 22 Aug 2008 09:43:14 -0600 Tom Tromey <tromey@redhat.com> wrote: 
>> 
Tom> I've wanted to roll the retags idea into etags itself.  It could read
Tom> a ".retags" file, daemonize, and use inotify to watch for changes in a
Tom> directory tree.  I never found the time to do it though.
>> 
Ted> I'd like the solution to be implemented in Emacs, not externally.
>> 
Tom> Why is that?
>> 
>> I like to avoid external dependencies.

LB> It might be a good idea also since inotify is not cross-platform.
LB> (Implementing it in a way that is not cross-platform will tie up
LB> developers resources.)

So I don't waste effort on this, does anyone know of existing inotify
support in Emacs?  Are there any packages that can use such support?

Ted

Re: auto-refresh TAGS file on ChangeLog mod?

[David House]

2008/8/25 Ted Zlatanov <tzz@lifelogs.com>:
> Thanks for clarifying.  That's something I did not consider.  It's
> probably easier to just have a Makefile that will track those
> dependencies automatically.  Actually, what I think may be appropriate
> is just a way to say "make tags" before running `find-tag'.  All the
> extra intelligence is just duplicating what the user already has in the
> Makefile, and what Make provides.  WDYT?

I think this is a good idea, as long as `make tags' is cheap, i.e., as
long as it only recalculates tags for the files which have changed.
This will require changes to etags as I described above, which Tom
says he's implemented.

Of course, the external command to run should be configurable, so
people could just call "find *.c | etags -", or use make if they
require more complicated dependency tracking, etc.

-- 
-David

Re: auto-refresh TAGS file on ChangeLog mod?

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> It's probably easier to just have a Makefile that will track
Ted> those dependencies automatically.  Actually, what I think may be
Ted> appropriate is just a way to say "make tags" before running
Ted> `find-tag'.  All the extra intelligence is just duplicating what
Ted> the user already has in the Makefile, and what Make provides.
Ted> WDYT?

The reason I did not do this is that I basically only work with
non-srcdir builds.  So, I would need to write code to tell Emacs how
to find the proper build directory from the source directory (which
for me is usually guessable -- though on occasion I work with multiple
builds from a given source).  Given that I wanted a "no fuss" setup, I
ruled this out.

The "work only from the source tree" approach does have one
disadvantage, which is that generated source files are not tagged.  In
practice I didn't mind sacrificing this -- I rarely need to look at
those files anyhow, and instead I tend to set up etags to tag the
original sources to the extent possible.

Finally, running make can do a lot more than just rebuild the TAGS
file.  It might re-run configure, rebuild all the Makefiles, etc.

Tom

Re: auto-refresh TAGS file on ChangeLog mod?

[David House]

Sorry Tom, meant this to go to the list in the first place, hope you
don't mind my quoting your answer.

2008/8/22 Tom Tromey <tromey@redhat.com>:
> David> Even on moderately large projects running etags can take a few
> David> seconds. An obvious optimisation would be to add support to etags to
> David> replace one part of the TAGS file, i.e., regnerate the tags for just a
> David> single file and splice them in to TAGS. This would make it feasible to
> David> run retags on every file save, for example.
>
> I was hoping for something with a bit less tweaking.  The nice thing
> about retags is I can make a config file once, then basically forget
> about it.

I don't see why my method would lead to more customisation. At the
moment, etags cannot replace the entries in TAGS for a given file. I
suggest adding this, so that we can call etags --replace index.php,
which will extract tags from index.php, and then replace index.php's
entry in TAGS with the tags found.

> Also, hooking into file save doesn't handle some cases, like "svn update".

Sorry, by "file save" I really meant "file update", using inotify or
similar. Or maybe just an idle-timer-hook that checked the filemtimes.


-- 
-David

Re: auto-refresh TAGS file on ChangeLog mod?

[Tom Tromey]

David> I don't see why my method would lead to more customisation.

The reason I chose the approach I did is that I thought it would be
convenient, in the long run, to have all source trees use the same
approach.  My thinking was a "make" based approach or the like means
teaching Emacs about the differences each time.

Also, you have to consider non-srcdir builds.  In this setup (which is
basically the only one I use daily, typical for GNU toolchain work),
Emacs doesn't know where to run make.  Instead, somebody has to tell
it.

Finally, in my view this was related to making Emacs more "eclipsey"
-- with eclipse you can check out a source tree and be ready to go.  I
thought it would be fun to do something like have Emacs notice that I
opened a file in a tree with a .retags file, then fire up retags and
open the TAGS file (in some new tree-local mode).

David> At the moment, etags cannot replace the entries in TAGS for a
David> given file. I suggest adding this, so that we can call etags
David> --replace index.php, which will extract tags from index.php,
David> and then replace index.php's entry in TAGS with the tags found.

I have a patch for this if you want it.  It makes -u work for the TAGS
case.

Tom

Re: auto-refresh TAGS file on ChangeLog mod?

[David House]

2008/8/22 Tom Tromey <tromey@redhat.com>:
> The reason I chose the approach I did is that I thought it would be
> convenient, in the long run, to have all source trees use the same
> approach.  My thinking was a "make" based approach or the like means
> teaching Emacs about the differences each time.
>
> Also, you have to consider non-srcdir builds.  In this setup (which is
> basically the only one I use daily, typical for GNU toolchain work),
> Emacs doesn't know where to run make.  Instead, somebody has to tell
> it.
>
> Finally, in my view this was related to making Emacs more "eclipsey"
> -- with eclipse you can check out a source tree and be ready to go.  I
> thought it would be fun to do something like have Emacs notice that I
> opened a file in a tree with a .retags file, then fire up retags and
> open the TAGS file (in some new tree-local mode).

My idea was not intended to replace yours. It was instead an
optimisation that would speed up running (r)etags every time we
noticed a file change. Otherwise I think a multi-second delay every
time you save a file is prohibitive.

> David> At the moment, etags cannot replace the entries in TAGS for a
> David> given file. I suggest adding this, so that we can call etags
> David> --replace index.php, which will extract tags from index.php,
> David> and then replace index.php's entry in TAGS with the tags found.
>
> I have a patch for this if you want it.  It makes -u work for the TAGS
> case.

This was all I was suggesting, so we seem to agree.

-- 
-David