Indexed search with grep-like output

Indexed search with grep-like output

[Lennart Borgman]

I have just added a file util/idxsearch.el to nXhtml that searches via
an index and gives a grep-like output.

Currently it only supports Windows Desktop Search on w32. It does that
by querying that index through a ruby file.

It would be nice if someone wanted to add support for for example
Lucene (or some other commonly used search engine on GNU/Linux) to the
ruby file.

Do we want this kind indexed search in Emacs?


This kind of search is quite nice for quickly looking up things in for
example Emacs code (or for looking up things in org-mode files which
is my primary use). Also I will add a "locate" style function for
locating files (which takes the burden away of updating a "locate"
index).

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 1 Jan 2011 20:24:53 +0100
> 
> This kind of search is quite nice for quickly looking up things in for
> example Emacs code

The GNU Id-utils package already does that for searching identifiers
in program source code.  The package comes with id-utils.el, which
implements the Emacs front-end for it, and produces grep-like output.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sat, Jan 1, 2011 at 9:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 1 Jan 2011 20:24:53 +0100
>>
>> This kind of search is quite nice for quickly looking up things in for
>> example Emacs code
>
> The GNU Id-utils package already does that for searching identifiers
> in program source code.  The package comes with id-utils.el, which
> implements the Emacs front-end for it, and produces grep-like output.


Nice. I have never seen it.

However that does not use a modern indexed database that is
automatically incrementally updated. (I would propose extracting the
scanner part from it and using that for other databases, but that is
another project, nothing for Emacs.)

And it does not handle text queries like those you are interested in
when you are using Org mode.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 1 Jan 2011 22:15:25 +0100
> Cc: emacs-devel@gnu.org
> 
> > The GNU Id-utils package already does that for searching identifiers
> > in program source code.  The package comes with id-utils.el, which
> > implements the Emacs front-end for it, and produces grep-like output.
> 
> 
> Nice. I have never seen it.
> 
> However that does not use a modern indexed database that is
> automatically incrementally updated.

Why does it matter?  mkid is lightning fast; I have it run once a week
when I'm usually away of my development machine, and it needs only 15
minutes to index all my tens of thousands of source files in 3
humongously large source repositories.  With such speed, why should
one care for incremental updates?  You are being brain-washed by
producers of slow indexing engines, if you think incremental indexing
is a must.

The same goes for `locate' DB updates: it needs only 9 minutes to
generate a DB of 758,000 files on my 2 large disks.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sat, Jan 1, 2011 at 11:06 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 1 Jan 2011 22:15:25 +0100
>> Cc: emacs-devel@gnu.org
>>
>> > The GNU Id-utils package already does that for searching identifiers
>> > in program source code.  The package comes with id-utils.el, which
>> > implements the Emacs front-end for it, and produces grep-like output.
>>
>>
>> Nice. I have never seen it.
>>
>> However that does not use a modern indexed database that is
>> automatically incrementally updated.
>
> Why does it matter?  mkid is lightning fast; I have it run once a week
> when I'm usually away of my development machine, and it needs only 15
> minutes to index all my tens of thousands of source files in 3
> humongously large source repositories.  With such speed, why should
> one care for incremental updates?  You are being brain-washed by
> producers of slow indexing engines, if you think incremental indexing
> is a must.
>
> The same goes for `locate' DB updates: it needs only 9 minutes to
> generate a DB of 758,000 files on my 2 large disks.


That is good.

However don't you think that most people already have an indexed
automatically updated search engines on their desktop that can be used
for this too? At least that is the case for w32 as you know.

Those search engines does of course not know about the symbol id
format. (But that can be corrected by adding indexing filters.) This
might be a problem, I am not sure.

On the other hand the indexing search engines are used for text
searching which many Org users might find useful.

It looks like less trouble to me installing ruby.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sat, Jan 1, 2011 at 10:15 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Jan 1, 2011 at 9:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sat, 1 Jan 2011 20:24:53 +0100
>>>
>>> This kind of search is quite nice for quickly looking up things in for
>>> example Emacs code
>>
>> The GNU Id-utils package already does that for searching identifiers
>> in program source code.  The package comes with id-utils.el, which
>> implements the Emacs front-end for it, and produces grep-like output.
>
>
> Nice. I have never seen it.
>
> However that does not use a modern indexed database that is
> automatically incrementally updated. (I would propose extracting the
> scanner part from it and using that for other databases, but that is
> another project, nothing for Emacs.)
>
> And it does not handle text queries like those you are interested in
> when you are using Org mode.

Does id-util.el allow you to restrict your query to a directory tree?

Re: Indexed search with grep-like output

[Stefan Monnier]

> I have just added a file util/idxsearch.el to nXhtml that searches via
> an index and gives a grep-like output.
[...]
> Do we want this kind indexed search in Emacs?

I think we already have something along these lines with M-x locate, and
there are others for specific backends (like id-utils).  I'm not sure to
what extent the UI can be shared between the various search engines, but
it would be good to try and come up with one (or a small set) of
commands that can dispatch to any/various backends.  Then id-utils.el
could simply provide the GNU id backend for that command.


        Stefan

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 3:09 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> I have just added a file util/idxsearch.el to nXhtml that searches via
>> an index and gives a grep-like output.
> [...]
>> Do we want this kind indexed search in Emacs?
>
> I think we already have something along these lines with M-x locate, and
> there are others for specific backends (like id-utils).  I'm not sure to
> what extent the UI can be shared between the various search engines, but
> it would be good to try and come up with one (or a small set) of
> commands that can dispatch to any/various backends.  Then id-utils.el
> could simply provide the GNU id backend for that command.

What I am saying is we could probably do it the other way round: Use
the same backend for the functionality of locate, id-util and in
addition free text search.

With just a little bit tweaking it is there in idxsearch.el in nXhtml.
(Though it requires an indexed search engine (which should be there
already in most cases I believe) and ruby.)

Re: Indexed search with grep-like output

[Stefan Monnier]

> What I am saying is we could probably do it the other way round: Use
> the same backend for the functionality of locate, id-util and in
> addition free text search.

I have no idea what that means, but I'll assume we agree: have a command
that can run a variety of external tools like `locate', or GNU id, or
Beagle, or ...

If you really mean to provide various commands that all end up running
one particular external tool, then I don't understand why you'd want
that, since AFAICT there is no standard "external search tool" (there
may exist such a thing on some platforms, but we'd want those commands
to work on all platforms, so they'll necessarily need to support various
backends).


        Stefan

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 3:59 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> What I am saying is we could probably do it the other way round: Use
>> the same backend for the functionality of locate, id-util and in
>> addition free text search.
>
> I have no idea what that means, but I'll assume we agree: have a command
> that can run a variety of external tools like `locate', or GNU id, or
> Beagle, or ...

Sure, we are just saying it a bit different. To be clear, here is an example:

- An Emacs command like "locate" which is meant to use a backend with
an index could use several backends for that. idxsearch.el could
provide a backend for it and so could the unix command "locate".

When idxsearch.el is used it can (hopefully) use different indexing
search engines. Currently, in the first version, it supports only
Windows Desktop Search (because that is what I have here), but I hope
someone else can tweak the ruby file idxsearch.rb for other indexing
search engines, for example Lucene.

Note: At the moment I tossed away the "locate" functionality in
idxsearch.el, but it will be there again soon.


The idea of an indexed "id" search could be thought of similar.


> If you really mean to provide various commands that all end up running
> one particular external tool, then I don't understand why you'd want
> that, since AFAICT there is no standard "external search tool" (there
> may exist such a thing on some platforms, but we'd want those commands
> to work on all platforms, so they'll necessarily need to support various
> backends).

Not one particular external tool, but one particular type of externa
tool, i.e. an indexed text search engine for the desktop (or
computer). I guess they all are similar enough to fit here. (But that
is a guess, some ideas used in implementation really surprises me in
that they lack useful things that easily could have been provided.)

Re: Indexed search with grep-like output

[Stefan Monnier]

> Sure, we are just saying it a bit different. To be clear, here is an example:

Good, thanks.

> Not one particular external tool, but one particular type of externa
> tool, i.e. an indexed text search engine for the desktop (or
> computer). I guess they all are similar enough to fit here.

I hope so, too, though there can be significant differences in the kind
of query they accept.


        Stefan

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 2 Jan 2011 00:03:46 +0100
> Cc: emacs-devel@gnu.org
> 
> Does id-util.el allow you to restrict your query to a directory tree?

Yes: you need to index each such tree separately.  `lid', the command
that queries the data base, accepts a file name of an index as its
command-line argument (and its Emacs front-end accepts that as well).

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 5:07 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 2 Jan 2011 00:03:46 +0100
>> Cc: emacs-devel@gnu.org
>>
>> Does id-util.el allow you to restrict your query to a directory tree?
>
> Yes: you need to index each such tree separately.  `lid', the command
> that queries the data base, accepts a file name of an index as its
> command-line argument (and its Emacs front-end accepts that as well).

Thanks.

I do not know about other search engines, but at least Windows Desktop
Search allows you to restrict the search later. (This is used in
idxsearch.el.)

Re: Indexed search with grep-like output

[Thierry Volpiatto]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Sun, Jan 2, 2011 at 5:07 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sun, 2 Jan 2011 00:03:46 +0100
>>> Cc: emacs-devel@gnu.org
>>>
>>> Does id-util.el allow you to restrict your query to a directory tree?
>>
>> Yes: you need to index each such tree separately.  `lid', the command
>> that queries the data base, accepts a file name of an index as its
>> command-line argument (and its Emacs front-end accepts that as well).
>
> Thanks.
>
> I do not know about other search engines, but at least Windows Desktop
> Search allows you to restrict the search later. (This is used in
> idxsearch.el.)
See http://www.voidtools.com/download.php
it's what i use for locate command (es) in windows.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 2 Jan 2011 05:12:32 +0100
> Cc: emacs-devel@gnu.org
> 
> I do not know about other search engines, but at least Windows Desktop
> Search allows you to restrict the search later.

Because it's dog slow.  `lid' is very fast, so if you don't want to
index each tree separately, you can filter the output (which includes
the full absolute file name) in Emacs, in the process filter that
accepts `lid's output.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 12:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 2 Jan 2011 05:12:32 +0100
>> Cc: emacs-devel@gnu.org
>>
>> I do not know about other search engines, but at least Windows Desktop
>> Search allows you to restrict the search later.
>
> Because it's dog slow.

Is it? Can you give an example?

I read that people have had good experience with Lucene and large
amounts of files. Have you tested Lucene?

> `lid' is very fast, so if you don't want to
> index each tree separately, you can filter the output (which includes
> the full absolute file name) in Emacs, in the process filter that
> accepts `lid's output.

Is not that slow since it requires a lot process switching?

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 2 Jan 2011 12:40:10 +0100
> Cc: emacs-devel@gnu.org
> 
> On Sun, Jan 2, 2011 at 12:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Lennart Borgman <lennart.borgman@gmail.com>
> >> Date: Sun, 2 Jan 2011 05:12:32 +0100
> >> Cc: emacs-devel@gnu.org
> >>
> >> I do not know about other search engines, but at least Windows Desktop
> >> Search allows you to restrict the search later.
> >
> > Because it's dog slow.
> 
> Is it? Can you give an example?

`lid' takes 0.1sec to return results of a query of a DB that indexes
12,800 source files of a large C++ software system.  What's your
timing of a comparable query with Windows Desktop?

> I read that people have had good experience with Lucene and large
> amounts of files. Have you tested Lucene?

Yes, I use one of the tools that builds on Lucene (see
http://www.methods.co.nz/docindexer/) to index MS Office documents
(some 17,000 of them) I have on my office machine.  It is also very
fast: just a few seconds to return a query.

> > `lid' is very fast, so if you don't want to
> > index each tree separately, you can filter the output (which includes
> > the full absolute file name) in Emacs, in the process filter that
> > accepts `lid's output.
> 
> Is not that slow since it requires a lot process switching?

Not slower than grep or compilation -- if you want the results to be
available to Emacs, and don't want to use synchronous subprocesses (so
that you could continue working while it runs, and perhaps even lookup
the first bunch of hits), you will always use this method.  (I don't
understand what you mean by "a lot of process switching" -- there are
two processes running in parallel.)

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 2:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 2 Jan 2011 12:40:10 +0100
>> Cc: emacs-devel@gnu.org
>>
>> On Sun, Jan 2, 2011 at 12:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> From: Lennart Borgman <lennart.borgman@gmail.com>
>> >> Date: Sun, 2 Jan 2011 05:12:32 +0100
>> >> Cc: emacs-devel@gnu.org
>> >>
>> >> I do not know about other search engines, but at least Windows Desktop
>> >> Search allows you to restrict the search later.
>> >
>> > Because it's dog slow.
>>
>> Is it? Can you give an example?
>
> `lid' takes 0.1sec to return results of a query of a DB that indexes
> 12,800 source files of a large C++ software system.  What's your
> timing of a comparable query with Windows Desktop?

Around 2 seconds.

But that is from within Emacs on a very slow computer. (And less than
8,000 files.)


>> I read that people have had good experience with Lucene and large
>> amounts of files. Have you tested Lucene?
>
> Yes, I use one of the tools that builds on Lucene (see
> http://www.methods.co.nz/docindexer/) to index MS Office documents
> (some 17,000 of them) I have on my office machine.  It is also very
> fast: just a few seconds to return a query.

Is that from within Emacs or?


>> > `lid' is very fast, so if you don't want to
>> > index each tree separately, you can filter the output (which includes
>> > the full absolute file name) in Emacs, in the process filter that
>> > accepts `lid's output.
>>
>> Is not that slow since it requires a lot process switching?
>
> Not slower than grep or compilation -- if you want the results to be
> available to Emacs, and don't want to use synchronous subprocesses (so
> that you could continue working while it runs, and perhaps even lookup
> the first bunch of hits), you will always use this method.  (I don't
> understand what you mean by "a lot of process switching" -- there are
> two processes running in parallel.)

The subprocess must be the current process when it sends the output to
Emacs and Emacs must be the current process when it receives the
output. So the operating system has to switch between them and as I
understands it that operation is rather costly on w32 especially. (But
I do not remember any comparable timings any longer.)

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 2 Jan 2011 14:46:45 +0100
> Cc: emacs-devel@gnu.org
> 
> > Yes, I use one of the tools that builds on Lucene (see
> > http://www.methods.co.nz/docindexer/) to index MS Office documents
> > (some 17,000 of them) I have on my office machine.  It is also very
> > fast: just a few seconds to return a query.
> 
> Is that from within Emacs or?

It doesn't matter.  Most of the time is to output a long list of
documents, i.e. I/O.

> The subprocess must be the current process when it sends the output to
> Emacs and Emacs must be the current process when it receives the
> output. So the operating system has to switch between them and as I
> understands it that operation is rather costly on w32 especially.

"Rather costly" compared to what?  I see no such issues on Windows (on
a 6-year old machine).  If they exist, they must be negligible
relative to other factors.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 2:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 2 Jan 2011 14:46:45 +0100
>> Cc: emacs-devel@gnu.org
>>
>> > Yes, I use one of the tools that builds on Lucene (see
>> > http://www.methods.co.nz/docindexer/) to index MS Office documents
>> > (some 17,000 of them) I have on my office machine.  It is also very
>> > fast: just a few seconds to return a query.
>>
>> Is that from within Emacs or?
>
> It doesn't matter.  Most of the time is to output a long list of
> documents, i.e. I/O.

Does that mean you actually see the same time inside of Emacs as
outside of Emacs? Or is it just what you expect?

>> The subprocess must be the current process when it sends the output to
>> Emacs and Emacs must be the current process when it receives the
>> output. So the operating system has to switch between them and as I
>> understands it that operation is rather costly on w32 especially.
>
> "Rather costly" compared to what?  I see no such issues on Windows (on
> a 6-year old machine).  If they exist, they must be negligible
> relative to other factors.

That is not how I remember it, but I do not have time to look it up now.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 2 Jan 2011 15:15:39 +0100
> Cc: emacs-devel@gnu.org
> 
> On Sun, Jan 2, 2011 at 2:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Lennart Borgman <lennart.borgman@gmail.com>
> >> Date: Sun, 2 Jan 2011 14:46:45 +0100
> >> Cc: emacs-devel@gnu.org
> >>
> >> > Yes, I use one of the tools that builds on Lucene (see
> >> > http://www.methods.co.nz/docindexer/) to index MS Office documents
> >> > (some 17,000 of them) I have on my office machine.  It is also very
> >> > fast: just a few seconds to return a query.
> >>
> >> Is that from within Emacs or?
> >
> > It doesn't matter.  Most of the time is to output a long list of
> > documents, i.e. I/O.
> 
> Does that mean you actually see the same time inside of Emacs as
> outside of Emacs?

Yes, the same (2.5 seconds for a query that returns 951 documents).

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sun, Jan 2, 2011 at 4:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sun, 2 Jan 2011 15:15:39 +0100
>> Cc: emacs-devel@gnu.org
>>
>> On Sun, Jan 2, 2011 at 2:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> From: Lennart Borgman <lennart.borgman@gmail.com>
>> >> Date: Sun, 2 Jan 2011 14:46:45 +0100
>> >> Cc: emacs-devel@gnu.org
>> >>
>> >> > Yes, I use one of the tools that builds on Lucene (see
>> >> > http://www.methods.co.nz/docindexer/) to index MS Office documents
>> >> > (some 17,000 of them) I have on my office machine.  It is also very
>> >> > fast: just a few seconds to return a query.
>> >>
>> >> Is that from within Emacs or?
>> >
>> > It doesn't matter.  Most of the time is to output a long list of
>> > documents, i.e. I/O.
>>
>> Does that mean you actually see the same time inside of Emacs as
>> outside of Emacs?
>
> Yes, the same (2.5 seconds for a query that returns 951 documents).


I think my estimation of the time required were a bit too high (though
it depends on whether it is "cold" or warm test).

I have now added support for Google Desktop Search too (the grepping
is not added there yet though). It feels snappier, but the timing
tells me it should be about the same as Windows Desktop Search. Wonder
if I did something wrong... ;-)

The Google version is a bit more cumbersome to setup, but on the other
hand you do not need to add ruby for it. And it is cross platform.

Both the Google and the Windows version seems fast enough at least for
the data set I have. On my old pc it takes no more than 1-2 seconds in
most cases to do the search.

It would be nice to have support for other search engines too, but I
am not going to add that myself. However if they are SQL-based then it
should be a simple matter of restructuring the ruby file a little bit,
generalizing the code a bit more, to add them.


Note that my part of the coding is not quite ready yet (some parts are
missing and there are some bugs), but the structure is there so it
should be ok to add new search engines to what there is now.

Re: Indexed search with grep-like output

[Stefan Monnier]

> I have now added support for Google Desktop Search too (the grepping
> is not added there yet though). It feels snappier, but the timing
> tells me it should be about the same as Windows Desktop Search. Wonder
> if I did something wrong... ;-)

Clearly, the wrong thing you did is focus on non-Free tools.


        Stefan

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 5:10 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> I have now added support for Google Desktop Search too (the grepping
>> is not added there yet though). It feels snappier, but the timing
>> tells me it should be about the same as Windows Desktop Search. Wonder
>> if I did something wrong... ;-)
>
> Clearly, the wrong thing you did is focus on non-Free tools.

;-)

That is not my fault. Those are the only tools I could find that are
easy to setup on w32.

Anyway I hope the structure is good enough to add free tools. (For
someone using GNU/Linux probably much easier. At least that is my
impression. Nearly all non-trivial free tools are a pain to setup on
w32. And that is part of the reason they are not gaining ground, at
least that is my guess.)

Re: Indexed search with grep-like output

[Daniel Colascione]

On 1/2/2011 8:10 PM, Stefan Monnier wrote:
>> I have now added support for Google Desktop Search too (the grepping
>> is not added there yet though). It feels snappier, but the timing
>> tells me it should be about the same as Windows Desktop Search. Wonder
>> if I did something wrong... ;-)
> 
> Clearly, the wrong thing you did is focus on non-Free tools.

If the index is there anyway, there's nothing wrong with providing
access to it. Emacs has never been designed to work exclusively with
free software, as desirable as that situation would be.

Re: Indexed search with grep-like output

[Stefan Monnier]

>> Clearly, the wrong thing you did is focus on non-Free tools.
> If the index is there anyway, there's nothing wrong with providing
> access to it.  Emacs has never been designed to work exclusively with
> free software, as desirable as that situation would be.

Emacs has always been designed around the idea that you shouldn't need
proprietary code to use a feature, so his code can't be integrated until
it is made to also support Free alternatives.

And regardless, I find it to be an error of his to prefer
ease-of-installation over freedom.  Of course, that's just my opinion
(tho I suspect I share it with several other people on this list).


        Stefan

Re: Indexed search with grep-like output

[Dimitri Fontaine]

Lennart Borgman <lennart.borgman@gmail.com> writes:
> On Mon, Jan 3, 2011 at 5:10 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Clearly, the wrong thing you did is focus on non-Free tools.
>
> That is not my fault. Those are the only tools I could find that are
> easy to setup on w32.

  http://strigi.sourceforge.net/

IIUC it should be installed with KDE for windows.

Regards,
-- 
dim

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Mon, 3 Jan 2011 05:14:19 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> > Clearly, the wrong thing you did is focus on non-Free tools.
> 
> ;-)
> 
> That is not my fault. Those are the only tools I could find that are
> easy to setup on w32.

I just mentioned 2 Free tools in this thread alone.  And there are
others.  So I don't see how you can say it's not your fault.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 10:31 AM, Dimitri Fontaine <dim@tapoueh.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>> On Mon, Jan 3, 2011 at 5:10 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> Clearly, the wrong thing you did is focus on non-Free tools.
>>
>> That is not my fault. Those are the only tools I could find that are
>> easy to setup on w32.
>
>  http://strigi.sourceforge.net/

Thanks, but there seems to be no binary distributions for w32. (And
since there is not I have the feeling that it is hard to build on w32.
I have seen several such cases.)


> IIUC it should be installed with KDE for windows.

Thanks, but it does not seem to be there.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 2:02 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Mon, 3 Jan 2011 05:14:19 +0100
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> > Clearly, the wrong thing you did is focus on non-Free tools.
>>
>> ;-)
>>
>> That is not my fault. Those are the only tools I could find that are
>> easy to setup on w32.
>
> I just mentioned 2 Free tools in this thread alone.  And there are
> others.  So I don't see how you can say it's not your fault.

Which ones? I mentioned Lucene. You mentioned GNU Id, but that does
not fit for me.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 7:14 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> Emacs has always been designed around the idea that you shouldn't need
> proprietary code to use a feature, so his code can't be integrated until
> it is made to also support Free alternatives.

I have nothing against that. That is what I expects, but I also expect
someone else to add the free alternatives of their choice. (I will
help with that of course.)

> And regardless, I find it to be an error of his to prefer
> ease-of-installation over freedom.

It is not an error. It is what I prefer.

And I do not expect free tools to succeed unless they are easy to use.
And I think I share that opinion with a lot of people who supports
free software.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> Date: Sun, 02 Jan 2011 21:40:30 -0800
> From: Daniel Colascione <dan.colascione@gmail.com>
> CC: Lennart Borgman <lennart.borgman@gmail.com>, 
>  Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> > Clearly, the wrong thing you did is focus on non-Free tools.
> 
> If the index is there anyway, there's nothing wrong with providing
> access to it.

But the index isn't "there anyway", at least not in the case of Google
Desktop: you need to install Google Desktop first.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 6:41 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sun, 02 Jan 2011 21:40:30 -0800
>> From: Daniel Colascione <dan.colascione@gmail.com>
>> CC: Lennart Borgman <lennart.borgman@gmail.com>,
>>  Eli Zaretskii <eliz@gnu.org>,
>>  emacs-devel@gnu.org
>>
>> > Clearly, the wrong thing you did is focus on non-Free tools.
>>
>> If the index is there anyway, there's nothing wrong with providing
>> access to it.
>
> But the index isn't "there anyway", at least not in the case of Google
> Desktop: you need to install Google Desktop first.

Though you may have done that before thinking about using it in Emacs.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Dimitri Fontaine <dim@tapoueh.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> User-Mail-Address: dim@tapoueh.org
> Date: Mon, 03 Jan 2011 10:31:18 +0100
> 
>   http://strigi.sourceforge.net/
> 
> IIUC it should be installed with KDE for windows.

It needs to be compiled first, because I don't see any Windows
binaries available there.

Also, doesn't look like it's actively maintained (last updated more
than a year ago).

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Mon, 3 Jan 2011 16:05:34 +0100
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >> That is not my fault. Those are the only tools I could find that are
> >> easy to setup on w32.
> >
> > I just mentioned 2 Free tools in this thread alone.  And there are
> > others.  So I don't see how you can say it's not your fault.
> 
> Which ones?

ID-utils and docindexer.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Mon, 3 Jan 2011 18:44:47 +0100
> Cc: Daniel Colascione <dan.colascione@gmail.com>, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > But the index isn't "there anyway", at least not in the case of Google
> > Desktop: you need to install Google Desktop first.
> 
> Though you may have done that before thinking about using it in Emacs.

Free Software is not only about Emacs.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 6:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Mon, 3 Jan 2011 16:05:34 +0100
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>>
>> >> That is not my fault. Those are the only tools I could find that are
>> >> easy to setup on w32.
>> >
>> > I just mentioned 2 Free tools in this thread alone.  And there are
>> > others.  So I don't see how you can say it's not your fault.
>>
>> Which ones?
>
> ID-utils and docindexer.

I am just testing docindexer, but the documentation for docindexer.exe
is very unclear. Two questions:

- How do I tell it that .org, .el etc files contains plain text?
- How do I add another directory root to the index? (Or how do I tell
it to index several separate directories.)

And for docsearch.exe:

- How do I tell it to search from several directory roots?

Re: Indexed search with grep-like output

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>>   http://strigi.sourceforge.net/
>> 
>> IIUC it should be installed with KDE for windows.
>
> It needs to be compiled first, because I don't see any Windows
> binaries available there.
>
> Also, doesn't look like it's actively maintained (last updated more
> than a year ago).

It's still maintained, but it doesn't seem to be available as
stand-alone project but only as part of KDE.  It was split in some
parts, and these now live in in several KDE git repositories:

https://projects.kde.org/projects/kdesupport/strigi/strigiutils/repository
https://projects.kde.org/projects/kdesupport/strigi/strigidaemon/repository
https://projects.kde.org/projects/kdesupport/strigi/strigiclient/repository
https://projects.kde.org/projects/kdesupport/strigi/libstreams/repository
https://projects.kde.org/projects/kdesupport/strigi/libstreamanalyzer/repository

Bye,
Tassilo

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 7:46 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>
> Hi Eli,
>
>>>   http://strigi.sourceforge.net/
>>>
>>> IIUC it should be installed with KDE for windows.
>>
>> It needs to be compiled first, because I don't see any Windows
>> binaries available there.
>>
>> Also, doesn't look like it's actively maintained (last updated more
>> than a year ago).
>
> It's still maintained, but it doesn't seem to be available as
> stand-alone project but only as part of KDE.

I can not find it in KDE for Windows Installer. So it seems not to be available.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 5:14 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Mon, Jan 3, 2011 at 5:10 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> I have now added support for Google Desktop Search too (the grepping
>>> is not added there yet though). It feels snappier, but the timing
>>> tells me it should be about the same as Windows Desktop Search. Wonder
>>> if I did something wrong... ;-)
>>
>> Clearly, the wrong thing you did is focus on non-Free tools.
>
> ;-)
>
> That is not my fault. Those are the only tools I could find that are
> easy to setup on w32.
>
> Anyway I hope the structure is good enough to add free tools. (For
> someone using GNU/Linux probably much easier. At least that is my
> impression. Nearly all non-trivial free tools are a pain to setup on
> w32. And that is part of the reason they are not gaining ground, at
> least that is my guess.)


As Eli suggested docindexer (which is free) and there is an installer
for that I added support for docindexer too.

It works nicely for what it does. However unfortunately it is still
unusable since the binary version of docindexer currently does not let
you tell which files extensions it should index as text. I have asked
the author/maintainer of docindexer, Stuart Rackham, for help and hope
he will fix it.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Tue, Jan 4, 2011 at 4:21 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Mon, Jan 3, 2011 at 5:14 AM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Mon, Jan 3, 2011 at 5:10 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>> I have now added support for Google Desktop Search too (the grepping
>>>> is not added there yet though). It feels snappier, but the timing
>>>> tells me it should be about the same as Windows Desktop Search. Wonder
>>>> if I did something wrong... ;-)
>>>
>>> Clearly, the wrong thing you did is focus on non-Free tools.
>>
>> ;-)
>>
>> That is not my fault. Those are the only tools I could find that are
>> easy to setup on w32.
>>
>> Anyway I hope the structure is good enough to add free tools. (For
>> someone using GNU/Linux probably much easier. At least that is my
>> impression. Nearly all non-trivial free tools are a pain to setup on
>> w32. And that is part of the reason they are not gaining ground, at
>> least that is my guess.)
>
>
> As Eli suggested docindexer (which is free) and there is an installer
> for that I added support for docindexer too.
>
> It works nicely for what it does. However unfortunately it is still
> unusable since the binary version of docindexer currently does not let
> you tell which files extensions it should index as text. I have asked
> the author/maintainer of docindexer, Stuart Rackham, for help and hope
> he will fix it.

I have got a reply from Stuart now and he confirms that you can not
use the easy installed binary distribution on w32 if you want to
change the file types that are indexed.

This may make it a less good alternative for those who want to use it
for Emacs on w32. However it is still usable with idxsearch.el.

To summarize:

- There are now support for 3 search engines:

  1) docindexer - free and easy to install, but you can't change the
file types indexed (easily).

  2) Google Desktop Search - easy to install and runs on free systems
too. You have to add a plugin to support new file types.

  3) Windows Desktop Search - comes with the system, but you have to
install ruby to use it with Emacs.

Of these Google Desktop Search seems to be the best alternative for a
user. (And it gives the best output too.)

I have not find any other indexed search engines that are easy to
install on w32. (Others might be very easy to install on GNU/Linux,
but I see no reason to install them on w32 since they require more of
your time.)


idxsearch.el and its companions are not yet quite ready though.
Comments are welcome.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Tue, 4 Jan 2011 04:21:22 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel devel <emacs-devel@gnu.org>, 
> 	Stuart Rackham <srackham@gmail.com>
> 
> It works nicely for what it does. However unfortunately it is still
> unusable since the binary version of docindexer currently does not let
> you tell which files extensions it should index as text.

In the directory where you installed docindexer, there's a file named
conf.py, a piece of Python code that describes the docindexer parser
configuration.  Its syntax should be self-explanatory; you can add
entries there for whatever source files you'd like to index.

Having said that, I don't think docindexer is the right tool for
indexing program source files.  Lucene text analyzers are biased
towards indexing plain text, so they typically ignore one-letter
words, like "a" and "i", words like "the", "in", "on", "some", etc. --
which could well be valid identifiers in a program.  It really isn't
the tool for this job.

For indexing source code, ID-utils is what you want; it doesn't
currently have an ELisp parser, but if you are willing to index *.el
files as plain text, you can tweak the id-util.map file to tell mkid
to treat *.el files as text (a plain text _is_ included in ID-utils).
If that's not good enough, writing an ELisp parser should be hard.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Tue, Jan 4, 2011 at 8:09 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Tue, 4 Jan 2011 04:21:22 +0100
>> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel devel <emacs-devel@gnu.org>,
>>       Stuart Rackham <srackham@gmail.com>
>>
>> It works nicely for what it does. However unfortunately it is still
>> unusable since the binary version of docindexer currently does not let
>> you tell which files extensions it should index as text.
>
> In the directory where you installed docindexer, there's a file named
> conf.py, a piece of Python code that describes the docindexer parser
> configuration.  Its syntax should be self-explanatory; you can add
> entries there for whatever source files you'd like to index.

No, you do not have that file if you used the installer and installed
the binary version. If you want to use that installer you can not
change the how files with different extensions are parsed by docindex.


> Having said that, I don't think docindexer is the right tool for
> indexing program source files.  Lucene text analyzers are biased
> towards indexing plain text, so they typically ignore one-letter
> words, like "a" and "i", words like "the", "in", "on", "some", etc. --
> which could well be valid identifiers in a program.  It really isn't
> the tool for this job.

It does not give an index of the kind you want, that is correct.
However I might still find it handy to quickly find parts of the code.


> For indexing source code, ID-utils is what you want; it doesn't
> currently have an ELisp parser, but if you are willing to index *.el
> files as plain text, you can tweak the id-util.map file to tell mkid
> to treat *.el files as text (a plain text _is_ included in ID-utils).
> If that's not good enough, writing an ELisp parser should be hard.

I did not look into ID-utils, but perhaps it can use other programs to
extract the id:s? In that case it could use Emacs.

If you want to then feel free to add support for ID-utils to
idxsearch.el. It should typically be a file on its own. The file
idxdocindex.el is a good starting example.

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Tue, 4 Jan 2011 08:22:09 +0100
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org, srackham@gmail.com
> 
> > In the directory where you installed docindexer, there's a file named
> > conf.py, a piece of Python code that describes the docindexer parser
> > configuration.  Its syntax should be self-explanatory; you can add
> > entries there for whatever source files you'd like to index.
> 
> No, you do not have that file if you used the installer and installed
> the binary version.

Well, I certainly did use the installer, and I do have that file.  Are
you sure you don't have it?

In any case, you can find it in the docindexer source distribution.

> If you want to use that installer you can not
> change the how files with different extensions are parsed by docindex.

But I just did change that.  Here's the exact recipe:

 . Find config.py in the docindexer installation directory and edit it
   to add a line for *.el files.
 . Find a file named library.zip in the docindexer installation
   directory.  This is the class library used by docindexer.
 . Replace the file docindexer/config.pyc in library.zip with the
   edited docindexer/config.py.  Note: the .pyc extension means that
   the file was compiled by Python; the corresponding .py file is not
   compiled, but it will be used anyway -- this is similar to what
   Emacs does with *.el and *.elc files.
 . Run "docindexer --config" and make sure you see the *.el line in
   the output.

After performing the above procedure, I have just indexed the entire
Emacs lisp/ directory.  It took 3 minutes (yes, the indexer is not
very fast, which is why it's scheduled to run at night when I'm away;
mkid does the same job 3 times faster).

Morale: Never underestimate the power of Free Software!  When you have
sources, _you_ are in control, not the software developer.  This is
what Free Software is all about.

> > Having said that, I don't think docindexer is the right tool for
> > indexing program source files.  Lucene text analyzers are biased
> > towards indexing plain text, so they typically ignore one-letter
> > words, like "a" and "i", words like "the", "in", "on", "some", etc. --
> > which could well be valid identifiers in a program.  It really isn't
> > the tool for this job.
> 
> It does not give an index of the kind you want, that is correct.
> However I might still find it handy to quickly find parts of the code.

Is it really handy?  Lisp identifiers include punctuation characters
such as `-', `>', `:', etc.  I'd guess that plain text indexing will
not index these identifiers as you'd want to.

> If you want to then feel free to add support for ID-utils to
> idxsearch.el. It should typically be a file on its own. The file
> idxdocindex.el is a good starting example.

I'd rather extend id-utils.el, and eventually add that to Emacs.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Tue, Jan 4, 2011 at 9:11 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Tue, 4 Jan 2011 08:22:09 +0100
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org, srackham@gmail.com
>>
>> > In the directory where you installed docindexer, there's a file named
>> > conf.py, a piece of Python code that describes the docindexer parser
>> > configuration.  Its syntax should be self-explanatory; you can add
>> > entries there for whatever source files you'd like to index.
>>
>> No, you do not have that file if you used the installer and installed
>> the binary version.
>
> Well, I certainly did use the installer, and I do have that file.  Are
> you sure you don't have it?

Yes, and I just told what Stuart told me.

> In any case, you can find it in the docindexer source distribution.
>
>> If you want to use that installer you can not
>> change the how files with different extensions are parsed by docindex.
>
> But I just did change that.  Here's the exact recipe:
>
>  . Find config.py in the docindexer installation directory and edit it
>   to add a line for *.el files.
>  . Find a file named library.zip in the docindexer installation
>   directory.  This is the class library used by docindexer.
>  . Replace the file docindexer/config.pyc in library.zip with the
>   edited docindexer/config.py.  Note: the .pyc extension means that
>   the file was compiled by Python; the corresponding .py file is not
>   compiled, but it will be used anyway -- this is similar to what
>   Emacs does with *.el and *.elc files.
>  . Run "docindexer --config" and make sure you see the *.el line in
>   the output.

Hm. Nice.


> After performing the above procedure, I have just indexed the entire
> Emacs lisp/ directory.  It took 3 minutes (yes, the indexer is not
> very fast, which is why it's scheduled to run at night when I'm away;
> mkid does the same job 3 times faster).
>
> Morale: Never underestimate the power of Free Software!  When you have
> sources, _you_ are in control, not the software developer.  This is
> what Free Software is all about.

It is nice, but normally you do not want to go through trouble just to
use a new bit of software. You might rightly suspect that it is a
waste of time to do that in many cases.


>> > Having said that, I don't think docindexer is the right tool for
>> > indexing program source files.  Lucene text analyzers are biased
>> > towards indexing plain text, so they typically ignore one-letter
>> > words, like "a" and "i", words like "the", "in", "on", "some", etc. --
>> > which could well be valid identifiers in a program.  It really isn't
>> > the tool for this job.
>>
>> It does not give an index of the kind you want, that is correct.
>> However I might still find it handy to quickly find parts of the code.
>
> Is it really handy?  Lisp identifiers include punctuation characters
> such as `-', `>', `:', etc.  I'd guess that plain text indexing will
> not index these identifiers as you'd want to.

It is a good point, I do not know. But I wrote this mainly for
org-mode files and thought it could be used for code too for quickly
finding something.

I still think it can - if I change one little thing: If the line
matcher uses AND it would be much better.


>> If you want to then feel free to add support for ID-utils to
>> idxsearch.el. It should typically be a file on its own. The file
>> idxdocindex.el is a good starting example.
>
> I'd rather extend id-utils.el, and eventually add that to Emacs.

Well, it might be good they stay separate.

Another way to handle it might be to use the parsers from ID-utils and
feed a regular search engine. Though you might perhaps then still have
trouble with the query language. (Which of course can be solved in
free software...)

Re: Indexed search with grep-like output

[Stefan Monnier]

>> If you want to then feel free to add support for ID-utils to
>> idxsearch.el. It should typically be a file on its own. The file
>> idxdocindex.el is a good starting example.
> I'd rather extend id-utils.el, and eventually add that to Emacs.

All things being equal, I'd rather add a package that has abstractions
to work with various indexers, than one that only works for one
particular kind of indexer, especially since we're unlikely to convince
all potential users of such a tool to switch to GNU id if they're
currently using something else.


        Stefan

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Lennart Borgman <lennart.borgman@gmail.com>, emacs-devel@gnu.org,
>         srackham@gmail.com
> Date: Wed, 05 Jan 2011 16:43:15 -0500
> 
> > I'd rather extend id-utils.el, and eventually add that to Emacs.
> 
> All things being equal, I'd rather add a package that has abstractions
> to work with various indexers, than one that only works for one
> particular kind of indexer

If it wasn't clear, that's what I was suggesting by "extend".  I just
think that id-utils.el will be a good starting point, since it already
uses the infrastructure provided by compile.el.

Re: Indexed search with grep-like output

[Stefan Monnier]

>> > I'd rather extend id-utils.el, and eventually add that to Emacs.
>> All things being equal, I'd rather add a package that has abstractions
>> to work with various indexers, than one that only works for one
>> particular kind of indexer.
> If it wasn't clear, that's what I was suggesting by "extend".  I just
> think that id-utils.el will be a good starting point,

Oh, I see, I had misunderstood.  Good, then.


        Stefan

Re: Indexed search with grep-like output

[Lennart Borgman]

On Thu, Jan 6, 2011 at 1:15 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> > I'd rather extend id-utils.el, and eventually add that to Emacs.
>>> All things being equal, I'd rather add a package that has abstractions
>>> to work with various indexers, than one that only works for one
>>> particular kind of indexer.
>> If it wasn't clear, that's what I was suggesting by "extend".  I just
>> think that id-utils.el will be a good starting point,
>
> Oh, I see, I had misunderstood.  Good, then.


The current idutils.el does not contain much. It is just a
define-compilation-mode (which adds nothing but adds one keyword) and
a a command gid which just does compilation-start (and does some
things that belongs to define-compile-command).

But since Eli says idutils are good (and I do not doubt that) I
started looking at it. I thought there was no binary download (it is
not mentioned on the GNU Id Utils home page at all), but there is one
in gnuwin32.

I installed that and tried it. However it either crashes or says
something like this

  C:\>mkid -o OUTFILE DIR
  mkid: can't get working directory: Permission denied

Eli, can you tell me what binary you are using and how you run mkid?

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Mon, 10 Jan 2011 11:22:35 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org, srackham@gmail.com
> 
> The current idutils.el does not contain much. It is just a
> define-compilation-mode (which adds nothing but adds one keyword) and
> a a command gid which just does compilation-start (and does some
> things that belongs to define-compile-command).

That's because lid, the command that performs the indexed search, has
an option that causes it to produce grep-like output.  So id-utils.el
doesn't need to do much, since everything is already in compile.el.
For other search programs, which don't support output formats that map
well to what compile.el groks, more massaging of the output will be
needed.

> I installed that and tried it. However it either crashes or says
> something like this
> 
>   C:\>mkid -o OUTFILE DIR
>   mkid: can't get working directory: Permission denied

Which is why I made my own port: the one on GnuWin32 is simply broken.
If you tell me how to upload a binary zip file to some site where you
can get it, I will.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 10, 2011 at 7:33 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Which is why I made my own port: the one on GnuWin32 is simply broken.
> If you tell me how to upload a binary zip file to some site where you
> can get it, I will.


Ok, since you have got an offer to upload your version to GnuWin32 I
will grab it from there when you have uploaded it. Please tell me when
you have done that.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Tue, Jan 4, 2011 at 8:09 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Tue, 4 Jan 2011 04:21:22 +0100
>> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-Devel devel <emacs-devel@gnu.org>,
>>       Stuart Rackham <srackham@gmail.com>
>>
>> It works nicely for what it does. However unfortunately it is still
>> unusable since the binary version of docindexer currently does not let
>> you tell which files extensions it should index as text.
>
> In the directory where you installed docindexer, there's a file named
> conf.py, a piece of Python code that describes the docindexer parser
> configuration.  Its syntax should be self-explanatory; you can add
> entries there for whatever source files you'd like to index.

I finally got around to do this and tested. It works nicely (but
indexing take 15-20 minutes).

However I got another problem with docsearch. I want to do things like this

  docsearch ext:pdf ext:org -a first second .

If I use both ext:pdf and ext:org they should be OR-ed but I get no
hits then. Do you have any idea what is wrong?

Re: Indexed search with grep-like output

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Mon, 14 Mar 2011 01:14:30 +0100
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org, srackham@gmail.com
> 
> However I got another problem with docsearch. I want to do things like this
> 
>   docsearch ext:pdf ext:org -a first second .
> 
> If I use both ext:pdf and ext:org they should be OR-ed but I get no
> hits then. Do you have any idea what is wrong?

Maybe it's a bug in docsearch.  But why do you care in which documents
the information lives?  I never use the ext: query, precisely
_because_ I don't care and don't want to remember in what format the
info is stored and where.  Just let it search everything, the search
is lightning fast anyway.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Mar 14, 2011 at 12:30 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Mon, 14 Mar 2011 01:14:30 +0100
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org, srackham@gmail.com
>>
>> However I got another problem with docsearch. I want to do things like this
>>
>>   docsearch ext:pdf ext:org -a first second .
>>
>> If I use both ext:pdf and ext:org they should be OR-ed but I get no
>> hits then. Do you have any idea what is wrong?
>
> Maybe it's a bug in docsearch.  But why do you care in which documents
> the information lives?  I never use the ext: query, precisely
> _because_ I don't care and don't want to remember in what format the
> info is stored and where.  Just let it search everything, the search
> is lightning fast anyway.

Thanks. I just found another way to do they same query with Lucene
syntax (which docindexer uses).

It is sometimes easier to find what you want if you do not search all files.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Mon, Jan 3, 2011 at 5:10 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> I have now added support for Google Desktop Search too (the grepping
>> is not added there yet though). It feels snappier, but the timing
>> tells me it should be about the same as Windows Desktop Search. Wonder
>> if I did something wrong... ;-)
>
> Clearly, the wrong thing you did is focus on non-Free tools.

It now supports DocIndexer, Google Desktop Search and Windows Desktop
Search (the latter via a rather general SQL-interface using ruby).

'This is now in the nXhtml repository if someone wants to test. The
easiest way to test is just:

- Get a checkout of nXhtml
- Run autostart.el in the top directory.
- Then try the search from the menus: nXhtml / Tools / Indexed Search.

DocIndexer is the default search engine unless you are on w32 where
Windows Desktop Search is default (since it is there by default). You
will need to tell where DocIndexer is, but you will get instructions
for that.

If you want to change search engine then change the variable
idxsearch-engine in idxsearch.el.

Re: Indexed search with grep-like output

[Stefan Monnier]

>> Clearly, the wrong thing you did is focus on non-Free tools.
> It now supports DocIndexer, Google Desktop Search and Windows Desktop
> Search (the latter via a rather general SQL-interface using ruby).

I recommend you submit it for inclusion in elpa.gnu.org.


        Stefan

Re: Indexed search with grep-like output

[Lennart Borgman]

On Fri, Mar 18, 2011 at 3:32 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>> Clearly, the wrong thing you did is focus on non-Free tools.
>> It now supports DocIndexer, Google Desktop Search and Windows Desktop
>> Search (the latter via a rather general SQL-interface using ruby).
>
> I recommend you submit it for inclusion in elpa.gnu.org.

I have not yet found out how to handle that.

Remember that nXhtml also includes a kind of packaging system. It can
download packages automatically as you need them. That allows for easy
integration, i.e. you can have things in menus without having them
installed. A not yet downloaded function can be called etc.

Since elpa can't do that yet I am not sure what to do.

Re: Indexed search with grep-like output

[Ted Zlatanov]

On Fri, 18 Mar 2011 11:55:54 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> On Fri, Mar 18, 2011 at 3:32 AM, Stefan Monnier
LB> <monnier@iro.umontreal.ca> wrote:
>>>> Clearly, the wrong thing you did is focus on non-Free tools.
>>> It now supports DocIndexer, Google Desktop Search and Windows Desktop
>>> Search (the latter via a rather general SQL-interface using ruby).
>> 
>> I recommend you submit it for inclusion in elpa.gnu.org.

LB> I have not yet found out how to handle that.

I'm sure Chong Yidong and I posted an answer to that question at least
twice recently.  I can do the Gmane search if you don't see it.

LB> Remember that nXhtml also includes a kind of packaging system. It can
LB> download packages automatically as you need them. That allows for easy
LB> integration, i.e. you can have things in menus without having them
LB> installed. A not yet downloaded function can be called etc.

LB> Since elpa can't do that yet I am not sure what to do.

I don't think ELPA itself has a problem with a package that downloads
things (I assume it uses featurep to avoid downloading things already in
the load-path).  nXhtml *could* use package.el functions to see if
package X is already available that way.

Ted

Re: Indexed search with grep-like output

[Stefan Monnier]

LB> Remember that nXhtml also includes a kind of packaging system.  It can
LB> download packages automatically as you need them.  That allows for easy
LB> integration, i.e. you can have things in menus without having them
LB> installed.  A not yet downloaded function can be called etc.

Doesn't seem relevant: I only talked about idxsearch.


        Stefan

Re: Indexed search with grep-like output

[Chong Yidong]

Lennart Borgman <lennart.borgman@gmail.com> writes:

>> I recommend you submit it for inclusion in elpa.gnu.org.
>
> I have not yet found out how to handle that.

Are the copyright assignments in order?  I asked you about that several
weeks ago, and the last I heard was that you were checking.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Sat, Mar 19, 2011 at 9:52 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>>> I recommend you submit it for inclusion in elpa.gnu.org.
>>
>> I have not yet found out how to handle that.
>
> Are the copyright assignments in order?  I asked you about that several
> weeks ago, and the last I heard was that you were checking.


Stefan replied in that thread that since I have signed papers for
Emacs then my part of nXhtml is in order too.


I just did a quick lock to see what files there are in nXhtml that I
have not written (or not written all of):

* alts subdir:
There is nothing I wrote (except an old copy of re-builder that I
forgotten to delete). Though I might have made some changes.


* etc subdir:
Here is a lot of things I did not wrote/create. There are for example
images (used by n-back) that I did not make, but that are free.

marktree.js is not mine (it is under an mit license).

The schema subdir is a bit troublesome and does not work as it should.
And the schemas are not mine. Jimmy Yuen Ho Wong contributed some of
them, see https://bugs.launchpad.net/nxhtml/+bug/409183

The wds subdir contains support for indexed search querying an sql
engine. trollop.rb is not mine. idxsearch.rb is basically mine (I do
not think there is anything left of what I started with).

viper-tut is a rewrite of some files from vi tutorial. I think they are free.


* nxhtml subdir:
** outline-magic.el: Carsten Dominique so it should not be a problem
** tidy-xhtml.el: From Kahlil Hodgson, with some changes by me. I do
not know if he has signed papers.


* related subdir:
** amrita.el: Roger Mason <rmason@mun.ca>. I do not know if he has
signed papers.
** csharp-mode.el: Dino Chiesa. I do not know if he has signed papers.
(And any way this should not be seen as part of nXhtml.)
** flymake*.el: All these are written by me after some ideas from
other places (mainly EmacsWiki). It implements a structure for taking
care of flymake for different major modes. This should be moved into
Emacs itself.
** moz.el: Massimiliana Mirra. I do not know if he has signed papers.
(And this should be coordinated with mozrepl.)
** php-imenu.el: Marcel Cary. I do not know if he has signed papers.
** smarty-mode.el: Vincent Debout. I do not know if he has signed papers.
** tt-mode.el: Dave Cross. I do not know if he has signed papers.
** visual-basic-mode.el: A lot of people. I do not know if they have
signed papers.


* util subdir:
** anchored-transpose.el: Originally by Rick Bielawski. I do not know
if he has signed papers.
** css-color.el, css-palette.el: Niels Giesen. We are discussing that elsewhere.
** edit-server.el: A lot of people might have contributed, not sure.
** ffip.el: Extracted from rinari and changed a bit. There is probably
something better in rinari now to use. (This is used to find a file in
a "site", which is essentially a directory tree.)
** fold-dwim.el: Peter Heslin. No idea if he has signed.
** gpl.el: Niels Giesen, see above.
** http-post-simple.el: Tom Schutzer-Weissmann. No idea if he has
signed. This file/functionality is so important so it should be in
Emacs IMO.
** org-odt.el: Just an old copy. Should be on its way into Emacs.
** pointback.el: Either this or for example winpoint.el (which I am
now using) or rebound.el should IMO be in Emacs. I have suggested to
the author of rebound.el to try to make this happen.
** powershell-mode.el: Frederic Perrin. I do not know if he has signed.
** rnc-mode.el: David Rosenberg. I do not know if he has signed.
** vline.el: Taiki Sugawara. I do not know if he has signed.
** zencoding.el: Chris Done. I do not know if he has signed.

Re: Indexed search with grep-like output

[Jambunathan K]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Sat, Mar 19, 2011 at 9:52 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
>> Lennart Borgman <lennart.borgman@gmail.com> writes:
>>
>>>> I recommend you submit it for inclusion in elpa.gnu.org.
>>>
>>> I have not yet found out how to handle that.
>>
>> Are the copyright assignments in order?  I asked you about that several
>> weeks ago, and the last I heard was that you were checking.
>

> ** org-odt.el: Just an old copy. Should be on its way into Emacs.

org-odt.el is by me. I have signed my Emacs papers. 

I am working with Bastien on getting this file in to org-mode and Emacs
proper.

Jambunathan K.

-- 

Re: Indexed search with grep-like output

[Chong Yidong]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> since I have signed papers for Emacs then my part of nXhtml is in
> order too.

Good.  Could you send an email to fsf-records@gnu.org stating that your
assignment should cover nXhtml, and change the copyright holder in the
copyright header to the FSF?

> I just did a quick lock to see what files there are in nXhtml that I
> have not written (or not written all of):

If the images and schema are released under free licenses, that is
acceptable.

If I understand your directory layout correctly, the files in nxhtml/
should be put into one single package, and the mumamo stuff in another
package.

Everything that nxhtml/*.el strictly needs should be placed in a
separate package; let's consider those pieces some other time.  So, we
will leave out everything in alts/, related/, and most of util/ for now.
Will this work?


As for your questions about contributors:

> ** tidy-xhtml.el: From Kahlil Hodgson, with some changes by me. I do
> not know if he has signed papers.

He has not.

> ** amrita.el: Roger Mason <rmason@mun.ca>. I do not know if he has
> signed papers.

No.

> ** csharp-mode.el: Dino Chiesa. I do not know if he has signed papers.
> (And any way this should not be seen as part of nXhtml.)

No.

> ** flymake*.el: All these are written by me after some ideas from
> other places (mainly EmacsWiki). It implements a structure for taking
> care of flymake for different major modes. This should be moved into
> Emacs itself.

Let's consider nxhtml for now.  Please submit these changes separately.

> ** moz.el: Massimiliana Mirra. I do not know if he has signed papers.
> (And this should be coordinated with mozrepl.)

He has not.

> ** php-imenu.el: Marcel Cary. I do not know if he has signed papers.

No.

> ** smarty-mode.el: Vincent Debout. I do not know if he has signed papers.

No.

> ** tt-mode.el: Dave Cross. I do not know if he has signed papers.

No.

> ** anchored-transpose.el: Originally by Rick Bielawski. I do not know
> if he has signed papers.

No.

> ** fold-dwim.el: Peter Heslin. No idea if he has signed.

He has.

> ** http-post-simple.el: Tom Schutzer-Weissmann.

No.

> ** powershell-mode.el: Frederic Perrin. I do not know if he has signed.

No.

> ** rnc-mode.el: David Rosenberg. I do not know if he has signed.

No.

> ** vline.el: Taiki Sugawara. I do not know if he has signed.

No.

> ** zencoding.el: Chris Done. I do not know if he has signed.

No.

Re: Indexed search with grep-like output

[Lennart Borgman]

On Thu, Mar 24, 2011 at 7:24 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
>
>> ** csharp-mode.el: Dino Chiesa. I do not know if he has signed papers.
>> (And any way this should not be seen as part of nXhtml.)
>
> No.

I forgot that Dylan R. E. Moonfire was the original author. Has he
signed papers for Emacs?

(Maybe csharp-mode.el is distributed another way now?)

Re: Indexed search with grep-like output

[Lennart Borgman]

On Thu, Mar 24, 2011 at 7:24 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
>
>> ** tidy-xhtml.el: From Kahlil Hodgson, with some changes by me. I do
>> not know if he has signed papers.
>
> He has not.

The mail address dorge@tpg.com.au for Kahlil does not work any more.
Does anyone know a new mail address for him?

Re: Indexed search with grep-like output

[Lennart Borgman]

On Thu, Mar 24, 2011 at 7:24 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
>
>> ** php-imenu.el: Marcel Cary. I do not know if he has signed papers.
>
> No.

The mail address Marcel Cary <marcel-cary of care2.com> does not work.
Does anyone has a better mail address for Marcel?

Re: Indexed search with grep-like output

[Lennart Borgman]

On Thu, Mar 24, 2011 at 7:24 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
>
>> ** http-post-simple.el: Tom Schutzer-Weissmann.
>
> No.

The mail address tom@schutzer-weissmann.net does not work. Anyone has
a better mail address?

(BTW this file should go into core Emacs in my opinion.)

Re: Indexed search with grep-like output

[Lennart Borgman]

On Thu, Mar 24, 2011 at 7:24 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> since I have signed papers for Emacs then my part of nXhtml is in
>> order too.
>
> Good.  Could you send an email to fsf-records@gnu.org stating that your
> assignment should cover nXhtml, and change the copyright holder in the
> copyright header to the FSF?

This is fixed.

>> I just did a quick lock to see what files there are in nXhtml that I
>> have not written (or not written all of):
>
> If the images and schema are released under free licenses, that is
> acceptable.

Good.

> If I understand your directory layout correctly, the files in nxhtml/
> should be put into one single package, and the mumamo stuff in another
> package.

Some parts of the mumamo stuff is in nxhtml/.

> Everything that nxhtml/*.el strictly needs should be placed in a
> separate package; let's consider those pieces some other time.  So, we
> will leave out everything in alts/, related/, and most of util/ for now.
> Will this work?

Maybe. I am a bit worried about keeping things working during the transition.

> As for your questions about contributors:

I try to summarize below. I have tried to contact all this
contributors. I doubt that those I have been able to send to that has
not answered will sign papers.

>> ** tidy-xhtml.el: From Kahlil Hodgson, with some changes by me. I do
>> not know if he has signed papers.
>
> He has not.

We need to find a mail address to him. (tidy-xhtml.el is useful, but
not a central part really. Users who want it could install it later.)

>> ** amrita.el: Roger Mason <rmason@mun.ca>. I do not know if he has
>> signed papers.
>
> No.

He will probably sign papers.

>> ** csharp-mode.el: Dino Chiesa. I do not know if he has signed papers.
>> (And any way this should not be seen as part of nXhtml.)
>
> No.

No answer (yet) from Dino. No big problem, csharp-mode is just used as
a submode for C# by mumamo. The user who wants it can install it.

>> ** flymake*.el: All these are written by me after some ideas from
>> other places (mainly EmacsWiki). It implements a structure for taking
>> care of flymake for different major modes. This should be moved into
>> Emacs itself.
>
> Let's consider nxhtml for now.  Please submit these changes separately.

Ok.

>> ** moz.el: Massimiliana Mirra. I do not know if he has signed papers.
>> (And this should be coordinated with mozrepl.)
>
> He has not.

I am not sure my mail has reached him yet. (This is not a big problem
either. It is an experimental part that maybe will stop working
because of changes in Firefox.)

>> ** php-imenu.el: Marcel Cary. I do not know if he has signed papers.
>
> No.

We need a mail address to him. (No big problem. php-mode.el is however.)

>> ** smarty-mode.el: Vincent Debout. I do not know if he has signed papers.
>
> No.

He will probably sign papers.

>> ** tt-mode.el: Dave Cross. I do not know if he has signed papers.
>
> No.

It looks like Dave Cross may not sign papers. The situation is similar
to that for csharp-mode.el (see above) so it is not a really big
problem.

>> ** anchored-transpose.el: Originally by Rick Bielawski. I do not know
>> if he has signed papers.
>
> No.

He will probably sign papers.

>> ** fold-dwim.el: Peter Heslin. No idea if he has signed.
>
> He has.
>
>> ** http-post-simple.el: Tom Schutzer-Weissmann.
>
> No.

We need a mail address to Tom. http-post-simple.el is very useful and
should in my opionon be in Emacs core.

>> ** powershell-mode.el: Frederic Perrin. I do not know if he has signed.
>
> No.

No answer (yet) from Perrin. This is no problem since
powershell-mode.el just was an optional test. It is not required and I
can remove it.

>> ** rnc-mode.el: David Rosenberg. I do not know if he has signed.
>
> No.

I hope he will sign.

>> ** vline.el: Taiki Sugawara. I do not know if he has signed.
>
> No.

He will probably sign papers.

>> ** zencoding.el: Chris Done. I do not know if he has signed.
>
> No.

There is one more contributor to this, Ron Panduwana. Both will
probably sign papers.

Re: Indexed search with grep-like output

[Stefan Monnier]

>>> ** powershell-mode.el: Frederic Perrin. I do not know if he has signed.
>> No.
> No answer (yet) from Perrin.  This is no problem since
> powershell-mode.el just was an optional test.  It is not required and
> I can remove it.

Especially since I don't think it should be in Emacs, given that it
provides support for proprietary-only software.  To the extent that
PowerShell is part of the base Windows OS, maybe we could make an
exception to this rule, but at least it's a negative point in
the balance.


        Stefan

Re: Indexed search with grep-like output

[Lennart Borgman]

On Fri, Apr 29, 2011 at 6:10 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>>> ** powershell-mode.el: Frederic Perrin. I do not know if he has signed.
>>> No.
>> No answer (yet) from Perrin.  This is no problem since
>> powershell-mode.el just was an optional test.  It is not required and
>> I can remove it.
>
> Especially since I don't think it should be in Emacs, given that it
> provides support for proprietary-only software.  To the extent that
> PowerShell is part of the base Windows OS, maybe we could make an
> exception to this rule, but at least it's a negative point in
> the balance.

I am not very positive to it either, but it can be an entrance to
Emacs and train for other OS:es.