RFC - cleaning up /etc

RFC - cleaning up /etc

[Eric S. Raymond]

This is an issue completely orthogonal to the git transition, but
also motivated by my concern that the Emacs codebase and project raise 
needless barriers to contribution.

/etc is a mess.  Over the years it's become a dumping ground for
miscellany.  There are no fewer that four different files of
miscellaneous jokes! And a cookie recipe! And...stuff.

I think this matters because newbies who try to browse it get lost.
Important things like the GNU Manifesto (yes, I agree it should 
be treated as important despite my well-known disagreements with it)
are half-submerged in trivia, clutter, and files that only programs
rather than humans care about.

I propose to fix this with a reorganization.  /etc should be the data
directory for files used by Lisp and programs in the suite.  The files
meant exclusively for human browsing - the jokes, the cookie recipe,
the Manifesto, the hacking guides - should move to a new directory
designated for such content.

That directory should have a proper README of it own mapping the
content, so  newbies can quickly grasp what is there and what they
might want to look at.

Discuss.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

"The power to tax involves the power to destroy;...the power to
destroy may defeat and render useless the power to create...."
	-- Chief Justice John Marshall, 1819.

Re: RFC - cleaning up /etc

[Samuel El-Borai]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

Has someone tried the cookie recipe?


2014/1/9 Eric S. Raymond <esr@thyrsus.com>

> This is an issue completely orthogonal to the git transition, but
> also motivated by my concern that the Emacs codebase and project raise
> needless barriers to contribution.
>
> /etc is a mess.  Over the years it's become a dumping ground for
> miscellany.  There are no fewer that four different files of
> miscellaneous jokes! And a cookie recipe! And...stuff.
>
> I think this matters because newbies who try to browse it get lost.
> Important things like the GNU Manifesto (yes, I agree it should
> be treated as important despite my well-known disagreements with it)
> are half-submerged in trivia, clutter, and files that only programs
> rather than humans care about.
>
> I propose to fix this with a reorganization.  /etc should be the data
> directory for files used by Lisp and programs in the suite.  The files
> meant exclusively for human browsing - the jokes, the cookie recipe,
> the Manifesto, the hacking guides - should move to a new directory
> designated for such content.
>
> That directory should have a proper README of it own mapping the
> content, so  newbies can quickly grasp what is there and what they
> might want to look at.
>
> Discuss.
> --
>                 <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
>
> "The power to tax involves the power to destroy;...the power to
> destroy may defeat and render useless the power to create...."
>         -- Chief Justice John Marshall, 1819.
>
>

[-- Attachment #2: Type: text/html, Size: 2082 bytes --]

Re: RFC - cleaning up /etc

[Samuel El-Borai]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

I don't know if it's The True Probleme that, if solved will bring to emacs
a whole new batch of serious developers ready to spend their life to
improve it.
I don't think it is the case.

But I'm with you on this issue. It's would be great to leave the jokes
outside the "real" data.


2014/1/9 Eric S. Raymond <esr@thyrsus.com>

> This is an issue completely orthogonal to the git transition, but
> also motivated by my concern that the Emacs codebase and project raise
> needless barriers to contribution.
>
> /etc is a mess.  Over the years it's become a dumping ground for
> miscellany.  There are no fewer that four different files of
> miscellaneous jokes! And a cookie recipe! And...stuff.
>
> I think this matters because newbies who try to browse it get lost.
> Important things like the GNU Manifesto (yes, I agree it should
> be treated as important despite my well-known disagreements with it)
> are half-submerged in trivia, clutter, and files that only programs
> rather than humans care about.
>
> I propose to fix this with a reorganization.  /etc should be the data
> directory for files used by Lisp and programs in the suite.  The files
> meant exclusively for human browsing - the jokes, the cookie recipe,
> the Manifesto, the hacking guides - should move to a new directory
> designated for such content.
>
> That directory should have a proper README of it own mapping the
> content, so  newbies can quickly grasp what is there and what they
> might want to look at.
>
> Discuss.
> --
>                 <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
>
> "The power to tax involves the power to destroy;...the power to
> destroy may defeat and render useless the power to create...."
>         -- Chief Justice John Marshall, 1819.
>
>

[-- Attachment #2: Type: text/html, Size: 2392 bytes --]

Re: RFC - cleaning up /etc

[Glenn Morris]

Eric S. Raymond wrote:

> I propose to fix this with a reorganization.  /etc should be the data
> directory for files used by Lisp and programs in the suite.  The files
> meant exclusively for human browsing - the jokes, the cookie recipe,
> the Manifesto, the hacking guides - should move to a new directory
> designated for such content.

Does not seem like a good idea to me.

Re: RFC - cleaning up /etc

[Eric S. Raymond]

Glenn Morris <rgm@gnu.org>:
> Eric S. Raymond wrote:
> 
> > I propose to fix this with a reorganization.  /etc should be the data
> > directory for files used by Lisp and programs in the suite.  The files
> > meant exclusively for human browsing - the jokes, the cookie recipe,
> > the Manifesto, the hacking guides - should move to a new directory
> > designated for such content.
> 
> Does not seem like a good idea to me.

That's not a useful response unless accompanied by an explanation.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: RFC - cleaning up /etc

[Glenn Morris]

"Eric S. Raymond" wrote:

>> Does not seem like a good idea to me.
>
> That's not a useful response unless accompanied by an explanation.

It seems necessary to respond quickly these days.

Anything in etc/ that is important is accessed via Emacs commands/menus.
M-x view-emacs-news, describe-no-warranty, describe-gnu-project, etc.
Moving the file does nothing but break any external references that
may exist.

The only reason that files like ORDERS still exist is for historical
reasons, in case of some external reference pointing to them.

Anything in there that is not important (COOKIES etc) is frankly only
there for sentimental historical reasons, because people will kick up
a fuss if they get removed. Moving it eg to an etc/misc subdirectory
is pointless IMO. And etc/jokes would just be leaden.
And an entire new top-level directory is an awful idea.

Basically, if a file is not pointless, there'll be an external
reference to it somewhere. If it is pointless, it does not matter
where it lives. So overall, better to leave alone IMO.

And again: not during a feature freeze, please.

Re: RFC - cleaning up /etc

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I propose to fix this with a reorganization.  /etc should be the data
    directory for files used by Lisp and programs in the suite.  The files
    meant exclusively for human browsing - the jokes, the cookie recipe,
    the Manifesto, the hacking guides - should move to a new directory
    designated for such content.

The idea makes sense to me, in principle.
I can also pick some old files to delete or move outside Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: RFC - cleaning up /etc

[Eric S. Raymond]

Richard Stallman <rms@gnu.org>:
>     I propose to fix this with a reorganization.  /etc should be the data
>     directory for files used by Lisp and programs in the suite.  The files
>     meant exclusively for human browsing - the jokes, the cookie recipe,
>     the Manifesto, the hacking guides - should move to a new directory
>     designated for such content.
> 
> The idea makes sense to me, in principle.
> I can also pick some old files to delete or move outside Emacs.

Please do that.  Meanwhile, I'll try to consolidate material now scattered
across multiple files where it makes sense to do so.  The miscellaneous
humor files are an obvious target for this.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: RFC - cleaning up /etc

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Please do that.  Meanwhile, I'll try to consolidate material now scattered
    across multiple files where it makes sense to do so.  The miscellaneous
    humor files are an obvious target for this.

Be careful about that!  Some of the humor files are in separate files
because programs refer to them.  I suggest you wait for me to identify
some to delete, before working on changes.

I suggest deleting these files, which were put here before we had a web site.

COOKIES
copying.paper
echo.msg
INTERVIEW
JOKES (but save the EMACS acronyms)
MAILINGLISTS
MOTIVATION
publicsuffix.txt
SERVICE

We can replace copying.paper, MAILINGLISTS and SERVICE by URLs.

We can also delete celebacy.1 and sex.6,
as well as the program meese.el which uses them.
I doubt many users will remember Attorney General Meese.

condom.1 can also be deleted; it is not actually related to those
two files, but without them, why put it here?
(Though the reference to setiud is funny.)


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: RFC - cleaning up /etc

[Eric S. Raymond]

Richard Stallman <rms@gnu.org>:
> Be careful about that!  Some of the humor files are in separate files
> because programs refer to them.

I'm well aware of this. Grepping in the elisp to make sure no dangling
references would be left behind was absolutely part of the procedure I
had in mind.

> I suggest deleting these files, which were put here before we had a web site.
> 
> COOKIES
> copying.paper
> echo.msg
> INTERVIEW
> JOKES (but save the EMACS acronyms)
> MAILINGLISTS
> MOTIVATION
> publicsuffix.txt
> SERVICE
> 
> We can replace copying.paper, MAILINGLISTS and SERVICE by URLs.
> 
> We can also delete celebacy.1 and sex.6,
> as well as the program meese.el which uses them.
> I doubt many users will remember Attorney General Meese.
> 
> condom.1 can also be deleted; it is not actually related to those
> two files, but without them, why put it here?
> (Though the reference to setiud is funny.)

What shall we do about future-bug?  I think that is much too funny to be
lost to posterity (take a bow, kfogel) but if we're going to exile JOKES
it should go too. 

I guess a lot of this stuff can go to the EmacsWiki humor section.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: RFC - cleaning up /etc

[Ulrich Mueller]

>>>>> On Sat, 11 Jan 2014, Richard Stallman wrote:

> We can also delete celebacy.1 and sex.6, as well as the program
> meese.el which uses them. I doubt many users will remember Attorney
> General Meese.

> condom.1 can also be deleted; it is not actually related to those
> two files, but without them, why put it here? (Though the reference
> to setiud is funny.)

If anyone misses them, these "man pages" (as well as echo.msg) are
also distributed via the funny-manpages package in some GNU/Linux
distros, e.g. Gentoo and Debian.

Ulrich

Re: RFC - cleaning up /etc

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I guess a lot of this stuff can go to the EmacsWiki humor section.

Ok with me.

    What shall we do about future-bug?  I think that is much too funny to be
    lost to posterity (take a bow, kfogel) but if we're going to exile JOKES
    it should go too. 

We don't have to exile ALL humor.  We can keep future-bug
and the acronyms for Emacs.  Those are very specificially related
to Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: RFC - cleaning up /etc

[Glenn Morris]

Richard Stallman wrote:

> MAILINGLISTS

Referenced in doc/.
What URL replaces it?

> publicsuffix.txt

What? This is used by lisp/url/url-domsuf.el.


How about obsoleting etc/GNU in favour of a pointer to
Info node `(emacs)Maninfesto'? It is identical content.

Re: RFC - cleaning up /etc

[Eric S. Raymond]

Glenn Morris <rgm@gnu.org>:
> > publicsuffix.txt
> 
> What? This is used by lisp/url/url-domsuf.el.

The deletion needs to be reverted, then.  I'll put it on my list.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: RFC - cleaning up /etc

[Eli Zaretskii]

> Date: Sat, 11 Jan 2014 15:59:25 -0500
> From: "Eric S. Raymond" <esr@thyrsus.com>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> Glenn Morris <rgm@gnu.org>:
> > > publicsuffix.txt
> > 
> > What? This is used by lisp/url/url-domsuf.el.
> 
> The deletion needs to be reverted, then.  I'll put it on my list.

I've done that.

Re: RFC - cleaning up /etc

[Eric S. Raymond]

Eli Zaretskii <eliz@gnu.org>:
> > Date: Sat, 11 Jan 2014 15:59:25 -0500
> > From: "Eric S. Raymond" <esr@thyrsus.com>
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> > 
> > Glenn Morris <rgm@gnu.org>:
> > > > publicsuffix.txt
> > > 
> > > What? This is used by lisp/url/url-domsuf.el.
> > 
> > The deletion needs to be reverted, then.  I'll put it on my list.
> 
> I've done that.

Thanks, removed from my list.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: RFC - cleaning up /etc

[Lars Magne Ingebrigtsen]

Uhm -- I thought we were in a feature freeze.  I.e., we should
concentrate on fixing bugs, and not do lots of non-bug-related changes.

Removing things from etc might be nice, but it doesn't seem to have
anything to do with fixing bugs.

Stefan, please clarify.

As for moving to git, I'm for it, even though git is inferior to bzr in
a few ways (no bound branches, totally icky sha revisions instead of
human-parseable revnos, a `C-x v g' that's so slow it's unusable (but
David is fixing that), worse vc-mode support).

But isn't all this also taking all focus from doing meaningful bug work?
How about waiting until the next feature period opens?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: RFC - cleaning up /etc

[Eric S. Raymond]

Lars Magne Ingebrigtsen <larsi@gnus.org>:
> Uhm -- I thought we were in a feature freeze.  I.e., we should
> concentrate on fixing bugs, and not do lots of non-bug-related changes.

That's true off in the code part of the universe.  All these changes
barely touch any of that.

> But isn't all this also taking all focus from doing meaningful bug work?

I wouldn't fixing bugs anyway, outside of VC.  I'll need to learn my way
back inside the rest of the codebase for that.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: RFC - cleaning up /etc

[David Kastrup]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Uhm -- I thought we were in a feature freeze.  I.e., we should
> concentrate on fixing bugs, and not do lots of non-bug-related changes.
>
> Removing things from etc might be nice, but it doesn't seem to have
> anything to do with fixing bugs.

Yes, that sounds like after-release material.  Also quite unrelated to
the git move, so it might make sense for Eric to take it off his
post-release work list.  He can still return to it when Git and VC have
stopped biting him.

> Stefan, please clarify.
>
> As for moving to git, I'm for it, even though git is inferior to bzr in
> a few ways (no bound branches, totally icky sha revisions instead of
> human-parseable revnos,

Well, for a _distributed_ version control system, I prefer icky.
"human-parseable" is not compatible with "automatically unique without
requiring coordination".

> a `C-x v g' that's so slow it's unusable (but David is fixing that),

Will still take a year to trickle down to typical users.  Man, my fix
better be good with everybody taking its efficacy for granted.

> worse vc-mode support).

That sounds more like a hen-and-egg problem than anything else.  I mean,
the hen-and-egg situation was a main motivator for moving Emacs to bzr
at a time when it was actually painful to do so.

> But isn't all this also taking all focus from doing meaningful bug
> work?  How about waiting until the next feature period opens?

It's one of the sad realities of software development that alternating
"stabilization" and "feature" periods mean that if you only start
working on features when the "feature" period is there, you won't finish
before "stabilization" sets in.  The "feature" period is when _prepared_
features get committed.

-- 
David Kastrup

Re: RFC - cleaning up /etc

[Stefan Monnier]

> Stefan, please clarify.

100% agreement.


        Stefan

Re: RFC - cleaning up /etc

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 12 Jan 2014 01:37:37 +0100
> 
> > But isn't all this also taking all focus from doing meaningful bug
> > work?  How about waiting until the next feature period opens?
> 
> It's one of the sad realities of software development that alternating
> "stabilization" and "feature" periods mean that if you only start
> working on features when the "feature" period is there, you won't finish
> before "stabilization" sets in.  The "feature" period is when _prepared_
> features get committed.

There's always a separate branch, on which one could work without
affecting the mainline development.  No need to do everything on the
trunk.

Re: RFC - cleaning up /etc

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > MAILINGLISTS

    Referenced in doc/.
    What URL replaces it?

I don't know, but I'd expect there to be something in the savannah Emacs
project or in gnu.org/software/emacs.

If such a page doesn't exist, we could create one.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: RFC - cleaning up /etc

[Glenn Morris]

Richard Stallman wrote:

> I don't know, but I'd expect there to be something in the savannah Emacs
> project or in gnu.org/software/emacs.

I think lists.gnu.org and http://savannah.gnu.org/mail/?group=emacs
covers it.

Progress report on git-blame (was: RFC - cleaning up /etc)

[David Kastrup]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

David Kastrup <dak@gnu.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> a `C-x v g' that's so slow it's unusable (but David is fixing that),
>
> Will still take a year to trickle down to typical users.  Man, my fix
> better be good with everybody taking its efficacy for granted.

Taking the following test job:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: blametest.sh --]
[-- Type: text/x-sh, Size: 235 bytes --]

#/bin/sh

cd /tmp
rm -rf testit
mkdir testit
cd testit
git init
for inc in 8 4 3 7 2 9 5 6 1
do
    seq 252000 -$inc 0 > testfile
    git add testfile
    git commit -m "Run with increment $inc"
done
time git blame testfile >/dev/null

[-- Attachment #3: Type: text/plain, Size: 1194 bytes --]


my current code (which is not ready for submission as it does not
support all options of git-blame yet) takes 2 seconds for the git-blame
step as opposed to the 40 seconds the system binary does.  That's at
least encouraging and one can expect some of that to be pure output
time.

For blaming src/xdisp.c, however, it only drops the execution time from
3:00 minutes to about half.  30 seconds of that is system time, and it
is basically identical to before.

So while I have made good progress on the "lots of tiny changes"
behavior, the "so slow it's unusable" angle for real-world files will
likely require meddling with when and how and why I/O is being done (or
conceivably memory allocation/deallocation).  And in that area, I've not
yet done or even profiled anything, so I have no clue how much can be
done.  It's possible that repository repacking could help independently.

At least having gotten the linear list operation thrashing out of the
way will help with further profiling.

Of course, the actual output is identical (or a progress report would
not have made much sense).  Top memory usage also is very much the same
(about 340MB for blaming src/xdisp.c).

-- 
David Kastrup

Re: Progress report on git-blame (was: RFC - cleaning up /etc)

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 25 Jan 2014 02:06:43 +0100
> 
> my current code (which is not ready for submission as it does not
> support all options of git-blame yet) takes 2 seconds for the git-blame
> step as opposed to the 40 seconds the system binary does.  That's at
> least encouraging and one can expect some of that to be pure output
> time.

Thank you for doing this.

Re: Progress report on git-blame

[David Engster]

David Kastrup writes:
> For blaming src/xdisp.c, however, it only drops the execution time from
> 3:00 minutes to about half.

"only"? That is pretty impressive in my book.

-David

Re: Progress report on git-blame

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sat, 25 Jan 2014 02:06:43 +0100
>> 
>> my current code (which is not ready for submission as it does not
>> support all options of git-blame yet) takes 2 seconds for the git-blame
>> step as opposed to the 40 seconds the system binary does.  That's at
>> least encouraging and one can expect some of that to be pure output
>> time.
>
> Thank you for doing this.

Well, I finally did "git gc --aggressive" on my Emacs repository and
tried again git-blame src/xdisp.c and the times got considerably _worse_
than before the packing (though the repository went to about a quarter
in size).  So after my fixes the run time in real use cases is probably
dominated by unpacking the various revisions, particularly with a
well-compressed repository.  The positive thing about it is that fixing
the blame algorithm will make the remaining culprits stand out more
prominently when profiling.

The downside, of course, is that I had not been banking on having to
rework more than the blame algorithm itself to move C-x v g into the
"tolerable" area.

[system version]
dak@lola:/usr/local/tmp/emacs$ time git blame src/xdisp.c >/dev/null

real	3m5.786s
user	2m21.168s
sys	0m43.956s

[my version]
dak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null

real	2m11.865s
user	1m13.372s
sys	0m57.656s

I am currently using commit time for prioritizing a breadth-first search
exhausting all later commits before going to previous revisions.  That
may result in worse object retainment patterns for the unpacking stage
while walking backwards in history.


Apart from trying to bully up general git performance and from trying to
get the unpacking to cache smarter, it would also be feasible to add a
"fast track" interface to git blame --interactive where Emacs sends
information about "currently viewed material corresponds to lines
xxx...yyy in the file" to the running git process which is used for
prioritizing the outstanding work.  That would be reasonably doable from
the git side, but of course it would require additional support from
vc-git.

I did that kind of thing in preview-latex (an in-document viewing mode
available in AUCTeX) for being able to work with large files and
four-figure amounts of included graphics while rendering all the
graphics with GhostScript on a 200MHz CPU.  It has sort of an
xroach-like effect where everything looks calm and finished until you
try scrolling around in the document.

While that stuff does not help with non-interactive use like M-x occur,
it should be useful even for incremental search as long as the search
concerns the _meat_ of the lines rather than the blame info.

But I definitely don't have the time to work on the Emacs component of
that sort of thing myself.

-- 
David Kastrup

Re: Progress report on git-blame

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> David Kastrup writes:
>> For blaming src/xdisp.c, however, it only drops the execution time from
>> 3:00 minutes to about half.
>
> "only"? That is pretty impressive in my book.

It's not a quantum leap, just a change from bad to bad.  So it will be
only moderately useful for pulling a "Hello GitHub, Microsoft,
plasticscm, beanstalk, [see the endorsement banner on
<URL:http://libgit2.github.com/>], if you want to use this code under a
license different from the core GPLv2 used in Git, like, say, in
libgit2's permissive licensing for binaries, fork over a cool 10k€"
number.  As my sole income is from working on free software, some extra
cash would have come in handy.

That was my basic self-justification for stalling on my work on GNU
LilyPond for several weeks.  Now I still need to finish the copy/move
detection parts of git-blame (while they are rarely used, the internals
they relied on are gone and their principal code has a #if 0/#endif
around it).

So collecting on the effort will not be a breeze but more like pulling
teeth and will likely not work at all.  As long as everybody will
_disable_ git-blame in web interfaces anyway, there's not much of a
_selling_ point to the code, never mind that it will be nice to have in
the canonical Git binaries under GPLv2.

-- 
David Kastrup

Re: Progress report on git-blame

[martin rudalics]

 > Well, I finally did "git gc --aggressive" on my Emacs repository and
 > tried again git-blame src/xdisp.c and the times got considerably _worse_
 > than before the packing (though the repository went to about a quarter
 > in size).  So after my fixes the run time in real use cases is probably
 > dominated by unpacking the various revisions, particularly with a
 > well-compressed repository.

Seems like the second law of thermodynamics still applies ...

martin

Re: Progress report on git-blame

[David Kastrup]

martin rudalics <rudalics@gmx.at> writes:

>> Well, I finally did "git gc --aggressive" on my Emacs repository and
>> tried again git-blame src/xdisp.c and the times got considerably _worse_
>> than before the packing (though the repository went to about a quarter
>> in size).  So after my fixes the run time in real use cases is probably
>> dominated by unpacking the various revisions, particularly with a
>> well-compressed repository.
>
> Seems like the second law of thermodynamics still applies ...

Yes, life is still easiest on the peanut galleries.

-- 
David Kastrup

Re: Progress report on git-blame

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> martin rudalics <rudalics@gmx.at> writes:
>
>>> Well, I finally did "git gc --aggressive" on my Emacs repository and
>>> tried again git-blame src/xdisp.c and the times got considerably _worse_
>>> than before the packing (though the repository went to about a quarter
>>> in size).  So after my fixes the run time in real use cases is probably
>>> dominated by unpacking the various revisions, particularly with a
>>> well-compressed repository.
>>
>> Seems like the second law of thermodynamics still applies ...
>
> Yes, life is still easiest on the peanut galleries.

At any rate, I forgot to set the comparison function on the governing
priority queue, turning it into a FIFO.  So the actual numbers on the
packed repository are not as disappointing as I first thought.

Here are the numbers, first the system version, then my own version:

dak@lola:/usr/local/tmp/emacs$ time git blame HEAD src/xdisp.c >/tmp/blame1

real	3m17.437s
user	2m23.676s
sys	0m52.560s
dak@lola:/usr/local/tmp/emacs$ time ../git/git blame HEAD src/xdisp.c >/tmp/blame2

real	1m24.628s
user	0m32.008s
sys	0m52.044s
dak@lola:/usr/local/tmp/emacs$ cmp /tmp/blame1 /tmp/blame2
dak@lola:/usr/local/tmp/emacs$ 

But it's obvious that significant further improvements will depend on
either or both
a) making git-blame respond to interactively determined requirements
(basically, reacting to expose events)
b) significantly decreasing the involved system time by employing a good
caching strategy
c) not letting the basic algorithm and the cached data progress
independently but try catering the workload order on what happens to be
in-memory at a given point of time.

c) is a very shady approach and would make debugging a nightmare.

At any rate, the user time already dropped by more than a factor of 4,
and I have not touched the inner work horse (the diff algorithm) at all,
just eliminated all the thrashing around it.  Now the diff algorithm
itself is a separate library by a separate author, with widespread use.
It would be rather optimistic to assume that there are significantly
low-hanging fruit to be found there.

Shrug.  I'll go see how I can reimplement the now defunct copy/move
detection not exercised without specific command line options in order
to get the current work into a shape where upstream can consider
accepting it.

-- 
David Kastrup

Re: Progress report on git-blame

[Lars Ingebrigtsen]

David Kastrup <dak@gnu.org> writes:

> Apart from trying to bully up general git performance and from trying to
> get the unpacking to cache smarter, it would also be feasible to add a
> "fast track" interface to git blame --interactive where Emacs sends
> information about "currently viewed material corresponds to lines
> xxx...yyy in the file" to the running git process which is used for
> prioritizing the outstanding work.

That sounds quite useful.  My common bug fixing use case is that I find
a function (or a bit of a function) that I think has a bug, and then I
`C-x v g' to find out who "who wrote this crap anyway!!!" (it usually
turns out to be myself), and then I try to see whether any of the lines
are suspiciously new and may have introduced a new bug.

So I'm usually just interested in a screenful of lines.  If we could
have a version of `C-x v g' that only does "blame" for the current
region, for instance, that would certainly fit my use case.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Progress report on git-blame

[David Kastrup]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Apart from trying to bully up general git performance and from trying to
>> get the unpacking to cache smarter, it would also be feasible to add a
>> "fast track" interface to git blame --interactive where Emacs sends
>> information about "currently viewed material corresponds to lines
>> xxx...yyy in the file" to the running git process which is used for
>> prioritizing the outstanding work.
>
> That sounds quite useful.  My common bug fixing use case is that I find
> a function (or a bit of a function) that I think has a bug, and then I
> `C-x v g' to find out who "who wrote this crap anyway!!!" (it usually
> turns out to be myself), and then I try to see whether any of the lines
> are suspiciously new and may have introduced a new bug.
>
> So I'm usually just interested in a screenful of lines.  If we could
> have a version of `C-x v g' that only does "blame" for the current
> region, for instance, that would certainly fit my use case.

It's worth noting that you already _can_ ask "git blame" for just a
region.  However, implementing this into a general update strategy will
mean that you'll usually have one git blame --incremental running for
the big picture, and one-shot git-blame instances for producing the
missing parts from the currently exposed area (probably sigstopping the
"big" process while the small updates run).  That means parallel work
that will ultimately go wasted.  It does have the advantage of working
with the currently available binaries.  A dynamically adjusted search
seems nicer.

-- 
David Kastrup

Re: Progress report on git-blame

[Óscar Fuentes]

Lars Ingebrigtsen <larsi@gnus.org> writes:

[snip]

> So I'm usually just interested in a screenful of lines.

Same here.

> If we could have a version of `C-x v g' that only does "blame" for the
> current region, for instance, that would certainly fit my use case.

Hmmm...

$ # ran this twice, for warming the cache:
$ time git blame -- src/xdisp.c > /dev/null

real    2m48.232s
user    2m45.233s
sys     0m2.829s


$ # `blame' on 20 lines starting at line 1000:
$ time git blame -L 1000,+20 -- src/xdisp.c > /dev/null

real    0m10.861s
user    0m10.296s
sys     0m0.547s


:-)


Some random data points:

$ time git blame -L 1000,+100 -- src/xdisp.c > /dev/null

real    0m11.034s
user    0m10.417s
sys     0m0.599s


$ time git blame -L 1000,+1000 -- src/xdisp.c > /dev/null

real    0m15.089s
user    0m14.297s
sys     0m0.767s


$ time git blame -L 1000,+10000 -- src/xdisp.c > /dev/null

real    1m5.144s
user    1m3.555s
sys     0m1.502s


It seems that there is not a lot of a difference among blaming 20 lines
and blaming 1000.

AFAIK current `blame' functionality on VC (and on git-blame.el) is based
on working on the whole file, so implementing partial blame would
require significant changes on how VC renders the `blame' output.

However, the speed gain is so much that IMO it is worth using M-x
compile "git blame -L ...." on the simple scenario you described.

Re: Progress report on git-blame

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Some random data points:
>
> $ time git blame -L 1000,+100 -- src/xdisp.c > /dev/null
>
> real    0m11.034s
> user    0m10.417s
> sys     0m0.599s
>
>
> $ time git blame -L 1000,+1000 -- src/xdisp.c > /dev/null
>
> real    0m15.089s
> user    0m14.297s
> sys     0m0.767s
>
>
> $ time git blame -L 1000,+10000 -- src/xdisp.c > /dev/null
>
> real    1m5.144s
> user    1m3.555s
> sys     0m1.502s
>
>
> It seems that there is not a lot of a difference among blaming 20 lines
> and blaming 1000.

It very much depends on _which_ lines.  Basically, lines forming a block
sticking together when doing a unified diff are not all that expensive
since they travel in a single data structure.  The worst are lots of
small changes introduced in different branches.

> AFAIK current `blame' functionality on VC (and on git-blame.el) is
> based on working on the whole file,

Sure?  If so, it maybe should at least be made to support git blame
--incremental usefully.

-- 
David Kastrup

Re: Progress report on git-blame

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 25 Jan 2014 19:59:50 +0100
> 
> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
> > Some random data points:
> >
> > $ time git blame -L 1000,+100 -- src/xdisp.c > /dev/null
> >
> > real    0m11.034s
> > user    0m10.417s
> > sys     0m0.599s
> >
> >
> > $ time git blame -L 1000,+1000 -- src/xdisp.c > /dev/null
> >
> > real    0m15.089s
> > user    0m14.297s
> > sys     0m0.767s

For the record, I get about 9 sec for the first of these and about 12
sec for the second.

> > It seems that there is not a lot of a difference among blaming 20 lines
> > and blaming 1000.
> 
> It very much depends on _which_ lines.  Basically, lines forming a block
> sticking together when doing a unified diff are not all that expensive
> since they travel in a single data structure.  The worst are lots of
> small changes introduced in different branches.

I tried this:

  time git blame -L 270,+1000 -- src/xdisp.c > /dev/null

(you will see that around those line numbers, there's a different
commit every couple of lines), and got

  real    0m11.484s
  user    0m0.015s
  sys     0m0.000s

A few more examples:

  $ time git blame -L 2523,+1000 -- src/xdisp.c > /dev/null

  real    0m15.000s
  user    0m0.015s
  sys     0m0.000s

  $ time git blame -L 9630,+1000 -- src/xdisp.c > /dev/null

  real    0m10.985s
  user    0m0.015s
  sys     0m0.015s

  $ time git blame -L 10997,+1000 -- src/xdisp.c > /dev/null

  real    0m15.172s
  user    0m0.015s
  sys     0m0.016s

So these times are fairly typical for xdisp.c.  Which is not
surprising, since xdisp.c sees a lot of changes, and thus there's
another commit every few lines almost anywhere you look.

Re: Progress report on git-blame

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

>> It seems that there is not a lot of a difference among blaming 20 lines
>> and blaming 1000.
>
> It very much depends on _which_ lines.  Basically, lines forming a block
> sticking together when doing a unified diff are not all that expensive
> since they travel in a single data structure.  The worst are lots of
> small changes introduced in different branches.

I suspected some effect like that, so before posting I tried other file
segments and most of them required the same time as the one showed, with
the rest requiring less time (7s vs 11s)

>> AFAIK current `blame' functionality on VC (and on git-blame.el) is
>> based on working on the whole file,
>
> Sure?

Not, really, but almost. :-) It seems quite clear that vc-annotate just
displays the output of the underlying VC tool.

> If so, it maybe should at least be made to support git blame
> --incremental usefully.

Of course --incremental can also apply to a line-range blame. From
trying a couple of times it seems that the most recently changed lines
were annotated first, so for Lars' case of checking if some line on the
range was recently changed, he would have the relevant info sooner (but
if all the lines are old, he would have to wait until some line is
annotated to know the fact; in that case there is no significant time
again over non-incremental annotation, so speeding up `git blame' is
worthy in any case.)

Re: Progress report on git-blame

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> annotated to know the fact; in that case there is no significant time
> again over non-incremental annotation,

--incremental performs exactly the same work as the normal blame does.
It just outputs the blame info for a line range as soon as it knows that
it is final rather than writing out everything in linear order at the
end of the run.

-- 
David Kastrup

Re: Progress report on git-blame

[Stefan Monnier]

> That sounds quite useful.  My common bug fixing use case is that I find
> a function (or a bit of a function) that I think has a bug, and then I
> `C-x v g' to find out who "who wrote this crap anyway!!!" (it usually
> turns out to be myself), and then I try to see whether any of the lines
> are suspiciously new and may have introduced a new bug.

> So I'm usually just interested in a screenful of lines.  If we could
> have a version of `C-x v g' that only does "blame" for the current
> region, for instance, that would certainly fit my use case.

Indeed, as mentioned in the past a few times in various contexts,
I generally don't really want "git blame" (or its equivalent in other
VCS), instead I want "git log-and-diff FILE:NN-MM" which gives me the
history (hunks and accompanying log comments) of the lines between NN
and MM of FILE.

I use "git blame" to get each step of this history by hand, but it'd
be much better to get that history directly, which is probably cheaper
than a single "git blame".


        Stefan

Re: Progress report on git-blame

[Stefan Monnier]

> So I'm usually just interested in a screenful of lines.  If we could
> have a version of `C-x v g' that only does "blame" for the current
> region, for instance, that would certainly fit my use case.

One more thing: "git blame" is of no use when you're trying to see not
"who wrote these lines when" but "who removed (and when) the lines that
aren't there any more".  A "git log-and-diff" would handle that case
just fine, OTOH.


        Stefan

Re: Progress report on git-blame

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> One more thing: "git blame" is of no use when you're trying to see not
> "who wrote these lines when" but "who removed (and when) the lines that
> aren't there any more".  A "git log-and-diff" would handle that case
> just fine, OTOH.

When you know the content's of the line that was added/removed (or a
peculiar enough part of it) you can use `git log -S <text>':

$ git log -S chmod

or, restricted to a path:

$ git log -S chmod -- *.h

for viewing the diffs too:

$ git log -S chmod -p -- *.h

There are lots of possibilities there. See `git help log' and search for
`pickaxe'.

All those cases are supported by Magit.

Re: Progress report on git-blame

[Stefan Monnier]

> When you know the content's of the line that was added/removed (or a
> peculiar enough part of it) you can use `git log -S <text>':

Of course, usually I don't know, because the whole point of using "git
blame" is indeed to find the history of that chunk of text.


        Stefan

Re: Progress report on git-blame

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So I'm usually just interested in a screenful of lines.  If we could
>> have a version of `C-x v g' that only does "blame" for the current
>> region, for instance, that would certainly fit my use case.
>
> One more thing: "git blame" is of no use when you're trying to see not
> "who wrote these lines when" but "who removed (and when) the lines that
> aren't there any more".

That's what the --reverse option of git blame is for.

> A "git log-and-diff" would handle that case just fine, OTOH.

git log -S "literal string" filename

is also quite helpful (it just outputs the commits where the "literal
string" appears or disappears).

-- 
David Kastrup

Re: Progress report on git-blame

[Aneesh Kumar K.V]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> That sounds quite useful.  My common bug fixing use case is that I find
>> a function (or a bit of a function) that I think has a bug, and then I
>> `C-x v g' to find out who "who wrote this crap anyway!!!" (it usually
>> turns out to be myself), and then I try to see whether any of the lines
>> are suspiciously new and may have introduced a new bug.
>
>> So I'm usually just interested in a screenful of lines.  If we could
>> have a version of `C-x v g' that only does "blame" for the current
>> region, for instance, that would certainly fit my use case.
>
> Indeed, as mentioned in the past a few times in various contexts,
> I generally don't really want "git blame" (or its equivalent in other
> VCS), instead I want "git log-and-diff FILE:NN-MM" which gives me the
> history (hunks and accompanying log comments) of the lines between NN
> and MM of FILE.

newer version of git log do support that

http://git-scm.com/docs/git-log

-L <start>,<end>:<file>
-L :<regex>:<file>

    Trace the evolution of the line range given by "<start>,<end>" (or
    the funcname regex <regex>) within the <file>. You may not give any
    pathspec limiters. This is currently limited to a walk starting from
    a single revision, i.e., you may only give zero or one positive
    revision arguments. You can specify this option more than once.
    

-aneesh

Re: Progress report on git-blame

[Stefan Monnier]

>>> So I'm usually just interested in a screenful of lines.  If we could
>>> have a version of `C-x v g' that only does "blame" for the current
>>> region, for instance, that would certainly fit my use case.
>> One more thing: "git blame" is of no use when you're trying to see not
>> "who wrote these lines when" but "who removed (and when) the lines that
>> aren't there any more".
> That's what the --reverse option of git blame is for.

Never tried it.  Can it always be used (i.e. do you have to know
beforehand that you need it)?

>> A "git log-and-diff" would handle that case just fine, OTOH.
> git log -S "literal string" filename
> is also quite helpful (it just outputs the commits where the "literal
> string" appears or disappears).

Right, there are various workarounds.
But the info I want (and that I typically construct by hand by
iterating between vc-annotate, vc-print-log, and vc-diff) is
"vc-print-diff&log-of-region".  And AFAIK the VCS could find that info
at least as easily as (if not more easily) "git log -S" or "git blame".


        Stefan

Re: Progress report on git-blame

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 25 Jan 2014 20:48:15 -0500
> Cc: emacs-devel@gnu.org
> 
> > When you know the content's of the line that was added/removed (or a
> > peculiar enough part of it) you can use `git log -S <text>':
> 
> Of course, usually I don't know, because the whole point of using "git
> blame" is indeed to find the history of that chunk of text.

Perhaps we should define what it means "find the history of that chunk
of text".  I'm not sure we all talk about the same use case.

Re: Progress report on git-blame

[Stefan Monnier]

> Perhaps we should define what it means "find the history of that chunk
> of text".  I'm not sure we all talk about the same use case.

Find the sequence of patches (with their corresponding log comments)
that results in that chunk of text (presented in reverse-chronological
order).


        Stefan

Re: Progress report on git-blame

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Sun, 26 Jan 2014 14:05:17 -0500
> 
> > Perhaps we should define what it means "find the history of that chunk
> > of text".  I'm not sure we all talk about the same use case.
> 
> Find the sequence of patches (with their corresponding log comments)
> that results in that chunk of text (presented in reverse-chronological
> order).

How do you define "that chunk of text"?  For the initial (latest)
revision, the answer is clear.  But, as you go backwards in time, the
text becomes less and less similar to what it is today, and can also
move around.  So which revisions are relevant to "that chunk of text"
becomes harder and harder to decide.  What is today 10 lines could
once be only one, and could have been moved from some other place, or
constructed from several different parts, each one taken from a
different place.  Or something that is a function today might have
been just a code fragment in the past, part of some other function.

When we the humans look at the results of applying changes in reverse,
we can make that decision because we know what we are looking for:
some specific trait of the original code in which we are interested.
But how can a program know that?  How can you explain it what you are
interested in?

The usual pattern I apply 'annotate' goes like this:

  a) annotate foo.c, find the revision that changed "that chunk of
     text"

  b) examine the relevant parts of the diffs for that revision

  c) if the diffs don't show what I'm looking for, annotate foo.c as
     it was before that revision, and go back to b)

The crucial parts here are (1) to examine the diffs, and (2) decide
where are "the relevant parts" to look.  How can we do this
programmatically, so that the "history of that chunk of text" will
indeed show what we want it to?

Re: Progress report on git-blame

[Stefan Monnier]

> How do you define "that chunk of text"?  For the initial (latest)
> revision, the answer is clear.  But, as you go backwards in time, the
> text becomes less and less similar to what it is today, and can also
> move around.

Right, as you go backward, the region to consider grows.  But the
meaning is still clear.


        Stefan

Re: Progress report on git-blame

[Stefan Monnier]

> newer version of git log do support that

> http://git-scm.com/docs/git-log

> -L <start>,<end>:<file>
> -L :<regex>:<file>

>     Trace the evolution of the line range given by "<start>,<end>" (or
>     the funcname regex <regex>) within the <file>. You may not give any
>     pathspec limiters. This is currently limited to a walk starting from
>     a single revision, i.e., you may only give zero or one positive
>     revision arguments. You can specify this option more than once.

Oh indeed, I hadn't seen that.  Looks like exactly what I'm after.
Super way cool,


        Stefan "looking for a Git checkout where he can try it out"

Re: Progress report on git-blame

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> martin rudalics <rudalics@gmx.at> writes:
>>
>>>> Well, I finally did "git gc --aggressive" on my Emacs repository and
>>>> tried again git-blame src/xdisp.c and the times got considerably _worse_
>>>> than before the packing (though the repository went to about a quarter
>>>> in size).  So after my fixes the run time in real use cases is probably
>>>> dominated by unpacking the various revisions, particularly with a
>>>> well-compressed repository.
>>>
>>> Seems like the second law of thermodynamics still applies ...
>>
>> Yes, life is still easiest on the peanut galleries.
>
> At any rate, I forgot to set the comparison function on the governing
> priority queue, turning it into a FIFO.  So the actual numbers on the
> packed repository are not as disappointing as I first thought.
>
> Here are the numbers, first the system version, then my own version:
>
> dak@lola:/usr/local/tmp/emacs$ time git blame HEAD src/xdisp.c >/tmp/blame1
>
> real	3m17.437s
> user	2m23.676s
> sys	0m52.560s
> dak@lola:/usr/local/tmp/emacs$ time ../git/git blame HEAD src/xdisp.c >/tmp/blame2
>
> real	1m24.628s
> user	0m32.008s
> sys	0m52.044s
> dak@lola:/usr/local/tmp/emacs$ cmp /tmp/blame1 /tmp/blame2

Aaaand one can shave off about 50 seconds from either by doing

git config --add core.deltaBaseCacheLimit 128m

(assuming that one has more than about 512mb of main memory, of course).

Which gets us to
dak@lola:/usr/local/tmp/emacs$ time git blame src/xdisp.c >/dev/null

real	2m26.690s
user	2m12.148s
sys	0m13.772s

for the current Git code, and

dak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null

real	0m27.519s
user	0m20.248s
sys	0m7.156s

for my version of git blame.  So once one has bumped
core.deltaBaseCacheLimit to a value where Git is not constantly stepping
on its own feet with a heavily packed repository, the changed blame
program will buy a further factor of 5.

At any rate, there is no point in _not_ saving the 50 seconds for
bumping the deltaBaseCacheLimit right now.  Certainly _iff_ you did
git gc --aggressive and thus got a ridiculously expensive to unpack
repository.

-- 
David Kastrup