Include necessary git tools in the Emacs distribution?

Include necessary git tools in the Emacs distribution?

[Lars Magne Ingebrigtsen]

To use git for Emacs development, you really kinda need merge-changelog
and git-new-workdir.  The former is a C file also included in gnulib, so
I would assume that that's not a licensing problem.

git-new-workdir is a shell script, and I can't find the license after
looking around for ten seconds.

Anyway, I don't think it makes much sense to make Emacs developers hunt
around for these things themselves when it's something that's needed if
you don't want to go insane while hacking Emacs.

I mean, more insane. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Include necessary git tools in the Emacs distribution?

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> To use git for Emacs development, you really kinda need
> merge-changelog and git-new-workdir.  The former is a C file also
> included in gnulib, so I would assume that that's not a licensing
> problem.

Yeah, I had to clone gnulib specificly for that.

> git-new-workdir is a shell script, and I can't find the license after
> looking around for ten seconds.

Isn't that installed with Git out of the box (just not in PATH)?  (At
least here on Arch GNU/Linux it is.)

Bye,
Tassilo

Re: Include necessary git tools in the Emacs distribution?

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 14 Nov 2014 15:39:52 +0100
> 
> To use git for Emacs development, you really kinda need merge-changelog
> and git-new-workdir.  The former is a C file also included in gnulib, so
> I would assume that that's not a licensing problem.
> 
> git-new-workdir is a shell script, and I can't find the license after
> looking around for ten seconds.
> 
> Anyway, I don't think it makes much sense to make Emacs developers hunt
> around for these things themselves

Those who are on MS-Windows don't need to hunt for
git-merge-changelog farther than this:

  http://sourceforge.net/projects/ezwinports/files/git-merge-changelog-0.1-2-w32-bin.zip/download

As for git-new-workdir, it isn't included in msysgit, and I wonder
whether it's because it somehow relies on features not available on
Windows.

Re: Include necessary git tools in the Emacs distribution?

[Ulrich Mueller]

>>>>> On Fri, 14 Nov 2014, Lars Magne Ingebrigtsen wrote:

> To use git for Emacs development, you really kinda need
> merge-changelog and git-new-workdir. [...]

These are development tools not related to Emacs in particular, so why
should they be bundled with Emacs? (Reminds me of early versions which
bundled a C preprocessor ...)

> Anyway, I don't think it makes much sense to make Emacs developers
> hunt around for these things themselves when it's something that's
> needed if you don't want to go insane while hacking Emacs.

Gentoo has dev-vcs/git-merge-changelog as a package. However, it would
certainly help if it was packaged upstream, without the need to first
extract it from gnulib.

Ulrich

Re: Include necessary git tools in the Emacs distribution?

[Lars Magne Ingebrigtsen]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Fri, 14 Nov 2014, Lars Magne Ingebrigtsen wrote:
>
>> To use git for Emacs development, you really kinda need
>> merge-changelog and git-new-workdir. [...]
>
> These are development tools not related to Emacs in particular, so why
> should they be bundled with Emacs? (Reminds me of early versions which
> bundled a C preprocessor ...)

merge-changelog is needed to do anything efficiently in Emacs, and it's
are difficult to get at otherwise.  The latter isn't the case with C
preprocessors any more.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Include necessary git tools in the Emacs distribution?

[Ulrich Mueller]

>>>>> On Sat, 15 Nov 2014, Lars Magne Ingebrigtsen wrote:

> merge-changelog is needed to do anything efficiently in Emacs, and
> it's are difficult to get at otherwise.

I am looking at it from a distro point of view. Generally, distros are
not happy when packages are bundling and installing unrelated tools,
but prefer keeping separate things in separate packages. One reason
for this is that bundled stuff tends to cause file collisions between
packages.

Ulrich

Re: Include necessary git tools in the Emacs distribution?

[Lars Magne Ingebrigtsen]

Ulrich Mueller <ulm@gentoo.org> writes:

> I am looking at it from a distro point of view. Generally, distros are
> not happy when packages are bundling and installing unrelated tools,
> but prefer keeping separate things in separate packages. One reason
> for this is that bundled stuff tends to cause file collisions between
> packages.

I don't really follow.  This is about putting things necessary to do
Emacs maintenance into the Emacs repository.  The tools would not be
distributed with the Emacs package in any distribution.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Include necessary git tools in the Emacs distribution?

[Kelvin White]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>> lrich Mueller <ulm@gentoo.org> writes:
>> I am looking at it from a distro point of view. Generally, distros are
>> not happy when packages are bundling and installing unrelated tools,
>> but prefer keeping separate things in separate packages. One reason
>> for this is that bundled stuff tends to cause file collisions between
>> packages.
> I don't really follow.  This is about putting things necessary to do
> Emacs maintenance into the Emacs repository.  The tools would not be
> distributed with the Emacs package in any distribution.

Just to clarify they would only be available in Emacs repositories,
 not bundled with the next release. Correct?

[-- Attachment #2: Type: text/html, Size: 1352 bytes --]

Re: Include necessary git tools in the Emacs distribution?

[Lars Magne Ingebrigtsen]

Kelvin White <kwhite@gnu.org> writes:

> Just to clarify they would only be available in Emacs repositories,
> not bundled with the next release. Correct?

Yes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Include necessary git tools in the Emacs distribution?

[Ulrich Mueller]

>>>>> On Sat, 15 Nov 2014, Lars Magne Ingebrigtsen wrote:

> Kelvin White <kwhite@gnu.org> writes:
>> Just to clarify they would only be available in Emacs repositories,
>> not bundled with the next release. Correct?

> Yes.

Then it is fine, of course. The Subject seemed to indicate something
else, though.

Ulrich

Re: Include necessary git tools in the Emacs distribution?

[Glenn Morris]

I'm against such bundling on principle.
Just include a clear statement on where to get whatever.

Re: Include necessary git tools in the Emacs distribution?

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

On Sat, Nov 15, 2014 at 9:07 PM, Lars Magne Ingebrigtsen <larsi@gnus.org>
wrote:

> Ulrich Mueller <ulm@gentoo.org> writes:
>
> > I am looking at it from a distro point of view. Generally, distros are
> > not happy when packages are bundling and installing unrelated tools,
> > but prefer keeping separate things in separate packages. One reason
> > for this is that bundled stuff tends to cause file collisions between
> > packages.
>
> I don't really follow.  This is about putting things necessary to do
> Emacs maintenance into the Emacs repository.  The tools would not be
> distributed with the Emacs package in any distribution.
>


I would be glad for something like this:

"Emacs error: The program 'grep' is not available. Install it automatically
(yes/no)?:"

On all platforms, of course.

In case that is not possible I prefer bundling. ;-)

[-- Attachment #2: Type: text/html, Size: 1850 bytes --]

Re: Include necessary git tools in the Emacs distribution?

[Rüdiger Sonderfeld]

On Friday 14 November 2014 17:13:35 Ulrich Mueller wrote:
> Gentoo has dev-vcs/git-merge-changelog as a package. However, it would
> certainly help if it was packaged upstream, without the need to first
> extract it from gnulib.

I've created a bug (wishlist) report for it to be added to Debian's git-extras 
package:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734527

But so far (7 Jan 2014) no reply.  I guess the best solution to deal with it, 
would be to move it from Gnulib to git/contrib.  But that would require the 
Gnulib and git maintainers to work together and agree on the issue.

(Since it is licensed GPL2+, there should at least be no license issue.)

Regards,
Rüdiger

Re: Include necessary git tools in the Emacs distribution?

[Paul Eggert]

Rüdiger Sonderfeld wrote:
> I guess the best solution to deal with it,
> would be to move it from Gnulib to git/contrib.

An even better solution would be to generate the ChangeLog automatically from 
the git commit log, as is done in several other GNU projects.  This is on our 
list of things to do for Emacs.  Once it's done, Emacs shouldn't need 
git-merge-changelog.

Generating the ChangeLog files from the commit messages (was: Include necessary git tools in the Emacs distribution?)

[Stefan Monnier]

> An even better solution would be to generate the ChangeLog automatically
> from the git commit log, as is done in several other GNU projects.

Agreed.


        Stefan

Re: Generating the ChangeLog files from the commit messages

[Ted Zlatanov]

On Tue, 18 Nov 2014 09:45:44 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> An even better solution would be to generate the ChangeLog automatically
>> from the git commit log, as is done in several other GNU projects.

SM> Agreed.

What's blocking that?  Just code or are you/we waiting for something
after the Git migration?

Ted

Re: Generating the ChangeLog files from the commit messages

[Rüdiger Sonderfeld]

On Tuesday 18 November 2014 09:59:23 Ted Zlatanov wrote:
> On Tue, 18 Nov 2014 09:45:44 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> 
wrote:
> >> An even better solution would be to generate the ChangeLog automatically
> >> from the git commit log, as is done in several other GNU projects.
> 
> SM> Agreed.
> 
> What's blocking that?  Just code or are you/we waiting for something
> after the Git migration?

I guess it makes sense to wait until everything has settled from the Git 
migration.  Not introducing too many changes to the workflow at once.

So far it seems the major argument against generating ChangeLogs from commit 
messages was, that the files can be easily edited and fixed.  I'm not sure how 
this can be addressed.  Git's tools to edit commit messages (history) are 
sadly destructive and not transparent.  The support for git-notes(1) seems to 
be rather poor.  I guess we have to look at how other projects have solved 
this.

Anyway, there is a gitlog-to-changelog script in Gnulib as well.  We might be 
able to use that.

http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=build-aux/gitlog-to-changelog

Regards,
Rüdiger

Re: Generating the ChangeLog files from the commit messages

[Glenn Morris]

Pretend I posted

http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00637.html

again. Obviously no-one cares and this will happen anyway, but I reckon
enough years have passed I can register the token objection again.

I did a lot of merging between branches with bzr, and had basically zero
problems with the ChangeLogs. But no-one cares about that either; and
it's someone else's problem now.

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Tue, 18 Nov 2014 12:10:37 -0500
> 
> 
> Pretend I posted
> 
> http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00637.html
> 
> again.

This part:

 3) ChangeLogs can be edited to correct mistakes, commit logs cannot.

can be taken care of when producing ChangeLog files from commit
messages: the gitlog-to-changelog script has a facility to perform
replacements while it generates ChangeLog, so we could maintain a list
of mistake corrections for that purpose.

This part:

 4) I have the impression that having to write ChangeLogs leads to
 higher quality entries than just using commit logs would.

worries me as well, especially since I've seen a couple of projects
where the quality of log messages is very low (not sure if that
started when they tossed ChangeLogs).

> I did a lot of merging between branches with bzr, and had basically zero
> problems with the ChangeLogs.

The same is true when using git-merge-changelog with git.

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

On 11/18/2014 09:40 AM, Eli Zaretskii wrote:
> I've seen a couple of projects
> where the quality of log messages is very low

That can be a problem, but it's the same as the problem where the 
quality of ChangeLog files is very low.  The two problems are solvable 
in the same way, namely by asking developers to write good commit log 
messages / ChangeLog entries, just as Stefan asked Lars to do a few 
minutes ago

You can judge the results by looking at the coreutils ChangeLog file 
(the first few entries are attached).  This was automatically generated 
from coreutils' commit log.These ChangeLog entries are just as 
high-quality as Emacs's, though admittedly coreutils is a smaller 
project and is easier to shepherd.

Glenn's objection (1) was not enough support in Emacs.  This depends on 
the workflow one uses.  I use vc-dwim and it works well for me.  Others 
may prefer to do other things, and this may require some Elisp hacking 
to work to their satisfaction, but it shouldn't be a big problem.

Glenn's objection (2) was that he sometimes wants more details in the 
commit log than in the ChangeLog.  He's right, this is a downside, but 
my own feeling is that it's a minor one.  We shouldn't be spending so 
much time on ChangeLog files that we worry about this sort of thing.  
Just write a decent entry and move on.


[-- Attachment #2: ChangeLog --]
[-- Type: text/plain, Size: 4451 bytes --]

2014-07-18  Pádraig Brady  <P@draigBrady.com>

	version 8.23
	* NEWS: Record release date.

	build: port new bootstrap script to POSIX shells
	* build-aux/gen-single-binary.sh: Avoid bash only constructs.
	Reported by Assaf Gordon

	tests: fix false failure with spaces in the test directory
	* tests/misc/env.sh: Skip if we can't execute the generated shebang,
	which would be the case if there are spaces in the directory hierarchy.
	This is triggered by `make distcheck`

2014-07-18  Pádraig Brady  <P@draigBrady.com>

	build: don't distribute generated coreutils.h
	This issue was identified by the manifest comparisons
	done by `make distcheck`

	* src/local.mk (noinst_HEADERS): Remove coreutils.h from this always
	distributed list.
	(nodist_src_coreutils_SOURCES): Add coreutils.h as its contents
	are determined at configure time, so pointless to distribute.
	(src_coreutils_SOURCES): Define explicitly so that the corresponding
	nodist_ variable is honored.
	(DISTCLEANFILES): Add coreutils.h to this rather than CLEANFILES,
	as its contents are determined at configure time.

2014-07-18  Pádraig Brady  <P@draigBrady.com>

	tests: fix false failure in cp --preserve=context test
	With libselinux-2.2.1-6.fc20.x86_64, kernel-3.12.6-300.fc20.x86_64
	`cp --preserve=context src dst` was seen to succeed when src and
	dst where on the same fixed context file system, as lsetfilecon()
	returned success in this case when the context wasn't being changed.

	* tests/cp/cp-a-selinux.sh: Copy from a different file system to
	most likely have a different context that will test context
	setting logic correctly.

2014-07-18  Pádraig Brady  <P@draigBrady.com>

	maint: avoid a syntax-check warning in previous commit
	* cfg.mk: Avoid warning for @AMDEP_TRUE@ usage in man/local.mk

2014-07-17  Paul Eggert  <eggert@cs.ucla.edu>

	build: port to non-GNU make
	This fixes a problem with native Solaris 'make', which does not
	grok '-include' lines (a GNU extension to POSIX 'make').
	* configure.ac (man/dynamic-deps.mk): Create it, with an old
	time stamp, if doing dynamic dependency tracking.
	* man/local.mk (DISTCLEANFILES): Put man/dynamic-deps.mk here,
	rather than in CLEANFILES.
	(man/dynamic-deps.mk): Don't create it read-only, so that we
	can easily touch it later.
	Include it with '@AMDEP_TRUE@@am__include@', not with '-include',
	as '-include' does not work with native Solaris 'make'.

2014-07-16  Assaf Gordon  <assafgordon@gmail.com>

	numfmt: fix isblank() usage for some unibyte locales
	* src/numfmt.c (simple_strtod_int): Replace isdigit() with c_isdigit()
	to avoid locale concerns and -Wchar-subscripts warnings on cygwin.
	Remove the now redundant locale guard.
	(simple_strtod_human): Cast characters to unsigned so that the promoted
	int value passed to isblank() is positive, allowing it to work correctly
	for all characters in unibyte locales.  Previously character 0xA0,
	i.e. non-breaking space, would be misclassified for example.
	(process_suffixed_number): Likewise.
	(skip_fields): Likewise.
	Both issues were triggered by the -Wchar-subscripts warning on GCC 4.8.3
	on cygwin, due to the is*() implementations used there, but the issue
	is present on all platforms defaulting to signed chars.
	* NEWS: Mention the bug fix.

	Reported by Eric Blake

2014-07-14  Pádraig Brady  <P@draigBrady.com>

	tests: fix portabilitiy issue in new od-endian test
	* tests/misc/od-endian.sh: '\n' is not generally supported
	in the replacement, so use the more portable "\\$NL".

	tests: fix portability issue in new dd/ascii test
	* tests/dd/ascii.sh: Quote so that '\\' is passed
	to printf rather than a single '\', as that's not portable.
	bash, dash, zsh and external solaris printf were seen to support
	  printf '\%03o' 1 2
	while ksh, freebsh shell and external GNU printf need the more portable
	  printf '\\%03o' 1 2
	Note we don't use env here to call the coreutils printf implementation,
	as there are many printf calls, so relying on portable shell
	implementations will be faster.

	tests: avoid hang when files created with wrong permissions
	* tests/split/b-chunk.sh: Never show the rm prompt which would
	hang the test suite.  This was seen when split erroneously
	created files with no persmissions, which was triggered by
	this compiler bug in clang 3.4:
	http://llvm.org/bugs/show_bug.cgi?id=18346

	maint: avoid clang -Wint-to-pointer-cast warning
	* src/chroot.c: Explicitly cast int to pointer type.

Re: Generating the ChangeLog files from the commit messages

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

() Paul Eggert <eggert@cs.ucla.edu>
() Tue, 18 Nov 2014 09:56:03 -0800

   Glenn's objection (2) was that he sometimes wants more details
   in the commit log than in the ChangeLog.  He's right, this is a
   downside, but my own feeling is that it's a minor one.  We
   shouldn't be spending so much time on ChangeLog files that we
   worry about this sort of thing.  Just write a decent entry and
   move on.

If we settle on an "official commit log format" that sets aside an
area for such details, then that area can be mechanically omitted
when generating the ChangeLog file.  I'm partial to:

 TITLE

 PARAGRAPH-1

 PARAGRAPH-2

 * FILE (FUNC): DESCRIPTION-OF-CHANGE.

The PARAGRAPHs are easily distinguishable; the generated ChangeLog
would be:

 YYYY-MM-DD  AUTHOR  <EMAIL>

 TITLE
 * FILE (FUNC): DESCRIPTION-OF-CHANGE.

(modulo some bol TABs), which is the current practice AFAICT.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Generating the ChangeLog files from the commit messages

[Stefan Monnier]

> so we could maintain a list of mistake corrections for that purpose.

Sounds a lot more complex than just "edit ChangeLog, and commit".

>  4) I have the impression that having to write ChangeLogs leads to
>  higher quality entries than just using commit logs would.

Yes, that's a risk.  We have to stay alert.

>> I did a lot of merging between branches with bzr, and had basically zero
>> problems with the ChangeLogs.
> The same is true when using git-merge-changelog with git.

We've developed ways to circumvent the problems and got used to them,
but that doesn't mean they don't exist.

One annoyance for me is the fact that we have several ChangeLog files,
so when I make a commit that touches many files, its description is
spread over several ChangeLogs, which is inconvenient (I almost always
end up forgetting to C-x C-s one of them, or to mark it in the *vc-dir*
buffer).


        Stefan

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

On 11/18/2014 01:12 PM, Stefan Monnier wrote:
> One annoyance for me is the fact that we have several ChangeLog files,

Likewise.  We should have just one.  We can do that at the same time 
that we fix things so that it's generated automatically.

Re: Generating the ChangeLog files from the commit messages

[Bastien]

I'm 100% with Glenn on this.

In Org, we generate the ChangeLog from the Git history and I often
wish we started by having a proper ChangeLog first.

If ChangeLog files are generated, maintainers will have to edit lots
of contributed commits manually before applying them -- this will be
boring.

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Bastien]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> One annoyance for me is the fact that we have several ChangeLog files,
> so when I make a commit that touches many files, its description is
> spread over several ChangeLogs, which is inconvenient (I almost always
> end up forgetting to C-x C-s one of them, or to mark it in the *vc-dir*
> buffer).

FWIW, I'm also all for merging ChangeLogs files.

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> One annoyance for me is the fact that we have several ChangeLog files,
> so when I make a commit that touches many files, its description is
> spread over several ChangeLogs, which is inconvenient (I almost always
> end up forgetting to C-x C-s one of them, or to mark it in the *vc-dir*
> buffer).

Yes, that's annoying.  And `C-c C-a' in the commit buffer doesn't always
find the entries from those deeply nested ChangeLog files.

If we had just one ChangeLog, I don't think I'd mind the ChangeLogs very
much.  But I think if we keep the ChangeLog, we really should distribute
git-merge-changelog in the Emacs git tree -- otherwise contributing to
Emacs becomes much harder than it has to be.

Count me in the "I'm fine with keeping the ChangeLog and I'm fine with
dumping 'em" camp.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Generating the ChangeLog files from the commit messages

[Rasmus]

Hi,

Bastien <bzg@gnu.org> writes:

> In Org, we generate the ChangeLog from the Git history and I often
> wish we started by having a proper ChangeLog first.

Interesting.

> If ChangeLog files are generated, maintainers will have to edit lots
> of contributed commits manually before applying them -- this will be
> boring.

As a offender I have observed that if a patch does not meet the commit
message/ChangeLog requirements one will (often) be asked to resubmit
following those prescriptions.  

What are the types of mistakes you see?  Could they be fixed by tougher
reviews?  Or is the issue with forgetful people with commit access?

—Rasmus

-- 
. . . The proofs are technical in nature and provides no real understanding

Re: Generating the ChangeLog files from the commit messages

[Bastien]

Hi Rasmus,

Rasmus <rasmus@gmx.us> writes:

>> If ChangeLog files are generated, maintainers will have to edit lots
>> of contributed commits manually before applying them -- this will be
>> boring.
>
> As a offender I have observed that if a patch does not meet the commit
> message/ChangeLog requirements one will (often) be asked to resubmit
> following those prescriptions.

Yes.  I've taught myself to ask this to contributors, but still,
I continue to accept patches for which I edit the changelog myself.

> What are the types of mistakes you see?

You can basically take each advice from this page

  http://orgmode.org/worg/org-contribute.html#unnumbered-10

and imagine the errors: an empty changelog, line 1 way too long,
the ChangeLog does not start with a capital letter, sentences do
not end with a fullstop, etc.

> Could they be fixed by tougher reviews?

Let me first say that I'm fine with a very low entry barrier for Org
contributors and I don't like the idea of a rigid review process for
such a fun project.  Also, I'm not angry with misformed patches, I
often commit typos and errors myself: I just try to understand what
would make everyone's life easier.

If we had a ChangeLog in Org, people would very quickly learn from it
by just *reading* it--because it is here.  Even doing a simple ~$ git
log is too much for new committers, because they just think they know
how to write a changelog entry, probably bringing habits from other
software communities.

> Or is the issue with forgetful people with commit access?

People get commit access when other committers are confident they
won't have to fix their logs or to restate the rules, so that's really
just a problem with occasionnal contributors.

Best,

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Óscar Fuentes]

Bastien <bzg@gnu.org> writes:

[snip]

> Let me first say that I'm fine with a very low entry barrier for Org
> contributors

[snip]

> If we had a ChangeLog in Org, people would very quickly learn from it
> by just *reading* it--because it is here.

Don't you think that having a ChangeLog highers the entry barrier?

[snip]

Re: Generating the ChangeLog files from the commit messages

[Bastien]

Óscar Fuentes <ofv@wanadoo.es> writes:

>> If we had a ChangeLog in Org, people would very quickly learn from it
>> by just *reading* it--because it is here.
>
> Don't you think that having a ChangeLog highers the entry barrier?

Compared to "having ChangeLog entries within commit messages"?

Definitely not.

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Yuri Khan]

On Wed, Nov 19, 2014 at 5:17 PM, Bastien <bzg@gnu.org> wrote:
>
> I've taught myself to ask this to contributors, but still,
> I continue to accept patches for which I edit the changelog myself.

So what’s the difference?

Workflow A: you receive a patch, apply it (it gets committed to a
temporary local branch), edit the changelog, commit -a --amend, and
push.

Workflow B: you receive a patch, apply it (it gets committed to a
temporary local branch), commit --amend (it lets you edit the commit
message), and push.

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> Date: Wed, 19 Nov 2014 23:29:11 +0700
> From: Yuri Khan <yuri.v.khan@gmail.com>
> Cc: Rasmus <rasmus@gmx.us>, Emacs developers <emacs-devel@gnu.org>
> 
> On Wed, Nov 19, 2014 at 5:17 PM, Bastien <bzg@gnu.org> wrote:
> >
> > I've taught myself to ask this to contributors, but still,
> > I continue to accept patches for which I edit the changelog myself.
> 
> So what’s the difference?
> 
> Workflow A: you receive a patch, apply it (it gets committed to a
> temporary local branch), edit the changelog, commit -a --amend, and
> push.
> 
> Workflow B: you receive a patch, apply it (it gets committed to a
> temporary local branch), commit --amend (it lets you edit the commit
> message), and push.

Workflow C: you receive a patch, and tell the contributor to go ahead
and push.

Re: Generating the ChangeLog files from the commit messages

[David Engster]

Bastien writes:
> I'm 100% with Glenn on this.
>
> In Org, we generate the ChangeLog from the Git history and I often
> wish we started by having a proper ChangeLog first.
>
> If ChangeLog files are generated, maintainers will have to edit lots
> of contributed commits manually before applying them -- this will be
> boring.

CEDET also generates Changelogs, and just recently I spent several hours
on a nice Sunday evening editing them, so I share your pain.  And yet,
I'm all for it. My hope is that this is the first step in getting rid of
the Changelog format altogether, and switching to commit messages that
value content over form.

-David

PS: Yeah I know, won't happen.

Re: Generating the ChangeLog files from the commit messages

[Yuri Khan]

On Wed, Nov 19, 2014 at 10:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Workflow C: you receive a patch, and tell the contributor to go ahead
> and push.

By doing so, you accept the patch as it is. And if the contributor has
push access, you already agreed that [s]he is knowledgeable enough to
write acceptable commit messages.

Re: Include necessary git tools in the Emacs distribution?

[David Engster]

Rüdiger Sonderfeld writes:
> On Friday 14 November 2014 17:13:35 Ulrich Mueller wrote:
>> Gentoo has dev-vcs/git-merge-changelog as a package. However, it would
>> certainly help if it was packaged upstream, without the need to first
>> extract it from gnulib.
>
> I've created a bug (wishlist) report for it to be added to Debian's git-extras 
> package:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734527
>
> But so far (7 Jan 2014) no reply.  I guess the best solution to deal with it, 
> would be to move it from Gnulib to git/contrib.  But that would require the 
> Gnulib and git maintainers to work together and agree on the issue.

Not sure if you know this, but git-merge-changelog is a package in
testing and will hence be part of Debian 8:

https://packages.debian.org/jessie/git-merge-changelog

You can install it from there into current stable with no problems.

-David

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> Date: Wed, 19 Nov 2014 23:52:00 +0700
> From: Yuri Khan <yuri.v.khan@gmail.com>
> Cc: Bastien <bzg@gnu.org>, Rasmus <rasmus@gmx.us>, Emacs developers <emacs-devel@gnu.org>
> 
> On Wed, Nov 19, 2014 at 10:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Workflow C: you receive a patch, and tell the contributor to go ahead
> > and push.
> 
> By doing so, you accept the patch as it is.

The pain is when the code is OK, but the log entry isn't.  In that
case, instead of just letting them push by themselves, I need to do
all the work for them.

Re: Generating the ChangeLog files from the commit messages

[Stefan Monnier]

> I'm all for it. My hope is that this is the first step in getting rid of
> the Changelog format altogether, and switching to commit messages that
> value content over form.

I'm not sure what other format you have in mind which would have
better content.

FWIW, I think an ideal commit message should look like:

    <location-summary> <change-summary>
    <explanation-if-needed>
    <bug-nb-if-applicable>
    <changelog-style details>

In most cases, the explanation will be empty because the bug-nb provides
the explanation or because the explanation can be placed in the code
instead (a better choice, when applicable).


        Stefan

Re: Generating the ChangeLog files from the commit messages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I separated change log files by directories to make them more
convenient.  Most changes only relate to one directory; a few changes
relate to src and lisp, but you can see that easily enough.

If they are automatically generated, I suppose people could generate
them combined or separated; but it is more convenient to distribute
them separated in the distros.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Generating the ChangeLog files from the commit messages

[David Engster]

Stefan Monnier writes:
>> I'm all for it. My hope is that this is the first step in getting rid of
>> the Changelog format altogether, and switching to commit messages that
>> value content over form.
>
> I'm not sure what other format you have in mind which would have
> better content.

That's easy: prose. Just browse through projects like Git or Linux,
which enforce meaningful commit messages. I'm interested in the 'Why'
and 'How', not in the 'Where' and 'What'.

> FWIW, I think an ideal commit message should look like:
>
>     <location-summary> <change-summary>
>     <explanation-if-needed>

Unless it's a trivial change, it's always needed. Even when you have a
bug report, at least a copy&paste of the relevant bit should be there,
so you can quickly see the problem without having to look it up.

>     <bug-nb-if-applicable>

Sure.

>     <changelog-style details>

That's precisely what I'd drop. It's just a verbose diff. I'm not saying
it's completely useless (although it comes close), but it's not worth
the effort.

> In most cases, the explanation will be empty because the bug-nb provides
> the explanation or because the explanation can be placed in the code
> instead (a better choice, when applicable).

I disagree. Comments and commit logs are orthogonal. A comment helps you
understand how code works. A commit log describes why code was changed
(unless it's new, but that's less common).

-David

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

Richard Stallman wrote:
> If they are automatically generated, I suppose people could generate
> them combined or separated; but it is more convenient to distribute
> them separated in the distros.

In Bug#19113 I just now proposed patches to generate the ChangeLogs combined. 
This was easier, as it's what other GNU projects do and I borrowed their code. 
If someone would like to change the generation procedure to tease out various 
subdirectories' changes, please feel free.  (There's no rush to do this, as the 
procedure matters only when making a new release.)  Other comments are welcome too.

Re: Generating the ChangeLog files from the commit messages

[Yuri Khan]

On Thu, Nov 20, 2014 at 6:15 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:

> In Bug#19113 I just now proposed patches to generate the ChangeLogs
> combined. This was easier, as it's what other GNU projects do and I borrowed
> their code. If someone would like to change the generation procedure to
> tease out various subdirectories' changes, please feel free.  (There's no
> rush to do this, as the procedure matters only when making a new release.)
> Other comments are welcome too.

I glanced through the gitlog-to-changelog script, and it appears to
pass all arguments after “--” to git log. git log accepts as
arguments, among other things, zero or more file or subdirectory paths
to limit the log to. To disambiguate paths from branch heads and tags,
one adds another “--” in between.

Example:

…/emacs $ gitlog-to-changelog --since=2014-11-18 -- foo -- lisp

should generate a changelog from the commits reachable from the branch
head “foo” authored on or after 2014-11-18 and touching any files in
the subdirectory “lisp”.

(I have not tried this.)

Re: Generating the ChangeLog files from the commit messages

[Yuri Khan]

On Thu, Nov 20, 2014 at 12:49 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> The pain is when the code is OK, but the log entry isn't.  In that
> case, instead of just letting them push by themselves, I need to do
> all the work for them.

If they have push access, you needn’t. Workflow D: you tell them “The
code looks good to me. Please fix the commit message <list of proposed
changes> and push”. They then git commit --amend && git push, and they
are left with a better understanding of how to write good commit
messages for Emacs.

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

Yuri Khan wrote:
> …/emacs $ gitlog-to-changelog --since=2014-11-18 -- foo -- lisp
>
> should generate a changelog from the commits reachable from the branch
> head “foo” authored on or after 2014-11-18 and touching any files in
> the subdirectory “lisp”.

Thanks, but that would generate duplicates in the ChangeLog files.  For example, 
if a patch updates both lisp/Makefile.in and src/Makefile.in, it would copy the 
patch's change log entry to both lisp/ChangeLog and src/ChangeLog.  Plus, it's 
not clear how to generate the "leftover" ChangeLog at the top level, which 
contains everything not in the subsidiary ChangeLogs.

Re: Generating the ChangeLog files from the commit messages

[Yuri Khan]

On Thu, Nov 20, 2014 at 9:15 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> Yuri Khan wrote:
>>
>> …/emacs $ gitlog-to-changelog --since=2014-11-18 -- foo -- lisp
>>
>> should generate a changelog from the commits reachable from the branch
>> head “foo” authored on or after 2014-11-18 and touching any files in
>> the subdirectory “lisp”.
>
> Thanks, but that would generate duplicates in the ChangeLog files.  For
> example, if a patch updates both lisp/Makefile.in and src/Makefile.in, it
> would copy the patch's change log entry to both lisp/ChangeLog and
> src/ChangeLog.

True. What is the existing practice for such cases? Does a changelog
entry go into the ChangeLog file of the deepest common ancestor of all
changes or what?

> Plus, it's not clear how to generate the "leftover"
> ChangeLog at the top level, which contains everything not in the subsidiary
> ChangeLogs.

I do not have an elegant solution to this problem.

A kludgy solution involves taking the list of top-level files,
filtering out directories which have separate ChangeLogs, and passing
the rest to git-log.

…/emacs$ git log -- $(ls | grep -vE
'leim|doc|lwlib|nt|msdos|lisp|admin|src|lib-src|oldXmenu|nextstep|test')

Re: Generating the ChangeLog files from the commit messages

[Bastien]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Wed, Nov 19, 2014 at 5:17 PM, Bastien <bzg@gnu.org> wrote:
>>
>> I've taught myself to ask this to contributors, but still,
>> I continue to accept patches for which I edit the changelog myself.
>
> So what’s the difference?

The difference is in creating incentives for new committers to write
proper ChangeLog files.

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Bastien]

Eli Zaretskii <eliz@gnu.org> writes:

> The pain is when the code is OK, but the log entry isn't.  In that
> case, instead of just letting them push by themselves, I need to do
> all the work for them.

And I expect this pain to grow with generated ChangeLog files,
as it cancels a good habit.

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Bastien]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Thu, Nov 20, 2014 at 12:49 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> The pain is when the code is OK, but the log entry isn't.  In that
>> case, instead of just letting them push by themselves, I need to do
>> all the work for them.
>
> If they have push access, you needn’t. Workflow D: you tell them “The
> code looks good to me. Please fix the commit message <list of proposed
> changes> and push”. They then git commit --amend && git push, and they
> are left with a better understanding of how to write good commit
> messages for Emacs.

From experience, I'd say that 50% of the committers will not fix their
changelogs and your task is then to (1) track patches in wait for a
better changelog entry and (2) apply them when the committer fixes the
changelog entry OR (2bis) fix it yourself... and waiting for (2) or
(2bis) to happen is so boring that you often fix the entry yourself
and stop encouraging committers to learn how to fix changelogs.

Or maybe I'm getting old :)

-- 
 Bastien

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

On 11/19/2014 08:04 PM, Yuri Khan wrote:
> What is the existing practice for such cases?

Existing practice elsewhere is to have just one generated ChangeLog 
file, at the top level.  If we had done this for Emacs from day one, 
we'd have a ChangeLog file of about 25 MB.  Although this would have 
been too large in the old days, nowadays it's not such a big deal, as 
computers typically have way more than 25 MB.  When I open a 25 MB 
ChangeLog file with Emacs, it appears instantaneously on my screen, and 
I can navigate through it without delays.

I prefer having a single ChangeLog file to having smaller ChangeLog* 
files in various locations, as it's a bit easier to scout through 
history that way.  The patch that I've proposed in Bug#19113 doesn't go 
that far, though; it merely arranges for future git commit messages to 
be copied into the top-level ChangeLog.  Perhaps coalescing the 
historical ChangeLog files could be done in a followon patch.

Re: Generating the ChangeLog files from the commit messages

[Kelvin White]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

"Paul Eggert" <eggert@cs.ucla.edu> wrote:
>
> On 11/19/2014 08:04 PM, Yuri Khan wrote:
>>
>> What is the existing practice for such cases?
>
>
> Existing practice elsewhere is to have just one generated ChangeLog file,
at the top level.  If we had done this for Emacs from day one, we'd have a
ChangeLog file of about 25 MB.  Although this would have been too large in
the old days, nowadays it's not such a big deal, as computers typically
have way more than 25 MB.  When I open a 25 MB ChangeLog file with Emacs,
it appears instantaneously on my screen, and I can navigate through it
without delays.
>
> I prefer having a single ChangeLog file to having smaller ChangeLog*
files in various locations, as it's a bit easier to scout through history
that way.  The patch that I've proposed in Bug#19113 doesn't go that far,
though; it merely arranges for future git commit messages to be copied into
the top-level ChangeLog.  Perhaps coalescing the historical ChangeLog files
could be done in a followon patch.

I agree. Also, there is a separate ChangeLog file in the lisp/erc
directory. How would I deal with this regarding this topic. Should it be
merged or removed?

[-- Attachment #2: Type: text/html, Size: 1369 bytes --]

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

Kelvin White wrote:
> there is a separate ChangeLog file in the lisp/erc
> directory. How would I deal with this regarding this topic. Should it be
> merged or removed?

The idea is that lisp/erc files could be maintained just as other files are, 
i.e., changes to them should be accompanied by commit messages in ChangeLog 
format.  This will require a change in practice when committing erc changes. 
For example, the message for your commit ea283d2 dated November 9 was merely 
"Update version to 24.5", and this commit altered lisp/erc/ChangeLog.  Under the 
proposed procedure, the commit message would instead use ChangeLog format 
preceded by a summary line and a blank line, e.g.:

   Update Emacs version for erc.

   * lisp/erc/erc-match.el, lisp/erc/erc-join.el, lisp/erc/erc-stamp.el:
   * lisp/erc/erc-backend.el, lisp/erc/erc.el, lisp/erc/erc-button.el:
   Update version to 24.5.

(but not indented), and there would be no change to the ChangeLog because there 
would be no ChangeLog file in the repository.

As a historical matter, lisp/erc/ChangeLog would be renamed to 
lisp/erc/ChangeLog.09 and wouldn't change thereafter.

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Wed, 19 Nov 2014 23:51:11 +0100
> Cc: Bastien <bzg@gnu.org>, emacs-devel@gnu.org
> 
> >     <changelog-style details>
> 
> That's precisely what I'd drop. It's just a verbose diff.

No, it's a summary of diffs.  As such, it many times presents a
bird-eye view that is hard to grasp by looking at the diffs.  It is
also much shorter, so if you want to decide whether a particular
changeset is of interest, you have much less text to read.

So I definitely disagree that this should be dropped.

Re: Generating the ChangeLog files from the commit messages

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Date: Wed, 19 Nov 2014 23:51:11 +0100
>> Cc: Bastien <bzg@gnu.org>, emacs-devel@gnu.org
>> 
>> >     <changelog-style details>
>> 
>> That's precisely what I'd drop. It's just a verbose diff.
>
> No, it's a summary of diffs.  As such, it many times presents a
> bird-eye view that is hard to grasp by looking at the diffs.

Well, this is also because our commits are often too large.

> It is also much shorter, so if you want to decide whether a particular
> changeset is of interest, you have much less text to read.

A good commit title is often enough to raise interest. This entails a
location prefix to mark which area of the code is affected, so that the
person who maintains that area can immediately see it, filter by it,
etc.

As I've said: the Changelog is not completely useless. But most of its
information is automatically generated from the diff. Just go through
the log and in your mind strip all the generated stuff (like filenames
and affected symbols), and see what information provided by the *author*
actually remains - it's usually very little, and often stating the
obvious, instead of saying *why* something was done. This is not
primarily the author's fault: since you have to break down your changes
function by function, you often cannot really describe the broader
picture these changes accomplish, so you just do the mandatory
book-keeping.

This makes our commit logs mostly useless for actually understanding how
code evolved. You may feel differently, but keep in mind there are not
many people who know the Emacs code base as well as you do. When I dig
into files which are unfamiliar to me (like recently into compile.el),
and hit 'l' on an annotated line, it's usually a pretty frustrating
experience.

I encourage anyone to browse through the commit logs from projects like
Git or Linux. If people then still tell me that our style is superior,
so be it.

-David

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: monnier@IRO.UMontreal.CA,  bzg@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 21 Nov 2014 17:19:30 +0100
> 
> Just go through the log and in your mind strip all the generated
> stuff (like filenames and affected symbols), and see what
> information provided by the *author* actually remains - it's usually
> very little, and often stating the obvious, instead of saying *why*
> something was done.

That's not what ChangeLog is supposed to tell.

> This makes our commit logs mostly useless for actually understanding how
> code evolved.

It's not intended for that.  The 'annotate' commands are for that.

Re: Generating the ChangeLog files from the commit messages

[Stefan Monnier]

> It's not intended for that.  The 'annotate' commands are for that.

No, same difference.  In many cases, a change is installed without
explaining why.  On this point I definitely agree with David.

I don't think it' due to the ChangeLog format per-se, but it's indeed
easy to write a ChangeLog comment which just paraphrases the diff
without really providing the explanation for the change.


        Stefan

Re: Generating the ChangeLog files from the commit messages

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: monnier@IRO.UMontreal.CA,  bzg@gnu.org,  emacs-devel@gnu.org
>> Date: Fri, 21 Nov 2014 17:19:30 +0100
>> 
>> Just go through the log and in your mind strip all the generated
>> stuff (like filenames and affected symbols), and see what
>> information provided by the *author* actually remains - it's usually
>> very little, and often stating the obvious, instead of saying *why*
>> something was done.
>
> That's not what ChangeLog is supposed to tell.

We're in violent agreement here.

>> This makes our commit logs mostly useless for actually understanding how
>> code evolved.
>
> It's not intended for that.  The 'annotate' commands are for that.

Yes. As I've written: we should improve the output of 'l' in an
annotated file from the Emacs repo: commit logs should tell why
something was done.

Anyway, I'm repeating myself, let's just disagree and drop this. It's
not like this is gonna change anytime soon.

-David

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

As a heads-up, it's looking like we won't be generating ChangeLogs automatically 
any time soon.

Any automated procedure would entail some minor changes to what ChangeLogs look 
like (e.g., spacing, order, commit dates), and this is likely to clash with at 
least one developer's preferences.  In Bug#19113 Stefan said he'd rather leave 
things alone unless we get explicit agreement from all the core maintainers, a 
bar sufficiently high that I'm taking it as a polite way of saying "let's not do 
this".

Re: Generating the ChangeLog files from the commit messages

[Jan D.]

Fwiw, I am for generating Changelogs. 

     Jan D. 



> 29 nov 2014 kl. 23:35 skrev Paul Eggert <eggert@cs.ucla.edu>:
> 
> As a heads-up, it's looking like we won't be generating ChangeLogs automatically any time soon.
> 
> Any automated procedure would entail some minor changes to what ChangeLogs look like (e.g., spacing, order, commit dates), and this is likely to clash with at least one developer's preferences.  In Bug#19113 Stefan said he'd rather leave things alone unless we get explicit agreement from all the core maintainers, a bar sufficiently high that I'm taking it as a polite way of saying "let's not do this".
> 

Re: Generating the ChangeLog files from the commit messages

[Jorgen Schaefer]

On Sat, 29 Nov 2014 14:35:07 -0800
Paul Eggert <eggert@cs.ucla.edu> wrote:

> As a heads-up, it's looking like we won't be generating ChangeLogs
> automatically any time soon.

Sad. Is there at least some consensus to merge the various ChangeLog
files into a single one, though?

Regards,
Jorgen

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

Jorgen Schaefer wrote:
> Is there at least some consensus to merge the various ChangeLog
> files into a single one, though?

No, sorry, Stefan didn't like that idea much.  FWIW, he was more positive about 
generating ChangeLogs automatically.

I can think of one improvement here, though, which we can make regardless of 
whether ChangeLogs are generated automatically.  For changes that affect files 
in different directories, I'd like to put the corresponding commit message into 
the ChangeLog at the closest ancestor to all the directories, instead of 
breaking the message into pieces and scattering it into little ChangeLog bits 
here and there that have to be pieced together later if I want to understand the 
change.  That would make my life easier, for both writing and reading the source 
code.

Re: Generating the ChangeLog files from the commit messages

[Lars Magne Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Any automated procedure would entail some minor changes to what
> ChangeLogs look like (e.g., spacing, order, commit dates), and this is
> likely to clash with at least one developer's preferences. 

Could we do an informal vote on this to clarify whether there are any
developers that really strenuously object to generating the ChangeLogs?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Generating the ChangeLog files from the commit messages

[Stefan Monnier]

>> Is there at least some consensus to merge the various ChangeLog
>> files into a single one, though?
> No, sorry, Stefan didn't like that idea much.

To be more precise, I don't like the idea of merging the *existing*
ChangeLog files.  We should generate a single ChangeLog, OTOH.


        Stefan

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Sun, 30 Nov 2014 13:11:18 -0500
> Cc: emacs-devel@gnu.org, Jorgen Schaefer <forcer@forcix.cx>
> 
> We should generate a single ChangeLog, OTOH.

Btw, there's another minor issue with a single large ChangeLog: the
size suggested by Paul (32MB AFAIR) is above the default value of
large-file-warning-threshold (currently 10,000,000 bytes).  So we
should either bump the default value of that variable or lower the
size of ChangeLog after which we roll it over, otherwise the users
will be annoyed by the large file warnings when visiting ChangeLog.

Re: Generating the ChangeLog files from the commit messages

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Btw, there's another minor issue with a single large ChangeLog: the
> size suggested by Paul (32MB AFAIR) is above the default value of
> large-file-warning-threshold (currently 10,000,000 bytes).  So we
> should either bump the default value of that variable or lower the
> size of ChangeLog after which we roll it over, otherwise the users
> will be annoyed by the large file warnings when visiting ChangeLog.

I think we should probably increase that limit.  10MB isn't a lot, these
days.

But I wonder whether it's feasible to make that a dynamic limit?  Do we
have a way to ask the OS how much (physical-ish) RAM it has?  If it's a
64GiB machine, it might make sense to make the limit higher than if it's
a 16MiB machine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  eggert@cs.ucla.edu,  forcer@forcix.cx,  emacs-devel@gnu.org
> Date: Sun, 30 Nov 2014 19:30:47 +0100
> 
> But I wonder whether it's feasible to make that a dynamic limit?  Do we
> have a way to ask the OS how much (physical-ish) RAM it has?  If it's a
> 64GiB machine, it might make sense to make the limit higher than if it's
> a 16MiB machine.

I'm not sure this is only about available memory.  This warning caught
my typing mistakes a few times.  It could also be a security feature.

Re: default large-file-warning-threshold (was: Generating the ChangeLog files ...)

[Paul Eggert]

Lars Magne Ingebrigtsen wrote:
> Do we have a way to ask the OS how much (physical-ish) RAM it has?

We could easily add that, using Gnulib's physmem module.  For example, GNU 
'sort' uses physmem to calculate an internal buffer size ranging from 1/8 to 3/4 
of physical memory, depending on some other factors.

Eli Zaretskii wrote:
 > It could also be a security feature.

It's largely a security feature insofar as it avoids denial-of-service problems, 
and deriving the limit from physical memory capacity helps to avoid these 
problems too.

The 10 MB limit is too small nowadays for typical machines.  I regularly run 
into it and it's a genuine (albeit minor) annoyance.  When the 10 MB limit was 
established back in 2003, machines typically had 64 MiB or so of RAM.  Nowadays 
8 GiB is closer to being typical and the 10 MB limit is way below the sweet spot 
for warnings.

If we were conservative and warned about files larger than 1/8 of physical 
memory, my circa-2011 8-GiB work desktop would warn about files larger than 1 
GiB, and my circa-2005 512-MiB laptop would warn about files larger than 64 MiB, 
and both numbers sound about right.

Re: default large-file-warning-threshold

[Óscar Fuentes]

Paul Eggert <eggert@cs.ucla.edu> writes:

[snip]

> If we were conservative and warned about files larger than 1/8 of
> physical memory, my circa-2011 8-GiB work desktop would warn about
> files larger than 1 GiB, and my circa-2005 512-MiB laptop would warn
> about files larger than 64 MiB, and both numbers sound about right.

The impact of visiting large files is not only about raw byte count.
Depending on Emacs configuration (active major/minor modes, etc)
operating on a large file can be impractical even for a high-end modern
machine.

Re: default large-file-warning-threshold

[Lars Magne Ingebrigtsen]

Óscar Fuentes <ofv@wanadoo.es> writes:

> The impact of visiting large files is not only about raw byte count.
> Depending on Emacs configuration (active major/minor modes, etc)
> operating on a large file can be impractical even for a high-end modern
> machine.

If a mode is so slow that visiting a big file is problematic, then that
sounds like a bug in that mode, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: default large-file-warning-threshold

[Óscar Fuentes]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> If a mode is so slow that visiting a big file is problematic, then that
> sounds like a bug in that mode, I think.

I don't think so. Some tasks are complex and hard/impractical to apply
lazily. If a feature needs to scan the whole buffer, and the scanning is
not trivial, visiting an 800 MB file would be unpleasant.

Re: Generating the ChangeLog files from the commit messages

[Stefan Monnier]

>> We should generate a single ChangeLog, OTOH.
> Btw, there's another minor issue with a single large ChangeLog: the
> size suggested by Paul (32MB AFAIR) is above the default value of
> large-file-warning-threshold (currently 10,000,000 bytes).

IIUC this figure was computed for a combined single file of the
whole history.  The generated file would start empty (since it wouldn't
include the part that's already present in the existing ChangeLog files).

It makes sense to roll those files on a release-number basis rather than
just size.

Paul said:
> The 10 MB limit is too small nowadays for typical machines.

Could be.  I rarely bump into it, personally.  But I wouldn't mind
increasing it a bit, tho probably not by a factor of ten.

> I regularly run into it and it's a genuine (albeit minor) annoyance.

What value would stop you from bumping into it, and on which kinds of
files do you typically bump into it.

> When the 10 MB limit was established back in 2003, machines typically
> had 64 MiB or so of RAM.

Reality check: my 2003-vintage X30 laptop (which I happily use when
doing presentations) came with 768MB of RAM.  This was fairly generous
back then (it's actually the max it can take, AFAIK).  The 2003 12"
PowerBook I got at the same time had 256MB (with a max of 1.25GB) and it
was considered "under provided" so I upgraded it to 786MB even before
I added a wifi card to it.  IOW I don't think 64MB was typical back then.

> Nowadays 8 GiB is closer to being typical and the 10 MB limit is way
> below the sweet spot for warnings.

But the size of human-edited files (the bread-and-butter for Emacs)
hasn't increased nearly as much.


        Stefan

Re: Generating the ChangeLog files from the commit messages

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org, forcer@forcix.cx
> Date: Sun, 30 Nov 2014 22:24:57 -0500
> 
> >> We should generate a single ChangeLog, OTOH.
> > Btw, there's another minor issue with a single large ChangeLog: the
> > size suggested by Paul (32MB AFAIR) is above the default value of
> > large-file-warning-threshold (currently 10,000,000 bytes).
> 
> IIUC this figure was computed for a combined single file of the
> whole history.  The generated file would start empty (since it wouldn't
> include the part that's already present in the existing ChangeLog files).

According to my estimations (see earlier in this thread), we will hit
the limit quite soon even if we start empty.

Re: default large-file-warning-threshold

[Paul Eggert]

Óscar Fuentes wrote:
> If a feature needs to scan the whole buffer, and the scanning is
> not trivial, visiting an 800 MB file would be unpleasant.

If the scanning is sufficiently nontrivial, the same argument would apply to a 
10 MB file, no?  So perhaps large-file-warning-threshold should be mode- or 
feature-dependent, as well as dependent on physical RAM size.

I agree that Emacs is too slow when used in default mode on large text files. 
On my circa-2011 desktop if I want reasonable efficiency on a multigigabyte text 
file I typically use find-file-literally, because plain find-file makes me wait 
for tooooo maaaannny seconds.  There's no reason in principle that Emacs must be 
that slow on large files, it's just that performance on large files has not been 
that high a priority, and the tiny default large-file-warning-threshold is to 
some extent a symptom of this.

Here's a little shell script that lets me reproduce the performance problem, if 
you're interested:

yes '#define assume_initialized(val) asm ("" : "=X" (val))' |
   head -n 30000000 >t
time emacs -Q -nw --execute '(find-file "t")' --kill
time emacs -Q -nw --execute '(find-file-literally "t")' --kill

On my desktop the find-file version takes 8.06 user CPU seconds and the delay is 
really annoying, whereas the find-file-literally version takes only 0.04 user 
CPU seconds.  It's like night and day.

Re: Generating the ChangeLog files from the commit messages

[Paul Eggert]

Stefan Monnier wrote:

> It makes sense to roll those files on a release-number basis rather than
> just size.

That sounds like a good idea.

> What value would stop you from bumping into it, and on which kinds of
> files do you typically bump into it.

It's usually log files.  For example, if I run 'git log >/tmp/log', and then 
visit /tmp/log, that's about 34 MB for Emac.  Likewise for things like strace 
output or for random other text debugging output.

>> When the 10 MB limit was established back in 2003, machines typically
>> had 64 MiB or so of RAM.
>
> Reality check: my 2003-vintage X30 laptop (which I happily use when
> doing presentations) came with 768MB of RAM.

You had bigger machines than I did.  Even my circa-2005 laptop (which I still 
use) has only 512 MB of RAM.  Was the limit tuned for your new laptop back in 
2003?  In that case, perhaps we should warn if the file is larger than 1/64th of 
physical RAM, though this sounds a tad conservative to me.

> the size of human-edited files (the bread-and-butter for Emacs)
> hasn't increased nearly as much.

I mostly run into the problem when viewing files, not editing them, and the 
files I view can get pretty large.  Sometimes I have to give up on Emacs 
entirely and use 'less', which is an annoying switch.

Re: Generating the ChangeLog files from the commit messages

[H. Dieter Wilhelm]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>
> Paul said:
>> The 10 MB limit is too small nowadays for typical machines.
>
> Could be.  I rarely bump into it, personally.  But I wouldn't mind
> increasing it a bit, tho probably not by a factor of ten.
>
>> I regularly run into it and it's a genuine (albeit minor) annoyance.
>
> What value would stop you from bumping into it, and on which kinds of
> files do you typically bump into it.
>

When *viewing* machine generated code files.

In my use case it's a finite element program called ANSYS which is
creating text files describing a complete FE model in a particular
"programming language APDL".  There is a special Emacs mode for editing
user generated APDL files:

http://www.emacswiki.org/AnsysMode

Nowadays these model files are always bigger than 10 MB, typically in
the order of 50 MB to 100 MB, because they have to include the node and
element descriptions.

        Dieter
-- 
Best wishes
H. Dieter Wilhelm
Darmstadt, Germany

Re: default large-file-warning-threshold

[Eli Zaretskii]

> Date: Sun, 30 Nov 2014 21:46:38 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> yes '#define assume_initialized(val) asm ("" : "=X" (val))' |
>   head -n 30000000 >t
> time emacs -Q -nw --execute '(find-file "t")' --kill
> time emacs -Q -nw --execute '(find-file-literally "t")' --kill

> On my desktop the find-file version takes 8.06 user CPU seconds and the delay is really annoying, whereas the find-file-literally version takes only 0.04 user CPU seconds. It's like night and day.

You are measuring here the one-time (per file) overhead of scanning
through the entire 1.5GB file in order to detect any non-ASCII
characters.  Doing that at 5 nanoseconds (which should be something
like 10 or 15 machine instructions) per byte is very reasonable for
the 2011 vintage CPU, don't you agree?

> There's no reason in principle that Emacs must be that slow on large files, it's just that performance on large files has not been that high a priority

Maybe this isn't high on our priority, but even if it were, I doubt
that the speed of decoding could be significantly increased.  We did
optimize it in 24.4 for ASCII and UTF-8 files.

And of course, the time it takes to read a file into a buffer is not
the most important measure: editing operations on such huge files are
at least as important, if not more important.  I think there's much
more place for optimization in the editing operations than in the
visiting department.

Re: default large-file-warning-threshold

[Achim Gratz]

Lars Magne Ingebrigtsen writes:
> If a mode is so slow that visiting a big file is problematic, then that
> sounds like a bug in that mode, I think.

Try fundamental mode with a line length of around 1MB some time, with
lines truncated at the window border, then.  And no, this isn't anything
spectactularly outrageous, just a CSV export of some tabulated data with
5000 columns.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: default large-file-warning-threshold

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Mon, 01 Dec 2014 18:06:04 +0100
> 
> Lars Magne Ingebrigtsen writes:
> > If a mode is so slow that visiting a big file is problematic, then that
> > sounds like a bug in that mode, I think.
> 
> Try fundamental mode with a line length of around 1MB some time, with
> lines truncated at the window border, then.  And no, this isn't anything
> spectactularly outrageous, just a CSV export of some tabulated data with
> 5000 columns.

A.k.a. bug #13675.

Re: default large-file-warning-threshold

[Paul Eggert]

On 12/01/2014 07:52 AM, Eli Zaretskii wrote:
> I doubt that the speed of decoding could be significantly increased.

We don't need to increase decoding speed to make Emacs perform much 
better on large files.  When I first visit a file, all I need to see is 
the start of the file on my screen.  Emacs needs only to decode the 
initial part of the file to do that.  It can decode the rest of the file 
as I'm looking at the first part.  Although this sort of thing would be 
a nontrivial change to Emacs, it's doable and it would already be done 
if performance on large files were a priority for Emacs developers.

I'm not the only one to run into this problem; see the thread 
"Intelligently opening large files in emacs" 
<http://lists.gnu.org/archive/html/help-gnu-emacs/2014-03/msg00341.html> 
in this mailing list earlier this year.  There, the proposed solution 
was to use find-file-literally for files larger than a certain size.  
Perhaps Emacs should do this by default -- it'd be a nicer user 
experience in many ways than what we have now, and it should be easy to 
implement.

Re: default large-file-warning-threshold

[Lars Magne Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I'm not the only one to run into this problem; see the thread
> "Intelligently opening large files in emacs"
> <http://lists.gnu.org/archive/html/help-gnu-emacs/2014-03/msg00341.html>
> in this mailing list earlier this year.  There, the proposed solution
> was to use find-file-literally for files larger than a certain size.
> Perhaps Emacs should do this by default -- it'd be a nicer user
> experience in many ways than what we have now, and it should be easy
> to implement.

The use case for opening huge files is log files and the like, and you
usually want to see those as "raw" as possible, I think.

What about a new, trivial mode for these buffers?  We open huge files
with `find-file-literally', display a message like "Press `M-something'
to put buffer into mode foo", and change to that new, trivial mode
(which would just have that one keybinding).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: default large-file-warning-threshold

[Paul Eggert]

On 12/01/2014 10:19 AM, Lars Magne Ingebrigtsen wrote:
> What about a new, trivial mode for these buffers?  We open huge files
> with `find-file-literally', display a message like "Press `M-something'
> to put buffer into mode foo", and change to that new, trivial mode
> (which would just have that one keybinding).
>

Thanks, I like this idea, and I think it'd help quite a bit for these 
use cases.

Re: default large-file-warning-threshold

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Achim Gratz <Stromeko@nexgo.de>
>> Date: Mon, 01 Dec 2014 18:06:04 +0100
>> 
>> Lars Magne Ingebrigtsen writes:
>> > If a mode is so slow that visiting a big file is problematic, then that
>> > sounds like a bug in that mode, I think.
>> 
>> Try fundamental mode with a line length of around 1MB some time, with
>> lines truncated at the window border, then.  And no, this isn't anything
>> spectactularly outrageous, just a CSV export of some tabulated data with
>> 5000 columns.
>
> A.k.a. bug #13675.

Does it help to do

(setq-default bidi-paragraph-direction nil)

?

-- 
David Kastrup

Re: default large-file-warning-threshold

[Eli Zaretskii]

> Date: Mon, 01 Dec 2014 10:13:57 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: ofv@wanadoo.es, emacs-devel@gnu.org
> 
> On 12/01/2014 07:52 AM, Eli Zaretskii wrote:
> > I doubt that the speed of decoding could be significantly increased.
> 
> We don't need to increase decoding speed to make Emacs perform much 
> better on large files.  When I first visit a file, all I need to see is 
> the start of the file on my screen.  Emacs needs only to decode the 
> initial part of the file to do that.  It can decode the rest of the file 
> as I'm looking at the first part.

Sounds like a good project to me.

> Although this sort of thing would be a nontrivial change to Emacs,
> it's doable and it would already be done if performance on large
> files were a priority for Emacs developers.

Well, you know that there's no such thing as "a priority for Emacs
developers", right?  There are itches to scratch, that's all.

Re: default large-file-warning-threshold

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 01 Dec 2014 19:33:55 +0100
> 
> > A.k.a. bug #13675.
> 
> Does it help to do
> 
> (setq-default bidi-paragraph-direction nil)
> 
> ?

Not nearly enough, unless you are on a very slow machine.

Re: default large-file-warning-threshold

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 01 Dec 2014 19:33:55 +0100
> 
> > A.k.a. bug #13675.
> 
> Does it help to do
> 
> (setq-default bidi-paragraph-direction nil)
> 
> ?

Not nearly enough, unless you are on a very slow machine.  And if you
are at the beginning of the buffer, this has no effect whatsoever.

(Searches for paragraph beginning are limited to some maximum number
of characters, so this factor has limited effect.)

Re: default large-file-warning-threshold (was: Generating the ChangeLog files ...)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The large file warning is not just a matter of the machine's memory size.
Large buffers are inconvenient for various reasons.

We may want to increase the large file warning threshold, since
machines in general are now both faster and larger than they were.
However, I don't think it should vary depending on the machine you are
using.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: default large-file-warning-threshold

[Ted Zlatanov]

On Mon, 01 Dec 2014 19:19:24 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> The use case for opening huge files is log files and the like, and you
LMI> usually want to see those as "raw" as possible, I think.

LMI> What about a new, trivial mode for these buffers?  We open huge files
LMI> with `find-file-literally', display a message like "Press `M-something'
LMI> to put buffer into mode foo", and change to that new, trivial mode
LMI> (which would just have that one keybinding).

Have you tried vlf?

Ted

Re: default large-file-warning-threshold

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Have you tried vlf?

Nope.  `M-x vlf TAB' doesn't say anything...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: default large-file-warning-threshold

[Rasmus]

Achim Gratz <Stromeko@nexgo.de> writes:

> Lars Magne Ingebrigtsen writes:
>> If a mode is so slow that visiting a big file is problematic, then
>> that sounds like a bug in that mode, I think.
>
> Try fundamental mode with a line length of around 1MB some time, with
> lines truncated at the window border, then.  And no, this isn't anything
> spectactularly outrageous, just a CSV export of some tabulated data with
> 5000 columns.

I get very bad performance when opening xml-files — even when using
emacs -q.

This shows, for instance, when opening the contents file of odt files
generated with Org.

—Rasmus

-- 
May the Force be with you

Re: default large-file-warning-threshold

[Ted Zlatanov]

On Mon, 01 Dec 2014 21:32:03 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Have you tried vlf?

LMI> Nope.  `M-x vlf TAB' doesn't say anything...

It's a package you can find on an ELPA archive :)

Ted

Re: default large-file-warning-threshold

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> Nope.  `M-x vlf TAB' doesn't say anything...
>
> It's a package you can find on an ELPA archive :)

Oh.  :-)

Hm, it does look interesting.  It loads bits of files into the buffer at
a time, but tries to still make re-search-forward etc work...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: default large-file-warning-threshold

[Lars Magne Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Lars Magne Ingebrigtsen wrote:
>> Do we have a way to ask the OS how much (physical-ish) RAM it has?
>
> We could easily add that, using Gnulib's physmem module.  For example,
> GNU 'sort' uses physmem to calculate an internal buffer size ranging
> from 1/8 to 3/4 of physical memory, depending on some other factors.

Here's a concrete suggestion that I think covers most of the issues
regarding the "look at log files" use case, as well as the security
issues involved with large files with complex modes.

0) We introduce a new, very trivial mode called `large-file-mode' which
is just `fundamental-mode' with one command: `M-m', which switches on
whatever mode would normally have been in that buffer.  The mode says on
startup "Use the command `M-m' to switch `foo-mode' on".

1) We set `large-file-warning-threshold' to 1/8th of the physical
memory.  If the user loads a file bigger than that, she gets queried
just as now, but the buffer is loaded with `find-file-literally' and
ends up in `large-file-mode'.

2) We add a new variable, `large-file-automatic-mode-threshold', which
is (max 40MB (/ large-file-warning-threshold 10)).  All files bigger
than this are loaded with `find-file-literally' and ends up in
`large-file-mode'.

3) Files smaller than this are handled like now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no