bug#19074: 24.4; Bug in auth-source.el's search of OS X Keychain

bug#19074: 24.4; Bug in auth-source.el's search of OS X Keychain

[John Mastro]

The library `auth-source.el' contains code to search OS X's Keychain
(the backends `macos-keychain-generic' and `macos-keychain-internet').

Both backends are implemented by parsing the results of OS X's
/usr/bin/security. This parsing is done by
`auth-source-macos-keychain-search-items'. However, there's currently a
problem with this code in both the master and emacs-24 branches.

Specifically, the function `auth-source-macos-keychain-result-append' is
called three times, but each time its result is ignored.

A precise recipe from `emacs -Q' is a little difficult, because it
depends on OS X and what you have in your keychain. However, the below
is an example of what will work with the fix but not with the current
code.

A simple patch (against the emacs-24 batch) is attached. I don't yet
have paperwork on file with FSF, but I believe this is short/trivial
enough to be accepted anyway.

    ;; Example. With appropriate user/host, will be nil before patch but
    ;; return expected output after patch.
    (progn (require 'auth-source)
           (auth-source-macos-keychain-search
            :backend (auth-source-backend-parse 'macos-keychain-internet)
            :user "MY-USER"
            :host "MY-HOST"))

Patch:

diff --git a/lisp/gnus/auth-source.el b/lisp/gnus/auth-source.el
index a50ad75..72ec5f4 100644
--- a/lisp/gnus/auth-source.el
+++ b/lisp/gnus/auth-source.el
@@ -1779,29 +1779,29 @@ entries for git.gnus.org:
         (while (not (eobp))
           (cond
            ((looking-at "^password: \"\\(.+\\)\"$")
-            (auth-source-macos-keychain-result-append
-             ret
-             keychain-generic
-             "secret"
-             (lexical-let ((v (match-string 1)))
-               (lambda () v))))
+            (setq ret (auth-source-macos-keychain-result-append
+                       ret
+                       keychain-generic
+                       "secret"
+                       (lexical-let ((v (match-string 1)))
+                         (lambda () v)))))
            ;; TODO: check if this is really the label
            ;; match 0x00000007 <blob>="AppleID"
            ((looking-at "^[ ]+0x00000007 <blob>=\"\\(.+\\)\"")
-            (auth-source-macos-keychain-result-append
-             ret
-             keychain-generic
-             "label"
-             (match-string 1)))
+            (setq ret (auth-source-macos-keychain-result-append
+                       ret
+                       keychain-generic
+                       "label"
+                       (match-string 1))))
            ;; match "crtr"<uint32>="aapl"
            ;; match "svce"<blob>="AppleID"
            ((looking-at "^[ ]+\"\\([a-z]+\\)\"[^=]+=\"\\(.+\\)\"")
-            (auth-source-macos-keychain-result-append
-             ret
-             keychain-generic
-             (match-string 1)
-             (match-string 2))))
-            (forward-line)))
+            (setq ret (auth-source-macos-keychain-result-append
+                       ret
+                       keychain-generic
+                       (match-string 1)
+                       (match-string 2)))))
+          (forward-line)))
       ;; return `ret' iff it has the :secret key
       (and (plist-get ret :secret) (list ret))))

In GNU Emacs 24.4.1 (x86_64-apple-darwin14.0.0, NS apple-appkit-1343.14)
 of 2014-11-09 on nebula.local
Windowing system distributor `Apple', version 10.3.1343
Configured using:
 `configure --prefix=/usr/local/Cellar/emacs/24.4
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/24.4/share/info/emacs --without-dbus
 --with-gnutls --with-imagemagick --with-ns --disable-ns-self-contained'

Important settings:
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  magit-auto-revert-mode: t
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  flyspell-mode: t
  paredit-mode: t
  eldoc-mode: t
  elisp-slime-nav-mode: t
  whitespace-mode: t
  global-company-mode: t
  company-mode: t
  smartparens-global-strict-mode: t
  projectile-global-mode: t
  projectile-mode: t
  ido-vertical-mode: t
  ido-ubiquitous-mode: t
  flx-ido-mode: t
  ido-everywhere: t
  savehist-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  guide-key-mode: t
  global-discover-mode: t
  discover-mode: t
  recentf-mode: t
  winner-mode: t
  global-auto-revert-mode: t
  global-page-break-lines-mode: t
  page-break-lines-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  global-hl-line-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
SPC f u n c t i o n SPC A-H-s-÷µ¹¶ M-o C-x b <return> 
M-= M-= M-w C-x b C-g C-v C-v C-x b <return> M-o C-q 
` C-y ' M-q SPC l o o p s SPC t h r o u A-H-s-÷µ¹¶ g h 
SPC C-a C-e a n d SPC u s e s SPC <backspace> <return> 
M-o C-x b <return> C-p C-p C-p M-f M-f M-f M-= M-= 
M-w C-x b <return> M-o C-q ` C-y ' SPC t o SPC c o 
l l e A-H-s-÷µ¹¶ c t SPC i n f o r m a t i o n . SPC H 
o w e v e r , SPC t h e SPC r e s A-H-s-÷µ¹¶ u l <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
C-h C-h C-g C-n C-k I n A-H-s-÷µ¹¶ C-c C-j C-c C-j M-^ 
M-e C-k , SPC t h e SPC r e t u r n SPC c <backspace> 
v a l u e SPC o f A-H-s-÷µ¹¶ C-n <M-backspace> <M-backspace> 
w a s SPC b e i n g SPC i g n o r e d . SPC T h i A-H-s-÷µ¹¶ 
s SPC c a u s e d SPC t h e SPC i n t e n d e d SPC 
r e s u l t A-H-s-÷µ¹¶ SPC n o t SPC t o SPC b e SPC r 
e t u r n e d . C-n C-a C-k C-k C-p C-p C-p C-c C-c 
C-x C-f <backspace> <backspace> <backspace> d r o <return> 
<return> e m <return> p a t C-n <return> C-h C-k q 
M-x r e p o r t - e m a c s - b u g <return>

Recent messages:
Quit
Indenting region...done
Indenting region...done
Indenting region...done
Indenting region...done
Indenting region...done
Indenting region...done
Saving file /Users/jbm/src/emacs/emacs/.git/COMMIT_EDITMSG...
Wrote /Users/jbm/src/emacs/emacs/.git/COMMIT_EDITMSG
Git finished

Load-path shadows:
None found.

Features:
(shadow emacsbug expand-region text-mode-expansions
the-org-mode-expansions er-basic-expansions expand-region-core
expand-region-custom magit-key-mode magit view tramp tramp-compat
tramp-loaddefs trampver git-rebase-mode git-commit-mode log-edit
pcvs-util add-log idomenu imenu eieio-opt ace-window ace-jump-mode
diff-mode gnutls network-stream starttls tls mailalias mail-extr sort
hippie-exp org-rmail org-mhe org-irc org-info org-gnus org-docview
org-bibtex bibtex org-bbdb org-w3m ob-sh shell ob-python ob-clojure org
org-macro org-footnote org-pcomplete pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint ob-core ob-eval
org-compat org-macs org-loaddefs cal-menu calendar cal-loaddefs
mule-util mu4e mu4e-speedbar speedbar sb-image ezimage dframe mu4e-main
mu4e-view epa derived epg browse-url mu4e-headers mu4e-compose
mu4e-draft mu4e-actions rfc2368 smtpmail sendmail mu4e-mark mu4e-message
html2text mu4e-proc mu4e-utils doc-view image-mode find-dired dired+
image-dired image-file dired-x dired-aux dired mu4e-lists mu4e-about
mu4e-vars message format-spec rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader mu4e-meta smex misearch multi-isearch
auth-source gnus-util mm-util mail-prsvr password-cache pp vc-git
flyspell ispell jka-compr disp-table paredit eldoc elisp-slime-nav
help-mode whitespace company-files company-keywords company-etags etags
company-gtags company-dabbrev-code company-dabbrev company-capf company
skewer-setup smartparens redshank-loader projectile ibuffer-vc ibuf-ext
ibuffer pkg-info find-func lisp-mnt epl grep compile comint thingatpt
ido-at-point ido-vertical-mode ido-ubiquitous warnings flx-ido flx ido
ibuf-macs savehist saveplace undo-tree diff key-chord guide-key
face-remap popwin discover makey man ansi-color recentf tree-widget
wid-edit browse-kill-ring winner ring diminish solarized-dark-theme
solarized-definitions cl autorevert filenotify page-break-lines delsel
paren hl-line server exec-path-from-shell rx easy-mmode advice help-fns
s ucs-normalize dash-functional dash subr-x pcase cl-macs gv finder-inf
eieio byte-opt bytecomp byte-compile cconv eieio-core edmacro kmacro
info easymenu slime-autoloads package epg-config cl-loaddefs cl-lib
time-date tooltip electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel ns-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process cocoa ns multi-tty emacs)

Memory information:
((conses 16 759686 83978)
 (symbols 48 45387 18)
 (miscs 40 1637 1304)
 (strings 32 105380 22526)
 (string-bytes 1 3126833)
 (vectors 16 136035)
 (vector-slots 8 3699462 25228)
 (floats 8 25664 1278)
 (intervals 56 3552 721)
 (buffers 960 35))

-- 
jbm

bug#19074: Bug in auth-source.el's search of OS X Keychain

[John Mastro]

I accidentally copied only part of the patch in my original email. The
full thing (via `git format-patch') follows.

-- 
jbm


From 2b968a29ff2a01e316e09faa4d765aca08cf0121 Mon Sep 17 00:00:00 2001
From: John Mastro <john.b.mastro@gmail.com>
Date: Sun, 16 Nov 2014 19:41:10 -0800
Subject: [PATCH] Fix auth-source.el bug regarding the OS X Keychain

In `auth-source-macos-keychain-search-items', the return value of
`auth-source-macos-keychain-result-append' was being ignored. This
caused the intended result not to be returned.
---
 lisp/gnus/auth-source.el | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/lisp/gnus/auth-source.el b/lisp/gnus/auth-source.el
index a50ad75..72ec5f4 100644
--- a/lisp/gnus/auth-source.el
+++ b/lisp/gnus/auth-source.el
@@ -1779,29 +1779,29 @@ entries for git.gnus.org:
         (while (not (eobp))
           (cond
            ((looking-at "^password: \"\\(.+\\)\"$")
-            (auth-source-macos-keychain-result-append
-             ret
-             keychain-generic
-             "secret"
-             (lexical-let ((v (match-string 1)))
-               (lambda () v))))
+            (setq ret (auth-source-macos-keychain-result-append
+                       ret
+                       keychain-generic
+                       "secret"
+                       (lexical-let ((v (match-string 1)))
+                         (lambda () v)))))
            ;; TODO: check if this is really the label
            ;; match 0x00000007 <blob>="AppleID"
            ((looking-at "^[ ]+0x00000007 <blob>=\"\\(.+\\)\"")
-            (auth-source-macos-keychain-result-append
-             ret
-             keychain-generic
-             "label"
-             (match-string 1)))
+            (setq ret (auth-source-macos-keychain-result-append
+                       ret
+                       keychain-generic
+                       "label"
+                       (match-string 1))))
            ;; match "crtr"<uint32>="aapl"
            ;; match "svce"<blob>="AppleID"
            ((looking-at "^[ ]+\"\\([a-z]+\\)\"[^=]+=\"\\(.+\\)\"")
-            (auth-source-macos-keychain-result-append
-             ret
-             keychain-generic
-             (match-string 1)
-             (match-string 2))))
-            (forward-line)))
+            (setq ret (auth-source-macos-keychain-result-append
+                       ret
+                       keychain-generic
+                       (match-string 1)
+                       (match-string 2)))))
+          (forward-line)))
       ;; return `ret' iff it has the :secret key
       (and (plist-get ret :secret) (list ret))))
 
-- 
2.1.3

bug#19074: Bug in auth-source.el's search of OS X Keychain

[Ted Zlatanov]

On Sun, 16 Nov 2014 21:38:32 -0800 John Mastro <jbm@jbm.io> wrote: 

JM> I accidentally copied only part of the patch in my original email. The
JM> full thing (via `git format-patch') follows.

Thanks, applied to emacs-24 branch as a bugfix:

commit a10e36a5d7fe95830e3f93dc7ae6f65507738978
Author: John Mastro <john.b.mastro@gmail.com>
Date:   Wed Nov 26 09:15:08 2014 -0500

    auth-source: Fix Mac OS X keychain lookups.
    
    * auth-source.el (auth-source-macos-keychain-search-items): Return
    result of `auth-source-macos-keychain-result-append' (bug#19074).

It will eventually get ported to Emacs master and to Gnus master as
well.  Glenn or Stefan, should I do that or wait for you?

Thanks!
Ted

bug#19074: Bug in auth-source.el's search of OS X Keychain

[Stefan Monnier]

> It will eventually get ported to Emacs master and to Gnus master as
> well.  Glenn or Stefan, should I do that or wait for you?

Feel free to merge emacs-24 into master any time you feel like it.

BUT: I really mean "merge the branch", not "cherrypick my change".


        Stefan

merging emacs-24 (was: bug#19074: Bug in auth-source.el's search of OS X Keychain)

[Stefan Monnier]

> Is it enough to merge laconically like you did:
[...]
> ...and resolve conflicts as needed?

Yes.  Be careful with the conflicts, tho: some of those patches should
simply not be applied (regardless if they'd create conflicts or not).

You might prefer to wait for the gitmerge.el  to appear (I thought it
was supposed to be done already, but I guess it's taking a bit more
time than expected).


        Stefan

Re: merging emacs-24

[David Engster]

Stefan Monnier writes:
>> Is it enough to merge laconically like you did:
> [...]
>> ...and resolve conflicts as needed?
>
> Yes.  Be careful with the conflicts, tho: some of those patches should
> simply not be applied (regardless if they'd create conflicts or not).
>
> You might prefer to wait for the gitmerge.el  to appear (I thought it
> was supposed to be done already, but I guess it's taking a bit more
> time than expected).

Seems like you overlooked this:

http://article.gmane.org/gmane.emacs.devel/178096

-David

Re: merging emacs-24

[Ted Zlatanov]

On Wed, 26 Nov 2014 20:02:22 +0100 David Engster <deng@randomsample.de> wrote: 

DE> Stefan Monnier writes:

>> You might prefer to wait for the gitmerge.el  to appear (I thought it
>> was supposed to be done already, but I guess it's taking a bit more
>> time than expected).

DE> Seems like you overlooked this:

DE> http://article.gmane.org/gmane.emacs.devel/178096

I'm probably not the right person to test this, having done no merges
before.  I'll do this one manually.

Ted

Re: merging emacs-24

[Stefan Monnier]

> Seems like you overlooked this:
> http://article.gmane.org/gmane.emacs.devel/178096

Indeed, I missed it.  Please install it into admin/gitmerge.el,
thank you.


        Stefan

Re: merging emacs-24

[David Engster]

Stefan Monnier writes:
>> Seems like you overlooked this:
>> http://article.gmane.org/gmane.emacs.devel/178096
>
> Indeed, I missed it.  Please install it into admin/gitmerge.el,

Done.

-David

Re: merging emacs-24

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Thu, 27 Nov 2014 18:27:48 +0100
> Cc: Ted Zlatanov <tzz@lifelogs.com>, emacs-devel@gnu.org
> 
> Stefan Monnier writes:
> >> Seems like you overlooked this:
> >> http://article.gmane.org/gmane.emacs.devel/178096
> >
> > Indeed, I missed it.  Please install it into admin/gitmerge.el,
> 
> Done.

Thanks, but why on master and not on emacs-24?

Re: merging emacs-24

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Engster <deng@randomsample.de>
>> Date: Thu, 27 Nov 2014 18:27:48 +0100
>> Cc: Ted Zlatanov <tzz@lifelogs.com>, emacs-devel@gnu.org
>> 
>> Stefan Monnier writes:
>> >> Seems like you overlooked this:
>> >> http://article.gmane.org/gmane.emacs.devel/178096
>> >
>> > Indeed, I missed it.  Please install it into admin/gitmerge.el,
>> 
>> Done.
>
> Thanks, but why on master and not on emacs-24?

Why emacs-24?  It's not installed, so you need to load it manually
anyway.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: David Engster <deng@randomsample.de>,  tzz@lifelogs.com,  monnier@IRO.UMontreal.CA,  emacs-devel@gnu.org
> Date: Thu, 27 Nov 2014 19:31:30 +0100
> 
> > Thanks, but why on master and not on emacs-24?
> 
> Why emacs-24?

So that I have it no matter which branch is checked-out.

Re: merging emacs-24

[David Engster]

Eli Zaretskii writes:
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: David Engster <deng@randomsample.de>, tzz@lifelogs.com,
>> monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>> Date: Thu, 27 Nov 2014 19:31:30 +0100
>> 
>> > Thanks, but why on master and not on emacs-24?
>> 
>> Why emacs-24?
>
> So that I have it no matter which branch is checked-out.

You'll have to checkout master if you want to merge into it, so I did
not see the point putting it into emacs-24. If you feel strongly about
it, I can backport it.

-David

Re: merging emacs-24

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Andreas Schwab <schwab@linux-m68k.org>,  tzz@lifelogs.com,  monnier@IRO.UMontreal.CA,  emacs-devel@gnu.org
> Date: Thu, 27 Nov 2014 19:46:02 +0100
> 
> Eli Zaretskii writes:
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: David Engster <deng@randomsample.de>, tzz@lifelogs.com,
> >> monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> >> Date: Thu, 27 Nov 2014 19:31:30 +0100
> >> 
> >> > Thanks, but why on master and not on emacs-24?
> >> 
> >> Why emacs-24?
> >
> > So that I have it no matter which branch is checked-out.
> 
> You'll have to checkout master if you want to merge into it

We had bzrmerge on both branches.

> If you feel strongly about it, I can backport it.

If I'm the only one who needs that, I can solve my problem myself.

Re: merging emacs-24

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: Andreas Schwab <schwab@linux-m68k.org>, tzz@lifelogs.com,
>> monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>
>> Date: Thu, 27 Nov 2014 19:46:02 +0100
>> 
>> Eli Zaretskii writes:
>> >> From: Andreas Schwab <schwab@linux-m68k.org>
>> >> Cc: David Engster <deng@randomsample.de>, tzz@lifelogs.com,
>> >> monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>> >> Date: Thu, 27 Nov 2014 19:31:30 +0100
>> >> 
>> >> > Thanks, but why on master and not on emacs-24?
>> >> 
>> >> Why emacs-24?
>> >
>> > So that I have it no matter which branch is checked-out.
>> 
>> You'll have to checkout master if you want to merge into it
>
> We had bzrmerge on both branches.

Its initial commit was on trunk. 

>> If you feel strongly about it, I can backport it.
>
> If I'm the only one who needs that, I can solve my problem myself.

OK.

-David

Re: merging emacs-24

[Stefan Monnier]

> You'll have to checkout master if you want to merge into it, so I did
> not see the point putting it into emacs-24. If you feel strongly about
> it, I can backport it.

No, it's fine on master, thanks.


        Stefan

Re: merging emacs-24

[David Engster]

Stefan Monnier writes:
>> You'll have to checkout master if you want to merge into it, so I did
>> not see the point putting it into emacs-24. If you feel strongly about
>> it, I can backport it.
>
> No, it's fine on master, thanks.

I've also added some documentation in admin/notes/git-workflow.

-David

Re: merging emacs-24

[Ted Zlatanov]

On Thu, 27 Nov 2014 23:24:45 +0100 David Engster <deng@randomsample.de> wrote: 

DE> Stefan Monnier writes:
>>> You'll have to checkout master if you want to merge into it, so I did
>>> not see the point putting it into emacs-24. If you feel strongly about
>>> it, I can backport it.
>> 
>> No, it's fine on master, thanks.

DE> I've also added some documentation in admin/notes/git-workflow.

I think gitmerge.el is really clever :)  I learned a lot reading it.

One request: maybe it could log what it does, especially the shell
commands, a little better.

Ted

Re: merging emacs-24

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Nov 2014 19:28:22 -0500
> 
> One request: maybe it could log what it does, especially the shell
> commands, a little better.

It's actually something I sorely miss in Git itself: a full log of all
commands.  Bazaar has ~/.bzr.log, which over the years was invaluable
for tracking past problems and also my own mistakes (and mistakes made
by others) in using the tool.  I hope Git will get such a feature, or
maybe it already does.

Re: merging emacs-24

[Michael Welsh Duggan]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 27 Nov 2014 19:28:22 -0500
>> 
>> One request: maybe it could log what it does, especially the shell
>> commands, a little better.
>
> It's actually something I sorely miss in Git itself: a full log of all
> commands.  Bazaar has ~/.bzr.log, which over the years was invaluable
> for tracking past problems and also my own mistakes (and mistakes made
> by others) in using the tool.  I hope Git will get such a feature, or
> maybe it already does.

I experimented with setting the GIT_TRACE and GIT_TRACE_SETUP
environment variables to ~/.git.log.  On the upside, you get a log of
what was done.  On the downside, you get a log of just about
_everything_ that was done.  It is difficult to tell at a glance what
command you typed in versus what commands your command is running under
the hood.  (Though git is usually fast enough that looking at the
timestamps can give you a very good indication of what was manual versus
automated.)

It's definitely not as nice as .bzr.log, and the output is more tuned to
debugging git than remembering what you did.  But you should probably
try it and see if its output is useful for you.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: merging emacs-24

[Eli Zaretskii]

> From: Michael Welsh Duggan <mwd@md5i.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Nov 2014 09:20:38 -0500
> 
> I experimented with setting the GIT_TRACE and GIT_TRACE_SETUP
> environment variables to ~/.git.log.  On the upside, you get a log of
> what was done.  On the downside, you get a log of just about
> _everything_ that was done.  It is difficult to tell at a glance what
> command you typed in versus what commands your command is running under
> the hood.  (Though git is usually fast enough that looking at the
> timestamps can give you a very good indication of what was manual versus
> automated.)
> 
> It's definitely not as nice as .bzr.log, and the output is more tuned to
> debugging git than remembering what you did.  But you should probably
> try it and see if its output is useful for you.

Thanks, will do.

Re: merging emacs-24

[Michael Welsh Duggan]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Welsh Duggan <mwd@md5i.com>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 28 Nov 2014 09:20:38 -0500
>> 
>> I experimented with setting the GIT_TRACE and GIT_TRACE_SETUP
>> environment variables to ~/.git.log.  On the upside, you get a log of
>> what was done.  On the downside, you get a log of just about
>> _everything_ that was done.  It is difficult to tell at a glance what
>> command you typed in versus what commands your command is running under
>> the hood.  (Though git is usually fast enough that looking at the
>> timestamps can give you a very good indication of what was manual versus
>> automated.)
>> 
>> It's definitely not as nice as .bzr.log, and the output is more tuned to
>> debugging git than remembering what you did.  But you should probably
>> try it and see if its output is useful for you.
>
> Thanks, will do.

As an extra note: if you set GIT_TRACE_PERFORMANCE to the same log, you
a) get even more output!  (Not really what you want, but...)
b) You get output when the command ends that you can trace back to the
   beginning, allowing you to automate chunking the data.  For example,
   from running 'git fetch':

10:05:38.995341 trace.c:315             setup: git_dir: .git
10:05:38.995375 trace.c:316             setup: worktree: /usr/local/home/md5i/src/emacs/master
10:05:38.995379 trace.c:317             setup: cwd: /usr/local/home/md5i/src/emacs/master
10:05:38.995383 trace.c:318             setup: prefix: (null)
10:05:38.995388 git.c:349               trace: built-in: git 'fetch'
10:05:39.187844 run-command.c:341       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
10:05:39.188061 exec_cmd.c:134          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
10:05:39.188772 trace.c:315             setup: git_dir: .git
10:05:39.188794 trace.c:316             setup: worktree: /usr/local/home/md5i/src/emacs/master
10:05:39.188797 trace.c:317             setup: cwd: /usr/local/home/md5i/src/emacs/master
10:05:39.188800 trace.c:318             setup: prefix: (null)
10:05:39.188806 git.c:349               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
10:05:39.191515 trace.c:414             performance: 0.002864780 s: git command: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
10:05:39.192073 run-command.c:341       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all'
10:05:39.192224 exec_cmd.c:134          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all'
10:05:39.192836 trace.c:315             setup: git_dir: .git
10:05:39.192856 trace.c:316             setup: worktree: /usr/local/home/md5i/src/emacs/master
10:05:39.192860 trace.c:317             setup: cwd: /usr/local/home/md5i/src/emacs/master
10:05:39.192863 trace.c:318             setup: prefix: (null)
10:05:39.192868 git.c:349               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all'
10:05:39.195159 trace.c:414             performance: 0.002428034 s: git command: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all'
10:05:39.196573 run-command.c:341       trace: run_command: 'gc' '--auto'
10:05:39.196690 exec_cmd.c:134          trace: exec: 'git' 'gc' '--auto'
10:05:39.197279 trace.c:315             setup: git_dir: .git
10:05:39.197297 trace.c:316             setup: worktree: /usr/local/home/md5i/src/emacs/master
10:05:39.197301 trace.c:317             setup: cwd: /usr/local/home/md5i/src/emacs/master
10:05:39.197304 trace.c:318             setup: prefix: (null)
10:05:39.197309 git.c:349               trace: built-in: git 'gc' '--auto'
10:05:39.197418 trace.c:414             performance: 0.000230295 s: git command: 'git' 'gc' '--auto'
10:05:39.197509 trace.c:414             performance: 0.202293483 s: git command: 'git' 'fetch'



-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: merging emacs-24

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 27 Nov 2014 19:28:22 -0500
>> 
>> One request: maybe it could log what it does, especially the shell
>> commands, a little better.
>
> It's actually something I sorely miss in Git itself: a full log of all
> commands.  Bazaar has ~/.bzr.log, which over the years was invaluable
> for tracking past problems and also my own mistakes (and mistakes made
> by others) in using the tool.  I hope Git will get such a feature, or
> maybe it already does.

Have you tried "git reflog" ?

-- 
David Kastrup

Re: merging emacs-24

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 28 Nov 2014 15:37:32 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Ted Zlatanov <tzz@lifelogs.com>
> >> Date: Thu, 27 Nov 2014 19:28:22 -0500
> >> 
> >> One request: maybe it could log what it does, especially the shell
> >> commands, a little better.
> >
> > It's actually something I sorely miss in Git itself: a full log of all
> > commands.  Bazaar has ~/.bzr.log, which over the years was invaluable
> > for tracking past problems and also my own mistakes (and mistakes made
> > by others) in using the tool.  I hope Git will get such a feature, or
> > maybe it already does.
> 
> Have you tried "git reflog" ?

I'm actually using it all the time.  It's a misunderstanding if you
think "git reflog" is the answer to what I'm looking for.

~/.bzr.log shows:

  . every bzr command, including the date/time when it was issued
  . every message displayed by every command on the console, including
    for example received/transmitted statistics
  . additional messages, such as the branch on which the command was
    working
  . every error message, including Python backtrace
  . for pull/update/merge commands, the list of changes in the same
    format used by "bzr status" and "git status --short"
  . each record comes with a time stamp relative to the time the
    command was invoked

In addition, bzr automatically roll over the log file when it gets too
large.

Re: merging emacs-24

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Andreas Schwab <schwab@linux-m68k.org>,  tzz@lifelogs.com,  emacs-devel@gnu.org
> Date: Thu, 27 Nov 2014 23:24:45 +0100
> 
> Stefan Monnier writes:
> >> You'll have to checkout master if you want to merge into it, so I did
> >> not see the point putting it into emacs-24. If you feel strongly about
> >> it, I can backport it.
> >
> > No, it's fine on master, thanks.
> 
> I've also added some documentation in admin/notes/git-workflow.

Thanks.  I've now added to the Wiki a section about working with the
master and the release branch, and that includes instructions on using
gitmerge.el.

I think we no longer need admin/notes/git-workflow.  It gives the same
instructions as the Wiki, but slightly differently, which might
confuse.  I think we should have a single place for these
instructions.  If someone thinks we need the instructions in the
repository, I suggest to make a text copy of the Wiki.  Otherwise, I
think we should remove admin/notes/git-workflow.

Any objections?

Re: merging emacs-24

[Glenn Morris]

Eli Zaretskii wrote:

> I think we no longer need admin/notes/git-workflow.  It gives the same
> instructions as the Wiki, but slightly differently, which might
> confuse.  I think we should have a single place for these
> instructions.  If someone thinks we need the instructions in the
> repository, I suggest to make a text copy of the Wiki.  Otherwise, I
> think we should remove admin/notes/git-workflow.
>
> Any objections?

Yes, I object.
I don't see why this information, which is critical to Emacs
development, is maintained in an external third-party wiki. I can only
assume it's because everyone loves to talk about version control.
I think the wiki version is too long (because everyone loves to talk
about version control) and I prefer the simple version that Lars wrote.

So I agree there should be one place, but IMO it should be in
admin/notes, and it should not try to say everything.

Re: merging emacs-24

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: David Engster <deng@randomsample.de>,  tzz@lifelogs.com,  schwab@linux-m68k.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 29 Nov 2014 15:01:54 -0500
> 
> I don't see why this information, which is critical to Emacs
> development

Is it?

> is maintained in an external third-party wiki.

It's a bit late to object to that: the corresponding Wiki for bzr was
written in 2009.

> I can only assume it's because everyone loves to talk about version
> control.

I don't see why you'd assume that.  I worked on that on the assumption
that it will be useful, not because I love to talk about Git (I
don't).

> I think the wiki version is too long (because everyone loves to talk
> about version control) and I prefer the simple version that Lars wrote.
> 
> So I agree there should be one place, but IMO it should be in
> admin/notes, and it should not try to say everything.

"Say everything" is IMO the exaggeration of the year.  But whatever.

Re: merging emacs-24

[Ted Zlatanov]

On Wed, 26 Nov 2014 13:59:33 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Is it enough to merge laconically like you did:
SM> [...]
>> ...and resolve conflicts as needed?

SM> Yes.  Be careful with the conflicts, tho: some of those patches should
SM> simply not be applied (regardless if they'd create conflicts or not).

OK, I pushed my proposed merge to the repo in branch
master-merge-emacs-24-2014-11-26 for your review.

The major change I'm not sure about is from Oscar Fuentes,
courtesy-CC-ed here. I'd appreciate a review for the MinGW64 stuff which
I don't understand enough to merge properly. The rest seems OK.

Ted

Re: merging emacs-24

[Óscar Fuentes]

Ted Zlatanov <tzz@lifelogs.com> writes:

[snip]

> The major change I'm not sure about is from Oscar Fuentes,
> courtesy-CC-ed here. I'd appreciate a review for the MinGW64 stuff which
> I don't understand enough to merge properly. The rest seems OK.

The MinGW64 change is non-critical and easy to fix in case there is any
breakage. I'll take care of it as soon as it enters master.

Re: merging emacs-24

[Ted Zlatanov]

On Thu, 27 Nov 2014 03:01:33 +0100 Óscar Fuentes <ofv@wanadoo.es> wrote: 

ÓF> Ted Zlatanov <tzz@lifelogs.com> writes:
ÓF> [snip]

>> The major change I'm not sure about is from Oscar Fuentes,
>> courtesy-CC-ed here. I'd appreciate a review for the MinGW64 stuff which
>> I don't understand enough to merge properly. The rest seems OK.

ÓF> The MinGW64 change is non-critical and easy to fix in case there is any
ÓF> breakage. I'll take care of it as soon as it enters master.

All right, merge pushed. I've deleted the
master-merge-emacs-24-2014-11-26 branch as it wasn't needed anymore.

Ted

Re: merging emacs-24

[Glenn Morris]

Thanks for trying.

The ChangeLogs don't look right. All merged new entries should have gone
to the top (this was trivial with bzr's changelog-merge plugin, probably
git has something similar) and got today's date.

Also, the ChangeLogs for the changes marked "backport" (etc, check for
any commit that matches bzrmerge-skip-regexp) have all been merged back
to trunk, leading to duplicate entries. The new ones should all be
removed. This is part of what bzrmerge.el handled.


BTW: with bzr, the logs for the merged changes would be visible with

  bzr log -n0

How does one do that with git log?

Re: merging emacs-24

[Glenn Morris]

More worryingly, some commits appear to be missing?

Eg I don't see

http://lists.gnu.org/archive/html/emacs-diffs/2014-11/msg00426.html

on master.

Re: merging emacs-24

[Andreas Schwab]

Glenn Morris <rgm@gnu.org> writes:

> More worryingly, some commits appear to be missing?

They are yet to be merged.  The merge commit merged a 10 day old branch.
Try "git show-branch origin/master origin/emacs-24".

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Ted Zlatanov]

On Wed, 26 Nov 2014 22:48:12 -0500 Glenn Morris <rgm@gnu.org> wrote: 

GM> The ChangeLogs don't look right. All merged new entries should have gone
GM> to the top (this was trivial with bzr's changelog-merge plugin, probably
GM> git has something similar) and got today's date.

I used the recommended gnulib driver:

[merge "merge-changelog"]
        name = GNU-style ChangeLog merge driver
        driver = /usr/local/bin/git-merge-changelog %O %A %B

GM> Also, the ChangeLogs for the changes marked "backport" (etc, check for
GM> any commit that matches bzrmerge-skip-regexp) have all been merged back
GM> to trunk, leading to duplicate entries. The new ones should all be
GM> removed. This is part of what bzrmerge.el handled.

I don't think Git merges can work that way, though. They bring in the
whole branch, you can't exclude some commits. You have to either
cherry-pick the ones you want instead of merging, or revert the ones you
don't want after merging.

GM> BTW: with bzr, the logs for the merged changes would be visible with

GM>   bzr log -n0

GM> How does one do that with git log?

You do it to the two branches that were merged.  So e.g.

commit ba4502fe1465f7803beca3ae187e41f0b25bef10
Merge: b121ef1 81e0cca
Author: Ted Zlatanov <tzz@lifelogs.com>
Date:   Wed Nov 26 21:31:11 2014 -0500

    Merge branch 'emacs-24'

means you do `git log b121ef1..81e0cca'

On Thu, 27 Nov 2014 11:45:14 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Glenn Morris <rgm@gnu.org> writes:

>> More worryingly, some commits appear to be missing?

AS> They are yet to be merged.  The merge commit merged a 10 day old branch.
AS> Try "git show-branch origin/master origin/emacs-24".

I can't believe I did that.  Sorry.  It's not harmful, but I was
careless.  I've now added this alias to my gitconfig:

[alias]
        pull-all = !"old=$(git rev-parse --abbrev-ref HEAD) ; for b in $(git for-each-ref refs/heads --format='%(refname)') ; do git checkout ${b#refs/heads/} ; git pull --ff-only ; done; git checkout ${old}"

I can revert my merge commit for the reasons Andreas and Glenn listed,
or it can be patched up. Reverting is probably better to keep the
ChangeLogs clean. Let me know, or go ahead and revert directly (it will
have a few conflicts, nothing too bad).

Ted

Re: merging emacs-24

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Nov 2014 08:02:12 -0500
> 
> GM> BTW: with bzr, the logs for the merged changes would be visible with
> 
> GM>   bzr log -n0
> 
> GM> How does one do that with git log?
> 
> You do it to the two branches that were merged.  So e.g.
> 
> commit ba4502fe1465f7803beca3ae187e41f0b25bef10
> Merge: b121ef1 81e0cca
> Author: Ted Zlatanov <tzz@lifelogs.com>
> Date:   Wed Nov 26 21:31:11 2014 -0500
> 
>     Merge branch 'emacs-24'
> 
> means you do `git log b121ef1..81e0cca'

AFAIU, this only shows the merge-commits, but doesn't otherwise
identify the commits that were done on another branch.

Re: merging emacs-24

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> You do it to the two branches that were merged.  So e.g.
>> 
>> commit ba4502fe1465f7803beca3ae187e41f0b25bef10
>> Merge: b121ef1 81e0cca
>> Author: Ted Zlatanov <tzz@lifelogs.com>
>> Date:   Wed Nov 26 21:31:11 2014 -0500
>> 
>>     Merge branch 'emacs-24'
>> 
>> means you do `git log b121ef1..81e0cca'
>
> AFAIU, this only shows the merge-commits, but doesn't otherwise
> identify the commits that were done on another branch.

It shows all commits reachable by 81e0cca that are not reachable by
b121ef1, which means all commits that were merged in.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Thu, 27 Nov 2014 18:11:32 +0100
> Cc: emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> You do it to the two branches that were merged.  So e.g.
> >> 
> >> commit ba4502fe1465f7803beca3ae187e41f0b25bef10
> >> Merge: b121ef1 81e0cca
> >> Author: Ted Zlatanov <tzz@lifelogs.com>
> >> Date:   Wed Nov 26 21:31:11 2014 -0500
> >> 
> >>     Merge branch 'emacs-24'
> >> 
> >> means you do `git log b121ef1..81e0cca'
> >
> > AFAIU, this only shows the merge-commits, but doesn't otherwise
> > identify the commits that were done on another branch.
> 
> It shows all commits reachable by 81e0cca that are not reachable by
> b121ef1, which means all commits that were merged in.

Yes, it shows them.  But I said "identify", i.e. I meant there's
(AFAIK) no indication on each commit that "git log" shows which says
if that commit was on the branch or on master.  Unless you use some
non-default switches to 'log'.

If that's not true, how do I tell which commits in the linear list
shown by "git log" were made on master, and which were committed to
emacs-24 and later merged?

Re: merging emacs-24

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> If that's not true, how do I tell which commits in the linear list
> shown by "git log" were made on master, and which were committed to
> emacs-24 and later merged?

By looking from which branch they are reachable.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 27 Nov 2014 18:56:15 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If that's not true, how do I tell which commits in the linear list
> > shown by "git log" were made on master, and which were committed to
> > emacs-24 and later merged?
> 
> By looking from which branch they are reachable.

How do I do that?

Re: merging emacs-24

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 27 Nov 2014 18:56:15 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > If that's not true, how do I tell which commits in the linear list
>> > shown by "git log" were made on master, and which were committed to
>> > emacs-24 and later merged?
>> 
>> By looking from which branch they are reachable.
>
> How do I do that?

If "git merge-base A B" outputs (the SHA1 of) B then you know that B is
reachable by A.  Another way to visualize this is to use "git
show-branch A B".

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Nov 2014 09:33:47 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: emacs-devel@gnu.org
> >> Date: Thu, 27 Nov 2014 18:56:15 +0100
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > If that's not true, how do I tell which commits in the linear list
> >> > shown by "git log" were made on master, and which were committed to
> >> > emacs-24 and later merged?
> >> 
> >> By looking from which branch they are reachable.
> >
> > How do I do that?
> 
> If "git merge-base A B" outputs (the SHA1 of) B then you know that B is
> reachable by A.  Another way to visualize this is to use "git
> show-branch A B".

Yes, OK.  Thanks.  I think "git log --graph" and "C-x v L" are also
fine.

My point was that just "git log" does not provide that info, although
(unlike bzr) it does by default show the commits from merged branches.

Re: merging emacs-24

[Sergey Organov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 28 Nov 2014 09:33:47 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andreas Schwab <schwab@linux-m68k.org>
>> >> Cc: emacs-devel@gnu.org
>> >> Date: Thu, 27 Nov 2014 18:56:15 +0100
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> > If that's not true, how do I tell which commits in the linear list
>> >> > shown by "git log" were made on master, and which were committed to
>> >> > emacs-24 and later merged?
>> >> 
>> >> By looking from which branch they are reachable.
>> >
>> > How do I do that?
>> 
>> If "git merge-base A B" outputs (the SHA1 of) B then you know that B is
>> reachable by A.  Another way to visualize this is to use "git
>> show-branch A B".
>
> Yes, OK.  Thanks.  I think "git log --graph" and "C-x v L" are also
> fine.

As was already said in earlier discussions, something like this is
probably most close to what you seem to expect from "git log":

$ git log --source --oneline orignin/master origin/emacs-24

> My point was that just "git log" does not provide that info,

Sure it doesn't provide that info by default. "git log" is simply
"starting from HEAD, go back through the DAG and show every commit." To
be fast, it doesn't even try (by default) to figure out from which
/other/ references any of listed commits are reachable.

> although (unlike bzr) it does by default show the commits from merged
> branches.

The fact that it "shows the commits from merged brahcnes" is a
side-effect of its simple default definition of showing all reachable
commits. That, e.g., makes it trivial (and fast) to see if given commit,
say git:ef97653, is "included" in any given release, say, rel-3.4.8:

$ git log --oneline rel-3.4.8 | grep ef97653

-- 
Sergey.

Re: merging emacs-24

[David Kastrup]

Sergey Organov <sorganov@gmail.com> writes:

> The fact that it "shows the commits from merged brahcnes" is a
> side-effect of its simple default definition of showing all reachable
> commits. That, e.g., makes it trivial (and fast) to see if given commit,
> say git:ef97653, is "included" in any given release, say, rel-3.4.8:
>
> $ git log --oneline rel-3.4.8 | grep ef97653

This can be done quite more efficiently.

git log -1 --exit-code rel-3.4.8..ef97653 > /dev/null


-- 
David Kastrup

Re: merging emacs-24

[Stefan Monnier]

GM> Also, the ChangeLogs for the changes marked "backport" (etc, check for
GM> any commit that matches bzrmerge-skip-regexp) have all been merged back
GM> to trunk, leading to duplicate entries. The new ones should all be
GM> removed. This is part of what bzrmerge.el handled.
> I don't think Git merges can work that way, though.

Glenn is talking about the content of the ChangeLog files.  To Git
(just as Bzr before) they're just files, not metadata, so of course you
can add/remove any entry you feel like.


        Stefan

Re: merging emacs-24

[David Engster]

Ted Zlatanov writes:
> I don't think Git merges can work that way, though. They bring in the
> whole branch, you can't exclude some commits.

You cannot exclude them, but you can merge them with merge strategy
'ours'. That's what gitmerge.el does.

> You have to either cherry-pick the ones you want instead of merging,

Which defies the point of merging, of course.

> or revert the ones you don't want after merging.

That's cumbersome when those commits have conflicts, which is very
common since it is often the reason they should be skipped in the first
place.

> I can't believe I did that.  Sorry.  It's not harmful, but I was
> careless.  I've now added this alias to my gitconfig:

I think the easiest way to avoid this is to merge the remote branch
instead of the local tracking one.

-David

Re: merging emacs-24

[Andreas Schwab]

David Engster <deng@randomsample.de> writes:

> Ted Zlatanov writes:
>> I can't believe I did that.  Sorry.  It's not harmful, but I was
>> careless.  I've now added this alias to my gitconfig:
>
> I think the easiest way to avoid this is to merge the remote branch
> instead of the local tracking one.

That should always be done anyway, since that makes sure you only merge
things that are already pushed.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Ted Zlatanov]

On Thu, 27 Nov 2014 18:32:54 +0100 David Engster <deng@randomsample.de> wrote: 

DE> Ted Zlatanov writes:
>> I don't think Git merges can work that way, though. They bring in the
>> whole branch, you can't exclude some commits.

DE> You cannot exclude them, but you can merge them with merge strategy
DE> 'ours'. That's what gitmerge.el does.

Yup, I see that.  Lovely; I had always applied the strategy to the whole
commit before.

>> I can't believe I did that.  Sorry.  It's not harmful, but I was
>> careless.  I've now added this alias to my gitconfig:

DE> I think the easiest way to avoid this is to merge the remote branch
DE> instead of the local tracking one.

Oh, that's very helpful. I have always had to remember to update the
local branch and then merge from it.  Thanks for the hints.

Ted

Re: merging emacs-24

[Stephen J. Turnbull]

David Engster writes:

 > > You have to either cherry-pick the ones you want instead of merging,
 > 
 > Which defies the point of merging, of course.
 > 
 > > or revert the ones you don't want after merging.
 > 
 > That's cumbersome when those commits have conflicts, which is very
 > common since it is often the reason they should be skipped in the first
 > place.

Sure, but *that is the choice*.  If you look at merge implementations
that claim to be able to skip commits, you will find that they amount
to "git rebase --interactive": ie, a sequence of *new* commits made
from diffs of the commits chosen.  If they're diligent, they may
identify an initial prefix of commits that can be merged unchanged,
but if they do, the resulting DAG will have two merges, which is
rarely what the user expects.

 > I think the easiest way to avoid this is to merge the remote branch
 > instead of the local tracking one.

It is mathematically impossible to merge a remote branch, at least if
you want a record of the history of commits in the local repo.[1]  A
merge always takes place in the repo being operated on, and the
relevant commits must be present there.  The difference between git
and other VCSes is that git leaves the copy of the remote branch and a
ref to it in the current repo, which is necessary if you want to
compare the actual history to the history constructed by a "merge"
which skips commits.  (Of course in the case of a true merge, the
tracking branch is only a ref -- there are no commits other than those
needed to complete the local branch in it.  In this case all of git,
hg, and bzr contain the complete remote branch in the local repo.)

Footnotes: 
[1]  That is, technically you could replace an actual commit with a
URL to a repo containing the needed commit and all dependent data, and
destroy all the corresponding data needed to construct conflicts after
the merge.  This is more or less a "shallow" repo in git, and it's
impossible to do operations on "sufficiently old" commits with a
shallow repo in git, and AFAIK it's not possible to deepen the repo --
you have reclone and then merge or even cherrypick new commits.

Re: merging emacs-24

[Stephen J. Turnbull]

Stephen J. Turnbull writes:
 > David Engster writes:

 >  > I think the easiest way to avoid this is to merge the remote branch
 >  > instead of the local tracking one.
 > 
 > It is mathematically impossible to merge a remote branch,

Oops, did you mean "git pull" here?

Re: merging emacs-24

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Fri, 28 Nov 2014 12:29:02 +0900
> 
> Stephen J. Turnbull writes:
>  > David Engster writes:
> 
>  >  > I think the easiest way to avoid this is to merge the remote branch
>  >  > instead of the local tracking one.
>  > 
>  > It is mathematically impossible to merge a remote branch,
> 
> Oops, did you mean "git pull" here?

I think he meant merging origin/emacs-24 after "git pull", as opposed
to merging emacs-24 (which requires "git checkout" followed by "git pull").

Re: merging emacs-24

[Ted Zlatanov]

On Thu, 27 Nov 2014 08:02:12 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I can revert my merge commit for the reasons Andreas and Glenn listed,
TZ> or it can be patched up. Reverting is probably better to keep the
TZ> ChangeLogs clean. Let me know, or go ahead and revert directly (it will
TZ> have a few conflicts, nothing too bad).

Maybe this question was lost in the helpful discussion of merging, but
it's only going to get harder to revert my "stupid" commit as more
commits pile on.

Ted

Re: merging emacs-24

[Paul Eggert]

Ted Zlatanov wrote:
> it's only going to get harder to revert my "stupid" commit as more
> commits pile on

I attempted to finish up the merge, and bring it up to date with the latest 
master and emacs-24 branches, without reverting anything (in the git sense), as 
git commit 0cce3623b169732a51f055a86fc926313b11a5ee on the master branch.  I did 
this by hand.  Please let me know of any errors.

Re: merging emacs-24

[Glenn Morris]

Backported changes were again duplicated in the ChangeLog.
(Please could whoever does this next keep this in mind.)
Jan's 2014-11-15 emacs-24 change to nsterm.m was duplicated and
somehow attributed to Ulf Jasper.

Re: generating Chan(was: merging emacs-24)

[Paul Eggert]

Glenn Morris wrote:
> Backported changes were again duplicated in the ChangeLog.
> (Please could whoever does this next keep this in mind.)

Sorry, that's my mistake; I got confused when trying to repair the confusion 
caused by the earlier incomplete merge from emacs-24.  Thanks for fixing it.

Re: merging emacs-24

[Ted Zlatanov]

On Thu, 27 Nov 2014 08:02:12 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Wed, 26 Nov 2014 22:48:12 -0500 Glenn Morris <rgm@gnu.org> wrote: 
GM> The ChangeLogs don't look right. All merged new entries should have gone
GM> to the top (this was trivial with bzr's changelog-merge plugin, probably
GM> git has something similar) and got today's date.

TZ> I used the recommended gnulib driver:

TZ> [merge "merge-changelog"]
TZ>         name = GNU-style ChangeLog merge driver
TZ>         driver = /usr/local/bin/git-merge-changelog %O %A %B

I'm not sure if this means that `git-merge-changelog' doesn't work the
way the Emacs maintainers want, or that I used it incorrectly. Hints
welcome. Most importantly, I need to know if I leave this in my
gitconfig but try to use gitmerge.el, the result will be bad from the
maintainers' perspective.

Ted

Re: merging emacs-24

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> TZ> I used the recommended gnulib driver:
>
> TZ> [merge "merge-changelog"]
> TZ>         name = GNU-style ChangeLog merge driver
> TZ>         driver = /usr/local/bin/git-merge-changelog %O %A %B
>
> I'm not sure if this means that `git-merge-changelog' doesn't work the
> way the Emacs maintainers want, or that I used it incorrectly. Hints
> welcome.

You need to add this to .git/info/attributes:

ChangeLog*	merge=merge-changelog

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: merging emacs-24

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Nov 2014 19:33:50 -0500
> 
> TZ> [merge "merge-changelog"]
> TZ>         name = GNU-style ChangeLog merge driver
> TZ>         driver = /usr/local/bin/git-merge-changelog %O %A %B
> 
> I'm not sure if this means that `git-merge-changelog' doesn't work the
> way the Emacs maintainers want, or that I used it incorrectly. Hints
> welcome. Most importantly, I need to know if I leave this in my
> gitconfig but try to use gitmerge.el, the result will be bad from the
> maintainers' perspective.

I think you should keep that entry, but you need to examine the
results in ChangeLog anyway.  git-merge-changelog does try to be smart
about where to put merged ChangeLog entries, but it cannot always do a
perfect job.  AFAICS, gitmerge.el doesn't fix that (maybe it should
try), so manual labor is still required.

Re: merging emacs-24

[Stefan Monnier]

> perfect job.  AFAICS, gitmerge.el doesn't fix that (maybe it should
> try), so manual labor is still required.

I haven't checked gitmerge.el, but bzrmerge.el had special code to do
the ChangeLog merge, and at some point at least it did it better (or at
least, more closely to our conventions) than Bzr's
"changelog-merge" plugin.


        Stefan

Re: merging emacs-24

[Glenn Morris]

Stefan Monnier wrote:

> I haven't checked gitmerge.el, but bzrmerge.el had special code to do
> the ChangeLog merge, and at some point at least it did it better (or at
> least, more closely to our conventions) than Bzr's
> "changelog-merge" plugin.

I think I disagree.
IIRC, bzrmerge used to look for ChangeLog conflicts, move those entries
to the top and redate them. I found that this did not work well for me
in practice. I found it better to use the changelog-merge plugin,
which moved new merged entried to the top, then re-date things by hand.

Re: merging emacs-24

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Wed, 26 Nov 2014 22:48:12 -0500
> 
> 
> BTW: with bzr, the logs for the merged changes would be visible with
> 
>   bzr log -n0
> 
> How does one do that with git log?

Git shows that by default, but it doesn't by default indent the merged
commits, or give any other hint that they are from another branch.  To
see that, I suggest to use "git log --graph" or "C-x v L".

Re: merging emacs-24

[Glenn Morris]

Eli Zaretskii wrote:

>> BTW: with bzr, the logs for the merged changes would be visible with
>> 
>>   bzr log -n0
>> 
>> How does one do that with git log?
>
> Git shows that by default

Oh, I see. The order in which it shows things seems weird to me.
(No need for anyone to try to explain/justify it to me. :) )

>, but it doesn't by default indent the merged commits, or give any
>other hint that they are from another branch. To see that, I suggest to
>use "git log --graph" or "C-x v L".

Thanks, I suppose --graph will do.

bug#19074: Bug in auth-source.el's search of OS X Keychain

[Katsumi Yamaoka]

On Wed, 26 Nov 2014 09:17:22 -0500, Ted Zlatanov wrote:
> Thanks, applied to emacs-24 branch as a bugfix:
[...]
>     auth-source: Fix Mac OS X keychain lookups.

>     * auth-source.el (auth-source-macos-keychain-search-items): Return
>     result of `auth-source-macos-keychain-result-append' (bug#19074).

> It will eventually get ported to Emacs master and to Gnus master as
> well.  Glenn or Stefan, should I do that or wait for you?

I've installed the fix in the Gnus master in advance of merging
it to the Emacs master. ;)  Thanks.

bug#19074: Bug in auth-source.el's search of OS X Keychain

[Ted Zlatanov]

On Thu, 27 Nov 2014 07:30:12 +0900 Katsumi Yamaoka <yamaoka@jpl.org> wrote: 

KY> On Wed, 26 Nov 2014 09:17:22 -0500, Ted Zlatanov wrote:
>> Thanks, applied to emacs-24 branch as a bugfix:
...
>> It will eventually get ported to Emacs master and to Gnus master as
>> well.  Glenn or Stefan, should I do that or wait for you?

KY> I've installed the fix in the Gnus master in advance of merging
KY> it to the Emacs master. ;)  Thanks.

Excellent. I started with emacs-24 because I wanted to make sure it
showed up in the next 24.x release and we're asked to make changes there
before master/trunk, that's why this commit was a bit unusual.

Thanks for your help!
Ted