My plans for VC mode

My plans for VC mode

[Unknown]

I have noted in previous mail that the VC code lacks a clean set of
primitives to deal with branching.  My goal is to fix that.

In the way is the fact that the VC backend API is still something of
a mess.  Everyone who has previously modified it (including me in 2007
when I did the fileset rewrite) has tried to preserve backward
compatibility to the year zero.  As a result, the upper-level code
has never gotten completely divorced from the file-oriented back ends.

The merge command is a particular trouble spot where the back-end model
pokes upward into what should be VCS-independent - as is revealed
by the ";; FIXME: This is CVS/RCS/SCCS specific." in vc/vc.el.
Because of the code that calls out, SVN merge in particular plain
doesn't work right - it assumes RCS behavior in SVN revision numbers.

A related effect of trying to preserve backward compatibility is that
the backend API has more entry points than it should and is harder to
understand than it should be. I had my nose rubbed in this writing the
support for SRC. Given the advantages of having written both VC mode
and SRC and the freedom to modify SRC to make VC-mode happy, it was
*way* more difficult than it should have been.

Accordingly, I have concluded that it is time to chuck backward
compatibility and rewrite tha back-end API, simplifying as radically
as I can and enforcing much stricter separation between the VC upper
layer and the back ends.

This is going to break out-of-tree back ends - not in any way that is
fundamentally difficult to fix, but it will break them. I've already had
one polite complaint about that in email, but explained that we can't
be stuck with the cruft and the previous design mistakes forever.

I'm explaining this, while "make bootstrap" runs to test a build,
because it may cause some minor disruptions.  I'm a little worried
about breaking CVS support; that back end is both the worst hairball
in the file-oriented group (SCCS/RCS/CVS/SRC) and the one I'm least
able to test well.  I could use some help with this if there is
anyone on the list using vc-cvs regularly.

While I do branching I'm also going to look at removing the elaborate
caching the older back ends use to avoid disk traffic as much as
possible.  That made sense when I first wrote it twenty-odd years
ago, but it's been a chronic source of TOCTOU bugs and other
coherency problems ever since.  Disks are much faster now; it's
probably time for the state-heuristic stuff to die.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

The saddest life is that of a political aspirant under democracy. His
failure is ignominious and his success is disgraceful.
        -- H.L. Mencken

Re: My plans for VC mode

[Jan D.]

Hi.

Do you have any plans for async checkin/commit from vc?

	Jan D.

> 22 nov 2014 kl. 14:33 skrev Eric S. Raymond <esr@snark.se>:
> 
> I have noted in previous mail that the VC code lacks a clean set of
> primitives to deal with branching.  My goal is to fix that.
> 
> In the way is the fact that the VC backend API is still something of
> a mess.  Everyone who has previously modified it (including me in 2007
> when I did the fileset rewrite) has tried to preserve backward
> compatibility to the year zero.  As a result, the upper-level code
> has never gotten completely divorced from the file-oriented back ends.
> 
> The merge command is a particular trouble spot where the back-end model
> pokes upward into what should be VCS-independent - as is revealed
> by the ";; FIXME: This is CVS/RCS/SCCS specific." in vc/vc.el.
> Because of the code that calls out, SVN merge in particular plain
> doesn't work right - it assumes RCS behavior in SVN revision numbers.
> 
> A related effect of trying to preserve backward compatibility is that
> the backend API has more entry points than it should and is harder to
> understand than it should be. I had my nose rubbed in this writing the
> support for SRC. Given the advantages of having written both VC mode
> and SRC and the freedom to modify SRC to make VC-mode happy, it was
> *way* more difficult than it should have been.
> 
> Accordingly, I have concluded that it is time to chuck backward
> compatibility and rewrite tha back-end API, simplifying as radically
> as I can and enforcing much stricter separation between the VC upper
> layer and the back ends.
> 
> This is going to break out-of-tree back ends - not in any way that is
> fundamentally difficult to fix, but it will break them. I've already had
> one polite complaint about that in email, but explained that we can't
> be stuck with the cruft and the previous design mistakes forever.
> 
> I'm explaining this, while "make bootstrap" runs to test a build,
> because it may cause some minor disruptions.  I'm a little worried
> about breaking CVS support; that back end is both the worst hairball
> in the file-oriented group (SCCS/RCS/CVS/SRC) and the one I'm least
> able to test well.  I could use some help with this if there is
> anyone on the list using vc-cvs regularly.
> 
> While I do branching I'm also going to look at removing the elaborate
> caching the older back ends use to avoid disk traffic as much as
> possible.  That made sense when I first wrote it twenty-odd years
> ago, but it's been a chronic source of TOCTOU bugs and other
> coherency problems ever since.  Disks are much faster now; it's
> probably time for the state-heuristic stuff to die.
> -- 
> 		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
> 
> The saddest life is that of a political aspirant under democracy. His
> failure is ignominious and his success is disgraceful.
>        -- H.L. Mencken

Re: My plans for VC mode

[Dmitry Gutov]

Hi Eric,

> Accordingly, I have concluded that it is time to chuck backward
> compatibility and rewrite tha back-end API, simplifying as radically
> as I can and enforcing much stricter separation between the VC upper
> layer and the back ends.

Here's hoping it won't be a full rewrite. Aside from serving as a
plumbing for a set of commands, VC is a helpful VCS-agnostic (to a
certain degree) wrapper over a set of commands which third-party
packages can use.

http://elpa.gnu.org/packages/diff-hl.html, for example, uses `vc-state',
as well as `diff', `dir-status' and `dir-status-files', and relies on
their current semantics (even if they're a bit broken in certain
implementations; I intend to improve that).

> While I do branching I'm also going to look at removing the elaborate
> caching the older back ends use to avoid disk traffic as much as
> possible.  That made sense when I first wrote it twenty-odd years
> ago, but it's been a chronic source of TOCTOU bugs and other
> coherency problems ever since.  Disks are much faster now; it's
> probably time for the state-heuristic stuff to die.

Not exactly sure which functions you're referring to here, but an
example of a function in a "newer" backend that uses caching is
`vc-git-root'.

Correctness should trump performance most of the time, but there are
still traffic-sensitive workflows out there, such as TRAMP.

Re: My plans for VC mode

[Stefan Monnier]

> This is going to break out-of-tree back ends - not in any way that is
> fundamentally difficult to fix, but it will break them.  I've already had
> one polite complaint about that in email, but explained that we can't
> be stuck with the cruft and the previous design mistakes forever.

I'm fine with breaking backward compatibility.  But then please try to
find "all" the out-of-tree backends for Free revision control systems,
and then try to find their maintainer to get them in the loop.
If they're unmaintained, then we should post a message to the
corresponding revision system's mailing-list telling them about the
expected breakage.

If you're about to redesign the backend API, I'll take the opportunity
to include my pet wishitems for VC which I never got to address:
- async operation for most operations (including "commit" and "status").
- generic support for shelve/stash.


        Stefan

Re: My plans for VC mode

[Eric S. Raymond]

Jan D. <jan.h.d@swipnet.se>:
> Do you have any plans for async checkin/commit from vc?

Not yet.  How do you want this to work?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Eric S. Raymond]

Dmitry Gutov <dgutov@yandex.ru>:
> http://elpa.gnu.org/packages/diff-hl.html, for example, uses `vc-state',
> as well as `diff', `dir-status' and `dir-status-files', and relies on
> their current semantics (even if they're a bit broken in certain
> implementations; I intend to improve that).

vc-state is likely to surivive with unaltered semantics.  So is diff.

`dir-status' and `dir-status-files', on the other hand...there is a comment
in vc.el that reads:

    ;; - Another important thing: merge all the status-like backend operations.
    ;;   We should remove dir-status, state, and dir-status-files, and
    ;;   replace them with just `status' which takes a fileset and a continuation
    ;;   (like dir-status) and returns a buffer in which the process(es) are run
    ;;   (or nil if it worked synchronously).

I think this is a good thing to aim at.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Eric S. Raymond]

Stefan Monnier <monnier@IRO.UMontreal.CA>:
> I'm fine with breaking backward compatibility.  But then please try to
> find "all" the out-of-tree backends for Free revision control systems,
> and then try to find their maintainer to get them in the loop.

I'd be happy to get the maintainers involved, but I don't have any idea
how to go about finding the out-of-tree back ends.  Help would be
appreciated.

> If you're about to redesign the backend API, I'll take the opportunity
> to include my pet wishitems for VC which I never got to address:
> - async operation for most operations (including "commit" and "status").
> - generic support for shelve/stash.

I think shelve-stash will be dead easy.

You're the second person to bring up async operation.  I don't have
a clear idea of what that means.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Jan D.]

Hi.

> 22 nov 2014 kl. 17:29 skrev Eric S. Raymond <esr@thyrsus.com>:
> 
> Jan D. <jan.h.d@swipnet.se>:
>> Do you have any plans for async checkin/commit from vc?
> 
> Not yet.  How do you want this to work?

Just mindstorming here, but I would expect something like make/grep, i.e. the output of the command goes to another buffer and you can continue working in other buffers.

Come to think of it, it would be nice for other commands like vc-dir, which I see lock up Emacs when working with slow connections to non-local repositories.  If this could be generalized,
i.e. if all vc-commands would be asynchronous it would improve the user experience.

	Jan D.

Re: My plans for VC mode

[Dieter Deyke]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> I'd be happy to get the maintainers involved, but I don't have any idea
> how to go about finding the out-of-tree back ends.  Help would be
> appreciated.

I use:

;;; vc-fossil.el --- VC backend for the fossil sofware configuraiton management system
;; Author: Venkat Iyer <venkat@comit.com>

;;; Commentary:

;; This file contains a VC backend for the fossil version control
;; system.

--
Dieter Deyke
mailto:dieter.deyke@gmail.com
GPG Key fingerprint = 70E8 D998 0E4D 6087 BE05 099B 50F4 BF3D B116 EA20

Re: My plans for VC mode

[Dmitry Gutov]

On 11/22/2014 06:37 PM, Eric S. Raymond wrote:

> vc-state is likely to surivive with unaltered semantics.  So is diff.

Ok. I'm also using `vc-diff-internal', `vc-buffer-sync' and 
`vc-exec-after', but these seem to be less relevant to the current subject.

> `dir-status' and `dir-status-files', on the other hand...there is a comment
> in vc.el that reads:
>
>      ;; - Another important thing: merge all the status-like backend operations.
>      ;;   We should remove dir-status, state, and dir-status-files, and
>      ;;   replace them with just `status' which takes a fileset and a continuation
>      ;;   (like dir-status) and returns a buffer in which the process(es) are run
>      ;;   (or nil if it worked synchronously).
>
> I think this is a good thing to aim at.

The above list includes `state', which you're saying won't be changed 
much. Other than that, do you mean `dir-status' and `dir-status-files' 
will be merged?

If that means that the new function will behave just like `dir-status' 
when passed a nil (or a directory) instead of a list of files, then how 
that's much better than keeping the functions separate? The downside is 
obvious, meanwhile (API breakage).

Re: My plans for VC mode

[Eric S. Raymond]

Dmitry Gutov <dgutov@yandex.ru>:
> On 11/22/2014 06:37 PM, Eric S. Raymond wrote:
> 
> >vc-state is likely to surivive with unaltered semantics.  So is diff.
> 
> Ok. I'm also using `vc-diff-internal', `vc-buffer-sync' and
> `vc-exec-after', but these seem to be less relevant to the current
> subject.

Agreed.

> The above list includes `state', which you're saying won't be
> changed much. Other than that, do you mean `dir-status' and
> `dir-status-files' will be merged?

That seems extremely likely. Unless I run out of time to work on this
before I get to it.

> If that means that the new function will behave just like
> `dir-status' when passed a nil (or a directory) instead of a list of
> files, then how that's much better than keeping the functions
> separate? The downside is obvious, meanwhile (API breakage).

It's better than keeping the functions separate because the
separation contributes in a significant way to making the
interface a hairball that is difficult to understand and maintain.

I should have been able to write SRC support in 90 minutes.  The fact
that it took the inventor of both SRC and VC mode *two days* told
me the API has gone very badly wrong.

It is obvious why; we have been succumbing to the temptation to just
glue on another knob for far too long.  Ideally, I would like to
*halve* the number of entry points. And ensure that each one has
simple, regular semantics that can be described in one or two
sentences.  Backward compatibility can go straight to the hell where
it belongs.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Michael Albinus]

> Accordingly, I have concluded that it is time to chuck backward
> compatibility and rewrite tha back-end API, simplifying as radically
> as I can and enforcing much stricter separation between the VC upper
> layer and the back ends.

Do you also plan to change vc-registered? This supports file name
handlers, and (at least) Tramp would be affected.

Best regards, Michael.

Re: My plans for VC mode

[Stefan Monnier]

> The above list includes `state', which you're saying won't be changed
> much.

He's saying vc-state won't be changed too much.  That's different from
saying that the `state' backend operation won't be changed too much.


        Stefan

Re: My plans for VC mode

[Stefan Monnier]

> I'd be happy to get the maintainers involved, but I don't have any idea
> how to go about finding the out-of-tree back ends.  Help would be
> appreciated.

I'd look at the list of revision control systems on wikipedia, ignoring
the non-free ones and then search for a corresponfing vc-<system>.el.

AFAICT that's basically, Fossil, Codeville, and DaRCS (since OpenCM
seems to be dead).


        Stefan

Re: My plans for VC mode

[Steinar Bang]

>>>>> Dmitry Gutov <dgutov@yandex.ru>:

> Hi Eric,
>> Accordingly, I have concluded that it is time to chuck backward
>> compatibility and rewrite tha back-end API, simplifying as radically
>> as I can and enforcing much stricter separation between the VC upper
>> layer and the back ends.

> Here's hoping it won't be a full rewrite.

Well, whether there will be a rewrite or not, here's a bug I would very
much like to see fixed: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8756

Ie. make vc-git.el's 'C-x v l' track history across renames (and make
stuff like annotate and diff from hashes work past renames).

Re: My plans for VC mode

[Dmitry Gutov]

On 11/23/2014 12:32 AM, Stefan Monnier wrote:

> He's saying vc-state won't be changed too much.  That's different from
> saying that the `state' backend operation won't be changed too much.

True enough. But the other two backend functions work asynchronously.

If `state' also becomes asynchronous, won't it make hard to keep 
`vc-state' synchronous?

Re: My plans for VC mode

[Dmitry Gutov]

On 11/22/2014 08:08 PM, Eric S. Raymond wrote:

> It's better than keeping the functions separate because the
> separation contributes in a significant way to making the
> interface a hairball that is difficult to understand and maintain.

Would that make the combined function's description shorter than the 
current `dir-status' one, in vc.el header? If yes, how?

I suspect that would make the details from `dir-status-files' get lost 
in the noise, whereas I like its "report on all requested files" 
semantics (here's why: 
https://github.com/dgutov/diff-hl/issues/25#issuecomment-57107335).

Even now, when the description is separate, almost every backend I tried 
currently fails to adhere to it in some respect, see 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18579#5 and 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18579#59.

> I should have been able to write SRC support in 90 minutes.  The fact
> that it took the inventor of both SRC and VC mode *two days* told
> me the API has gone very badly wrong.

Some things are just hard. And, backwards compatibility aside, usually 
we don't want to discard existing features.

Re: My plans for VC mode

[Eric S. Raymond]

Stefan Monnier <monnier@IRO.UMontreal.CA>:
> > The above list includes `state', which you're saying won't be changed
> > much.
> 
> He's saying vc-state won't be changed too much.  That's different from
> saying that the `state' backend operation won't be changed too much.

Yes.  I think the inentory of states it returns is reasonable.  Though
I will try to reduce it if I can find a way.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Eric S. Raymond]

Michael Albinus <michael.albinus@gmx.de>:
> 
> > Accordingly, I have concluded that it is time to chuck backward
> > compatibility and rewrite tha back-end API, simplifying as radically
> > as I can and enforcing much stricter separation between the VC upper
> > layer and the back ends.
> 
> Do you also plan to change vc-registered? This supports file name
> handlers, and (at least) Tramp would be affected.

I don't know for sure yet, but that seems to me to be one of the
least likely pieces to change.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Michael Albinus]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> I don't know for sure yet, but that seems to me to be one of the
> least likely pieces to change.

Thanks, this makes me more relaxed :-)

Best regards, Michael.

Re: My plans for VC mode

[Ivan Shmakov]

>>>>> Eric S Raymond <esr@thyrsus.com> writes:
>>>>> Stefan Monnier <monnier@IRO.UMontreal.CA>:

 >> I'm fine with breaking backward compatibility.  But then please try
 >> to find "all" the out-of-tree backends for Free revision control
 >> systems, and then try to find their maintainer to get them in the
 >> loop.

 > I'd be happy to get the maintainers involved, but I don't have any
 > idea how to go about finding the out-of-tree back ends.  Help would
 > be appreciated.

	FWIW, as a maintainer of vc-mw.el, I’m already following this
	thread (albeit loosely.)

	Sure, I’m all for simplifying the VC-to-backend interface, yet
	I’m somewhat concerned over whether I’d be able to provide
	support for /both/ Emacs 24 and the Git ‘master’ in my code.

	At the very least, I hope that the calling conventions for the
	most basic interface functions my backend provides (including
	registered, state, working-revision, find-revision, checkin,
	print-log) are changed in a kind of a compatible way.  (For
	instance, I could easily make ‘state’ also accept a list of
	files instead of just a single one.  Or so I hope, anyway.)

	Otherwise, I’d rather appreciate if the updated interface could
	use some entirely new function names where the semantics changes
	drastically, so that I can provide functions implementing both
	the old and the new interface.  (I do not think it’s necessary
	for the new VC to support the old interface in this case.)

[…]

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: My plans for VC mode

[Eric S. Raymond]

Stefan Monnier <monnier@IRO.UMontreal.CA>:
> I'd look at the list of revision control systems on wikipedia, ignoring
> the non-free ones and then search for a corresponfing vc-<system>.el.
> 
> AFAICT that's basically, Fossil, Codeville, and DaRCS (since OpenCM
> seems to be dead).

I've sent email to the maintainer of the Fossil back end to bring
him in on the process.  Google turns up no evidence of modes for
DaRCS or Codeville.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Eric S. Raymond]

Steinar Bang <sb@dod.no>:
> Well, whether there will be a rewrite or not, here's a bug I would very
> much like to see fixed: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8756
> 
> Ie. make vc-git.el's 'C-x v l' track history across renames (and make
> stuff like annotate and diff from hashes work past renames).

That would be *quite* hard, and is out of scope of my plans.  The
extent to which VCSes make it even possible to track across renames
varies a lot; the mechanisms for it, where they exist, are complex and
tend to be flaky.

Even if we restrict attention to git, log --follow only works on
single files...
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Eric S. Raymond]

Ivan Shmakov <ivan@siamics.net>:
> 	At the very least, I hope that the calling conventions for the
> 	most basic interface functions my backend provides (including
> 	registered, state, working-revision, find-revision, checkin,
> 	print-log) are changed in a kind of a compatible way.  (For
> 	instance, I could easily make ‘state’ also accept a list of
> 	files instead of just a single one.  Or so I hope, anyway.)
> 
> 	Otherwise, I’d rather appreciate if the updated interface could
> 	use some entirely new function names where the semantics changes
> 	drastically, so that I can provide functions implementing both
> 	the old and the new interface.  (I do not think it’s necessary
> 	for the new VC to support the old interface in this case.)

These are completely reasonable requests, and I will attempt to
fulfill them.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Eli Zaretskii]

> Date: Sun, 23 Nov 2014 15:02:40 -0500
> From: "Eric S. Raymond" <esr@thyrsus.com>
> 
> Even if we restrict attention to git, log --follow only works on
> single files...

"C-x v l" _is_ for a single file.

Re: My plans for VC mode

[Dieter Deyke]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Google turns up no evidence of modes for
> DaRCS or Codeville.

melpa has vc-darcs:

vc-darcs is an available package.

     Status: Available from melpa -- Install
    Archive: melpa
    Version: 20141122.1326
    Summary: a VC backend for darcs
   Keywords: vc
    Other versions: 1.12 (marmalade).

Darcs is David's Advanced Revision Control System, available at
http://www.darcs.net/

-- 
Dieter Deyke
mailto:dieter.deyke@gmail.com
GPG Key fingerprint = 70E8 D998 0E4D 6087 BE05 099B 50F4 BF3D B116 EA20

Re: My plans for VC mode

[Eric S. Raymond]

Dieter Deyke <dieter.deyke@gmail.com>:
> vc-darcs is an available package.

I know nothing about melpa.  How di I find the maintainer's email?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Kan-Ru Chen (陳侃如)]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Dieter Deyke <dieter.deyke@gmail.com>:
>> vc-darcs is an available package.
>
> I know nothing about melpa.  How di I find the maintainer's email?

http://www.pps.univ-paris-diderot.fr/~jch/ is the maintainer's homepage

Kanru

Re: My plans for VC mode

[Eric S. Raymond]

Kan-Ru Chen (陳侃如) <kanru@kanru.info>:
> "Eric S. Raymond" <esr@thyrsus.com> writes:
> 
> > Dieter Deyke <dieter.deyke@gmail.com>:
> >> vc-darcs is an available package.
> >
> > I know nothing about melpa.  How di I find the maintainer's email?
> 
> http://www.pps.univ-paris-diderot.fr/~jch/ is the maintainer's homepage
> 
> Kanru

Thanks, I have sent him email.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: My plans for VC mode

[Stefan Monnier]

> Google turns up no evidence of modes for DaRCS or Codeville.

Searching for

    "vc-darcs.el"

(with or without the quotes) in duckduckgo points me to the emacswiki as
the first entry.  I'd expect Google to give similar results.


        Stefan

Re: My plans for VC mode

[Eric S. Raymond]

Stefan Monnier <monnier@IRO.UMontreal.CA>:
> (with or without the quotes) in duckduckgo points me to the emacswiki as
> the first entry.  I'd expect Google to give similar results.

And now it does.  The ways of Google are mysterious.  I had tried some
permutation of "Darcs Emacs mode" and got nothing, but even that hits
EmacsWiki two days later.

I sent email to the maintainer yesterday.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>