VC and too long command lines

VC and too long command lines

[Deniz Dogan]

Recently I came across a problem when using VC in Emacs.  I had just 
added a whole bunch of files to my Bazaar repository and wanted to 
commit them all.  I chose to mark them all in vc-dir and then tried to 
commit them.

However...

File error: "Spawning child process", "exec format error"

Of course, the command line was simply too long (approximately 66,000 
characters) for my operating system to handle (Windows 7 64-bit).

What should Emacs do when this happens?

I think the most appropriate thing would be to add some functionality 
which determines the maximum command line length for the current system 
and if the call exceeds that length, tell the user about it.  Another 
way would be to e.g. divide the commit into several smaller commits and 
appending something along the lines of "(commit N of M)" to each commit 
message.

What do you think?  I think this is quite an important issue.

Deniz

Re: VC and too long command lines

[Eli Zaretskii]

> Date: Thu, 21 Jul 2011 14:55:28 +0200
> From: Deniz Dogan <deniz@dogan.se>
> 
> Of course, the command line was simply too long (approximately 66,000 
> characters) for my operating system to handle (Windows 7 64-bit).
> 
> What should Emacs do when this happens?
> 
> I think the most appropriate thing would be to add some functionality 
> which determines the maximum command line length for the current system 
> and if the call exceeds that length, tell the user about it.  Another 
> way would be to e.g. divide the commit into several smaller commits and 
> appending something along the lines of "(commit N of M)" to each commit 
> message.

The former.  The latter is impossible, because to know how to break a
command into several pieces you need to understand the semantics of
that command.  E.g., all pieces but the first might need some
additional command-line switch to tell the program to append the
results to the previous ones, rather than overwriting them.  As
another example, you'd need to use ">>" instead of ">" in all but the
first command.  Etc., etc.

Re: VC and too long command lines

[Deniz Dogan]

On 2011-07-21 16:12, Eli Zaretskii wrote:
>> Date: Thu, 21 Jul 2011 14:55:28 +0200
>> From: Deniz Dogan<deniz@dogan.se>
>>
>> Of course, the command line was simply too long (approximately 66,000
>> characters) for my operating system to handle (Windows 7 64-bit).
>>
>> What should Emacs do when this happens?
>>
>> I think the most appropriate thing would be to add some functionality
>> which determines the maximum command line length for the current system
>> and if the call exceeds that length, tell the user about it.  Another
>> way would be to e.g. divide the commit into several smaller commits and
>> appending something along the lines of "(commit N of M)" to each commit
>> message.
>
> The former.  The latter is impossible, because to know how to break a
> command into several pieces you need to understand the semantics of
> that command.  E.g., all pieces but the first might need some
> additional command-line switch to tell the program to append the
> results to the previous ones, rather than overwriting them.  As
> another example, you'd need to use ">>" instead of">" in all but the
> first command.  Etc., etc.
>

So we agree then!  Do you happen to know of any reliable way to 
determine the maximum command line length of the user system?

Re: VC and too long command lines

[Eli Zaretskii]

> Date: Thu, 21 Jul 2011 16:16:17 +0200
> From: Deniz Dogan <deniz@dogan.se>
> 
> Do you happen to know of any reliable way to determine the maximum
> command line length of the user system?

There isn't any, AFAIK.  However, anything longer than 32K bytes will
fail with 100% certainty.

Please make a prominent note to revisit this code once we move to
Unicode APIs, though, because the limitations there are different.

TIA

Re: VC and too long command lines

[Nix]

On 21 Jul 2011, Eli Zaretskii said:

>> Date: Thu, 21 Jul 2011 16:16:17 +0200
>> From: Deniz Dogan <deniz@dogan.se>
>> 
>> Do you happen to know of any reliable way to determine the maximum
>> command line length of the user system?
>
> There isn't any, AFAIK.

There is, but it's time-consuming. Repeatedly retry a longer and longer
harmless command until one fails. (libtool does it.)

(Note that just because a command succeeds doesn't mean a later one
won't fail: many Unix systems have a fixed length for
cmdline+environment, so if the environment grows then the command-line
space necessarily must shrink. So this probably isn't actually much use
because it means we can't check in advance, and it's too slow to check
at the time. I suppose we could check in advance, and only recheck if
someone calls `setenv'...)

-- 
NULL && (void)

Re: VC and too long command lines

[Eli Zaretskii]

> From: Nix <nix@esperi.org.uk>
> Cc: Deniz Dogan <deniz@dogan.se>, emacs-devel@gnu.org
> Emacs: there's a reason it comes with a built-in psychotherapist.
> Date: Fri, 22 Jul 2011 21:37:50 +0100
> 
> On 21 Jul 2011, Eli Zaretskii said:
> 
> >> Date: Thu, 21 Jul 2011 16:16:17 +0200
> >> From: Deniz Dogan <deniz@dogan.se>
> >> 
> >> Do you happen to know of any reliable way to determine the maximum
> >> command line length of the user system?
> >
> > There isn't any, AFAIK.
> 
> There is, but it's time-consuming. Repeatedly retry a longer and longer
> harmless command until one fails. (libtool does it.)

Obviously, such "way" is inappropriate for a facility that is supposed
to run a command of a given size, because the result of that will
still be a failure.  And if you thought of doing this test only once,
then (a) it will be time consuming, and (b) its result will be
unreliable, for the same reasons it is on Unix:

> (Note that just because a command succeeds doesn't mean a later one
> won't fail: many Unix systems have a fixed length for
> cmdline+environment, so if the environment grows then the command-line
> space necessarily must shrink. So this probably isn't actually much use
> because it means we can't check in advance, and it's too slow to check
> at the time. I suppose we could check in advance, and only recheck if
> someone calls `setenv'...)

Except that on Windows, using `setenv' is not the only reason that
could invalidate previous results.  I'm not aware of any documentation
about the specific factors which influence this, but in my experience
there are such factors.  If someone can find this documented (and
verify the accuracy of that documentation), please be sure to post
here, TIA.

Re: VC and too long command lines

[Richard Stallman]

    > There is, but it's time-consuming. Repeatedly retry a longer and longer
    > harmless command until one fails. (libtool does it.)

    Obviously, such "way" is inappropriate for a facility that is supposed
    to run a command of a given size, because the result of that will
    still be a failure.

It would be possible to test this in advance in the build process, and
record the answer.  If all the systems that the same binary, or same
package, will get installed on will have the same limit, then that
will be reliable.

Even if they don't all have the same limit, this approach may be less
unreliable than the present code.

However, it might be better to make a conservative assumption about
how big the command line can be, and always keep generated commands
under that size.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: VC and too long command lines

[Eli Zaretskii]

> Date: Sat, 23 Jul 2011 06:36:40 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org
> 
> However, it might be better to make a conservative assumption about
> how big the command line can be, and always keep generated commands
> under that size.

In my experience, 30K bytes is such a conservative assumption on
MS-Windows.

Re: VC and too long command lines

[Nix]

On 23 Jul 2011, Eli Zaretskii uttered the following:

>> Date: Sat, 23 Jul 2011 06:36:40 -0400
>> From: Richard Stallman <rms@gnu.org>
>> CC: emacs-devel@gnu.org
>> 
>> However, it might be better to make a conservative assumption about
>> how big the command line can be, and always keep generated commands
>> under that size.
>
> In my experience, 30K bytes is such a conservative assumption on
> MS-Windows.

Unfortunately on operating systems that share the environment and
command line, the maximum command line length can get *very* small, and
of course can vary. I recall HP-UX 10.20 refusing to run anything
because the environment was 8190 bytes... and on that system we
regularly had environments >8000 bytes, so things broke all the time :(

(I too would be fascinated to learn of other constraints on command-line
length on Windows. I suspect the Libtool people might like to know as
well.)

-- 
NULL && (void)

Re: VC and too long command lines

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

On 7/21/2011 5:55 AM, Deniz Dogan wrote:
> Of course, the command line was simply too long (approximately 66,000
> characters) for my operating system to handle (Windows 7 64-bit).
> 
> What should Emacs do when this happens?
> 
> I think the most appropriate thing would be to add some functionality
> which determines the maximum command line length for the current system
> and if the call exceeds that length, tell the user about it.  Another
> way would be to e.g. divide the commit into several smaller commits and
> appending something along the lines of "(commit N of M)" to each commit
> message.
> 
> What do you think?  I think this is quite an important issue.

There is a third option, at least if you use Cygwin:

http://www.cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-at


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: VC and too long command lines

[Stefan Monnier]

> I think the most appropriate thing would be to add some functionality which
> determines the maximum command line length for the current system and if the
> call exceeds that length, tell the user about it.  Another way would be to
> e.g. divide the commit into several smaller commits and appending something
> along the lines of "(commit N of M)" to each commit message.

> What do you think?  I think this is quite an important issue.

No, the problem here is that VC should encourage you (and better
support) to mark directories rather than marking each and every file.

Splitting the command into several commands is right since a commit is
a single atomic entity which in general can't be split.


        Stefan