Problems with GNU Parallel and ulimit

Problems with GNU Parallel and ulimit

[Arun Isaac]

Does `parallel' work for anybody? When I run

$ ls | parallel echo

I get the following error message, and nothing happens.

parallel: Warning: No more processes: Decreasing number of running jobs to 1.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Error: No more processes: cannot run a single job. Something is wrong at file-name.

Any idea about what's happening?

Re: Problems with GNU Parallel and ulimit

[Ben Woodcroft]

On 14/08/17 12:31, Arun Isaac wrote:
> Does `parallel' work for anybody? When I run
>
> $ ls | parallel echo
>
> I get the following error message, and nothing happens.
It works fine for me using Guix on a foreign OS, and has since I can 
remember.
ben

Re: Problems with GNU Parallel and ulimit

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
> On 14/08/17 12:31, Arun Isaac wrote:
> > Does `parallel' work for anybody? When I run
> > 
> > $ ls | parallel echo
> > 
> > I get the following error message, and nothing happens.
> It works fine for me using Guix on a foreign OS, and has since I can
> remember.
> ben

Same here. Arun, can you give some more details about your system?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Problems with GNU Parallel and ulimit

[Arun Isaac]

Leo Famulari writes:

> On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
>> On 14/08/17 12:31, Arun Isaac wrote:
>> > Does `parallel' work for anybody? When I run
>> > 
>> > $ ls | parallel echo
>> > 
>> > I get the following error message, and nothing happens.
>> It works fine for me using Guix on a foreign OS, and has since I can
>> remember.
>> ben
>
> Same here. Arun, can you give some more details about your system?

This is probably a problem only on GuixSD systems. Could you try on
GuixSD instead of using Guix on a foreign distro?

Re: Problems with GNU Parallel and ulimit

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

Arun Isaac <arunisaac@systemreboot.net> writes:

> Leo Famulari writes:
>
>> On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
>>> On 14/08/17 12:31, Arun Isaac wrote:
>>> > Does `parallel' work for anybody? When I run
>>> > 
>>> > $ ls | parallel echo
>>> > 
>>> > I get the following error message, and nothing happens.
>>> It works fine for me using Guix on a foreign OS, and has since I can
>>> remember.
>>> ben
>>
>> Same here. Arun, can you give some more details about your system?
>
> This is probably a problem only on GuixSD systems. Could you try on
> GuixSD instead of using Guix on a foreign distro?

I tested it on GuixSD, works fine. Are you trying the command in a
folder with > 1024 files perhaps? What does `ulimit -n` print?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Problems with GNU Parallel and ulimit

[myglc2]

On 08/15/2017 at 00:14 Arun Isaac writes:

> Leo Famulari writes:
>
>> On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
>>> On 14/08/17 12:31, Arun Isaac wrote:
>>> > Does `parallel' work for anybody? When I run
>>> > 
>>> > $ ls | parallel echo
>>> > 
>>> > I get the following error message, and nothing happens.
>>> It works fine for me using Guix on a foreign OS, and has since I can
>>> remember.
>>> ben
>>
>> Same here. Arun, can you give some more details about your system?
>
> This is probably a problem only on GuixSD systems. Could you try on
> GuixSD instead of using Guix on a foreign distro?

I think you are correct. I run GuixSD and, after installing parallel I
get ...

***
g1@g1 ~$ ls | parallel echo
Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence this citation notice: run 'parallel --citation'.

parallel: Warning: No more processes: Decreasing number of running jobs to 7.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 6.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 5.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 4.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 3.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 2.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Warning: No more processes: Decreasing number of running jobs to 1.
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Error: No more processes: cannot run a single job. Something is wrong at bin.
g1@g1 ~$ guix --version
guix (GNU Guix) 0.13.0.1765-9ef59
Copyright (C) 2017 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
g1@g1 ~$ 
***

I am not a parallel user so I can't comment further.

HTH - George

Re: Problems with GNU Parallel and ulimit

[ericbavier]

> On 08/15/2017 at 00:14 Arun Isaac writes:
> 
>> Leo Famulari writes:
>>
>>> On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
>>>> On 14/08/17 12:31, Arun Isaac wrote:
>>>> > Does `parallel' work for anybody? When I run
>>>> > 
>>>> > $ ls | parallel echo
>>>> > 
>>>> > I get the following error message, and nothing happens.
>>>> It works fine for me using Guix on a foreign OS, and has since I can
>>>> remember.
>>>> ben
>>>
>>> Same here. Arun, can you give some more details about your system?
>>
>> This is probably a problem only on GuixSD systems. Could you try on
>> GuixSD instead of using Guix on a foreign distro?
> 
> I think you are correct. I run GuixSD and, after installing parallel I
> get ...
> 
> ***
> g1@g1 ~$ ls | parallel echo
> ...
> parallel: Warning: No more processes: Decreasing number of running jobs to 7.
> parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.

The problem appears if you do not have 'perl' available in $PATH.  GNU parallel calls it recursively for a few tasks:

$ guix package -I | grep "perl"
$ echo $?
1
$ parallel echo ::: 1 2 3
...
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Error: No more processes: cannot run a single job. Something is wrong at 1.
$ guix environment --ad-hoc perl
[dev]$ parallel echo ::: 1 2 3
1
2
3

There is some path patching that goes on in the package recipe, but it must have gone out of sync in latest versions.  The 'post-install-test' phase wouldn't catch this particular error without '(setenv "PATH" "")'.

I wouldn't feel quite comfortable doing a "blind" update of the regex, since some mentions of "perl" I think are meant to be executed by remote hosts and so shouldn't be patched to absolute store references.

It might be useful to get Ole's opinion on this.

`~Eric

> I am not a parallel user so I can't comment further.
> 
> HTH - George

Re: Problems with GNU Parallel and ulimit

[Eric Bavier]

Sending response to the list this time...

> Arun Isaac <address@hidden> writes:
> 
> > Leo Famulari writes:
> >
> >> On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
> >>> On 14/08/17 12:31, Arun Isaac wrote:
> >>> > Does `parallel' work for anybody? When I run
> >>> > 
> >>> > $ ls | parallel echo
> >>> > 
> >>> > I get the following error message, and nothing happens.
> >>> It works fine for me using Guix on a foreign OS, and has since I
> >>> can remember.
> >>> ben
> >>
> >> Same here. Arun, can you give some more details about your system?
> >
> > This is probably a problem only on GuixSD systems. Could you try on
> > GuixSD instead of using Guix on a foreign distro?
> 
> I tested it on GuixSD, works fine. Are you trying the command in a
> folder with > 1024 files perhaps? What does `ulimit -n` print?

The problem appears if you do not have 'perl' available in $PATH. GNU
parallel calls it recursively for a few tasks:

$ guix package -I | grep "perl"
$ echo $?
1
$ parallel echo ::: 1 2 3
...
parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
parallel: Error: No more processes: cannot run a single job. Something is wrong at 1.
$ guix environment --ad-hoc perl
[dev]$ parallel echo ::: 1 2 3
1
2
3

There is some path patching that goes on in the package recipe, but it
must have gone out of sync in latest versions.  The 'post-install-test'
phase wouldn't catch this particular error without '(setenv "PATH" "")'.

I wouldn't feel quite comfortable doing a "blind" update of the regex,
since some mentions of "perl" I think are meant to be executed by
remote hosts and so shouldn't be patched to absolute store references.

It might be useful to get Ole's opinion on this.

`~Eric

Re: Problems with GNU Parallel and ulimit

[Arun Isaac]

Eric Bavier writes:

> The problem appears if you do not have 'perl' available in $PATH. GNU
> parallel calls it recursively for a few tasks:

This is correct. I have verified it. When I have perl in my PATH, GNU
parallel works fine.

> There is some path patching that goes on in the package recipe, but it
> must have gone out of sync in latest versions.  The 'post-install-test'
> phase wouldn't catch this particular error without '(setenv "PATH" "")'.
>
> I wouldn't feel quite comfortable doing a "blind" update of the regex,
> since some mentions of "perl" I think are meant to be executed by
> remote hosts and so shouldn't be patched to absolute store references.

Why does parallel need to execute perl on remote hosts?

> It might be useful to get Ole's opinion on this.

Yes, could you please do this?

Thanks!

Re: Problems with GNU Parallel and ulimit

[Clément Lassieur]

Eric Bavier <ericbavier@centurylink.net> writes:

> Sending response to the list this time...
>
>> Arun Isaac <address@hidden> writes:
>> 
>> > Leo Famulari writes:
>> >
>> >> On Mon, Aug 14, 2017 at 12:46:25PM +1000, Ben Woodcroft wrote:
>> >>> On 14/08/17 12:31, Arun Isaac wrote:
>> >>> > Does `parallel' work for anybody? When I run
>> >>> > 
>> >>> > $ ls | parallel echo
>> >>> > 
>> >>> > I get the following error message, and nothing happens.
>> >>> It works fine for me using Guix on a foreign OS, and has since I
>> >>> can remember.
>> >>> ben
>> >>
>> >> Same here. Arun, can you give some more details about your system?
>> >
>> > This is probably a problem only on GuixSD systems. Could you try on
>> > GuixSD instead of using Guix on a foreign distro?
>> 
>> I tested it on GuixSD, works fine. Are you trying the command in a
>> folder with > 1024 files perhaps? What does `ulimit -n` print?
>
> The problem appears if you do not have 'perl' available in $PATH. GNU
> parallel calls it recursively for a few tasks:
>
> $ guix package -I | grep "perl"
> $ echo $?
> 1
> $ parallel echo ::: 1 2 3
> ...
> parallel: Warning: Raising ulimit -u or /etc/security/limits.conf may help.
> parallel: Error: No more processes: cannot run a single job. Something is wrong at 1.
> $ guix environment --ad-hoc perl
> [dev]$ parallel echo ::: 1 2 3
> 1
> 2
> 3
>
> There is some path patching that goes on in the package recipe, but it
> must have gone out of sync in latest versions.  The 'post-install-test'
> phase wouldn't catch this particular error without '(setenv "PATH" "")'.
>
> I wouldn't feel quite comfortable doing a "blind" update of the regex,
> since some mentions of "perl" I think are meant to be executed by
> remote hosts and so shouldn't be patched to absolute store references.

Hi,

This is fixed by 4e7950fdce5b83db018c44a80ec4a4059612d716 (it adds a
wrap-program phase).

Clément

Re: Problems with GNU Parallel and ulimit

[Arun Isaac]

> This is fixed by 4e7950fdce5b83db018c44a80ec4a4059612d716 (it adds a
> wrap-program phase).

I just verified this. It works. Thanks! :-)