bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Zefram]

GUILE_INSTALL_LOCALE=1 breaks some of the robustness of non-locale-using
programs, marring their stderr output if the environment's locale settings
are faulty.

Suppose you have a program written in Guile Scheme that doesn't use any
locale facilities.  To be portable to the GUILE_INSTALL_LOCALE=1 situation
(which the documentation threatens will become the default in Guile 2.2),
it must be prepared to start up with some locale already selected, and
reconfigure from there as required.  Being a conscientious programmer,
you are of course willing to add the (setlocale LC_ALL "C") and whatever
other invocations are required to recover the non-locale state.  But then
this situation arises:

$ LANG=wibble GUILE_INSTALL_LOCALE=1 guile-2.0 -c '(setlocale LC_ALL "C") (write "hi") (newline)'
guile: warning: failed to install locale
"hi"

The warning shown goes to the program's stderr.  It does not come from
the program's setlocale call, which is succeeding and would signal a
perfectly ordinary (catchable) exception if it failed.  The warning comes
from the implicit setlocale call triggered by GUILE_INSTALL_LOCALE=1,
before the program gains control.  As far as I can see, there is no way
for the program to prevent the failing setlocale attempt or to muffle
the warning.  Or even to detect that this has happened.

Guile should not be saying anything on the program's stderr.  This is
damaging the program's visible behaviour, and making it impossible to
effectively port non-locale programs to new Guile versions.

If Guile must attempt this implicit setlocale and continue to run the
program if it fails, then it needs to keep quiet about the failure.
This is no disadvantage to a program that actually wants to use the
environmental locale, because the program is free to call (setlocale
LC_ALL "") itself and handle its failure in whatever manner it finds
appropriate.  Indeed, any such program predating Guile 2.0 must already
be performing that call itself, because the implicit setlocale didn't
occur then.  The same for any program portable to pre-2.0 Guiles.  And on
Guile 2.0+ such a program still really needs to perform the call itself,
because it can't predict how GUILE_INSTALL_LOCALE will be set in the
environment, so still can't rely on the implicit setlocale happening.

However, if it is deemed to be essential that Guile attempt the implicit
setlocale and gripe about its failure, then the message should not
precede or otherwise mix with the actual program run.  The message should
be emitted *instead of* running the program, declaring the absolute
incompatibility of the Guile framework with this environmental condition.

-zefram

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Andy Wingo]

On Fri 04 Mar 2016 15:13, Zefram <zefram@fysh.org> writes:

> GUILE_INSTALL_LOCALE=1 breaks some of the robustness of non-locale-using
> programs, marring their stderr output if the environment's locale settings
> are faulty.

I believe this is consistent with other programs which call setlocale,
notably Perl and Bash.  I think the right way to avoid the output is to
avoid the call to setlocale, and Guile offers the GUILE_INSTALL_LOCALE=0
knob to do this.

> However, if it is deemed to be essential that Guile attempt the implicit
> setlocale and gripe about its failure, then the message should not
> precede or otherwise mix with the actual program run.  The message should
> be emitted *instead of* running the program, declaring the absolute
> incompatibility of the Guile framework with this environmental condition.

Your suggestion is not what Perl or Bash does, FWIW.  But it is
possible -- we could emit an error and require the user to run with
GUILE_INSTALL_LOCALE=0.  Or we could add that suggestion to the
warning.  Probably adding the suggestion to the warning is the right
thing; wdyt?  Try "LC_ALL=fasdada perl" for an example.

Andy

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Zefram]

Andy Wingo wrote:
>I believe this is consistent with other programs which call setlocale,
>notably Perl and Bash.

It is consistent with them, but the fact that others get it wrong isn't
an excuse.

>avoid the call to setlocale, and Guile offers the GUILE_INSTALL_LOCALE=0
>knob to do this.

That knob is not available to the program.  If you provide a knob that
the program can control, independent of the environment, with backward
compatibility to Guile 1.8, then we can consider the setlocale call
avoidable.

>          Probably adding the suggestion to the warning is the right
>thing; wdyt?

No, that's not an improvement.  Emitting a warning and then running
the program anyway is fundamentally broken behaviour, and tweaking the
content of the warning doesn't help.

Some way for the program to detect that you've screwed up its output,
so that it can decide to abort rather than continue with faulty output,
would be another middle way.

-zefram

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Andy Wingo]

Hello,

On Mon 08 Aug 2016 01:00, Zefram <zefram@fysh.org> writes:

> Andy Wingo wrote:
>>I believe this is consistent with other programs which call setlocale,
>>notably Perl and Bash.
>
> It is consistent with them, but the fact that others get it wrong isn't
> an excuse.

I would like to let you know that I am working on this in my free time
and in that context I don't respond well to blame, "excuses", and so on.
If you would like for me to work on your bugs then I would appreciate it
if you would keep things constructive.  Thanks :)

>>avoid the call to setlocale, and Guile offers the GUILE_INSTALL_LOCALE=0
>>knob to do this.
>
> That knob is not available to the program.  If you provide a knob that
> the program can control, independent of the environment, with backward
> compatibility to Guile 1.8, then we can consider the setlocale call
> avoidable.

Serious question tho: what sort of back-compatibility can there be with
a Guile that only supports latin-1 strings?  What property is it that
you are going for here?

> Emitting a warning and then running the program anyway is
> fundamentally broken behaviour

My guess is that many people want exactly this behavior.  Only a guess
of course.

What about GUILE_INSTALL_LOCALE=require or something like that?  Does
that suit your needs?  If not, then:

> Some way for the program to detect that you've screwed up its output,
> so that it can decide to abort rather than continue with faulty output,
> would be another middle way.

How would this work?

Regards,

Andy

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Zefram]

Andy Wingo wrote:
>If you would like for me to work on your bugs then I would appreciate it
>if you would keep things constructive.  Thanks :)

I'm sorry that that bit came across badly.  I do appreciate your efforts.

>Serious question tho: what sort of back-compatibility can there be with
>a Guile that only supports latin-1 strings?

I'd expect that almost any program that runs on Guile 1.8 ought to be
portable, with only minimal modifications, to later versions of Guile.
Obviously this wouldn't work the other way round: if a program relies
on 2.0's non-Latin-1 strings then it can't be easily ported back to 1.8.
But lots of programs work fine on 1.8, either not processing non-Latin-1
data or processing it in forms other than the builtin string type.
Scheme was a good programming language long before Unicode came along.

>                                             What property is it that
>you are going for here?

In that bit, I'm going for it being possible for a program to run on
both Guile 1.8 and Guile 2.N while avoiding the new locale warning
from Guile 2.N.  This should be a single program file, starting with a
"#!/usr/bin/guile" line, where /usr/bin/guile may refer to either version
of Guile.  This would be especially relevant for a program originally
written for Guile 1.8, but more generally is relevant for any program
that doesn't need any of 2.0's new capabilities.

The particular problem that arises is that a possible form for a
warning-muffling switch would be a command-line switch that goes on the
#! line.  Any new switch of that nature wouldn't be recognised by Guile
1.8, and would cause an error when attempting to run the program on 1.8.

>What about GUILE_INSTALL_LOCALE=require or something like that?

In the environment?  That's still not controllable by the program.  The
environment is the wrong place for any switch that needs to be the choice
of the program.  Whether to engage with the environmentally-suggested
locale ought to be the choice of the program.

>How would this work?

I imagine a builtin function that returns a truth value saying whether
the Guile framework has emitted a warning before running the program.
Suppose it's called "program-running-with-unclean-output".  Then those
who particularly want clean output can write something like

    (when (program-running-with-unclean-output)
      (error "can't run after warnings"))

This doesn't avoid the warning appearing, but does avoid treating a
run marred by the warning as a successful program run.  The program's
checking code can easily be made portable back to Guile versions lacking
the new function, by using cond-expand, false-if-exception, or other
metaprogramming facilities.

-zefram

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Andy Wingo]

On Mon 08 Aug 2016 23:02, Zefram <zefram@fysh.org> writes:

> Andy Wingo wrote:
>>Serious question tho: what sort of back-compatibility can there be with
>>a Guile that only supports latin-1 strings?
>
> I'd expect that almost any program that runs on Guile 1.8 ought to be
> portable, with only minimal modifications, to later versions of Guile.

I'm with you :)  We did (and do) our best but some problems remain.  I'm
not sure which ones are incidental and which ones are essential.
Probably some are essential though and need to be minimized and then
documented.

> The particular problem that arises is that a possible form for a
> warning-muffling switch would be a command-line switch that goes on the
> #! line.  Any new switch of that nature wouldn't be recognised by Guile
> 1.8, and would cause an error when attempting to run the program on 1.8.

Just thinking out loud here, what about something like:

#!/bin/sh
export FOO=bar
exec guile $0 "$@"
!#

Various FOO=bar things that could work for you:

  LC_ALL=C
  GUILE_INSTALL_LOCALE=0

It doesn't work if you run "guile myscript.scm" but it would work as a
./myscript.scm.

>>How would this work?
>
> I imagine a builtin function that returns a truth value saying whether
> the Guile framework has emitted a warning before running the program.
> Suppose it's called "program-running-with-unclean-output".  Then those
> who particularly want clean output can write something like
>
>     (when (program-running-with-unclean-output)
>       (error "can't run after warnings"))
>
> This doesn't avoid the warning appearing, but does avoid treating a
> run marred by the warning as a successful program run.  The program's
> checking code can easily be made portable back to Guile versions lacking
> the new function, by using cond-expand, false-if-exception, or other
> metaprogramming facilities.

This is certainly possible to do.  Actually I would guess that this
works:

  (setlocale LC_ALL "")

E.g.

$ GUILE_INSTALL_LOCALE=1 LC_ALL=sadfadf guile -c '(setlocale LC_ALL "")'
guile: warning: failed to install locale
Backtrace:
In ice-9/boot-9.scm:
 157: 7 [catch #t #<catch-closure 24c1900> ...]
In unknown file:
   ?: 6 [apply-smob/1 #<catch-closure 24c1900>]
In ice-9/boot-9.scm:
  63: 5 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 4 [eval # #]
In unknown file:
   ?: 3 [call-with-input-string "(setlocale LC_ALL \"\")" ...]
In ice-9/command-line.scm:
 180: 2 [#<procedure 24d2400 at ice-9/command-line.scm:175:6 (port)> #<input: string 2271680>]
In unknown file:
   ?: 1 [eval (setlocale LC_ALL "") #<directory (guile-user) 24b4c60>]
   ?: 0 [setlocale 6 ""]

ERROR: In procedure setlocale:
ERROR: In procedure setlocale: Invalid argument

Does any of this work for you?

Andy

bug#22905: GUILE_INSTALL_LOCALE produces unavoidable noise

[Zefram]

Andy Wingo wrote:
>#!/bin/sh
>export FOO=bar
>exec guile $0 "$@"
>!#

That introduces all the complexity of using another language interpreter,
one I've chosen not to write my program in.  I don't much fancy working
round a gotcha by importing another series of gotchas.

Fundamentally, it seems like an admission of defeat.  With care it would
work, but means that Guile is not itself the platform on which to write
a Unix program.  Maybe you're OK with the idea that Guile programs aren't
meant to run in their own right.  Would you be OK with documenting it?

It also means that the Guile program isn't actually seeing the user's
environment, and so doesn't accurately pass that environment through to
anything that it runs in turn.  Working around that would involve some
hairy and error-prone shell code.

>This is certainly possible to do.  Actually I would guess that this
>works:
>
>  (setlocale LC_ALL "")

That succeeds in signalling an error in any case where the environmental
locale doesn't exist, but that's not really what I want.  If the
framework didn't perform an implicit setlocale, and so didn't mar my
output, I don't then want to make things break.

That approach is also totally specific to the setlocale warning.
If program-running-with-unclean-output were to exist, it should also
cover uncleanliness due to auto-compile banners (bug#16364).  It would
be the solution (though not a great one) to both problems.

>Does any of this work for you?

Shell script wrapper is the closest so far, but it's nasty.  You haven't
proposed any real solution.

The really simple solution would be to remove this switch from the
environment entirely, and remove the implicit setlocale from the startup
sequence entirely.  The environment was always the wrong place for the
switch, and there's no benefit in the implicit setlocale being as early
as it is.  The decision on whether to engage with the user's locale is
then made entirely by the program, as part of its ordinary execution.
If it wants to use the user's locale, it executes (setlocale LC_ALL
"").  If it wants non-default handling of errors, it executes that in
the dynamic scope of whatever throw or catch handler it likes.  If it
doesn't want to use the user's locale, it doesn't execute that.  Bonus:
works identically on older Guile versions.

If you won't go for the simple solution, then a proper solution
that maintains the default implicit setlocale would be to have
a switch in a magic comment in the program file.  Something like
"#!GUILE_INSTALL_LOCALE=0\n!#\n" immediately following the program's
initial #!...!# block.  This is ignored as a comment by older Guile
versions.  The semantic on newer versions would be that the setting given
there (which may be 0 or 1) determines conclusively whether the implicit
setlocale happens.  The environment variable would take effect as it
currently does only for programs not containing this kind of setting.

-zefram