sh-script.el and magic numbers

sh-script.el and magic numbers

[Luc Teirlinck]

As I mentioned in an earlier report, there seems to be quite some
evidence that sh-script.el assumes, at several places and in several
ways, that shell scripts start with a magic number (#! line), unless
it is a special standard configuration file, obviously intended to be
sourced, style .bashrc.  (The concrete definition being "matching
`executable-magicless-file-regexp'".)  Even in that case sh-script.el
seems to assume the file to start with some kind of comment.  The only
person who responded to my earlier report (Glenn Morris) seemed to
believe that these assumptions should not be made.

There are at least three separate problems (one bug, reported before,
and two inconveniences) associated with not following these
conventions.  Clearly I am not going to even describe the other two
problems (inconveniences rather than real bugs) if it is decided that
sh-scripts.el's assumptions are perfectly reasonable, since in that
case all these "problems" are irrelevant.

So, is there a policy concerning whether or not we care about files
not following the above conventions?  If there is not, should we have
a discussion or is the answer "obvious"?

Glenn's argument is that some files are intended to be sourced, not
executed, which, of course, is obvious.  My sysadmin seems to defend
the conventions, his argument being that a magic number does no harm,
even if the file is to be sourced and indicates to several facilities
(including sh-mode) what shell the script is written for.  He does
put magic comments in .bashrc and similar files.  My Red Hat provided
.bashrc contains no magic number, but seems to very deliberately start
with a comment (which means it is OK in as far as sh-script.el goes):
# .bashrc

One argument against the conventions, in addition to Glenn's earlier
argument, might be that it is possible to use special history files
that look very much like shell-scripts and interactively C-o (in
Readline) one's way through them.  One may want to put such files in
sh-mode, without starting them with a magic number.

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, 18 Jan 2003 21:24:03 -0600 (CST), Luc Teirlinck <teirllm@dms.auburn.edu> wrote:
> 
> As I mentioned in an earlier report, there seems to be quite some
> evidence that sh-script.el assumes, at several places and in several
> ways, that shell scripts start with a magic number (#! line), unless
> it is a special standard configuration file, obviously intended to be
> sourced, style .bashrc.  (The concrete definition being "matching
> `executable-magicless-file-regexp'".)  

The short answer is that the "#!" magic number should be a requirement.
You can use it in sourced files (both in sh/bash and csh/tcsh). It is
good programming practice to add it. Leave it as it is.

Ehud.


- -- 
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 mailto:ehud@unix.mvs.co.il                  Better  Safe  Than  Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE+Kp5FLFvTvpjqOY0RAtUGAJ98dNsblho77u9h2NHudxBReu35swCeJRzb
E3TbeAtXr7ambCcmEcawZfA=
=ikn1
-----END PGP SIGNATURE-----

Re: sh-script.el and magic numbers

[Glenn Morris]

"Ehud Karni" wrote:

> The short answer is that the "#!" magic number should be a requirement.

I would disagree - it seems to me a matter of personal taste. It's not
essential, but it does no harm either. So I probably need the longer answer
as to why it should be a requirement. Though perhaps that is OT in
emacs-devel...

The main Emacs-related gripe I have about adding it is that I like to use

(add-hook 'after-save-hook 'executable-make-buffer-file-executable-if-script-p)

which makes files that contain #! magic executable on save. So if I put
#!/bin/bash in my ~/.bashrc, for example, Emacs insists on making it
executable, which annoys me, as it's not *meant* to be executed.

> You can use it in sourced files (both in sh/bash and csh/tcsh).

You can't *use* it. You can put it in the file and it doesn't do anything -
doesn't seem very useful to me.

I note that none of the system config files in the RedHat distribution I'm
currently using have it. 

> It is good programming practice to add it.

Why?

Re: sh-script.el and magic numbers

[Ehud Karni]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 19 Jan 2003 23:48:19 +0000, Glenn Morris <gmorris+mail@ast.cam.ac.uk> wrote:
> 
> The main Emacs-related gripe I have about adding it is that I like to use
> 
> (add-hook 'after-save-hook 'executable-make-buffer-file-executable-if-script-p)
> 
> which makes files that contain #! magic executable on save. So if I put
> #!/bin/bash in my ~/.bashrc, for example, Emacs insists on making it
> executable, which annoys me, as it's not *meant* to be executed.

The solution to that is simple. Don't put the "#!" magic number,
instead change the 1st line to something like this:
# <your comment> -*- mode: sh -*-   

> > You can use it in sourced files (both in sh/bash and csh/tcsh).
> 
> You can't *use* it. You can put it in the file and it doesn't do anything -
> doesn't seem very useful to me.

That's what I meant, that it does not interfere with sourcing.
It is useful. It is clearer to the reader which shell will run it.

> I note that none of the system config files in the RedHat distribution I'm
> currently using have it. 

I don't think so, take a look at the scripts in /etc/init.d (almost all 
have #!/bin/bash in them FOR A REASON (see below).

> > It is good programming practice to add it.
> 
> Why?

Because if you don't have the magic number, when you run the script,
it is run in the CURRENT shell which may be bash, but also any of the
other useful shells (ksh, zsh, csh/tcsh, etc.). The differences may
be very small (ash vs. bash) or too large to run (bash vs. csh).

Most of the configuration scripts are sourced (it is run by the current
shell) so specifying shell does not do anything. The init scripts (in
/etc/init.d) may be run (and are run) by any shell without problems.

Some are intended for very specific shell (.bashrc, .cshrc), although
it is not necessary to put the magic number, it is advisable for 
debugging purposes (you can run it as normal script to check it).

One last remark: Don't generalize from the GNU/Linux to all UNIX/POSIX
systems. e.g. In GNU/Linux sh is synonym for bash, but there are POSIX
systems where this is not true.

Ehud.


- -- 
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 mailto:ehud@unix.mvs.co.il                  Better  Safe  Than  Sorry
-----BEGIN PGP SIGNATURE-----
Comment: use http://www.keyserver.net/ to get my key (and others)

iD8DBQE+K6xTLFvTvpjqOY0RArojAKCKcNk6/9fLxyigpCtWQrOl40DS1gCeJHYD
xuE88KJ4f3HB2046d9u5Ytc=
=3yJ4
-----END PGP SIGNATURE-----

Re: sh-script.el and magic numbers

[Luc Teirlinck]

Ehud Karni wrote:

   The solution to that is simple. Don't put the "#!" magic number,
   instead change the 1st line to something like this:
   # <your comment> -*- mode: sh -*-

In certain situations it can get somewhat less simple.   

There is a problem here that is still relevant, even though (actually
especially since) a decision has been made on the #! problem.  A
problem I referred to but did not elaborate on, because I thought
(like you) that it could trivially be solved with file local variables
is, in fact, not really as easily solved (for people not familiar with
the sh-script.el source code) as I thought.

Please remember in the following that people could be experienced
shell programmers and even experienced Emacs users without knowing any
Elisp beyond routine customization.

Suppose your default sh-shell-file is bash and are working on a script
intended for csh.  You do not want to use a magic number and maybe do
not want to make the file executable, but you still want all of
sh-script.el to know that you are programming for csh.

The problem is that C-c : will not allow to change sh-shell-file
without inserting a magic number.  Somebody familiar with the
sh-script.el source code can get around this by executing, say 
M-: (sh-set-shell "/bin/csh"), but this is not mentioned in the mode
documentation.  It is also does not persist through different Emacs
sessions, anyway.  Do not try to add .csh or similar to the file name,
it is going to be ignored.  So obviously we try file local variables:

# <your comment> -*- mode: sh; sh-shell-file: /bin/csh; -*-  

Does not work.  sh-shell-file gets set after sh-mode has ran.  The
mode line still says [bash] and indeed, just try, for instance: 
C-c C-c, you get bash's version of case, not csh's switch.  What you 
have to do is:

# <your comment> -*- mode: sh; eval: (sh-set-shell "/bin/csh"); -*-

So it probably would be useful to mention in the mode documentation
that, to tell sh-mode the shell you are programming for, without using
a magic number, you have to do something like 
M-: (sh-set-shell "/bin/csh")
if you want it on a temporary basis and (probably more relevantly),
you have to put something like the previously mentioned line at the
beginning of the script, if you want it to persist.  There still is
the problem that enable-local-eval has to be non-nil (t or maybe) for
the line to work.  I do not know what to do about that.  At least, the
value does not need to be t.  One could, of course, make clear that
using a magic number is way simpler and, in nearly all situations,
vastly preferable.

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Glenn Morris]

Luc Teirlinck wrote:

> Suppose your default sh-shell-file is bash and are working on a script
> intended for csh.  You do not want to use a magic number and maybe do
> not want to make the file executable, but you still want all of
> sh-script.el to know that you are programming for csh.
[No easy way to do that]


Yes! I was asking about this in gnu.emacs.help not so long ago
(e7bs5txfox.fsf@ast.cam.ac.uk thread). Kevin Rodgers pointed out to me that
one could set the variable `sh-shell' as a file local variable to have a
given script recognized as a given shell type. This works (mostly), but
still leaves the mode-line saying "[bash]" in all cases. That can be fixed
with mode-line-process: "[tcsh]" (or whatever).

I agree it would be nice if there was some easy way to switch sh-mode
between the various different shell flavours with no side effects.

Re: sh-script.el and magic numbers

[Luc Teirlinck]

Glenn Morris wrote:

   one could set the variable `sh-shell' as a file local variable to have a
   given script recognized as a given shell type. This works (mostly), but
   still leaves the mode-line saying "[bash]" in all cases. That can be fixed
   with mode-line-process: "[tcsh]" (or whatever).

Setting mode-line-process is purely cosmetic.  It solves the symptom,
not the problem.  I assume you mean:

# -*- mode: sh; sh-shell: /bin/csh; mode-line-process: "[tcsh]"-*-

You get [tcsh] in the mode line all right but do C-c C-c: bash's case,
not csh's switch.  As I pointed out, what you have to do is:

# <your comment> -*- mode: sh; eval: (sh-set-shell "/bin/csh"); -*-

   I agree it would be nice if there was some easy way to switch sh-mode
   between the various different shell flavours with no side effects.

You mean interactively (temporarily) or persistently?

Interactively:

M-: (sh-set-shell "/bin/csh")

Persistently, see above.

It seems to me that all of this is not too difficult once one knows
it, but the main problem is that you need some familiarity with the
sh-script.el source code to figure this out.  As I said, it should be
mentioned in the mode documentation you get with C-h m.  Would you
consider this to be a sufficient solution?  There also is the problem
that the persistent solution requires enable-local-eval to be non-nil
(t or maybe).

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Glenn Morris]

Luc Teirlinck wrote:

> Setting mode-line-process is purely cosmetic.  It solves the symptom,
> not the problem.  I assume you mean:
>
> # -*- mode: sh; sh-shell: /bin/csh; mode-line-process: "[tcsh]"-*-

No, I mean what was said in the thread I referred to, namely:

# -*- mode: sh; sh-shell: csh; mode-line-process: "[csh]"-*-

which works without the need for enabling eval of local variables. But it's
a little ugly, and doesn't work right, since it still prints "indentation
set up for shell type bash". I'd prefer just a single variable that
controls what shell flavour a file is.

Re: sh-script.el and magic numbers

[Luc Teirlinck]

Glenn Morris wrote:
   
   No, I mean what was said in the thread I referred to, namely:

   # -*- mode: sh; sh-shell: csh; mode-line-process: "[csh]"-*-

   which works without the need for enabling eval of local variables. But it's
   a little ugly, and doesn't work right, since it still prints "indentation
   set up for shell type bash". I'd prefer just a single variable that
   controls what shell flavour a file is.

and:

   and doesn't work right, since it still prints "indentation
   set up for shell type bash".

The problem is not just that it prints that, worse: it is actually
true.

Do C-c C-c.  Answer "a" or whatever to all questions. You now indeed
get switch.  But, look at the indentation.  Then use my mode line. Do
the same thing over.  Does the indentation not look a lot better?

While the solution you describe does not involve eval, it still
involves a hook local variable.

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Luc Teirlinck]

Glenn Morris wrote;

   No, I mean what was said in the thread I referred to, namely:

   # -*- mode: sh; sh-shell: csh; mode-line-process: "[csh]"-*-

   which works without the need for enabling eval of local variables. But it's
   a little ugly, and doesn't work right, since it still prints "indentation
   set up for shell type bash". I'd prefer just a single variable that
   controls what shell flavour a file is.

That solution really does not seem solid at all.  Not only is the
indentation wrong, but moreover sh-shell is now csh, whereas
sh-shell-file still has its usual value, in my case
"/usr/local/bin/bash".  This means half of sh-script.el thinks you are
programming for csh and the other half thinks you are programming for
bash.  This is bad.  I doubt that the problems caused by that are
limited to the indentation problems (which are bad enough anyway.)

I personally believe that the only solid way to switch shells is to
call sh-set-shell.  Otherwise, you have to emulate everything
sh-set-shell does, including (run-hooks 'sh-set-shell-hook).  This
seems hopeless.

Is having enable-local-eval set to maybe (the default-value) that bad?
(It is not going to execute any evals without asking you.)  If it is,
one could have sh-mode look for a definition of sh-shell-file in the
local variable line, if there is no magic number.  Then you could just
set sh-shell-file and sh-mode would itself call sh-set-shell with the
desired argument.

Can anybody think of a simple, but thoroughly solid, solution, not
involving code changes, that works when enable-local-eval is nil?

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Luc Teirlinck]

I believe that sh-shell-file should be made into a permanent local.
The variable is always defined, but only makes a difference if the
buffer is in sh-mode, so we only have to worry about what happens
while the buffer is in sh-mode.  Suppose that for some reason we want
to put a buffer in sh-mode temporarily in another mode.  When we
switch back to sh-mode again, should that reset sh-shell-file, and
hence sh-script.el's notion of which shell we are programming for, to
its global default?  I personally do not believe so.

This question is irrelevant if there is a magic number, because that
is going to determine the value of sh-shell-file anyway, but without
magic number it is relevant.

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Richard Stallman]

    I believe that sh-shell-file should be made into a permanent local.

I think your argument is valid.

Re: sh-script.el and magic numbers

[Luc Teirlinck]

>From my earlier message:

   Is having enable-local-eval set to maybe (the default-value) that bad?
   (It is not going to execute any evals without asking you.)  If it is,
   one could have sh-mode look for a definition of sh-shell-file in the
   local variable line, if there is no magic number.  Then you could just
   set sh-shell-file and sh-mode would itself call sh-set-shell with the
   desired argument.

I now believe that, if we absolutely want to handle the problem
without using a local eval, it would be better to have sh-mode, if
there is no magic number, add a buffer-local hook function to
find-file-hook.  That function would check the value of a buffer-local
variable, say sh-local-shell-file (or whatever), and, if the variable
is non-nil, call (sh-set-shell sh-local-shell-file).  This would
eliminate the need for the eval and

# <your comment> -*- mode: sh; sh-local-shell-file: /bin/csh; -*-

would work.  Defining sh-local-shell-file in a local variables list at
the end of the file would work too.

(The problem we want to circumvent is that file local variables get set
after the mode function has already ran.  So we use find-file-hook,
which runs after the local variables have been set.)

I am not going to worry further about implementation details before a
decision is made on whether this would be desirable.  Again, the only
purpose of this would be to avoid having to use a local eval.  If we
do not mind about using a local eval, all that is required is adding a
line or two to the mode's documentation string.

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Kai Großjohann]

Glenn Morris <gmorris+mail@ast.cam.ac.uk> writes:

> I agree it would be nice if there was some easy way to switch sh-mode
> between the various different shell flavours with no side effects.

Why doesn't the file define sh-mode, csh-mode, tcsh-mode, and so on?
-- 
Ambibibentists unite!

Re: sh-script.el and magic numbers

[Luc Teirlinck]

Kai Grossjohann wrote:

   Why doesn't the file define sh-mode, csh-mode, tcsh-mode, and so
   on?

Because:

Ignoring unknown mode `csh-mode'

(from the *Messages* buffer)

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Kai Großjohann]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Kai Grossjohann wrote:
>
>    Why doesn't the file define sh-mode, csh-mode, tcsh-mode, and so
>    on?
>
> Because:
>
> Ignoring unknown mode `csh-mode'
>
> (from the *Messages* buffer)

I meant, why doesn't sh-script.el contain (defun sh-mode ...) and
(defun csh-mode ...) and so on?  If it did, you could just put "-*-
csh -*-" in the first line, and Bob's your uncle.
-- 
Ambibibentists unite!

Re: sh-script.el and magic numbers

[Kai Großjohann]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Kai Grossjohann wrote:
>
>    Why doesn't the file define sh-mode, csh-mode, tcsh-mode, and so
>    on?
>
> Because:
>
> Ignoring unknown mode `csh-mode'
>
> (from the *Messages* buffer)

It seems my last message didn't go through.  What I meant is, why
doesn't sh-script.el contain (defun sh-mode ...) and (defun csh-mode
...) and (defun tcsh-mode ...)?

It seems to me, if it did, then people could just put "-*- csh -*-"
in the first line of their script and Bob's their uncle.
-- 
Ambibibentists unite!

Re: sh-script.el and magic numbers

[Luc Teirlinck]

Kai Grossjohann wrote:

   It seems my last message didn't go through.  What I meant is, why
   doesn't sh-script.el contain (defun sh-mode ...) and (defun csh-mode
   ...) and (defun tcsh-mode ...)?

   It seems to me, if it did, then people could just put "-*- csh -*-"
   in the first line of their script and Bob's their uncle.

sh-script.el was not written that way, because it assumes that a
shell-script either has a #! line or is an obvious configuration file,
style .bashrc.  We are now trying to make some changes to
sh-script.el, to make it usable without these assumptions.

I believe that defining a bunch of derived modes of sh-mode would
correspond to a substantial rewrite of sh-script.el and, actually a
radical change in style and philosophy.

I do not believe it is necessary to go that far.  What seems needed is
fixing a concrete bug in sh-get-indent-info, maybe some similar not
yet discovered bugs elsewhere in the indentation machinery (let us
hope not), making sh-shell-file a permanent local, adding a couple of
lines to the mode documentation and maybe make sh-mode add a small and
simple function to find-file-hook.  These are all small local changes,
whereas the change you propose would be a lot more radical.  Let us
also not forget that in most, be it definitely not all, situations the
implicit assumptions made by sh-script.el are actually satisfied.

Sorry for misunderstanding your original message. 

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Luc Teirlinck]

I forgot a cc to emacs-devel in a previous message:

Kai Grossjohann wrote:
   
   I thought it would be a small change.  Hm.  I was hoping that
   something along the lines of

   (defun csh-mode ()
     "docstring"
     (interactive)
     (shell-script-mode)
     (sh-set-shell "csh"))

   would be sufficient. 

Actually it does work.  I believe I again misunderstood you.  I
thought you wanted to make csh-mode and friends into genuine major
modes.  I somehow forgot to notice that you could bypass eval for any
expression whatsoever by doing (defun fake-mode () expression) and
putting -*- fake -*- in the first line of the file.  fake-mode does
not need to be a "legitimate" major or minor mode.

I still believe there is a problem.  We need one of these "fake" modes
for every interpreter currently supported to any degree and keep
adding new fake modes for every new interpreter or variation on
interpreter that would be supported in the future.  In other words,
the "fake mode" trick does not work well if you are trying to pass a
variable to a function, as we are trying to do here.

I still believe using find-file-hook is more natural.

Sincerely,

Luc.

Re: sh-script.el and magic numbers

[Kevin Rodgers]

Luc Teirlinck wrote:

> I believe that defining a bunch of derived modes of sh-mode would
> correspond to a substantial rewrite of sh-script.el and, actually a
> radical change in style and philosophy.
> 
> I do not believe it is necessary to go that far.


It may not be necessary to go that far to fix this particular bug, but it is the
right thing to do in the long term.

> What seems needed is
> fixing a concrete bug in sh-get-indent-info, maybe some similar not
> yet discovered bugs elsewhere in the indentation machinery (let us
> hope not), making sh-shell-file a permanent local, adding a couple of
> lines to the mode documentation and maybe make sh-mode add a small and
> simple function to find-file-hook.  These are all small local changes,
> whereas the change you propose would be a lot more radical.  Let us
> also not forget that in most, be it definitely not all, situations the
> implicit assumptions made by sh-script.el are actually satisfied.



-- 
<a href="mailto:&lt;kevin.rodgers&#64;ihs.com&gt;">Kevin Rodgers</a>

Re: sh-script.el and magic numbers

[Richard Stallman]

My decision is that we should support editing scripts that do not have
a #! line.  It may be a good idea to use a #! line, perhaps, but Emacs
should operate properly on scripts that don't have it.