non-scheme scripts: proposed solutions and their pros/cons

non-scheme scripts: proposed solutions and their pros/cons

[Ian Price]

As promised in the other thread, here is my list. This was really a
response to the even the earlier thread I started, which I
(unfortunately) didn't reply to at the time.

First off, they important question "why do we need this?". Well, guile
is a multi-language vm in principle, even if Scheme is it's first and
foremost citizen. If guile is to be a full-fledged vm for these other
languages, they need all (or at least most of) the rights and privileges
scheme does. This includes the ability to be run as scripts.

So, what are the solutions? Well, a few have cropped up, but I'm not
sure that individually any of them constitute a complete
solution. They are: a command-line argument, some marker in the
module, a different executable, and heuristics based on file
extension.

I shall treat these in order, though I can't pretend I will be
comprehensive. Clarifications and additions welcome.


* a command line argument
This was the position I initially proposed, and so had some bias
towards. The idea is very simple, we add a --language extension, the
argument to which would be the language of the file(s) we are trying
to execute.

** Pros
- Existing source files in the language do not need to be modified to
  use this.
- Requires (in theory) modifying only the command-line parsing

** Cons
- Has nothing to say about how e.g. scheme modules interact with elisp modules.
- #! only gives you one argument, and not all languages are going to
  support a \ type solution

* a file marker
The idea here it to have some way of marking in a file which language
it is written in, with some token like #!javascript, or perhaps
something like #!language javascript

** Pros
- works well with current module system
  e.g. say we have a file foo/bar.js and that had a #!javascript
  marker, then I could (use-modules (foo bar)) and guile would notice
  the marker, switch to javascript mode for it.
- we already do this for switching to curly-infix and r6rs reader
  modes

** Cons
- requires modifying existing source code, I couldn't just import an
  existing elisp (or whatever) file and use as is.
- the existing mechanism allows switching reader mode at an arbitrary
  part of the file. Not a big con, for #!language, I would simply say
  we disallow it in arbitrary places. However, people that extend
  guile as a language (i.e. lilypond), might disagree and would want
  to take advantage of this.

* different executable
This was a position posed in response to mine, that I was initially
against, but am somewhat more open to now. The idea is that for every
language $foo, we have a script guile-$foo which invokes that
language.
Technically it need not be a different executable, but one whose
action depends on argv[0] and just performs the appropriate action,
but in many respects these can be treated the same.

** Pros
- Existing source files in the language do not need to be modified to
  use this, they can change the existing languages symlink.
- We can handle common switches used by the language
- If you go for argv[0], may only require modifying the command-line
  parsing.

** Cons
- proliferation of names, even if just symlinks
- has nothing to say about cross-language interoperability


* heuristics
This always gets proposed, and I never like it, but hey ho. The idea
is to have some simple way to guess what language a file is written
in, for example, with a mapping of file extensions to languages.

** Pros
Best case scenario, you do nothing. You type guile foo.js, and it just
works.
- same excutable name
- don't need to modify existing code

** Cons
*** guessing on file extension
- guile allows user defined extensions
  Need to have a way of associating modes with new extensions
- some language share extensions
  I'm thinking of .pl for perl and prolog, but I'm sure there are
  other conflicts. Might not be a big con in practice
- Many files don't have an extension, think running with ./foo
*** guessing on content type
- Seems complicated to implement, especially for syntactically close
  languages. Probably undecidable in general.


My thanks to William Leslie[0], Neil Jerram[1], and quotemstr on #emacs for
pointing out various pros/cons.


So, the million dollar question: what do we do?

Well, I don't know. :)

Maybe the argv[0] solution, with some extension for requiring modules
in other languages built into use-modules (similarly for other
languages). But that's less nice since now the calling module needs to
know the language of the module it's requiring.

0. https://lists.gnu.org/archive/html/guile-devel/2012-07/msg00067.html
1. https://lists.gnu.org/archive/html/guile-devel/2012-07/msg00068.html

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Re: non-scheme scripts: proposed solutions and their pros/cons

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 6223 bytes --]

As you say, the only real solution is to do more than one of these things.

For instance, I think it's really important to be able to load modules
written in other languages. However, this may be language-dependent to a
certain extent, because some languages (Python) already have ways to define
modules. In those cases we should stick with their conventions, and use our
other methods for figuring out what language the file is in.

However, if we're using Guile in one language and loading an executable in
a different language, then we can't use a command-line argument or a
different executable to signal it. The only choices left are heuristics and
explicit markers. I think the only reasonable choice is both - use
heuristics, and let the user supply a marker if the heuristics are wrong.
(In the module case, one can imagine a heuristic based on having a
language-specific load path for each language, which might be very
effective.)

But for using Guile as an interpreter for different languages, a
command-line argument or argv[0] switch make a lot of sense.

What do you think?
Noah



On Tue, Nov 20, 2012 at 7:15 AM, Ian Price <ianprice90@googlemail.com>wrote:

>
> As promised in the other thread, here is my list. This was really a
> response to the even the earlier thread I started, which I
> (unfortunately) didn't reply to at the time.
>
> First off, they important question "why do we need this?". Well, guile
> is a multi-language vm in principle, even if Scheme is it's first and
> foremost citizen. If guile is to be a full-fledged vm for these other
> languages, they need all (or at least most of) the rights and privileges
> scheme does. This includes the ability to be run as scripts.
>
> So, what are the solutions? Well, a few have cropped up, but I'm not
> sure that individually any of them constitute a complete
> solution. They are: a command-line argument, some marker in the
> module, a different executable, and heuristics based on file
> extension.
>
> I shall treat these in order, though I can't pretend I will be
> comprehensive. Clarifications and additions welcome.
>
>
> * a command line argument
> This was the position I initially proposed, and so had some bias
> towards. The idea is very simple, we add a --language extension, the
> argument to which would be the language of the file(s) we are trying
> to execute.
>
> ** Pros
> - Existing source files in the language do not need to be modified to
>   use this.
> - Requires (in theory) modifying only the command-line parsing
>
> ** Cons
> - Has nothing to say about how e.g. scheme modules interact with elisp
> modules.
> - #! only gives you one argument, and not all languages are going to
>   support a \ type solution
>
> * a file marker
> The idea here it to have some way of marking in a file which language
> it is written in, with some token like #!javascript, or perhaps
> something like #!language javascript
>
> ** Pros
> - works well with current module system
>   e.g. say we have a file foo/bar.js and that had a #!javascript
>   marker, then I could (use-modules (foo bar)) and guile would notice
>   the marker, switch to javascript mode for it.
> - we already do this for switching to curly-infix and r6rs reader
>   modes
>
> ** Cons
> - requires modifying existing source code, I couldn't just import an
>   existing elisp (or whatever) file and use as is.
> - the existing mechanism allows switching reader mode at an arbitrary
>   part of the file. Not a big con, for #!language, I would simply say
>   we disallow it in arbitrary places. However, people that extend
>   guile as a language (i.e. lilypond), might disagree and would want
>   to take advantage of this.
>
> * different executable
> This was a position posed in response to mine, that I was initially
> against, but am somewhat more open to now. The idea is that for every
> language $foo, we have a script guile-$foo which invokes that
> language.
> Technically it need not be a different executable, but one whose
> action depends on argv[0] and just performs the appropriate action,
> but in many respects these can be treated the same.
>
> ** Pros
> - Existing source files in the language do not need to be modified to
>   use this, they can change the existing languages symlink.
> - We can handle common switches used by the language
> - If you go for argv[0], may only require modifying the command-line
>   parsing.
>
> ** Cons
> - proliferation of names, even if just symlinks
> - has nothing to say about cross-language interoperability
>
>
> * heuristics
> This always gets proposed, and I never like it, but hey ho. The idea
> is to have some simple way to guess what language a file is written
> in, for example, with a mapping of file extensions to languages.
>
> ** Pros
> Best case scenario, you do nothing. You type guile foo.js, and it just
> works.
> - same excutable name
> - don't need to modify existing code
>
> ** Cons
> *** guessing on file extension
> - guile allows user defined extensions
>   Need to have a way of associating modes with new extensions
> - some language share extensions
>   I'm thinking of .pl for perl and prolog, but I'm sure there are
>   other conflicts. Might not be a big con in practice
> - Many files don't have an extension, think running with ./foo
> *** guessing on content type
> - Seems complicated to implement, especially for syntactically close
>   languages. Probably undecidable in general.
>
>
> My thanks to William Leslie[0], Neil Jerram[1], and quotemstr on #emacs for
> pointing out various pros/cons.
>
>
> So, the million dollar question: what do we do?
>
> Well, I don't know. :)
>
> Maybe the argv[0] solution, with some extension for requiring modules
> in other languages built into use-modules (similarly for other
> languages). But that's less nice since now the calling module needs to
> know the language of the module it's requiring.
>
> 0. https://lists.gnu.org/archive/html/guile-devel/2012-07/msg00067.html
> 1. https://lists.gnu.org/archive/html/guile-devel/2012-07/msg00068.html
>
> --
> Ian Price -- shift-reset.com
>
> "Programming is like pinball. The reward for doing it well is
> the opportunity to do it again" - from "The Wizardy Compiled"
>
>

[-- Attachment #2: Type: text/html, Size: 7434 bytes --]

Re: non-scheme scripts: proposed solutions and their pros/cons

[Ian Price]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> For instance, I think it's really important to be able to load modules written in
> other languages. However, this may be language-dependent to a certain extent, because
> some languages (Python) already have ways to define modules. In those cases we should
> stick with their conventions, and use our other methods for figuring out what
> language the file is in.
For now, I think we can kinda punt on this issue. Mixing languages is
hard; getting an interpreter running isn't, and the two seem orthogonal
enough we can tackle them separately.


> However, if we're using Guile in one language and loading an executable in a
> different language, then we can't use a command-line argument or a different
> executable to signal it. The only choices left are heuristics and explicit markers. I
> think the only reasonable choice is both - use heuristics, and let the user supply a
> marker if the heuristics are wrong. (In the module case, one can imagine a heuristic
> based on having a language-specific load path for each language, which might be very
> effective.)
I said as much in my summary, but I really don't like any of the
solutions to the mixed-modules problem, but I think we are going to need
some proof-of-concepts and some experience for these before we know how
good/bad they are really going to be.

> But for using Guile as an interpreter for different languages, a command-line
> argument or argv[0] switch make a lot of sense.
I've had quite a few votes for argv[0] so that's probably what it's
going to be. --lang might be worth adding as an option under the "guile"
name (or --from to match "guild compile")

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Re: non-scheme scripts: proposed solutions and their pros/cons

[Ian Price]

So, here's the "plan of attack" I'm envisioning for this.

Right now, questions of cross-language module referencing can be
ignored. I think it is mostly orthogonal to the current goal of running
non-scheme scripts.

First, I'm going to try and write a proof-of-concept guile-elisp
executable. This shouldn't be too hard, I think, and may shed some light
on expected difficulties.

Secondly, the more serious version. While I can't know all the details
till I've done the previous stage, here is what I think I need to do.

command-line parsing needs to be moved under the language/$lang
directories, and a command line parser field added to guile's language
objects. Languages that wish to customise it can add one, or we can have
a default one that mostly mimics guile's current interface, and allows
executing scripts, lines, etc.

We may also want to all language objects to specify one or a set of
allowed argv[0] names.

While I do not know if anyone actually uses it, we will probably need to
keep ice-9/command-line.scm functionally intact for backwards
compatibility.

With this additional layer of indirection[0], we now dispatch to the
correct command-line parser based on the argv[0] name, set the repl
language etc. etc.

One important issue with this is that it is likely to slow down the time
it takes guile to boot, however, is should accommodate any user-written
languages.

The other thing is, I'm not sure how this is going to affect the C code
for guile. It _might_ be possible to do this all in Scheme, but until
I've tried, I'm going to remain sceptical.

If you have a better suggestion please tell me :)

0. as we all know, "in computer science, every problem can be solved
with an additional layer of indirection"

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Re: non-scheme scripts: proposed solutions and their pros/cons

[Ian Price]

Ian Price <ianprice90@googlemail.com> writes:

> First, I'm going to try and write a proof-of-concept guile-elisp
> executable. This shouldn't be too hard, I think, and may shed some light
> on expected difficulties.

I was distracted by the pfds release so it's taken me longer than it
should have, but as expected, it wasn't difficult _once I knew where to
look_. The general template is

#!/usr/local/bin/guile -s
!#

(use-modules (system base compile)
             (system repl repl))

(let* ((args (command-line))
       (argv0 (car args))
       (files (cdr args))
       (load-file (lambda (file)
                    (compile-file file #:from 'elisp #:to 'value))))
  ;; ^^ Imagine sophisticated command-line parsing :)
  (if (null? files)
      (start-repl 'elisp)
      (for-each load-file files)))

Be aware you get lots of warnings if you actually run this, since elisp
overrides a bunch of bindings.
      
Extending it to handle different argv0s seems obvious. Handling -c would
involve loading the relevant reader, and using compile (maybe write that
as a compile-from-string function). -e is a little tricker.

-- 
Ian Price -- shift-reset.com

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"