[bug#46231] Add emilua

[bug#46231] Add emilua

[Vinícius dos Santos Oliveira]

Hi,

I've created a new package for guix. Here's its definition:
https://gitlab.com/emilua/emilua/-/snippets/2068658

Please let me know if it has any issues so I can fix them until it's
ready for inclusion in the main repository.

-- 
Vinícius dos Santos Oliveira
https://vinipsmaker.github.io/

[bug#46231] Add emilua

[Nicolas Goaziou]

Hello,

Vinícius dos Santos Oliveira <vini.ipsmaker@gmail.com> writes:

> I've created a new package for guix. Here's its definition:
> https://gitlab.com/emilua/emilua/-/snippets/2068658

Thank you!

> Please let me know if it has any issues so I can fix them until it's
> ready for inclusion in the main repository.

There are some issue to fix:
- the name of the custom luajit2 package should be changed. Maybe
  openresty-luajit?
- that package should have a proper version and build tag out of it.
- I see that emilua uses #:recursive? set to #true. Could some
  submodules could be unbundled by any chance?
- You disabled tests. I would be nice to provide a reason for that.
- You don't need `string-append' in the description. Just write the full
  textin a single string.

Could you split this into two patches and send them here, for easier review?

Regards,
-- 
Nicolas Goaziou

[bug#46231] Add emilua

[Vinícius dos Santos Oliveira]

Em sáb., 20 de fev. de 2021 às 22:21, Nicolas Goaziou
<mail@nicolasgoaziou.fr> escreveu:
> There are some issue to fix:
> - the name of the custom luajit2 package should be changed. Maybe
>   openresty-luajit?
> - that package should have a proper version and build tag out of it.

I've used "define" instead "define-public" so it wouldn't be really
available elsewhere in the system. However after some thought I do
agree that this dependency can be generally useful to more people and
should have a proper name.

What do you think about the name openresty-luajit-lua52?

> - I see that emilua uses #:recursive? set to #true. Could some
>   submodules could be unbundled by any chance?

Some, yes, but not all. A Debian packager asked me the same and I
already started to unbundle the dependencies where it can make sense
to use the one found in the system. I can offer more details on why a
few dependencies still must be provided as submodules if requested.
Next emilua version (a couple of weeks away) should use more
dependencies from the system and less bundled dependencies, but you'll
still need #:recursive? set to #true for the bundled dependencies that
will remain.

> - You disabled tests. I would be nice to provide a reason for that.

This is how the tests are run:

1. The emilua binary is generated.
2. The emilua binary is executed against every *.lua file inside the test dir.
3. An output is generated and compared against the matching *.out file
inside the same dir.
4. If they differ, the test fails.

There are two types of tests:

* Good test cases. No error expected.
* Bad test cases. Buggy code is feeded to the runtime to see if it is
able to detect the error.

On exceptions (the bad test cases), Lua throws an error string and
Emilua will print the stacktrace. The stacktrace is generated by the
Lua VM and contains paths to the files involved. This path will be
different in every machine you run the tests as you probably will
store the source code in different paths. Here's an example:

$ emilua test/interrupt7.lua
Main fiber from VM 0x7f1f133d1380 panicked: 'Invalid argument'
stack traceback:
  /home/me/emilua/test/interrupt7.lua:5: in main chunk
  [C]: in function ''
  [string "?"]: in function <[string "?"]:0>

So I filter the test output using an AWK script that replaces the
filename by "input.lua". Here's the translation line:
https://gitlab.com/emilua/emilua/-/blob/ded59fa9d4bb8977aed40fb7b683c6403b462f28/test/run_test.awk#L60

This filter ensures that a path in the likes of
"/home/me/emilua/test/interrupt7.lua" will be spit as "input.lua".
That makes the output predictable so I can write the *.out files that
reference input.lua. Here's the matching test/interrupt7.out from the
previous run:

Main fiber from VM 0x1 panicked: 'Invalid argument'
stack traceback:
  input.lua:5: in main chunk
  [C]: in function ''
  [string "?"]: in function <[string "?"]:0>

Unfortunately this strategy is not enough. The Lua VM will shorten
paths by excluding some of its parts and replace them by "..." when
they are too long. That's why the script has this rule:
https://gitlab.com/emilua/emilua/-/blob/ded59fa9d4bb8977aed40fb7b683c6403b462f28/test/run_test.awk#L4

However running the tests from the guix build unveiled yet another
problem with this script. If even longer paths are at play (as in the
guix build), the Lua VM will apply a different shortening algorithm to
the paths sometimes. So the tests run, the emilua binary will do the
right thing, but the translation script won't work on one of the test
cases and will raise a false negative. I intend to solve this issue
and I already started to investigate so it'll eventually be solved,
but in the meantime I just disabled the tests. I don't control the
paths the Lua VM generates so improving the AWK script is really my
only choice here.

> - You don't need `string-append' in the description. Just write the full
>   textin a single string.

Okay.

> Could you split this into two patches and send them here, for easier review?

Sure. Let's get the openresty-luajit-lua52 package in first and then
I'll work on the second patch.

Here's the patch for openresty-luajit-lua52 alone:
https://gitlab.com/emilua/emilua/-/snippets/2079960


-- 
Vinícius dos Santos Oliveira
https://vinipsmaker.github.io/

[bug#46231] Add emilua

[Nicolas Goaziou]

Hello,

Vinícius dos Santos Oliveira <vini.ipsmaker@gmail.com> writes:

Thank you for your detailed answer! Some comments follow.

> I've used "define" instead "define-public" so it wouldn't be really
> available elsewhere in the system.

I know. But this does not prevent to choose an explicit name.

> However after some thought I do agree that this dependency can be
> generally useful to more people and should have a proper name.
>
> What do you think about the name openresty-luajit-lua52?

The trend in other distributions is to use luajit-openresty, in this
order. So maybe luajit-lua52-openresty? Or luajit-openresty if that is
unambiguous. It is not a big issue anyway.

>> - I see that emilua uses #:recursive? set to #true. Could some
>>   submodules could be unbundled by any chance?
>
> Some, yes, but not all. A Debian packager asked me the same and I
> already started to unbundle the dependencies where it can make sense
> to use the one found in the system. I can offer more details on why a
> few dependencies still must be provided as submodules if requested.
> Next emilua version (a couple of weeks away) should use more
> dependencies from the system and less bundled dependencies, but you'll
> still need #:recursive? set to #true for the bundled dependencies that
> will remain.

Then could you add a comment explaining what will be required, and what
can be removed later on (as a TODO)?

> I intend to solve this issue and I already started to investigate so
> it'll eventually be solved, but in the meantime I just disabled the
> tests. I don't control the paths the Lua VM generates so improving the
> AWK script is really my only choice here.

OK. Then could you add a short comment explaining the issue, and that it
is being worked on?

> Sure. Let's get the openresty-luajit-lua52 package in first and then
> I'll work on the second patch.

Great!

> Here's the patch for openresty-luajit-lua52 alone:
> https://gitlab.com/emilua/emilua/-/snippets/2079960

The version should be "2.1-20201027" and the commit (string-append "v"
version). Don't forget to remove the string-append in the description.
Also, the description itself is not particularly clear. Maybe

     This package provides the official OpenResty branch of LuaJIT.  It
     is not to be considered a fork, since changes are regularly
     synchronized from the upstream LuaJIT project.

Could you send this to this ML using git format-patch?

Regards,
-- 
Nicolas Goaziou

[bug#46231] Add emilua

[Vinícius dos Santos Oliveira]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

Em seg., 22 de fev. de 2021 às 07:04, Nicolas Goaziou
<mail@nicolasgoaziou.fr> escreveu:
> The trend in other distributions is to use luajit-openresty, in this
> order. So maybe luajit-lua52-openresty? Or luajit-openresty if that is
> unambiguous. It is not a big issue anyway.

luajit-openresty might be ambiguous as not every package might depend
on the lua52 configure flag, so I renamed it to
luajit-lua52-openresty.

> Could you send this to this ML using git format-patch?

You may find the patch with the requested fixes attached to this
email. I'll implement the changes to the other package once we have
luajit-lua52-openresty in place.

Thanks for acting as the reviewer of this patch.


-- 
Vinícius dos Santos Oliveira
https://vinipsmaker.github.io/

[-- Attachment #2: 0001-Add-luajit-lua52-openresty.patch --]
[-- Type: text/x-patch, Size: 1877 bytes --]

From c5d11f986595637ae72d551acf0bf2cd15390a39 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20dos=20Santos=20Oliveira?=
 <vini.ipsmaker@gmail.com>
Date: Tue, 23 Feb 2021 17:25:07 -0300
Subject: [PATCH] Add luajit-lua52-openresty

---
 gnu/packages/lua.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/lua.scm b/gnu/packages/lua.scm
index b029400cb2..8fbcc0b317 100644
--- a/gnu/packages/lua.scm
+++ b/gnu/packages/lua.scm
@@ -157,6 +157,35 @@ language.  It may be embedded or used as a general-purpose, stand-alone
 language.")
     (license license:x11)))
 
+(define-public luajit-lua52-openresty
+  (package
+   (inherit luajit)
+   (name "luajit-lua52-openresty")
+   (version "2.1-20201027")
+   (source (origin
+            (method git-fetch)
+            (uri (git-reference
+                  (url "https://github.com/openresty/luajit2.git")
+                  (commit (string-append "v" version))))
+            (sha256
+             (base32
+              "115gixlbik4fx62bv8m1psfvv48alcvg3hki5kz3hcpa830gd7zp"))
+            (patches (search-patches
+                      "luajit-enable-lua52compat.patch"))))
+   (arguments
+    `(#:tests? #f
+      #:phases (modify-phases %standard-phases
+                              (delete 'configure))
+      #:make-flags (list
+                    (string-append "PREFIX=" (assoc-ref %outputs "out")))))
+   (synopsis "OpenResty's Branch of LuaJIT 2")
+   (description
+    "This package provides the official OpenResty branch of LuaJIT. It is not to
+be considered a fork, since changes are regularly synchronized from the upstream
+LuaJIT project. This package also enables the Lua 5.2 compat mode needed by some
+projects.")
+   (home-page "https://github.com/openresty/luajit2")))
+
 (define (make-lua-expat name lua)
   (package
     (name name)
-- 
2.30.1

[bug#46231] Add emilua

[Nicolas Goaziou]

Hello,

Vinícius dos Santos Oliveira <vini.ipsmaker@gmail.com> writes:

> luajit-openresty might be ambiguous as not every package might depend
> on the lua52 configure flag, so I renamed it to
> luajit-lua52-openresty.

OK.

> You may find the patch with the requested fixes attached to this
> email. I'll implement the changes to the other package once we have
> luajit-lua52-openresty in place.

You didn't include "luajit-enable-lua52compat.patch". Could you send it
too?

Also, there is a new release: 2.1-20201229. You may want to send an
updated version. I can also update it.

Thanks!

Regards,
-- 
Nicolas Goaziou

[bug#46231] Add emilua

[Vinícius dos Santos Oliveira]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

Em qua., 24 de fev. de 2021 às 10:07, Nicolas Goaziou
<mail@nicolasgoaziou.fr> escreveu:
> > You may find the patch with the requested fixes attached to this
> > email. I'll implement the changes to the other package once we have
> > luajit-lua52-openresty in place.
>
> You didn't include "luajit-enable-lua52compat.patch". Could you send it
> too?

Oops, my bad. You may find it attached to this email.

> Also, there is a new release: 2.1-20201229. You may want to send an
> updated version. I can also update it.

I'll let you do the update then.


-- 
Vinícius dos Santos Oliveira
https://vinipsmaker.github.io/

[-- Attachment #2: luajit-enable-lua52compat.patch --]
[-- Type: text/x-patch, Size: 882 bytes --]

From 8573754371e5c34fe8e2a4e88f33573daedafb83 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20dos=20Santos=20Oliveira?=
 <vini.ipsmaker@gmail.com>
Date: Mon, 1 Feb 2021 00:11:55 -0300
Subject: [PATCH] Enable -DLUAJIT_ENABLE_LUA52COMPAT

---
 src/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/Makefile b/src/Makefile
index 3d7d2542..1a3a0324 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -99,7 +99,7 @@ XCFLAGS=
 # enabled by default. Some other features that *might* break some existing
 # code (e.g. __pairs or os.execute() return values) can be enabled here.
 # Note: this does not provide full compatibility with Lua 5.2 at this time.
-#XCFLAGS+= -DLUAJIT_ENABLE_LUA52COMPAT
+XCFLAGS+= -DLUAJIT_ENABLE_LUA52COMPAT
 #
 # Disable the JIT compiler, i.e. turn LuaJIT into a pure interpreter.
 #XCFLAGS+= -DLUAJIT_DISABLE_JIT
-- 
2.30.0

[bug#46231] Add emilua

[Nicolas Goaziou]

Vinícius dos Santos Oliveira <vini.ipsmaker@gmail.com> writes:

> Oops, my bad. You may find it attached to this email.

OK. Since it was trivial, I used a phase instead.

>> Also, there is a new release: 2.1-20201229. You may want to send an
>> updated version. I can also update it.
>
> I'll let you do the update then.

Done.

I applied the patch. We can now proceed with emilua.

Thanks!

Regards,

[bug#46231] Add emilua

[Vinícius dos Santos Oliveira]

[-- Attachment #1: Type: text/plain, Size: 2977 bytes --]

Em qua., 24 de fev. de 2021 às 12:49, Nicolas Goaziou
<mail@nicolasgoaziou.fr> escreveu:
> > I'll let you do the update then.
> Done.

Thanks.

> I applied the patch. We can now proceed with emilua.

You may find the updated patch (using git format) attached in this email.

Btw, if you don't mind, I'd like to use the space to also make a few
questions guix-related. Emilua is a software similar to NodeJS, but it
doesn't have its own npm, pip, gem, or anything alike. I'd like to use
guix as the package manager in emilua. I don't want to create another
npm. Therefore all the code I'm writing for the emilua's next version
was done with guix in mind.

I've learned about native-search-paths and how it can be used to fill
environment variables such as GST_PLUGIN_PATH. I've learned about the
approach to "merge" directories from each package to fill the guix
environment/profile and how crucial it is for such environment
variables to guide the software running inside it (e.g. python and
PYTHONPATH). That's pretty much solved for me (unless there are other
approaches in guix that I'm missing and they would fit better).

Next step for me is to better understand what's the proper take when
dealing with GStreamer native-search-paths and pkg-config. I'm talking
about GStreamer here because it faces a similar problem and I believe
it's easier to talk on issues from existing and packaged software.

GStreamer defines native-search-paths to
GST_PLUGIN_SYSTEM_PATH=lib/gstreamer-1.0. As far as I tested, if a
package gstreamer-foobar depends on gstreamer and install files to
${prefix}/lib/gstreamer-1.0, then GST_PLUGIN_SYSTEM_PATH will be
automatically updated to also contain gstreamer-foobar's
/gnu/store/*/lib/gstreamer-1.0.

My question here is: how should the software packaged in
gstreamer-foobar find out the proper directory to install its plugins?
Should it be ${prefix}/lib/gstreamer-1.0? Should it be
${prefix}/lib/gstreamer-1.1? That's something that shouldn't be
hardcoded. As far as I see, it can just use the pluginsdir from
GStreamer's pkg-config definition. Like so:

$ pkg-config --variable=pluginsdir gstreamer-1.0

However this command will print the wrong dir in guix. It'll print
something like:

/gnu/store/9if71w58d5mkxfxyc7fpz289qssnkqsv-gstreamer-1.18.2/lib/gstreamer-1.0

But that's the "namespace" for the gstreamer package, not for the
gstreamer-foobar package. A solution would be to invoke pkg-config as
follows:

$ pkg-config --define-variable="prefix=${prefix}"
--variable=pluginsdir gstreamer-1.0

This will actually print the proper ${prefix}/lib/gstreamer-1.0. But
that's my question here. How should pkg-config be invoked properly? I
could just as well invoke it as:

$ pkg-config --define-variable="prefix=${prefix}"
--define-variable=libdir='${prefix}/lib' --variable=pluginsdir
gstreamer-1.0


-- 
Vinícius dos Santos Oliveira
https://vinipsmaker.github.io/

[-- Attachment #2: 0001-Add-emilua.patch --]
[-- Type: text/x-patch, Size: 3839 bytes --]

From 10c04b9b70886b14377c8a80165978601c8617ce Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20dos=20Santos=20Oliveira?=
 <vini.ipsmaker@gmail.com>
Date: Thu, 25 Feb 2021 10:22:20 -0300
Subject: [PATCH] Add emilua

---
 gnu/packages/lua.scm | 60 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/gnu/packages/lua.scm b/gnu/packages/lua.scm
index c03eea3c70..b98427cbde 100644
--- a/gnu/packages/lua.scm
+++ b/gnu/packages/lua.scm
@@ -37,8 +37,11 @@
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system cmake)
+  #:use-module (guix build-system meson)
   #:use-module (guix build-system trivial)
   #:use-module (gnu packages)
+  #:use-module (gnu packages gcc)
+  #:use-module (gnu packages build-tools)
   #:use-module (gnu packages readline)
   #:use-module (gnu packages m4)
   #:use-module (gnu packages tls)
@@ -47,6 +50,11 @@
   #:use-module (gnu packages libevent)
   #:use-module (gnu packages libffi)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages boost)
+  #:use-module (gnu packages tls)
+  #:use-module (gnu packages ncurses)
+  #:use-module (gnu packages vim)
+  #:use-module (gnu packages re2c)
   #:use-module (gnu packages xorg)
   #:use-module (gnu packages gtk))
 
@@ -1151,3 +1159,55 @@ shell command executions.")
 simplicity, and reach of Lua with the flexibility of a Lisp syntax and macro
 system.")
     (license license:expat)))
+
+(define-public emilua
+  (package
+   (name "emilua")
+   (version "0.2.1")
+   (source (origin
+            (method git-fetch)
+            (uri (git-reference
+                  (url "https://gitlab.com/emilua/emilua.git")
+                  (commit (string-append "v" version))
+                  ; Current version requires bundled CLI11 and fmt, but at some
+                  ; future release the ones found in the system could be used
+                  ; instead. Current version also requires Trial.Protocol and
+                  ; the HTTP lib developed as part of GSoC 2014 for Boost, but
+                  ; these are dependencies unlikely to be "unbundled" in future
+                  ; releases.
+                  (recursive? #t)))
+            (sha256
+             (base32
+              "1d6k5v6x85fbvz2ijq1imnfdwvqmsav4xp021a5v3ah4mgy7yann"))))
+   (build-system meson-build-system)
+   (arguments
+    `(#:meson ,meson-0.55
+      ; Tests are disabled for now due to an issue that affecs guix:
+      ; <https://gitlab.com/emilua/emilua/-/issues/22>
+      #:configure-flags '("-Denable_http=false" "-Denable_tests=false")))
+   (inputs
+    `(("boost" ,boost)
+      ("boost-static" ,boost-static)
+      ; LuaJIT has a 2GiB addressing limit[1] that has been fixed on OpenResty
+      ; fork. Emilua is severely affected by this limit, so the upstream package
+      ; is avoided. Emilua also depends on the -DLUAJIT_ENABLE_LUA52COMPAT
+      ; configure flag[2] for some features to work (e.g. __pairs on HTTP
+      ; headers).
+      ;
+      ; [1] <http://hacksoflife.blogspot.com/2012/12/integrating-luajit-with-x-plane-64-bit.html>
+      ; [2] <http://luajit.org/extensions.html#lua52>
+      ("luajit-lua52-openresty" ,luajit-lua52-openresty)
+      ("openssl" ,openssl)
+      ("ncurses" ,ncurses)))
+   (native-inputs
+    `(("pkg-config" ,pkg-config)
+      ("gcc" ,gcc-10) ; gcc-7 is too old for our C++17 needs
+      ("luajit-lua52-openresty" ,luajit-lua52-openresty)
+      ("xxd" ,xxd)
+      ("re2c" ,re2c)))
+   (synopsis "Lua execution engine")
+   (description
+    "A LuaJIT-based Lua execution engine that supports async IO, fibers and
+actor-inspired threading. The experimental builtin HTTP module is enabled.")
+   (home-page "https://gitlab.com/emilua/emilua")
+   (license license:boost1.0)))
-- 
2.30.1

bug#46231: Add emilua

[Nicolas Goaziou]

Hello,

Vinícius dos Santos Oliveira <vini.ipsmaker@gmail.com> writes:

> You may find the updated patch (using git format) attached in this
> email.

Thank you!

I re-ordered inputs alphabetically, fixed comment syntax (Lisp uses two
semicolons for full line comments), added a copyright line and a proper
commit message, and applied your patch.

> Btw, if you don't mind, I'd like to use the space to also make a few
> questions guix-related.

I think a more appropriate place would be Guix Devel mailing list
<mailto:guix-devel@gnu.org>. May I ask you to post again your questions
there? Your message will get more visibility from people way more
competent than I am on this subject.

Regards,
-- 
Nicolas Goaziou