Web development

Web development

[Zelphir Kaltstahl]

Hello Guile Users!

Today I wanted to try developing a website in Guile again. I took a look
at GNU Artanis, but unfortunately it seems to be not compatible with
Guile 3 or at least not yet released for Guile 3. At least version 0.4.1
on Guix seems to still depend on Guile-2.2.x. So I thought "Hmmm I had
that example project once, but the issue was with serving static files
not being a good idea using Guile's web server." So that leads me to the
next point: What HTTP server to use?

- I could use NGINX, but then again I don't really like NGINX
configurations, even though I am familiar with it.

- I could try and get familiar with HAProxy. This seems to have the
advantage of being GPLv2.

But how to create a development setup? I'd like to not install Docker on
this particular system (so far it is very clean, only using free
software), but still keep my system clean. Perhaps I could use Guix
somehow (Guix containers?). Both, NGINX and HAProxy, seem available on Guix.

I seem to remember, that sending static files via Guile's web server
would be slow or insecure and that other HTTP servers make use of
sendfile or something, to be fast. Also they take care of MIME types
etc. I would be OK with being a bit slower, if I don't have to make a
huge effort to handle MIME types and whatever other stuff correctly. But
if there was something already implemented in Guile somewhere, which I
could copy for my project, perhaps that would work too.

I would like to have a local development environment, where I can start
a server (and the Guile service behind it) with a command and shut it
down again, but keeping my system rather clean.

So I have a few questions:

(1) How do you do your Guile web development? What's the setup?

(2) What do you use to serve static files (securely)? If you use Guile's
web server, how exactly do you do it? Do you have the code somewhere?

(3) Perhaps there is a minimalistic option instead of NGINX or HAProxy
out there, which is also free software? Is there perhaps even anything
in Guile, which I could use, that is suitable for serving static files?

Best regards,
Zelphir

-- 
repositories: https://notabug.org/ZelphirKaltstahl

Re: [EXT] Web development

[Thompson, David]

On Fri, Sep 4, 2020 at 11:44 AM Zelphir Kaltstahl
<zelphirkaltstahl@posteo.de> wrote:
>
> Hello Guile Users!
>
> Today I wanted to try developing a website in Guile again. I took a look
> at GNU Artanis, but unfortunately it seems to be not compatible with
> Guile 3 or at least not yet released for Guile 3. At least version 0.4.1
> on Guix seems to still depend on Guile-2.2.x. So I thought "Hmmm I had
> that example project once, but the issue was with serving static files
> not being a good idea using Guile's web server." So that leads me to the
> next point: What HTTP server to use?
>
> - I could use NGINX, but then again I don't really like NGINX
> configurations, even though I am familiar with it.
>
> - I could try and get familiar with HAProxy. This seems to have the
> advantage of being GPLv2.
>
> But how to create a development setup? I'd like to not install Docker on
> this particular system (so far it is very clean, only using free
> software), but still keep my system clean. Perhaps I could use Guix
> somehow (Guix containers?). Both, NGINX and HAProxy, seem available on Guix.
>
> I seem to remember, that sending static files via Guile's web server
> would be slow or insecure and that other HTTP servers make use of
> sendfile or something, to be fast. Also they take care of MIME types
> etc. I would be OK with being a bit slower, if I don't have to make a
> huge effort to handle MIME types and whatever other stuff correctly. But
> if there was something already implemented in Guile somewhere, which I
> could copy for my project, perhaps that would work too.
>
> I would like to have a local development environment, where I can start
> a server (and the Guile service behind it) with a command and shut it
> down again, but keeping my system rather clean.
>
> So I have a few questions:
>
> (1) How do you do your Guile web development? What's the setup?
>
> (2) What do you use to serve static files (securely)? If you use Guile's
> web server, how exactly do you do it? Do you have the code somewhere?
>
> (3) Perhaps there is a minimalistic option instead of NGINX or HAProxy
> out there, which is also free software? Is there perhaps even anything
> in Guile, which I could use, that is suitable for serving static files?

I have never run a guile web application in a production environment
but the same general pattern for web applications written in other
languages would apply here:

* nginx (or equivalent) sits in front and all requests go through it first
* nginx handles all static file requests because it is very good at
serving static files
* all other requests are passed to the guile web application via reverse proxy

Artanis is the only web framework I know of for Guile, but I've never
used it. I wrote the initial version of the 'guix publish' tool using
Guile's built-in web server and used the built-in pattern matcher for
handling the very simple request routing it needed.

Hope this helps,

- Dave

Re: [EXT] Web development

[Zelphir Kaltstahl]

Hi David!

On 9/4/20 5:56 PM, Thompson, David wrote:
> On Fri, Sep 4, 2020 at 11:44 AM Zelphir Kaltstahl
> <zelphirkaltstahl@posteo.de> wrote:
>> Hello Guile Users!
>>
>> Today I wanted to try developing a website in Guile again. I took a look
>> at GNU Artanis, but unfortunately it seems to be not compatible with
>> Guile 3 or at least not yet released for Guile 3. At least version 0.4.1
>> on Guix seems to still depend on Guile-2.2.x. So I thought "Hmmm I had
>> that example project once, but the issue was with serving static files
>> not being a good idea using Guile's web server." So that leads me to the
>> next point: What HTTP server to use?
>>
>> - I could use NGINX, but then again I don't really like NGINX
>> configurations, even though I am familiar with it.
>>
>> - I could try and get familiar with HAProxy. This seems to have the
>> advantage of being GPLv2.
>>
>> But how to create a development setup? I'd like to not install Docker on
>> this particular system (so far it is very clean, only using free
>> software), but still keep my system clean. Perhaps I could use Guix
>> somehow (Guix containers?). Both, NGINX and HAProxy, seem available on Guix.
>>
>> I seem to remember, that sending static files via Guile's web server
>> would be slow or insecure and that other HTTP servers make use of
>> sendfile or something, to be fast. Also they take care of MIME types
>> etc. I would be OK with being a bit slower, if I don't have to make a
>> huge effort to handle MIME types and whatever other stuff correctly. But
>> if there was something already implemented in Guile somewhere, which I
>> could copy for my project, perhaps that would work too.
>>
>> I would like to have a local development environment, where I can start
>> a server (and the Guile service behind it) with a command and shut it
>> down again, but keeping my system rather clean.
>>
>> So I have a few questions:
>>
>> (1) How do you do your Guile web development? What's the setup?
>>
>> (2) What do you use to serve static files (securely)? If you use Guile's
>> web server, how exactly do you do it? Do you have the code somewhere?
>>
>> (3) Perhaps there is a minimalistic option instead of NGINX or HAProxy
>> out there, which is also free software? Is there perhaps even anything
>> in Guile, which I could use, that is suitable for serving static files?
> I have never run a guile web application in a production environment
> but the same general pattern for web applications written in other
> languages would apply here:
>
> * nginx (or equivalent) sits in front and all requests go through it first
> * nginx handles all static file requests because it is very good at
> serving static files
> * all other requests are passed to the guile web application via reverse proxy
>
> Artanis is the only web framework I know of for Guile, but I've never
> used it. I wrote the initial version of the 'guix publish' tool using
> Guile's built-in web server and used the built-in pattern matcher for
> handling the very simple request routing it needed.
>
> Hope this helps,
>
> - Dave

OK thanks for the pointer, I'll look at guix publish!

Best regards,
Zelphir

Re: [EXT] Web development

[Joshua Branson via General Guile related discussions]

I've actually been making videos of me doing guile web development with
nginx and David's Haunt.  My latest video (I don't know if I've uploaded
it yet), is starting to build a static site, build by haunt, which has a
submission form, which is processed by a running guile program.

https://video.hardlimit.com/accounts/joshua_branson/video-channels

The videos are titled "Learning Guile Web Development...", and they may
be pretty basic, because honestly I'm still learning.

Joshua,

P.S.  I would love to learn artanis at some point.

--
Joshua Branson
Sent from Emacs and Gnus

Re: Web development

[Ricardo Wurmus]

Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> writes:

> So I have a few questions:
>
> (1) How do you do your Guile web development? What's the setup?

I don’t use a framework in my web projects; just the included web
modules and sometimes the fibers web server.  I keep everything
stateless so that I can spin up many workers (i.e. Guile processes
with a web server listening on one of a range of local ports) and
dispatch to them from a reverse proxy such as Nginx or Lighttpd.

> (2) What do you use to serve static files (securely)? If you use Guile's
> web server, how exactly do you do it? Do you have the code somewhere?

I configure an assets directory and define a procedure that sanitizes
the requested file name to serve it from that directory.  I don’t do
much with files so I don’t usually do anything other than

    (call-with-input-file file-name get-bytevector-all)

for the body of the response.  But if I had to send large files I’d use
“sendfile” directly.

> (3) Perhaps there is a minimalistic option instead of NGINX or HAProxy
> out there, which is also free software? Is there perhaps even anything
> in Guile, which I could use, that is suitable for serving static files?

Lighttpd seems to be lighter than Nginx.  If all you want is serve
static files upon request I’d use just Guile, matching on the request,
looking up the file in a declared directory, and using sendfile to push
it to the requester.

-- 
Ricardo

Re: Web development

[Zelphir Kaltstahl]

Hi Ricardo!

On 9/4/20 6:44 PM, Ricardo Wurmus wrote:
> Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> writes:
>
>> So I have a few questions:
>>
>> (1) How do you do your Guile web development? What's the setup?
> I don’t use a framework in my web projects; just the included web
> modules and sometimes the fibers web server.  I keep everything
> stateless so that I can spin up many workers (i.e. Guile processes
> with a web server listening on one of a range of local ports) and
> dispatch to them from a reverse proxy such as Nginx or Lighttpd.
>
>> (2) What do you use to serve static files (securely)? If you use Guile's
>> web server, how exactly do you do it? Do you have the code somewhere?
> I configure an assets directory and define a procedure that sanitizes
> the requested file name to serve it from that directory.  I don’t do
> much with files so I don’t usually do anything other than
>
>     (call-with-input-file file-name get-bytevector-all)
>
> for the body of the response.  But if I had to send large files I’d use
> “sendfile” directly.

Don't you need to also check what kind of file type it is and select the
appropriate MIME type for answering the request? (Do you have some code
public somewhere, showing how to do this?)

And how do you use sendfile directly? (Code example for this would also
be great to see.)

(I've checked https://github.com/rekado and https://git.elephly.net/,
but could not find a web server project.)

Is it still necessary to sanitize the requested file names, when an HTTP
server handles requests for assets before the Guile application is hit?


>> (3) Perhaps there is a minimalistic option instead of NGINX or HAProxy
>> out there, which is also free software? Is there perhaps even anything
>> in Guile, which I could use, that is suitable for serving static files?
> Lighttpd seems to be lighter than Nginx.  If all you want is serve
> static files upon request I’d use just Guile, matching on the request,
> looking up the file in a declared directory, and using sendfile to push
> it to the requester.


Best regards,
Zelphir

Re: Web development

[Ricardo Wurmus]

Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> writes:

>>> (2) What do you use to serve static files (securely)? If you use Guile's
>>> web server, how exactly do you do it? Do you have the code somewhere?
>> I configure an assets directory and define a procedure that sanitizes
>> the requested file name to serve it from that directory.  I don’t do
>> much with files so I don’t usually do anything other than
>>
>>     (call-with-input-file file-name get-bytevector-all)
>>
>> for the body of the response.  But if I had to send large files I’d use
>> “sendfile” directly.
>
> Don't you need to also check what kind of file type it is and select the
> appropriate MIME type for answering the request?

Yes, you need to send along the correct MIME type.  In my projects I
know the MIME type of all files I’ll send ahead of time, so I can get
away with looking up the file extension in an alist of extensions to
MIME types.

Something like that:

--8<---------------cut here---------------start------------->8---
(define (render-static-file root path)
  ;; PATH is a list of path components
  (let ((file-name (string-join (cons* root path) "/")))
    (if (and (not (any (cut string-contains <> "..") path))
             (file-exists? file-name)
             (not (directory? file-name)))
        (list `((content-type . ,(assoc-ref file-mime-types
                                            (file-extension file-name))))
              (call-with-input-file file-name get-bytevector-all))
        (not-found (build-uri 'http
                              #:host (assoc-ref %config 'host)
                              #:port (assoc-ref %config 'port)
                              #:path (string-join path "/" 'prefix))))))
--8<---------------cut here---------------end--------------->8---

There’s probably a better way and a fail-safe way to do this, but this
has worked fine for me.

> And how do you use sendfile directly? (Code example for this would also
> be great to see.)

I’ve got no example for that in my projects, but “guix publish” provides
an implementation of “http-write” that uses sendfile.  See
guix/scripts/publish.scm in the Guix source tree.

> Is it still necessary to sanitize the requested file names, when an HTTP
> server handles requests for assets before the Guile application is hit?

I’d still do that because I’d want the application to never do the wrong
thing, even when the deployment details change.

-- 
Ricardo