Fixing "stringly typed" data structures in Artanis (Was: Learning Guile web. Stuck on returning an image.)

Fixing "stringly typed" data structures in Artanis (Was: Learning Guile web. Stuck on returning an image.)

[Thompson, David]

[ Changing the subject for this little rant below ]

On Thu, Dec 10, 2015 at 2:44 PM, Martyn Smith
<martyn.developer@googlemail.com> wrote:

[snip]

> (get "/image"
>      (lambda (rc)
>        (let* ((port (open-file "s.jpg" "r"))
>                (bytes (get-bytevector-all port)))
>      (close-port port)
>      (response-emit bytes #:headers '((content-type image/jpg))))))
>
> (run #:port 1234)
>
>
> For my personal website, I pass in an id (ie "/image/{id}") which returns a
> record from the database, containing the location of the file in question...
> plus other things (tags, uploader, etc)  -- once working I stick it inside
> an image tag - job done!

I guess this is as good a time as any to voice a concern that I have
with Artanis before too many people depend on it and its not feasible
to change it.  In Scheme, we have the luxury of being able to easily
create embedded domain specific languages or otherwise take advantage
of s-expressions to avoid the tiring work of writing parsers and
interpreters for new languages.  However, Artanis features a URI
routing language written in strings, like "/user/:id".  This
particular string means: Match a URI beginning with "/user/", followed
by any string and associate it with the name "id".  Generally
speaking, this is a pattern matcher.  Unfortunately, unlike Guile's
built-in pattern matcher, these patterns must be string encoded and
cannot do more sophisticated matching techniques such as making sure
"id" is a string representation of an integer.  Doing this type of
string-based URI matching is the status quo in many web frameworks
written in Ruby, Python, etc., but in Scheme we have the opportunity
to do *much* better.  For an example that uses more proper Guile
Scheme idioms, take a look the source for the 'guix publish' tool in
GNU Guix. [0]  By viewing a URI as a list strings, we can represent
"/user/1" as the list ("image" "1").  From there, it's easy to use a
pattern matcher to match this route:

    (match uri (("user" id) (display-user-info id)))

From there we can get more advanced and assert various things about
"id" in the match expression, as 'guix publish' does with its routes.

There are also security issues to think about when you use "stringly
typed" data.  It doesn't apply to this particular case, but working
with strings that are actually not strings (like HTML or SQL) opens
the door for injection attacks. [1] Fortunately, we can avoid such
issues entirely by choosing more appropriate data types.

I hope this has made some sense.  I think Artanis is a great project,
and I hope issues like this can be fixed so web developers looking at
Guile can truly see how much better it is in Lisp land.

- Dave

[0] http://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.scm#n309
[1] http://www.more-magic.net/posts/structurally-fixing-injection-bugs.html

Re: Fixing "stringly typed" data structures in Artanis

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On Thu, Dec 10, 2015 at 17:02:52 -0500, Thompson, David wrote:
>     (match uri (("user" id) (display-user-info id)))

I agree with this.  I understand where Artanis has derived its
conventions from, (e.g. Sinatra/Ruby; Express/Node.js; Symfony/PHP;
Spark/Java; and countless others), and how useful those conventions are
in those languages, but I agree that we have better options here.

> I think Artanis is a great project,

Certainly. :)  I haven't had the chance to put it to any practical use
yet, but I fully intend to!

> [1] http://www.more-magic.net/posts/structurally-fixing-injection-bugs.html

Yes, it's silly for Schemers to have to worry about these issues.  Which
I make obnoxiously clear to my PHP co-workers on a frequent basis.

-- 
Mike Gerwitz
Free Software Hacker | GNU Maintainer
http://mikegerwitz.com
FSF Member #5804 | GPG Key ID: 0x8EE30EAB

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Fixing "stringly typed" data structures in Artanis (Was: Learning Guile web. Stuck on returning an image.)

[Nala Ginrut]

Actually, all of these issues (better rule parsing, anti
SQL-injection...) have been concerning when developing Artanis.
And I've prepared something for them in Artanis, but all these new
solutions are still premature. It's why I didn't announce them.

1. For better rule parsing
In the beginning, I plan to mimic the way of mainstream web framework to
provide string rule parsing for URL-remapping. This would be easier for
newcomers to learn Artanis from other framework (Django/Rails...).

This string rule parsing implementation is complete and stable now.
So yes, it's time to think about a better solution now, which may show
some benefit when people come to Scheme.

Fortunately, Artanis is flexible enough to add new list parsing for
URL-remapping. So we may provide both in Artanis, say:
(get '(img s.jpg) ...)
or
(get '(img (: jpgfile)) ...)

Folks may dislike (: key) to indicate binding in URL, but it's trivial
here. Anyway, it's possible to provide list matching for URL-remapping
without refactoring Artanis.
(provide both string and list pattern is seen in Irregex, I think it's
cool)

The only problem is how to design a good list pattern for URL-mapping.
IMO, it's a new Domain-Specific-Language too, just like the current one.
But it's more natural and expressive in list.


2. DB security and efficiency, anti-SQL-injection

This is a big and hard topic, although one may achieve something in
limited situations.

Yes, there's possibility that we can detect/filter bad strings in URL
rules within list matching.
But that's bad solution.

DB operation should be decoupled from URL-remapping. So I'll be unlikely
to add such filter in list-URL pattern matching, although it's trivial
to add.

The solution Artanis provided is SQL-mapping, which is undocumented yet.
Because it's still premature.
It'll provide a mechanism for user to define their own filter to make
sure every fields/values is safe before passing it to Database. Besides,
there could be a syntax analyzer to check if the whole concatenated SQL
string is safe (yes it's hard, but it's the policy, here I'm just
talking about mechanism).

Beyond, as we know, Relational-Mapping is not silver bullet for any
cases. Personally, I've seen huge SQL code block in hundreds lines in
product environment. If your business need such complex query, it's
never Relation-Mapping can solve. People should write fancy SQL code
manually.
But how to interact with users input from client? And how to do proper
check/filter for SQL? How can such a mapping be used in users code
easily? That's what SQL-mapping trying to solve.

3. Conclusion

I plan to add list matching for URL-remapping in future release. Maybe
it's a good time to discuss how to design the pattern (actually it's a
new DSL) now?

The proper design is important for Artanis, since most of the URL routes
would be generated automatically if users use CLI tools to save their
time (maybe it's trivial since the generated rules unnecessary to be
list). Of course, it is never a problem for Scheme because of the
expressiveness. ;-) 


On Thu, 2015-12-10 at 17:02 -0500, Thompson, David wrote:
> I guess this is as good a time as any to voice a concern that I have
> with Artanis before too many people depend on it and its not feasible
> to change it.  In Scheme, we have the luxury of being able to easily
> create embedded domain specific languages or otherwise take advantage
> of s-expressions to avoid the tiring work of writing parsers and
> interpreters for new languages.  However, Artanis features a URI
> routing language written in strings, like "/user/:id".  This
> particular string means: Match a URI beginning with "/user/", followed
> by any string and associate it with the name "id".  Generally
> speaking, this is a pattern matcher.  Unfortunately, unlike Guile's
> built-in pattern matcher, these patterns must be string encoded and
> cannot do more sophisticated matching techniques such as making sure
> "id" is a string representation of an integer.  Doing this type of
> string-based URI matching is the status quo in many web frameworks
> written in Ruby, Python, etc., but in Scheme we have the opportunity
> to do *much* better.  For an example that uses more proper Guile
> Scheme idioms, take a look the source for the 'guix publish' tool in
> GNU Guix. [0]  By viewing a URI as a list strings, we can represent
> "/user/1" as the list ("image" "1").  From there, it's easy to use a
> pattern matcher to match this route:
> 
>     (match uri (("user" id) (display-user-info id)))
> 
> From there we can get more advanced and assert various things about
> "id" in the match expression, as 'guix publish' does with its routes.
> 
> There are also security issues to think about when you use "stringly
> typed" data.  It doesn't apply to this particular case, but working
> with strings that are actually not strings (like HTML or SQL) opens
> the door for injection attacks. [1] Fortunately, we can avoid such
> issues entirely by choosing more appropriate data types.
> 
> I hope this has made some sense.  I think Artanis is a great project,
> and I hope issues like this can be fixed so web developers looking at
> Guile can truly see how much better it is in Lisp land.
> 
> - Dave
> 
> [0] http://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.scm#n309
> [1] http://www.more-magic.net/posts/structurally-fixing-injection-bugs.html

Re: Fixing "stringly typed" data structures in Artanis (Was: Learning Guile web. Stuck on returning an image.)

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Dec 10, 2015 at 05:02:52PM -0500, Thompson, David wrote:
> [ Changing the subject for this little rant below ]
> 
> On Thu, Dec 10, 2015 at 2:44 PM, Martyn Smith
> <martyn.developer@googlemail.com> wrote:
> 
> [snip]
> 
> > (get "/image"
> >      (lambda (rc)
> >        (let* ((port (open-file "s.jpg" "r"))
> >                (bytes (get-bytevector-all port)))
> >      (close-port port)
> >      (response-emit bytes #:headers '((content-type image/jpg))))))
> >
> > (run #:port 1234)
> >
> >
> > For my personal website, I pass in an id (ie "/image/{id}") which returns a
> > record from the database, containing the location of the file in question...
> > plus other things (tags, uploader, etc)  -- once working I stick it inside
> > an image tag - job done!
> 
> I guess this is as good a time as any to voice a concern that I have
> with Artanis before too many people depend on it and its not feasible
> to change it.  In Scheme, we have the luxury of being able to easily
> create embedded domain specific languages or otherwise take advantage
> of s-expressions to avoid the tiring work of writing parsers and
> interpreters for new languages. [...]

I'd like to throw in a word of caution. There are those two worlds
"everything is a string" (excuse me the Tcl pun) and "everything is
a structure". They both have their strengths, and over-committing to
one can immobilize too early.

In Guix this may make more sense, since we're talking about a more or
less controlled environment. But in the WWW, the only currency of exchange
is an URL which is... a string! (it can be interpreted as a serialization
of an intrinsic data structure, but what the RFCs talk about is a
string).

For a current discusion of this kind of topic, see [1]

And yes, David: your concerns about (not only SQL) injection are
definitely valid; I guess this is a force field we'll be living in,
at least in this trade. Too much in one corner and it becomes
too dangerous, too much in the other and it becomes boring. Gödel
or something.

[1] <http://lambda-the-ultimate.org/node/5289>

regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlZqgO8ACgkQBcgs9XrR2kZTaQCeM42GhOsKjNJa8xVBGv2D3caS
Y88An0UVbd/zL7HaY3UDhmutbh7byILm
=xwJZ
-----END PGP SIGNATURE-----

Re: Fixing "stringly typed" data structures in Artanis (Was: Learning Guile web. Stuck on returning an image.)

[Christopher Allan Webber]

Speaking of expressiveness and investigating these things, I've found no
better series than the ones by Peter Bex:

  http://www.more-magic.net/posts/lispy-dsl-scss.html
  http://www.more-magic.net/posts/lispy-dsl-sxml.html
  http://www.more-magic.net/posts/lispy-dsl-sre.html
  http://www.more-magic.net/posts/lispy-dsl-ssql.html

Probably of interest!

Re: Fixing "stringly typed" data structures in Artanis (Was: Learning Guile web. Stuck on returning an image.)

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Dec 11, 2015 at 09:25:57AM -0600, Christopher Allan Webber wrote:
> Speaking of expressiveness and investigating these things, I've found no
> better series than the ones by Peter Bex:
> 
>   http://www.more-magic.net/posts/lispy-dsl-scss.html
>   http://www.more-magic.net/posts/lispy-dsl-sxml.html
>   http://www.more-magic.net/posts/lispy-dsl-sre.html
>   http://www.more-magic.net/posts/lispy-dsl-ssql.html

Thanks for the nice reading!
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlZrIPUACgkQBcgs9XrR2kYRtQCdF2VbeuSZ1pIDt0EvAlKiIHk0
ziQAnR33gnag4Gmv8FGvg7F3OF0CA5UL
=3TBr
-----END PGP SIGNATURE-----

Re: Fixing "stringly typed" data structures in Artanis

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> Doing this type of
> string-based URI matching is the status quo in many web frameworks
> written in Ruby, Python, etc., but in Scheme we have the opportunity
> to do *much* better.

+1

See also Andy’s take on that in the manual (info "(guile) Types and the
Web").

Ludo’.

Re: Fixing "stringly typed" data structures in Artanis

[Amirouche Boubekki]

Le 2015-12-10 23:02, Thompson, David a écrit :
> [ Changing the subject for this little rant below ]
> 
> I guess this is as good a time as any to voice a concern that I have
> with Artanis before too many people depend on it and its not feasible
> to change it.

Artanis is alpha nobody expects to code against it without having to 
change things
here and there.

> In Scheme, we have the luxury of being able to easily
> create embedded domain specific languages or otherwise take advantage
> of s-expressions to avoid the tiring work of writing parsers and
> interpreters for new languages.

tldr: Artanis should use a sexp instead of a string to declare URL 
patterns.

> For an example that uses more proper Guile
> Scheme idioms, take a look the source for the 'guix publish' tool in
> GNU Guix. [0]  By viewing a URI as a list strings, we can represent
> "/user/1" as the list ("image" "1").  From there, it's easy to use a
> pattern matcher to match this route:
> 
>     (match uri (("user" id) (display-user-info id)))
> 
> From there we can get more advanced and assert various things about
> "id" in the match expression, as 'guix publish' does with its routes.

It's more proper Guile (Scheme?) idiom but it's not enough. In 
particular
the code of guix publish doesn't handle various cases like

   '/article/1337-diy-a-web-framework'

where the split is not only done around slash. I'm not saying it's not
possible but the solution is not that simple.

AFAIK working with Artanis is still a win compared to getting guix 
publish
code working.

> [...]

> 
> I hope this has made some sense.  I think Artanis is a great project,
> and I hope issues like this can be fixed [...]
> 

The rant falls short I guess.

This is a rather scary mail which would better fit the bug report 
format.

Re: Fixing "stringly typed" data structures in Artanis

[Thompson, David]

On Sun, Dec 13, 2015 at 2:41 PM, Amirouche Boubekki
<amirouche@hypermove.net> wrote:
> Le 2015-12-10 23:02, Thompson, David a écrit :
>>
>> [ Changing the subject for this little rant below ]
>>
>> I guess this is as good a time as any to voice a concern that I have
>> with Artanis before too many people depend on it and its not feasible
>> to change it.
>
> Artanis is alpha nobody expects to code against it without having to change
> things
> here and there.

Sure, which is why I want to get these issues taken care of before
things are stable.

>> For an example that uses more proper Guile
>> Scheme idioms, take a look the source for the 'guix publish' tool in
>> GNU Guix. [0]  By viewing a URI as a list strings, we can represent
>> "/user/1" as the list ("image" "1").  From there, it's easy to use a
>> pattern matcher to match this route:
>>
>>     (match uri (("user" id) (display-user-info id)))
>>
>> From there we can get more advanced and assert various things about
>> "id" in the match expression, as 'guix publish' does with its routes.
>
> It's more proper Guile (Scheme?) idiom but it's not enough. In particular
> the code of guix publish doesn't handle various cases like
>
>   '/article/1337-diy-a-web-framework'
>
> where the split is not only done around slash. I'm not saying it's not
> possible but the solution is not that simple.

The point is that there's a language that we can express with Scheme
code, not strings, that is suitable for the domain of URI routing.  It
need not be simply using (ice-9 match), but generally speaking a
pattern matcher is what we want here.  That said, it's not true that
the above URI wouldn't work with a simple pattern matching strategy.
'guix publish' actually already has an example of this:

              ;; /<hash>.narinfo
              (((= extract-narinfo-hash (? string? hash)))
               (render-narinfo store request hash))

Where 'extract-narinfo-hash' matches a string like
"0001mfr72xdjw284dm1dw067zzylf2p0.narinfo" and extracts just the hash,
or returns #f  if the string doesn't match in which case the route
match fails.

> This is a rather scary mail which would better fit the bug report format.

Scary?  Definitely not the intent.

- Dave