* Guile web server example serving static files @ 2020-09-17 21:45 Zelphir Kaltstahl 2020-09-18 5:56 ` divoplade 2020-09-18 7:47 ` Dr. Arne Babenhauserheide 0 siblings, 2 replies; 10+ messages in thread From: Zelphir Kaltstahl @ 2020-09-17 21:45 UTC (permalink / raw) To: Guile User Hello Guile Users! I finally managed to create an example for using Guile's web server and serving static files. A rather silly bug kept me for a few days from making progress, but finally today I fixed it. I tried to implement some security checks about the path of the requested static assets. If anyone wants to look at it and point out issues with it, I will try to fix it, or you could make a pull request. If there are any other issues, it would also be great to know them : ) Here is the code in my repository: https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets I tried to comment most stuff, so that the code can be understood more easily. And here is a pointer to the path security stuff: https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets/web-path-handling.scm#L50 (Perhaps I overlooked something fundamental with regard to safe path handling?) So I guess I can soon start developing another example, perhaps already a blog, which makes use of all the code : ) I have not yet made use of Guile's sendfile procedure and keyboard interrupt is not yet caught. Regards, Zelphir -- repositories: https://notabug.org/ZelphirKaltstahl ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-17 21:45 Guile web server example serving static files Zelphir Kaltstahl @ 2020-09-18 5:56 ` divoplade 2020-09-19 10:39 ` Zelphir Kaltstahl 2020-09-20 7:48 ` tomas 2020-09-18 7:47 ` Dr. Arne Babenhauserheide 1 sibling, 2 replies; 10+ messages in thread From: divoplade @ 2020-09-18 5:56 UTC (permalink / raw) To: Zelphir Kaltstahl, Guile User Hello Zelphir, Le jeudi 17 septembre 2020 à 23:45 +0200, Zelphir Kaltstahl a écrit : > I finally managed to create an example for using Guile's web server > and > serving static files. A rather silly bug kept me for a few days from > making progress, but finally today I fixed it. > > I tried to implement some security checks about the path of the > requested static assets. If anyone wants to look at it and point out > issues with it, I will try to fix it, or you could make a pull > request. > If there are any other issues, it would also be great to know them : > ) > > Here is the code in my repository: > > https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets > > I tried to comment most stuff, so that the code can be understood > more > easily. > > And here is a pointer to the path security stuff: > > https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets/web-path-handling.scm#L50 As for why guile avoid reasoning about "paths", see https://www.gnu.org/prep/standards/standards.html#GNU-Manuals https://www.gnu.org/prep/standards/standards.html#GNU-Manuals: Please do not use the term “pathname” that is used in Unix documentation; use “file name” (two words) instead. We use the term “path” only for search paths, which are lists of directory names. Also, your functions "absolute-path" and "complex-path?" in path- handling.scm https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets/path-handling.scm do not seem to me that they would work correctly when passed something starting with "../" (as opposed to containing "/../"). I think that with a little bit of work you could accept "../" in arguments and tweak path-join to go up (by discarding anything in path1 after the last '/' and go to the next part, if there is something to discard). Also I am not sure how it would remove inclusions of '/./' or leading './' in the name. The URI RFC (https://tools.ietf.org/html/rfc3986#section-5) describes an algorithm in section 5.2. Relative resolution that does the canonization of an URI relative to an absolute URI (you just need to ignore the scheme, authority, query and fragment parts and focus on the path). This is similar to canonicalization of file names, except for the \\ difficulty. In particular, see 5.2.4, Remove dot segments. Also, you should refrain from checking if a file exists, because it could be deleted between your call to file-exists? and when you actually open the file. Thus, passing the file-exists? test will not guarantee that the file will exist when you want to use it, and even less that you will be able to open it and read it. Finally, you don't need to check if a file name is "safe" at all. the file procedures do not interpret or substitute variables or ~ or `` (try it: change directory to /tmp and write to files named ~root, `pwd`, $PATH, '*', ... just be aware that you will have a hard time deleting them from bash!), and there is nothing special with files named as a series of dots. That's good, otherwise you would also need to check for '%' in mingw and whatever stuff microsoft invented to change the file name experience. Best regards, divoplade ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-18 5:56 ` divoplade @ 2020-09-19 10:39 ` Zelphir Kaltstahl 2020-09-20 7:48 ` tomas 1 sibling, 0 replies; 10+ messages in thread From: Zelphir Kaltstahl @ 2020-09-19 10:39 UTC (permalink / raw) To: divoplade; +Cc: Guile User Hello divoplade, On 18.09.20 07:56, divoplade wrote: > Hello Zelphir, > > Le jeudi 17 septembre 2020 à 23:45 +0200, Zelphir Kaltstahl a écrit : >> I finally managed to create an example for using Guile's web server >> and >> serving static files. A rather silly bug kept me for a few days from >> making progress, but finally today I fixed it. >> >> I tried to implement some security checks about the path of the >> requested static assets. If anyone wants to look at it and point out >> issues with it, I will try to fix it, or you could make a pull >> request. >> If there are any other issues, it would also be great to know them : >> ) >> >> Here is the code in my repository: >> >> https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets >> >> I tried to comment most stuff, so that the code can be understood >> more >> easily. >> >> And here is a pointer to the path security stuff: >> >> https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets/web-path-handling.scm#L50 > As for why guile avoid reasoning about "paths", see > https://www.gnu.org/prep/standards/standards.html#GNU-Manuals > https://www.gnu.org/prep/standards/standards.html#GNU-Manuals: > > Please do not use the term “pathname” that is used in Unix > documentation; use “file name” (two words) instead. We use the term > “path” only for search paths, which are lists of directory names. I see! I was not aware of any of that. That explains the naming. Perhaps I need to think of another name than "path" or "file". It seems still weird to me, to call everything a "file" when it is actually also for directories, symlinks and whatever else there is. Is there any common idea for an alternative name? But perhaps on the lower level those are really the same? I have no idea how a typical file system or OS handles directories. > Also, your functions "absolute-path" and "complex-path?" in path- > handling.scm > https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets/path-handling.scm > > do not seem to me that they would work correctly when passed something > starting with "../" (as opposed to containing "/../"). I think that is a remnant from thinking, that anything that contains anything with special meaning like ".." should be considered "complex" and refusing to handle it. That is probably, why I did not think about it. Perhaps I overlooked this case. I need to check my code. I'll add test cases for those. > I think that > with a little bit of work you could accept "../" in arguments and tweak > path-join to go up (by discarding anything in path1 after the last '/' > and go to the next part, if there is something to discard). When a "../" is encountered, it should go one level up, so that means discarding one part (one "thing between the separators", have no better name for it right now), so I would have to not use fold or do something special inside fold to modify the already accumulated part, I guess. I think you are correct in that this function is in the path-handling module and should work properly also for special cases like "../" parts. It is an abstraction layer below the web-path-handling module. I should probably completely separate it out into its own example directory and make it more comprehensive, adding things whenever I realize something is missing. > Also I am not sure how it would remove inclusions of '/./' or leading > './' in the name. I think for that case such "./" could be skipped, as they do not change the meaning or result. If a part begins with a "/" it causes the previously accumulated to be discarded, as it is itself an absolute path. I think this is the behavior of the Python 3 function and what I took as an example of what the results of my own function should be. I should add test cases for this and then fix it. > The URI RFC (https://tools.ietf.org/html/rfc3986#section-5) describes > an algorithm in section 5.2. Relative resolution that does the > canonization of an URI relative to an absolute URI (you just need to > ignore the scheme, authority, query and fragment parts and focus on the > path). This is similar to canonicalization of file names, except for > the \\ difficulty. In particular, see 5.2.4, Remove dot segments. That is a lot to read. Seems like the whole problem calls for an extra path handling library. I did not look in detail at the code in the RFC yet, but hopefully it will be readable, understandable and not too difficult to implement, so that I can create that library and call it "RFC conform" or something. Seems like I opened a can of worms with the whole path stuff. It's always those little things that explode ... > Also, you should refrain from checking if a file exists, because it > could be deleted between your call to file-exists? and when you > actually open the file. Thus, passing the file-exists? test will not > guarantee that the file will exist when you want to use it, and even > less that you will be able to open it and read it. That makes sense. Thanks for pointing it out. I shostringuld rather be handling the exception when/if the file happens to be not existing when I try to read it. > Finally, you don't need to check if a file name is "safe" at all. the > file procedures do not interpret or substitute variables or ~ or `` > (try it: change directory to /tmp and write to files named ~root, > `pwd`, $PATH, '*', ... just be aware that you will have a hard time > deleting them from bash!), and there is nothing special with files > named as a series of dots. That's good, otherwise you would also need > to check for '%' in mingw and whatever stuff microsoft invented to > change the file name experience. Ha! I simply assumed I would need to do this. Good to know! > Best regards, > > divoplade All good input, thank you! Regards, Zelphir -- repositories: https://notabug.org/ZelphirKaltstahl ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-18 5:56 ` divoplade 2020-09-19 10:39 ` Zelphir Kaltstahl @ 2020-09-20 7:48 ` tomas 2020-09-20 7:52 ` divoplade 1 sibling, 1 reply; 10+ messages in thread From: tomas @ 2020-09-20 7:48 UTC (permalink / raw) To: divoplade; +Cc: Guile User [-- Attachment #1: Type: text/plain, Size: 611 bytes --] On Fri, Sep 18, 2020 at 07:56:04AM +0200, divoplade wrote: > Hello Zelphir, [...] > As for why guile avoid reasoning about "paths", see > https://www.gnu.org/prep/standards/standards.html#GNU-Manuals > https://www.gnu.org/prep/standards/standards.html#GNU-Manuals: > > Please do not use the term “pathname” [...] For completeness's sake, one should add that `path' is customary in UNIX in the sense Zelphir used it (that's even mentioned in the references you gave). Something to keep in mind when talking to "other cultures" -- they are our neighbours, after all :-) Cheers - t [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-20 7:48 ` tomas @ 2020-09-20 7:52 ` divoplade 2020-09-20 8:29 ` tomas 0 siblings, 1 reply; 10+ messages in thread From: divoplade @ 2020-09-20 7:52 UTC (permalink / raw) To: tomas; +Cc: Guile User Le dimanche 20 septembre 2020 à 09:48 +0200, tomas@tuxteam.de a écrit : > Something to keep in mind when talking to "other cultures" -- > they are our neighbours, after all :-) I did not recommend using this term. There was a question in the source code asking why guile used a different terminology; I put the explanation why, with the relevant citation (because the section is huge). That's all. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-20 7:52 ` divoplade @ 2020-09-20 8:29 ` tomas 2020-09-20 8:54 ` divoplade 0 siblings, 1 reply; 10+ messages in thread From: tomas @ 2020-09-20 8:29 UTC (permalink / raw) To: divoplade; +Cc: Guile User [-- Attachment #1: Type: text/plain, Size: 841 bytes --] On Sun, Sep 20, 2020 at 09:52:08AM +0200, divoplade wrote: > Le dimanche 20 septembre 2020 à 09:48 +0200, tomas@tuxteam.de a écrit : > > Something to keep in mind when talking to "other cultures" -- > > they are our neighbours, after all :-) > > I did not recommend using this term. There was a question in the source > code asking why guile used a different terminology; I put the > explanation why, with the relevant citation (because the section is > huge). That's all. Don't get me wrong: I think your hint is relevant and appropriate. It makes sense to keep a consistent terminology whithin one context. I just wanted to complement a bit: we should keep in mind that others use a slightly different terminology (so a little clarification here and there won't hurt). Building bridges :-) Cheers & thanks - t [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-20 8:29 ` tomas @ 2020-09-20 8:54 ` divoplade 2020-09-20 9:07 ` tomas 0 siblings, 1 reply; 10+ messages in thread From: divoplade @ 2020-09-20 8:54 UTC (permalink / raw) To: tomas; +Cc: Guile User Le dimanche 20 septembre 2020 à 10:29 +0200, tomas@tuxteam.de a écrit : > > I did not recommend using this term. There was a question in the > source > > code asking why guile used a different terminology; I put the > > explanation why, with the relevant citation (because the section is > > huge). That's all. > > Don't get me wrong: I think your hint is relevant and appropriate. It > makes sense to keep a consistent terminology whithin one context. Then I don't understand what hint you are discussing. > we should keep in mind that others use a slightly different > terminology Yes, that's why I try to clarify where the strange "file name" terminology comes from. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-20 8:54 ` divoplade @ 2020-09-20 9:07 ` tomas 0 siblings, 0 replies; 10+ messages in thread From: tomas @ 2020-09-20 9:07 UTC (permalink / raw) To: divoplade; +Cc: Guile User [-- Attachment #1: Type: text/plain, Size: 214 bytes --] On Sun, Sep 20, 2020 at 10:54:50AM +0200, divoplade wrote: [...] > Yes, that's why I try to clarify where the strange "file name" > terminology comes from. We are in violent agreement, it seems :-) thanks - t [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-17 21:45 Guile web server example serving static files Zelphir Kaltstahl 2020-09-18 5:56 ` divoplade @ 2020-09-18 7:47 ` Dr. Arne Babenhauserheide 2020-09-19 10:57 ` Zelphir Kaltstahl 1 sibling, 1 reply; 10+ messages in thread From: Dr. Arne Babenhauserheide @ 2020-09-18 7:47 UTC (permalink / raw) To: Zelphir Kaltstahl; +Cc: guile-user [-- Attachment #1: Type: text/plain, Size: 744 bytes --] Hi Zelphir, I see that you’re getting the full file as bytevector Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> writes: > https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets I see that you’re getting the full file as bytevector in response-utils. When I tested this in wispserve with multi-gigabyte files, I got gc-errors. I do string-conversion, though, because I got into problems with just putting a bytevector — does avoiding that avoid the problem? https://hg.sr.ht/~arnebab/wispserve/browse/wispserve/serve.w?rev=4541f36df0b0#L411 Best wishes, Arne -- Unpolitisch sein heißt politisch sein ohne es zu merken [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Guile web server example serving static files 2020-09-18 7:47 ` Dr. Arne Babenhauserheide @ 2020-09-19 10:57 ` Zelphir Kaltstahl 0 siblings, 0 replies; 10+ messages in thread From: Zelphir Kaltstahl @ 2020-09-19 10:57 UTC (permalink / raw) To: Dr. Arne Babenhauserheide; +Cc: guile-user Hello Arne! Thanks for the link. So far I have only tried with a tiny CSS file, so I have no idea, whether I would run into a problem, besides memory consumption, for big files. I wonder, whether I should simply use sendfile described in https://www.gnu.org/software/guile/manual/html_node/File-System.html#File-System. I think it was mentioned to be in GUIX somewhere, as an example. So far I did not get to look at it. Regard, Zelphir On 18.09.20 09:47, Dr. Arne Babenhauserheide wrote: > Hi Zelphir, > > I see that you’re getting the full file as bytevector > > Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> writes: >> https://notabug.org/ZelphirKaltstahl/guile-examples/src/65ba7cead2983f1ceb8aa2d4eedfe37734e5ca56/web-development/example-03-serve-static-assets > I see that you’re getting the full file as bytevector in response-utils. > When I tested this in wispserve with multi-gigabyte files, I got > gc-errors. > > I do string-conversion, though, because I got into problems with just > putting a bytevector — does avoiding that avoid the problem? > > https://hg.sr.ht/~arnebab/wispserve/browse/wispserve/serve.w?rev=4541f36df0b0#L411 > > Best wishes, > Arne -- repositories: https://notabug.org/ZelphirKaltstahl ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-09-20 9:07 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-09-17 21:45 Guile web server example serving static files Zelphir Kaltstahl 2020-09-18 5:56 ` divoplade 2020-09-19 10:39 ` Zelphir Kaltstahl 2020-09-20 7:48 ` tomas 2020-09-20 7:52 ` divoplade 2020-09-20 8:29 ` tomas 2020-09-20 8:54 ` divoplade 2020-09-20 9:07 ` tomas 2020-09-18 7:47 ` Dr. Arne Babenhauserheide 2020-09-19 10:57 ` Zelphir Kaltstahl
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).