Questions about the (web client) module.

Questions about the (web client) module.

[Roel Janssen]

Dear Guilers,

When I use http-post, and I want to change the HTTP header called
"Content-Type", I seem to need to spell it as "content-type" in the
#:headers part of the 'http-post' section.  Other headers like "Accept"
do not seem to follow the same lowercase style route.

More confusingly, using something like:
  #:headers '((Content-Type . "text/csv"))

leads to outputting the "Content-Type" header twice.
Why is "content-type" special?


Then my next question is about "multipart/form-data" content types.
My code looks like this:

  #:headers `((content-type . ,(string-append
                                "multipart/form-data; boundary=" boundary))
              (Accept . "*/*"))

But that does not work:
  scheme@(guile-user)> 
  web/request.scm:184:10: In procedure build-request:
  web/request.scm:184:10: Bad request: Bad value for header content-type: "multipart/form-data; boundary=..."

This is, however, a valid Content-Type.

So, why does content-type need to be spelled out lowercased?
And why can't it handle "multipart/form-data"?

Kind regards,
Roel Janssen

Re: Questions about the (web client) module.

[Ludovic Courtès]

Hi Roel,

Roel Janssen <roel@gnu.org> skribis:

> When I use http-post, and I want to change the HTTP header called
> "Content-Type", I seem to need to spell it as "content-type" in the
> #:headers part of the 'http-post' section.  Other headers like "Accept"
> do not seem to follow the same lowercase style route.
>
> More confusingly, using something like:
>   #:headers '((Content-Type . "text/csv"))
>
> leads to outputting the "Content-Type" header twice.
> Why is "content-type" special?

‘sanitize-request’ in (web client) adds a ‘Content-Type’ header if there
is none.

As for lower-case, perhaps the ‘request’ procedure in (web client)
should automatically convert to lower-case, or perhaps we should simply
clarify the documentation here.

> Then my next question is about "multipart/form-data" content types.
> My code looks like this:
>
>   #:headers `((content-type . ,(string-append
>                                 "multipart/form-data; boundary=" boundary))
>               (Accept . "*/*"))
>
> But that does not work:
>   scheme@(guile-user)> 
>   web/request.scm:184:10: In procedure build-request:
>   web/request.scm:184:10: Bad request: Bad value for header content-type: "multipart/form-data; boundary=..."
>
> This is, however, a valid Content-Type.

What’s the value of ‘boundary’?  At first sight it looks good to me:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(web http)
scheme@(guile-user)> (valid-header? 'content-type (parse-header 'content-type "text/plain"))
$13 = #t
scheme@(guile-user)> (valid-header? 'content-type (parse-header 'content-type "multipart/form-data; boundary=2"))
$14 = #t
scheme@(guile-user)> (valid-header? 'content-type (parse-header 'content-type "multipart/form-data; boundary=sdfd"))
$15 = #t
scheme@(guile-user)> (parse-header 'content-type "multipart/form-data; boundary=sdfd")
$16 = (multipart/form-data (boundary . "sdfd"))
--8<---------------cut here---------------end--------------->8---

HTH!
Ludo’.

Re: Questions about the (web client) module.

[Roel Janssen]

Hi Ludo’,

Thanks for your response.

Ludovic Courtès writes:

> Hi Roel,
>
> Roel Janssen <roel@gnu.org> skribis:
>
>> When I use http-post, and I want to change the HTTP header called
>> "Content-Type", I seem to need to spell it as "content-type" in the
>> #:headers part of the 'http-post' section.  Other headers like "Accept"
>> do not seem to follow the same lowercase style route.
>>
>> More confusingly, using something like:
>>   #:headers '((Content-Type . "text/csv"))
>>
>> leads to outputting the "Content-Type" header twice.
>> Why is "content-type" special?
>
> ‘sanitize-request’ in (web client) adds a ‘Content-Type’ header if there
> is none.

But 'Content-Type' and 'content-type' is not the same.. Adding a
'Content-Type' header results in the request having two 'Content-Type'
headers.  So it doesn't add 'Content-Type' when it isn't there, it
converts 'content-type' to 'Content-Type', so it adds 'Content-Type'
when 'content-type' isn't there.  This seems wrong to me.

>
> As for lower-case, perhaps the ‘request’ procedure in (web client)
> should automatically convert to lower-case, or perhaps we should simply
> clarify the documentation here.

Or perhaps do not treat some HTTP headers different than others, and
just keep it 'Content-Type' (as used in HTTP) instead of 'content-type'.

>
>> Then my next question is about "multipart/form-data" content types.
>> My code looks like this:
>>
>>   #:headers `((content-type . ,(string-append
>>                                 "multipart/form-data; boundary=" boundary))
>>               (Accept . "*/*"))
>>
>> But that does not work:
>>   scheme@(guile-user)> 
>>   web/request.scm:184:10: In procedure build-request:
>>   web/request.scm:184:10: Bad request: Bad value for header content-type: "multipart/form-data; boundary=..."
>>
>> This is, however, a valid Content-Type.
>
> What’s the value of ‘boundary’?  At first sight it looks good to me:

The value I use is "guile-virtuoso-driver".  All of the below
functions work as expected with this value.

Could it be that something else is wrong?
Here's the entire code I use:

--8<---------------cut here---------------start------------->8---
(define* (sparql-query-post query
                       #:key
                       (host "localhost")
                       (port 8890)
                       (graph "")
                       (type "json"))
  (let ((base-uri (string-append "http://" host ":" (number->string port) ))
        (boundary "guile-virtuoso-driver"))
    (let ((body (string-append
                 (format #f
                  "--~a~%Content-Disposition: form-data; name=~s~%~%text/csv~%"
                  boundary "format")
                  (format #f
                  "--~a~%Content-Disposition: form-data; name=~s~%~%~a~%--~a~%"
                  boundary "query" query boundary))))
      (http-post
       (string-append base-uri "/sparql")
       #:body body
       #:headers `((content-length . ,(string-length body))
                   (content-type . "multipart/form-data; boundary=guile-virtuoso-driver")
                   (Accept . "*/*"))))))
  (define %query-get-all-triplets
  "SELECT * FROM <http://localhost:8890/packages>
WHERE
{
  ?s ?p ?o
}")

(sparql-query-post %query-get-all-triplets)
--8<---------------cut here---------------end--------------->8---

Which results in:

--8<---------------cut here---------------start------------->8---
web/request.scm:184:10: In procedure build-request:
web/request.scm:184:10: Bad request: Bad value for header content-type: "multipart/form-data; boundary=guile-virtuoso-driver"
--8<---------------cut here---------------end--------------->8---

Thanks!

Kind regards,
Roel Janssen

Re: Questions about the (web client) module.

[Mark H Weaver]

Roel Janssen <roel@gnu.org> writes:

>>> Then my next question is about "multipart/form-data" content types.
>>> My code looks like this:
>>>
>>>   #:headers `((content-type . ,(string-append
>>>                                 "multipart/form-data; boundary=" boundary))
>>>               (Accept . "*/*"))
>>>
>>> But that does not work:
>>>   scheme@(guile-user)> 
>>>   web/request.scm:184:10: In procedure build-request:
>>>   web/request.scm:184:10: Bad request: Bad value for header content-type: "multipart/form-data; boundary=..."
>>>
>>> This is, however, a valid Content-Type.
>>
>> What’s the value of ‘boundary’?  At first sight it looks good to me:
>
> The value I use is "guile-virtuoso-driver".  All of the below
> functions work as expected with this value.
>
> Could it be that something else is wrong?
> Here's the entire code I use:
>
> (define* (sparql-query-post query
>                        #:key
>                        (host "localhost")
>                        (port 8890)
>                        (graph "")
>                        (type "json"))
>   (let ((base-uri (string-append "http://" host ":" (number->string port) ))
>         (boundary "guile-virtuoso-driver"))
>     (let ((body (string-append
>                  (format #f
>                   "--~a~%Content-Disposition: form-data; name=~s~%~%text/csv~%"
>                   boundary "format")
>                   (format #f
>                   "--~a~%Content-Disposition: form-data; name=~s~%~%~a~%--~a~%"
>                   boundary "query" query boundary))))
>       (http-post
>        (string-append base-uri "/sparql")
>        #:body body
>        #:headers `((content-length . ,(string-length body))
>                    (content-type . "multipart/form-data; boundary=guile-virtuoso-driver")

The line above should be replaced with this:

  (content-type . (multipart/form-data (boundary . ,boundary)))

>                    (Accept . "*/*"))))))

and this should be:

  (accept . ((*/*)))

These are the "parsed" form of HTTP headers, where the header names are
lowercase symbols and the associated values are not usually strings but
rather a Scheme representation of the value.  See section 7.3.4 (HTTP
Headers) in the Guile manual for more information.

      Mark

Re: Questions about the (web client) module.

[Mark H Weaver]

Roel Janssen <roel@gnu.org> writes:

> Here's the entire code I use:
>
> (define* (sparql-query-post query
>                        #:key
>                        (host "localhost")
>                        (port 8890)
>                        (graph "")
>                        (type "json"))
>   (let ((base-uri (string-append "http://" host ":" (number->string port) ))
>         (boundary "guile-virtuoso-driver"))
>     (let ((body (string-append
>                  (format #f
>                   "--~a~%Content-Disposition: form-data; name=~s~%~%text/csv~%"
>                   boundary "format")
>                   (format #f
>                   "--~a~%Content-Disposition: form-data; name=~s~%~%~a~%--~a~%"
>                   boundary "query" query boundary))))
>       (http-post
>        (string-append base-uri "/sparql")
>        #:body body
>        #:headers `((content-length . ,(string-length body))

Another thing: you can omit the 'content-length' header here.  It will
be added automatically, and in fact the value you use here is incorrect
unless the body is ASCII-only, because it's supposed to be the length in
bytes, but here you are giving the length in characters.

      Mark