bug#14128: web/http.scm: bad-header date check (UTC?)

bug#14128: web/http.scm: bad-header date check (UTC?)

[Aleix Conchillo Flaqué]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Hi,

I was doing a test using gnutls and connecting to twitter api
services. Example attached.

I successfully passed the TLS/SSL part, but then I got this HTTP bad
header issue. I am running Guile 2.0.5 from Debian/unstable but
current stable-2.0 branch also suffers from the safe problem (I
think).

=============
184: 1 [read-header #<input-output: gnutls-session-port 2e94c30>]
 754: 0 [parse-asctime-date "Wed, 03 Apr 2013 07:29:49 UTC"]

web/http.scm:754:6: In procedure parse-asctime-date:
web/http.scm:754:6: Throw to key `bad-header' with args `(date "Wed,
03 Apr 2013 07:29:49 UTC")'.
=============

Basically, the problem is that web/http.scm only accepts a GMT suffix
and the twitter server sends UTC. I think the RFC 822 standard accepts
UT instead of UTC, so it would be wrong as well.

Really, not sure what's the right way to fix this considering that
even the twitter server does not send a correct day (as far as I
understood the RFC).

Thanks,

Aleix

[-- Attachment #2: twitter-user-timeline.scm --]
[-- Type: application/octet-stream, Size: 1210 bytes --]


(use-modules (gnutls)
             (ice-9 format)
             (srfi srfi-8)
             (rnrs bytevectors)
             (web uri)
             (web client))

(define (twitter-user-timeline url)
  (let* ((uri (string->uri url))
         (socket (open-socket-for-uri uri))
         (session (make-session connection-end/client)))

  (set-log-level! 0)

  (set-log-procedure!
   (lambda (level msg) (format #t "|<~d>| ~a" level msg)))

  ;; Use the file 8descriptor that underlies SOCKET.
  (set-session-transport-fd! session (fileno socket))

  ;; Use the default settings.
  (set-session-priorities! session "NORMAL")

  ;; Create anonymous credentials.
  (set-session-credentials! session
                            (make-anonymous-client-credentials))
  (set-session-credentials! session
                            (make-certificate-credentials))

  ;; Perform the TLS handshake with the server.
  (handshake session)

  (receive (response body)
      (http-get uri
                #:port (session-record-port session)
                #:keep-alive? #t)
    (pk (utf8->string body)))

  (bye session close-request/rdwr)))

(twitter-user-timeline "https://api.twitter.com/1.1/statuses/user_timeline.json")

bug#14128: web/http.scm: bad-header date check (UTC?)

[Daniel Hartwig]

On 3 April 2013 15:59, Aleix Conchillo Flaqué <aconchillo@gmail.com> wrote:
> Hi,
>
> I was doing a test using gnutls and connecting to twitter api
> services. Example attached.
>
> I successfully passed the TLS/SSL part, but then I got this HTTP bad
> header issue. I am running Guile 2.0.5 from Debian/unstable but
> current stable-2.0 branch also suffers from the safe problem (I
> think).
>
> =============
> 184: 1 [read-header #<input-output: gnutls-session-port 2e94c30>]
>  754: 0 [parse-asctime-date "Wed, 03 Apr 2013 07:29:49 UTC"]
>
> web/http.scm:754:6: In procedure parse-asctime-date:
> web/http.scm:754:6: Throw to key `bad-header' with args `(date "Wed,
> 03 Apr 2013 07:29:49 UTC")'.
> =============
>
> Basically, the problem is that web/http.scm only accepts a GMT suffix
> and the twitter server sends UTC. I think the RFC 822 standard accepts
> UT instead of UTC, so it would be wrong as well.

RFC 2616 _requires_ http date values have a suffix of " GMT", other
time zone values are not valid.  RFC 822 has no relevance.  Some times
you see these “API” or “web services” breaking HTTP spec. in various
ways and claiming “well we are not HTTP just very-HTTP-like” but that
is not useful for people trying to interact.

Turns out there are a few broken http servers around.  2616 recommends
to convert non-GMT date values to GMT where such conversion is
unambiguous, and in Guile we have done this a bit.  Apparently we are
supposed to do this a bit more and accomodate yet another
non-compliant service?

Erk.  What is the point of defining protocols and standards then?

bug#14128: web/http.scm: bad-header date check (UTC?)

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

() Daniel Hartwig <mandyke@gmail.com>
() Wed, 3 Apr 2013 17:47:01 +0800

   Apparently we are supposed to do this a bit more and accomodate yet
   another non-compliant service?

Maybe that stuff should be exposed to the user.  Do a best effort
conversion and if not successful, return a pair ‘(raw-string . STRING)’
or whatever -- it's enough that it has a different type and that the
type is documented.  This way, you avoid carrying forward lots of cruft.
Do it now before it's too late!

   Erk.  What is the point of defining protocols and standards then?

One purpose is to challenge programmers to learn how and when to say "no".

-- 
Thien-Thi Nguyen
GPG key: 4C807502

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

bug#14128: web/http.scm: bad-header date check (UTC?)

[Daniel Hartwig]

On 3 April 2013 18:33, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> () Daniel Hartwig <mandyke@gmail.com>
> () Wed, 3 Apr 2013 17:47:01 +0800
>
>    Apparently we are supposed to do this a bit more and accomodate yet
>    another non-compliant service?
>
> Maybe that stuff should be exposed to the user.  Do a best effort
> conversion and if not successful, return a pair ‘(raw-string . STRING)’
> or whatever -- it's enough that it has a different type and that the
> type is documented. This way, you avoid carrying forward lots of cruft.
> Do it now before it's too late!

Interesting.  Though this does gradually erode the type barrier
erected by the web module.  I am reluctant to cede this territory.
Instead of this cruft accumulating in a few places (the web modules),
it becomes gratutiously spread around and duplicated in other
programs.

It was previously suggested to implement a permissive flag that, while
not passing unparsed data to the users, will at least not raise errors
and stop.

>
>
>    Erk.  What is the point of defining protocols and standards then?
>
> One purpose is to challenge programmers to learn how and when to say "no".
>

:-)

bug#14128: web/http.scm: bad-header date check (UTC?)

[Ludovic Courtès]

Daniel Hartwig <mandyke@gmail.com> skribis:

> RFC 2616 _requires_ http date values have a suffix of " GMT"

What about adding an exception for “UTC”?  :-)

It’s the same timezone, only with a different name, so it shouldn’t cost
us much.

WDYT?

Ludo’.

bug#14128: web/http.scm: bad-header date check (UTC?)

[Aleix Conchillo Flaqué]

On Wed, Apr 3, 2013 at 5:34 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> Daniel Hartwig <mandyke@gmail.com> skribis:
>
>> RFC 2616 _requires_ http date values have a suffix of " GMT"
>
> What about adding an exception for “UTC”?  :-)
>
> It’s the same timezone, only with a different name, so it shouldn’t cost
> us much.
>
> WDYT?
>

This is what I did locally so I could continue testing. But
unfortunately, it's is very likely that there are more broken servers
around. The permissive flag seems like an "all-or-nothing", so I think
Thien-Thi's suggestion sounds pretty reasonable: for all headers
checks, give a default but let user specify their owns (if I
understood that correctly). In my case I would just provide a #t for
the date check.

Best,

Aleix

bug#14128: web/http.scm: bad-header date check (UTC?)

[Aleix Conchillo Flaqué]

On Wed, Apr 3, 2013 at 7:57 AM, Aleix Conchillo Flaqué
<aconchillo@gmail.com> wrote:
>
> This is what I did locally so I could continue testing. But
> unfortunately, it's is very likely that there are more broken servers
> around. The permissive flag seems like an "all-or-nothing", so I think
> Thien-Thi's suggestion sounds pretty reasonable: for all headers
> checks, give a default but let user specify their owns (if I
> understood that correctly). In my case I would just provide a #t for
> the date check.
>

I have to add, that it might be also true that you end up with all
checks returning true, to make every server happy.

So, all in all, I am a bit confused, may be the permissive flag is not
that bad. The user could still read the headers after that, right? And
decide whether they are good or bad.

Aleix

bug#14128: web/http.scm: bad-header date check (UTC?)

[Andy Wingo]

On Wed 03 Apr 2013 09:59, Aleix Conchillo Flaqué <aconchillo@gmail.com> writes:

> =============
> 184: 1 [read-header #<input-output: gnutls-session-port 2e94c30>]
>  754: 0 [parse-asctime-date "Wed, 03 Apr 2013 07:29:49 UTC"]
>
> web/http.scm:754:6: In procedure parse-asctime-date:
> web/http.scm:754:6: Throw to key `bad-header' with args `(date "Wed,
> 03 Apr 2013 07:29:49 UTC")'.
> =============

Just reported this:
https://twitter.com/andywingo/status/319916865642168320

Let's see if they fix it before doing anything on our side.

Andy
-- 
http://wingolog.org/

bug#14128: web/http.scm: bad-header date check (UTC?)

[Daniel Hartwig]

On 3 April 2013 20:32, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> () Daniel Hartwig <mandyke@gmail.com>
> () Wed, 3 Apr 2013 18:47:22 +0800
>
>    Interesting.  Though this does gradually erode the type barrier
>    erected by the web module.  I am reluctant to cede this territory.
>
> I suppose playing whack-a-mole is fun sometimes, for a while.
>
>    Instead of this cruft accumulating in a few places (the web modules),
>    it becomes gratutiously spread around and duplicated in other
>    programs.
>
> Well more precisely, in those programs which talk to such servers, and
> actually care about that particular data.  (Many programs do not.)
>
>    It was previously suggested to implement a permissive flag that,
>    while not passing unparsed data to the users, will at least not raise
>    errors and stop.
>
> In this direction, you might as well go higher-order.  Client code
> specifies the conversion procedure, and there is a reasonable default
> (also exported, so that the user can compose it w/ their own bad-input
> detection/handling).  This is more Schemely, in my mind.

That is ‘declare-header!’.

bug#14128: web/http.scm: bad-header date check (UTC?)

[Mark H Weaver]

Andy Wingo agreed on IRC to accepting UTC, since we've since discovered
that not only various web servers, but also some transparent web proxies
put UTC in the date header, which was causing problems for many people.

So, I've pushed a change to stable-2.0 to accept UTC.

http://git.savannah.gnu.org/gitweb/?p=guile.git;a=commit;h=8904b7a9362ede0de324e55701d8ea1d018f8738

I'm closing this bug now.  Thanks for the report!

      Mark