bug#39800: gnutls guile bug receiving https data

bug#39800: gnutls guile bug receiving https data

[franco.rcr]

Hello,
I installed gnutls for guile and checked the gnutls module with this 
simple code:


;;Guile version 3.0 and gnutls  from git

,show version
GNU Guile 3.0.0.15-ff14b7

(gnutls-version)
$6 = "3.6.12"

;;Now, submitting this simple https request, you get an exception
(http-request "https://www.google.com")
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `gnutls-error' with args `(#<gnutls-error-enum La 
connessione TLS non è stata terminata in modo corretto.> 
read_from_session_record_port)'.

;;instead, without https there are no errors
(http-request "http://www.google.com") ;;works fine.


The error happens only on https://www.google.com and does not throw with 
a lot of other https web sites.
Furthermore the error is not throw if the method is HEAD, so it is 
related to the data part of the https answer.
I tried to enter in the internal implementation of the http web client 
but after some tests I decided to do some simple tests at application level.
I rewrote the get-bytevector-all, with a loop that reads one byte per 
time and the error was thrown anyway.
I catched the error and I've got the complete answer from the google web 
server.
In the following there is my applicative solution, where I rewrote the 
get-bytevector-all by adding the error checking and specifiyng to 
http-request that the data has to be returned as a port (#:streaming? #t).


;;A macro to catch errors
(define-syntax my-noerr
   (syntax-rules ()
     ((_ __error-return exp ...)
      (let
          ((__st #f))
        (catch #t
          (lambda() exp ...)
          (lambda (k . p) __error-return))))))

;;The rewriting of get-bytevector-all
(defun get-bytevector-all (port)
   (u8-list->bytevector (let loop ((port port))
      (let ((v (my-noerr (eof-object) (get-u8 port))))
        (if (eof-object? v)
        #nil
        (cons v (loop port)))))))

;;the piece of code that now gives the correct result
   (let-values (((a b)(http-request "https://www.google.com" 
#:streaming? #t)))
     (bytevector->string (get-bytevector-all b) "ISO-8859-1"))

As conclusion I can say that web modules read correctly the http answers 
and, with some (one for me, the google web site) https sites there is a 
misinterpretation of EOF in the layer between http and https.

Franco.

bug#39800: gnutls guile bug receiving https data

[Andy Wingo]

Thanks very much for the report!

I think this one may be a good one for Ludovic, if I may be so bold.
Apologies for the top-post but I couldn't clip your excellent report.

Cheers,

Andy

"franco.rcr@gmail.com" <franco.rcr@gmail.com> writes:

> Hello,
> I installed gnutls for guile and checked the gnutls module with this
> simple code:
>
>
> ;;Guile version 3.0 and gnutls  from git
>
> ,show version
> GNU Guile 3.0.0.15-ff14b7
>
> (gnutls-version)
> $6 = "3.6.12"
>
> ;;Now, submitting this simple https request, you get an exception
> (http-request "https://www.google.com")
> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
> Throw to key `gnutls-error' with args `(#<gnutls-error-enum La
> connessione TLS non è stata terminata in modo corretto.>
> read_from_session_record_port)'.
>
> ;;instead, without https there are no errors
> (http-request "http://www.google.com") ;;works fine.
>
>
> The error happens only on https://www.google.com and does not throw with
> a lot of other https web sites.
> Furthermore the error is not throw if the method is HEAD, so it is
> related to the data part of the https answer.
> I tried to enter in the internal implementation of the http web client
> but after some tests I decided to do some simple tests at application
> level.
> I rewrote the get-bytevector-all, with a loop that reads one byte per
> time and the error was thrown anyway.
> I catched the error and I've got the complete answer from the google web
> server.
>
> In the following there is my applicative solution, where I rewrote the
> get-bytevector-all by adding the error checking and specifiyng to
> http-request that the data has to be returned as a port (#:streaming?
> #t).
>
>
> ;;A macro to catch errors
> (define-syntax my-noerr
>   (syntax-rules ()
>     ((_ __error-return exp ...)
>      (let
>          ((__st #f))
>        (catch #t
>          (lambda() exp ...)
>          (lambda (k . p) __error-return))))))
>
> ;;The rewriting of get-bytevector-all
> (defun get-bytevector-all (port)
>   (u8-list->bytevector (let loop ((port port))
>      (let ((v (my-noerr (eof-object) (get-u8 port))))
>        (if (eof-object? v)
>        #nil
>        (cons v (loop port)))))))
>
> ;;the piece of code that now gives the correct result
>   (let-values (((a b)(http-request "https://www.google.com" #:streaming?
> #t)))
>     (bytevector->string (get-bytevector-all b) "ISO-8859-1"))
>
> As conclusion I can say that web modules read correctly the http answers
> and, with some (one for me, the google web site) https sites there is a
> misinterpretation of EOF in the layer between http and https.
>
> Franco.

bug#39800: gnutls guile bug receiving https data

[Ludovic Courtès]

Hi,

"franco.rcr@gmail.com" <franco.rcr@gmail.com> skribis:

> ;;Now, submitting this simple https request, you get an exception
> (http-request "https://www.google.com")
> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
> Throw to key `gnutls-error' with args `(#<gnutls-error-enum La
> connessione TLS non è stata terminata in modo corretto.>
> read_from_session_record_port)'.

That happens when passing #:keep-alive? #f (the default), specifically
when reading the response body:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (http-request "https://www.google.com" #:keep-alive? #f #:streaming? #t)
$1 = #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers: ((date . #<date nanosecond: 0 second: 56 minute: 45 hour: 21 day: 6 month: 3 year: 2020 zone-offset: 0>) (expires . #<date nanosecond: 0 second: 0 minute: 0 hour: 0 day: 1 month: 1 year: 1970 zone-offset: 0>) (cache-control private (max-age . 0)) (content-type text/html (charset . "ISO-8859-1")) (p3p . "CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"") (server . "gws") (x-xss-protection . "0") (x-frame-options . "SAMEORIGIN") (set-cookie . "1P_JAR=2020-03-06-21; expires=Sun, 05-Apr-2020 21:45:56 GMT; path=/; domain=.google.com; Secure") (set-cookie . "NID=199=yXgE_KAGvxJbZAIGEXLt8CsEe3pre-RRLm1Jqap3b3iJRqZZq_PJ9wCT798mfDZ2TC5_3mKnM5KABSh8CguI64SsNoWHIc9EsW2osFltsIJnMswXhrtjQFDpfm_fUb6RDrWrqKHkOuvkG7Izp5im1Ys1TzGdztrFmOQV4FOraJk; expires=Sat, 05-Sep-2020 21:45:56 GMT; path=/; domain=.google.com; HttpOnly") (alt-svc . "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q050=\":443\"; ma=2592000,h3-Q049=\":443\"; ma=2592000,h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000") (accept-ranges none) (vary accept-encoding) (connection close)) port: #<input-output: file 7f3ae7dd3540>>
$2 = #<input-output: file 7f3ae7dd3540>
scheme@(guile-user)> (define bv (get-bytevector-all $2))
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `gnutls-error' with args `(#<gnutls-error-enum The TLS connection was non-properly terminated.> read_from_session_record_port)'.
--8<---------------cut here---------------end--------------->8---

The reason for this is that google.com closes the connection right away,
without sending a proper TLS “bye” message as is conventionally done.

Fixed in commit 076276c4f580368b4106316a77752d69c8f1494a, which will be
in 3.0.1.

Thanks,
Ludo’.