From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Vivien Kraus Newsgroups: gmane.lisp.guile.user Subject: Re: Some help needed to use curl lib to download binary file Date: Fri, 29 Jul 2022 12:25:53 +0200 Message-ID: References: <874jz01erv.fsf@univ-rouen.fr> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="6168"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Evolution 3.42.1 To: =?ISO-8859-1?Q?S=E9bastien?= Rey-Coyrehourcq , guile-user@gnu.org Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Fri Jul 29 12:28:07 2022 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oHNDz-0001Pk-9F for guile-user@m.gmane-mx.org; Fri, 29 Jul 2022 12:28:07 +0200 Original-Received: from localhost ([::1]:56294 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oHNDy-0008Gg-3A for guile-user@m.gmane-mx.org; Fri, 29 Jul 2022 06:28:06 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:57500) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oHNCH-0008F6-6p for guile-user@gnu.org; Fri, 29 Jul 2022 06:26:24 -0400 Original-Received: from planete-kraus.eu ([2a00:5881:4008:2810::309]:45676) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1oHNCE-0006Ol-H2 for guile-user@gnu.org; Fri, 29 Jul 2022 06:26:20 -0400 Original-Received: from planete-kraus.eu (localhost.lan [127.0.0.1]) by planete-kraus.eu (OpenSMTPD) with ESMTP id 47474a4c; Fri, 29 Jul 2022 10:26:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=planete-kraus.eu; h= message-id:subject:from:to:date:in-reply-to:references :content-type:mime-version:content-transfer-encoding; s=dkim; bh=/JNaVLQ2DvlmH/O4Q/gHfFdvVRo=; b=LprInwGtNsuh3OLMJ4Fzp6lKTH5j DOb0xkipX9HlaAgsYBOGJLk03+lqb5gUkymOQZSGoAavMJyEQawXu99WzKTF+Uw8 SrqKEQxPU60Up/LllU4i8+DQcr8YaIWQ/MNfJAdCfCDPKNHcPPm3wk1YZeNmzNy7 IUSSWZPpVtmuOvw= Original-Received: by planete-kraus.eu (OpenSMTPD) with ESMTPSA id ba62fecb (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Fri, 29 Jul 2022 10:25:53 +0000 (UTC) In-Reply-To: <874jz01erv.fsf@univ-rouen.fr> Received-SPF: pass client-ip=2a00:5881:4008:2810::309; envelope-from=vivien@planete-kraus.eu; helo=planete-kraus.eu X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:18464 Archived-At: Hello, I see in the paste: > ;; function taken on > https://gist.github.com/amirouche/138a27bdbef5a672a0135f90ca26ec41 > ;; then adapted to use cookie jar > (define-public (http-get url cookie-exist) >   ;; Create a Curl handle >   (let ((handle (curl-easy-init))) >     ;; Set the URL from which to get the data >     (curl-easy-setopt handle 'url url) >     (if cookie-exist >         (curl-easy-setopt handle 'cookie "cookie.txt") >         (curl-easy-setopt handle 'cookiejar "cookie.txt")) > >     ;; Request that the HTTP headers be included in the response >     (curl-easy-setopt handle 'header #t) >     ;; Get the result as a Latin-1 string >     (let* ((response-string (curl-easy-perform handle)) >            ;; Create a string port from the response >            (response-port (open-input-string response-string)) >            ;; Have the (web response) module to parse the response >            (response (read-response response-port)) >            (body (utf8->string (read-response-body response)))) >       (close response-port) >       ;; Have the (web response) module extract the body from the >       ;; response >       (values response body)))) So here the call expects the response to be UTF-8 text. If it is a binary file that you are downloading, the function will raise an exception. Guile has that python3 feeling where you are supposed to know in advance whether what you are using is text or binary, which is hurting you here.  However, you can avoid the problem by either having bytevectors everywhere, so removing the call to utf8->string and bind the "body" variable directly to (read-response-body response), or pretend that you know better than guile and pretend that it is latin-1-encoded, so you lose no information and guile won’t complain. In that case, load (ice-9 iconv) and replace (utf8->string …) with (bytevector->string … "ISO- 8859-1"). If you go the first route, you get a bytevector back. If you go the second one, you get a string, but you must remember that it contains raw bytes and not text (unless the response body was indeed text). There are some cases when you might want to have such strings-that- contain-binary-or-text, such as if you want to use the strings API on them and the bytevector API does not provide what you want, or you want to interface with NUL-terminated strings. In other cases you might prefer bytevectors. Anyway, there is no encoding cost to convert between strings-that-contain-binary-or-toxt and bytevectors, it is as simple as a copy. > ;; write content into file > (call-with-output-file “download.zip” (lambda (current-output-port) >                                         (get-file get-file-link) >                                         (put-bytevector (current- > output-port) body))) call-with-output-file calls its function argument with a port. Plus, you just ignore the result of get-file, so instead you should do something with it. If you chose to have get-file return a bytevector, you can do: (call-with-output-file "download.zip"   (lambda (port)     (put-bytevector port (get-file get-file-link)))   #:binary #t) If get-file returns a string, you can do: (call-with-output-file "download.zip"   (lambda (port)     (put-string port (get-file get-file-link)))   #:encoding "ISO-8859-1") Best regards, Vivien