From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Joseph Turner Newsgroups: gmane.emacs.devel Subject: Re: How to get buffer byte length (not number of characters)? Date: Thu, 22 Aug 2024 00:24:45 -0700 Message-ID: <87ed6hdnpe.fsf@ushin.org> References: <87wmkbekjp.fsf@ushin.org> <86o75nwilg.fsf@gnu.org> <87bk1lhkvg.fsf@ushin.org> <86y14pu5rp.fsf@gnu.org> <871q2hfn7c.fsf@ushin.org> <86plq1td4n.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="28979"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org, schwab@suse.de, adam@alphapapa.net To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Aug 22 12:02:43 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sh4eQ-0007Jo-BB for ged-emacs-devel@m.gmane-mx.org; Thu, 22 Aug 2024 12:02:43 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sh4dW-0003cl-9j; Thu, 22 Aug 2024 06:01:46 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sh2Bm-0000VM-OK for emacs-devel@gnu.org; Thu, 22 Aug 2024 03:24:58 -0400 Original-Received: from out-178.mta1.migadu.com ([95.215.58.178]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sh2Bk-0002Ae-4I for emacs-devel@gnu.org; Thu, 22 Aug 2024 03:24:58 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ushin.org; s=key1; t=1724311492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TvKEBcjUZ7o47DcqGXUomq4HTQIsSXNbhcu5zoDL8HQ=; b=UMsilnzMdbf/fUUi0yk/1B0UNRfAM4NbTFwIAFe+7jQ6z7msWPT9vyt/ARv+KCbYoMBpdM iZxm/PsYzcYhvcyvwzDdB/Pod8R2dcfPtRzehaEuAYwwvIzejwj/eHNmXAUHI8OKuquZjJ Ebo/RJ8O4DhFD+6bay6VeEQSyMNRnLCYJSmV0vHeRmTOq6CqcI09pl6lcv0lIfjK5XwWda 1rpymllJwlFG5lfvKaUFWQK1Pxj+Slvwj0ZStjNmzJLJok+DChK2s0f1ZEUcKPjWJ1GYLD Y71YMtHB3s5D10JGySPGC0N4vDgdRzb+w3DxRp1SUp6xsNG4/OWurURPdfOz6Q== In-Reply-To: <86plq1td4n.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 22 Aug 2024 07:06:32 +0300") X-Migadu-Flow: FLOW_OUT Received-SPF: pass client-ip=95.215.58.178; envelope-from=joseph@ushin.org; helo=out-178.mta1.migadu.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Thu, 22 Aug 2024 06:01:36 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:323040 Archived-At: --=-=-= Content-Type: text/plain Eli Zaretskii writes: >> From: Joseph Turner >> Cc: emacs-devel@gnu.org, schwab@suse.de, adam@alphapapa.net >> Date: Wed, 21 Aug 2024 16:52:39 -0700 >> >> Eli Zaretskii writes: >> >> >> Currently, plz.el always creates the curl subprocess like so: >> >> >> >> (make-process :coding 'binary ...) >> >> >> >> https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/plz.el?h=externals-release/plz#n519 >> >> >> >> Does this DTRT? >> > >> > It could be TRT if plz.el encodes the buffer text "by hand" before >> > sending the results to curl and decodes it when it receives text from >> > curl. Which I think is what happens there. >> >> plz.el does not manually encode buffer text *within Emacs* when sending >> requests to curl, but by default, plz.el sends data to curl with --data, >> which tells curl to strip CR and newlines. With the :body-type 'binary >> argument, plz.el instead uses --data-binary, which does no conversion. > > Newlines is a relatively minor issue (although it, too, needs to be > considered). My main concern is with the text encoding. How can it > be TRT to use 'binary when sending buffer text to curl? that would > mean we are more-or-less always sending the internal representation of > characters, which is superset of UTF-8. If the data was originally > encoded in anything but UTF-8, reading it into Emacs and then sending > it back will change the byte sequences from that other encoding to > UTF-8. Moreover, 'binary does not guarantee that the result is valid > UTF-8. > > So maybe I misunderstand how these plz.el facilities are used, but up > front this sounds like a mistake. It could be. Eli, Adam, what do you think about the default coding systems for encoding the request body in the attached patch? >> We don't want to strip newlines from hyperdrive files, so we always use >> :body-type 'binary when sending buffer contents. Should hyperdrive.el >> encode data with `buffer-file-coding-system' before passing to plz.el? > > I would think so, but maybe we should bring the plz.el developers on > board of this discussion. I've CC'd Adam. >> When receiving text from curl, plz.el optionally decodes the text >> according to the charset in the 'Content-Type' header, e.g., "text/html; >> charset=utf-8" or utf-8 if no charset is found. > > By "optionally" you mean that it doesn't always happen, except if the > caller requests that? If so, the caller of plz.el should decode the > text manually before using it in user-facing features. By default, `plz' decodes response body according to the 'Content-Type' charset (or utf-8 as fallback). Passing `:decode nil' stops that. >> Perhaps hyperdrive.el should check the 'Content-Type' header charset, >> then fallback to guessing the coding system based on filename and file >> contents with `set-auto-coding' (to avoid decoding images, etc.), and >> then finally fallback to something else? > > Probably. But then I don't know anything about hyperdrive.el, either. > If it copies text between files or URLs without showing it to the > user, then the best strategy is indeed not to decode and encode stuff, > but handle it as a stream of raw bytes. (In that case, my suggestion > would be to use unibyte buffers and strings for temporarily storing > and processing these raw bytes in Emacs.) But if the text is somehow > shown to the user, it must be decoded to be displayed correctly by > Emacs. And then it must be encoded back when writing it back to the > external storage. Thanks! Good to know about unibyte buffers and strings for that. hyperdrive.el does show text to the user, so we'll likely do something like what I described above. What fallback encoding should we use if there's no 'Content-Type' charset and `set-auto-coding' returns nil? IIUC, there's no foolproof way to guess the encoding of unknown bytes. default-file-name-coding-system? Thank you!! I feel more solid in my understanding of encodings now. Joseph --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Add-plz-BODY-CODING-argument-Add-default-encoding.patch >From a684ff680ab05f359b628623159b4d3392eb448e Mon Sep 17 00:00:00 2001 From: Joseph Turner Date: Thu, 22 Aug 2024 00:02:19 -0700 Subject: [PATCH] Add: (plz) BODY-CODING argument; Add default encoding Previously, strings and buffers were sent to curl as the internal Emacs representation. Now strings and buffers are encoded, and the BODY-CODING argument can used to override the default coding systems. --- plz.el | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/plz.el b/plz.el index 903d71e..91d41d2 100644 --- a/plz.el +++ b/plz.el @@ -323,7 +323,7 @@ (defalias 'plz--generate-new-buffer ;;;;; Public -(cl-defun plz (method url &rest rest &key headers body else filter finally noquery timeout +(cl-defun plz (method url &rest rest &key headers body else filter finally noquery timeout body-coding (as 'string) (then 'sync) (body-type 'text) (decode t decode-s) (connect-timeout plz-connect-timeout)) @@ -340,6 +340,13 @@ (cl-defun plz (method url &rest rest &key headers body else filter finally noque BODY-TYPE may be `text' to send BODY as text, or `binary' to send it as binary. +BODY-CODING may a coding system used to encode BODY before +passing it to curl. BODY-CODING has no effect when BODY is a +list like `(file FILENAME)'. If nil and BODY is a string, the +default process I/O output coding system is used. If nil and +BODY is a buffer, the buffer-local value of +`buffer-file-coding-system' is used. + AS selects the kind of result to pass to the callback function THEN, or the kind of result to return for synchronous requests. It may be: @@ -416,6 +423,19 @@ (cl-defun plz (method url &rest rest &key headers body else filter finally noque (declare (indent defun)) (setf decode (if (and decode-s (not decode)) nil decode)) + (unless body-coding + (pcase-exhaustive body + (`(file ,filename) + ;; Don't set BODY-CODING; files are passed as-is to curl. + (setf body-coding nil)) + ((pred stringp) + ;; Use default output coding for processes. + (setf body-coding (cdr default-process-coding-system))) + ((and (pred bufferp) buffer) + ;; Use buffer-local coding. + (setf body-coding + (buffer-local-value 'buffer-file-coding-system buffer))))) + ;; NOTE: By default, for PUT requests and POST requests >1KB, curl sends an ;; "Expect:" header, which causes servers to send a "100 Continue" response, which ;; we don't want to have to deal with, so we disable it by setting the header to @@ -553,8 +573,11 @@ (cl-defun plz (method url &rest rest &key headers body else filter finally noque (process-send-string process curl-config) (when body (cl-typecase body - (string (process-send-string process body)) - (buffer (with-current-buffer body + (string (process-send-string + process (encode-coding-string body body-coding t))) + (buffer (with-temp-buffer + (insert-buffer-substring-no-properties body) + (encode-coding-region (point-min) (point-max) body-coding) (process-send-region process (point-min) (point-max)))))) (process-send-eof process) (if sync-p -- 2.41.0 --=-=-=--