How to contribute new package to GNU ELPA?

How to contribute new package to GNU ELPA?

[stardiviner]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

As this issue https://github.com/stardiviner/kiwix.el/issues/3 suggested.
I would like to contribute my package kiwix.el to GNU ELPA.
I followed this wiki manual https://www.emacswiki.org/emacs/ELPA
Here is the steps I have done:

- [X] assign FSF copyright assignment. my name is "Christoper M. Miles"
- [X] created account "stardiviner" on Savannah.gnu.org. Also I request to
be member in group "emacs" in page https://savannah.gnu.org/my/groups.php
Still waiting for approval.
- [ ] I try to clone ELPA member repo with following command:
```
git clone stardiviner@git.savannah.gnu.org:/srv/git/emacs/elpa.git elpa
```
But don't have permission. Don't know need what permission.

- [ ] About my package "kiwix.el", what do I need to modify in my source
code file?

Thanks for your help. :smile:

[stardiviner]           <Hack this world!>      GPG key ID: 47C32433
IRC(freeenode): stardiviner                     Twitter:  @numbchild
Key fingerprint = 9BAA 92BC CDDD B9EF 3B36  CB99 B8C4 B8E5 47C3 2433
Blog: http://stardiviner.github.io/

[-- Attachment #2: Type: text/html, Size: 2904 bytes --]

Re: How to contribute new package to GNU ELPA?

[Stefan Monnier]

stardiviner [2020-12-17 20:11:39] wrote:
> As this issue https://github.com/stardiviner/kiwix.el/issues/3 suggested.
> I would like to contribute my package kiwix.el to GNU ELPA.

I've just started to look at it.  I have done some of the preliminary
work, but there are some problems to solve:

1- The package uses `request` which is neither in Emacs nor in GNU ELPA.
2- The PNG images which are useful for the Homepage make the package
   much larger for no good reason, so I think we should add
   a `.elpaignore` file so as not to include those PNG imagines in the
   package's tarball.

3- Even with `request`, compiling the package fails because
   `org-kiwix.el` requirex `kiwix` and loading `kiwix` signal an error:

      (file-missing "Opening directory" "Aucun fichier ou dossier de ce type" "<HOMEDIR>/.www.kiwix.org/kiwix")

   So, I think `file-accessible-directory-p` is not the function you
   want to use.

The patch below fixes some of those problems, but I still get warnings
about a missing .www.kiwix... directory when I load the file, which can
happen even if the user never intended to actually use any of that
package's functionality: those warnings should be delayed to when we
actually call some of the package's functions.

The main problem, tho, is the `request` dependency.  There are two
possible solutions to that: either we add `request` to GNU ELPA, or we
rewrite the code so as not to use `request` (IIUC `request` is
a reasonably thing wrapper above the `url` package, so it might not be
too bad to do).  The best course is probably to add `request` to
GNU ELPA.  Could you look into that, see how much work it would take to
include `request` in GNU ELPA?


        Stefan


diff --git a/kiwix.el b/kiwix.el
index 5b765b5d3c..377772ee2e 100644
--- a/kiwix.el
+++ b/kiwix.el
@@ -1,5 +1,5 @@
-;;; kiwix.el --- Searching offline Wikipedia through Kiwix.
-;;; -*- coding: utf-8 -*-
+;;; kiwix.el --- Searching offline Wikipedia through Kiwix.  -*- lexical-binding: t; -*-
+;; -*- coding: utf-8 -*-
 
 ;; Copyright (C) 2019-2020  Free Software Foundation, Inc.
 
@@ -9,7 +9,7 @@
 ;; URL: https://github.com/stardiviner/kiwix.el
 ;; Created: 23th July 2016
 ;; Version: 1.0.0
-;; Package-Requires: ((emacs "24.4") (cl-lib "0.5") (request "0.3.0"))
+;; Package-Requires: ((emacs "24.4") (request "0.3.0"))
 
 ;; This file is part of GNU Emacs.
 
@@ -28,13 +28,13 @@
 
 ;;; Commentary:
 
-;;; This currently only works for Linux, not tested for Mac OS X and Windows.
+;; This currently only works for Linux, not tested for Mac OS X and Windows.
 
-;;; Kiwix installation
+;;;; Kiwix installation
 ;;
 ;; http://www.kiwix.org
 
-;;; Config:
+;;;; Config:
 ;;
 ;; (use-package kiwix
 ;;   :ensure t
@@ -45,7 +45,7 @@
 ;;               kiwix-server-port 8080
 ;;               kiwix-default-library "wikipedia_zh_all_2015-11.zim"))
 
-;;; Usage:
+;;;; Usage:
 ;;
 ;; 1. [M-x kiwix-launch-server] to launch Kiwix server.
 ;; 2. [M-x kiwix-at-point] to search the word under point or the region selected string.
@@ -58,7 +58,7 @@
 (require 'subr-x)
 (require 'thingatpt)
 (require 'json)
-(if (featurep 'ivy) (require 'ivy))
+(if (featurep 'ivy) (require 'ivy))     ;FIXME: That's a no-op!
 
 (defgroup kiwix-mode nil
   "Kiwix customization options."
@@ -67,19 +67,16 @@
 (defcustom kiwix-server-use-docker nil
   "Using Docker container for kiwix-serve or not?"
   :type 'boolean
-  :safe #'booleanp
-  :group 'kiwix-mode)
+  :safe #'booleanp)
 
 (defcustom kiwix-server-port 8000
   "Specify default kiwix-serve server port."
   :type 'number
-  :safe #'numberp
-  :group 'kiwix-mode)
+  :safe #'numberp)
 
 (defcustom kiwix-server-url (format "http://127.0.0.1:%s" kiwix-server-port)
   "Specify Kiwix server URL."
-  :type 'string
-  :group 'kiwix-mode)
+  :type 'string)
 
 (defcustom kiwix-server-command
   (cond
@@ -90,44 +87,40 @@
    ((string-equal system-type "windows-nt")
     (warn "You need to specify Windows Kiwix path. And send a PR to my repo.")))
   "Specify kiwix server command."
-  :type 'string
-  :group 'kiwix-mode)
+  :type 'string)
 
 (defun kiwix-dir-detect ()
   "Detect Kiwix profile directory exist."
-  (let ((kiwix-dir (concat (getenv "HOME") "/.www.kiwix.org/kiwix")))
-    (if (file-accessible-directory-p kiwix-dir)
+  (let ((kiwix-dir "~/.www.kiwix.org/kiwix"))
+    (if (file-directory-p kiwix-dir)
         kiwix-dir
-      (warn "ERROR: Kiwix profile directory \".www.kiwix.org/kiwix\" is not accessible."))))
+      (warn "ERROR: Kiwix profile directory \"~/.www.kiwix.org/kiwix\" is not accessible.")
+      nil)))
 
 (defcustom kiwix-default-data-profile-name
   (when (kiwix-dir-detect)
-    (car (directory-files
-          (concat (getenv "HOME") "/.www.kiwix.org/kiwix") nil ".*\\.default")))
+    (car (directory-files "~/.www.kiwix.org/kiwix" nil ".*\\.default")))
   "Specify the default Kiwix data profile path."
-  :type 'string
-  :group 'kiwix-mode)
+  :type 'string)
 
 (defcustom kiwix-default-data-path
   (when (kiwix-dir-detect)
-    (concat (getenv "HOME") "/.www.kiwix.org/kiwix/" kiwix-default-data-profile-name))
+    (expand-file-name kiwix-default-data-profile-name
+                      "~/.www.kiwix.org/kiwix/"))
   "Specify the default Kiwix data path."
   :type 'string
-  :safe #'stringp
-  :group 'kiwix-mode)
+  :safe #'stringp)
 
 (defcustom kiwix-default-library-path (file-name-directory
                                        (concat kiwix-default-data-path "/data/library/library.xml"))
   "Kiwix libraries path."
   :type 'string
-  :safe #'stringp
-  :group 'kiwix-mode)
+  :safe #'stringp)
 
 (defcustom kiwix-default-completing-read 'ivy
   "Kiwix default completion frontend. Currently Ivy ('ivy) and Helm ('helm) both supported."
   :type 'symbol
-  :safe #'symbolp
-  :group 'kiwix-mode)
+  :safe #'symbolp)
 
 (defcustom kiwix-default-browser-function browse-url-browser-function
   "Set default browser for open kiwix query result URL."
@@ -139,8 +132,7 @@
           (const :tag "Google Chrome web browser" browse-url-chrome)
           (const :tag "Conkeror web browser" browse-url-conkeror)
           (const :tag "xwidget browser" xwidget-webkit-browse-url))
-  :safe #'symbolp
-  :group 'kiwix-mode)
+  :safe #'symbolp)
 
 ;;;###autoload
 (defun kiwix--get-library-name (file)
@@ -179,19 +171,16 @@ Like in function `kiwix-ajax-search-hints'.")
 (defcustom kiwix-default-library "wikipedia_en_all.zim"
   "The default kiwix library when library fragment in link not specified."
   :type 'string
-  :safe #'stringp
-  :group 'kiwix-mode)
+  :safe #'stringp)
 
 (defcustom kiwix-search-interactively t
   "`kiwix-at-point' search interactively."
   :type 'boolean
-  :safe #'booleanp
-  :group 'kiwix-mode)
+  :safe #'booleanp)
 
 (defcustom kiwix-mode-prefix nil
   "Specify kiwix-mode keybinding prefix before loading."
-  :type 'kbd
-  :group 'kiwix-mode)
+  :type 'kbd)
 
 ;; update kiwix server url and port
 (defun kiwix-server-url-update ()
@@ -258,7 +247,7 @@ Like in function `kiwix-ajax-search-hints'.")
                 (when (string-equal (cdr error-thrown) "exited abnormally with code 7\n")
                   (warn "kiwix.el failed to connect to host. exited abnormally with status code: 7."))))
       :success (cl-function
-                (lambda (&key data &allow-other-keys)
+                (lambda (&key _data &allow-other-keys) ;FIXME: Why not `&rest _'?
                   (setq kiwix-server-available? t)))
       :status-code '((404 . (lambda (&rest _) (message (format "Endpoint %s does not exist." url))))
                      (500 . (lambda (&rest _) (message (format "Error from  %s." url))))))))
@@ -290,12 +279,12 @@ list and return a list result."
           (mapcar 'cdar data)))))
 
 ;;;###autoload
-(defun kiwix-at-point (&optional interactively)
+(defun kiwix-at-point ()
   "Search for the symbol at point with `kiwix-query'.
 
 Or When prefix argument `INTERACTIVELY' specified, then prompt
 for query string and library interactively."
-  (interactive "P")
+  (interactive)
   (unless (kiwix-ping-server)
     (kiwix-launch-server))
   (if kiwix-server-available?
@@ -368,7 +357,6 @@ for query string and library interactively."
   :require 'kiwix-mode
   :init-value nil
   :lighter " Kiwix"
-  :group 'kiwix-mode
   :keymap kiwix-mode-map
   (if kiwix-mode (kiwix-mode-enable) (kiwix-mode-disable)))
 

Re: How to contribute new package to GNU ELPA?

[stardiviner]

On 2020/12/19 下午2:22, Stefan Monnier wrote:
> stardiviner [2020-12-17 20:11:39] wrote:
>> As this issue https://github.com/stardiviner/kiwix.el/issues/3 suggested.
>> I would like to contribute my package kiwix.el to GNU ELPA.
> I've just started to look at it.  I have done some of the preliminary
> work, but there are some problems to solve:
>
> 1- The package uses `request` which is neither in Emacs nor in GNU ELPA.
> 2- The PNG images which are useful for the Homepage make the package
>     much larger for no good reason, so I think we should add
>     a `.elpaignore` file so as not to include those PNG imagines in the
>     package's tarball.
>
> 3- Even with `request`, compiling the package fails because
>     `org-kiwix.el` requirex `kiwix` and loading `kiwix` signal an error:
>
>        (file-missing "Opening directory" "Aucun fichier ou dossier de ce type" "<HOMEDIR>/.www.kiwix.org/kiwix")
>
>     So, I think `file-accessible-directory-p` is not the function you
>     want to use.
>
> The patch below fixes some of those problems, but I still get warnings
> about a missing .www.kiwix... directory when I load the file, which can
> happen even if the user never intended to actually use any of that
> package's functionality: those warnings should be delayed to when we
> actually call some of the package's functions.
>
> The main problem, tho, is the `request` dependency.  There are two
> possible solutions to that: either we add `request` to GNU ELPA, or we
> rewrite the code so as not to use `request` (IIUC `request` is
> a reasonably thing wrapper above the `url` package, so it might not be
> too bad to do).  The best course is probably to add `request` to
> GNU ELPA.  Could you look into that, see how much work it would take to
> include `request` in GNU ELPA?
>
>
>          Stefan
>
>
> diff --git a/kiwix.el b/kiwix.el
> index 5b765b5d3c..377772ee2e 100644
> --- a/kiwix.el
> +++ b/kiwix.el
> @@ -1,5 +1,5 @@
> -;;; kiwix.el --- Searching offline Wikipedia through Kiwix.
> -;;; -*- coding: utf-8 -*-
> +;;; kiwix.el --- Searching offline Wikipedia through Kiwix.  -*- lexical-binding: t; -*-
> +;; -*- coding: utf-8 -*-
>   
>   ;; Copyright (C) 2019-2020  Free Software Foundation, Inc.

lexical-binding added.

>   
> @@ -9,7 +9,7 @@
>   ;; URL: https://github.com/stardiviner/kiwix.el
>   ;; Created: 23th July 2016
>   ;; Version: 1.0.0
> -;; Package-Requires: ((emacs "24.4") (cl-lib "0.5") (request "0.3.0"))
> +;; Package-Requires: ((emacs "24.4") (request "0.3.0"))

cl-lib should be required for `cl-function`.

>   
>   ;; This file is part of GNU Emacs.
>   
> @@ -28,13 +28,13 @@
>   
>   ;;; Commentary:
>   
> -;;; This currently only works for Linux, not tested for Mac OS X and Windows.
> +;; This currently only works for Linux, not tested for Mac OS X and Windows.
>   
> -;;; Kiwix installation
> +;;;; Kiwix installation
>   ;;
>   ;; http://www.kiwix.org
>   
> -;;; Config:
> +;;;; Config:
>   ;;
>   ;; (use-package kiwix
>   ;;   :ensure t
> @@ -45,7 +45,7 @@
>   ;;               kiwix-server-port 8080
>   ;;               kiwix-default-library "wikipedia_zh_all_2015-11.zim"))
>   
> -;;; Usage:
> +;;;; Usage:

Why use four `;` instead of three `;`? I search many packages, they all 
use three `;`. I don't know which one is the standard.

>   ;;
>   ;; 1. [M-x kiwix-launch-server] to launch Kiwix server.
>   ;; 2. [M-x kiwix-at-point] to search the word under point or the region selected string.
> @@ -58,7 +58,7 @@
>   (require 'subr-x)
>   (require 'thingatpt)
>   (require 'json)
> -(if (featurep 'ivy) (require 'ivy))
> +(if (featurep 'ivy) (require 'ivy))     ;FIXME: That's a no-op!

What does the "no-op" mean? Because some user might have not installed 
ivy, so I added one condition to detect it here.

>   
>   (defgroup kiwix-mode nil
>     "Kiwix customization options."
> @@ -67,19 +67,16 @@
>   (defcustom kiwix-server-use-docker nil
>     "Using Docker container for kiwix-serve or not?"
>     :type 'boolean
> -  :safe #'booleanp
> -  :group 'kiwix-mode)
> +  :safe #'booleanp)
>   
>   (defcustom kiwix-server-port 8000
>     "Specify default kiwix-serve server port."
>     :type 'number
> -  :safe #'numberp
> -  :group 'kiwix-mode)
> +  :safe #'numberp)
>   
>   (defcustom kiwix-server-url (format "http://127.0.0.1:%s" kiwix-server-port)
>     "Specify Kiwix server URL."
> -  :type 'string
> -  :group 'kiwix-mode)
> +  :type 'string)
>   
>   (defcustom kiwix-server-command
>     (cond
> @@ -90,44 +87,40 @@
>      ((string-equal system-type "windows-nt")
>       (warn "You need to specify Windows Kiwix path. And send a PR to my repo.")))
>     "Specify kiwix server command."
> -  :type 'string
> -  :group 'kiwix-mode)
> +  :type 'string)
>   

Why deleted all `:group 'kiwix-mode`? If think correct, the :group is 
used by `customize-group`. So It should be necessary.

>   (defun kiwix-dir-detect ()
>     "Detect Kiwix profile directory exist."
> -  (let ((kiwix-dir (concat (getenv "HOME") "/.www.kiwix.org/kiwix")))
> -    (if (file-accessible-directory-p kiwix-dir)
> +  (let ((kiwix-dir "~/.www.kiwix.org/kiwix"))
> +    (if (file-directory-p kiwix-dir)

Use `file-accessible-directory-p` because also can detect folder 
permission. If user can't read folder, that's a problem too.

I prefer to use `$HOME` environment variable, it should be more 
cross-platformed.

>           kiwix-dir
> -      (warn "ERROR: Kiwix profile directory \".www.kiwix.org/kiwix\" is not accessible."))))
> +      (warn "ERROR: Kiwix profile directory \"~/.www.kiwix.org/kiwix\" is not accessible.")
> +      nil)))
Applied
>   
>   (defcustom kiwix-default-data-profile-name
>     (when (kiwix-dir-detect)
> -    (car (directory-files
> -          (concat (getenv "HOME") "/.www.kiwix.org/kiwix") nil ".*\\.default")))
> +    (car (directory-files "~/.www.kiwix.org/kiwix" nil ".*\\.default")))
Same as upper.
>     "Specify the default Kiwix data profile path."
> -  :type 'string
> -  :group 'kiwix-mode)
> +  :type 'string)
>   
>   (defcustom kiwix-default-data-path
>     (when (kiwix-dir-detect)
> -    (concat (getenv "HOME") "/.www.kiwix.org/kiwix/" kiwix-default-data-profile-name))
> +    (expand-file-name kiwix-default-data-profile-name
> +                      "~/.www.kiwix.org/kiwix/"))
>     "Specify the default Kiwix data path."
>     :type 'string
> -  :safe #'stringp
> -  :group 'kiwix-mode)
> +  :safe #'stringp)
>   
Same as upper.
>   (defcustom kiwix-default-library-path (file-name-directory
>                                          (concat kiwix-default-data-path "/data/library/library.xml"))
>     "Kiwix libraries path."
>     :type 'string
> -  :safe #'stringp
> -  :group 'kiwix-mode)
> +  :safe #'stringp)
>   
>   (defcustom kiwix-default-completing-read 'ivy
>     "Kiwix default completion frontend. Currently Ivy ('ivy) and Helm ('helm) both supported."
>     :type 'symbol
> -  :safe #'symbolp
> -  :group 'kiwix-mode)
> +  :safe #'symbolp)
>   
>   (defcustom kiwix-default-browser-function browse-url-browser-function
>     "Set default browser for open kiwix query result URL."
> @@ -139,8 +132,7 @@
>             (const :tag "Google Chrome web browser" browse-url-chrome)
>             (const :tag "Conkeror web browser" browse-url-conkeror)
>             (const :tag "xwidget browser" xwidget-webkit-browse-url))
> -  :safe #'symbolp
> -  :group 'kiwix-mode)
> +  :safe #'symbolp)
>   
>   ;;;###autoload
>   (defun kiwix--get-library-name (file)
> @@ -179,19 +171,16 @@ Like in function `kiwix-ajax-search-hints'.")
>   (defcustom kiwix-default-library "wikipedia_en_all.zim"
>     "The default kiwix library when library fragment in link not specified."
>     :type 'string
> -  :safe #'stringp
> -  :group 'kiwix-mode)
> +  :safe #'stringp)
>   
>   (defcustom kiwix-search-interactively t
>     "`kiwix-at-point' search interactively."
>     :type 'boolean
> -  :safe #'booleanp
> -  :group 'kiwix-mode)
> +  :safe #'booleanp)
>   
>   (defcustom kiwix-mode-prefix nil
>     "Specify kiwix-mode keybinding prefix before loading."
> -  :type 'kbd
> -  :group 'kiwix-mode)
> +  :type 'kbd)
>   
Same as upper.
>   ;; update kiwix server url and port
>   (defun kiwix-server-url-update ()
> @@ -258,7 +247,7 @@ Like in function `kiwix-ajax-search-hints'.")
>                   (when (string-equal (cdr error-thrown) "exited abnormally with code 7\n")
>                     (warn "kiwix.el failed to connect to host. exited abnormally with status code: 7."))))
>         :success (cl-function
> -                (lambda (&key data &allow-other-keys)
> +                (lambda (&key _data &allow-other-keys) ;FIXME: Why not `&rest _'?
For later success message output.
>                     (setq kiwix-server-available? t)))
>         :status-code '((404 . (lambda (&rest _) (message (format "Endpoint %s does not exist." url))))
>                        (500 . (lambda (&rest _) (message (format "Error from  %s." url))))))))
> @@ -290,12 +279,12 @@ list and return a list result."
>             (mapcar 'cdar data)))))
>   
>   ;;;###autoload
> -(defun kiwix-at-point (&optional interactively)
> +(defun kiwix-at-point ()
>     "Search for the symbol at point with `kiwix-query'.
>   
>   Or When prefix argument `INTERACTIVELY' specified, then prompt
>   for query string and library interactively."
> -  (interactive "P")
> +  (interactive)
This is necessary for later command definition `kiwix-at-point-interactive`.
>     (unless (kiwix-ping-server)
>       (kiwix-launch-server))
>     (if kiwix-server-available?
> @@ -368,7 +357,6 @@ for query string and library interactively."
>     :require 'kiwix-mode
>     :init-value nil
>     :lighter " Kiwix"
> -  :group 'kiwix-mode
>     :keymap kiwix-mode-map
>     (if kiwix-mode (kiwix-mode-enable) (kiwix-mode-disable)))
>   
>
Same as upper.

Re: How to contribute new package to GNU ELPA?

[Stefan Monnier]

Hi 

>> -;; Package-Requires: ((emacs "24.4") (cl-lib "0.5") (request "0.3.0"))
>> +;; Package-Requires: ((emacs "24.4") (request "0.3.0"))
>
> cl-lib should be required for `cl-function`.

cl-lib comes bundles with Emacs>=24.3, so your Emacs-24.4 dependency
already ensures cl-lib is available.

>>     ;; This file is part of GNU Emacs.
>>   @@ -28,13 +28,13 @@
>>     ;;; Commentary:
>>   -;;; This currently only works for Linux, not tested for Mac OS
>> X and Windows.
>> +;; This currently only works for Linux, not tested for Mac OS X and Windows.
>>   -;;; Kiwix installation
>> +;;;; Kiwix installation
>>   ;;
>>   ;; http://www.kiwix.org
>>   -;;; Config:
>> +;;;; Config:
>>   ;;
>>   ;; (use-package kiwix
>>   ;;   :ensure t
>> @@ -45,7 +45,7 @@
>>   ;;               kiwix-server-port 8080
>>   ;;               kiwix-default-library "wikipedia_zh_all_2015-11.zim"))
>>   -;;; Usage:
>> +;;;; Usage:
>
> Why use four `;` instead of three `;`? I search many packages, they all use
> three `;`. I don't know which one is the standard.

Because it's a sub-heading of "Commentary:" (e.g. we'd use five
semi-colons for subsubheadings).

>> +(if (featurep 'ivy) (require 'ivy))     ;FIXME: That's a no-op!
> What does the "no-op" mean? Because some user might have not installed ivy,
> so I added one condition to detect it here.

If (featurep 'ivy) is non-nil, then (require 'ivy) won't do anything.
I think your (featurep 'ivy) intended to test is Ivy is installed,
whereas it tests if Ivy is already loaded.  But in any case you don't
need any of that: if Ivy is installed, then `ivy-read` will be
auto-loaded, so the rest of your code will "just work" regardless if Ivy
is already loaded or not.

You might want to use something like:

    (defcustom kiwix-default-completing-read
      (cond ((fboundp 'ivy-read) 'ivy)
            ((fboundp 'helm) 'helm))

OTOH.

> Why deleted all `:group 'kiwix-mode`? If think correct, the :group is used
> by `customize-group`. So It should be necessary.

These are redundant because by default vars are placed in the group that
was last `defgroup`d.

>>   (defun kiwix-dir-detect ()
>>     "Detect Kiwix profile directory exist."
>> -  (let ((kiwix-dir (concat (getenv "HOME") "/.www.kiwix.org/kiwix")))
>> -    (if (file-accessible-directory-p kiwix-dir)
>> +  (let ((kiwix-dir "~/.www.kiwix.org/kiwix"))
>> +    (if (file-directory-p kiwix-dir)
> Use `file-accessible-directory-p` because also can detect folder
> permission. If user can't read folder, that's a problem too.

Then use something better (e.g. (and (file-directory-p kiwix-dir)
(file-readable-p kiwix-dir))), but (file-accessible-directory-p kiwix-dir)
returns non-nil when the directory is absent, in which case the user
can't read the directory either, so it's not testing what you want.

> I prefer to use `$HOME` environment variable, it should be more
> cross-platformed.

Fine by me.  I just thought it was an "obvious simplification".

Actually, it's the other way around: HOME is just an env-var and its
meaning is defined by the platform (it just happens to work fine with the
currently supported platforms), whereas "~/" is defined by Emacs so
we (Emacs maintainers) have to make sure it works on all platforms.  ;-)

>>         :success (cl-function
>> -                (lambda (&key data &allow-other-keys)
>> +                (lambda (&key _data &allow-other-keys) ;FIXME: Why not `&rest _'?
> For later success message output.

But that's another function, so it's unrelated, AFAICT.
So, I guess what you mean is that you put it for documentation purposes,
to remind the reader what args are passed to the success function (and
ignored in this particular success function).
FWIW, I find this to be a perfectly good reason (even thought it costs
a bit extra because the `cl-lib` will actually "work" to extract the `data`
argument even though it's not used).

>> -(defun kiwix-at-point (&optional interactively)
>> +(defun kiwix-at-point ()
>>     "Search for the symbol at point with `kiwix-query'.
>>     Or When prefix argument `INTERACTIVELY' specified, then prompt
>>   for query string and library interactively."
>> -  (interactive "P")
>> +  (interactive)
> This is necessary for later command definition `kiwix-at-point-interactive`.

No it's not.  What this did is add an `interactively` argument and pass
it the value of `current-prefix-arg`.  Since you don't use the variable
`interactively`, you don't need that: the variable `current-prefix-arg`
is not affected by the above changes.

This said, maybe I'm missing something: I don't see any place where you
implement the "When prefix argument `INTERACTIVELY' specified" test,
instead it seems your code just always "prompts for query string and
library interactively".

More importantly: what do you want to do about `request`?


        Stefan

Re: How to contribute new package to GNU ELPA?

[stardiviner]

[-- Attachment #1: Type: text/plain, Size: 6143 bytes --]

Thanks for your detailed answering. @Stefan Monnier
<monnier@iro.umontreal.ca>  It's really helpful! I tried to give up after
first email. :smile:

[stardiviner]           <Hack this world!>      GPG key ID: 47C32433
IRC(freeenode): stardiviner                     Twitter:  @numbchild
Key fingerprint = 9BAA 92BC CDDD B9EF 3B36  CB99 B8C4 B8E5 47C3 2433
Blog: http://stardiviner.github.io/


On Sat, Dec 19, 2020 at 11:35 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> Hi
>
> >> -;; Package-Requires: ((emacs "24.4") (cl-lib "0.5") (request "0.3.0"))
> >> +;; Package-Requires: ((emacs "24.4") (request "0.3.0"))
> >
> > cl-lib should be required for `cl-function`.
>
> cl-lib comes bundles with Emacs>=24.3, so your Emacs-24.4 dependency
> already ensures cl-lib is available.
>
Applied.

>
> >>     ;; This file is part of GNU Emacs.
> >>   @@ -28,13 +28,13 @@
> >>     ;;; Commentary:
> >>   -;;; This currently only works for Linux, not tested for Mac OS
> >> X and Windows.
> >> +;; This currently only works for Linux, not tested for Mac OS X and
> Windows.
> >>   -;;; Kiwix installation
> >> +;;;; Kiwix installation
> >>   ;;
> >>   ;; http://www.kiwix.org
> >>   -;;; Config:
> >> +;;;; Config:
> >>   ;;
> >>   ;; (use-package kiwix
> >>   ;;   :ensure t
> >> @@ -45,7 +45,7 @@
> >>   ;;               kiwix-server-port 8080
> >>   ;;               kiwix-default-library
> "wikipedia_zh_all_2015-11.zim"))
> >>   -;;; Usage:
> >> +;;;; Usage:
> >
> > Why use four `;` instead of three `;`? I search many packages, they all
> use
> > three `;`. I don't know which one is the standard.
>
Applied.

>
> Because it's a sub-heading of "Commentary:" (e.g. we'd use five
> semi-colons for subsubheadings).
>
> >> +(if (featurep 'ivy) (require 'ivy))     ;FIXME: That's a no-op!
> > What does the "no-op" mean? Because some user might have not installed
> ivy,
> > so I added one condition to detect it here.
>
> If (featurep 'ivy) is non-nil, then (require 'ivy) won't do anything.
> I think your (featurep 'ivy) intended to test is Ivy is installed,
> whereas it tests if Ivy is already loaded.  But in any case you don't
> need any of that: if Ivy is installed, then `ivy-read` will be
> auto-loaded, so the rest of your code will "just work" regardless if Ivy
> is already loaded or not.
>
> You might want to use something like:
>
>     (defcustom kiwix-default-completing-read
>       (cond ((fboundp 'ivy-read) 'ivy)
>             ((fboundp 'helm) 'helm))
>
> OTOH.
>
> Applied.

> > Why deleted all `:group 'kiwix-mode`? If think correct, the :group is
> used
> > by `customize-group`. So It should be necessary.
>
> These are redundant because by default vars are placed in the group that
> was last `defgroup`d.
>
> >>   (defun kiwix-dir-detect ()
> >>     "Detect Kiwix profile directory exist."
> >> -  (let ((kiwix-dir (concat (getenv "HOME") "/.www.kiwix.org/kiwix")))
> >> -    (if (file-accessible-directory-p kiwix-dir)
> >> +  (let ((kiwix-dir "~/.www.kiwix.org/kiwix"))
> >> +    (if (file-directory-p kiwix-dir)
> > Use `file-accessible-directory-p` because also can detect folder
> > permission. If user can't read folder, that's a problem too.
>
> Then use something better (e.g. (and (file-directory-p kiwix-dir)
> (file-readable-p kiwix-dir))), but (file-accessible-directory-p kiwix-dir)
> returns non-nil when the directory is absent, in which case the user
> can't read the directory either, so it's not testing what you want.
>
> > I prefer to use `$HOME` environment variable, it should be more
> > cross-platformed.
>
> Fine by me.  I just thought it was an "obvious simplification".
>
> Actually, it's the other way around: HOME is just an env-var and its
> meaning is defined by the platform (it just happens to work fine with the
> currently supported platforms), whereas "~/" is defined by Emacs so
> we (Emacs maintainers) have to make sure it works on all platforms.  ;-)
>
Applied.

>
> >>         :success (cl-function
> >> -                (lambda (&key data &allow-other-keys)
> >> +                (lambda (&key _data &allow-other-keys) ;FIXME: Why not
> `&rest _'?
> > For later success message output.
>
> But that's another function, so it's unrelated, AFAICT.
> So, I guess what you mean is that you put it for documentation purposes,
> to remind the reader what args are passed to the success function (and
> ignored in this particular success function).
> FWIW, I find this to be a perfectly good reason (even thought it costs
> a bit extra because the `cl-lib` will actually "work" to extract the `data`
> argument even though it's not used).
>
Hmm, let's keep this. I just applied "_data" to hint it's not used.

>
> >> -(defun kiwix-at-point (&optional interactively)
> >> +(defun kiwix-at-point ()
> >>     "Search for the symbol at point with `kiwix-query'.
> >>     Or When prefix argument `INTERACTIVELY' specified, then prompt
> >>   for query string and library interactively."
> >> -  (interactive "P")
> >> +  (interactive)
> > This is necessary for later command definition
> `kiwix-at-point-interactive`.
>
> No it's not.  What this did is add an `interactively` argument and pass
> it the value of `current-prefix-arg`.  Since you don't use the variable
> `interactively`, you don't need that: the variable `current-prefix-arg`
> is not affected by the above changes.
>
> This said, maybe I'm missing something: I don't see any place where you
> implement the "When prefix argument `INTERACTIVELY' specified" test,
> instead it seems your code just always "prompts for query string and
> library interactively".
>

Applied too. I removed command `kiwix-at-point-interactive`.

> More importantly: what do you want to do about `request`?
>
> About this, I really suggest ELPA can include it. Because I hate to write
more complex url requests with `url.el`.
If someone want to adopt kiwix.el request to use `url.el`. I also can't
accept it. Because I still need to write code on it.
Also as you said, a wrapper on `url.el` is good for developer. Emacs will
need it. Really helpful.

At the end, thanks for your patient kind help. Thanks very much!

>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 10306 bytes --]

Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Adam Porter]

stardiviner <numbchild@gmail.com> writes:

>  More importantly: what do you want to do about `request`?
>
> About this, I really suggest ELPA can include it. Because I hate to
> write more complex url requests with `url.el`.  If someone want to
> adopt kiwix.el request to use `url.el`. I also can't accept
> it. Because I still need to write code on it.  Also as you said, a
> wrapper on `url.el` is good for developer. Emacs will need it. Really
> helpful.

Emacs definitely needs an alternative to url.el.  It has some issues
that are not easily solved nor worked around, such as potentially
calling callbacks multiple times[0], requests not timing out, and some
of its API being in the form of undocumented variables that must be
let-bound around calls to its functions.

I wrote a macro that wraps `url-retrieve' to help alleviate some of the
issues.[1]  Using it in matrix-client.el seems to make its use of url.el
much more reliable (requests time out properly, etc).  But I don't know
if it would be suitable for general use, and it basically "papers over"
some of these issues.

request.el is a widely used library, and it works well for many things.
However, it has its own not-easily-resolved issues that, IME, forced me
to use url.el in the end--except for the things which url.el couldn't
handle properly, such as some file uploads, which forced me to use
request.el for those.  Having to use two different HTTP libraries in
matrix-client.el is...not ideal.

Besides that, I doubt that request.el could be easily merged into ELPA
since, apparently, not all of its contributors have done FSF copyright
assignment.  An issue that was raised about merging it into ELPA was
closed without discussion or consideration.[2]  And that leads to another
concern, one I raised about its current governance, which has been
ignored for a year now.[3]

As Chris Wellons noted in some discussions I've had with him[4], it's
relatively simple to write a Curl wrapper in Elisp that precludes having
to use url.el.  He did this for Elfeed, and it works very well.
However, quite understandably, he isn't interested in factoring it out
and maintaining it as a library.

So in Ement.el[5], I've been using a simple Curl wrapper library that
I've started.[6]  It's rather primitive, but it seems to work well.
It's relatively easy to use, IMHO.  I haven't written documentation yet
(other than docstrings), but you can see how it's used in the tests.[7]
If it were useful, I'd be glad to contribute it to ELPA.

Of course, it would be ideal if Emacs had a built-in HTTP library that
didn't require an external dependency like Curl, but Curl does work
well.  The only issue I have with it is that certain data can only be
passed via either the command line (leaving it visible to the whole
system) or temporary files (which have to be carefully handled), which
is a concern for, e.g. cookies, authentication tokens, passwords, etc.
Unfortunately, when that issue was raised and a solution proposed in
2003, it was quickly rejected.[8]

It's primarily that issue that has left me hesitant to advertise plz.el,
because I wouldn't want to recommend that anyone use it for sensitive
applications, even if the risk is probably minimal on one's personal
system.  Nevertheless, that issue remains in request.el when it uses its
Curl backend, so it's not worse in that respect.

Well, food for thought.  :)

Adam

0: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20159
1: https://github.com/alphapapa/elexandria/blob/83a1b08d0711fdce07a5b33525535cc3a457c6ee/elexandria.el#L429
2: https://github.com/tkf/emacs-request/issues/112
3: https://github.com/tkf/emacs-request/issues/172
4: https://github.com/skeeto/elisp-latch/issues/1#issuecomment-397910988
5: https://github.com/alphapapa/ement.el
6: https://github.com/alphapapa/plz.el
7: https://github.com/alphapapa/plz.el/blob/master/tests/test-plz.el
8: https://curl.se/mail/archive-2003-08/0099.html

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Daniel Brooks]

Adam Porter <adam@alphapapa.net> writes:

> Of course, it would be ideal if Emacs had a built-in HTTP library that
> didn't require an external dependency like Curl, but Curl does work
> well.  The only issue I have with it is that certain data can only be
> passed via either the command line (leaving it visible to the whole
> system) or temporary files (which have to be carefully handled), which
> is a concern for, e.g. cookies, authentication tokens, passwords, etc.
> Unfortunately, when that issue was raised and a solution proposed in
> 2003, it was quickly rejected.[8]
>
> 8: https://curl.se/mail/archive-2003-08/0099.html

For a built-in library, it would be better to use libcurl. This ensures
that there won't be any complications from hiding this type of
information, and the curl command-line program is just a simple wrapper
around libcurl anyway. In fact, even an installable library could
include libcurl in a dynamically-loadable module.

db48x

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[David Engster]

> stardiviner <numbchild@gmail.com> writes:
>
>>  More importantly: what do you want to do about `request`?
>>
>> About this, I really suggest ELPA can include it. Because I hate to
>> write more complex url requests with `url.el`.  If someone want to
>> adopt kiwix.el request to use `url.el`. I also can't accept
>> it. Because I still need to write code on it.  Also as you said, a
>> wrapper on `url.el` is good for developer. Emacs will need it. Really
>> helpful.
>
> Emacs definitely needs an alternative to url.el.

See

https://lists.gnu.org/archive/html/emacs-devel/2016-12/msg01070.html

Lars has pushed this work as scratch/with-url and
scratch/with-fetched-url, but AFAICS it wasn't merged (yet?).

-David

Re: How to contribute new package to GNU ELPA?

[Stefan Monnier]

>> More importantly: what do you want to do about `request`?
>>
> About this, I really suggest ELPA can include it.

I'm glad we agree.  AFAICT the main contributor is Takafumi Arakaki who
hasn't signed the copyright paperwork.  Any chance you could contact him
and ask him to sign that paperwork?

AFAICT `request` is a fairly popular library so it would be very nice to
have it in GNU ELPA.


        Stefan

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Stefan Monnier]

> Emacs definitely needs an alternative to url.el.  It has some issues
> that are not easily solved nor worked around, such as potentially
> calling callbacks multiple times[0], requests not timing out, and some
> of its API being in the form of undocumented variables that must be
> let-bound around calls to its functions.

We should improve the API, indeed.  Would you be interested in working
on that?

Extra bonus points if we can write a simple "`request` emulation layer"
on top of it ;-)

We should also fix the bugs: the multiple callbacks issue shouldn't be
hard to trackdown and fix.

> So in Ement.el[5], I've been using a simple Curl wrapper library that
> I've started.[6]  It's rather primitive, but it seems to work well.

I think it's important for Emacs to have a good HTTP API that works
without any external dependency.  It (technically) could be in any of
the forms:
- Including `curl` in Emacs' tarball.
- Linking to some more-or-less standard HTTP library and exposing some
  API to ELisp.
- Write it all in ELisp (like url.el does).

If it can be done by improving url.el and growing a new API on top of
it, that will be less painful.


        Stefan

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-12-20 17:37]:
> > Emacs definitely needs an alternative to url.el.  It has some issues
> > that are not easily solved nor worked around, such as potentially
> > calling callbacks multiple times[0], requests not timing out, and some
> > of its API being in the form of undocumented variables that must be
> > let-bound around calls to its functions.
> 
> We should improve the API, indeed.  Would you be interested in working
> on that?
> 
> Extra bonus points if we can write a simple "`request` emulation layer"
> on top of it ;-)
> 
> We should also fix the bugs: the multiple callbacks issue shouldn't be
> hard to trackdown and fix.
> 
> > So in Ement.el[5], I've been using a simple Curl wrapper library that
> > I've started.[6]  It's rather primitive, but it seems to work well.
> 
> I think it's important for Emacs to have a good HTTP API that works
> without any external dependency.  It (technically) could be in any of
> the forms:
> - Including `curl` in Emacs' tarball.
> - Linking to some more-or-less standard HTTP library and exposing some
>   API to ELisp.
> - Write it all in ELisp (like url.el does).
> 
> If it can be done by improving url.el and growing a new API on top of
> it, that will be less painful.

Please see if this library could maybe help:
https://gnunet.org/en/gnurl.html

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Helmut Eller]

On Sun, Dec 20 2020, Stefan Monnier wrote:

>> Emacs definitely needs an alternative to url.el.  It has some issues
>> that are not easily solved nor worked around, such as potentially
>> calling callbacks multiple times[0], requests not timing out, and some
>> of its API being in the form of undocumented variables that must be
>> let-bound around calls to its functions.
>
> We should improve the API, indeed.  Would you be interested in working
> on that?
>
> Extra bonus points if we can write a simple "`request` emulation layer"
> on top of it ;-)
>
> We should also fix the bugs: the multiple callbacks issue shouldn't be
> hard to trackdown and fix.

I think it's worth looking at the fetch API[1] for inspiration.
Especially using Javascript-like promises consistently would help.  Many
people will be familiar with the Javascript API and it's just a lot
easier to steal a (non-perfect) design than to come up with something
better.

Helmut

[1] https://fetch.spec.whatwg.org/#fetch-api

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Daniel Brooks]

Helmut Eller <eller.helmut@gmail.com> writes:

> I think it's worth looking at the fetch API[1] for inspiration.
> Especially using Javascript-like promises consistently would help.  Many
> people will be familiar with the Javascript API and it's just a lot
> easier to steal a (non-perfect) design than to come up with something
> better.
>
> Helmut
>
> [1] https://fetch.spec.whatwg.org/#fetch-api

That is a good idea; the fetch API is quite nice. The URL[1] and
URLSearchParams[2] APIs are also really important; they make it
unnecessary to ever parse urls or build urls up through string
concatenation.

[1] https://developer.mozilla.org/en-US/docs/Web/API/URL
[2] https://developer.mozilla.org/en-US/docs/Web/API/URLSearchParams

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Lars Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> Lars has pushed this work as scratch/with-url and
> scratch/with-fetched-url, but AFAICS it wasn't merged (yet?).

Yeah, I keep meaning to get around to merging it...  One of these days.
:-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Would people please tell me what new capabilities you aim to have
in this new facility?  How would it be better than using url.el
or running curl as a subprocess?

What jobs do you think of using it for?

I'm concerned that it may add a substantial amount of complexity
that we could do without.  I'm also concerned about how it might
deal with Javascript code.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: How to contribute new package to GNU ELPA?

[stardiviner]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

That's great.
I see, I will try to contact him, as soon as he replied, I will notify you
by update here.

[stardiviner]           <Hack this world!>      GPG key ID: 47C32433
IRC(freeenode): stardiviner                     Twitter:  @numbchild
Key fingerprint = 9BAA 92BC CDDD B9EF 3B36  CB99 B8C4 B8E5 47C3 2433
Blog: http://stardiviner.github.io/


On Sun, Dec 20, 2020 at 10:18 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> >> More importantly: what do you want to do about `request`?
> >>
> > About this, I really suggest ELPA can include it.
>
> I'm glad we agree.  AFAICT the main contributor is Takafumi Arakaki who
> hasn't signed the copyright paperwork.  Any chance you could contact him
> and ask him to sign that paperwork?
>
> AFAICT `request` is a fairly popular library so it would be very nice to
> have it in GNU ELPA.
>
>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 1744 bytes --]

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Stefan Monnier]

> I'm concerned that it may add a substantial amount of complexity
> that we could do without.  I'm also concerned about how it might
> deal with Javascript code.

AFAIK Javascript has not yet crept into HTTP, so this is not an issue.


        Stefan

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Would people please tell me what new capabilities you aim to have
> in this new facility?  How would it be better than using url.el
> or running curl as a subprocess?
>
> What jobs do you think of using it for?
>
> I'm concerned that it may add a substantial amount of complexity
> that we could do without.  I'm also concerned about how it might
> deal with Javascript code.

I think the main issue is that url.el can often be rather slow,
sometimes taking half a minute or more to complete a request that would
take curl or other libraries a blink of an eye.

If curl or whatever other method could be used to accelerate url.el, a
lot of application in Emacs would become more feasible to use, without
having to rewrite anything.

Request.el is another story, and I can't comment on that. It's probably
just like with dash/s/f/... where non-elisp developers dislike the
built-in API, but I'm just guessing.

-- 
	Philip K.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: "Philip K." <philipk@posteo.net>
> Date: Mon, 21 Dec 2020 17:59:00 +0100
> Cc: Adam Porter <adam@alphapapa.net>, emacs-devel@gnu.org
> 
> I think the main issue is that url.el can often be rather slow,
> sometimes taking half a minute or more to complete a request that would
> take curl or other libraries a blink of an eye.

Before this is taken as an indication that using one of these
libraries will automagically make Emacs as fast as the applications
like curl, we should carefully profile url.el and find out which
part(s) of it cause the slowness.  Because it could well be that what
makes url.el slow will also make Emacs using libcurl slow.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

> Before this is taken as an indication that using one of these
> libraries will automagically make Emacs as fast as the applications
> like curl, we should carefully profile url.el and find out which
> part(s) of it cause the slowness.  Because it could well be that what
> makes url.el slow will also make Emacs using libcurl slow.

Maybe this library can be of help while testing/profiling

https://tkf.github.io/emacs-request/

"Request.el is a HTTP request library with multiple backends. It supports url.el which is shipped with Emacs and curl command line program. User can use curl when s/he has it, as curl is more reliable than url.el. Library author can use request.el to avoid imposing external dependencies such as curl to users while giving richer experience for users who have curl."

https://github.com/tkf/emacs-request

Sorry for the github link; but it can be just cloned without running
their js.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Mon, 21 Dec 2020 18:41:02 +0100
> Cc: adam@alphapapa.net, "Philip K." <philipk@posteo.net>, rms@gnu.org,
>  emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Before this is taken as an indication that using one of these
> > libraries will automagically make Emacs as fast as the applications
> > like curl, we should carefully profile url.el and find out which
> > part(s) of it cause the slowness.  Because it could well be that what
> > makes url.el slow will also make Emacs using libcurl slow.
> 
> Maybe this library can be of help while testing/profiling

We have a built-in profiler in Emacs.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Mon, 21 Dec 2020 18:41:02 +0100
>> Cc: adam@alphapapa.net, "Philip K." <philipk@posteo.net>, rms@gnu.org,
>>  emacs-devel@gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Before this is taken as an indication that using one of these
>> > libraries will automagically make Emacs as fast as the applications
>> > like curl, we should carefully profile url.el and find out which
>> > part(s) of it cause the slowness.  Because it could well be that what
>> > makes url.el slow will also make Emacs using libcurl slow.
>> 
>> Maybe this library can be of help while testing/profiling
>
> We have a built-in profiler in Emacs.

It's a nice and easy wrapper around curl, you can call built-in profiler on.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Mon, 21 Dec 2020 18:41:02 +0100
>> Cc: adam@alphapapa.net, "Philip K." <philipk@posteo.net>, rms@gnu.org,
>>  emacs-devel@gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Before this is taken as an indication that using one of these
>> > libraries will automagically make Emacs as fast as the applications
>> > like curl, we should carefully profile url.el and find out which
>> > part(s) of it cause the slowness.  Because it could well be that what
>> > makes url.el slow will also make Emacs using libcurl slow.
>> 
>> Maybe this library can be of help while testing/profiling
>
> We have a built-in profiler in Emacs.

In that case, gnutls-negotiate seems to be the most suspicious function,
using over 50%-60% of the CPU time, at least on my machine. This also
makes sense, as TLS sites seem to take longer to load than regular,
non-encrypted ones.

-- 
	Philip K.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Lars Ingebrigtsen]

"Philip K." <philipk@posteo.net> writes:

> In that case, gnutls-negotiate seems to be the most suspicious function,
> using over 50%-60% of the CPU time, at least on my machine.

I haven't checked, but I seem to remember there being a busy-wait in
gnutls-negotiate somewhere?

> This also makes sense, as TLS sites seem to take longer to load than
> regular, non-encrypted ones.

TLS sites have more round trips than non-TLS sites (and there's more
data to transfer), so that's natural for smaller documents.  If the
document being pulled down is large, though, there shouldn't be a big
time differential (if nothing's broken in our code).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: "Philip K." <philipk@posteo.net>
> Cc: arthur.miller@live.com, adam@alphapapa.net, rms@gnu.org,
> 	emacs-devel@gnu.org
> Date: Tue, 22 Dec 2020 00:51:08 +0100
> 
> >> > Before this is taken as an indication that using one of these
> >> > libraries will automagically make Emacs as fast as the applications
> >> > like curl, we should carefully profile url.el and find out which
> >> > part(s) of it cause the slowness.  Because it could well be that what
> >> > makes url.el slow will also make Emacs using libcurl slow.
> >> 
> >> Maybe this library can be of help while testing/profiling
> >
> > We have a built-in profiler in Emacs.
> 
> In that case, gnutls-negotiate seems to be the most suspicious function,
> using over 50%-60% of the CPU time, at least on my machine. This also
> makes sense, as TLS sites seem to take longer to load than regular,
> non-encrypted ones.

Please show the code you profiled and the fully expanded profile.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I'm concerned that it may add a substantial amount of complexity
  > > that we could do without.  I'm also concerned about how it might
  > > deal with Javascript code.

  > AFAIK Javascript has not yet crept into HTTP, so this is not an issue.

Javascript is potentially an issue.  HTTP typically transfers HTML,
and where HTML goes, Javascript tends to go with it.  So this program
may cause pressure to support Javascript in some way.

Or it may not.  But we need to look a few steps down this road before
setting out on it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think the main issue is that url.el can often be rather slow,
  > sometimes taking half a minute or more to complete a request that would
  > take curl or other libraries a blink of an eye.

  > If curl or whatever other method could be used to accelerate url.el, a
  > lot of application in Emacs would become more feasible to use, without
  > having to rewrite anything.

I see.  No problem in making that faster.

  > Request.el is another story, and I can't comment on that. It's probably
  > just like with dash/s/f/... where non-elisp developers dislike the
  > built-in API, but I'm just guessing.

From what people have said here, it sounds unproblematical.
But could you explain what you mean by comparing it with dash/s/f/?
Would someone please tell me what names the entry points of request.el have?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I think the main issue is that url.el can often be rather slow,
>   > sometimes taking half a minute or more to complete a request that would
>   > take curl or other libraries a blink of an eye.
>
>   > If curl or whatever other method could be used to accelerate url.el, a
>   > lot of application in Emacs would become more feasible to use, without
>   > having to rewrite anything.
>
> I see.  No problem in making that faster.
>
>   > Request.el is another story, and I can't comment on that. It's probably
>   > just like with dash/s/f/... where non-elisp developers dislike the
>   > built-in API, but I'm just guessing.
>
> From what people have said here, it sounds unproblematical.
> But could you explain what you mean by comparing it with dash/s/f/?
> Would someone please tell me what names the entry points of request.el have?

[-- Attachment #2: request.el --]
[-- Type: text/plain, Size: 53672 bytes --]

;;; request.el --- Compatible layer for URL request in Emacs -*- lexical-binding: t; -*-

;; Copyright (C) 2012 Takafumi Arakaki
;; Copyright (C) 1985-1986, 1992, 1994-1995, 1999-2012
;;   Free Software Foundation, Inc.

;; Author: Takafumi Arakaki <aka.tkf at gmail.com>
;; URL: https://github.com/tkf/emacs-request
;; Package-Version: 20201026.2324
;; Package-Commit: 0183da84cb45eb94da996cd2eab714ef0d7504cc
;; Package-Requires: ((emacs "24.4"))
;; Version: 0.3.2

;; This file is NOT part of GNU Emacs.

;; request.el is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; request.el is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with request.el.
;; If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; Request.el is a HTTP request library with multiple backends.  It
;; supports url.el which is shipped with Emacs and curl command line
;; program.  User can use curl when s/he has it, as curl is more reliable
;; than url.el.  Library author can use request.el to avoid imposing
;; external dependencies such as curl to users while giving richer
;; experience for users who have curl.

;; Following functions are adapted from GNU Emacs source code.
;; Free Software Foundation holds the copyright of them.
;; * `request--process-live-p'
;; * `request--url-default-expander'

;;; Code:

(eval-when-compile
  (defvar url-http-method)
  (defvar url-http-response-status))

(require 'cl-lib)
(require 'url)
(require 'mail-utils)
(require 'autorevert)
(require 'auth-source)

(defgroup request nil
  "Compatible layer for URL request in Emacs."
  :group 'comm
  :prefix "request-")

(defconst request-version "0.3.0")

\f
;;; Customize variables

(defcustom request-storage-directory
  (concat (file-name-as-directory user-emacs-directory) "request")
  "Directory to store data related to request.el."
  :type 'directory)

(defcustom request-curl "curl"
  "Executable for curl command."
  :type 'string)

(defcustom request-curl-options nil
  "curl command options.

List of strings that will be passed to every curl invocation. You can pass
extra options here, like setting the proxy."
  :type '(repeat string))

(defcustom request-backend (if (executable-find request-curl)
                               'curl
                             'url-retrieve)
  "Backend to be used for HTTP request.
Automatically set to `curl' if curl command is found."
  :type '(choice (const :tag "cURL backend" curl)
                 (const :tag "url-retrieve backend" url-retrieve)))

(defcustom request-timeout nil
  "Default request timeout in second.
`nil' means no timeout."
  :type '(choice (integer :tag "Request timeout seconds")
                 (boolean :tag "No timeout" nil)))

(defcustom request-temp-prefix "emacs-request"
  "Prefix for temporary files created by Request."
  :type 'string
  :risky t)

(defcustom request-log-level -1
  "Logging level for request.
One of `error'/`warn'/`info'/`verbose'/`debug'/`trace'/`blather'.
-1 means no logging."
  :type '(choice (integer :tag "No logging" -1)
                 (const :tag "Level error" error)
                 (const :tag "Level warn" warn)
                 (const :tag "Level info" info)
                 (const :tag "Level Verbose" verbose)
                 (const :tag "Level DEBUG" debug)
                 (const :tag "Level TRACE" trace)
                 (const :tag "Level BLATHER" blather)))

(defcustom request-message-level 'warn
  "Logging level for request.
See `request-log-level'."
  :type '(choice (integer :tag "No logging" -1)
                 (const :tag "Level error" error)
                 (const :tag "Level warn" warn)
                 (const :tag "Level info" info)
                 (const :tag "Level Verbose" verbose)
                 (const :tag "Level DEBUG" debug)
                 (const :tag "Level TRACE" trace)
                 (const :tag "Level BLATHER" blather)))

\f
;;; Utilities

(defun request--safe-apply (function &rest arguments)
  "Apply FUNCTION with ARGUMENTS, suppressing any errors."
  (condition-case nil
      (apply #'apply function arguments)
    ((debug error))))

(defun request--safe-call (function &rest arguments)
  (request--safe-apply function arguments))

;; (defun request--url-no-cache (url)
;;   "Imitate `cache=false' of `jQuery.ajax'.
;; See: http://api.jquery.com/jQuery.ajax/"
;;   ;; FIXME: parse URL before adding ?_=TIME.
;;   (concat url (format-time-string "?_=%s")))

(defmacro request--document-function (function docstring)
  "Document FUNCTION with DOCSTRING.  Use this for defstruct accessor etc."
  (declare (indent defun)
           (doc-string 2))
  `(put ',function 'function-documentation ,docstring))

(defun request--process-live-p (process)
  "Copied from `process-live-p' for backward compatibility (Emacs < 24).
Adapted from lisp/subr.el.
FSF holds the copyright of this function:
  Copyright (C) 1985-1986, 1992, 1994-1995, 1999-2012
    Free Software Foundation, Inc."
  (memq (process-status process) '(run open listen connect stop)))

\f
;;; Logging

(defconst request--log-level-def
  '(;; debugging
    (blather . 60) (trace . 50) (debug . 40)
    ;; information
    (verbose . 30) (info . 20)
    ;; errors
    (warn . 10) (error . 0))
  "Named logging levels.")

(defun request--log-level-as-int (level)
  (if (integerp level)
      level
    (or (cdr (assq level request--log-level-def))
        0)))

(defvar request-log-buffer-name " *request-log*")

(defun request--log-buffer ()
  (get-buffer-create request-log-buffer-name))

(defmacro request-log (level fmt &rest args)
  (declare (indent 1))
  `(let ((level (request--log-level-as-int ,level))
         (log-level (request--log-level-as-int request-log-level))
         (msg-level (request--log-level-as-int request-message-level)))
     (when (<= level (max log-level msg-level))
       (let ((msg (format "[%s] %s" ,level
                          (condition-case err
                              (format ,fmt ,@args)
                            (error (format "
!!! Logging error while executing:
%S
!!! Error:
%S"
                                           ',args err))))))
         (when (<= level log-level)
           (with-current-buffer (request--log-buffer)
             (setq buffer-read-only t)
             (let ((inhibit-read-only t))
               (goto-char (point-max))
               (insert msg "\n"))))
         (when (<= level msg-level)
           (message "%s" msg))))))

\f
;;; HTTP specific utilities

(defconst request--url-unreserved-chars
  '(?a ?b ?c ?d ?e ?f ?g ?h ?i ?j ?k ?l ?m ?n ?o ?p ?q ?r ?s ?t ?u ?v ?w ?x ?y ?z
    ?A ?B ?C ?D ?E ?F ?G ?H ?I ?J ?K ?L ?M ?N ?O ?P ?Q ?R ?S ?T ?U ?V ?W ?X ?Y ?Z
    ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9
    ?- ?_ ?. ?~)
  "`url-unreserved-chars' copied from Emacs 24.3 release candidate.
This is used for making `request--urlencode-alist' RFC 3986 compliant
for older Emacs versions.")

(defun request--urlencode-alist (alist)
  ;; FIXME: make monkey patching `url-unreserved-chars' optional
  (let ((url-unreserved-chars request--url-unreserved-chars))
    (cl-loop for sep = "" then "&"
             for (k . v) in alist
             concat sep
             concat (url-hexify-string (format "%s" k))
             concat "="
             concat (url-hexify-string (format "%s" v)))))

\f
;;; Header parser

(defun request--parse-response-at-point ()
  "Parse the first header line such as \"HTTP/1.1 200 OK\"."
  (when (re-search-forward "\\=[ \t\n]*HTTP/\\([0-9\\.]+\\) +\\([0-9]+\\)" nil t)
    (list :version (match-string 1)
          :code (string-to-number (match-string 2)))))

(defun request--goto-next-body (&optional noerror)
  (re-search-forward "^\r\n" nil noerror))

\f
;;; Response object

(cl-defstruct request-response
  "A structure holding all relevant information of a request."
  status-code history data error-thrown symbol-status url
  done-p settings
  ;; internal variables
  -buffer -raw-header -timer -backend -tempfiles)

(defmacro request--document-response (function docstring)
  (declare (indent defun)
           (doc-string 2))
  `(request--document-function ,function ,(concat docstring "

.. This is an accessor for `request-response' object.

\(fn RESPONSE)")))

(request--document-response request-response-status-code
  "Integer HTTP response code (e.g., 200).")

(request--document-response request-response-history
  "Redirection history (a list of response object).
The first element is the oldest redirection.

You can use restricted portion of functions for the response
objects in the history slot.  It also depends on backend.  Here
is the table showing what functions you can use for the response
objects in the history slot.

==================================== ============== ==============
Slots                                          Backends
------------------------------------ -----------------------------
\\                                    curl           url-retrieve
==================================== ============== ==============
request-response-url                  yes            yes
request-response-header               yes            no
other functions                       no             no
==================================== ============== ==============
")

(request--document-response request-response-data
  "Response parsed by the given parser.")

(request--document-response request-response-error-thrown
  "Error thrown during request.
It takes the form of ``(ERROR-SYMBOL . DATA)``, which can be
re-raised (`signal'ed) by ``(signal ERROR-SYMBOL DATA)``.")

(request--document-response request-response-symbol-status
  "A symbol representing the status of request (not HTTP response code).
One of success/error/timeout/abort/parse-error.")

(request--document-response request-response-url
  "Final URL location of response.")

(request--document-response request-response-done-p
  "Return t when the request is finished or aborted.")

(request--document-response request-response-settings
  "Keyword arguments passed to `request' function.
Some arguments such as HEADERS is changed to the one actually
passed to the backend.  Also, it has additional keywords such
as URL which is the requested URL.")

(defun request-response-header (response field-name)
  "Fetch the values of RESPONSE header field named FIELD-NAME.

It returns comma separated values when the header has multiple
field with the same name, as :RFC:`2616` specifies.

Examples::

  (request-response-header response
                           \"content-type\") ; => \"text/html; charset=utf-8\"
  (request-response-header response
                           \"unknown-field\") ; => nil
"
  (let ((raw-header (request-response--raw-header response)))
    (when raw-header
      (with-temp-buffer
        (erase-buffer)
        (insert raw-header)
        ;; ALL=t to fetch all fields with the same name to get comma
        ;; separated value [#rfc2616-sec4]_.
        (mail-fetch-field field-name nil t)))))
;; .. [#rfc2616-sec4] RFC2616 says this is the right thing to do
;;    (see http://tools.ietf.org/html/rfc2616.html#section-4.2).
;;    Python's requests module does this too.

\f
;;; Backend dispatcher

(defconst request--backend-alist
  '((url-retrieve
     . ((request             . request--url-retrieve)
        (request-sync        . request--url-retrieve-sync)
        (terminate-process   . delete-process)
        (get-cookies         . request--url-retrieve-get-cookies)))
    (curl
     . ((request             . request--curl)
        (request-sync        . request--curl-sync)
        (terminate-process   . interrupt-process)
        (get-cookies         . request--curl-get-cookies))))
  "Map backend and method name to actual method (symbol).

It's alist of alist, of the following form::

    ((BACKEND . ((METHOD . FUNCTION) ...)) ...)

It would be nicer if I can use EIEIO.  But as CEDET is included
in Emacs by 23.2, using EIEIO means abandon older Emacs versions.
It is probably necessary if I need to support more backends.  But
let's stick to manual dispatch for now.")
;; See: (view-emacs-news "23.2")

(defun request--choose-backend (method)
  "Return `fucall'able object for METHOD of current `request-backend'."
  (assoc-default
   method
   (or (assoc-default request-backend request--backend-alist)
       (error "%S is not valid `request-backend'." request-backend))))

\f
;;; Cookie

(defun request-cookie-string (host &optional localpart secure)
  "Return cookie string (like `document.cookie').

Example::

 (request-cookie-string \"127.0.0.1\" \"/\")  ; => \"key=value; key2=value2\"
"
  (mapconcat (lambda (nv) (concat (car nv) "=" (cdr nv)))
             (request-cookie-alist host localpart secure)
             "; "))

(defun request-cookie-alist (host &optional localpart secure)
  "Return cookies as an alist.

Example::

 (request-cookie-alist \"127.0.0.1\" \"/\")  ; => ((\"key\" . \"value\") ...)
"
  (funcall (request--choose-backend 'get-cookies) host localpart secure))

\f
;;; Main

(cl-defun request-default-error-callback (url &key symbol-status
                                              &allow-other-keys)
  (request-log 'error
    "request-default-error-callback: %s %s" url symbol-status))

(cl-defun request (url &rest settings
                       &key
                       (params nil)
                       (data nil)
                       (headers nil)
                       (encoding 'utf-8)
                       (error nil)
                       (sync nil)
                       (response (make-request-response))
                       &allow-other-keys)
  "Send request to URL.

Request.el has a single entry point.  It is `request'.

==================== ========================================================
Keyword argument      Explanation
==================== ========================================================
TYPE          (string)   type of request to make: POST/GET/PUT/DELETE
PARAMS         (alist)   set \"?key=val\" part in URL
DATA    (string/alist)   data to be sent to the server
FILES          (alist)   files to be sent to the server (see below)
PARSER        (symbol)   a function that reads current buffer and return data
HEADERS        (alist)   additional headers to send with the request
ENCODING      (symbol)   encoding for request body (utf-8 by default)
SUCCESS     (function)   called on success
ERROR       (function)   called on error
COMPLETE    (function)   called on both success and error
TIMEOUT       (number)   timeout in second
STATUS-CODE    (alist)   map status code (int) to callback
SYNC            (bool)   If `t', wait until request is done.  Default is `nil'.
==================== ========================================================


* Callback functions

Callback functions STATUS, ERROR, COMPLETE and `cdr's in element of
the alist STATUS-CODE take same keyword arguments listed below.  For
forward compatibility, these functions must ignore unused keyword
arguments (i.e., it's better to use `&allow-other-keys' [#]_).::

    (CALLBACK                      ; SUCCESS/ERROR/COMPLETE/STATUS-CODE
     :data          data           ; whatever PARSER function returns, or nil
     :error-thrown  error-thrown   ; (ERROR-SYMBOL . DATA), or nil
     :symbol-status symbol-status  ; success/error/timeout/abort/parse-error
     :response      response       ; request-response object
     ...)

.. [#] `&allow-other-keys' is a special \"markers\" available in macros
   in the CL library for function definition such as `cl-defun' and
   `cl-function'.  Without this marker, you need to specify all arguments
   to be passed.  This becomes problem when request.el adds new arguments
   when calling callback functions.  If you use `&allow-other-keys'
   (or manually ignore other arguments), your code is free from this
   problem.  See info node `(cl) Argument Lists' for more information.

Arguments data, error-thrown, symbol-status can be accessed by
`request-response-data', `request-response-error-thrown',
`request-response-symbol-status' accessors, i.e.::

    (request-response-data RESPONSE)  ; same as data

Response object holds other information which can be accessed by
the following accessors:
`request-response-status-code',
`request-response-url' and
`request-response-settings'

* STATUS-CODE callback

STATUS-CODE is an alist of the following format::

    ((N-1 . CALLBACK-1)
     (N-2 . CALLBACK-2)
     ...)

Here, N-1, N-2,... are integer status codes such as 200.


* FILES

FILES is an alist of the following format::

    ((NAME-1 . FILE-1)
     (NAME-2 . FILE-2)
     ...)

where FILE-N is a list of the form::

    (FILENAME &key PATH BUFFER STRING MIME-TYPE)

FILE-N can also be a string (path to the file) or a buffer object.
In that case, FILENAME is set to the file name or buffer name.

Example FILES argument::

    `((\"passwd\"   . \"/etc/passwd\")                ; filename = passwd
      (\"scratch\"  . ,(get-buffer \"*scratch*\"))    ; filename = *scratch*
      (\"passwd2\"  . (\"password.txt\" :file \"/etc/passwd\"))
      (\"scratch2\" . (\"scratch.txt\"  :buffer ,(get-buffer \"*scratch*\")))
      (\"data\"     . (\"data.csv\"     :data \"1,2,3\\n4,5,6\\n\")))

.. note:: FILES is implemented only for curl backend for now.
   As furl.el_ supports multipart POST, it should be possible to
   support FILES in pure elisp by making furl.el_ another backend.
   Contributions are welcome.

   .. _furl.el: http://code.google.com/p/furl-el/


* PARSER function

PARSER function takes no argument and it is executed in the
buffer with HTTP response body.  The current position in the HTTP
response buffer is at the beginning of the buffer.  As the HTTP
header is stripped off, the cursor is actually at the beginning
of the response body.  So, for example, you can pass `json-read'
to parse JSON object in the buffer.  To fetch whole response as a
string, pass `buffer-string'.

When using `json-read', it is useful to know that the returned
type can be modified by `json-object-type', `json-array-type',
`json-key-type', `json-false' and `json-null'.  See docstring of
each function for what it does.  For example, to convert JSON
objects to plist instead of alist, wrap `json-read' by `lambda'
like this.::

    (request
     \"http://...\"
     :parser (lambda ()
               (let ((json-object-type 'plist))
                 (json-read)))
     ...)

This is analogous to the `dataType' argument of jQuery.ajax_.
Only this function can access to the process buffer, which
is killed immediately after the execution of this function.

* SYNC

Synchronous request is functional, but *please* don't use it
other than testing or debugging.  Emacs users have better things
to do rather than waiting for HTTP request.  If you want a better
way to write callback chains, use `request-deferred'.

If you can't avoid using it (e.g., you are inside of some hook
which must return some value), make sure to set TIMEOUT to
relatively small value.

Due to limitation of `url-retrieve-synchronously', response slots
`request-response-error-thrown', `request-response-history' and
`request-response-url' are unknown (always `nil') when using
synchronous request with `url-retrieve' backend.

* Note

API of `request' is somewhat mixture of jQuery.ajax_ (Javascript)
and requests.request_ (Python).

.. _jQuery.ajax: http://api.jquery.com/jQuery.ajax/
.. _requests.request: http://docs.python-requests.org
"
  (declare (indent defun))
  ;; FIXME: support CACHE argument (if possible)
  ;; (unless cache
  ;;   (setq url (request--url-no-cache url)))
  (unless error
    (setq error (apply-partially #'request-default-error-callback url))
    (setq settings (plist-put settings :error error)))
  (unless (or (stringp data)
              (null data)
              (assoc-string "Content-Type" headers t))
    (setq data (request--urlencode-alist data))
    (setq settings (plist-put settings :data data)))
  (when params
    (cl-assert (listp params) nil "PARAMS must be an alist.  Given: %S" params)
    (setq url (concat url (if (string-match-p "\\?" url) "&" "?")
                      (request--urlencode-alist params))))
  (setq settings (plist-put settings :url url))
  (setq settings (plist-put settings :response response))
  (setq settings (plist-put settings :encoding encoding))
  (setf (request-response-settings response) settings)
  (setf (request-response-url      response) url)
  (setf (request-response--backend response) request-backend)
  ;; Call `request--url-retrieve'(`-sync') or `request--curl'(`-sync').
  (apply (if sync
             (request--choose-backend 'request-sync)
           (request--choose-backend 'request))
         url settings)
  response)

(defun request--clean-header (response)
  "Strip off carriage returns in the header of REQUEST."
  (let* ((buffer (request-response--buffer response))
         (backend (request-response--backend response))
         ;; FIXME: a workaround when `url-http-clean-headers' fails...
         (sep-regexp (if (eq backend 'url-retrieve) "^\r?$" "^\r$")))
    (when (buffer-live-p buffer)
      (with-current-buffer buffer
        (goto-char (point-min))
        (when (and (re-search-forward sep-regexp nil t)
                   (not (equal (match-string 0) "")))
          (request-log 'trace "request--clean-header: cleaning\n%s"
                       (buffer-substring (save-excursion
                                           (forward-line -1)
                                           (line-beginning-position))
                                         (save-excursion
                                           (forward-line 1)
                                           (line-end-position))))
          (while (re-search-backward "\r$" (point-min) t)
            (replace-match "")))))))

(defun request--cut-header (response)
  "Cut the first header part in the buffer of RESPONSE and move it to
raw-header slot."
  (let ((buffer (request-response--buffer response)))
    (when (buffer-live-p buffer)
      (with-current-buffer buffer
        (goto-char (point-min))
        (when (re-search-forward "^$" nil t)
          (setf (request-response--raw-header response)
                (buffer-substring (point-min) (point)))
          (request-log 'trace "request--cut-header: cutting\n%s"
                       (buffer-substring (point-min) (min (1+ (point)) (point-max))))
          (delete-region (point-min) (min (1+ (point)) (point-max))))))))

(defun request-untrampify-filename (file)
  "Return FILE as the local file name."
  (or (file-remote-p file 'localname) file))

(defun request--parse-data (response encoding parser)
  "For buffer held by RESPONSE, first decode via user's ENCODING elective,
then send to PARSER."
  (let ((buffer (request-response--buffer response)))
    (when (buffer-live-p buffer)
      (with-current-buffer buffer
        (request-log 'trace "request--parse-data: %s" (buffer-string))
        (unless (eq (request-response-status-code response) 204)
          (recode-region (point-min) (point-max) encoding 'no-conversion)
          (goto-char (point-min))
          (setf (request-response-data response)
                (if parser (funcall parser) (buffer-string))))))))

(defsubst request-url-file-p (url)
  "Return non-nil if URL looks like a file URL."
  (let ((scheme (and (stringp url) (url-type (url-generic-parse-url url)))))
    (and (stringp scheme)
         (not (string-match-p "^http" scheme)))))

(cl-defun request--callback (buffer
                             &key
                             parser success error complete
                             status-code response
                             encoding
                             &allow-other-keys)
  (request-log 'debug "request--callback: UNPARSED\n%s"
               (when (buffer-live-p buffer)
                 (with-current-buffer buffer (buffer-string))))

  ;; Sometimes BUFFER given as the argument is different from the
  ;; buffer already set in RESPONSE.  That's why it is reset here.
  ;; FIXME: Refactor how BUFFER is passed around.
  (setf (request-response--buffer response) buffer)
  (request-response--cancel-timer response)
  (cl-symbol-macrolet
      ((error-thrown (request-response-error-thrown response))
       (symbol-status (request-response-symbol-status response))
       (data (request-response-data response))
       (done-p (request-response-done-p response)))
    (let* ((response-url (request-response-url response))
           (curl-file-p (and (eq (request-response--backend response) 'curl)
                             (request-url-file-p response-url))))
      (unless curl-file-p
        (request--clean-header response)
        (request--cut-header response)))

    ;; Parse response even if `error-thrown' is set, e.g., timeout
    (condition-case err
        (request--parse-data response encoding parser)
      (error (unless error-thrown (setq error-thrown err))
             (unless symbol-status (setq symbol-status 'parse-error))))
    (kill-buffer buffer)

    ;; Ensuring `symbol-status' and `error-thrown' are consistent
    ;; is why we should get rid of `symbol-status'
    ;; (but downstream apps might ill-advisedly rely on it).
    (if error-thrown
        (progn
          (request-log 'error "request--callback: %s"
                       (error-message-string error-thrown))
          (unless symbol-status (setq symbol-status 'error)))
      (unless symbol-status (setq symbol-status 'success))
      (request-log 'debug "request--callback: PARSED\n%s" data))

    (let ((args (list :data data
                      :symbol-status symbol-status
                      :error-thrown error-thrown
                      :response response)))
      (let* ((success-p (eq symbol-status 'success))
             (cb (if success-p success error))
             (name (if success-p "success" "error")))
        (when cb
          (request-log 'debug "request--callback: executing %s" name)
          (request--safe-apply cb args)))

      (let ((cb (cdr (assq (request-response-status-code response)
                           status-code))))
        (when cb
          (request-log 'debug "request--callback: executing status-code")
          (request--safe-apply cb args)))

      (when complete
        (request-log 'debug "request--callback: executing complete")
        (request--safe-apply complete args)))

    (setq done-p t)

    ;; Remove temporary files
    ;; FIXME: Make tempfile cleanup more reliable.  It is possible
    ;;        callback is never called.
    (request--safe-delete-files (request-response--tempfiles response))))

(cl-defun request-response--timeout-callback (response)
  (setf (request-response-symbol-status response) 'timeout)
  (setf (request-response-error-thrown response)  '(error . ("Timeout")))
  (let* ((buffer (request-response--buffer response))
         (proc (and (buffer-live-p buffer) (get-buffer-process buffer))))
    (if proc
        ;; This will call `request--callback':
        (funcall (request--choose-backend 'terminate-process) proc)
      (cl-symbol-macrolet ((done-p (request-response-done-p response)))
        (unless done-p
          (when (buffer-live-p buffer)
            (cl-destructuring-bind (&key code &allow-other-keys)
                (with-current-buffer buffer
                  (goto-char (point-min))
                  (request--parse-response-at-point))
              (setf (request-response-status-code response) code)))
          (apply #'request--callback
                 buffer
                 (request-response-settings response))
          (setq done-p t))))))

(defun request-response--cancel-timer (response)
  (cl-symbol-macrolet ((timer (request-response--timer response)))
    (when timer
      (cancel-timer timer)
      (setq timer nil))))


(defun request-abort (response)
  "Abort request for RESPONSE (the object returned by `request').
Note that this function invoke ERROR and COMPLETE callbacks.
Callbacks may not be called immediately but called later when
associated process is exited."
  (cl-symbol-macrolet ((buffer (request-response--buffer response))
                       (symbol-status (request-response-symbol-status response))
                       (done-p (request-response-done-p response)))
    (let ((process (get-buffer-process buffer)))
      (unless symbol-status             ; should I use done-p here?
        (setq symbol-status 'abort)
        (setq done-p t)
        (when (and
               (processp process) ; process can be nil when buffer is killed
               (request--process-live-p process))
          (funcall (request--choose-backend 'terminate-process) process))))))

\f
;;; Backend: `url-retrieve'

(cl-defun request--url-retrieve-preprocess-settings
    (&rest settings &key type data files headers &allow-other-keys)
  (when files
    (error "`url-retrieve' backend does not support FILES."))
  (when (and (equal type "POST")
             data
             (not (assoc-string "Content-Type" headers t)))
    (push '("Content-Type" . "application/x-www-form-urlencoded") headers)
    (setq settings (plist-put settings :headers headers)))
  settings)

(cl-defun request--url-retrieve (url &rest settings
                                     &key type data timeout response
                                     &allow-other-keys
                                     &aux headers)
  (setq settings (apply #'request--url-retrieve-preprocess-settings settings))
  (setq headers (plist-get settings :headers))
  (let* ((url-request-extra-headers headers)
         (url-request-method type)
         (url-request-data data)
         (buffer (url-retrieve url #'request--url-retrieve-callback
                               (nconc (list :response response) settings) t))
         (proc (get-buffer-process buffer)))
    (request--install-timeout timeout response)
    (setf (request-response--buffer response) buffer)
    (process-put proc :request-response response)
    (set-process-query-on-exit-flag proc nil)))

(cl-defun request--url-retrieve-callback (status &rest settings
                                                 &key response url
                                                 &allow-other-keys)
  (when (featurep 'url-http)
    (setf (request-response-status-code response) url-http-response-status))
  (let ((redirect (plist-get status :redirect)))
    (when redirect
      (setf (request-response-url response) redirect)))
  ;; Construct history slot
  (cl-loop for v in
           (cl-loop with first = t
                    with l = nil
                    for (k v) on status by 'cddr
                    when (eq k :redirect)
                    if first
                    do (setq first nil)
                    else
                    do (push v l)
                    finally do (cons url l))
           do (let ((r (make-request-response :-backend 'url-retrieve)))
                (setf (request-response-url r) v)
                (push r (request-response-history response))))

  (cl-symbol-macrolet ((error-thrown (request-response-error-thrown response))
                       (status-error (plist-get status :error)))
    (when status-error
      (request-log 'warn "request--url-retrieve-callback: %s" status-error)
      (unless error-thrown
        (setq error-thrown status-error))))

  (apply #'request--callback (current-buffer) settings))

(cl-defun request--url-retrieve-sync (url &rest settings
                                          &key type data timeout response
                                          &allow-other-keys
                                          &aux headers)
  (setq settings (apply #'request--url-retrieve-preprocess-settings settings))
  (setq headers (plist-get settings :headers))
  (let* ((url-request-extra-headers headers)
         (url-request-method type)
         (url-request-data data)
         (buffer (if timeout
                     (with-timeout
                         (timeout
                          (setf (request-response-symbol-status response)
                                'timeout)
                          (setf (request-response-done-p response) t)
                          nil)
                       (url-retrieve-synchronously url t))
                   (url-retrieve-synchronously url t))))
    (setf (request-response--buffer response) buffer)
    ;; It seems there is no way to get redirects and URL here...
    (when buffer
      ;; Fetch HTTP response code
      (with-current-buffer buffer
        (goto-char (point-min))
        (cl-destructuring-bind (&key code &allow-other-keys)
            (request--parse-response-at-point)
          (setf (request-response-status-code response) code)))
      ;; Parse response body, etc.
      (apply #'request--callback buffer settings)))
  response)

(defun request--url-retrieve-get-cookies (host localpart secure)
  (mapcar
   (lambda (c) (cons (url-cookie-name c) (url-cookie-value c)))
   (url-cookie-retrieve host localpart secure)))

\f
;;; Backend: curl

(defvar request--curl-cookie-jar nil
  "Override what the function `request--curl-cookie-jar' returns.
Currently it is used only for testing.")

(defun request--curl-cookie-jar ()
  "Cookie storage for curl backend."
  (or request--curl-cookie-jar
      (expand-file-name "curl-cookie-jar" request-storage-directory)))

(defvar request--curl-capabilities-cache
  (make-hash-table :test 'eq :weakness 'key)
  "Used to avoid invoking curl more than once for version info.  By skeeto/elfeed.")

(defun request--curl-capabilities ()
  "Return capabilities plist for curl.  By skeeto/elfeed.
:version     -- cURL's version string
:compression -- non-nil if --compressed is supported."
  (let ((cache-value (gethash request-curl request--curl-capabilities-cache)))
    (if cache-value
        cache-value
      (with-temp-buffer
        (call-process request-curl nil t nil "--version")
        (let ((version
               (progn
                 (setf (point) (point-min))
                 (when (re-search-forward "[.0-9]+" nil t)
                   (match-string 0))))
              (compression
               (progn
                 (setf (point) (point-min))
                 (not (null (re-search-forward "libz\\>" nil t))))))
          (setf (gethash request-curl request--curl-capabilities-cache)
                `(:version ,version :compression ,compression)))))))

(defconst request--curl-write-out-template
  (if (eq system-type 'windows-nt)
      "\\n(:num-redirects %{num_redirects} :url-effective %{url_effective})"
    "\\n(:num-redirects %{num_redirects} :url-effective \"%{url_effective}\")"))

(defun request--curl-mkdir-for-cookie-jar ()
  (ignore-errors
    (make-directory (file-name-directory (request--curl-cookie-jar)) t)))

(cl-defun request--curl-command
    (url &key type data headers response files* unix-socket encoding auth
         &allow-other-keys
         &aux (cookie-jar (convert-standard-filename
                           (expand-file-name (request--curl-cookie-jar)))))
  "BUG: Simultaneous requests are a known cause of cookie-jar corruption."
  (append
   (list request-curl
         "--silent" "--location"
         "--cookie" cookie-jar "--cookie-jar" cookie-jar)
   (when auth
     (let* ((host (url-host (url-generic-parse-url url)))
            (auth-source-creation-prompts `((user . ,(format "%s user: " host))
                                            (secret . "Password for %u: ")))
            (cred (car (auth-source-search
                        :host host :require '(:user :secret) :create t :max 1))))
       (split-string (format "--%s --user %s:%s"
                             auth
                             (plist-get cred :user)
                             (let ((secret (plist-get cred :secret)))
                               (if (functionp secret)
                                   (funcall secret)
                                 secret))))))
   (unless (request-url-file-p url)
     (list "--include" "--write-out" request--curl-write-out-template))
   request-curl-options
   (when (plist-get (request--curl-capabilities) :compression) (list "--compressed"))
   (when unix-socket (list "--unix-socket" unix-socket))
   (cl-loop for (name filename path mime-type) in files*
            collect "--form"
            collect (format "%s=@%s;filename=%s%s" name
                            (request-untrampify-filename path) filename
                            (if mime-type
                                (format ";type=%s" mime-type)
                              "")))
   (when data
     (let ((tempfile (request--make-temp-file)))
       (push tempfile (request-response--tempfiles response))
       ;; We dynamic-let the global `buffer-file-coding-system' to `no-conversion'
       ;; in case the user-configured `encoding' doesn't fly.
       ;; If we do not dynamic-let the global, `select-safe-coding-system' would
       ;; plunge us into an undesirable interactive dialogue.
       (let ((buffer-file-coding-system-orig
              (default-value 'buffer-file-coding-system))
             (select-safe-coding-system-accept-default-p
              (lambda (&rest _) t)))
         (unwind-protect
             (progn
               (setf (default-value 'buffer-file-coding-system) 'no-conversion)
               (with-temp-file tempfile
                 (setq-local buffer-file-coding-system encoding)
                 (insert data)))
           (setf (default-value 'buffer-file-coding-system)
                 buffer-file-coding-system-orig)))
       (list "--data-binary" (concat  "@" (request-untrampify-filename tempfile)))))
   (when type (if (equal "head" (downcase type))
		  (list "--head")
		(list "--request" type)))
   (cl-loop for (k . v) in headers
            collect "--header"
            collect (format "%s: %s" k v))
   (list url)))

(defun request--curl-normalize-files-1 (files get-temp-file)
  (cl-loop for (name . item) in files
           collect
           (cl-destructuring-bind
               (filename &key file buffer data mime-type)
               (cond
                ((stringp item) (list (file-name-nondirectory item) :file item))
                ((bufferp item) (list (buffer-name item) :buffer item))
                (t item))
             (unless (= (cl-loop for v in (list file buffer data) if v sum 1) 1)
               (error "Only one of :file/:buffer/:data must be given.  Got: %S"
                      (cons name item)))
             (cond
              (file
               (list name filename file mime-type))
              (buffer
               (let ((tf (funcall get-temp-file)))
                 (with-current-buffer buffer
                   (write-region (point-min) (point-max) tf nil 'silent))
                 (list name filename tf mime-type)))
              (data
               (let ((tf (funcall get-temp-file)))
                 (with-temp-buffer
                   (erase-buffer)
                   (insert data)
                   (write-region (point-min) (point-max) tf nil 'silent))
                 (list name filename tf mime-type)))))))


(declare-function tramp-get-remote-tmpdir "tramp")
(declare-function tramp-dissect-file-name "tramp")

(defun request--make-temp-file ()
  "Create a temporary file."
  (if (file-remote-p default-directory)
      (let ((temporary-file-directory
	     (tramp-get-remote-tmpdir (tramp-dissect-file-name default-directory))))
	(make-temp-file request-temp-prefix))
    (make-temp-file request-temp-prefix)))

(defun request--curl-normalize-files (files)
  "Change FILES into a list of (NAME FILENAME PATH MIME-TYPE).
This is to make `request--curl-command' cleaner by converting
FILES to a homogeneous list.  It returns a list (FILES* TEMPFILES)
where FILES* is a converted FILES and TEMPFILES is a list of
temporary file paths."
  (let (tempfiles noerror)
    (unwind-protect
        (let* ((get-temp-file (lambda ()
                                (let ((tf (request--make-temp-file)))
                                  (push tf tempfiles)
                                  tf)))
               (files* (request--curl-normalize-files-1 files get-temp-file)))
          (setq noerror t)
          (list files* tempfiles))
      (unless noerror
        ;; Remove temporary files only when an error occurs
        (request--safe-delete-files tempfiles)))))

(defun request--safe-delete-files (files)
  "Remove FILES but do not raise error when failed to do so."
  (mapc (lambda (f) (condition-case err
                        (delete-file f)
                      (error (request-log 'error
                               "request--safe-delete-files: %s %s"
                               f (error-message-string err)))))
        files))

(defun request--install-timeout (timeout response)
  "Out-of-band trigger after TIMEOUT seconds to prevent hangs."
  (when (numberp timeout)
    (setf (request-response--timer response)
          (run-at-time timeout nil
                       #'request-response--timeout-callback response))))

(defun request--curl-occlude-secret (command)
  "Simple regex filter on anything looking like a secret."
  (let ((matched
         (string-match (concat (regexp-quote "--user") "\\s-*\\(\\S-+\\)") command)))
    (if matched
        (replace-match "elided" nil nil command 1)
      command)))

(cl-defun request--curl (url &rest settings
                             &key files timeout response encoding semaphore
                             &allow-other-keys)
  "cURL-based request backend.

Redirection handling strategy
-----------------------------

curl follows redirection when --location is given.  However,
all headers are printed when it is used with --include option.
Number of redirects is printed out sexp-based message using
--write-out option (see `request--curl-write-out-template').
This number is used for removing extra headers and parse
location header from the last redirection header.

Sexp at the end of buffer and extra headers for redirects are
removed from the buffer before it is shown to the parser function.
"
  (request--curl-mkdir-for-cookie-jar)
  (let* (;; Use pipe instead of pty.  Otherwise, curl process hangs.
         (process-connection-type nil)
         ;; Avoid starting program in non-existing directory.
         (home-directory (or (file-remote-p default-directory) "~/"))
         (default-directory (expand-file-name home-directory))
         (buffer (generate-new-buffer " *request curl*"))
         (command (cl-destructuring-bind
                      (files* tempfiles)
                      (request--curl-normalize-files files)
                    (setf (request-response--tempfiles response) tempfiles)
                    (apply #'request--curl-command url :files* files*
                           :response response :encoding encoding settings)))
         (proc (apply #'start-process "request curl" buffer command)))
    (request--install-timeout timeout response)
    (request-log 'debug "request--curl: %s"
                 (request--curl-occlude-secret (mapconcat 'identity command " ")))
    (setf (request-response--buffer response) buffer)
    (process-put proc :request-response response)
    (set-process-coding-system proc 'no-conversion 'no-conversion)
    (set-process-query-on-exit-flag proc nil)
    (let ((callback-2 (apply-partially #'request--curl-callback url)))
      (if semaphore
          (set-process-sentinel proc (lambda (&rest args)
                                       (apply callback-2 args)
                                       (apply semaphore args)))
        (set-process-sentinel proc callback-2)))))

(defun request--curl-read-and-delete-tail-info ()
  "Read a sexp at the end of buffer and remove it and preceding character.
This function moves the point at the end of buffer by side effect.
See also `request--curl-write-out-template'."
  (let (forward-sexp-function)
    (goto-char (point-max))
    (forward-sexp -1)
    (let ((beg (1- (point))))
      (prog1
          (read (current-buffer))
        (delete-region beg (point-max))))))

(defconst request--cookie-reserved-re
  (mapconcat
   (lambda (x) (concat "\\(^" x "\\'\\)"))
   '("comment" "commenturl" "discard" "domain" "max-age" "path" "port"
     "secure" "version" "expires")
   "\\|")
  "Uninterested keys in cookie.
See \"set-cookie-av\" in http://www.ietf.org/rfc/rfc2965.txt")

(defun request--consume-100-continue ()
  "Remove \"HTTP/* 100 Continue\" header at the point."
  (cl-destructuring-bind (&key code &allow-other-keys)
      (save-excursion (request--parse-response-at-point))
    (when (equal code 100)
      (request-log 'debug "request--consume-100-continue: consuming\n%s"
                   (buffer-substring (point)
                                     (save-excursion
                                       (request--goto-next-body t)
                                       (point))))
      (delete-region (point) (progn (request--goto-next-body) (point)))
      ;; FIXME: Does this make sense?  Is it possible to have multiple 100?
      (request--consume-100-continue))))

(defun request--consume-200-connection-established ()
  "Remove \"HTTP/* 200 Connection established\" header at the point."
  (when (looking-at-p "HTTP/1\\.[0-1] 200 Connection established")
    (delete-region (point) (progn (request--goto-next-body) (point)))))

(defun request--curl-preprocess (&optional url)
  "Pre-process current buffer before showing it to user."
  (let (history)
    (cl-destructuring-bind (&key num-redirects url-effective)
        (if (request-url-file-p url)
            `(:num-redirects 0 :url-effective ,url)
          (request--curl-read-and-delete-tail-info))
      (goto-char (point-min))
      (request--consume-100-continue)
      (request--consume-200-connection-established)
      (when (> num-redirects 0)
        (cl-loop with case-fold-search = t
                 repeat num-redirects
                 ;; Do not store code=100 headers:
                 do (request--consume-100-continue)
                 do (let ((response (make-request-response
                                     :-buffer (current-buffer)
                                     :-backend 'curl)))
                      (request--clean-header response)
                      (request--cut-header response)
                      (push response history))))

      (goto-char (point-min))
      (nconc (list :num-redirects num-redirects :url-effective url-effective
                   :history (nreverse history))
             (request--parse-response-at-point)))))

(defun request--curl-absolutify-redirects (start-url redirects)
  "Convert relative paths in REDIRECTS to absolute URLs.
START-URL is the URL requested."
  (cl-loop for prev-url = start-url then url
           for url in redirects
           unless (string-match url-nonrelative-link url)
           do (setq url (url-expand-file-name url prev-url))
           collect url))

(defun request--curl-absolutify-location-history (start-url history)
  "Convert relative paths in HISTORY to absolute URLs.
START-URL is the URL requested."
  (when history
    (setf (request-response-url (car history)) start-url))
  (cl-loop for url in (request--curl-absolutify-redirects
                       start-url
                       (mapcar (lambda (response)
                                 (or (request-response-header response "location")
                                     (request-response-url response)))
                               history))
           for response in (cdr history)
           do (setf (request-response-url response) url)))

(defun request--curl-callback (url proc event)
  (let* ((buffer (process-buffer proc))
         (response (process-get proc :request-response))
         (settings (request-response-settings response)))
    (request-log 'debug "request--curl-callback: event %s" event)
    (request-log 'trace "request--curl-callback: raw-bytes=\n%s"
                 (when (buffer-live-p buffer)
                   (with-current-buffer buffer (buffer-string))))
    (cond
     ((and (memq (process-status proc) '(exit signal))
           (/= (process-exit-status proc) 0))
      (setf (request-response-error-thrown response) (cons 'error event))
      (apply #'request--callback buffer settings))
     ((cl-search "finished" event)
      (cl-destructuring-bind (&key code history error url-effective &allow-other-keys)
          (condition-case err
              (with-current-buffer buffer
                (request--curl-preprocess url))
            ((debug error)
             (list :error err)))
        (request--curl-absolutify-location-history (plist-get settings :url)
                                                   history)
        (setf (request-response-status-code  response) code)
        (setf (request-response-url          response) url-effective)
        (setf (request-response-history      response) history)
        (setf (request-response-error-thrown response)
              (or error (and (numberp code) (>= code 400) `(error . (http ,code)))))
        (apply #'request--callback buffer settings))))))

(defun request-auto-revert-notify-rm-watch ()
  "Backport of M. Engdegard's fix of `auto-revert-notify-rm-watch'."
  (let ((desc auto-revert-notify-watch-descriptor)
        (table (if (boundp 'auto-revert--buffers-by-watch-descriptor)
                   auto-revert--buffers-by-watch-descriptor
                 (when (boundp 'auto-revert-notify-watch-descriptor-hash-list)
                   auto-revert-notify-watch-descriptor-hash-list))))
    (when desc
      (let ((buffers (delq (current-buffer) (gethash desc table))))
        (if buffers
            (puthash desc buffers table)
          (remhash desc table)))
      (condition-case nil ;; ignore-errors doesn't work for me, sorry
	(file-notify-rm-watch desc)
        (error))
      (remove-hook 'kill-buffer-hook #'auto-revert-notify-rm-watch t)))
  (setq auto-revert-notify-watch-descriptor nil
	auto-revert-notify-modified-p nil))

(cl-defun request--curl-sync (url &rest settings &key response &allow-other-keys)
  (let (finished)
    (prog1 (apply #'request--curl url
                  :semaphore (lambda (&rest _) (setq finished t))
                  settings)
      (let* ((proc (get-buffer-process (request-response--buffer response)))
	     (interval 0.05)
	     (timeout 5)
	     (maxiter (truncate (/ timeout interval))))
        (auto-revert-set-timer)
        (when auto-revert-use-notify
          (dolist (buf (buffer-list))
            (with-current-buffer buf
              (request-auto-revert-notify-rm-watch))))
        (with-local-quit
          (cl-loop with iter = 0
                   until (or (>= iter maxiter) finished)
                   do (accept-process-output nil interval)
                   unless (request--process-live-p proc)
                     do (cl-incf iter)
                   end
                   finally (when (>= iter maxiter)
                             (let ((m "request--curl-sync: semaphore never called"))
                               (princ (format "%s\n" m) #'external-debugging-output)
                               (request-log 'error m)))))))))

(defun request--curl-get-cookies (host localpart secure)
  (request--netscape-get-cookies (request--curl-cookie-jar)
                                 host localpart secure))

\f
;;; Netscape cookie.txt parser

(defun request--netscape-cookie-parse ()
  "Parse Netscape/Mozilla cookie format."
  (goto-char (point-min))
  (let ((tsv-re (concat "^\\(#HttpOnly_\\)?"
                        (cl-loop repeat 6 concat "\\([^\t\n]+\\)\t")
                        "\\(.*\\)"))
        cookies)
    (while (not (eobp))
      ;; HttpOnly cookie starts with '#' but its line is not comment line(#60)
      (cond ((and (looking-at-p "^#") (not (looking-at-p "^#HttpOnly_"))) t)
            ((looking-at-p "^$") t)
            ((looking-at tsv-re)
             (let ((cookie (cl-loop for i from 1 to 8 collect (match-string i))))
               (push cookie cookies))))
      (forward-line 1))
    (setq cookies (nreverse cookies))
    (cl-loop for (http-only domain flag path secure expiration name value) in cookies
             collect (list domain
                           (equal flag "TRUE")
                           path
                           (equal secure "TRUE")
                           (null (not http-only))
                           (string-to-number expiration)
                           name
                           value))))

(defun request--netscape-filter-cookies (cookies host localpart secure)
  (cl-loop for (domain _flag path secure-1 _http-only _expiration name value) in cookies
           when (and (equal domain host)
                     (equal path localpart)
                     (or secure (not secure-1)))
           collect (cons name value)))

(defun request--netscape-get-cookies (filename host localpart secure)
  (when (file-readable-p filename)
    (with-temp-buffer
      (erase-buffer)
      (insert-file-contents filename)
      (request--netscape-filter-cookies (request--netscape-cookie-parse)
                                        host localpart secure))))

(provide 'request)

;;; request.el ends here

[-- Attachment #3: Type: text/plain, Size: 2530 bytes --]



"Request.el has a single entry point.  It is `request'."


(cl-defun request (url &rest settings
                       &key
                       (params nil)
                       (data nil)
                       (headers nil)
                       (encoding 'utf-8)
                       (error nil)
                       (sync nil)
                       (response (make-request-response))
                       &allow-other-keys)
  "Send request to URL.

Request.el has a single entry point.  It is `request'.

==================== ========================================================
Keyword argument      Explanation
==================== ========================================================
TYPE          (string)   type of request to make: POST/GET/PUT/DELETE
PARAMS         (alist)   set \"?key=val\" part in URL
DATA    (string/alist)   data to be sent to the server
FILES          (alist)   files to be sent to the server (see below)
PARSER        (symbol)   a function that reads current buffer and return data
HEADERS        (alist)   additional headers to send with the request
ENCODING      (symbol)   encoding for request body (utf-8 by default)
SUCCESS     (function)   called on success
ERROR       (function)   called on error
COMPLETE    (function)   called on both success and error
TIMEOUT       (number)   timeout in second
STATUS-CODE    (alist)   map status code (int) to callback
SYNC            (bool)   If `t', wait until request is done.  Default is `nil'.
==================== ========================================================


* Callback functions

Callback functions STATUS, ERROR, COMPLETE and `cdr's in element of
the alist STATUS-CODE take same keyword arguments listed below.  For
forward compatibility, these functions must ignore unused keyword
arguments (i.e., it's better to use `&allow-other-keys' [#]_).::

    (CALLBACK                      ; SUCCESS/ERROR/COMPLETE/STATUS-CODE
     :data          data           ; whatever PARSER function returns, or nil
     :error-thrown  error-thrown   ; (ERROR-SYMBOL . DATA), or nil
     :symbol-status symbol-status  ; success/error/timeout/abort/parse-error
     :response      response       ; request-response object
     ...)

___________________

I mentioned it just because it is already a wrapper around both built
url and curl (the process not libcurl), so it already abstracts
everything needed to profile (I think).

I have used it privately in my code when I wanted to download some html
for text processing.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Philip K." <philipk@posteo.net>
>> Cc: arthur.miller@live.com, adam@alphapapa.net, rms@gnu.org,
>> 	emacs-devel@gnu.org
>> Date: Tue, 22 Dec 2020 00:51:08 +0100
>> 
>> >> > Before this is taken as an indication that using one of these
>> >> > libraries will automagically make Emacs as fast as the applications
>> >> > like curl, we should carefully profile url.el and find out which
>> >> > part(s) of it cause the slowness.  Because it could well be that what
>> >> > makes url.el slow will also make Emacs using libcurl slow.
>> >> 
>> >> Maybe this library can be of help while testing/profiling
>> >
>> > We have a built-in profiler in Emacs.
>> 
>> In that case, gnutls-negotiate seems to be the most suspicious function,
>> using over 50%-60% of the CPU time, at least on my machine. This also
>> makes sense, as TLS sites seem to take longer to load than regular,
>> non-encrypted ones.
>
> Please show the code you profiled and the fully expanded profile.

I sadly coudln't reproduce it, but this time the critical section looked
something like this:

        - url-retrieve-synchronously                               212  52%
         - url-retrieve                                            149  36%
          - url-retrieve-internal                                  149  36%
           - url-http                                              136  33%
            - url-http-find-free-connection                        135  33%
             - url-open-stream                                     135  33%
                open-network-stream                                134  33%

when evaluating 

	(url-retrieve-synchronously "http://textboard.org/sexp/prog/index")

in the *scratch* buffer. I used emacs -Q (GNU Emacs 27.1 (build 1,
x86_64-redhat-linux-gnu, GTK+ Version 3.24.22, cairo version 1.16.0) of
2020-08-21), but I don't know why that should make any
difference. I repeated the same test on the master branch and the
results were basically the same (±5%).

Either way, this simple request took over 2.5 minutes, whereas curl
requires a quarter of a second. Note that this is even unencrypted, so
this is not even taking the encryption overhead into account.

(FYI: I took textboard.org as an example, because I wrote a client for
      this site in Elisp, that often takes a while to load, so I could
      count on it for this test. An interesting observation is that the
      first request might take a while, but sometimes the connection
      time suddenly drops to basically nothing. I have yet to figure
      out what the reason is. :/)

-- 
	Philip K.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Richard Stallman <rms@gnu.org> writes:

>   > Request.el is another story, and I can't comment on that. It's probably
>   > just like with dash/s/f/... where non-elisp developers dislike the
>   > built-in API, but I'm just guessing.
>
> From what people have said here, it sounds unproblematical.
> But could you explain what you mean by comparing it with dash/s/f/?

In my experience, people use dash/s/f when they are either unfamiliar
with the built-in functions and macros, or when they find them too
cumbersome to use (I'm sure there are exceptions, but again, this is my
impression). So they want to write (s-join "+" '("abc" "def" "ghi"))
instead of (mapconcat #'identity '("abc" "def" "ghi") "+"), and add an
external dependency for this minor convenience.

And again, in my experience this often motivates people to use request,
though it might be a better example, because it actually does useful
stuff, even though I don't always think it's idiomatic. The built-in
url.el could use this as an inspiration, to add more macros/functions
that simplify code.

One notable difference is that it will use curl, the binary, instead of
url.el if it's installed, which does notably accelerate network
requests.

-- 
	Philip K.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Jean Louis]

* Philip K. <philipk@posteo.net> [2020-12-22 13:53]:
> One notable difference is that it will use curl, the binary, instead of
> url.el if it's installed, which does notably accelerate network
> requests.

Do you think Emacs could use some C library like gnurl or something
else that could accelerate Emacs's network requests?

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

"Philip K." <philipk@posteo.net> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>   > Request.el is another story, and I can't comment on that. It's probably
>>   > just like with dash/s/f/... where non-elisp developers dislike the
>>   > built-in API, but I'm just guessing.
>>
>> From what people have said here, it sounds unproblematical.
>> But could you explain what you mean by comparing it with dash/s/f/?
>
> In my experience, people use dash/s/f when they are either unfamiliar
> with the built-in functions and macros, or when they find them too
> cumbersome to use (I'm sure there are exceptions, but again, this is my
> impression). So they want to write (s-join "+" '("abc" "def" "ghi"))
> instead of (mapconcat #'identity '("abc" "def" "ghi") "+"), and add an
> external dependency for this minor convenience.

Indeed. I think one problem is dicoverability. Built-in functions and
macros does not start always with what people would use as their first
guess. Thinngs like s-* orc -* are easier to look up (just one page on
github), easier to guess the name etc. Then we have things like
string-replace and replace-string: which one do you use if you are not
familiar with elisp? It asks for doc lookup, lookup breaks the workflow
and thought process, and so people probably find it easier to open
github page for s.el then bother to figure out how things are done
idiomatically in Elisp. 

> And again, in my experience this often motivates people to use request,
> though it might be a better example, because it actually does useful
> stuff, even though I don't always think it's idiomatic. The built-in
> url.el could use this as an inspiration, to add more macros/functions
> that simplify code.
>
> One notable difference is that it will use curl, the binary, instead of
> url.el if it's installed, which does notably accelerate network
> requests.

Yes, I think it was main goal of the the author - to use curl when
possible for the performance reasons; but I don't know for sure. Maybe
the author is reading this list and can step in and tell us.

I don't think it was same motivation as behind s/dash etc. It defines
asynchronous framework for use with networking. It is an "old" package,
at least since 2012, I don't remember if Emacs back then had threads or
not.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Jean Louis <bugs@gnu.support> writes:

> * Philip K. <philipk@posteo.net> [2020-12-22 13:53]:
>> One notable difference is that it will use curl, the binary, instead of
>> url.el if it's installed, which does notably accelerate network
>> requests.
>
> Do you think Emacs could use some C library like gnurl or something
> else that could accelerate Emacs's network requests?

I don't know enough about how Emacs' networking currently works to say
for sure.

-- 
	Philip K.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: "Philip K." <philipk@posteo.net>
> Cc: arthur.miller@live.com, adam@alphapapa.net, rms@gnu.org,
> 	emacs-devel@gnu.org
> Date: Tue, 22 Dec 2020 11:38:31 +0100
> 
> >> In that case, gnutls-negotiate seems to be the most suspicious function,
> >> using over 50%-60% of the CPU time, at least on my machine. This also
> >> makes sense, as TLS sites seem to take longer to load than regular,
> >> non-encrypted ones.
> >
> > Please show the code you profiled and the fully expanded profile.
> 
> I sadly coudln't reproduce it, but this time the critical section looked
> something like this:
> 
>         - url-retrieve-synchronously                               212  52%
>          - url-retrieve                                            149  36%
>           - url-retrieve-internal                                  149  36%
>            - url-http                                              136  33%
>             - url-http-find-free-connection                        135  33%
>              - url-open-stream                                     135  33%
>                 open-network-stream                                134  33%
> 
> when evaluating 
> 
> 	(url-retrieve-synchronously "http://textboard.org/sexp/prog/index")
> 
> in the *scratch* buffer. I used emacs -Q (GNU Emacs 27.1 (build 1,
> x86_64-redhat-linux-gnu, GTK+ Version 3.24.22, cairo version 1.16.0) of
> 2020-08-21), but I don't know why that should make any
> difference. I repeated the same test on the master branch and the
> results were basically the same (±5%).
> 
> Either way, this simple request took over 2.5 minutes, whereas curl
> requires a quarter of a second. Note that this is even unencrypted, so
> this is not even taking the encryption overhead into account.

That's strange, because I get here much faster times: 0.6 sec (with 3
GC cycles) on the first attempt, and less than 0.1 sec afterwards.
How come it's so slow on your system?

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Philip K." <philipk@posteo.net>
>> Cc: arthur.miller@live.com, adam@alphapapa.net, rms@gnu.org,
>> 	emacs-devel@gnu.org
>> Date: Tue, 22 Dec 2020 11:38:31 +0100
>> 
>> >> In that case, gnutls-negotiate seems to be the most suspicious function,
>> >> using over 50%-60% of the CPU time, at least on my machine. This also
>> >> makes sense, as TLS sites seem to take longer to load than regular,
>> >> non-encrypted ones.
>> >
>> > Please show the code you profiled and the fully expanded profile.
>> 
>> I sadly coudln't reproduce it, but this time the critical section looked
>> something like this:
>> 
>>         - url-retrieve-synchronously                               212  52%
>>          - url-retrieve                                            149  36%
>>           - url-retrieve-internal                                  149  36%
>>            - url-http                                              136  33%
>>             - url-http-find-free-connection                        135  33%
>>              - url-open-stream                                     135  33%
>>                 open-network-stream                                134  33%
>> 
>> when evaluating 
>> 
>> 	(url-retrieve-synchronously "http://textboard.org/sexp/prog/index")
>> 
>> in the *scratch* buffer. I used emacs -Q (GNU Emacs 27.1 (build 1,
>> x86_64-redhat-linux-gnu, GTK+ Version 3.24.22, cairo version 1.16.0) of
>> 2020-08-21), but I don't know why that should make any
>> difference. I repeated the same test on the master branch and the
>> results were basically the same (±5%).
>> 
>> Either way, this simple request took over 2.5 minutes, whereas curl
>> requires a quarter of a second. Note that this is even unencrypted, so
>> this is not even taking the encryption overhead into account.
>
> That's strange, because I get here much faster times: 0.6 sec (with 3
> GC cycles) on the first attempt, and less than 0.1 sec afterwards.

Why did it take three GC cycles? benchmark-run says

    (130.773534384 0 0.0)

and 

    (129.933227402 0 0.0)

for the first two runs, and

    (0.044592597 0 0.0)

for the third one, a bit later.

> How come it's so slow on your system?

I can't say for sure, but I don't think it is due to some specific local
configuration. I reran the code on a university computer a few
kilometres away, and the behaviour stays basically the same, even though
their network is a lot better than mine. But when I ran it on a VPS in
Berlin (I'm currently in southern Germany), it was a lot faster, despite
probably running the same build (whatever it in the Debian Buster
repository). I looked up where textboard.org is located, and it seems to
be in France, or it at the very least has to pass through some Proxy in
France. Another interesting observation is that connections from my home
to my university are basically instantaneous using Eww, and vice
versa. The Server in Berlin takes a lot longer to connect to my local
network and my university server. The only exception to this rule is
that eww doesn't seem to take that long to connect to my VPS.

None of these effects are observable using curl.

I guess this should be reported as a proper bug, instead of being
discussed here.

-- 
	Philip K.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: "Philip K." <philipk@posteo.net>
> Cc: arthur.miller@live.com, adam@alphapapa.net, rms@gnu.org,
> 	emacs-devel@gnu.org
> Date: Tue, 22 Dec 2020 17:59:19 +0100
> 
> >> Either way, this simple request took over 2.5 minutes, whereas curl
> >> requires a quarter of a second. Note that this is even unencrypted, so
> >> this is not even taking the encryption overhead into account.
> >
> > That's strange, because I get here much faster times: 0.6 sec (with 3
> > GC cycles) on the first attempt, and less than 0.1 sec afterwards.
> 
> Why did it take three GC cycles?

There's no way for me to know.  It could well be that the previous
commands, which had nothing to do with the benchmark, produced enough
garbage for GC to kick in at that point.

> benchmark-run says
> 
>     (130.773534384 0 0.0)
> 
> and 
> 
>     (129.933227402 0 0.0)
> 
> for the first two runs, and
> 
>     (0.044592597 0 0.0)
> 
> for the third one, a bit later.

The last one sounds comparable to what I see.  So maybe the long time
has nothing to do with Emacs, but with something else.  Perhaps this
is because DNS is asynchronous on your system?

> I guess this should be reported as a proper bug, instead of being
> discussed here.

Yep.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Do you think Emacs could use some C library like gnurl or something
  > > else that could accelerate Emacs's network requests?

I think it is quite undesirable to link more facilities into Emacs.
If fork and exec of a program like curl are adequate, we should do it
that way, for simplicity.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Does request.el indeed duplicate built-in functionality?
I don't know for certain what it does, but if it is just
"try curl, then url.el", that may still be an improvement
over what we have now.

But maybe it could be added very simply to url.el.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Does request.el indeed duplicate built-in functionality?
I wouldn't think so. It models an asychronous framework for network
requests.

> I don't know for certain what it does, but if it is just
> "try curl, then url.el", that may still be an improvement
> over what we have now.
>
> But maybe it could be added very simply to url.el.
Sure, you could do own wrapper, but the man already modelled one, why
not just use it?

Author is Arakaki Takafumi, I don't know if he has signed FSF policy or
not, the package itself is GPL:ed.

I have cc:ed him, so let see if he answers, he can maybe best answer all
the questions.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Does request.el indeed duplicate built-in functionality?
  > I wouldn't think so. It models an asychronous framework for network
  > requests.

That is very abstract -- concretely what does it mean?

  > > But maybe it could be added very simply to url.el.

  > Sure, you could do own wrapper,

Why do you say "wrapper"?  I don't know what you mean,
but I have a feeling we are miscommunicating.

                                    but the man already modelled one, why
  > not just use it?

What do you mean by "modelled"?  

I have never used url.el, so I am making some surmises.
I could be mistaken.  But I can't tell from what you've said
whether I was mistaken.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Does request.el indeed duplicate built-in functionality?
>   > I wouldn't think so. It models an asychronous framework for network
>   > requests.
>
> That is very abstract -- concretely what does it mean?
It gives a little API to do http requests, and it can switch between
curl and built-in url as backends. I thought it was already done, so
easy to use when profiling. I thought it would be useful, so I just
pointed to it, I didn't thought it would cause so much discussion.

>   > > But maybe it could be added very simply to url.el.
>
>   > Sure, you could do own wrapper,
>
> Why do you say "wrapper"?  I don't know what you mean,
An abstraction on top of both, as he did.

> but I have a feeling we are miscommunicating.
Maybe, easy on the internet. I am not always the best to communicate
with either. Sorry, probably my fault.
>                                     but the man already modelled one, why
>   > not just use it?
>
> What do you mean by "modelled"?  
Architect, designed, created .... it was just a word.

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Instead of a new layer above url.el, how about a small change _inside_ url.el
to optionally use curl instead?

Adding layers tends to make code bigger and slower, so we shouldn't
make that our go-to approach by default.  Sometimes the new layer
is worth the cost, but we should compare with other approaches.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: How to contribute new package to GNU ELPA?

[stardiviner]

I got response from my proposed issue https://github.com/tkf/emacs-request/issues/195

Seems request.el will not be included in GNU ELPA.

https://github.com/tkf/emacs-request/issues/112

I'm thinking, if request.el will not be included in GNU ELPA, then will Emacs
development team write a similar library?

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> More importantly: what do you want to do about `request`?
>>>
>> About this, I really suggest ELPA can include it.
>
> I'm glad we agree.  AFAICT the main contributor is Takafumi Arakaki who
> hasn't signed the copyright paperwork.  Any chance you could contact him
> and ask him to sign that paperwork?
>
> AFAICT `request` is a fairly popular library so it would be very nice to
> have it in GNU ELPA.
>
>
>         Stefan


-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: How to contribute new package to GNU ELPA?

[dick.r.chiang]

s> Seems request.el will not be included in GNU ELPA.
s> https://github.com/tkf/emacs-request/issues/112

I disagree with your conclusion.

Just because some random contributor (dickmao) is unwilling to solicit
Mr. Arakaki's written release does not mean you must do the same.  After all,
his acquaintance with Mr. Arakaki is just as non-existent as yours.

An importunate email campaign to the effect of "I will continue emailing you once
per week until you respond definitively yes or no.  Apologies in advance."
seems apropos.

Re: How to contribute new package to GNU ELPA?

[Adam Porter]

dick.r.chiang@gmail.com writes:

> s> Seems request.el will not be included in GNU ELPA.
> s> https://github.com/tkf/emacs-request/issues/112
>
> I disagree with your conclusion.
>
> Just because some random contributor (dickmao) is unwilling to solicit
> Mr. Arakaki's written release does not mean you must do the same.
> After all, his acquaintance with Mr. Arakaki is just as non-existent
> as yours.
>
> An importunate email campaign to the effect of "I will continue
> emailing you once per week until you respond definitively yes or no.
> Apologies in advance."  seems apropos.

Mr. Arakaki has not commited to the repo in over 6.5 years:
https://github.com/tkf/emacs-request/commits?author=tkf  Since, according
to GitHub, he has made over 4,000 commits to other projects in the past
year, it seems likely that he is uninterested in the project.

As I pointed out in my message of a few days ago about "Emacs HTTP
libraries," Request.el seems to have some issues with governance and
maintenance: https://github.com/tkf/emacs-request/issues/172

So although it's generally usable, considering 1) the bugs I encountered
in using it, ones that forced me to use url.el instead, 2) its
maintenance issues, and 3) the issues I pointed out regarding Curl and
handling of sensitive data on the command line and in temporary files, I
don't think it should be added to ELPA.  (If you have more questions,
please refer to the message I posted, which I thoroughly footnoted.)

Instead, I would encourage Lars to continue the work on his
with-fetched-url branch, which looks very promising.

Re: How to contribute new package to GNU ELPA?

[Stefan Monnier]

> Instead, I would encourage Lars to continue the work on his
> with-fetched-url branch, which looks very promising.

We need a solution in the short term which lets us include in (Non)GNU
ELPA packages which currently rely on the `request.el` API.

An easy short term fix is to include request.el into NonGNU ELPA.
But that will reduce the pressure to find a better fix.

Still, if there's no better suggestion, that's probably what I'll do.


        Stefan

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1440 bytes --]

Daniel Brooks <db48x@db48x.net> writes:

1+ on libcurl, this discussion happened a while ago here though and ran
aground, dont remember why.
But independent of the ability down the road to native-compile elisp, I
still think re-using the work of the curl project to get an HTTP library
that the emacs project doesn't have to maintain would be a big win

> Adam Porter <adam@alphapapa.net> writes:
>
>> Of course, it would be ideal if Emacs had a built-in HTTP library that
>> didn't require an external dependency like Curl, but Curl does work
>> well.  The only issue I have with it is that certain data can only be
>> passed via either the command line (leaving it visible to the whole
>> system) or temporary files (which have to be carefully handled), which
>> is a concern for, e.g. cookies, authentication tokens, passwords, etc.
>> Unfortunately, when that issue was raised and a solution proposed in
>> 2003, it was quickly rejected.[8]
>>
>> 8: https://curl.se/mail/archive-2003-08/0099.html
>
> For a built-in library, it would be better to use libcurl. This ensures
> that there won't be any complications from hiding this type of
> information, and the curl command-line program is just a simple wrapper
> around libcurl anyway. In fact, even an installable library could
> include libcurl in a dynamically-loadable module.
>
> db48x
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Vladimir Sedach]

T.V Raman <raman@google.com> writes:

> Daniel Brooks <db48x@db48x.net> writes:
>
> 1+ on libcurl, this discussion happened a while ago here though and ran
> aground, dont remember why.
> But independent of the ability down the road to native-compile elisp, I
> still think re-using the work of the curl project to get an HTTP library
> that the emacs project doesn't have to maintain would be a big win

libwget from GNU Wget2 is a better alternative. It is a GNU
project and shares the libgnutls dependency with Emacs.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la

Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

> libwget from GNU Wget2 is a better alternative. It is a GNU
> project and shares the libgnutls dependency with Emacs.

Are you sure about that one?

https://gitlab.com/gnuwget/wget2/-/wikis/GNU-vs.-non-GNU

However, Emacs already links against other non GNU libs, I am
not sure it is an argument. Curl can use gnutls if desired, but maybe
Emacs can get rid of gnutls dependency if using libcurl for the
network stack?

Libcurl is well known, tested and well documented (there are
even books about it); libwget seems like new as a library.
I don't know, I am not familiar with libwget, didn't know wget had
an API and a library; they didn't used to before. Maybe it is technically
better then libcurl, it can be worth investigation.

Whichever emacs devs would use I think it would be nice to have
a good networking stack accessible from elisp.
________________________________
Från: Emacs-devel <emacs-devel-bounces+arthur.miller=live.com@gnu.org> för Vladimir Sedach <vas@oneofus.la>
Skickat: den 29 mars 2021 00:42
Till: T.V Raman <raman@google.com>
Kopia: Daniel Brooks <db48x@db48x.net>; Adam Porter <adam@alphapapa.net>; emacs-devel@gnu.org <emacs-devel@gnu.org>
Ämne: Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]


T.V Raman <raman@google.com> writes:

> Daniel Brooks <db48x@db48x.net> writes:
>
> 1+ on libcurl, this discussion happened a while ago here though and ran
> aground, dont remember why.
> But independent of the ability down the road to native-compile elisp, I
> still think re-using the work of the curl project to get an HTTP library
> that the emacs project doesn't have to maintain would be a big win

libwget from GNU Wget2 is a better alternative. It is a GNU
project and shares the libgnutls dependency with Emacs.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la


[-- Attachment #2: Type: text/html, Size: 6147 bytes --]

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Jose E. Marchesi]

>> libwget from GNU Wget2 is a better alternative. It is a GNU
>> project and shares the libgnutls dependency with Emacs.
>
> Are you sure about that one?
>
> https://gitlab.com/gnuwget/wget2/-/wikis/GNU-vs.-non-GNU

wget2 is definitely a GNU program.
I'm CCing Darshit, the maintainer.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Darshit Shah]

On 30.03.21 00:03, Jose E. Marchesi wrote:
> 
>>> libwget from GNU Wget2 is a better alternative. It is a GNU
>>> project and shares the libgnutls dependency with Emacs.
>>
>> Are you sure about that one?
>>
>> https://gitlab.com/gnuwget/wget2/-/wikis/GNU-vs.-non-GNU
> 
> wget2 is definitely a GNU program.
> I'm CCing Darshit, the maintainer.
>

I am the maintainer of GNU Wget and Wget2 is the spiritual successor to
Wget itself. It is not a new / different project. Wget2 is written as a
frontend to the library, libwget the same way as cURL is the frontend ot
libcurl.

Yes, libwget is not as battle tested as libcurl is, but we're working
towards getting more projects to use it. There are other GNU projects
that are also interested in integrating libwget for their HTTP client
library needs

From a technological point of view, please evaluate libwget and if you
find it short / missing in some aspect, let us know and we can work with
you to add the needed features. From a philosophical point of view,
you'd be supporting a free software project that better aligns with the
GNU ideologies.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Daniel Brooks]

"Jose E. Marchesi" <jemarch@gnu.org> writes:

>>> libwget from GNU Wget2 is a better alternative. It is a GNU
>>> project and shares the libgnutls dependency with Emacs.
>>
>> Are you sure about that one?
>>
>> https://gitlab.com/gnuwget/wget2/-/wikis/GNU-vs.-non-GNU
>
> wget2 is definitely a GNU program.
> I'm CCing Darshit, the maintainer.

I suspect that Arthur was wondering how sure Vladimir is that libwget is
better than libcurl. There isn’t much question that Wget2 is a GNU
project, or what its dependencies are.

db48x

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

Darshit Shah <darnir@gnu.org> writes:

> On 30.03.21 00:03, Jose E. Marchesi wrote:
>> 
>>>> libwget from GNU Wget2 is a better alternative. It is a GNU
>>>> project and shares the libgnutls dependency with Emacs.
>>>
>>> Are you sure about that one?
>>>
>>> https://gitlab.com/gnuwget/wget2/-/wikis/GNU-vs.-non-GNU
>> 
>> wget2 is definitely a GNU program.
>> I'm CCing Darshit, the maintainer.
>>
>
> I am the maintainer of GNU Wget and Wget2 is the spiritual successor to
> Wget itself. It is not a new / different project. Wget2 is written as a

Thanks for the answer. No idea if there is a consideration to include
new library or not, I got impression from some other discussions on this
list, it is rather not.

I saw the wiki page about GNU vs Non GNU and the remark on the bottom of
the page about being Non GNU 2017, but I see you are using LGPL license.

For the technical side, no idea how libwget is comparable to
libcurl. About libcurl I know it is avialable included on Windows and
MacOS and avialable for probaly any GNU/Linux *BSD distro and it
supports lots of protocols, SSL implementations (GnuTLS, OpenSSL, NSS
etc) and has very good documentation including some published books.

I am sure libwget has its own merits too, I am just not familiar with
it, so I am not saying neither for or against.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Daniel Martín]

Arthur Miller <arthur.miller@live.com> writes:

>
> Thanks for the answer. No idea if there is a consideration to include
> new library or not, I got impression from some other discussions on this
> list, it is rather not.

I also think that having a better networking library could be a great
addition to Emacs.  But before we decide on introducing a curl or wget
dependency, I think we should do some analysis on url.el, to be sure
about the trade-offs of the proposed new approach to do networking,
compared to what we already have.  There's two main points that people
wanted to improve in url.el, according to the comments in this thread:

- url.el has strange/buggy behavior in some scenarios.  For example,
  callbacks are run multiple times, or not at all.  This blog post[1]
  presents some of those problems, but it doesn't offer much details
  about why they happen.  The blog post is from 2016, so things might
  have changed since then.

- url.el is slow compared to using curl as an inferior process.  Again,
  the blog post cites some possible culprits, which I don't know if they
  still apply to Emacs 28, so I think we should profile url.el to
  identify its bottlenecks more precisely.

Once we have concrete numbers that we can compare against a prototype
implementation using libwget2 or libcurl, I think it'd be easier to
justify adding a new dependency to Emacs, or search for other solutions.

[1]: https://nullprogram.com/blog/2016/06/16/

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Stefan Kangas]

Daniel Martín <mardani29@yahoo.es> writes:

> I also think that having a better networking library could be a great
> addition to Emacs.  But before we decide on introducing a curl or wget
> dependency, I think we should do some analysis on url.el, to be sure
> about the trade-offs of the proposed new approach to do networking,
> compared to what we already have.

There is also the `scratch/with-fetched-url' branch.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Daniel Martín <mardani29@yahoo.es>
> Cc: Darshit Shah <darnir@gnu.org>,  "T.V Raman" <raman@google.com>,  Daniel
>  Brooks <db48x@db48x.net>,  Adam Porter <adam@alphapapa.net>,
>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>,  "Jose E. Marchesi"
>  <jemarch@gnu.org>
> Date: Tue, 30 Mar 2021 14:03:58 +0200
> 
> Arthur Miller <arthur.miller@live.com> writes:
> 
> >
> > Thanks for the answer. No idea if there is a consideration to include
> > new library or not, I got impression from some other discussions on this
> > list, it is rather not.
> 
> I also think that having a better networking library could be a great
> addition to Emacs.  But before we decide on introducing a curl or wget
> dependency, I think we should do some analysis on url.el, to be sure
> about the trade-offs of the proposed new approach to do networking,
> compared to what we already have.

It's actually more than that: to start discussing this seriously,
Someone™ should study the issue and present a design proposal: which
functions currently implemented in process.c and/or in Lisp are
proposed to be replaced or redesigned/rewritten.  It is not enough to
say let's use this networking library, because network communications
in Emacs are tightly coupled with process objects and with primitives
that perform background I/O, and also with the filter and sentinel
functions.  The best (and perhaps also the only practical) way of
using such a library is as a drop-in replacement for some functions,
but then someone should make concrete proposal regarding what is to be
replaced and how.

> - url.el has strange/buggy behavior in some scenarios.  For example,
>   callbacks are run multiple times, or not at all.  This blog post[1]
>   presents some of those problems, but it doesn't offer much details
>   about why they happen.  The blog post is from 2016, so things might
>   have changed since then.
> 
> - url.el is slow compared to using curl as an inferior process.  Again,
>   the blog post cites some possible culprits, which I don't know if they
>   still apply to Emacs 28, so I think we should profile url.el to
>   identify its bottlenecks more precisely.

Much has happened since 2016; in particular the network code is much
more asynchronous these days on modern platforms.  Not sure how that
blog measures against what we have now.

One issue that one must keep in mind when talking about external
libraries is support for non-ASCII characters: such libraries usually
support only the current locale, so if that is not UTF-8, it may mean
restrictions and subtle issues.  By contrast, Emacs's own code is
independent of the locale, and completely under control of Lisp
programs and users.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Lars Ingebrigtsen]

Daniel Martín <mardani29@yahoo.es> writes:

> - url.el is slow compared to using curl as an inferior process.  Again,
>   the blog post cites some possible culprits, which I don't know if they
>   still apply to Emacs 28, so I think we should profile url.el to
>   identify its bottlenecks more precisely.

url.el is slow (and slightly buggy and has an awkward interface); a new
implementation is on the scratch/with-url branch, which I will merge
any...  month...  now.

The other issue is that Emacs has a generally slow process interface due
to the way it's geared towards inputting artefacts from the file
systems, with all that entails for coding systems etc -- which is 100%
irrelevant for the networking protocols that are dominant today, none of
which are character oriented, but only deal with streams of octets.  (In
addition, the interfaces are slow in themselves, with process filter
functions being the primary culprit in my tests.)

These things are solvable and will result in all Emacs networking
processes not being as slow as they are now...  but the work just has to
be done.

Starting to work on this has been my priority for ... a year, but things
keep taking precedence.

I don't see adding a new C level library for HTTP/1.1 being very useful,
but for HTTP/3 it might make more sense.  Unless somebody sees
implementing HTTP/3 on the Lisp level as being a fun challenge, which
might happen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 30 Mar 2021 15:14:35 +0200
> Cc: Arthur Miller <arthur.miller@live.com>,
>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>, Daniel Brooks <db48x@db48x.net>,
>  Darshit Shah <darnir@gnu.org>, Adam Porter <adam@alphapapa.net>,
>  "T.V Raman" <raman@google.com>, "Jose E. Marchesi" <jemarch@gnu.org>
> 
> The other issue is that Emacs has a generally slow process interface due
> to the way it's geared towards inputting artefacts from the file
> systems, with all that entails for coding systems etc -- which is 100%
> irrelevant for the networking protocols that are dominant today, none of
> which are character oriented, but only deal with streams of octets.

We have the ability not to attempt decoding at all, don't we?  So why
is that still a problem?

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We have the ability not to attempt decoding at all, don't we?  So why
> is that still a problem?

I don't know that it is.  The last time I poked at this (which is a
couple years ago), I determined that the Emacs network/process handling
code was slow, but I got distracted before I found out why.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mardani29@yahoo.es,  arthur.miller@live.com,  emacs-devel@gnu.org,
>   db48x@db48x.net,  darnir@gnu.org,  adam@alphapapa.net,  raman@google.com,
>   jemarch@gnu.org
> Date: Tue, 30 Mar 2021 15:48:20 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We have the ability not to attempt decoding at all, don't we?  So why
> > is that still a problem?
> 
> I don't know that it is.  The last time I poked at this (which is a
> couple years ago), I determined that the Emacs network/process handling
> code was slow, but I got distracted before I found out why.

I'd be surprised if the reason was decoding.  I think, if it's still
slow, the reason is elsewhere.  But in any case, the effect of
decoding is simple to measure, by binding coding-system's to
'no-conversion when opening the network stream.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

I just tried

  (benchmark-run 1 (url-retrieve-synchronously
                    "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))

in Emacs, and compared that with download by wget.  On my relatively
slow home connection I get almost the same times, down to less than
2%.  On fencepost.gnu.org, which AFAIU is on the same LAN as
ftp.gnu.org, Emacs got that in 2.3 sec while wget took 1.7 sec.  Is
this what is considered "slow"?

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> On fencepost.gnu.org, which AFAIU is on the same LAN as ftp.gnu.org,
> Emacs got that in 2.3 sec while wget took 1.7 sec.  Is this what is
> considered "slow"?

Yup.  If I remember correctly, my experiments showed that Emacs uses 2x
as much time to download a file from localhost than wget (but like I
said, it's been a while, so I may be misremembering).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: arthur.miller@live.com,  mardani29@yahoo.es,  emacs-devel@gnu.org,
>   db48x@db48x.net,  darnir@gnu.org,  adam@alphapapa.net,  raman@google.com,
>   jemarch@gnu.org
> Date: Tue, 30 Mar 2021 16:34:38 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > On fencepost.gnu.org, which AFAIU is on the same LAN as ftp.gnu.org,
> > Emacs got that in 2.3 sec while wget took 1.7 sec.  Is this what is
> > considered "slow"?
> 
> Yup.  If I remember correctly, my experiments showed that Emacs uses 2x
> as much time to download a file from localhost than wget

Is fetching from localhost an interesting use case for URL retrieval?

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Clément Pit-Claudel]

On 3/30/21 10:25 AM, Eli Zaretskii wrote:
> I just tried
> 
>   (benchmark-run 1 (url-retrieve-synchronously
>                     "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
> 
> in Emacs, and compared that with download by wget.  On my relatively
> slow home connection I get almost the same times, down to less than
> 2%.

On my decently fast home connection, Emacs takes 4.5s-5s to download that file using the form you posted, and wget takes 1.4s-2s to download the same file:

(benchmark-run 1 (url-retrieve-synchronously "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
⇒ (4.875848555 13 3.7572300619999623)

(benchmark-run 1 (shell-command-to-string "wget https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
⇒ (1.5106374 0 0.0)

Clément.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 30 Mar 2021 12:13:55 -0400
> 
> >   (benchmark-run 1 (url-retrieve-synchronously
> >                     "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
> > 
> > in Emacs, and compared that with download by wget.  On my relatively
> > slow home connection I get almost the same times, down to less than
> > 2%.
> 
> On my decently fast home connection, Emacs takes 4.5s-5s to download that file using the form you posted, and wget takes 1.4s-2s to download the same file:
> 
> (benchmark-run 1 (url-retrieve-synchronously "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
> ⇒ (4.875848555 13 3.7572300619999623)

But note that the actual Emacs download time was 4.876 - 3.757 = 1.319
sec, the rest was taken by GC.  What happens if you set
gc-cons-threshold to a very large value?

Also, nowadays we have a variable to tune how much stuff do we read in
one go (by default just 4K), so might as well test the effect of that
on download times.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Clément Pit-Claudel]

On 3/30/21 12:21 PM, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 30 Mar 2021 12:13:55 -0400
>>
>>>   (benchmark-run 1 (url-retrieve-synchronously
>>>                     "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
>>>
>>> in Emacs, and compared that with download by wget.  On my relatively
>>> slow home connection I get almost the same times, down to less than
>>> 2%.
>>
>> On my decently fast home connection, Emacs takes 4.5s-5s to download that file using the form you posted, and wget takes 1.4s-2s to download the same file:
>>
>> (benchmark-run 1 (url-retrieve-synchronously "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz"))
>> ⇒ (4.875848555 13 3.7572300619999623)
> 
> But note that the actual Emacs download time was 4.876 - 3.757 = 1.319
> sec, the rest was taken by GC.  What happens if you set
> gc-cons-threshold to a very large value?

Good point:

(let ((gc-cons-threshold (expt 2 32)))
  (benchmark-run 1 (url-retrieve-synchronously "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz")))
⇒ (3.181631628 0 0.0)
⇒ (1.46930467 0 0.0)

> Also, nowadays we have a variable to tune how much stuff do we read in
> one go (by default just 4K), so might as well test the effect of that
> on download times.

You're completely right:

(let ((gc-cons-threshold (expt 2 32))
      (read-process-output-max (expt 2 22)))
  (benchmark-run 1 (url-retrieve-synchronously
                      "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz")))
⇒ (1.446506607 0 0.0)
⇒ (2.441543274 0 0.0)

Adding back GC:

(let ((read-process-output-max (expt 2 22)))
  (benchmark-run 1 (url-retrieve-synchronously
                      "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz")))
⇒ (2.719771409 7 1.9299083920000157)

Although, to be fair, I'm not sure what to make of these numbers.  With the GC change only, for example, Emacs reports 1.45s after freezing for 7 to 8 seconds, not 1.45 — I guess that's because the GC runs after the benchmarking completes?

Clément.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 30 Mar 2021 12:49:25 -0400
> 
> (let ((read-process-output-max (expt 2 22)))
>   (benchmark-run 1 (url-retrieve-synchronously
>                       "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz")))
> ⇒ (2.719771409 7 1.9299083920000157)
> 
> Although, to be fair, I'm not sure what to make of these numbers.  With the GC change only, for example, Emacs reports 1.45s after freezing for 7 to 8 seconds, not 1.45 — I guess that's because the GC runs after the benchmarking completes?

Yes, that's what I'd expect.

So it seems like Emacs networking is not so slow, at least in this
simple benchmark.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I don't see adding a new C level library for HTTP/1.1 being very useful,
> but for HTTP/3 it might make more sense.  Unless somebody sees
> implementing HTTP/3 on the Lisp level as being a fun challenge, which
> might happen.

Seems like libcurl does (or will) support http/3.

https://github.com/curl/curl/blob/master/docs/HTTP3.md

Don't know how experiemntal it is.

What about other protocols it supports, is that of interest to have in
Emacs?

Here seems to be a list of supported ones:

https://everything.curl.dev/protocols/curl

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Clément Pit-Claudel]

On 3/30/21 1:03 PM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 30 Mar 2021 12:49:25 -0400
>>
>> (let ((read-process-output-max (expt 2 22)))
>>   (benchmark-run 1 (url-retrieve-synchronously
>>                       "https://ftp.gnu.org/gnu/emacs/emacs-27.2.tar.xz")))
>> ⇒ (2.719771409 7 1.9299083920000157)
>>
>> Although, to be fair, I'm not sure what to make of these numbers.  With the GC change only, for example, Emacs reports 1.45s after freezing for 7 to 8 seconds, not 1.45 — I guess that's because the GC runs after the benchmarking completes?
> 
> Yes, that's what I'd expect.
> 
> So it seems like Emacs networking is not so slow, at least in this
> simple benchmark.

Not if you turn off garbage collection, indeed (except for the fact that you pay for it later on), but wouldn't a native library remove that GC pressure?

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 231 bytes --]

This is useful.

While all other things get discussed, could we perhaps have url.el be
updated to locally bind read-process-output-max to a larger value, say
1mb?

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Stefan Monnier]

> Not if you turn off garbage collection, indeed (except for the fact that you
> pay for it later on), but wouldn't a native library remove that GC pressure?

Not clear.  There's a good chance that the GC pressure is due to the
strings generated and passed to the process filter and then inserted in
a temp buffer.

So there's a chance that even going through a super-efficient C library,
as long as the end result is returned via output sent to a process
filter for insertion into a temp buffer the result won't necessarily be
much better.


        Stefan "who does think that we'll want to reuse an existing
                HTTP/3 library than try and implement it ourselves in
                ELisp"

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 30 Mar 2021 15:53:21 -0400
> 
> > So it seems like Emacs networking is not so slow, at least in this
> > simple benchmark.
> 
> Not if you turn off garbage collection, indeed (except for the fact that you pay for it later on), but wouldn't a native library remove that GC pressure?

I don't see how: eventually, you'd need to cons Lisp objects from the
stuff received from the wire, this way or another.

In any case, if the conclusion is that the problem is GC (or maybe
something related, like reallocating the buffer into which we
accumulate the received stuff), we can probably find an easy enough
solution.  But I'm not sure yet this is the problem.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 758 bytes --]

> From: "T.V Raman" <raman@google.com>
> Cc: Cl¨¦ment Pit-Claudel <cpitclaudel@gmail.com>,
>   emacs-devel@gnu.org
> Date: Tue, 30 Mar 2021 13:53:02 -0700
> 
> While all other things get discussed, could we perhaps have url.el be
> updated to locally bind read-process-output-max to a larger value, say
> 1mb?

I don't think this would be TRT, because that variable affects all the
communications with all the subprocesses running at that time.

To do something less blunt, we need a per-process attribute.  But that
is only justified if indeed the effect in most scenarios related to
url.el is profoundly positive.  Just seeing the improvement in one
synthetic test case is not enough for drawing project-wide conclusions
about the default behavior, IMO.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 30 Mar 2021 21:08:43 -0400
> 
> > Not if you turn off garbage collection, indeed (except for the fact that you
> > pay for it later on), but wouldn't a native library remove that GC pressure?
> 
> Not clear.  There's a good chance that the GC pressure is due to the
> strings generated and passed to the process filter and then inserted in
> a temp buffer.
> 
> So there's a chance that even going through a super-efficient C library,
> as long as the end result is returned via output sent to a process
> filter for insertion into a temp buffer the result won't necessarily be
> much better.

We could add a special kind of filter that puts the text directly into
a buffer, without going through a string.  In that special filter, we
could make arrangements to preallocate a large enough gap for that
buffer, so that repeated reallocations won't get in the way, either.
Then GC won't be a problem even when doing this entirely in Emacs
code.

But I think this all is premature optimization.  We have only one
synthetic use case; we should base our design decisions on more solid
grounds, after looking into more uses and analyzing the reason(s) for
the slowness.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Robert Pluim]

>>>>> On Tue, 30 Mar 2021 20:08:20 +0200, Arthur Miller <arthur.miller@live.com> said:

    Arthur> Lars Ingebrigtsen <larsi@gnus.org> writes:
    >> I don't see adding a new C level library for HTTP/1.1 being very useful,
    >> but for HTTP/3 it might make more sense.  Unless somebody sees
    >> implementing HTTP/3 on the Lisp level as being a fun challenge, which
    >> might happen.

    Arthur> Seems like libcurl does (or will) support http/3.

    Arthur> https://github.com/curl/curl/blob/master/docs/HTTP3.md

    Arthur> Don't know how experiemntal it is.

    Arthur> What about other protocols it supports, is that of interest to have in
    Arthur> Emacs?

    Arthur> Here seems to be a list of supported ones:

    Arthur> https://everything.curl.dev/protocols/curl

Iʼm curious, what does 'support' mean in this case? I mean, SMTP or
IMAP is just TCP (perhaps with TLS). You still need the emacs client
code to actually issue SMTP or IMAP commands and interpret them.

If there was a way to use libcurl to provide a generic tcp/tls stream,
that could be useful, and remove the requirement to update our GnuTLS code,
but as Eli pointed out elsewhere thatʼs not a small change.

Robert
-- 

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Is fetching from localhost an interesting use case for URL retrieval?

Not particularly, except as a benchmark.  The interesting thing is other
local communication (like with an LSP server), and fetching things remotely.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

read-process-output-max (was: Emacs HTTP libraries)

[Stefan Monnier]

>> While all other things get discussed, could we perhaps have url.el be
>> updated to locally bind read-process-output-max to a larger value, say
>> 1mb?
>
> I don't think this would be TRT, because that variable affects all the
> communications with all the subprocesses running at that time.

BTW, what are the factors at play here?
I understand there is a per-chunk overhead which pushes towards a higher
value, but I'm not sure what factors are pushing towards smaller values.
I can imagine that a 1MB buffer could be considered more than useful
(hence wasteful) but I don't immediately see what would be the downside
of setting it to 64kB instead of 4096B.

The docstring seems to strongly recommend not to increase it, so
I suspect there's a much stronger reason that "not be wasteful" in play.


        Stefan

Re: read-process-output-max (was: Emacs HTTP libraries)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: "T.V Raman" <raman@google.com>,  cpitclaudel@gmail.com,
>   emacs-devel@gnu.org
> Date: Wed, 31 Mar 2021 12:01:35 -0400
> 
> I understand there is a per-chunk overhead which pushes towards a higher
> value, but I'm not sure what factors are pushing towards smaller values.

Values larger that some threshold will cause us use malloc instead of
alloca, that's one reason to use smaller buffers for performance
reasons.

Another potential issue, in programs that display the arriving stuff
as it is received, is that showing the text in small chunks is
generally better (UX-wise) than blasting a 1MB chunk of text to the
screen in one go.

> The docstring seems to strongly recommend not to increase it, so
> I suspect there's a much stronger reason that "not be wasteful" in play.

The effect of the value was never studied in detail, and so the
optimal value could be different from the default.  The doc string
says what it says to prevent people from getting themselves into
trouble when they don't really need to change the value for their
application: after all, the default value was used for many years, and
should be considered safe enough.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Daniel Brooks]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 30 Mar 2021 20:08:20 +0200, Arthur Miller <arthur.miller@live.com> said:
>     Arthur> What about other protocols it supports, is that of interest to have in
>     Arthur> Emacs?
>
>     Arthur> Here seems to be a list of supported ones:
>
>     Arthur> https://everything.curl.dev/protocols/curl
>
> Iʼm curious, what does 'support' mean in this case? I mean, SMTP or
> IMAP is just TCP (perhaps with TLS). You still need the emacs client
> code to actually issue SMTP or IMAP commands and interpret them.

Give this a whirl, with appropriate subtitutions:

    curl "imap://<username>:<password>@<your imap server>/INBOX;UID=<NNNN>"

It’s a little bit odd, but I guess someone was feeling completionist
that day.

db48x

Re: read-process-output-max

[Stefan Monnier]

>> I understand there is a per-chunk overhead which pushes towards a higher
>> value, but I'm not sure what factors are pushing towards smaller values.
> Values larger that some threshold will cause us use malloc instead of
> alloca, that's one reason to use smaller buffers for performance
> reasons.

That's indeed a good reason not to push the limit to far up, indeed.

> Another potential issue, in programs that display the arriving stuff
> as it is received, is that showing the text in small chunks is
> generally better (UX-wise) than blasting a 1MB chunk of text to the
> screen in one go.

If 64kB already arrived, will part of the result really reach the eyes
of the user faster if we process them as 16 chunks of 4kB or do we skip
redisplay between the chunks (just like we skip redisplay when there
are pending input events)?

> The effect of the value was never studied in detail, and so the
> optimal value could be different from the default.  The doc string
> says what it says to prevent people from getting themselves into
> trouble when they don't really need to change the value for their
> application: after all, the default value was used for many years, and
> should be considered safe enough.

Makes sense, indeed.  It might be worth having people play with the
value for a while (and not just for bulk-bandwidth tests but also for
interactive use like `M-x shell`, `M-x compile`, `M-x grep`, including
with largish outputs) and report back.

I just put (setq read-process-output-max (max 65536 read-process-output-max))
into my init file, and I'll see if I notice a difference.
[ BTW, I think a value >=32kB would be desirable since it avoids
  breaking UDP datagrams.  ]


        Stefan

Re: read-process-output-max

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: raman@google.com,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 31 Mar 2021 19:05:59 -0400
> 
> > Another potential issue, in programs that display the arriving stuff
> > as it is received, is that showing the text in small chunks is
> > generally better (UX-wise) than blasting a 1MB chunk of text to the
> > screen in one go.
> 
> If 64kB already arrived, will part of the result really reach the eyes
> of the user faster if we process them as 16 chunks of 4kB or do we skip
> redisplay between the chunks (just like we skip redisplay when there
> are pending input events)?

The result of inserting large chunks of text into a buffer that's
displayed will be a continuously scrolling window, whereas if we do
that in smaller chunks, the text will be static for long enough to
allow at least some of it being read (at least with the default
scrolling behavior).  I think the latter is less annoying, at least in
some use cases (think the likes of "M-x compile" and "M-x grep").

> [ BTW, I think a value >=32kB would be desirable since it avoids
>   breaking UDP datagrams.  ]

If we want to do that, we should bump the alloca threshold in
SAFE_ALLOCA higher, it is currently set at 16KB.

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Robert Pluim]

>>>>> On Wed, 31 Mar 2021 10:21:09 -0700, Daniel Brooks <db48x@db48x.net> said:

    Daniel> Robert Pluim <rpluim@gmail.com> writes:
    >>>>>>> On Tue, 30 Mar 2021 20:08:20 +0200, Arthur Miller <arthur.miller@live.com> said:
    Arthur> What about other protocols it supports, is that of interest to have in
    Arthur> Emacs?
    >> 
    Arthur> Here seems to be a list of supported ones:
    >> 
    Arthur> https://everything.curl.dev/protocols/curl
    >> 
    >> Iʼm curious, what does 'support' mean in this case? I mean, SMTP or
    >> IMAP is just TCP (perhaps with TLS). You still need the emacs client
    >> code to actually issue SMTP or IMAP commands and interpret them.

    Daniel> Give this a whirl, with appropriate subtitutions:

    Daniel>     curl "imap://<username>:<password>@<your imap server>/INBOX;UID=<NNNN>"

    Daniel> It’s a little bit odd, but I guess someone was feeling completionist
    Daniel> that day.

OK, so itʼs "pretend IMAP is a weird kind of http". Finding out what
UID to use requires a little more work.

Robert
-- 

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Daniel Brooks]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Wed, 31 Mar 2021 10:21:09 -0700, Daniel Brooks <db48x@db48x.net> said:
>     Daniel>     curl "imap://<username>:<password>@<your imap server>/INBOX;UID=<NNNN>"
>
> OK, so itʼs "pretend IMAP is a weird kind of http". Finding out what
> UID to use requires a little more work.

That’s right. Useful if you find yourself needing to automate an email
task using shell scripts, but not much more. There are nicer ways to
write that same automation inside Emacs already, so I doubt there would
be much call for it. On the other hand, I’m sure someone can think of a
use for the RTSP support.

Similarly, there are features that libwget has which can already be done
in Emacs, such as parsing HTML and CSS.

db48x

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Thu, Apr 01, 2021 at 04:23:52PM +0200, Robert Pluim wrote:
> >>>>> On Wed, 31 Mar 2021 10:21:09 -0700, Daniel Brooks <db48x@db48x.net> said:

[...]

>     Daniel> It’s a little bit odd, but I guess someone was feeling completionist
>     Daniel> that day.
> 
> OK, so itʼs "pretend IMAP is a weird kind of http". Finding out what
> UID to use requires a little more work.

Well, curl speaks IMAP (among quite a shovelful of other protocols).

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Sv: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Robert Pluim]

>>>>> On Thu, 01 Apr 2021 09:09:20 -0700, Daniel Brooks <db48x@db48x.net> said:

    Daniel> Similarly, there are features that libwget has which can already be done
    Daniel> in Emacs, such as parsing HTML and CSS.

For html we already punt to libxml :-)

Robert
--