bug#19556: eww: make URI rewriting fully customizable

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

Package:  emacs
Severity: minor

	As of 88bc8332eb14 (2015-01-07 13:51:41 +0000), the ‘eww’
	command follows a hard-coded set of rules to rewrite the
	argument given into a URI suitable for url-retrieve.

	I hereby suggest to make these rules fully customizable, per the
	patch MIMEd.

	* lisp/net/eww.el
	(eww-uri-rewrite-hook): New customizable variable.
	(eww): Use eww-uri-rewrite-hook.
	(eww-search-words): Use eww-uri-search.
	(eww-uri-file, eww-uri-not-supported, eww-uri-search)
	(eww-uri-remote-prepend-http): New functions, split off eww.

	FWIW, removing eww-search-words from the hook allows one to type
	something like https://en.wikipedia.org/wiki/Free software RET
	(note the blank) at the prompt and be with that.

	On top of this, eww-uri-file may then be changed slightly to
	deal with #19472.

-- 
FSF associate member #7257  np. Tree of Love — Jami Sieber   … B6A0 230E 334A

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/diff, Size: 4491 bytes --]

--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -75,6 +75,26 @@ defcustom eww-suggest-uris
 	     url-get-url-at-point
 	     eww-current-url))
 
+(defcustom eww-uri-rewrite-hook
+  '(eww-uri-file
+    eww-uri-not-supported
+    eww-uri-remote-prepend-http
+    eww-uri-search)
+  "List of functions called to deal with the argument to `eww'.
+These functions will be called in order, with the argument given to
+`eww' passed as their only argument, until one returns non-nil.  The
+value returned will be used as the URI to fetch.
+
+Should no function of those listed return non-nil, `eww' will use its
+argument unaltered."
+  :version "25.1"
+  :group 'eww
+  :type 'hook
+  :options '(eww-uri-file
+	     eww-uri-not-supported
+	     eww-uri-remote-prepend-http
+	     eww-uri-search))
+
 (defcustom eww-bookmarks-directory user-emacs-directory
   "Directory where bookmark files will be stored."
   :version "25.1"
@@ -250,23 +270,9 @@ defun eww (url)
 			  ": ")))
      (list (read-string prompt nil nil uris))))
   (setq url (string-trim url))
-  (cond ((string-match-p "\\`file:/" url))
-	;; Don't mangle file: URLs at all.
-        ((string-match-p "\\`ftp://" url)
-         (user-error "FTP is not supported."))
-        (t
-         (if (and (= (length (split-string url)) 1)
-		  (or (and (not (string-match-p "\\`[\"\'].*[\"\']\\'" url))
-			   (> (length (split-string url "[.:]")) 1))
-		      (string-match eww-local-regex url)))
-             (progn
-               (unless (string-match-p "\\`[a-zA-Z][-a-zA-Z0-9+.]*://" url)
-                 (setq url (concat "http://" url)))
-               ;; some site don't redirect final /
-               (when (string= (url-filename (url-generic-parse-url url)) "")
-                 (setq url (concat url "/"))))
-           (setq url (concat eww-search-prefix
-                             (replace-regexp-in-string " " "+" url))))))
+  (when-let ((new (run-hook-with-args-until-success
+		   'eww-uri-rewrite-hook url)))
+    (setq url new))
   (if (eq major-mode 'eww-mode)
       (when (or (plist-get eww-data :url)
 		(plist-get eww-data :dom))
@@ -292,11 +299,11 @@ defun eww-open-file (file)
 	       (expand-file-name file))))
 
 ;;;###autoload
-(defun eww-search-words (&optional beg end)
+(defun eww-search-words (beg end)
   "Search the web for the text between the point and marker.
 See the `eww-search-prefix' variable for the search engine used."
   (interactive "r")
-  (eww (buffer-substring beg end)))
+  (eww (eww-uri-search (buffer-substring beg end))))
 
 (defun eww-render (status url &optional point buffer encode)
   (let ((redirect (plist-get status :redirect)))
@@ -551,6 +559,46 @@ defun eww-links-at-point (&optional pt)
 	(list (get-text-property (point) 'shr-url)
 	      (get-text-property (point) 'image-url))))
 
+(defun eww-uri-file (uri)
+  "Return URI unaltered if it looks like a valid file: scheme URI."
+  (and (string-match-p "\\`file:/" uri)
+       uri))
+
+(defun eww-uri-not-supported (uri)
+  "Raise an error if URI uses the ftp: scheme."
+  (when (string-match-p "\\`ftp://" url)
+    (user-error "FTP is not supported.")))
+
+(defun eww-uri-remote-prepend-http (uri)
+  "Return URI if it looks like a reference to a remote resource.
+Specifically, return URI if it contains no whitespace characters
+ (except for leading and trailing whitespace, which is ignored) and
+either matches `eww-local-regex', or contains a . or :, but no single-
+or double-quoted parts.
+
+Prepend http:// to URI unless it already has a scheme."
+  (when (and (= (length (split-string uri)) 1)
+	     (or (and (not (string-match-p "\\`[\"\'].*[\"\']\\'" uri))
+		      (> (length (split-string uri "[.:]")) 1))
+		 (string-match eww-local-regex uri)))
+    (unless (string-match-p "\\`[a-zA-Z][-a-zA-Z0-9+.]*://" uri)
+      (setq uri (concat "http://" uri)))
+    (setq uri (replace-regexp-in-string " " "%20" uri))
+    ;; Some sites don't redirect final /.
+    (if (string= (url-filename (url-generic-parse-url uri)) "")
+	(concat uri "/")
+      uri)))
+
+(defun eww-uri-search (keywords)
+  "Return an URI for the search given by KEYWORDS.
+KEYWORDS may be either a list or a string.
+
+See the `eww-search-prefix' variable for the search engine used."
+  (concat eww-search-prefix
+	  (if (listp keywords)
+	      (mapconcat #'identity keywords "+")
+	    (subst-char-in-string 32 ?+ keywords))))
+
 (defun eww-view-source ()
   "View the HTML source code of the current page."
   (interactive)

bug#19556: eww: make URI rewriting fully customizable

[Eli Zaretskii]

> From: Ivan Shmakov <ivan@siamics.net>
> Date: Sat, 10 Jan 2015 13:17:55 +0000
> 
> +(defcustom eww-uri-rewrite-hook
> +  '(eww-uri-file
> +    eww-uri-not-supported
> +    eww-uri-remote-prepend-http
> +    eww-uri-search)
> +  "List of functions called to deal with the argument to `eww'.
> +These functions will be called in order, with the argument given to
> +`eww' passed as their only argument, until one returns non-nil.  The
> +value returned will be used as the URI to fetch.
> +
> +Should no function of those listed return non-nil, `eww' will use its
> +argument unaltered."
> +  :version "25.1"
> +  :group 'eww
> +  :type 'hook
> +  :options '(eww-uri-file
> +	     eww-uri-not-supported
> +	     eww-uri-remote-prepend-http
> +	     eww-uri-search))

Why put the "standard" rules into the defcustom? why not leave them in
place as "plan B", and leave the hook for customizations only?  That's
what hooks are normally for -- _modifying_ the default behavior, not
supplanting it.  E.g., with your suggestion, what happens if someone
customizes the value to nil?

Thanks.

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>> From: Ivan Shmakov  Date: Sat, 10 Jan 2015 13:17:55 +0000

[…]

 > Why put the "standard" rules into the defcustom?

	So to make the tricks played by EWW on unsuspecting URIs more
	obvious to the user.  Cf. eww-suggest-uris, BTW.

 > why not leave them in place as "plan B", and leave the hook for
 > customizations only?

	It’s certainly a possibility; frankly, I have no strong
	preference for doing it either way.

 > That's what hooks are normally for -- _modifying_ the default
 > behavior, not supplanting it.

	Are they?  For instance, when run with -Q, my find-file-hooks
	includes ange-ftp-set-buffer-mode, epa-file-find-file-hook,
	vc-find-file-hook, – which I’d consider pretty much a
	prerequisite for the “default” behavior.

	With these functions being explicitly listed, however, I could
	easily drop everything but vc-find-file-hook off the list to get
	rid of the functionality that tends to get in my way.

 > E. g., with your suggestion, what happens if someone customizes the
 > value to nil?

	That’s simple: M-x eww will assume that the URIs it’s given
	never need any special treatment.  One consequence I’ve already
	mentioned:

 >> FWIW, removing eww-search-words from the hook allows one to type
 >> something like https://en.wikipedia.org/wiki/Free software RET (note
 >> the blank) at the prompt and be with that.

	Removing just eww-search-words leads to more or less the same
	effect.  Dropping eww-uri-not-supported resolves #17959.  And
	so on.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Eli Zaretskii]

> From: Ivan Shmakov <ivan@siamics.net>
> Date: Sat, 10 Jan 2015 14:40:57 +0000
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> >>>>> From: Ivan Shmakov  Date: Sat, 10 Jan 2015 13:17:55 +0000
> 
> […]
> 
>  > Why put the "standard" rules into the defcustom?
> 
> 	So to make the tricks played by EWW on unsuspecting URIs more
> 	obvious to the user.

How does that make it more obvious?

>  > That's what hooks are normally for -- _modifying_ the default
>  > behavior, not supplanting it.
> 
> 	Are they?  For instance, when run with -Q, my find-file-hooks
> 	includes ange-ftp-set-buffer-mode, epa-file-find-file-hook,
> 	vc-find-file-hook

None of these replace the default behavior of their hookee functions.
They _add_ something to it.  The default behavior is not in the hook,
it's in the code that calls the hook.

And anyway, I said "normally", so a couple of examples to the contrary
doesn't yet invalidate what I said.

> 	With these functions being explicitly listed, however, I could
> 	easily drop everything but vc-find-file-hook off the list to get
> 	rid of the functionality that tends to get in my way.

It sounds like we have some deep disagreement about what the hooks are
for.  You want a hook to _remove_ some behaviors, but that's not what
hooks are for.

> 
>  > E. g., with your suggestion, what happens if someone customizes the
>  > value to nil?
> 
> 	That’s simple: M-x eww will assume that the URIs it’s given
> 	never need any special treatment.

Which is clearly wrong with some URIs.

Please let's not go that way.

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>> From: Ivan Shmakov  Date: Sat, 10 Jan 2015 14:40:57 +0000
>>>>> Eli Zaretskii <eliz@gnu.org> writes:

 >>> Why put the "standard" rules into the defcustom?

 >> So to make the tricks played by EWW on unsuspecting URIs more
 >> obvious to the user.

 > How does that make it more obvious?

	By showing them in the “Customize Group” buffer?

 >>> That's what hooks are normally for -- _modifying_ the default
 >>> behavior, not supplanting it.

 >> Are they?  For instance, when run with -Q, my find-file-hooks
 >> includes ange-ftp-set-buffer-mode, epa-file-find-file-hook,
 >> vc-find-file-hook

 > None of these replace the default behavior of their hookee functions.
 > They _add_ something to it.  The default behavior is not in the hook,
 > it's in the code that calls the hook.

	We seem to be disagreeing on the terms.  To me, the way
	unmodified Emacs behaves when started with ‘-Q’ is the default.

	To get different (custom) behavior, the user may add elements to
	hooks, lists of other kinds (including associative), keymaps, –
	or /remove/ from them; load or /unload/ packages; add, redefine
	or advise commands or functions; etc.

	For instance, electric-indent-mode is the default.  Using
	(electric-indent-mode -1) in one’s ~/.emacs is a customization.
	Calling log-edit-insert-message-template as part of
	log-edit-hook is the default.  Setting log-edit-hook to nil is a
	(perfectly valid) customization.  Providing a rich set of
	filename suffix to Emacs major mode pairs in auto-mode-alist is
	the default.  Tailoring that alist according to the modes one
	does and does not use is customization.  Etc.

	I see no reason for EWW to be different in that its default
	behavior cannot be customized – whether that’d mean adding
	elements to some list, removing others from there, or doing
	something else.

 > And anyway, I said "normally", so a couple of examples to the
 > contrary doesn't yet invalidate what I said.

	Should I look for a couple more examples on top of that couple
	within the Emacs source tree just for the sake of this argument?

 >> With these functions being explicitly listed, however, I could
 >> easily drop everything but vc-find-file-hook off the list to get rid
 >> of the functionality that tends to get in my way.

 > It sounds like we have some deep disagreement about what the hooks
 > are for.  You want a hook to _remove_ some behaviors, but that's not
 > what hooks are for.

	Except for those which are.

	Besides, what’d be the point of run-hook-with-args-until-success
	if not to allow the user to prepend his or her own function to
	override the behavior of the (default) functions coming later in
	that same hook?

 >>> E. g., with your suggestion, what happens if someone customizes the
 >>> value to nil?

 >> That’s simple: M-x eww will assume that the URIs it’s given never
 >> need any special treatment.

 > Which is clearly wrong with some URIs.

	I see no way it could be wrong for /URIs proper./  Yes, that’d
	mean that M-x eww RET mary had a little lamb RET fails.  But so
	does Lynx, Wget and Curl, url-retrieve, C-x C-f, and what not.

	On the contrary, I find the eagerness of EWW to send what could
	easily be a mistyped URI to a third party somewhat disturbing.

	Customizing eww-search-prefix to refer to some non-existing
	service (or just setting it to a non-nil symbol) may be a decent
	work-around for the issue, but it also disables eww-search-words
	as an unfortunate side-effect.

 > Please let's not go that way.

	Sure.  Let’s instead make C-x C-f go DDG for completion and
	provide no option for the user to opt-out.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Eli Zaretskii]

> From: Ivan Shmakov <ivan@siamics.net>
> Date: Sat, 10 Jan 2015 17:20:42 +0000
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> >>>>> From: Ivan Shmakov  Date: Sat, 10 Jan 2015 14:40:57 +0000
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
>  >>> Why put the "standard" rules into the defcustom?
> 
>  >> So to make the tricks played by EWW on unsuspecting URIs more
>  >> obvious to the user.
> 
>  > How does that make it more obvious?
> 
> 	By showing them in the “Customize Group” buffer?

That buffer is too large to make anything obvious.

>  >>> That's what hooks are normally for -- _modifying_ the default
>  >>> behavior, not supplanting it.
> 
>  >> Are they?  For instance, when run with -Q, my find-file-hooks
>  >> includes ange-ftp-set-buffer-mode, epa-file-find-file-hook,
>  >> vc-find-file-hook
> 
>  > None of these replace the default behavior of their hookee functions.
>  > They _add_ something to it.  The default behavior is not in the hook,
>  > it's in the code that calls the hook.
> 
> 	We seem to be disagreeing on the terms.  To me, the way
> 	unmodified Emacs behaves when started with ‘-Q’ is the default.

We are not talking about the Emacs behavior, we are talking about the
behavior of a certain function.  Things that are central to that
function's job are that function's default behavior.  find-file's job
is to find and visit files, and that functionality should not be in a
hook, available for removal by users.

Similarly, eww's job is to find and visit a URL, which includes
processing file:// URLs, and that functionality should always be in
the function, not in any hook.

> 	I see no reason for EWW to be different in that its default
> 	behavior cannot be customized – whether that’d mean adding
> 	elements to some list, removing others from there, or doing
> 	something else.

It makes very little sense to me to customize out something that is an
inherent part of a function's job.

>  > And anyway, I said "normally", so a couple of examples to the
>  > contrary doesn't yet invalidate what I said.
> 
> 	Should I look for a couple more examples on top of that couple
> 	within the Emacs source tree just for the sake of this argument?

If you wish.  But I'd rather you invested your time and talent where
it could benefit Emacs more than by looking for those examples.

>  > It sounds like we have some deep disagreement about what the hooks
>  > are for.  You want a hook to _remove_ some behaviors, but that's not
>  > what hooks are for.
> 
> 	Except for those which are.
> 
> 	Besides, what’d be the point of run-hook-with-args-until-success
> 	if not to allow the user to prepend his or her own function to
> 	override the behavior of the (default) functions coming later in
> 	that same hook?

Hooks do not override behavior.  They let the user run some code
_in_addition_ to the normal behavior.  It's therefore no accident that
a hook is called near the end of the function which provides it, after

To _override_ behavior, we provide function variables that get used
_instead_ of the default.  Like indent-region-function, for example.
So in that case, you'd need to make 'eww' call the value of
eww-mangle-uri-function, whose default value is a function that does
what the current code does inside eww itself, and then users could
override that by providing their own URI-mangling function.

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

[…]

 > Similarly, eww's job is to find and visit a URL, which includes
 > processing file:// URLs, and that functionality should always be in
 > the function, not in any hook.

	file: URIs’ processing is the responsibility of url-file.el.
	There’s nothing in ‘eww’ which ever deals with such URIs, –
	/except/ for a simple guard which tells: “if it begins with
	‘file:/’, do /not/ never, ever mangle it.”

	Personally, I’d rather appreciate having such a guard for https:
	and http: URIs at the least, – if only for the sake of
	consistency.

[…]

 > To _override_ behavior, we provide function variables that get used
 > _instead_ of the default.  Like indent-region-function, for example.
 > So in that case, you'd need to make 'eww' call the value of
 > eww-mangle-uri-function, whose default value is a function that does
 > what the current code does inside eww itself, and then users could
 > override that by providing their own URI-mangling function.

	Please consider the patch MIMEd.

	* lisp/net/eww.el
	(eww-uri-rewrite-function): New customizable variable.
	(eww): Use eww-uri-rewrite-function.
	(eww-mangle-uri): New function, split off eww.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/diff, Size: 3192 bytes --]

--- a/lisp/net/eww.el	2015-01-10 16:26:37 +0000
+++ b/lisp/net/eww.el	2015-01-10 17:59:44 +0000
@@ -75,6 +75,15 @@
 	     url-get-url-at-point
 	     eww-current-url))
 
+(defcustom eww-uri-rewrite-function
+  'eww-uri-rewrite
+  "A function to call to pre-process the argument to `eww'."
+  :version "25.1"
+  :group 'eww
+  :type 'hook
+  :options '(eww-uri-rewrite
+	     identity))
+
 (defcustom eww-bookmarks-directory user-emacs-directory
   "Directory where bookmark files will be stored."
   :version "25.1"
@@ -249,28 +258,7 @@
 			  (if uris (format " (default %s)" (car uris)) "")
 			  ": ")))
      (list (read-string prompt nil nil uris))))
-  (setq url (string-trim url))
-  (cond ((string-match-p "\\`file:/" url))
-	;; Don't mangle file: URLs at all.
-        ((string-match-p "\\`ftp://" url)
-         (user-error "FTP is not supported."))
-        (t
-         (if (or (string-match "\\`https?:" url)
-		 ;; Also try to match "naked" URLs like
-		 ;; en.wikipedia.org/wiki/Free software
-		 (string-match "\\`[A-Za-z_]+\\.[A-Za-z._]+/" url)
-		 (and (= (length (split-string url)) 1)
-		      (or (and (not (string-match-p "\\`[\"\'].*[\"\']\\'" url))
-			       (> (length (split-string url "[.:]")) 1))
-			  (string-match eww-local-regex url))))
-             (progn
-               (unless (string-match-p "\\`[a-zA-Z][-a-zA-Z0-9+.]*://" url)
-                 (setq url (concat "http://" url)))
-               ;; Some sites do not redirect final /
-               (when (string= (url-filename (url-generic-parse-url url)) "")
-                 (setq url (concat url "/"))))
-           (setq url (concat eww-search-prefix
-                             (replace-regexp-in-string " " "+" url))))))
+  (setq url (funcall eww-uri-rewrite-function (string-trim url)))
   (if (eq major-mode 'eww-mode)
       (when (or (plist-get eww-data :url)
 		(plist-get eww-data :dom))
@@ -556,6 +545,31 @@
 	(list (get-text-property (point) 'shr-url)
 	      (get-text-property (point) 'image-url))))
 
+(defun eww-uri-rewrite (url)
+  (cond ((string-match-p "\\`file:/" url))
+	;; Don't mangle file: URLs at all.
+        ((string-match-p "\\`ftp://" url)
+         (user-error "FTP is not supported."))
+        (t
+         (if (or (string-match "\\`https?:" url)
+		 ;; Also try to match "naked" URLs like
+		 ;; en.wikipedia.org/wiki/Free software
+		 (string-match "\\`[A-Za-z_]+\\.[A-Za-z._]+/" url)
+		 (and (= (length (split-string url)) 1)
+		      (or (and (not (string-match-p "\\`[\"\'].*[\"\']\\'" url))
+			       (> (length (split-string url "[.:]")) 1))
+			  (string-match eww-local-regex url))))
+             (progn
+               (unless (string-match-p "\\`[a-zA-Z][-a-zA-Z0-9+.]*://" url)
+                 (setq url (concat "http://" url)))
+               ;; Some sites do not redirect final /
+               (when (string= (url-filename (url-generic-parse-url url)) "")
+                 (setq url (concat url "/"))))
+           (setq url (concat eww-search-prefix
+                             (replace-regexp-in-string " " "+" url))))))
+  ;; .
+  url)
+
 (defun eww-view-source ()
   "View the HTML source code of the current page."
   (interactive)

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Ivan Shmakov <ivan@siamics.net> writes:

> 	FWIW, removing eww-search-words from the hook allows one to type
> 	something like https://en.wikipedia.org/wiki/Free software RET
> 	(note the blank) at the prompt and be with that.

That sounds like a bug that should be fixed.

I don't see a use case for letting users customise this stuff, and it
sounds like it would be error-prone.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ivan Shmakov <ivan@siamics.net> writes:

 >> FWIW, removing eww-search-words from the hook allows one to type
 >> something like https://en.wikipedia.org/wiki/Free software RET (note
 >> the blank) at the prompt and be with that.

 > That sounds like a bug that should be fixed.

 > I don't see a use case for letting users customise this stuff,

	How about allowing for out-of-tree packages to define !whatever
	keywords as shortcuts for whatever search engines and what not?

	Think of, say, “!wh Example” being rewritten into
	https://en.wikipedia.org/wiki/Example?action=history.

 > and it sounds like it would be error-prone.

	How so?

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Ivan Shmakov <ivan@siamics.net> writes:

> 	How about allowing for out-of-tree packages to define !whatever
> 	keywords as shortcuts for whatever search engines and what not?
>
> 	Think of, say, “!wh Example” being rewritten into
> 	https://en.wikipedia.org/wiki/Example?action=history.

That does sound nice, but implementing something like that as a list of
functions to be called is overly complicated.  Other web browsers allow
people to type the abbreviations and their expansions into a handy form.

So it sounds like a separate functionality from the `M-x eww' light DWIM
code.

>  > and it sounds like it would be error-prone.
>
> 	How so?

People will send bug reports saying "M-x eww does this, and it's wrong",
and what it does depends on what they've customised.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ivan Shmakov <ivan@siamics.net> writes:

 >> How about allowing for out-of-tree packages to define !whatever
 >> keywords as shortcuts for whatever search engines and what not?

 >> Think of, say, “!wh Example” being rewritten into
 >> https://en.wikipedia.org/wiki/Example?action=history.

 > That does sound nice, but implementing something like that as a list
 > of functions to be called is overly complicated.

	Leaving the obvious customizations aside, it doesn’t seem to be
	anywhere near “complicated” to me.

(defun my-eww-wh-uri (keywords)
  "Return an English Wikipedia history page URI for KEYWORDS.
Return nil if the KEYWORDS string does not begin with \"!wh\"."
  (save-match-data
    (when (string-match "^ *!wh\\> *" keywords)
      (concat "https://en.wikipedia.org/wiki/"
              (substring keywords (match-end 0))
              "?action=history"))))

(add-to-list 'eww-uri-rewrite-hook 'my-eww-wh-uri)

 > Other web browsers allow people to type the abbreviations and their
 > expansions into a handy form.

 > So it sounds like a separate functionality from the `M-x eww' light
 > DWIM code.

	When I /do/ mistype an URI, I /never/ mean for the browser to
	refer to a third-party to correct me.  Lynx never does that, and
	Firefox allows the behavior to be configured via its
	‘keyword.enabled’ option.  Is there any good reason EWW should
	have this functionality nailed down right to its core?

	Otherwise, how do I add my own abbreviation-processing code to
	EWW as currently implemented apart from resorting to defadvice?

 >>> and it sounds like it would be error-prone.

 >> How so?

 > People will send bug reports saying "M-x eww does this, and it's
 > wrong", and what it does depends on what they've customised.

	How’s that different to, say, find-file-hook or auto-mode-alist?

-- 
FSF associate member #7257  np. Because — David Modica  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Ivan Shmakov <ivan@siamics.net> writes:

> 	Leaving the obvious customizations aside, it doesn’t seem to be
> 	anywhere near “complicated” to me.
>
> (defun my-eww-wh-uri (keywords)
>   "Return an English Wikipedia history page URI for KEYWORDS.
> Return nil if the KEYWORDS string does not begin with \"!wh\"."
>   (save-match-data
>     (when (string-match "^ *!wh\\> *" keywords)
>       (concat "https://en.wikipedia.org/wiki/"
>               (substring keywords (match-end 0))
>               "?action=history"))))
>
> (add-to-list 'eww-uri-rewrite-hook 'my-eww-wh-uri)

Customisations that require users to write code like this is seldom
used, and you could just define your own `eww' function if you want
something else to happen.

> 	Otherwise, how do I add my own abbreviation-processing code to
> 	EWW as currently implemented apart from resorting to defadvice?

Resorting to defadvice is perfectly fine.

And eww should have built-in abbreviation functionality.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

And, as always, making users customise stuff to make bugs (and
undesirable behaviour) go away is the wrong approach.  The right thing
to do is to fix the bug instead of wall-papering over the bug by
allowing customisations.

Which I have done here by making the DWIM code behave more reasonably on
URLs with embedded spaces.  So I don't think there's more to be done on
this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

 > Customisations that require users to write code like this is seldom
 > used,

	That’s why I’ve explicitly mentioned out-of-tree packages.
	The user would not be required write code like that, – just to
	run M-x package-install over the relevant *ELPA package.

 > and you could just define your own `eww' function if you want
 > something else to happen.

	Such an approach has drawbacks on its own.

 > And, as always, making users customise stuff to make bugs (and
 > undesirable behaviour) go away is the wrong approach.  The right
 > thing to do is to fix the bug instead of wall-papering over the bug
 > by allowing customisations.

	The desirability is subjective and may easily depend on the
	specific situation.  This applies to EWW no more and no less
	than it does to the more usual Emacs facilities, – such as, say,
	VC, EPA, Font-Lock, etc.

	Personally, I find eww DWIM undesirable, and would appreciate
	for an easy way to get rid of it.  A way which would not
	interfere with $ git merge, that is.

[…]

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Lars Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ivan Shmakov <ivan@siamics.net> writes:
>
>> 	FWIW, removing eww-search-words from the hook allows one to type
>> 	something like https://en.wikipedia.org/wiki/Free software RET
>> 	(note the blank) at the prompt and be with that.
>
> That sounds like a bug that should be fixed.
>
> I don't see a use case for letting users customise this stuff, and it
> sounds like it would be error-prone.

This is still my view, so I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ivan Shmakov <ivan@siamics.net> writes:

 >>> FWIW, removing eww-search-words from the hook allows one to type
 >>> something like https://en.wikipedia.org/wiki/Free software RET
 >>> (note the blank) at the prompt and be with that.

 >> That sounds like a bug that should be fixed.

 >> I don't see a use case for letting users customise this stuff, and
 >> it sounds like it would be error-prone.

 > This is still my view, so I'm closing this bug report.

	I believe that when there’s a disagreement on how (and if) the
	feature should be implemented, the respective bugs should not
	generally be closed, although they surely may be tagged
	‘wontfix’ at the maintainer’s discretion.

	The justification is that the maintainer cannot decide what
	features /the users/ desire – and it’s what the bug tracking
	system is ought to reflect – but only the features that he or
	she volunteers – or not – to actually implement and support.

	JFTR, my feature request is as follows.  First, the “automatic”
	use of a search engine may lead to inadvertent information
	disclosure, should the user type (or yank) some text into the
	URI prompt by accident (and it happened to me more than once.)
	Explicitly setting eww-search-prefix to nil will work around
	this issue, although as currently implemented, the respective
	code path doesn’t lead to a clear error message.  (Neither the
	eww-search-prefix defcustom suggests nil as a possible value.)

	The other part is that the authors of EWW extensions may benefit
	from an easy way to hook into the EWW URI handling.  The patch
	I’ve suggested does provide such a way, although implements it
	badly, and so should not be used as it is.

-- 
FSF associate member #7257  http://am-1.org/~ivan/      … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Ivan Shmakov <ivan@siamics.net> writes:

> 	FWIW, removing eww-search-words from the hook allows one to type
> 	something like https://en.wikipedia.org/wiki/Free software RET
> 	(note the blank) at the prompt and be with that.

This has now been fixed on trunk.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ivan Shmakov <ivan@siamics.net> writes:

 >> FWIW, removing eww-search-words from the hook allows one to type
 >> something like https://en.wikipedia.org/wiki/Free software RET (note
 >> the blank) at the prompt and be with that.

 > This has now been fixed on trunk.

	Still, M-x eww RET news:87sifi22rg.fsf@violet.siamics.net RET
	results in the URI being rewritten into
	http://news:87sifi22rg.fsf@violet.siamics.net/.

	With the DWYM code disabled (and url-news-server set to
	news.gmane.org), the command shows me the body of the message
	instead.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Ivan Shmakov <ivan@siamics.net> writes:

> 	Still, M-x eww RET news:87sifi22rg.fsf@violet.siamics.net RET
> 	results in the URI being rewritten into
> 	http://news:87sifi22rg.fsf@violet.siamics.net/.

Fixed now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ivan Shmakov <ivan@siamics.net> writes:
>
>> 	Still, M-x eww RET news:87sifi22rg.fsf@violet.siamics.net RET
>> 	results in the URI being rewritten into
>> 	http://news:87sifi22rg.fsf@violet.siamics.net/.
>
> Fixed now.

(As an aside, my magic crystal ball says that web browsers in the future
will support none of these marginal protocols, but only HTTP, and will
render nothing but HTML and multimedia.  This will cut down this
essentially useless code base drastically.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> Ivan Shmakov <ivan@siamics.net> writes:

 >>> Still, M-x eww RET news:87sifi22rg.fsf@violet.siamics.net RET
 >>> results in the URI being rewritten into
 >>> http://news:87sifi22rg.fsf@violet.siamics.net/.

 >> Fixed now.

	What about президент.рф?

	… Well, right, – Emacs by itself doesn’t seem to offer IDN
	support right now, but I guess that will change.  Eventually.

 > (As an aside, my magic crystal ball says that web browsers in the
 > future will support none of these marginal protocols, but only HTTP,
 > and will render nothing but HTML and multimedia.  This will cut down
 > this essentially useless code base drastically.)

	Given that there still seem to be an add-on for Firefox
	implementing the Gopher protocol support, – I’d give NNTP at
	least quarter a century starting from now.

	But my point is: Firefox doesn’t try to interfere with add-ons
	which implement protocols other than HTTP.  Unlike EWW.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Eli Zaretskii]

> From: Ivan Shmakov <ivan@siamics.net>
> Date: Sun, 11 Jan 2015 18:22:22 +0000
> 
> 	What about президент.рф?
> 
> 	… Well, right, – Emacs by itself doesn’t seem to offer IDN
> 	support right now, but I guess that will change.  Eventually.

There's already idna.el, and Emacs uses it when it's available.

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>> From: Ivan Shmakov  Date: Sun, 11 Jan 2015 18:22:22 +0000

 >> What about президент.рф?

 >> … Well, right, – Emacs by itself doesn’t seem to offer IDN support
 >> right now, but I guess that will change.  Eventually.

 > There's already idna.el, and Emacs uses it when it's available.

	Indeed, I’ve seen references to idna.el under lisp/gnus/, but
	neither any references to it outside of that directory, nor
	idna.el itself in the Emacs tree.  I thus presume the latter is
	supplied by GNU Libidn, and not GNU Emacs itself.

	In relation to my comment above, I see no mention of IDN under
	lisp/url/ or in lisp/net/eww.el.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#19556: eww: make URI rewriting fully customizable

[Eli Zaretskii]

> From: Ivan Shmakov <ivan@siamics.net>
> Date: Sun, 11 Jan 2015 20:17:49 +0000
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> >>>>> From: Ivan Shmakov  Date: Sun, 11 Jan 2015 18:22:22 +0000
> 
>  >> What about президент.рф?
> 
>  >> … Well, right, – Emacs by itself doesn’t seem to offer IDN support
>  >> right now, but I guess that will change.  Eventually.
> 
>  > There's already idna.el, and Emacs uses it when it's available.
> 
> 	Indeed, I’ve seen references to idna.el under lisp/gnus/, but
> 	neither any references to it outside of that directory, nor
> 	idna.el itself in the Emacs tree.

idna.el comes with libidn.  And one of the references you see is in
message.el, which is the default MUA in Emacs.  So not just Gnus uses
it.

> 	In relation to my comment above, I see no mention of IDN under
> 	lisp/url/ or in lisp/net/eww.el.

Volunteers are welcome to fill that gap.  The infrastructure is there
already, it just needs to be used.

bug#19556: eww: make URI rewriting fully customizable

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> idna.el comes with libidn.  And one of the references you see is in
> message.el, which is the default MUA in Emacs.  So not just Gnus uses
> it.

I've wondered why the IDNA stuff isn't implemented natively in Elisp.  I
haven't looked at the spec closely, but from a cursory scan of it a few
years back, it seemed like a not-very-complicated transform.

But my impression may be mistaken...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#19556: eww: make URI rewriting fully customizable

[Stefan Monnier]

> +(defcustom eww-uri-rewrite-hook
> +  '(eww-uri-file
> +    eww-uri-not-supported
> +    eww-uri-remote-prepend-http
> +    eww-uri-search)

A hook defined with defcustom, with a non-nil default value, and
which is not defined in a preloaded file, is asking for trouble.


        Stefan

bug#19556: eww: make URI rewriting fully customizable

[Ivan Shmakov]

>>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

 >> +(defcustom eww-uri-rewrite-hook
 >> +  '(eww-uri-file
 >> +    eww-uri-not-supported
 >> +    eww-uri-remote-prepend-http
 >> +    eww-uri-search)

 > A hook defined with defcustom, with a non-nil default value, and
 > which is not defined in a preloaded file, is asking for trouble.

	If you mean that I should’ve marked it for autoloading, – yes.
	(And the same goes for eww-suggest-uris.)

	However, I’ve already provided a variant of the change [1] which
	is free of this deficiency.

[1] news:87y4pazf33.fsf@violet.siamics.net
    http://debbugs.gnu.org/19556#44

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A