[ELPA] New package: luwak

[ELPA] New package: luwak

[Yuchen Pei]

Hello,

It has been a while since Emacs won the Editor War^[Citations needed].
Do you, like me, sometimes wake up in the middle of the night, feeling
something is amiss, only to realise after M-x list-packages that there
are not enough web browsers in ELPA for a Browser War (inside Emacs)?

luwak[1] is a simple web browser in Emacs, utilising the power of lynx
-dump[2].  It is currently text-only and GET-only.

Features:

- Asynchronous loading
- Some usual browser features: open, reload, search with a search
  engine, follow links, go forward / backward in history, copy url of
  the current page or link at point
- Completion from persistent history in prompt to open a url
- Multiple ways of rendering links: numbered, forward-sexp or hide
  altogether
- Quickly open a link on the page with completion for url / link id
- imenu support, from all unindented strings (which look like
  headings)
- Support of storing and capturing for org mode, guessing the title
  (first imenu item)
- Write the dump of the current page to a file
- Render a buffer containing a lynx dump in the luwak mode
- Browse with or without torsocks

[1] https://g.ypei.me/luwak.git/about/
[2] https://lynx.invisible-island.net

It is a rather simple package, there's no nonfree javascript issue as
lynx does not support javascript.  I would like to submit it to ELPA -
let me know what you think, thanks.

Best,
Yuchen

-- 
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          <https://ypei.org/assets/ypei-pubkey.txt>

Re: [ELPA] New package: luwak

[Jean Louis]

Lynx without Emacs works fine, and luwak can't get the page as it
tries using torsocks automatically:

1666322739 PERROR torsocks[21360]: socks5 libc connect: Connection refused (in socks5_connect() at socks5.c:202)

Looking up html.duckduckgo.com
Unable to locate remote host html.duckduckgo.com.
Alert!: Unable to connect to remote host.

lynx: Can't access startfile https://html.duckduckgo.com/html?q=something%20about%20luwak

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: luwak

[Yuchen Pei]

On Fri 2022-10-21 06:26:51 +0300, Jean Louis wrote:

> Lynx without Emacs works fine, and luwak can't get the page as it
> tries using torsocks automatically:

Thanks for trying it out.

Now I think of it, perhaps it makes more sense for the tor-switch should
be the other around.  I have pushed a commit to toggle the default value
of luwak-tor-switch.

>
> 1666322739 PERROR torsocks[21360]: socks5 libc connect: Connection
> refused (in socks5_connect() at socks5.c:202)

This looks like tor is not on.  Are you able to get output from say

torsocks wget -qO- https://check.torproject.org/ | grep -i congratulations

>
> Looking up html.duckduckgo.com
> Unable to locate remote host html.duckduckgo.com.
> Alert!: Unable to connect to remote host.
>
> lynx: Can't access startfile
> https://html.duckduckgo.com/html?q=something%20about%20luwak


Best,
Yuchen

-- 
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          <https://ypei.org/assets/ypei-pubkey.txt>

Re: [ELPA] New package: luwak

[Jean Louis]

* Yuchen Pei <id@ypei.org> [2022-10-21 06:50]:
> On Fri 2022-10-21 06:26:51 +0300, Jean Louis wrote:
> 
> > Lynx without Emacs works fine, and luwak can't get the page as it
> > tries using torsocks automatically:
> 
> Thanks for trying it out.
> 
> Now I think of it, perhaps it makes more sense for the tor-switch should
> be the other around.  I have pushed a commit to toggle the default value
> of luwak-tor-switch.
> 
> >
> > 1666322739 PERROR torsocks[21360]: socks5 libc connect: Connection
> > refused (in socks5_connect() at socks5.c:202)
> 
> This looks like tor is not on.  Are you able to get output from say
> 
> torsocks wget -qO- https://check.torproject.org/ | grep -i congratulations

Because Tor is not started:

$ torsocks wget -qO- https://check.torproject.org/ | grep -i congratulations
1666335386 PERROR torsocks[30029]: socks5 libc connect: Connection refused (in socks5_connect() at socks5.c:202)

I do not think that package that is supposed to provide browsing shall
supposedly automate activation of Tor. That shall be optional. It
needs instructions for user to understand that it requires Tor, but
why it should require it? There is no reason for it. Using Tor is
generally useful, but there is no reason for browsing package to
require it.

After:

$ sudo rc-service tor start
 * /run/tor: creating directory
 * /run/tor: correcting owner
 * Starting tor ...                                                                           

$ torsocks wget -qO- https://check.torproject.org/ | grep -i congratulations

      Congratulations. This browser is configured to use Tor.
      Congratulations. This browser is configured to use Tor.

If you only say that `torsocks' is to activate it all, then there is
problem, as Tor does not run in background. 

Think of usability of the package for browsing.

Further, it seems I can't access any of the links:

error in process sentinel: save-excursion: Args out of range: #<buffer *luwak*>, 0, 70
error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 70

Further, I think that numbered links shall be activated by using
numbers: 12 RET to activate link number 12.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: luwak

[Yuchen Pei]

On Fri 2022-10-21 10:03:29 +0300, Jean Louis wrote:

>
> I do not think that package that is supposed to provide browsing shall
> supposedly automate activation of Tor. That shall be optional. It
> needs instructions for user to understand that it requires Tor, but
> why it should require it? There is no reason for it. Using Tor is
> generally useful, but there is no reason for browsing package to
> require it.

Yeah, I gathered it could be confusing to use tor by default, which is
why I pushed that commit to flip the default to not use tor.

> Further, it seems I can't access any of the links:
>
> error in process sentinel: save-excursion: Args out of range: #<buffer *luwak*>, 0, 70
> error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 70

Hmm that is strange as this has never happened to me - do you mind
sharing the url(s) with which the page has this error?  By "can't access
any of the links" do you mean this error happens when you try to follow
any links?

Best,
Yuchen

-- 
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          <https://ypei.org/assets/ypei-pubkey.txt>

Re: [ELPA] New package: luwak

[Jean Louis]

* Yuchen Pei <id@ypei.org> [2022-10-21 13:27]:
> > error in process sentinel: save-excursion: Args out of range: #<buffer *luwak*>, 0, 70
> > error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 70
> 
> Hmm that is strange as this has never happened to me - do you mind
> sharing the url(s) with which the page has this error?  By "can't access
> any of the links" do you mean this error happens when you try to follow
> any links?

I used:

{M-x luwak-search RET something emacs RET}

then I get this URL:

https://html.duckduckgo.com/html?q=something%20emacs

I see like this:

   #[1]DuckDuckGo (HTML)

   ____________________

   something emacs_____ Submit
   [All Regions_____]
   [Any Time__]

[2]An Emacs Tutorial: Beginner's Guide to Emacs

   [3]www.masteringemacs.org/article/beginners-guide-to-emacs
   [4]Emacs is self-documenting. When you write code or extend Emacs, it's
   customary to summarize what your code does as a documentation string.
   Emacs pioneered docstrings in the 70s, and today it's supported by most
   modern programming languages. My point is that you can always ask Emacs
   for help: from the mundane to the esoteric.

[5]Basic Emacs Editor Commands - Colorado State University

   [6]www.cs.colostate.edu/helpdocs/emacs.html
   [7]What is Emacs? Emacs is another editor available in UNIX. Like vi ,
   emacs is a screen editor. Unlike vi, emacs is not an insertion mode
   editor, meaning that any character typed in emacs is automatically
   inserted into the file, unless it includes a command prefix.

Then I press enter on [5]

Then I see:

Loading https://duckduckgo.com/l/?uddg=https://www.cs.colostate.edu/helpdocs/emacs.html&rut=f525d859acb76aae4e7e78fd2958fbc0d33a10e5351f3a61f9e68a85f9c71a6d... Done.
error in process sentinel: save-excursion: Args out of range: #<buffer *luwak*>, 0, 71
error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 71

and then new buffer opens:

   REFRESH(0 sec): [1]https://www.cs.colostate.edu/helpdocs/emacs.html

and buffer is empty

I need to wait, then something appears and I need to click on that
link, then I get:

error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 29

and then I see page:

   Colorado State University

                                                  Basic Emacs Editor Commands
     __________________________________________________________________
     __________________________________________________________________

  What is Emacs?

          Emacs is another editor available in UNIX. Like vi, emacs is a
          screen editor. Unlike vi, emacs is not an insertion mode editor,
          meaning that any character typed in emacs is automatically
          inserted into the file, unless it includes a command prefix.



-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: luwak

[Jean Louis]

Another issue is how do you simply open URL?


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: [ELPA] New package: luwak

[Yuchen Pei]

Hello,

Thanks a lot for the detailed instruction on how to reproduce the bug.
On Fri 2022-10-21 13:54:55 +0300, Jean Louis wrote:

>
> [5]Basic Emacs Editor Commands - Colorado State University
>
>    [6]www.cs.colostate.edu/helpdocs/emacs.html
>    [7]What is Emacs? Emacs is another editor available in UNIX. Like vi ,
>    emacs is a screen editor. Unlike vi, emacs is not an insertion mode
>    editor, meaning that any character typed in emacs is automatically
>    inserted into the file, unless it includes a command prefix.
>
> Then I press enter on [5]
>
> Then I see:
>
> Loading https://duckduckgo.com/l/?uddg=https://www.cs.colostate.edu/helpdocs/emacs.html&rut=f525d859acb76aae4e7e78fd2958fbc0d33a10e5351f3a61f9e68a85f9c71a6d... Done.
> error in process sentinel: save-excursion: Args out of range: #<buffer *luwak*>, 0, 71
> error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 71

Good catch.  It is a bug: luwak cannot guess the title as all lines were
indented.  I've pushed a fix.

>
> and then new buffer opens:
>
>    REFRESH(0 sec): [1]https://www.cs.colostate.edu/helpdocs/emacs.html

This is an example where a url rewrite function would be handy to save
one click.  I just added an example rewrite function that does exactly
this (luwak-rewrite-ddg-result).

You can do (setq luwak-url-rewrite-function 'luwak-rewrite-ddg-result)
to have it in effect.

>
> and buffer is empty
>
> I need to wait, then something appears and I need to click on that
> link, then I get:
>
> error in process sentinel: Args out of range: #<buffer *luwak*>, 0, 29

Same problem as the previous "error in proces sentinel" error.

Best,
Yuchen

-- 
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          <https://ypei.org/assets/ypei-pubkey.txt>

Re: [ELPA] New package: luwak

[Yuchen Pei]

On Fri 2022-10-21 13:55:50 +0300, Jean Louis wrote:

> Another issue is how do you simply open URL?

M-x luwak-open

Bound to the o key in luwak mode.

Best,
Yuchen

-- 
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          <https://ypei.org/assets/ypei-pubkey.txt>

Re: [ELPA] New package: luwak

[Stefan Monnier]

> It has been a while since Emacs won the Editor War^[Citations needed].
> Do you, like me, sometimes wake up in the middle of the night, feeling
> something is amiss, only to realise after M-x list-packages that there
> are not enough web browsers in ELPA for a Browser War (inside Emacs)?

It shouldn't take much work to resuscitate W3, if needed.

> luwak[1] is a simple web browser in Emacs, utilising the power of lynx
> -dump[2].  It is currently text-only and GET-only.

Sounds good.  Except I'd much prefer it tried to improve on EWW rather
than reinvent a wheel that's just as square but in a different way.

> Features:
>
> - Asynchronous loading
> - Some usual browser features: open, reload, search with a search
>   engine, follow links, go forward / backward in history, copy url of
>   the current page or link at point
> - Completion from persistent history in prompt to open a url
> - Multiple ways of rendering links: numbered, forward-sexp or hide
>   altogether
> - Quickly open a link on the page with completion for url / link id
> - imenu support, from all unindented strings (which look like
>   headings)
> - Support of storing and capturing for org mode, guessing the title
>   (first imenu item)
> - Write the dump of the current page to a file
> - Render a buffer containing a lynx dump in the luwak mode
> - Browse with or without torsocks

Many of those features seem independent from whether the rendering is
done by w3m, lynx, shr, ...
So could you try and merge this code with that of EWW.
More specifically:
- Divide your code into a "backend" part that deals with calling lynx, and
  a frontend part which provides features by analysing the rendered text.
- Add your frontend features to EWW.
- change EWW so it can call your backend instead of shr.  This last part
  may be more difficult depending on how easy it is to align the details
  of shr's rendering with those of lynx.

Admittedly, we'd then still suffer from "not enough web browsers", but
at least users wouldn't have to choose between incomparable sets
of features.

WDYT?

In the mean time I've added `luwak` to `elpa.git`.
You might want to install and/or look at the patch below as well.


        Stefan


diff --git a/.gitignore b/.gitignore
index 628aad4cc8..b3b5372679 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,4 @@
 *~
-*.elc
\ No newline at end of file
+*.elc
+/luwak-autoloads.el
+/luwak-pkg.el
diff --git a/luwak.el b/luwak.el
index 236a25f42e..a283e13536 100644
--- a/luwak.el
+++ b/luwak.el
@@ -37,34 +37,39 @@
 (defvar luwak-history-file "~/.emacs.d/luwak-history")
 
 (defun luwak-lynx-buffer (url) (format "*luwak-lynx %s*" url))
+
+(defgroup luwak ()
+  "Web browser based on lynx -dump."
+  :group 'web)
+
 (defcustom luwak-search-engine "https://html.duckduckgo.com/html?q=%s"
   "Default search engine for use in 'luwak-search'."
-  :group 'luwak :type '(string))
+  :type '(string))
 (defcustom luwak-url-rewrite-function 'identity
   "Function to rewrite url before loading."
-  :group 'luwak :type '(function))
+  :type '(function))
 (defcustom luwak-tor-switch t
   "Switch behaviour of prefix arg concerning the use of tor.
 
 When nil, use tor by default (requires a tor daemon having been
 started in the system), and not use it with a prefix arg.  When
 non-nill, swap the tor-switch in prefix-arg effect."
-  :group 'luwak :type '(boolean))
+  :type '(boolean))
 (defcustom luwak-max-history-length 100
   "Maximum history length."
-  :group 'luwak :type '(natnum))
+  :type '(natnum))
 (defcustom luwak-render-link-function 'luwak-render-link-id
   "Function to render a link."
-  :group 'luwak :type '(choice (const luwak-render-link-id)
-                               (const luwak-render-link-forward-sexp)
-                               (const luwak-render-link-hide-link)))
+  :type '(choice (const luwak-render-link-id)
+          (const luwak-render-link-forward-sexp)
+          (const luwak-render-link-hide-link)))
 (defcustom luwak-keep-history t
   "If non-nil, will keep history in 'luwak-history-file'."
-  :group 'luwak :type '(boolean))
+  :type '(boolean))
 (defcustom luwak-use-history t
   "If non-nil, will use history from the 'luwak-history-file' when invoking
 'luwak-open'."
-  :group 'luwak :type '(boolean))
+  :type '(boolean))
 
 (put luwak-history 'history-length luwak-max-history-length)
 
@@ -81,18 +86,18 @@ non-nill, swap the tor-switch in prefix-arg effect."
 
 (defvar luwak-mode-map
   (let ((kmap (make-sparse-keymap)))
-    (define-key kmap "\t" 'forward-button)
-    (define-key kmap [backtab] 'backward-button)
-    (define-key kmap "g" 'luwak-reload)
-    (define-key kmap "l" 'luwak-history-backward)
-    (define-key kmap "r" 'luwak-history-forward)
-    (define-key kmap "w" 'luwak-copy-url)
-    (define-key kmap "o" 'luwak-open)
-    (define-key kmap "s" 'luwak-search)
-    (define-key kmap "d" 'luwak-save-dump)
-    (define-key kmap "j" 'imenu)
-    (define-key kmap "t" 'luwak-toggle-links)
-    (define-key kmap "a" 'luwak-follow-numbered-link)
+    (define-key kmap "\t" #'forward-button)
+    (define-key kmap [backtab] #'backward-button)
+    (define-key kmap "g" #'luwak-reload)
+    (define-key kmap "l" #'luwak-history-backward)
+    (define-key kmap "r" #'luwak-history-forward)
+    (define-key kmap "w" #'luwak-copy-url)
+    (define-key kmap "o" #'luwak-open)
+    (define-key kmap "s" #'luwak-search)
+    (define-key kmap "d" #'luwak-save-dump)
+    (define-key kmap "j" #'imenu)
+    (define-key kmap "t" #'luwak-toggle-links)
+    (define-key kmap "a" #'luwak-follow-numbered-link)
     kmap))
 
 (define-derived-mode luwak-mode special-mode (luwak-mode-name)
@@ -120,14 +125,15 @@ non-nill, swap the tor-switch in prefix-arg effect."
       (buffer-substring-no-properties (1- (point))
                                       (progn (end-of-line 1) (point))))))
 
-(when (require 'org nil t)
-  (defun luwak-org-store-link ()
-    (when (derived-mode-p 'luwak-mode)
-      (org-link-store-props
-       :type "luwak"
-       :link (plist-get luwak-data :url)
-       :description (luwak-guess-title))))
+(defun luwak-org-store-link ()
+  (when (derived-mode-p 'luwak-mode)
+    (org-link-store-props
+     :type "luwak"
+     :link (plist-get luwak-data :url)
+     :description (luwak-guess-title))))
 
+;; FIXME: `org' is always available, so this should never fail!
+(when (require 'org nil t)
   (org-link-set-parameters "luwak"
                            :follow #'luwak-open
                            :store #'luwak-org-store-link))
@@ -138,13 +144,14 @@ non-nill, swap the tor-switch in prefix-arg effect."
   (interactive
    (list
     (if luwak-use-history
-        (car
+        (car            ;FIXME: Why throw away everything after space?
          (split-string
-          (completing-read "Url to open: " (luwak-history-collection-from-file))))
+          (completing-read "Url to open: "
+                           (luwak-history-collection-from-file))))
       (read-string "Url to open: "))))
   (luwak-open-url
    (url-encode-url url)
-   (xor luwak-tor-switch current-prefix-arg) 'luwak-add-to-history))
+   (xor luwak-tor-switch current-prefix-arg) #'luwak-add-to-history))
 
 (defun luwak-history-collection-from-file ()
   (split-string
@@ -162,7 +169,7 @@ non-nill, swap the tor-switch in prefix-arg effect."
 
 ;;;###autoload
 (defun luwak-search (query)
-  "Search QUERY using 'luwak-search-engine'."
+  "Search QUERY using `luwak-search-engine'."
   (interactive "sLuwak search query: ")
   (luwak-open (format luwak-search-engine query)))
 
@@ -265,7 +272,7 @@ non-nill, swap the tor-switch in prefix-arg effect."
 (defun luwak-follow-link (marker)
   (let ((url (get-text-property marker 'url)))
     (luwak-open-url
-     url (plist-get luwak-data :no-tor) 'luwak-add-to-history)))
+     url (plist-get luwak-data :no-tor) #'luwak-add-to-history)))
 
 (defun luwak-render-links (urls)
   (with-current-buffer luwak-buffer
@@ -335,7 +342,7 @@ non-nill, swap the tor-switch in prefix-arg effect."
       (goto-char (point-min))
       (re-search-forward "^References\n\n\\(\\ *Visible links:\n\\)?" nil t)
       (delete-region (point-min) (match-end 0))
-      (seq-filter 'identity
+      (seq-filter #'identity            ;`delq nil' ?
        (mapcar (lambda (s)
                 (when (string-match "^\\ *\\([0-9]+\\)\\. \\(.*\\)" s)
                   (concat (match-string 1 s) " " (match-string 2 s))))
@@ -347,11 +354,9 @@ non-nill, swap the tor-switch in prefix-arg effect."
    (list (completing-read "Select link to open: " (luwak-collect-links) nil t)))
   (luwak-open (cadr (split-string link))))
 
-(defun luwak-start-process-with-torsocks (no-tor name buffer program &rest program-args)
-  (if no-tor
-      (apply 'start-process (append (list name buffer program) program-args))
-    (apply 'start-process
-           (append (list name buffer "torsocks" program) program-args))))
+(defun luwak-start-process-with-torsocks (no-tor name buffer &rest cmd)
+  (apply #'start-process name buffer
+         (if no-tor cmd `("torsocks" ,@cmd))))
 
 (defun luwak-save-dump (file-name)
   "Write dump of the current luwak buffer to FILE-NAME."

Re: [ELPA] New package: luwak

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 608 bytes --]

The other axis (ie other than who/what renders the HTML) is how the http
content is fetched; in the case of the now defunct W3 and the present
EWW, this uses the url library, whereas w3m uses w3m to fetch
content. If I had a vote, I would  have Emacs to integrate something
like CURL's libcurl in some shape into Emacs and provide a light-weight
Lisp binding 
to that functionality -- would improve the Web  that is Emacs usable
much better (here, Emacs-usable Web ==Web -- JS and parts of CSS)

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: [ELPA] New package: luwak

[Stefan Kangas]

"T.V Raman" <raman@google.com> writes:

> If I had a vote, I would have Emacs to integrate something like CURL's
> libcurl in some shape into Emacs and provide a light-weight Lisp
> binding to that functionality

That would be very nice, indeed.

Re: [ELPA] New package: luwak

[Yuchen Pei]

Hi,

Sorry for the late reply.  I've had a very busy week.
On Tue 2022-10-25 15:09:06 -0400, Stefan Monnier wrote:

>
> WDYT?

I like the idea, I think it's doable, but also a rather ambitious plan.
I'll think more about it.

>
> In the mean time I've added `luwak` to `elpa.git`.
> You might want to install and/or look at the patch below as well.

Thank you - I'll do that before making a 1.0.0 release.

Best,
Yuchen

-- 
PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
          <https://ypei.org/assets/ypei-pubkey.txt>