bug#44338: 27.1; EWW can't download and view pdf

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1: Type: text/plain, Size: 4768 bytes --]

Using the EWW browser, I can navigate around to webpages, but navigating
to a pdf fails. It appears to download an unreadable pdf. Downloading
and viewing the pdf manually in emacs (pdf-view-mode) works fine.

I've tried these solutions to no avail:
- Adding "application/pdf; emacsclient %s" to a ~/.mailcap file. This
  makes the process hang indefinitely.
- Running the following elisp code:
  `(add-to-list 'mailcap-user-mime-data
                 '((type . "application/pdf")
                   (viewer . pdf-view-mode)))`
  This doesn't seem to have an effect.


--System Info--
In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
 of 2020-08-21 built on CIRROCUMULUS
Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
Repository branch: HEAD
Windowing system distributor 'Microsoft Corp.', version 10.0.19041
System Description: Microsoft Windows 10 Home (v10.0.2004.19041.572)

Recent messages:
[yas] Prepared just-in-time loading of snippets successfully.
For information about GNU Emacs and the GNU system, type C-h C-a.
Contacting host: www.restek.com:443
error in process filter: mailcap-view-mime: Symbol’s function definition is
void: nil
error in process filter: Symbol’s function definition is void: nil
Making completion list... [2 times]

Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Fundamental

Minor modes in effect:
  pyvenv-mode: t
  shell-dirtrack-mode: t
  global-linum-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired dired-loaddefs rfc822 mml
mml-sec epa derived epg epg-config mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader sendmail gnutls network-stream url-http
mail-parse rfc2231 url-gw nsm rmc url-cache url-auth eww mm-url gnus
nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
mail-utils mm-util mail-prsvr url-queue url url-proxy url-privacy
url-expand url-methods url-history mailcap shr text-property-search
url-cookie url-domsuf url-util puny svg xml dom cl-extra yasnippet
highlight-indentation flymake-proc flymake warnings thingatpt
company-capf company pcase help-fns radix-tree help-mode elpy advice
edmacro kmacro elpy-rpc pyvenv eshell esh-cmd esh-ext esh-opt esh-proc
esh-io esh-arg esh-module esh-groups esh-util elpy-shell elpy-profile
elpy-django s elpy-refactor python tramp-sh tramp tramp-loaddefs
trampver tramp-integration tramp-compat shell pcomplete parse-time
iso8601 time-date format-spec ido grep compile comint ansi-color files-x
etags fileloop generator xref project ring cus-edit cus-start cus-load
wid-edit linum material-theme finder-inf package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads w32notify w32
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 286983 9790)
 (symbols 48 22710 1)
 (strings 32 96281 3012)
 (string-bytes 1 2696643)
 (vectors 16 30701)
 (vector-slots 8 394315 14882)
 (floats 8 139 245)
 (intervals 56 410 0)
 (buffers 1000 15))

[-- Attachment #2: Type: text/html, Size: 5170 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Nicholas Harrison <nicholasharrison222@gmail.com> writes:

> Using the EWW browser, I can navigate around to webpages, but navigating
> to a pdf fails. It appears to download an unreadable pdf. Downloading
> and viewing the pdf manually in emacs (pdf-view-mode) works fine.
>
> I've tried these solutions to no avail:
> - Adding "application/pdf; emacsclient %s" to a ~/.mailcap file. This
>   makes the process hang indefinitely.
> - Running the following elisp code:
>   `(add-to-list 'mailcap-user-mime-data
>                  '((type . "application/pdf")
>                    (viewer . pdf-view-mode)))`
>   This doesn't seem to have an effect.

Seems to have been fixed on master:

1. emacs -Q
2. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
3. C-s extra pdf RET RET

This opens the "Specialized Emacs Features" manual in a *eww pdf* buffer
in doc-view-mode.

With PDF Tools and the mailcap-user-mime-data setting above installed,
it opens in pdf-view-mode instead.

I'm surprised the setting for mailcap-user-mime-data is needed though,
since pdf-view-mode appears before doc-view-mode in both
mailcap-mime-data and my version of auto-mode-alist.

Lars?

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Jean Louis]

* Basil L. Contovounesios <contovob@tcd.ie> [2020-10-31 16:45]:
> Nicholas Harrison <nicholasharrison222@gmail.com> writes:
> 
> > Using the EWW browser, I can navigate around to webpages, but navigating
> > to a pdf fails. It appears to download an unreadable pdf. Downloading
> > and viewing the pdf manually in emacs (pdf-view-mode) works fine.
> >
> > I've tried these solutions to no avail:
> > - Adding "application/pdf; emacsclient %s" to a ~/.mailcap file. This
> >   makes the process hang indefinitely.
> > - Running the following elisp code:
> >   `(add-to-list 'mailcap-user-mime-data
> >                  '((type . "application/pdf")
> >                    (viewer . pdf-view-mode)))`
> >   This doesn't seem to have an effect.
> 
> Seems to have been fixed on master:
> 
> 1. emacs -Q
> 2. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
> 3. C-s extra pdf RET RET

When external editor is used, buffer for eww pdf remains there in
background *eww pdf* and neither l or q key bindings work, it would be
expected to go back to the previous page from that buffer.


-- 
Thanks,
Jean Louis
⎔ λ 🄯 𝍄 𝌡 𝌚

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1.1: Type: text/plain, Size: 1601 bytes --]

This is what I get after following those steps:
[image: image.png]

These errors still show in the Messages:
error in process filter: mailcap-view-mime: Symbol’s function definition is
void: nil
error in process filter: Symbol’s function definition is void: nil

Nicholas

On Sat, Oct 31, 2020 at 10:35 AM Jean Louis <bugs@gnu.support> wrote:

> * Basil L. Contovounesios <contovob@tcd.ie> [2020-10-31 16:45]:
> > Nicholas Harrison <nicholasharrison222@gmail.com> writes:
> >
> > > Using the EWW browser, I can navigate around to webpages, but
> navigating
> > > to a pdf fails. It appears to download an unreadable pdf. Downloading
> > > and viewing the pdf manually in emacs (pdf-view-mode) works fine.
> > >
> > > I've tried these solutions to no avail:
> > > - Adding "application/pdf; emacsclient %s" to a ~/.mailcap file. This
> > >   makes the process hang indefinitely.
> > > - Running the following elisp code:
> > >   `(add-to-list 'mailcap-user-mime-data
> > >                  '((type . "application/pdf")
> > >                    (viewer . pdf-view-mode)))`
> > >   This doesn't seem to have an effect.
> >
> > Seems to have been fixed on master:
> >
> > 1. emacs -Q
> > 2. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
> > 3. C-s extra pdf RET RET
>
> When external editor is used, buffer for eww pdf remains there in
> background *eww pdf* and neither l or q key bindings work, it would be
> expected to go back to the previous page from that buffer.
>
>
> --
> Thanks,
> Jean Louis
> ⎔ λ 🄯 𝍄 𝌡 𝌚
>

[-- Attachment #1.2: Type: text/html, Size: 2420 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 75450 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1.1: Type: text/plain, Size: 1842 bytes --]

If it was fixed on master, can you tell me which commit fixed it?

Thanks,
Nicholas

On Sat, Oct 31, 2020 at 11:14 AM Nicholas Harrison <
nicholasharrison222@gmail.com> wrote:

> This is what I get after following those steps:
> [image: image.png]
>
> These errors still show in the Messages:
> error in process filter: mailcap-view-mime: Symbol’s function definition
> is void: nil
> error in process filter: Symbol’s function definition is void: nil
>
> Nicholas
>
> On Sat, Oct 31, 2020 at 10:35 AM Jean Louis <bugs@gnu.support> wrote:
>
>> * Basil L. Contovounesios <contovob@tcd.ie> [2020-10-31 16:45]:
>> > Nicholas Harrison <nicholasharrison222@gmail.com> writes:
>> >
>> > > Using the EWW browser, I can navigate around to webpages, but
>> navigating
>> > > to a pdf fails. It appears to download an unreadable pdf. Downloading
>> > > and viewing the pdf manually in emacs (pdf-view-mode) works fine.
>> > >
>> > > I've tried these solutions to no avail:
>> > > - Adding "application/pdf; emacsclient %s" to a ~/.mailcap file. This
>> > >   makes the process hang indefinitely.
>> > > - Running the following elisp code:
>> > >   `(add-to-list 'mailcap-user-mime-data
>> > >                  '((type . "application/pdf")
>> > >                    (viewer . pdf-view-mode)))`
>> > >   This doesn't seem to have an effect.
>> >
>> > Seems to have been fixed on master:
>> >
>> > 1. emacs -Q
>> > 2. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
>> > 3. C-s extra pdf RET RET
>>
>> When external editor is used, buffer for eww pdf remains there in
>> background *eww pdf* and neither l or q key bindings work, it would be
>> expected to go back to the previous page from that buffer.
>>
>>
>> --
>> Thanks,
>> Jean Louis
>> ⎔ λ 🄯 𝍄 𝌡 𝌚
>>
>

[-- Attachment #1.2: Type: text/html, Size: 2919 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 75450 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> I'm surprised the setting for mailcap-user-mime-data is needed though,
> since pdf-view-mode appears before doc-view-mode in both
> mailcap-mime-data and my version of auto-mode-alist.
>
> Lars?

Hm...  I'm not getting any difference whether the add-to-list has been
done or not?  But I don't have pdf-view-mode installed here, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Nicholas Harrison <nicholasharrison222@gmail.com> writes:

> If it was fixed on master, can you tell me which commit fixed it?

I don't know, but the following are some potential candidates.

Try to fix mailcap parsing again to respect Emacs defaults
eab636c7eb 2020-08-02 09:04:31 +0200
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=eab636c7eb25c4e1cbfeb0fc48cc1274e1c55222

Fix viewing PDFs from eww with external viewers
a34a80a878 2020-09-11 14:06:07 +0200
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=a34a80a878f37832cc5e59f2c26ea1779eca5cf8

Tweak previous mailcap patch (for external viewers)
bde93182bf 2020-09-11 15:37:00 +0200
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=bde93182bf07251f66d571d9667a6c21b6af1930

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> I'm surprised the setting for mailcap-user-mime-data is needed though,
>> since pdf-view-mode appears before doc-view-mode in both
>> mailcap-mime-data and my version of auto-mode-alist.
>>
>> Lars?
>
> Hm...  I'm not getting any difference whether the add-to-list has been
> done or not?  But I don't have pdf-view-mode installed here, I think.

Right, I'm only talking about the case when pdf-view-mode (from the
pdf-tools package on MELPA) is installed.

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Right, I'm only talking about the case when pdf-view-mode (from the
> pdf-tools package on MELPA) is installed.

Just to check -- you didn't install pdf-view-mode between the first and
second test, by any chance?  Then that would explain why it started
working with the seemingly superfluous add-to-list...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> Right, I'm only talking about the case when pdf-view-mode (from the
>> pdf-tools package on MELPA) is installed.
>
> Just to check -- you didn't install pdf-view-mode between the first and
> second test, by any chance?  Then that would explain why it started
> working with the seemingly superfluous add-to-list...

Here's what I'm doing, with the pdf-tools package already installed
under package-user-dir:

0. emacs -Q
1. M-x package-initialize RET
2. M-x pdf-tools-install RET
   (this autoloads pdf-view-mode, registers it on auto-mode-alist, etc.)
3. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
4. C-s extra pdf RET RET

This uses doc-view-mode, whereas C-x C-f path/to/pdf RET uses
pdf-view-mode.

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Here's what I'm doing, with the pdf-tools package already installed
> under package-user-dir:
>
> 0. emacs -Q
> 1. M-x package-initialize RET
> 2. M-x pdf-tools-install RET
>    (this autoloads pdf-view-mode, registers it on auto-mode-alist, etc.)
> 3. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
> 4. C-s extra pdf RET RET
>
> This uses doc-view-mode, whereas C-x C-f path/to/pdf RET uses
> pdf-view-mode.

Thanks for the recipe; I'm seeing this bug, too.

(pp mailcap--computed-mime-data (current-buffer))
=>

  ("pdf"
   (viewer . doc-view-mode)
   (type . "application/pdf")
   (test . window-system))
  ("pdf"
   (viewer . pdf-view-mode)
   (type . "application/pdf")
   (test . window-system))

So doc-view-mode is ahead of pdf-view-mode in the computed structure...

Ah; mailcap-mime-data entries are handled in opposite order -- the final
matching entry is the one that wins, not the first one?  Looks like it.

I've now noted this in the doc string, and I've also moved pdf-view-mode
later, because it makes sense to prefer that if it's installed (since
doc-view-mode is build in.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> (pp mailcap--computed-mime-data (current-buffer))
> =>
>
>   ("pdf"
>    (viewer . doc-view-mode)
>    (type . "application/pdf")
>    (test . window-system))
>   ("pdf"
>    (viewer . pdf-view-mode)
>    (type . "application/pdf")
>    (test . window-system))
>
> So doc-view-mode is ahead of pdf-view-mode in the computed structure...
>
> Ah; mailcap-mime-data entries are handled in opposite order -- the final
> matching entry is the one that wins, not the first one?  Looks like it.

Yes, mailcap--computed-mime-data is in reverse order w.r.t. both major
and minor mime types.

> I've now noted this in the doc string, and I've also moved pdf-view-mode
> later, because it makes sense to prefer that if it's installed (since
> doc-view-mode is build in.

Alternatively, mailcap--computed-mime-data could be kept in canonical
order, e.g. in mailcap-add-mailcap-entry?

[BTW, I don't see the function mailcap-add used anywhere.]
[And neither do I see your changes on Savannah. ;]

Thanks,

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

Jean Louis <bugs@gnu.support> writes:

> When external editor is used, buffer for eww pdf remains there in
> background *eww pdf* and neither l or q key bindings work, it would be
> expected to go back to the previous page from that buffer.

Indeed, popping to an empty *eww pdf* buffer when using an external
viewer is suboptimal.  Even less optimal is that external viewers are
called synchronously, so Emacs cannot be used simultaneously.

How's the attached?

-- 
Basil


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Improve-eww-support-for-externally-viewed-PDFs.patch --]
[-- Type: text/x-diff, Size: 4572 bytes --]

From 4595e0f159fffc93fc5be750478e52db4885d373 Mon Sep 17 00:00:00 2001
From: "Basil L. Contovounesios" <contovob@tcd.ie>
Date: Tue, 3 Nov 2020 22:54:34 +0000
Subject: [PATCH] Improve eww support for externally viewed PDFs

The *eww pdf* buffer is only needed when viewing PDFs within Emacs,
e.g., with doc-view-mode.  External PDF viewers are called with a
temporary file, so the buffer is not needed in that case.  What's
more, mailcap-view-mime erased the buffer and left it in
fundamental-mode until now, so the user was left staring at a
useless, empty buffer.  To make things even worse, external viewers
were invoked synchronously until now, so the user could not browse
the PDF file and use Emacs simultaneously.

* lisp/net/mailcap.el (mailcap--async-shell): New function.
(mailcap-view-mime): Use it to invoke external viewers
asynchronously.  Mention erasure of current buffer in that case in
docstring.  Add a period between the temporary file name and its
extension.

* lisp/net/eww.el (eww-display-pdf): Pop to *eww pdf* buffer only if
it is used for displaying a document; otherwise kill it (bug#44338).
Simplify buffer-substring+insert as insert-buffer-substring.
---
 lisp/net/eww.el     | 21 +++++++++++++--------
 lisp/net/mailcap.el | 30 ++++++++++++++++++++----------
 2 files changed, 33 insertions(+), 18 deletions(-)

diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index ebc75e0e8a..e9763ef6df 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -811,14 +811,19 @@ eww-display-image
 
 (declare-function mailcap-view-mime "mailcap" (type))
 (defun eww-display-pdf ()
-  (let ((data (buffer-substring (point) (point-max))))
-    (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
-    (let ((coding-system-for-write 'raw-text)
-	  (inhibit-read-only t))
-      (erase-buffer)
-      (insert data)
-      (mailcap-view-mime "application/pdf")))
-  (goto-char (point-min)))
+  (let ((buf (current-buffer))
+        (pos (point)))
+    (with-current-buffer (get-buffer-create "*eww pdf*")
+      (let ((coding-system-for-write 'raw-text)
+            (inhibit-read-only t))
+        (erase-buffer)
+        (insert-buffer-substring buf pos)
+        (mailcap-view-mime "application/pdf"))
+      (if (zerop (buffer-size))
+          ;; Buffer contents passed to shell command via temporary file.
+          (kill-buffer)
+        (goto-char (point-min))
+        (pop-to-buffer-same-window (current-buffer))))))
 
 (defun eww-setup-buffer ()
   (when (or (plist-get eww-data :url)
diff --git a/lisp/net/mailcap.el b/lisp/net/mailcap.el
index 94cd9e2156..ad715c4b0e 100644
--- a/lisp/net/mailcap.el
+++ b/lisp/net/mailcap.el
@@ -1128,20 +1128,30 @@ mailcap-file-default-commands
             res)))
        (nreverse res)))))
 
+(defun mailcap--async-shell (command file)
+  "Asynchronously call MIME viewer shell COMMAND.
+Replace %s in COMMAND with FILE, as per `mailcap-mime-data'.
+Delete FILE once COMMAND exits."
+  (let ((buf (get-buffer-create " *mailcap shell*")))
+    (async-shell-command (format command file) buf)
+    (add-function :after (process-sentinel (get-buffer-process buf))
+                  (lambda (proc _msg)
+                    (when (memq (process-status proc) '(exit signal))
+                      (delete-file file))))))
+
 (defun mailcap-view-mime (type)
   "View the data in the current buffer that has MIME type TYPE.
-`mailcap--computed-mime-data' determines the method to use."
+The variable `mailcap--computed-mime-data' determines the method
+to use.  If the method is a shell command string, erase the
+current buffer after passing its contents to the shell command."
   (let ((method (mailcap-mime-info type)))
     (if (stringp method)
-        (let ((file (make-temp-file "emacs-mailcap" nil
-                                    (cadr (split-string type "/")))))
-          (unwind-protect
-              (let ((coding-system-for-write 'binary))
-                (write-region (point-min) (point-max) file nil 'silent)
-                (delete-region (point-min) (point-max))
-                (shell-command (format method file)))
-            (when (file-exists-p file)
-              (delete-file file))))
+        (let* ((ext (concat "." (cadr (split-string type "/"))))
+               (file (make-temp-file "emacs-mailcap" nil ext))
+               (coding-system-for-write 'binary))
+          (write-region nil nil file nil 'silent)
+          (delete-region (point-min) (point-max))
+          (mailcap--async-shell method file))
       (funcall method))))
 
 (provide 'mailcap)
-- 
2.28.0

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1.1: Type: text/plain, Size: 2705 bytes --]

I'll make a couple more comments on the original problem I explained. It
looks like you may have identified additional improvements in the process.

I believe the first problem for me is that both mailcap-mime-data
and mailcap-user-mime-data are nil. This causes the `error in process
filter: mailcap-view-mime: Symbol’s function definition is void: nil` and
makes the pdf download and appear in Fundamental mode. This occurs whether
I will be using doc-view-mode or pdf-view-mode. I'll use DocView for the
rest of this example.

1. emacs -Q
2. M-x eww
3. https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf

This results in the error message and the following:
[image: image.png]

This can be (partially) corrected by running the following code before the
steps 2 and 3:
(add-to-list 'mailcap-user-mime-data
             '((type . "application/pdf")
               (viewer . doc-view-mode)))

This chooses a view mode for the pdf but that brings the second problem.
This selects the default encoding of raw-text and the conversion fails:
[image: image.png]

Instead I choose doc-view-mode manually for the eww pdf buffer:
[image: image.png]

Then selecting binary for the encoding finally gets a viewable pdf:
[image: image.png]

I hope this is in some way helpful.

Nicholas

On Sat, Oct 31, 2020 at 7:43 AM Basil L. Contovounesios <contovob@tcd.ie>
wrote:

> Nicholas Harrison <nicholasharrison222@gmail.com> writes:
>
> > Using the EWW browser, I can navigate around to webpages, but navigating
> > to a pdf fails. It appears to download an unreadable pdf. Downloading
> > and viewing the pdf manually in emacs (pdf-view-mode) works fine.
> >
> > I've tried these solutions to no avail:
> > - Adding "application/pdf; emacsclient %s" to a ~/.mailcap file. This
> >   makes the process hang indefinitely.
> > - Running the following elisp code:
> >   `(add-to-list 'mailcap-user-mime-data
> >                  '((type . "application/pdf")
> >                    (viewer . pdf-view-mode)))`
> >   This doesn't seem to have an effect.
>
> Seems to have been fixed on master:
>
> 1. emacs -Q
> 2. M-x eww RET https://www.gnu.org/software/emacs/manual/emacs.html RET
> 3. C-s extra pdf RET RET
>
> This opens the "Specialized Emacs Features" manual in a *eww pdf* buffer
> in doc-view-mode.
>
> With PDF Tools and the mailcap-user-mime-data setting above installed,
> it opens in pdf-view-mode instead.
>
> I'm surprised the setting for mailcap-user-mime-data is needed though,
> since pdf-view-mode appears before doc-view-mode in both
> mailcap-mime-data and my version of auto-mode-alist.
>
> Lars?
>
> --
> Basil
>

[-- Attachment #1.2: Type: text/html, Size: 3921 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 73558 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 93307 bytes --]

[-- Attachment #4: image.png --]
[-- Type: image/png, Size: 37758 bytes --]

[-- Attachment #5: image.png --]
[-- Type: image/png, Size: 38711 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Nicholas Harrison <nicholasharrison222@gmail.com> writes:

> I'll make a couple more comments on the original problem I
> explained. It looks like you may have identified additional
> improvements in the process.
>
> I believe the first problem for me is that both mailcap-mime-data and
> mailcap-user-mime-data are nil. This causes the `error in process
> filter: mailcap-view-mime: Symbol’s function definition is void: nil`
> and makes the pdf download and appear in Fundamental mode. This occurs
> whether I will be using doc-view-mode or pdf-view-mode. I'll use
> DocView for the rest of this example.
>
> 1. emacs -Q
> 2. M-x eww
> 3. https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf
>
> This results in the error message and the following:
> image.png
>
> This can be (partially) corrected by running the following code before the steps 2 and 3:
> (add-to-list 'mailcap-user-mime-data
>              '((type . "application/pdf")
>                (viewer . doc-view-mode)))
>
> This chooses a view mode for the pdf but that brings the second
> problem. This selects the default encoding of raw-text and the
> conversion fails:
>
> Instead I choose doc-view-mode manually for the eww pdf buffer:
>
> Then selecting binary for the encoding finally gets a viewable pdf:
>
> I hope this is in some way helpful.

Thanks.  I cannot reproduce these on what will be Emacs 27.2 or Emacs
28.1 on GNU/Linux.  Perhaps they have been fixed already, or are
specific to MS Windows.  If someone on MS Windows could check whether
they still occur on master and emacs-27, that would be helpful.

-- 
Basil

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars)
 of 2020-11-03 built on thunk
Repository revision: f9d6e463d310db0e1931f26609d938531c56f9c3
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
 --prefix=/home/blc/.local --with-x-toolkit=lucid
 --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS
LIBSYSTEMD JSON PDUMPER LCMS2

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml easymenu mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 51798 5028)
 (symbols 48 6742 1)
 (strings 32 18899 1840)
 (string-bytes 1 612322)
 (vectors 16 12192)
 (vector-slots 8 168066 8842)
 (floats 8 23 45)
 (intervals 56 221 0)
 (buffers 992 10))

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Nicholas Harrison <nicholasharrison222@gmail.com> writes:

> I believe the first problem for me is that both mailcap-mime-data and
> mailcap-user-mime-data are nil.

I forgot to say that the former is probably due to
https://debbugs.gnu.org/40247.

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1: Type: text/plain, Size: 5316 bytes --]

In WSL on 27.1 the mailcap-mime-data problem remains, but past that the
conversion when doc-view-mode is enabled works with both binary and the
default raw-text.

On Tue, Nov 3, 2020 at 5:23 PM Basil L. Contovounesios <contovob@tcd.ie>
wrote:

> Nicholas Harrison <nicholasharrison222@gmail.com> writes:
>
> > I'll make a couple more comments on the original problem I
> > explained. It looks like you may have identified additional
> > improvements in the process.
> >
> > I believe the first problem for me is that both mailcap-mime-data and
> > mailcap-user-mime-data are nil. This causes the `error in process
> > filter: mailcap-view-mime: Symbol’s function definition is void: nil`
> > and makes the pdf download and appear in Fundamental mode. This occurs
> > whether I will be using doc-view-mode or pdf-view-mode. I'll use
> > DocView for the rest of this example.
> >
> > 1. emacs -Q
> > 2. M-x eww
> > 3. https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf
> >
> > This results in the error message and the following:
> > image.png
> >
> > This can be (partially) corrected by running the following code before
> the steps 2 and 3:
> > (add-to-list 'mailcap-user-mime-data
> >              '((type . "application/pdf")
> >                (viewer . doc-view-mode)))
> >
> > This chooses a view mode for the pdf but that brings the second
> > problem. This selects the default encoding of raw-text and the
> > conversion fails:
> >
> > Instead I choose doc-view-mode manually for the eww pdf buffer:
> >
> > Then selecting binary for the encoding finally gets a viewable pdf:
> >
> > I hope this is in some way helpful.
>
> Thanks.  I cannot reproduce these on what will be Emacs 27.2 or Emacs
> 28.1 on GNU/Linux.  Perhaps they have been fixed already, or are
> specific to MS Windows.  If someone on MS Windows could check whether
> they still occur on master and emacs-27, that would be helpful.
>
> --
> Basil
>
> In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
> version 1.16.0, Xaw3d scroll bars)
>  of 2020-11-03 built on thunk
> Repository revision: f9d6e463d310db0e1931f26609d938531c56f9c3
> Repository branch: master
> Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
> System Description: Debian GNU/Linux bullseye/sid
>
> Configured using:
>  'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
>  --prefix=/home/blc/.local --with-x-toolkit=lucid
>  --with-file-notification=yes --with-x'
>
> Configured features:
> XAW3D XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB
> NOTIFY INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
> LIBOTF ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS
> LIBSYSTEMD JSON PDUMPER LCMS2
>
> Important settings:
>   value of $LANG: en_IE.UTF-8
>   value of $XMODIFIERS: @im=ibus
>   locale-coding-system: utf-8-unix
>
> Major mode: Lisp Interaction
>
> Minor modes in effect:
>   tooltip-mode: t
>   global-eldoc-mode: t
>   eldoc-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   line-number-mode: t
>   transient-mark-mode: t
>
> Load-path shadows:
> None found.
>
> Features:
> (shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
> rfc822 mml easymenu mml-sec epa derived epg epg-config gnus-util rmail
> rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
> eieio-loaddefs password-cache json map text-property-search time-date
> subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
> mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
> cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
> tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
> mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
> regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
> lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
> timer select scroll-bar mouse jit-lock font-lock syntax facemenu
> font-core term/tty-colors frame minibuffer cl-generic cham georgian
> utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
> japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
> ethiopic indian cyrillic chinese composite charscript charprop
> case-table epa-hook jka-cmpr-hook help simple abbrev obarray
> cl-preloaded nadvice button loaddefs faces cus-face macroexp files
> window text-properties overlay sha1 md5 base64 format env code-pages
> mule custom widget hashtable-print-readable backquote threads dbusbind
> inotify lcms2 dynamic-setting system-font-setting font-render-setting
> cairo x-toolkit x multi-tty make-network-process emacs)
>
> Memory information:
> ((conses 16 51798 5028)
>  (symbols 48 6742 1)
>  (strings 32 18899 1840)
>  (string-bytes 1 612322)
>  (vectors 16 12192)
>  (vector-slots 8 168066 8842)
>  (floats 8 23 45)
>  (intervals 56 221 0)
>  (buffers 992 10))
>

[-- Attachment #2: Type: text/html, Size: 6219 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: Nicholas Harrison <nicholasharrison222@gmail.com>
> Date: Tue, 3 Nov 2020 16:52:40 -0700
> Cc: 44338@debbugs.gnu.org
> 
> This can be (partially) corrected by running the following code before the steps 2 and 3:
> (add-to-list 'mailcap-user-mime-data
>              '((type . "application/pdf")
>                (viewer . doc-view-mode)))
> 
> This chooses a view mode for the pdf but that brings the second problem. This selects the default encoding
> of raw-text and the conversion fails:

You say it selects raw-text, but the screenshot you sent clearly shows
that Emacs was trying to use iso-latin-1-dos.  In which case the
failure is easily understandable, but I don't immediately see where
did that value come from (it's most probably the default value of
buffer-file-coding-system for you, but since eww-display-pdf binds
coding-system-for-write, the question is why that value isn't being
used).  Could you perhaps produce a backtrace from that situation?
For example, like this:

  M-: (debug-on-entry 'select-safe-coding-system) RET

and then repeat the recipe.

In any case, this isn't right:

  (defun eww-display-pdf ()
    (let ((data (buffer-substring (point) (point-max))))
      (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
      (let ((coding-system-for-write 'raw-text)  <<<<<<<<<<<<<<<<<<<<<<
	    (inhibit-read-only t))
	(erase-buffer)
	(insert data)
	(mailcap-view-mime "application/pdf")))
    (goto-char (point-min)))

We should use 'raw-text-unix here, since the buffer contents is a
stream of raw bytes.

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Wed, 04 Nov 2020 00:23:39 +0000
> Cc: 44338@debbugs.gnu.org
> 
> Thanks.  I cannot reproduce these on what will be Emacs 27.2 or Emacs
> 28.1 on GNU/Linux.  Perhaps they have been fixed already, or are
> specific to MS Windows.  If someone on MS Windows could check whether
> they still occur on master and emacs-27, that would be helpful.

I asked for a backtrace to see where do we try using Latin-1 to encode
this buffer.  Given that information, I think it will be quite easy to
find a solution.

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

> In any case, this isn't right:
>
>   (defun eww-display-pdf ()
>     (let ((data (buffer-substring (point) (point-max))))
>       (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
>       (let ((coding-system-for-write 'raw-text)  <<<<<<<<<<<<<<<<<<<<<<
> 	    (inhibit-read-only t))
> 	(erase-buffer)
> 	(insert data)
> 	(mailcap-view-mime "application/pdf")))
>     (goto-char (point-min)))
>
> We should use 'raw-text-unix here, since the buffer contents is a
> stream of raw bytes.

Thanks, I've made the change in my patch, and will push in a few days if
I don't hear otherwise.

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1: Type: text/plain, Size: 5826 bytes --]

Not sure if this is much help, but here is the backtrace given when I do
the following steps:

1. emacs -Q
2. M-: (debug-on-entry 'select-safe-coding-system) RET
3. M-x eww RET https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf
RET
(no backtrace here)
4. M-x doc-view-mode RET

Debugger entered--entering a function:
* select-safe-coding-system(1 381654 iso-latin-1-dos nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
  write-region(nil nil "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!")
  doc-view-mode()
  funcall-interactively(doc-view-mode)
  call-interactively(doc-view-mode record nil)
  command-execute(doc-view-mode record)
  execute-extended-command(nil "doc-view-mode" "doc-view-mo")
  funcall-interactively(execute-extended-command nil "doc-view-mode"
"doc-view-mo")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

5. ESC ESC ESC
6. RET (it asks to choose an encoding, chose default raw-text)

Debugger entered--returning value: raw-text
  select-safe-coding-system(1 381654 iso-latin-1-dos nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww...")
  write-region(nil nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww...")
  doc-view-mode()
  funcall-interactively(doc-view-mode)
  call-interactively(doc-view-mode record nil)
  command-execute(doc-view-mode record)
  execute-extended-command(nil "doc-view-mode" "doc-view-mo")
  funcall-interactively(execute-extended-command nil "doc-view-mode"
"doc-view-mo")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Debugger entered--entering a function:
* select-safe-coding-system(1 383892 no-conversion nil)
  md5(#<buffer  *temp*>)
  doc-view--current-cache-dir()
  doc-view-already-converted-p()
  doc-view-initiate-display()
  doc-view-mode()
  funcall-interactively(doc-view-mode)
  call-interactively(doc-view-mode record nil)
  command-execute(doc-view-mode record)
  execute-extended-command(nil "doc-view-mode" "doc-view-mo")
  funcall-interactively(execute-extended-command nil "doc-view-mode"
"doc-view-mo")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Debugger entered--returning value: no-conversion
  select-safe-coding-system(1 383892 no-conversion nil)
  md5(#<buffer  *temp*>)
  doc-view--current-cache-dir()
  doc-view-already-converted-p()
  doc-view-initiate-display()
  doc-view-mode()
  funcall-interactively(doc-view-mode)
  call-interactively(doc-view-mode record nil)
  command-execute(doc-view-mode record)
  execute-extended-command(nil "doc-view-mode" "doc-view-mo")
  funcall-interactively(execute-extended-command nil "doc-view-mode"
"doc-view-mo")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Debugger entered--entering a function:
* select-safe-coding-system("100" nil prefer-utf-8 nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
  write-region("100" nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution...." nil silently)
  #f(compiled-function () #<bytecode 0x42abe9>)()
  doc-view-sentinel(#<process pdf/ps->png> "finished\n")

Debugger entered--returning value: prefer-utf-8-dos
  select-safe-coding-system("100" nil prefer-utf-8 nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww...")
  write-region("100" nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww..." nil silently)
  #f(compiled-function () #<bytecode 0x42abe9>)()
  doc-view-sentinel(#<process pdf/ps->png> "finished\n")

Debugger entered--returning value: prefer-utf-8-dos
  select-safe-coding-system("100" nil prefer-utf-8 nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww...")
  write-region("100" nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww..." nil silently)
  #f(compiled-function () #<bytecode 0x42abe9>)()
  doc-view-sentinel(#<process pdf/ps->png> "finished\n")

  Let me know if I was supposed to do something differently.

Nicholas

On Wed, Nov 4, 2020 at 8:07 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Nicholas Harrison <nicholasharrison222@gmail.com>
> > Date: Tue, 3 Nov 2020 16:52:40 -0700
> > Cc: 44338@debbugs.gnu.org
> >
> > This can be (partially) corrected by running the following code before
> the steps 2 and 3:
> > (add-to-list 'mailcap-user-mime-data
> >              '((type . "application/pdf")
> >                (viewer . doc-view-mode)))
> >
> > This chooses a view mode for the pdf but that brings the second problem.
> This selects the default encoding
> > of raw-text and the conversion fails:
>
> You say it selects raw-text, but the screenshot you sent clearly shows
> that Emacs was trying to use iso-latin-1-dos.  In which case the
> failure is easily understandable, but I don't immediately see where
> did that value come from (it's most probably the default value of
> buffer-file-coding-system for you, but since eww-display-pdf binds
> coding-system-for-write, the question is why that value isn't being
> used).  Could you perhaps produce a backtrace from that situation?
> For example, like this:
>
>   M-: (debug-on-entry 'select-safe-coding-system) RET
>
> and then repeat the recipe.
>
> In any case, this isn't right:
>
>   (defun eww-display-pdf ()
>     (let ((data (buffer-substring (point) (point-max))))
>       (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
>       (let ((coding-system-for-write 'raw-text)  <<<<<<<<<<<<<<<<<<<<<<
>             (inhibit-read-only t))
>         (erase-buffer)
>         (insert data)
>         (mailcap-view-mime "application/pdf")))
>     (goto-char (point-min)))
>
> We should use 'raw-text-unix here, since the buffer contents is a
> stream of raw bytes.
>

[-- Attachment #2: Type: text/html, Size: 7661 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: Nicholas Harrison <nicholasharrison222@gmail.com>
> Date: Wed, 4 Nov 2020 16:43:13 -0700
> Cc: "Basil L. Contovounesios" <contovob@tcd.ie>, 44338@debbugs.gnu.org
> 
> Not sure if this is much help, but here is the backtrace given when I do the following steps:
> 
> 1. emacs -Q
> 2. M-: (debug-on-entry 'select-safe-coding-system) RET
> 3. M-x eww RET https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf RET
> (no backtrace here)
> 4. M-x doc-view-mode RET
> 
> Debugger entered--entering a function:
> * select-safe-coding-system(1 381654 iso-latin-1-dos nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
>   write-region(nil nil "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
>   doc-view-mode()
>   funcall-interactively(doc-view-mode)
>   call-interactively(doc-view-mode record nil)
>   command-execute(doc-view-mode record)
>   execute-extended-command(nil "doc-view-mode" "doc-view-mo")
>   funcall-interactively(execute-extended-command nil "doc-view-mode" "doc-view-mo")
>   call-interactively(execute-extended-command nil nil)
>   command-execute(execute-extended-command)

Thanks.  This tells part of the story, but not all of it.  What I
wanted to see was the backtrace when doc-view-mode is invoked by EWW.
AFAIU, that requires you to augment mailcap-user-mime-data as this:

 (add-to-list 'mailcap-user-mime-data
              '((type . "application/pdf")
                (viewer . doc-view-mode)))

before performing the reproduction steps.

To give you more background: when you invoke doc-view-mode manually in
a buffer produced by "M-x eww", the buffer's buffer-file-coding-system
is the platform default, in your case iso-latin-1-dos.  That is what
doc-view-mode uses to write the PDF bytestream to a temporary file,
and that fails because iso-latin-1-dos cannot encode the raw bytes in
the binary content.  But eww-display-pdf binds coding-system-for-write
to 'raw-text, and then doc-view-mode ought to use that to write to the
temporary file, and yet in your screenshot I still see it tried to use
iso-latin-1-dos, which I cannot explain.  So I'd like to see the
backtrace when you invoke doc-view-mode via EWW, after augmenting
mailcap-user-mime-data, to try to understand why it uses the wrong
encoding.

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: Nicholas Harrison <nicholasharrison222@gmail.com>
> Date: Wed, 4 Nov 2020 16:43:13 -0700
> Cc: "Basil L. Contovounesios" <contovob@tcd.ie>, 44338@debbugs.gnu.org
> 
> Not sure if this is much help, but here is the backtrace given when I do the following steps:
> 
> 1. emacs -Q
> 2. M-: (debug-on-entry 'select-safe-coding-system) RET
> 3. M-x eww RET https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf RET
> (no backtrace here)
> 4. M-x doc-view-mode RET
> 
> Debugger entered--entering a function:
> * select-safe-coding-system(1 381654 iso-latin-1-dos nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
>   write-region(nil nil "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
>   doc-view-mode()
>   funcall-interactively(doc-view-mode)
>   call-interactively(doc-view-mode record nil)
>   command-execute(doc-view-mode record)
>   execute-extended-command(nil "doc-view-mode" "doc-view-mo")
>   funcall-interactively(execute-extended-command nil "doc-view-mode" "doc-view-mo")
>   call-interactively(execute-extended-command nil nil)
>   command-execute(execute-extended-command)

Thanks.  This tells part of the story, but not all of it.  What I
wanted to see was the backtrace when doc-view-mode is invoked by EWW.
AFAIU, that requires you to augment mailcap-user-mime-data as this:

 (add-to-list 'mailcap-user-mime-data
              '((type . "application/pdf")
                (viewer . doc-view-mode)))

before performing the reproduction steps.

To give you more background: when you invoke doc-view-mode manually in
a buffer produced by "M-x eww", the buffer's buffer-file-coding-system
is the platform default, in your case iso-latin-1-dos.  That is what
doc-view-mode uses to write the PDF bytestream to a temporary file,
and that fails because iso-latin-1-dos cannot encode the raw bytes in
the binary content.  But eww-display-pdf binds coding-system-for-write
to 'raw-text, and then doc-view-mode ought to use that to write to the
temporary file, and yet in your screenshot I still see it tried to use
iso-latin-1-dos, which I cannot explain.  So I'd like to see the
backtrace when you invoke doc-view-mode via EWW, after augmenting
mailcap-user-mime-data, to try to understand why it uses the wrong
encoding.

Can you please produce the backtrace under those  modified conditions?

bug#44338: 27.1; EWW can't download and view pdf

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Alternatively, mailcap--computed-mime-data could be kept in canonical
> order, e.g. in mailcap-add-mailcap-entry?

I guess that would make sense, but this code has been re-fixed so
much that I'd rather ... leave it alone for a while.  :-)

> [BTW, I don't see the function mailcap-add used anywhere.]

It's for usage by users, I think?

> [And neither do I see your changes on Savannah. ;]

Forgot to push; done now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44338: 27.1; EWW can't download and view pdf

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> How's the attached?

Haven't tried it, but it sounds like a good change to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1: Type: text/plain, Size: 3611 bytes --]

After running the code you gave and using eww to open a pdf, this is what I
get:

Debugger entered--entering a function:
* select-safe-coding-system("100" nil prefer-utf-8 nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
  write-region("100" nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el" nil silently)
  #f(compiled-function () #<bytecode 0x1ff19f1>)()
  doc-view-sentinel(#<process pdf/ps->png> "finished\n")

Debugger entered--returning value: prefer-utf-8-dos
  select-safe-coding-system("100" nil prefer-utf-8 nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
  write-region("100" nil
"c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el" nil silently)
  #f(compiled-function () #<bytecode 0x1ff19f1>)()
  doc-view-sentinel(#<process pdf/ps->png> "finished\n")

The buffer it ends up with says:
Cannot display this page!
Maybe because of a conversion failure!

On Thu, Nov 5, 2020 at 6:42 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Nicholas Harrison <nicholasharrison222@gmail.com>
> > Date: Wed, 4 Nov 2020 16:43:13 -0700
> > Cc: "Basil L. Contovounesios" <contovob@tcd.ie>, 44338@debbugs.gnu.org
> >
> > Not sure if this is much help, but here is the backtrace given when I do
> the following steps:
> >
> > 1. emacs -Q
> > 2. M-: (debug-on-entry 'select-safe-coding-system) RET
> > 3. M-x eww RET
> https://www.gnu.org/software/emacs/manual/pdf/emacs-xtra.pdf RET
> > (no backtrace here)
> > 4. M-x doc-view-mode RET
> >
> > Debugger entered--entering a function:
> > * select-safe-coding-system(1 381654 iso-latin-1-dos nil
> > "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
> >   write-region(nil nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww pdf!")
> >   doc-view-mode()
> >   funcall-interactively(doc-view-mode)
> >   call-interactively(doc-view-mode record nil)
> >   command-execute(doc-view-mode record)
> >   execute-extended-command(nil "doc-view-mode" "doc-view-mo")
> >   funcall-interactively(execute-extended-command nil "doc-view-mode"
> "doc-view-mo")
> >   call-interactively(execute-extended-command nil nil)
> >   command-execute(execute-extended-command)
>
> Thanks.  This tells part of the story, but not all of it.  What I
> wanted to see was the backtrace when doc-view-mode is invoked by EWW.
> AFAIU, that requires you to augment mailcap-user-mime-data as this:
>
>  (add-to-list 'mailcap-user-mime-data
>               '((type . "application/pdf")
>                 (viewer . doc-view-mode)))
>
> before performing the reproduction steps.
>
> To give you more background: when you invoke doc-view-mode manually in
> a buffer produced by "M-x eww", the buffer's buffer-file-coding-system
> is the platform default, in your case iso-latin-1-dos.  That is what
> doc-view-mode uses to write the PDF bytestream to a temporary file,
> and that fails because iso-latin-1-dos cannot encode the raw bytes in
> the binary content.  But eww-display-pdf binds coding-system-for-write
> to 'raw-text, and then doc-view-mode ought to use that to write to the
> temporary file, and yet in your screenshot I still see it tried to use
> iso-latin-1-dos, which I cannot explain.  So I'd like to see the
> backtrace when you invoke doc-view-mode via EWW, after augmenting
> mailcap-user-mime-data, to try to understand why it uses the wrong
> encoding.
>
> Can you please produce the backtrace under those  modified conditions?
>

[-- Attachment #2: Type: text/html, Size: 4814 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: Nicholas Harrison <nicholasharrison222@gmail.com>
> Date: Thu, 5 Nov 2020 08:18:19 -0700
> Cc: 44338@debbugs.gnu.org
> 
> After running the code you gave and using eww to open a pdf, this is what I get:
> 
> Debugger entered--entering a function:
> * select-safe-coding-system("100" nil prefer-utf-8 nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
>   write-region("100" nil "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el" nil silently)
>   #f(compiled-function () #<bytecode 0x1ff19f1>)()
>   doc-view-sentinel(#<process pdf/ps->png> "finished\n")
> 
> Debugger entered--returning value: prefer-utf-8-dos
>   select-safe-coding-system("100" nil prefer-utf-8 nil "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
>   write-region("100" nil "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el" nil silently)
>   #f(compiled-function () #<bytecode 0x1ff19f1>)()
>   doc-view-sentinel(#<process pdf/ps->png> "finished\n")

Hmm, that's not the problem you reported originally, and it doesn't
look like Emacs asked you to select an encoding here, did it?

> The buffer it ends up with says:
> Cannot display this page!
> Maybe because of a conversion failure!

So I guess I'm confused now and don't know what is the problem, sorry.

A stub in the dark: if you replace raw-text with raw-text-unix here:

  (defun eww-display-pdf ()
    (let ((data (buffer-substring (point) (point-max))))
      (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
      (let ((coding-system-for-write 'raw-text)  <<<<<<<<<<<<<<<<<<<
	    (inhibit-read-only t))
	(erase-buffer)
	(insert data)
	(mailcap-view-mime "application/pdf")))
    (goto-char (point-min)))

does the problem go away?

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> Alternatively, mailcap--computed-mime-data could be kept in canonical
>> order, e.g. in mailcap-add-mailcap-entry?
>
> I guess that would make sense, but this code has been re-fixed so
> much that I'd rather ... leave it alone for a while.  :-)

I understand. :)

>> [BTW, I don't see the function mailcap-add used anywhere.]
>
> It's for usage by users, I think?

Right.

>> [And neither do I see your changes on Savannah. ;]
>
> Forgot to push; done now.

Thanks.

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> How's the attached?
>
> Haven't tried it, but it sounds like a good change to me.

Thanks, pushed to ease testing. ;)

Improve eww support for externally viewed PDFs
bfd3124202 2020-11-05 17:34:23 +0000
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=bfd31242025cde90c8252db92dc54d0be4115c91

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

No, it didn't ask me for an encoding.

Good stab in the dark. I ran your new function code and the
mailcap-user-mime-data code again (after loading eww). No debugger
triggered. It converted and showed the pdf correctly.

On Thu, Nov 5, 2020 at 8:50 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Nicholas Harrison <nicholasharrison222@gmail.com>
> > Date: Thu, 5 Nov 2020 08:18:19 -0700
> > Cc: 44338@debbugs.gnu.org
> >
> > After running the code you gave and using eww to open a pdf, this is
> what I get:
> >
> > Debugger entered--entering a function:
> > * select-safe-coding-system("100" nil prefer-utf-8 nil
> > "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> > pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
> >   write-region("100" nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> > pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el" nil silently)
> >   #f(compiled-function () #<bytecode 0x1ff19f1>)()
> >   doc-view-sentinel(#<process pdf/ps->png> "finished\n")
> >
> > Debugger entered--returning value: prefer-utf-8-dos
> >   select-safe-coding-system("100" nil prefer-utf-8 nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> > pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el")
> >   write-region("100" nil
> "c:/Users/nicho/AppData/Local/Temp/docview1001/!eww
> > pdf!-2072e1249b26ee28e656f1a01f0cb4a9/resolution.el" nil silently)
> >   #f(compiled-function () #<bytecode 0x1ff19f1>)()
> >   doc-view-sentinel(#<process pdf/ps->png> "finished\n")
>
> Hmm, that's not the problem you reported originally, and it doesn't
> look like Emacs asked you to select an encoding here, did it?
>
> > The buffer it ends up with says:
> > Cannot display this page!
> > Maybe because of a conversion failure!
>
> So I guess I'm confused now and don't know what is the problem, sorry.
>
> A stub in the dark: if you replace raw-text with raw-text-unix here:
>
>   (defun eww-display-pdf ()
>     (let ((data (buffer-substring (point) (point-max))))
>       (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
>       (let ((coding-system-for-write 'raw-text)  <<<<<<<<<<<<<<<<<<<
>             (inhibit-read-only t))
>         (erase-buffer)
>         (insert data)
>         (mailcap-view-mime "application/pdf")))
>     (goto-char (point-min)))
>
> does the problem go away?
>

[-- Attachment #2: Type: text/html, Size: 3256 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: Nicholas Harrison <nicholasharrison222@gmail.com>
> Date: Thu, 5 Nov 2020 10:52:10 -0700
> Cc: 44338@debbugs.gnu.org
> 
> No, it didn't ask me for an encoding.
> 
> Good stab in the dark. I ran your new function code and the mailcap-user-mime-data code again (after
> loading eww). No debugger triggered. It converted and showed the pdf correctly.

Great, then the change Basil already made locally will also solve the
last part of the problem.

Thanks.

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicholas Harrison <nicholasharrison222@gmail.com>
>> Date: Thu, 5 Nov 2020 10:52:10 -0700
>> Cc: 44338@debbugs.gnu.org
>> 
>> No, it didn't ask me for an encoding.
>> 
>> Good stab in the dark. I ran your new function code and the mailcap-user-mime-data code again (after
>> loading eww). No debugger triggered. It converted and showed the pdf correctly.
>
> Great, then the change Basil already made locally will also solve the
> last part of the problem.

The change is no longer local, which prompted some off-list comments
from Stefan that confused me.

mailcap-view-mime already binds coding-system-for-write to 'binary
before writing to a file and spawning a subshell, so why does the
coding-system-for-write matter in its caller eww-display-pdf?

In other words, can we remove the binding altogether from
eww-display-pdf and make the *eww pdf* buffer unibyte, as Stefan
suggested?

Could something funny happen / be happening during insertion from one
buffer into the other in eww-display-pdf?

Thanks,

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Nicholas Harrison <nicholasharrison222@gmail.com>,  44338@debbugs.gnu.org
> Date: Thu, 05 Nov 2020 19:04:55 +0000
> 
> > Great, then the change Basil already made locally will also solve the
> > last part of the problem.
> 
> The change is no longer local, which prompted some off-list comments
> from Stefan that confused me.
> 
> mailcap-view-mime already binds coding-system-for-write to 'binary
> before writing to a file and spawning a subshell, so why does the
> coding-system-for-write matter in its caller eww-display-pdf?

I don't think this part of mailcap-view-mime matters in this case:

  (defun mailcap-view-mime (type)
    "View the data in the current buffer that has MIME type TYPE.
  `mailcap--computed-mime-data' determines the method to use."
    (let ((method (mailcap-mime-info type)))
      (if (stringp method)
	  (let ((file (make-temp-file "emacs-mailcap" nil
				      (cadr (split-string type "/")))))
	    (unwind-protect
		(let ((coding-system-for-write 'binary))
		  (write-region (point-min) (point-max) file nil 'silent)
		  (delete-region (point-min) (point-max))
		  (shell-command (format method file)))
	      (when (file-exists-p file)
		(delete-file file))))
	(funcall method))))

As you see, mailcap-view-mime only binds coding-system-for-write if
mailcap-mime-info returns a string, which should then be a shell
command.  But in our case, mailcap-mime-info returns doc-view-mode, a
symbol of a function, and in that case mailcap-mime-info just calls
the function.

That said, I don't think I understand well enough what exactly
happened in the problematic case, because I didn't succeed in getting
a backtrace which matched the problem description.  So the above is
based on some looking into a crystal ball, and thus might be wrong.

> In other words, can we remove the binding altogether from
> eww-display-pdf and make the *eww pdf* buffer unibyte, as Stefan
> suggested?

If we want to remove the binding from eww-display-pdf, it should go to
doc-view-mode, because only it knows what it needs.  mailcap-view-mime
is right not to do anything when it calls the function, since it
cannot know what that function will or will not do.

Please also note that doc-view-mode expects to be called on a unibyte
buffer with 'no-conversion' as its buffer-file-coding-system, because
we have ("\\.pdf\\'" . no-conversion) in auto-coding-alist.  So an
alternative solution would be for eww to arrange for the buffer to be
unibyte; then no binding of coding-system-for-write will be needed.

> Could something funny happen / be happening during insertion from one
> buffer into the other in eww-display-pdf?

In principle, maybe, but I don't see what could happen in the case in
point, given the circumstances.

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> [BTW, I don't see the function mailcap-add used anywhere.]
>
> It's for usage by users, I think?

When I do:

0. emacs -Q
1. (progn
     (require 'mailcap)
     (mailcap-add "application/pdf" "mupdf %s")
     mailcap-user-mime-data)
2. C-j

This results in:

  (("application"
    ("pdf"
     (viewer . "mupdf %s")
     (test . t)
     (type . "application/pdf"))))

But mailcap-user-mime-data is documented as being a list of

  ((viewer . VIEWER)
   (type   . MIME-TYPE)
   (test   . TEST))

Isn't this a bug?  The documentation of mailcap-add doesn't exactly
welcome the user either...

Thanks,

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> In other words, can we remove the binding altogether from
> eww-display-pdf and make the *eww pdf* buffer unibyte, as Stefan
> suggested?

Yes, this is the correct fix.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> Cc: Nicholas Harrison <nicholasharrison222@gmail.com>,  44338@debbugs.gnu.org
>> Date: Thu, 05 Nov 2020 19:04:55 +0000
>> 
>> > Great, then the change Basil already made locally will also solve the
>> > last part of the problem.
>> 
>> The change is no longer local, which prompted some off-list comments
>> from Stefan that confused me.
>> 
>> mailcap-view-mime already binds coding-system-for-write to 'binary
>> before writing to a file and spawning a subshell, so why does the
>> coding-system-for-write matter in its caller eww-display-pdf?
>
> I don't think this part of mailcap-view-mime matters in this case:
>
>   (defun mailcap-view-mime (type)
>     "View the data in the current buffer that has MIME type TYPE.
>   `mailcap--computed-mime-data' determines the method to use."
>     (let ((method (mailcap-mime-info type)))
>       (if (stringp method)
> 	  (let ((file (make-temp-file "emacs-mailcap" nil
> 				      (cadr (split-string type "/")))))
> 	    (unwind-protect
> 		(let ((coding-system-for-write 'binary))
> 		  (write-region (point-min) (point-max) file nil 'silent)
> 		  (delete-region (point-min) (point-max))
> 		  (shell-command (format method file)))
> 	      (when (file-exists-p file)
> 		(delete-file file))))
> 	(funcall method))))
>
> As you see, mailcap-view-mime only binds coding-system-for-write if
> mailcap-mime-info returns a string, which should then be a shell
> command.  But in our case, mailcap-mime-info returns doc-view-mode, a
> symbol of a function, and in that case mailcap-mime-info just calls
> the function.

Right, I got confused between the problem in the OP and my changes for
async shell commands.

I'd also forgotten that doc-view-mode writes non-visiting buffers to a
temporary file.

> That said, I don't think I understand well enough what exactly
> happened in the problematic case, because I didn't succeed in getting
> a backtrace which matched the problem description.  So the above is
> based on some looking into a crystal ball, and thus might be wrong.
>
>> In other words, can we remove the binding altogether from
>> eww-display-pdf and make the *eww pdf* buffer unibyte, as Stefan
>> suggested?
>
> If we want to remove the binding from eww-display-pdf, it should go to
> doc-view-mode, because only it knows what it needs.  mailcap-view-mime
> is right not to do anything when it calls the function, since it
> cannot know what that function will or will not do.
>
> Please also note that doc-view-mode expects to be called on a unibyte
> buffer with 'no-conversion' as its buffer-file-coding-system, because
> we have ("\\.pdf\\'" . no-conversion) in auto-coding-alist.  So an
> alternative solution would be for eww to arrange for the buffer to be
> unibyte; then no binding of coding-system-for-write will be needed.

Everyone seems to agree that that's TRT, so I've now done so on master.

>> Could something funny happen / be happening during insertion from one
>> buffer into the other in eww-display-pdf?
>
> In principle, maybe, but I don't see what could happen in the case in
> point, given the circumstances.

Me neither, I just thought it might contribute to the snafu on MS
Windows somehow.

Thanks,

-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

tags 44338 fixed
close 44338 28.1
quit

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> In other words, can we remove the binding altogether from
>> eww-display-pdf and make the *eww pdf* buffer unibyte, as Stefan
>> suggested?
>
> Yes, this is the correct fix.

Thanks, done.

Fix coding system in eww-display-pdf
4610241a9b 2020-11-05 21:06:39 +0000
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4610241a9b3fbddd1f0973bf49f7008ed09ab955

I'm therefore marking this bug as fixed in 28.1.

Nicholas, here's the cumulative change to the function eww-display-pdf,
in case you want to patch/advise yours on Emacs 27.1:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: eww.diff --]
[-- Type: text/x-diff, Size: 1157 bytes --]

diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index d6f850ca3b..43405fbd9c 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -667,14 +811,19 @@ eww-display-image
 
 (declare-function mailcap-view-mime "mailcap" (type))
 (defun eww-display-pdf ()
-  (let ((data (buffer-substring (point) (point-max))))
-    (pop-to-buffer-same-window (get-buffer-create "*eww pdf*"))
-    (let ((coding-system-for-write 'raw-text)
-	  (inhibit-read-only t))
-      (erase-buffer)
-      (insert data)
-      (mailcap-view-mime "application/pdf")))
-  (goto-char (point-min)))
+  (let ((buf (current-buffer))
+        (pos (point)))
+    (with-current-buffer (get-buffer-create "*eww pdf*")
+      (let ((inhibit-read-only t))
+        (erase-buffer)
+        (set-buffer-multibyte nil)
+        (insert-buffer-substring buf pos)
+        (mailcap-view-mime "application/pdf"))
+      (if (zerop (buffer-size))
+          ;; Buffer contents passed to shell command via temporary file.
+          (kill-buffer)
+        (goto-char (point-min))
+        (pop-to-buffer-same-window (current-buffer))))))
 
 (defun eww-setup-buffer ()
   (when (or (plist-get eww-data :url)

[-- Attachment #3: Type: text/plain, Size: 11 bytes --]


-- 
Basil

bug#44338: 27.1; EWW can't download and view pdf

[Nicholas Harrison]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

Awesome, thanks!

On Thu, Nov 5, 2020 at 2:25 PM Basil L. Contovounesios <contovob@tcd.ie>
wrote:

> tags 44338 fixed
> close 44338 28.1
> quit
>
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> > "Basil L. Contovounesios" <contovob@tcd.ie> writes:
> >
> >> In other words, can we remove the binding altogether from
> >> eww-display-pdf and make the *eww pdf* buffer unibyte, as Stefan
> >> suggested?
> >
> > Yes, this is the correct fix.
>
> Thanks, done.
>
> Fix coding system in eww-display-pdf
> 4610241a9b 2020-11-05 21:06:39 +0000
>
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4610241a9b3fbddd1f0973bf49f7008ed09ab955
>
> I'm therefore marking this bug as fixed in 28.1.
>
> Nicholas, here's the cumulative change to the function eww-display-pdf,
> in case you want to patch/advise yours on Emacs 27.1:
>
>
> --
> Basil
>

[-- Attachment #2: Type: text/html, Size: 1520 bytes --]

bug#44338: 27.1; EWW can't download and view pdf

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: nicholasharrison222@gmail.com,  44338@debbugs.gnu.org
> Date: Thu, 05 Nov 2020 21:17:22 +0000
> 
> >> Could something funny happen / be happening during insertion from one
> >> buffer into the other in eww-display-pdf?
> >
> > In principle, maybe, but I don't see what could happen in the case in
> > point, given the circumstances.
> 
> Me neither, I just thought it might contribute to the snafu on MS
> Windows somehow.

The only Windows-related snafu here could be (1) the EOL issue due to
using raw-text instead of raw-text-unix; and (2) the fact that on
Windows systems the default encoding is not UTF-8.