Bug in 9.5.3 org--file-default-apps

Bug in 9.5.3 org--file-default-apps

[Craig STCR]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

I believe a change to the last line of org--file-default-apps introduced 
in 9.5.3 is a bug.  For example, the change prevents shell scripts from 
being recognized correctly, since the mailcap logic in org-file-apps-gnu 
is no longer included in org--file-default-apps.

Best wishes,
-Craig Stcr1


org.el 9.5.2

    (defun org--file-default-apps ()
       "Return the default applications for this operating system."
       (pcase system-type
         (`darwin org-file-apps-macos)
         (`windows-nt org-file-apps-windowsnt)
         ('org-file-apps-gnu)))


org.el 9.5.3

    (defun org--file-default-apps ()
       "Return the default applications for this operating system."
       (pcase system-type
         (`darwin org-file-apps-macos)
         (`windows-nt org-file-apps-windowsnt)
         (_ org-file-apps-gnu)))


diff

    8701c8700
    <     ('org-file-apps-gnu)))
    ---
     >     (_ org-file-apps-gnu)))


[-- Attachment #2: Type: text/html, Size: 1363 bytes --]

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Craig STCR <craig.stcr1@gmail.com> writes:

> I believe a change to the last line of org--file-default-apps introduced 
> in 9.5.3 is a bug.  For example, the change prevents shell scripts from 
> being recognized correctly, since the mailcap logic in org-file-apps-gnu 
> is no longer included in org--file-default-apps.

Can you elaborate? In the past version org--file-default-apps returned
nil on Linux, while now it returns the value of org-file-apps-gnu, as
intended. What else did you expect?

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Craig STCR <craig.stcr1@gmail.com> writes:

> 9.5.3 does not return org-file-apps-gnu because org-file-apps-gnu is not 
> quoted.  Should be (and was in 9.5.2):
>
>     'org-file-apps-gnu
>
> but in 9.5.3 it is:
>
>     _ org-file-apps-gnu

Please try to run the following form:

(pcase 'gnu/linux
    (`darwin org-file-apps-macos)
    (`windows-nt org-file-apps-windowsnt)
    ('org-file-apps-gnu)) ;; => nil
   
and then

(pcase 'gnu/linux
    (`darwin org-file-apps-macos)
    (`windows-nt org-file-apps-windowsnt)
    (_ org-file-apps-gnu)) ;; => ((remote . emacs) (system . mailcap) (t . mailcap))

The second case is returning a list, which org--file-default-apps
supposed to return. The previous behaviour was erroneous. You may refer
to M-x describe-function <RET> pcase <RET> to understand the code.

Please, provide more details on the actual error you ran into. The
change in the org--file-default-apps is not a bug.

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Craig STCR <craig.stcr1@gmail.com> writes:

> OK, I'll take a look as you suggested as soon as I can.
>
> So the form in 9.5.2 was a bug?

Yes, 9.5.2 version of that function was a bug.

> The problem I encounter with the new form in 9.5.3 is that when opening 
> a shell script -- no file extension, e.g. /home/user/myscript -- 9.5.2 
> would consult mailcap and open the script in Emacs.  The mailcap entry is:
>
>     application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"
>
> But with the new form in 9.5.3, /home/user/myscript is opened by 
> /bin/less, not emacs.  I assume mailcap is not consulted.  Which does 
> not work well.  These behaviors are only for org.  Outside of org, emacs 
> behaves correctly.

mailcap does get consulted. What you are seeing happens because
mailcap.el (built-in Emacs library) is only able to recognise mime-types
by extension. So, your file is likely recognised as "nil" mimetype thus
making Org mode fallback to default mailcap handler, which is /bin/less
in your case. In Org 9.5.2 the error in org--file-default-apps made Org
mode skip using mailcap and use the last possible fallback, which is
opening in emacs. That fallback just happened to be the same with your
setting in mailcap file.

I guess that Org can also try to use `file' command (when available) to
determine the mime type. Though ideally, it should be all handled by
mailcap.el Would you mind writing to emacs-devel mailing list and asking
to add the feature of using `file' command into mailcap.el?

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

[-- Attachment #1: Type: text/plain, Size: 4426 bytes --]

[ With 'Reply All' ]

Here's a summary of what I currently observe:

1)  I was mistaken about the change from 9.5.2 to 9.5.3.  You are 
correct.  As you noted, the 9.5.2, 9.5.3 diff I previously mentioned was 
erroneous.

2) The problem I encounter with both 9.5.2 and 9.5.3 is that when 
opening a shell script -- no file extension, e.g. /home/user/myscript -- 
mailcap should be consulted and org should open the script file in 
Emacs.  The mailcap entry is:

    application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"

But instead, org opens the script file using /bin/less, not emacs.

3) The misbehavior in #2 is new.  I don't know exactly when it started 
misbehaving but I suspect it happened after updating org to 9.5.3.  But 
other packages -- elpa, emacs binaries, OS binaries -- may have been 
updated at the same time, or before that, or after that, so hard to pin 
it down.  I can't even say for sure if the upgrade to org 9.5.3 was from 
org 9.5.2.  It may have been an upgrade from an org version older than 
9.5.2.

4) Throughout this discussion, when I say "org opens" I am referring to 
the key sequence "C-c C-o" bound to org-open-at-point.  It opens script 
file with /bin/less, which is not what I would expect. <mouse-1>, bound 
to org-open-at-mouse (which calls org-open-file?) has the same misbehavior.

5) <mouse-3>, bound to org-find-file-at-mouse, *works as expected* and 
opens the script file in emacs.  Also, using a single prefix argument, 
"C-u C-c C-o" *works as expected* and opens the script file in emacs.

6) Last but not lease, when I *manually edit* org.el and change the last 
line of the org--file-default-apps function definition, removing 
underscore and space, and replacing with single quote like this:

    diff

        8701c8700
        <     (_ org-file-apps-gnu)))
        ---
         > ('org-file-apps-gnu)))

I no longer observe the misbehavior described in #2 and #4. Instead, org 
*works as expected*, like it does with #5, and opens the script file in 
emacs, not /bin/less.

I know your time is valuable.  No need for you to spend a lot of time on 
this if I am the only one having this problem.  It could be some random 
artifact in my installation, maybe something in my ~/.emacs.  My 
~/.emacs has a lot in it.  And I have a work-around, since I can use #5.

Best wishes,
-C

On 5/16/22 6:29 AM, Craig STCR wrote:
> It's possible my elpa is FUBAR.  I will uninstall, rm .elc, 
> re-install, and re-compile org 9.5.3 when I get a chance.
>
> On 5/16/22 6:08 AM, Craig STCR wrote:
>> OK, I'll take a look as you suggested as soon as I can.
>>
>> So the form in 9.5.2 was a bug?
>>
>> The problem I encounter with the new form in 9.5.3 is that when 
>> opening a shell script -- no file extension, e.g. /home/user/myscript 
>> -- 9.5.2 would consult mailcap and open the script in Emacs.  The 
>> mailcap entry is:
>>
>>     application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"
>>
>> But with the new form in 9.5.3, /home/user/myscript is opened by 
>> /bin/less, not emacs.  I assume mailcap is not consulted.  Which does 
>> not work well.  These behaviors are only for org.  Outside of org, 
>> emacs behaves correctly.
>>
>> I'll take a look as you suggested as soon as I can.
>>
>> Thanks, Ihor.
>>
>>
>> On 5/16/22 4:33 AM, Ihor Radchenko wrote:
>>> Craig STCR<craig.stcr1@gmail.com>  writes:
>>>
>>>> 9.5.3 does not return org-file-apps-gnu because org-file-apps-gnu is not
>>>> quoted.  Should be (and was in 9.5.2):
>>>>
>>>>      'org-file-apps-gnu
>>>>
>>>> but in 9.5.3 it is:
>>>>
>>>>      _ org-file-apps-gnu
>>> Please try to run the following form:
>>>
>>> (pcase 'gnu/linux
>>>      (`darwin org-file-apps-macos)
>>>      (`windows-nt org-file-apps-windowsnt)
>>>      ('org-file-apps-gnu)) ;; => nil
>>>     
>>> and then
>>>
>>> (pcase 'gnu/linux
>>>      (`darwin org-file-apps-macos)
>>>      (`windows-nt org-file-apps-windowsnt)
>>>      (_ org-file-apps-gnu)) ;; => ((remote . emacs) (system . mailcap) (t . mailcap))
>>>
>>> The second case is returning a list, which org--file-default-apps
>>> supposed to return. The previous behaviour was erroneous. You may refer
>>> to M-x describe-function <RET> pcase <RET> to understand the code.
>>>
>>> Please, provide more details on the actual error you ran into. The
>>> change in the org--file-default-apps is not a bug.
>>>
>>> Best,
>>> Ihor
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6228 bytes --]

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 16/05/2022 18:57, Ihor Radchenko wrote:
> Craig STCR writes:
>> But with the new form in 9.5.3, /home/user/myscript is opened by
>> /bin/less, not emacs.  I assume mailcap is not consulted.  Which does
>> not work well.  These behaviors are only for org.  Outside of org, emacs
>> behaves correctly.
> 
> mailcap does get consulted. What you are seeing happens because
> mailcap.el (built-in Emacs library) is only able to recognise mime-types
> by extension. So, your file is likely recognised as "nil" mimetype thus
> making Org mode fallback to default mailcap handler, which is /bin/less
> in your case.

Sounds reasonable. However I just have tried a [[file:~/path/to/script]] 
link running Org main HEAD and the file is opened in emacs (26.3) other 
window.

> I guess that Org can also try to use `file' command (when available) to
> determine the mime type. Though ideally, it should be all handled by
> mailcap.el Would you mind writing to emacs-devel mailing list and asking
> to add the feature of using `file' command into mailcap.el?

https://lists.gnu.org/archive/html/emacs-devel/2009-08/msg01057.html
Re: using libmagic in Emacs?
From: 	joakim
Subject: 	Re: using libmagic in Emacs?
Date: 	Mon, 24 Aug 2009 14:30:33 +0200

A never committed patch. It is sour from my point of view.

https://lists.gnu.org/archive/html/emacs-devel/2009-08/msg01368.html
From: 	Eli Zaretskii
Subject: 	Re: using libmagic in Emacs?
Date: 	Sun, 30 Aug 2009 06:09:23 +0300
>> From: Juri Linkov
>> Date: Sun, 30 Aug 2009 02:19:12 +0300
>> 
>> I agree that running `file' is a simpler solution.
> 
> PLEASE do not base Emacs infrastructure on external programs, unless
> they come with Emacs.  `file' is not available on every platform, and
> even on those it is, the quality and extent of its database is unclear
> and so cannot be relied upon.

https://lists.gnu.org/archive/html/emacs-devel/2009-08/msg01072.html
From: 	Richard Stallman
> How hard would it be to change the code in Emacs to recognize these
> using the existing mechanism?

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

On 5/16/22 11:14 AM, Max Nikulin wrote:
> However I just have tried a [[file:~/path/to/script]] link running Org 
> main HEAD and the file is opened in emacs (26.3) other window.

Hmmm.  That's interesting.  I upgraded from Emacs 26.x to 27.x at some 
point in the not-too-distant past.  Maybe that explains it.

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

On 5/16/22 11:14 AM, Max Nikulin wrote:
> Sounds reasonable. However I just have tried a 
> [[file:~/path/to/script]] link running Org main HEAD and the file is 
> opened in emacs (26.3) other window.

Hmmm.  That's interesting.  I upgraded from Emacs 26.x to 27.x at some 
point in the not-too-distant past.  Maybe that explains it. I'm on 
ubuntu bionic, using emacs 27.1 from kelleyk emacs ppa.

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 17/05/2022 02:29, Craig STCR wrote:
> On 5/16/22 11:14 AM, Max Nikulin wrote:
>> Sounds reasonable. However I just have tried a 
>> [[file:~/path/to/script]] link running Org main HEAD and the file is 
>> opened in emacs (26.3) other window.
> 
> Hmmm.  That's interesting.  I upgraded from Emacs 26.x to 27.x at some 
> point in the not-too-distant past.  Maybe that explains it. I'm on 
> ubuntu bionic, using emacs 27.1 from kelleyk emacs ppa.

Craig, Have you tried it in Emacs-27 without your default config? I mean 
something like

     emacs -Q -L ~/src/org-mode/lisp test.org

I do not think emacs version matters here. It works even with emacs-27 
for me.

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 16/05/2022 18:57, Ihor Radchenko wrote:
>> Craig STCR writes:
>>> But with the new form in 9.5.3, /home/user/myscript is opened by
>>> /bin/less, not emacs.  I assume mailcap is not consulted.  Which does
>>> not work well.  These behaviors are only for org.  Outside of org, emacs
>>> behaves correctly.
>> 
>> mailcap does get consulted. What you are seeing happens because
>> mailcap.el (built-in Emacs library) is only able to recognise mime-types
>> by extension. So, your file is likely recognised as "nil" mimetype thus
>> making Org mode fallback to default mailcap handler, which is /bin/less
>> in your case.
>
> Sounds reasonable. However I just have tried a [[file:~/path/to/script]] 
> link running Org main HEAD and the file is opened in emacs (26.3) other 
> window.

The behaviour is dependent on the OS-level mailcap defaults. What does
(mailcap-mime-info nil) return for you?

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Craig STCR <craig.stcr1@gmail.com> writes:

> I know your time is valuable.  No need for you to spend a lot of time on 
> this if I am the only one having this problem.  It could be some random 
> artifact in my installation, maybe something in my ~/.emacs.  My 
> ~/.emacs has a lot in it.  And I have a work-around, since I can use #5.

Using #5 may surprise you if you actually set something via mailcap.
Can you instead try to put
text/plain; emacs27 %s; test=test -n "$DISPLAY"

into your mailcap file and report if it helps?

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 18/05/2022 18:36, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Sounds reasonable. However I just have tried a [[file:~/path/to/script]]
>> link running Org main HEAD and the file is opened in emacs (26.3) other
>> window.
> 
> The behaviour is dependent on the OS-level mailcap defaults. What does
> (mailcap-mime-info nil) return for you?

view-mode

run-mailcap --norun ~/path/to/script

reports less.

Craig wrote that he runs ubuntu as well (however even older LTS), so I 
do not expect significant difference.

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 18/05/2022 18:36, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>>> Sounds reasonable. However I just have tried a [[file:~/path/to/script]]
>>> link running Org main HEAD and the file is opened in emacs (26.3) other
>>> window.
>> 
>> The behaviour is dependent on the OS-level mailcap defaults. What does
>> (mailcap-mime-info nil) return for you?
>
> view-mode
>
> run-mailcap --norun ~/path/to/script

Note that mailcap.el does _not_ use run-mailcap or anything external. It
is written purely in elisp.

So, can you please report the output of M-: (mailcap-mime-info nil)?

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 19/05/2022 20:39, Ihor Radchenko wrote:
> Max Nikulin writes:
>> On 18/05/2022 18:36, Ihor Radchenko wrote:
>>>
>>> The behaviour is dependent on the OS-level mailcap defaults. What does
>>> (mailcap-mime-info nil) return for you?
>>
>> view-mode
    ^^^^^^^^^

>> run-mailcap --norun ~/path/to/script
> 
> Note that mailcap.el does _not_ use run-mailcap or anything external. It
> is written purely in elisp.
> 
> So, can you please report the output of M-: (mailcap-mime-info nil)?

view-mode

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> run-mailcap --norun ~/path/to/script
>> 
>> Note that mailcap.el does _not_ use run-mailcap or anything external. It
>> is written purely in elisp.
>> 
>> So, can you please report the output of M-: (mailcap-mime-info nil)?
>
> view-mode

Sorry, I did not consider that you customised mailcap-user-mime-data (I
guess). No doubt your Emacs opens the script in other window.

What about using emacs -Q?

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 20/05/2022 15:22, Ihor Radchenko wrote:
> Max Nikulin <manikulin@gmail.com> writes:
> 
>>>> run-mailcap --norun ~/path/to/script
>>>
>>> Note that mailcap.el does _not_ use run-mailcap or anything external. It
>>> is written purely in elisp.
>>>
>>> So, can you please report the output of M-: (mailcap-mime-info nil)?
>>
>> view-mode
> 
> Sorry, I did not consider that you customised mailcap-user-mime-data (I
> guess). No doubt your Emacs opens the script in other window.
> 
> What about using emacs -Q?

It is the same with and without -Q and no, I have not customized 
`mailcap-user-mime-data', its value is nil, easy customization interface 
tells that it is the standard value. There is a chance that debian has 
some patch, but most of debian specific is disabled when -Q option is used.

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 20/05/2022 15:22, Ihor Radchenko wrote:
>> Max Nikulin <manikulin@gmail.com> writes:
>> 
>>>>> run-mailcap --norun ~/path/to/script
>>>>
>>>> Note that mailcap.el does _not_ use run-mailcap or anything external. It
>>>> is written purely in elisp.
>>>>
>>>> So, can you please report the output of M-: (mailcap-mime-info nil)?
>>>
>>> view-mode
>> 
>> Sorry, I did not consider that you customised mailcap-user-mime-data (I
>> guess). No doubt your Emacs opens the script in other window.
>> 
>> What about using emacs -Q?
>
> It is the same with and without -Q and no, I have not customized 
> `mailcap-user-mime-data', its value is nil, easy customization interface 
> tells that it is the standard value. There is a chance that debian has 
> some patch, but most of debian specific is disabled when -Q option is used.

Interesting. In any case, it confirms that mailcap behaviour depends on
both Emacs settings and also system settings.

As I understand now:
1. (mailcap-mime-info nil) in Emacs 26 returns 'view-mode disregarding
   ~/.mailcap
2. (mailcap-mime-info nil) in Emacs 27 returns "less %s" on my system
   unless I have ~/.mailcap, in which case it takes text/plain handler
   from ~/.mailcap
3. (mailcap-mime-info nil) in Emacs 28/29 returns 'fundamental-mode
   unless I have ~/.mailcap, in which case it takes text/plain handler
   from there

Dear Craig, your issue will kind of disappear if you upgrade Emacs or
provide plain/text handler in ~/.mailcap.

On Org side, the very reason we call (mailcap-mime-info nil) with _nil_
argument is because mailcap.el does not recognise file types without
extension. I think that we can prefer the output of `file' utility if it
is available on the system. However, I am not sure what to do on
Windows/Mac.

Best,
Ihor
   

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 21/05/2022 08:44, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> It is the same with and without -Q and no, I have not customized
>> `mailcap-user-mime-data', its value is nil, easy customization interface
>> tells that it is the standard value. There is a chance that debian has
>> some patch, but most of debian specific is disabled when -Q option is used.
> 
> Interesting. In any case, it confirms that mailcap behaviour depends on
> both Emacs settings and also system settings.

You should be even more surprised, if I say that [[file:~/tstorg]] link 
is opened in emacs buffer in shell-script mode (emacs-27.1, org main 
HEAD) despite

    (mailcap-mime-info nil)
    nil

actually it is a minimal LXC container with no mime-support package 
installed, so /etc/mailcap file is absent.

Notice `mailcap-mime-data' has the following entries for a long time
     ("text"
      ("plain"
       (viewer  . view-mode)
       (type    . "text/plain"))
      ("plain"
       (viewer  . fundamental-mode)
       (type    . "text/plain"))

The source of the problem is that Emacs-27 was released with the 
following bug:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40247
mailcap-mime-data erased when parsing mime parts

So `mailcap-parse-mailcaps' called from `mailcap-mime-info' erases 
predefined associations in Emacs-27.

[PATCH] Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> On 21/05/2022 08:44, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>>> It is the same with and without -Q and no, I have not customized
>>> `mailcap-user-mime-data', its value is nil, easy customization interface
>>> tells that it is the standard value. There is a chance that debian has
>>> some patch, but most of debian specific is disabled when -Q option is used.
>> 
>> Interesting. In any case, it confirms that mailcap behaviour depends on
>> both Emacs settings and also system settings.
>
> You should be even more surprised, if I say that [[file:~/tstorg]] link 
> is opened in emacs buffer in shell-script mode (emacs-27.1, org main 
> HEAD) despite
>
>     (mailcap-mime-info nil)
>     nil

>
> The source of the problem is that Emacs-27 was released with the 
> following bug:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40247
> mailcap-mime-data erased when parsing mime parts
>
> So `mailcap-parse-mailcaps' called from `mailcap-mime-info' erases 
> predefined associations in Emacs-27.

If I understand correctly, this extra complication does not affect most
of the systems. I am not sure if we need to work around it.

Also, I am attaching a patch to address the original issue. We can just
use file command when available. WDYT?

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-open-file-Use-file-command-to-determine-mime-typ.patch --]
[-- Type: text/x-patch, Size: 1518 bytes --]

From f69824559d62cb6963ff30f11ceb46303bf1b3d4 Mon Sep 17 00:00:00 2001
Message-Id: <f69824559d62cb6963ff30f11ceb46303bf1b3d4.1653192368.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 22 May 2022 12:04:35 +0800
Subject: [PATCH] org-open-file: Use file command to determine mime type, when
 available

* lisp/org.el (org-open-file): Prefer file command to determine file
type instead of relying on `mailcap-extension-to-mime'.  Only fallback
to the latter when file command is not available on the system.

Fixes https://list.orgmode.org/874k1n2hpv.fsf@localhost/T/#mcc10df4ea7937360671e6dea8061153dee518807
---
 lisp/org.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/org.el b/lisp/org.el
index d7da8efc4..3102fe611 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -7975,7 +7975,12 @@ (defun org-open-file (path &optional in-emacs line search)
     (when (eq cmd 'mailcap)
       (require 'mailcap)
       (mailcap-parse-mailcaps)
-      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
+      (let* ((mime-type (if (executable-find "file")
+                            (shell-command-to-string
+                             (format "%s --brief --mime-type %s"
+                                     (executable-find "file")
+                                     file))
+                          (mailcap-extension-to-mime (or ext ""))))
 	     (command (mailcap-mime-info mime-type)))
 	(if (stringp command)
 	    (setq cmd command)
-- 
2.35.1

Re: [PATCH] Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 22/05/2022 11:10, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> The source of the problem is that Emacs-27 was released with the
>> following bug:
>>
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40247
>> mailcap-mime-data erased when parsing mime parts
>>
>> So `mailcap-parse-mailcaps' called from `mailcap-mime-info' erases
>> predefined associations in Emacs-27.
> 
> If I understand correctly, this extra complication does not affect most
> of the systems. I am not sure if we need to work around it.

I would say that view-mode is quite reasonable default to open a 
"text/plain" file and this bug broke it. I do not think that Craig 
really wants a new emacs session for every followed link:

     application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"

> Also, I am attaching a patch to address the original issue. We can just
> use file command when available. WDYT?

Ihor, have you manged to reproduce the original issue? Are links with 
explicit .txt suffix [[file:file.txt]] affected by the same problem? My 
environments sometimes behave in a way unexpected to you and I have not 
setup any tool to quickly launch transient virtual machines with no fear 
to "broke" current state, so I have not tried to debug the reported 
issue in its original form.

I may be excessively suspicious.

> diff --git a/lisp/org.el b/lisp/org.el
> index d7da8efc4..3102fe611 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -7975,7 +7975,12 @@ (defun org-open-file (path &optional in-emacs line search)
>      (when (eq cmd 'mailcap)
>        (require 'mailcap)
>        (mailcap-parse-mailcaps)
> -      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
> +      (let* ((mime-type (if (executable-find "file")
> +                            (shell-command-to-string
> +                             (format "%s --brief --mime-type %s"
> +                                     (executable-find "file")
> +                                     file))

I hate elisp API related to executing of external processes because it 
encourages proliferation of unsafe code. What if the linked file name 
has some peculiarities and characters interpreted by shell?

    See [[file:/tmp/`touch /tmp/hacked`/test][here]]

I can not say that I fully understand `org-open-file' code, so I am 
unsure if remote file name can appear here, e.g. /ssh:user@host:testfie 
or a file form an archive due to a relative link [[file:testfile]] from 
a remote .org file. When remote files are not an issue, it is safer to 
use functions that takes command arguments as a list of string, not the 
command as a ready to execute string. Unfortunately there is no helper 
returning a string and accepting a command as a list.

> +                          (mailcap-extension-to-mime (or ext ""))))
>  	     (command (mailcap-mime-info mime-type)))
>  	(if (stringp command)
>  	    (setq cmd command)

P.S. `org-open-file' already has some problems with handling of some 
file names:

Maxim Nikulin to emacs-orgmode. greedy substitution in org-open-file. 
Wed, 20 Jan 2021 23:08:35 +0700. 
https://list.orgmode.org/ru9ki4$t5e$1@ciao.gmane.io

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

Thanks all for your help!

On 5/20/22 9:44 PM, Ihor Radchenko wrote:
> Dear Craig, ... or provide plain/text handler in ~/.mailcap.
>
OK, I did a first-try on this and was unsuccessful, but I'm sure it's 
user error.  I need to refresh my knowledge on how to customize 
user-local mime database, and that will write-out a new ~/.mailcap, etc, 
I think?  I've done it before, but it was awhile ago, and I wasn't 
paying attention to ~/.mailcap when I did it.  I know for Gnome I can 
create a .desktop file.  But I know there's a way to customize 
user-local mime database without Gnome desktop.  I'll take a closer look 
when I have a little more time.


On 5/20/22 9:44 PM, Ihor Radchenko wrote:
> However, I am not sure what to do on Windows/Mac.
Maybe try a quick-and-dirty, cross-platform solution that checks 
non-binary files for a first-line shebang?  Could use existing Emacs 
hooks that determine major-mode when opening files.

Again, thanks all for your help!

Best,
-Craig

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

I guess maybe I should have given a little better description of what I 
tried that did NOT work?  But it's a little off-topic for this mailing 
list.  Nevertheless, here it is...

I created a ~/.mailcap file and put this in it, which I cut and pasted 
from /etc/mailcap:

    application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"

But obviously that's not going to change anything, since it's already in 
the system mailcap file, /etc/mailcap.  DOH!  And sure enough, running 
'>$ run-mailcap myscript' invokes 'less'.  But what I wasn't expecting 
is that running '>$ update-mime -- local' gives me: "Error: 
'/home/user/.mailcap' is not in required format -- not updated".   Not 
sure why I'm getting that when I cut-and-pasted from /etc/mailcap.

No worries.  I'll look in more depth later.


On 5/23/22 8:40 AM, Craig STCR wrote:
> Thanks all for your help!
>
> On 5/20/22 9:44 PM, Ihor Radchenko wrote:
>> Dear Craig, ... or provide plain/text handler in ~/.mailcap.
>>
> OK, I did a first-try on this and was unsuccessful, but I'm sure it's 
> user error.  I need to refresh my knowledge on how to customize 
> user-local mime database, and that will write-out a new ~/.mailcap, 
> etc, I think?  I've done it before, but it was awhile ago, and I 
> wasn't paying attention to ~/.mailcap when I did it.  I know for Gnome 
> I can create a .desktop file.  But I know there's a way to customize 
> user-local mime database without Gnome desktop. I'll take a closer 
> look when I have a little more time.
>
>
> On 5/20/22 9:44 PM, Ihor Radchenko wrote:
>> However, I am not sure what to do on Windows/Mac.
> Maybe try a quick-and-dirty, cross-platform solution that checks 
> non-binary files for a first-line shebang?  Could use existing Emacs 
> hooks that determine major-mode when opening files.
>
> Again, thanks all for your help!
>
> Best,
> -Craig
>

[-- Attachment #2: Type: text/html, Size: 2671 bytes --]

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

[-- Attachment #1: Type: text/plain, Size: 2598 bytes --]

Everything I mentioned so far has been my Bionic desktop.  Which, 
incidentally was an upgrade from Xenial, not a clean install. But... in 
my Focal headless container, if I run:

     >$ run-mailcap myscript

it invokes emacs.  Yay!  It works!  DISPLAY, EDITOR, and VISUAL are all 
unset or empty.  There's nothing in /etc/mailcap for emacs, and what 
*IS* in /etc/mailcap is (amongst others of course):

    application/x-shellscript; vim %s; needsterminal

So I really need to spend some time reading man pages about mailcap, 
mime, etc because that's not what I would expect, and I'm really confused.

On 5/23/22 8:59 AM, Craig STCR wrote:
> I guess maybe I should have given a little better description of what 
> I tried that did NOT work?  But it's a little off-topic for this 
> mailing list.  Nevertheless, here it is...
>
> I created a ~/.mailcap file and put this in it, which I cut and pasted 
> from /etc/mailcap:
>
>     application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"
>
> But obviously that's not going to change anything, since it's already 
> in the system mailcap file, /etc/mailcap.  DOH!  And sure enough, 
> running '>$ run-mailcap myscript' invokes 'less'.  But what I wasn't 
> expecting is that running '>$ update-mime -- local' gives me: "Error: 
> '/home/user/.mailcap' is not in required format -- not updated".   Not 
> sure why I'm getting that when I cut-and-pasted from /etc/mailcap.
>
> No worries.  I'll look in more depth later.
>
>
> On 5/23/22 8:40 AM, Craig STCR wrote:
>> Thanks all for your help!
>>
>> On 5/20/22 9:44 PM, Ihor Radchenko wrote:
>>> Dear Craig, ... or provide plain/text handler in ~/.mailcap.
>>>
>> OK, I did a first-try on this and was unsuccessful, but I'm sure it's 
>> user error.  I need to refresh my knowledge on how to customize 
>> user-local mime database, and that will write-out a new ~/.mailcap, 
>> etc, I think?  I've done it before, but it was awhile ago, and I 
>> wasn't paying attention to ~/.mailcap when I did it.  I know for 
>> Gnome I can create a .desktop file.  But I know there's a way to 
>> customize user-local mime database without Gnome desktop.  I'll take 
>> a closer look when I have a little more time.
>>
>>
>> On 5/20/22 9:44 PM, Ihor Radchenko wrote:
>>> However, I am not sure what to do on Windows/Mac.
>> Maybe try a quick-and-dirty, cross-platform solution that checks 
>> non-binary files for a first-line shebang?  Could use existing Emacs 
>> hooks that determine major-mode when opening files.
>>
>> Again, thanks all for your help!
>>
>> Best,
>> -Craig
>>
>

[-- Attachment #2: Type: text/html, Size: 3739 bytes --]

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]



On 5/23/22 10:14 AM, Craig STCR wrote:
>
>     >$ run-mailcap myscript
>
> it invokes emacs.  Yay!  It works!

Double DOH!  Forget what I said.  It invokes vim, lol.  That *is* what I 
would expect.  Sorry for the noise.  I need to think a little more 
before I hit <SEND>.

[-- Attachment #2: Type: text/html, Size: 750 bytes --]

Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

Craig, discussing the issue with Ihor, I sent some messages to the mail 
list only without your address in Cc. You may check complete thread at 
https://list.orgmode.org

On 23/05/2022 19:40, Craig STCR wrote:
>
> OK, I did a first-try on this and was unsuccessful, but I'm sure it's 
> user error.  I need to refresh my knowledge on how to customize 
> user-local mime database, and that will write-out a new ~/.mailcap, etc, 
> I think?  I've done it before, but it was awhile ago, and I wasn't 
> paying attention to ~/.mailcap when I did it.  I know for Gnome I can 
> create a .desktop file.  But I know there's a way to customize 
> user-local mime database without Gnome desktop.  I'll take a closer look 
> when I have a little more time.

Debian package scripts (that work on Ubuntu as well) extract MIME info 
from *system* .desktop files to add them to mailcap database.

There were some mailcap related changes and bugs around Emacs-27 
release, e.g. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36771

> application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"

Notice that such entry will be ignored when DISPLAY is not set due to 
specified "test" property. You need to pass X socket and pass or set 
DISPLAY environment to you "headless" container.

update-mime likely assumes that you created ~/.mailcap.order to generate 
~/.mailcap from it. Emacs may just read ~/.mailcap, so if you created 
this file, nothing more is required.

Actually I do not think you wish to launch another emacs session in 
response to following a link in an .org file. I suppose, you experienced
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40247
otherwise (mailcap-mime-info nil) (or "text/plain") would return 
view-mode and your scripts would be opened in another emacs window. It 
should work without mailcap entries.

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Craig STCR <craig.stcr1@gmail.com> writes:

> On 5/20/22 9:44 PM, Ihor Radchenko wrote:
>> However, I am not sure what to do on Windows/Mac.
> Maybe try a quick-and-dirty, cross-platform solution that checks 
> non-binary files for a first-line shebang?  Could use existing Emacs 
> hooks that determine major-mode when opening files.

Good idea. Currently, Org only supports auto-mode-alist (see
org-file-apps docstring) checking the filename. I think that the same
option can be extended to use magic-mode-alist and such.

I will leave it for others to implement. Patches are welcome!

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Craig STCR <craig.stcr1@gmail.com> writes:

> '>$ run-mailcap myscript'

One comment. I do not personally have run-mailcap command on my system,
but searching for its docstring reveals
(http://www.linux-commands-examples.com/run-mailcap):

>>  If the mime-type is omitted, an attempt to determine the type is
>>  made by trying to match the file’s extension with those in the
>>  mime.types files.

So, run-mailcap itself does not look inside the file and only checks the
extension. AFAIU. Correct me if I am wrong.

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"
>
> Notice that such entry will be ignored when DISPLAY is not set due to 
> specified "test" property. You need to pass X socket and pass or set 
> DISPLAY environment to you "headless" container.

Also, note that mailcap.el contains `mailcap-mailcap-entry-passes-test'
and that function is ... interesting. At least, it should work fine with
the test above, but only very limited number of tests appears to be
supported.

Best,
Ihor

Re: Bug in 9.5.3 org--file-default-apps

[Craig STCR]

Yeah I was wondering about that 'test' line.  I was wondering if, like 
you say, it gets ignored.  Or if the true / false value gets passed to 
some other logic that would, for example, add a '-nw' to the emacs 
command line.  I have no idea.

On 5/25/22 2:24 AM, Ihor Radchenko wrote:
> Max Nikulin <manikulin@gmail.com> writes:
>
>>> application/x-shellscript; emacs27 %s; test=test -n "$DISPLAY"
>> Notice that such entry will be ignored when DISPLAY is not set due to
>> specified "test" property. You need to pass X socket and pass or set
>> DISPLAY environment to you "headless" container.
> Also, note that mailcap.el contains `mailcap-mailcap-entry-passes-test'
> and that function is ... interesting. At least, it should work fine with
> the test above, but only very limited number of tests appears to be
> supported.
>
> Best,
> Ihor

[PATCH v2] Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 4534 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> On 22/05/2022 11:10, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>>> The source of the problem is that Emacs-27 was released with the
>>> following bug:
>>>
>>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40247
>>> mailcap-mime-data erased when parsing mime parts
>>>
>>> So `mailcap-parse-mailcaps' called from `mailcap-mime-info' erases
>>> predefined associations in Emacs-27.
>> 
>> If I understand correctly, this extra complication does not affect most
>> of the systems. I am not sure if we need to work around it.
>
> I would say that view-mode is quite reasonable default to open a 
> "text/plain" file and this bug broke it.

I agree. However, I am already a bit lost with all the complications.
This particular one does not appear to be directly relevant to the
initial problem with a file with no extension. I'd prefer if this
Emacs-27 issue were reported in a separate thread.

>> Also, I am attaching a patch to address the original issue. We can just
>> use file command when available. WDYT?
>
> Ihor, have you manged to reproduce the original issue? Are links with 
> explicit .txt suffix [[file:file.txt]] affected by the same problem? My 
> environments sometimes behave in a way unexpected to you and I have not 
> setup any tool to quickly launch transient virtual machines with no fear 
> to "broke" current state, so I have not tried to debug the reported 
> issue in its original form.
>
> I may be excessively suspicious.

Yes, I have managed to reproduce the original issue. Kind of.

We have the following in org-open-file:

(when (eq cmd 'mailcap)
      (require 'mailcap)
      (mailcap-parse-mailcaps)
      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
	     (command (mailcap-mime-info mime-type)))
	(if (stringp command)
	    (setq cmd command)
	  (setq cmd 'emacs))))

with EXT being bound to file extension.
When file does not have an extension (the original issue),
(mailcap-extension-to-mime) returns nil.
Then, (mailcap-mime-info nil) returns the same result with
(mailcap-mime-info "text/plain"). It is hard-coded inside
mailcap-mime-info.

So, with the current default value of org-file-apps-gnu, files with no
extension always use mime handler assigned to "text/plain" mimetype.
In a way, it is not wrong - mailcap spec only considers file extension
and has undefined behavior for files with no extension.
However, it does not make sense from the user perspective. Hence, I am
suggesting this patch. =file= command (when available) is more powerful
and looks at the file contents, not just into its extension.

>> +      (let* ((mime-type (if (executable-find "file")
>> +                            (shell-command-to-string
>> +                             (format "%s --brief --mime-type %s"
>> +                                     (executable-find "file")
>> +                                     file))
>
> I hate elisp API related to executing of external processes because it 
> encourages proliferation of unsafe code. What if the linked file name 
> has some peculiarities and characters interpreted by shell?
>
>     See [[file:/tmp/`touch /tmp/hacked`/test][here]]

Thanks for the heads up! Updated version of the patch is attached.

> I can not say that I fully understand `org-open-file' code, so I am 
> unsure if remote file name can appear here, e.g. /ssh:user@host:testfie 
> or a file form an archive due to a relative link [[file:testfile]] from 
> a remote .org file. When remote files are not an issue, it is safer to 
> use functions that takes command arguments as a list of string, not the 
> command as a ready to execute string. Unfortunately there is no helper 
> returning a string and accepting a command as a list.

org-file-apps-gnu defaults to

((remote . emacs)
 (system . mailcap)
 (t . mailcap))

All the "remote" files (including /ssh) will be opened in emacs. We
should have no issue in this area.

>> +                          (mailcap-extension-to-mime (or ext ""))))
>>  	     (command (mailcap-mime-info mime-type)))
>>  	(if (stringp command)
>>  	    (setq cmd command)
>
> P.S. `org-open-file' already has some problems with handling of some 
> file names:
>
> Maxim Nikulin to emacs-orgmode. greedy substitution in org-open-file. 
> Wed, 20 Jan 2021 23:08:35 +0700. 
> https://list.orgmode.org/ru9ki4$t5e$1@ciao.gmane.io

Agree. Though it is not directly related. Maybe you can bump that thread
and mark is as a bug report to be listed on
https://updates.orgmode.org/?

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-org-open-file-Use-file-command-to-determine-mime-.patch --]
[-- Type: text/x-patch, Size: 1572 bytes --]

From 4aeff613c07d9025c5fc1f0b3851870a42d36869 Mon Sep 17 00:00:00 2001
Message-Id: <4aeff613c07d9025c5fc1f0b3851870a42d36869.1653538199.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 22 May 2022 12:04:35 +0800
Subject: [PATCH v2] org-open-file: Use file command to determine mime type,
 when available

* lisp/org.el (org-open-file): Prefer file command to determine file
type instead of relying on `mailcap-extension-to-mime'.  Only fallback
to the latter when file command is not available on the system.

Fixes https://list.orgmode.org/874k1n2hpv.fsf@localhost/T/#mcc10df4ea7937360671e6dea8061153dee518807
---
 lisp/org.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/org.el b/lisp/org.el
index d7da8efc4..45a179a8a 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -7975,7 +7975,12 @@ (defun org-open-file (path &optional in-emacs line search)
     (when (eq cmd 'mailcap)
       (require 'mailcap)
       (mailcap-parse-mailcaps)
-      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
+      (let* ((mime-type (if (executable-find "file")
+                            (shell-command-to-string
+                             (format "%s --brief --mime-type %s"
+                                     (executable-find "file")
+                                     (shell-quote-argument (convert-standard-filename file))))
+                          (mailcap-extension-to-mime (or ext ""))))
 	     (command (mailcap-mime-info mime-type)))
 	(if (stringp command)
 	    (setq cmd command)
-- 
2.35.1

Re: [PATCH v2] Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

Ihor,

Your patch may be an improvement, but it requires an additional fix, see 
below.

However, in my opinion, it does not address original Craig's issue. The 
patch improves handling of *non-text* files requiring *external* 
viewers, while Craig complained that behavior for shell script is 
incorrect and his problem is tightly bound to erased `mailcap-mime-data'.

I can not reproduce behavior he observed exactly, Org does not opens 
shell scripts by less, but it tries and silently (it is expected) fails.

My results (emacs-27):
1. If there is no mailcap files at all, the script is opened
    in the same emacs session that is correct from my point of view.
2. If I add ~/.mailcap
        text/plain; less '%s'; needsterminal
    then I get silent failures
        Running less /etc/profile...done
3. With your patch and the following additional entry in ~/.mailcap
    (notice "text" instead of "application" and just "emacs")
        text/x-shellscript; emacs %s; test=test -n "$DISPLAY"
    new Emacs session is started. It is a problem but partially
    it is caused by incorrect mailcap configuration.
    Unlike "text/plain" that would be handled by view-mode
    unless `mailcap-mime-data' were erased in Emacs-27,
    "text/x-shellscript" is handled by less on my main system
    due to mailcap while I would expect same Emacs session.

I read implementation of `org-open-file' once more and now I see that 
currently remote files can not be processed by mailcap code path even 
with custom `org-file-apps', so thank you for explanation. In some cases 
it may be handy to launch remote viewer from a host accessed through 
ssh, but let's leave it aside.

> -      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
> +      (let* ((mime-type (if (executable-find "file")

I would consider (and ... (not remp)) despite currently it is redundant. 
Just to mitigate consequences if other parts of this complicated 
function will be modified. On the other hand 
`start-process-shell-command' is not ready for remote files anyway, so 
major cleanup would be required.

> +                            (shell-command-to-string
> +                             (format "%s --brief --mime-type %s"
> +                                     (executable-find "file")
> +                                     (shell-quote-argument (convert-standard-filename file))))

It is not enough to cure my paranoia. However the following case is 
rather pathological

mkdir 'Program Files'
ln -s /usr/bin/file 'Program Files'/
PATH="$HOME/Program Files:$PATH" \
    emacs -Q -L ~/src/org-mode/lisp/ ~/examples/org/open-script.org &

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
   string-match("/" nil 0)
   split-string(nil "/")
   mailcap-mime-info("/bin/bash: line 1: /home/ubuntu/Program: No such 
f...")
   (let* ((mime-type (if (executable-find "file") 
(shell-command-to-string (format "%s --brief --mime-type %s" 
(executable-find "file") (shell-quote-argument 
(convert-standard-filename file)))) (mailcap-extension-to-mime (or ext 
"")))) (command (mailcap-mime-info mime-type))) (if (stringp command) 
(setq cmd command) (setq cmd 'emacs)))


> +                          (mailcap-extension-to-mime (or ext ""))))

> Agree. Though it is not directly related. Maybe you can bump that thread
> and mark is as a bug report to be listed on
> https://updates.orgmode.org/?

It is already tracked there as
"org-open-file & org-file-apps multiple substitution bug"
but my point was that great care is required otherwise a lot of issues 
may happen with shell command.

Ihor Radchenko. Bug in 9.5.3 org--file-default-apps.
Wed, 25 May 2022 14:18:10 +0800.
https://list.orgmode.org/8735gy2l0d.fsf@localhost

>> '>$ run-mailcap myscript'
> 
> One comment. I do not personally have run-mailcap command on my system,
> but searching for its docstring reveals
> (http://www.linux-commands-examples.com/run-mailcap):
> 
>>>  If the mime-type is omitted, an attempt to determine the type is
>>>  made by trying to match the file’s extension with those in the
>>>  mime.types files.
> 
> So, run-mailcap itself does not look inside the file and only checks the
> extension. AFAIU. Correct me if I am wrong.

It is a Debian extension. Local man page has the following statement 
(confirmed by code and strace):

> If no  mime-type is found, a last attempt will be done by running the
> file command, if available.

Re: [PATCH v2] Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 26/05/2022 11:23, Ihor Radchenko wrote:
> diff --git a/lisp/org.el b/lisp/org.el
> index d7da8efc4..45a179a8a 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -7975,7 +7975,12 @@ (defun org-open-file (path &optional in-emacs line search)
>      (when (eq cmd 'mailcap)
>        (require 'mailcap)
>        (mailcap-parse-mailcaps)
> -      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
> +      (let* ((mime-type (if (executable-find "file")
> +                            (shell-command-to-string
> +                             (format "%s --brief --mime-type %s"

Another corner case:

    file --brief --mime-type tstorg-sh-symlink
    inode/symlink
    file --brief --mime-type --dereference tstorg-sh-symlink
    text/x-shellscript

> +                                     (executable-find "file")
> +                                     (shell-quote-argument (convert-standard-filename file))))
> +                          (mailcap-extension-to-mime (or ext ""))))

Actually MIME type for shell scripts varies a lot

(mailcap-extension-to-mime "sh") => "text/x-sh"

run-mailcap --norun examples/org/script/tstorg.sh
Error: no "view" mailcap rules found for type "application/x-sh"

And "text/x-shellscript" as above.

[PATCH v3] Re: Bug in 9.5.3 org--file-default-apps

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 5563 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> However, in my opinion, it does not address original Craig's issue. The 
> patch improves handling of *non-text* files requiring *external* 
> viewers, while Craig complained that behavior for shell script is 
> incorrect and his problem is tightly bound to erased `mailcap-mime-data'.

Ideally, we need a feedback from him.
For emacs-27 specifically, we might need to work around the bug.
However, I am not sure what would be the best way to do it. The easiest
can be changing the default value of org-file-apps-gnu on Emacs 27
specifically to not use mailcap at all. But I am pretty sure that we can
do better.

> I can not reproduce behavior he observed exactly, Org does not opens 
> shell scripts by less, but it tries and silently (it is expected) fails.
>
> My results (emacs-27):
> 1. If there is no mailcap files at all, the script is opened
>     in the same emacs session that is correct from my point of view.
> 2. If I add ~/.mailcap
>         text/plain; less '%s'; needsterminal
>     then I get silent failures
>         Running less /etc/profile...done
> 3. With your patch and the following additional entry in ~/.mailcap
>     (notice "text" instead of "application" and just "emacs")
>         text/x-shellscript; emacs %s; test=test -n "$DISPLAY"
>     new Emacs session is started. It is a problem but partially
>     it is caused by incorrect mailcap configuration.
>     Unlike "text/plain" that would be handled by view-mode
>     unless `mailcap-mime-data' were erased in Emacs-27,
>     "text/x-shellscript" is handled by less on my main system
>     due to mailcap while I would expect same Emacs session.

I am confused here. org-file-apps-gnu says that we rely on mailcap:

((remote . emacs)
 (system . mailcap)
 (t . mailcap))

So, is (3) following what you would expect from mailcap (regardless
whether it is incorrectly configured or not)? Wrong configuration of
mailcap is none of Org business - we need not to be "smart" and fix user
"errors". They may be deliberate.

> I read implementation of `org-open-file' once more and now I see that 
> currently remote files can not be processed by mailcap code path even 
> with custom `org-file-apps', so thank you for explanation. In some cases 
> it may be handy to launch remote viewer from a host accessed through 
> ssh, but let's leave it aside.

This is exactly why you can always customize org-file-apps. 

>> -      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
>> +      (let* ((mime-type (if (executable-find "file")
>
> I would consider (and ... (not remp)) despite currently it is redundant. 
> Just to mitigate consequences if other parts of this complicated 
> function will be modified. On the other hand 
> `start-process-shell-command' is not ready for remote files anyway, so 
> major cleanup would be required.

I would be in favor of a cleanup (by someone™), but I am against
redundancy. Such redundancy may mask bugs making them difficult to
debug. Not to mention code readability.

>> +                            (shell-command-to-string
>> +                             (format "%s --brief --mime-type %s"
>> +                                     (executable-find "file")
>> +                                     (shell-quote-argument (convert-standard-filename file))))
>
> It is not enough to cure my paranoia. However the following case is 
> rather pathological
>
> mkdir 'Program Files'
> ln -s /usr/bin/file 'Program Files'/
> PATH="$HOME/Program Files:$PATH" \
>     emacs -Q -L ~/src/org-mode/lisp/ ~/examples/org/open-script.org &
>
> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>    string-match("/" nil 0)
>    split-string(nil "/")
>    mailcap-mime-info("/bin/bash: line 1: /home/ubuntu/Program: No such 
> f...")
>    (let* ((mime-type (if (executable-find "file") 
> (shell-command-to-string (format "%s --brief --mime-type %s" 
> (executable-find "file") (shell-quote-argument 
> (convert-standard-filename file)))) (mailcap-extension-to-mime (or ext 
> "")))) (command (mailcap-mime-info mime-type))) (if (stringp command) 
> (setq cmd command) (setq cmd 'emacs)))

Well. If we want to be this paranoid, could you write a generic safe
shell-command wrapper that takes care of various edge cases? Then, we
can add that wrapper to org-macs and reuse it in various places where we
need to run external command.

> Another corner case:
>
>     file --brief --mime-type tstorg-sh-symlink
>     inode/symlink
>     file --brief --mime-type --dereference tstorg-sh-symlink
>     text/x-shellscript

I added the extra argument as you suggested. See the new version of the
patch. Though my man tells me that --dereference is the default. Not on
your system apparently.

>> +                                     (executable-find "file")
>> +                                     (shell-quote-argument (convert-standard-filename file))))
>> +                          (mailcap-extension-to-mime (or ext ""))))
>
> Actually MIME type for shell scripts varies a lot
>
> (mailcap-extension-to-mime "sh") => "text/x-sh"
>
> run-mailcap --norun examples/org/script/tstorg.sh
> Error: no "view" mailcap rules found for type "application/x-sh"
>
> And "text/x-shellscript" as above.

This should not matter for us. As long as mailcap-mime-info returns
something meaningful, we should be good to go. Unless mailcap-mime-info
itself is buggy.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-org-open-file-Use-file-command-to-determine-mime-.patch --]
[-- Type: text/x-patch, Size: 1586 bytes --]

From 3dac7d62b7c85bde31d27ef48569df9d07ae6eec Mon Sep 17 00:00:00 2001
Message-Id: <3dac7d62b7c85bde31d27ef48569df9d07ae6eec.1653918246.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 22 May 2022 12:04:35 +0800
Subject: [PATCH v3] org-open-file: Use file command to determine mime type,
 when available

* lisp/org.el (org-open-file): Prefer file command to determine file
type instead of relying on `mailcap-extension-to-mime'.  Only fallback
to the latter when file command is not available on the system.

Fixes https://list.orgmode.org/874k1n2hpv.fsf@localhost/T/#mcc10df4ea7937360671e6dea8061153dee518807
---
 lisp/org.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/org.el b/lisp/org.el
index 95dff27ad..b9a7b4b2e 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -7981,7 +7981,12 @@ (defun org-open-file (path &optional in-emacs line search)
     (when (eq cmd 'mailcap)
       (require 'mailcap)
       (mailcap-parse-mailcaps)
-      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
+      (let* ((mime-type (if (executable-find "file")
+                            (shell-command-to-string
+                             (format "%s --brief --mime-type --dereference %s"
+                                     (executable-find "file")
+                                     (shell-quote-argument (convert-standard-filename file))))
+                          (mailcap-extension-to-mime (or ext ""))))
 	     (command (mailcap-mime-info mime-type)))
 	(if (stringp command)
 	    (setq cmd command)
-- 
2.35.1

Re: [PATCH v3] Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 30/05/2022 21:00, Ihor Radchenko wrote:
> -      (let* ((mime-type (mailcap-extension-to-mime (or ext "")))
> +      (let* ((mime-type (if (executable-find "file")
> +                            (shell-command-to-string
> +                             (format "%s --brief --mime-type --dereference %s"
> +                                     (executable-find "file")

     (shell-quote-argument (executable-find "file"))

For the case of a directory with spaces or other characters interpreted 
by shell in the PATH environment.

Unsure if "file" command variants exist that do not support 
--dereference. Another pitfalls with `shell-command-to-string' is that 
it provides no way to handle errors. I would consider 2>/dev/null to not 
consider error message as MIME type and `mailcap-extension-to-mime' as 
fallback when "file" command fails.

(let* ((file-executable (executable-find "file"))
        (mime-type-file
         (and file-executable
          (shell-command-to-string
           (format "%s --brief --mime-type --dereference %s 2>/dev/null"
                  ; ...
         )))
        (mime-type (if (org-string-nw-p mime-type-file)
                       mime-type-file
                     (mailcap-extension-to-mime (or ext "")))

> +                                     (shell-quote-argument (convert-standard-filename file))))
> +                          (mailcap-extension-to-mime (or ext ""))))

> Max Nikulin writes:
> 
>> 3. With your patch and the following additional entry in ~/.mailcap
>>      (notice "text" instead of "application" and just "emacs")
>>          text/x-shellscript; emacs %s; test=test -n "$DISPLAY"
>>      new Emacs session is started. It is a problem but partially
>>      it is caused by incorrect mailcap configuration.
>>      Unlike "text/plain" that would be handled by view-mode
>>      unless `mailcap-mime-data' were erased in Emacs-27,
>>      "text/x-shellscript" is handled by less on my main system
>>      due to mailcap while I would expect same Emacs session.
> 
> I am confused here. org-file-apps-gnu says that we rely on mailcap:
> 
> ((remote . emacs)
>   (system . mailcap)
>   (t . mailcap))
> 
> So, is (3) following what you would expect from mailcap (regardless
> whether it is incorrectly configured or not)? Wrong configuration of
> mailcap is none of Org business - we need not to be "smart" and fix user
> "errors". They may be deliberate.

Ihor, I am afraid that your patch may break some subtle equilibrium 
existing despite discrepancy in MIME type names for shell script. Worst 
scenario: without the patch shell scripts are opened in the same Emacs 
session, with it attempt to open a script silently fails due to "less" 
handler requiring a terminal.

I admit that your patch may improve handling of e.g. images, however it 
is more rare case when an image file has no suffix, while it is quite 
common for various scripts (shell, python, perl, etc.)

I have not tried emacs-28 yet or some other full fledged desktop 
environment (in VM).

There is no official MIME type for shell scripts
http://www.iana.org/assignments/media-types/media-types.xhtml
so definitions "text" or "application", "x-sh" or "x-shellscript" vary. 
E.g. https://wiki.debian.org/ShellScript :
"The MIME type of shell scripts is nowadays text/x-shellscript but other 
systems may still use application/x-shellscript."

Emacs uses application/x-sh for the .sh suffix. There is no association 
for this type in `mailcap-mime-data'. The "file" command reports 
"text/x-shellscript". I have neither "application/x-sh" nor 
"text/x-shellscript" in /etc/mailcap (Ubuntu-20.04), there are only 
several entries for "application/x-shellscript". That is why your patch 
should not make handling of shell scripts worse... till mailcap and 
"file" will start to use the same type.

>> In some cases
>> it may be handy to launch remote viewer from a host accessed through
>> ssh, but let's leave it aside.
> 
> This is exactly why you can always customize org-file-apps.

I am confused by this phrase. `org-file-apps' is ignored for remote 
files, otherwise more care would be required for executing "file".

> I added the extra argument as you suggested. See the new version of the
> patch. Though my man tells me that --dereference is the default. Not on
> your system apparently.

I have no idea. Quick search have not provided a changelog, but I have 
noticed https://bugs.astron.com/view.php?id=29

>> (mailcap-extension-to-mime "sh") => "text/x-sh"
>>
>> run-mailcap --norun examples/org/script/tstorg.sh
>> Error: no "view" mailcap rules found for type "application/x-sh"
>>
>> And "text/x-shellscript" as above.
> 
> This should not matter for us. As long as mailcap-mime-info returns
> something meaningful, we should be good to go. Unless mailcap-mime-info
> itself is buggy.

Following a link to a shell script from an org file (without 
customization) I expect that it is opened in the same Emacs session. 
Maybe my expectation is wrong and system-wide handler (gedit, kate, 
etc.) should be launched. It seems current status quo (at least for some 
linux distributions) is consistent with my expectations, but only due to 
inconsistency in naming of the MIME type.

Re: [PATCH v3] Re: Bug in 9.5.3 org--file-default-apps

[Max Nikulin]

On 30/05/2022 22:38, Max Nikulin wrote:
> (let* ((file-executable (executable-find "file"))
>         (mime-type-file
>          (and file-executable
>           (shell-command-to-string
>            (format "%s --brief --mime-type --dereference %s 2>/dev/null"
>                   ; ...
>          )))
>         (mime-type (if (org-string-nw-p mime-type-file)
>                        mime-type-file
>                      (mailcap-extension-to-mime (or ext "")))

There is an implementation of "file" that does not support --mime-type 
in particular and long options at all in general: 
https://landley.net/toybox/help.html#file toybox that is installed on 
Android. Windows port http://gnuwin32.sourceforge.net/packages/file.htm 
looks dead, but anyway for cmd.exe "2> nul" should be used instead of 
">/dev/null". It may be safer to check
    (string-match-p "\\`[-a-zA-Z0-9+_.]+/[-a-zA-Z0-9+_.]+\\'"
                    mime-file-type)
instead of error redirection. I still think that 
`mailcap-extension-to-mime' should be fallback, not just alternative to 
trying file command.

[DISCUSSION, default settings] Using mailcap as default handler for opening file links (was: [PATCH v3] Re: Bug in 9.5.3 org--file-default-apps)

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> I am confused here. org-file-apps-gnu says that we rely on mailcap:
>> 
>> ((remote . emacs)
>>   (system . mailcap)
>>   (t . mailcap))
>> 
>> So, is (3) following what you would expect from mailcap (regardless
>> whether it is incorrectly configured or not)? Wrong configuration of
>> mailcap is none of Org business - we need not to be "smart" and fix user
>> "errors". They may be deliberate.
>
> Ihor, I am afraid that your patch may break some subtle equilibrium 
> existing despite discrepancy in MIME type names for shell script. Worst 
> scenario: without the patch shell scripts are opened in the same Emacs 
> session, with it attempt to open a script silently fails due to "less" 
> handler requiring a terminal.
>
> I admit that your patch may improve handling of e.g. images, however it 
> is more rare case when an image file has no suffix, while it is quite 
> common for various scripts (shell, python, perl, etc.)

As Craig found out, Org 9.5.2 didn't try to use mailcap at all (because
of typo). So, the equilibrium you are talking about only exists since
Org 9.5.3 (April, 2022).

Before Org 9.5.3, the default behavior on Linux (not Windows or Mac) was
opening links in Emacs

or, more precisely
(funcall (cdr (assq 'file org-link-frame-setup)) file)

Since Org 9.5.3, the default behavior on Linux is using mailcap with all
the side-effects we are observing.

It appears that using mailcap is giving us more trouble than benefits.
I am not sure about the situation on Windows and Mac though.

Should we change the default file handlers to Emacs globally (unless
user customizes otherwise)? Should we continue efforts to work around
mailcap issues? Maybe there is yet another alternative generic way to
open files?

Best,
Ihor

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links (was: [PATCH v3] Re: Bug in 9.5.3 org--file-default-apps)

[Greg Minshall]

hi, Ihor,

> It appears that using mailcap is giving us more trouble than benefits.
> I am not sure about the situation on Windows and Mac though.

> Should we change the default file handlers to Emacs globally (unless
> user customizes otherwise)? Should we continue efforts to work around
> mailcap issues? Maybe there is yet another alternative generic way to
> open files?

do you, or anyone else who might chime in, have a sense to what extent
"mainline emacs development" is committed to pushing the mailcap
solution?  and, to what extent are they running up against, and trying
to solve, problems similar to the ones you mention?

if "commitment + yes, trying", it might make sense to have, at least,
the *goal* of going mailcap.

my 2c.

cheers, Greg

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

On 04/06/2022 20:42, Ihor Radchenko wrote:
> 
> It appears that using mailcap is giving us more trouble than benefits.
> I am not sure about the situation on Windows and Mac though.
> 
> Should we change the default file handlers to Emacs globally (unless
> user customizes otherwise)? Should we continue efforts to work around
> mailcap issues? Maybe there is yet another alternative generic way to
> open files?

First of all, does someone has reproducible examples when 
`org-open-file' behaves against expectations in *default* configuration?

My current impression is that even despite serious problems with wiping 
of `mailcap-mime-data' in Emacs-27, "most" files are still opened in 
Emacs due to `auto-mode-alist'. Mailcap is used more rare than I expected.

I believe, there are enough issues with mailcap implementation in Emacs, 
but do we have some alternative? There is no support of queries to 
mimeapps.list files in Emacs (XDG). Like Chrome it is possible to call 
xdg-open for any type that can not be handled internally. Maybe it 
possible to leave it in Org as is or with the patch to call "file" 
utility (after some fixes).

At least Arch and Debian with Ubuntu have packages for SEMI 
(emacs-mime), but I am unsure what it is
http://git.chise.org/elisp/semi/
http://git.chise.org/gitweb/?p=elisp/semi.git;a=tree

P.S. Some observations.

MIME is mess. On my system I have in the /etc/mime.types file
application/x-sh				sh
application/x-shellscript
text/x-sh					sh

Notice that for a file with no extension MIME type is different. The 
"file" utility reports another variant: "text/x-shellscript". To be on 
the safe side users should configure all variants...

Unsure if the latest version is the same
https://gitlab.freedesktop.org/xdg/shared-mime-info/-/raw/master/data/freedesktop.org.xml.in
   <mime-type type="application/x-shellscript">
     <comment>shell script</comment>
     <sub-class-of type="application/x-executable"/>
     <sub-class-of type="text/plain"/>
     <alias type="text/x-sh"/>
     <generic-icon name="text-x-script"/>
     <magic>
       <!-- ... -->
     </magic>
     <glob pattern="*.sh"/>
   </mime-type>

Emacs behavior for (mailcap-mime-info "text/plain") when no mailcap 
files are present in the system
- 26: view-mode
- 27: nil since initial value of `mailcap-mime-data' erased
- 28: fundamental-mode because when `mailcap-mime-data' is copied to 
`mailcap--computed-mime-data' order of fundamental-mode and view-mode is 
reverted.

By default user's ~/.mailcap has higher priority than initial 
`mailcap-mime-data' configuration in Emacs-28, but it was not so in 
Emacs-26.

Though it should not matter due to `auto-mode-alist'.

P.P.S. I had a hope that recent Fedora-36 release has Emacs-28 packaged, 
so it would be possible to test live image in qemu to quickly check 
behavior in full-fledged desktop environment, but version 27 is really 
packaged there.

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

> On 04/06/2022 20:42, Ihor Radchenko wrote:
>>
>> Should we change the default file handlers to Emacs globally (unless
>> user customizes otherwise)? Should we continue efforts to work around
>> mailcap issues? Maybe there is yet another alternative generic way to
>> open files?

Ihor, back to your patch introducing invocation of the "file" utility. 
To get consistent behavior it should be done much earlier, when it 
becomes known that the file is not a remote one, not only as a part of 
mailcap code path (perhaps as a fallback for a file with no extension). 
Unfortunately it requires more work since Emacs mostly uses file 
suffixes, not MIME types, so the determined type should be converted to 
file extension to query e.g. auto-mode-alist.

Ideally both MIME type and file suffix should be taken into account 
since e.g. .xpi mozilla extensions are recognized as regular zip archives.

Ihor Radchenko. [PATCH v3] Re: Bug in 9.5.3 org--file-default-apps. Mon, 
30 May 2022 22:00:27 +0800. 
https://list.orgmode.org/8735gr15ok.fsf@localhost
> 
> ((remote . emacs)
>  (system . mailcap)
>  (t . mailcap))
> 
> So, is (3) following what you would expect from mailcap (regardless
> whether it is incorrectly configured or not)? Wrong configuration of
> mailcap is none of Org business - we need not to be "smart" and fix user
> "errors". They may be deliberate.

I was trying to say that mailcap.el has some predefined associations 
that are intended to handle some file types by Emacs instead of external 
handlers configured in mailcap files, unless `mailcap-user-mime-data' or 
`mailcap-prefer-mailcap-viewers' prescribes another behavior.

> Max Nikulin writes:
>> I read implementation of `org-open-file' once more and now I see that 
>> currently remote files can not be processed by mailcap code path even 
>> with custom `org-file-apps', so thank you for explanation. In some cases 
>> it may be handy to launch remote viewer from a host accessed through 
>> ssh, but let's leave it aside.
> 
> This is exactly why you can always customize org-file-apps. 

Have I missed something of `org-file-apps' are ignored for remote files?

P.S. Earlier it was discussed whether run-mailcap inspects file content 
or relies solely on file suffix. A side note: originally mailcap 
processes parts of mail messages and MIME type is specified by the 
Cotntent-Type header. So handling of standalone files with no original 
MIME type is not a really native way to use mailcap. On the other hand 
sometimes there is no better way than reusing existing database.

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Bhavin Gandhi]

Hello Max,

On Mon, 6 Jun 2022 at 19:09, Max Nikulin <manikulin@gmail.com> wrote:
> P.P.S. I had a hope that recent Fedora-36 release has Emacs-28 packaged,
> so it would be possible to test live image in qemu to quickly check
> behavior in full-fledged desktop environment, but version 27 is really
> packaged there.

I think we will have Emacs 28 in Fedora 37 (that's because major changes
are not introduced in beta/released versions)[1].

I can test the Emacs 28 + Fedora 36 combination, if you give me steps to
perform. Sorry, I don't have full context on how MIME type stuff applies
to Org mode etc., so I can try the steps and report back the behavior.

If you want to try it, run these commands when you boot into Fedora
36. This Copr is maintained by me, and is built from the official RPM
spec file + modifications for Emacs 28[2].

sudo dnf copr enable bhavin192/emacs-pretest
sudo dnf install emacs

[1] https://gitlab.com/bhavin192/emacs-pretest-rpm/-/issues/3
[2] https://src.fedoraproject.org/rpms/emacs/pull-request/12#comment-105727

-- 
Regards,
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

On 07/06/2022 00:47, Bhavin Gandhi wrote:
> On Mon, 6 Jun 2022 at 19:09, Max Nikulin <manikulin@gmail.com> wrote:
>> P.P.S. I had a hope that recent Fedora-36 release has Emacs-28 packaged,
>> so it would be possible to test live image in qemu to quickly check
>> behavior in full-fledged desktop environment, but version 27 is really
>> packaged there.
> 
> I think we will have Emacs 28 in Fedora 37 (that's because major changes
> are not introduced in beta/released versions)[1].

I have noticed your activity in search results but finally I decided to 
revive existing minimal LXC container with Arch.

> I can test the Emacs 28 + Fedora 36 combination, if you give me steps to
> perform. Sorry, I don't have full context on how MIME type stuff applies
> to Org mode etc., so I can try the steps and report back the behavior.

Thank you for offering a hand. Actually there are too many variables to 
ask for particular actions. I was trying various functions and stepping 
through them in debugger having side by side windows of Emacs-26 from 
Ubuntu-20.04 and Emacs-28 from ArchLinux (Emacs-27 has a bug in 
mailcap.el rather serious in this case). Content of /etc/mime.types 
(e.g. shell script definitions) and /etc/mailcap (presence of */* less 
%s) varies in Linux distributions, `mailcap-mime-apps' is not the only 
map of handlers withing Emacs, `org-open-file' uses `auto-mode-alist' 
and `org-file-apps'...

It is even more off-topic here, but since packaging Emacs for next 
Fedora has been mentioned, I have another question. To test my patches 
for Org I have LXC containers with various versions of Ubuntu and 
installing Emacs there pulls reasonable amount of X11 libraries. Several 
months ago I tried to install Emacs in a Fedora-35 container. I gave up 
because of enormous amount of requirements including wayland, libraries 
related to hardware acceleration and maybe even drivers. Often I do not 
mount /dev/dri inside containers, so a lot of packages were completely 
useless for me. Amount of required libraries in Arch is not so huge.

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 04/06/2022 20:42, Ihor Radchenko wrote:
>> 
>> It appears that using mailcap is giving us more trouble than benefits.
>> I am not sure about the situation on Windows and Mac though.
>> 
>> Should we change the default file handlers to Emacs globally (unless
>> user customizes otherwise)? Should we continue efforts to work around
>> mailcap issues? Maybe there is yet another alternative generic way to
>> open files?
>
> ...
> I believe, there are enough issues with mailcap implementation in Emacs, 
> but do we have some alternative? There is no support of queries to 
> mimeapps.list files in Emacs (XDG). Like Chrome it is possible to call 
> xdg-open for any type that can not be handled internally. Maybe it 
> possible to leave it in Org as is or with the patch to call "file" 
> utility (after some fixes).

I have been digging more on this issue recently, and I have found that
Carsten once attempted to tweak this default to use xdg-mime:

https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=15ae89b39

Use `xdg-open' to open files where available
* lisp/org.el (org-file-apps-defaults-gnu): Use `xdg-open' to open files
  where available.

 (defconst org-file-apps-defaults-gnu
-  '((remote . emacs)
-    (system . mailcap)
-    (t . mailcap))
+  (append
+   '((remote . emacs))
+   (if (executable-find "xdg-open")
+       '((system . "xdg-open %s")
+	 (t . "xdg-open %s"))
+     '((system . mailcap)
+       (t . mailcap))))

However, that commit had been reverted because xdg-open did not work for
some users, "failing silently":
https://list.orgmode.org/CAN_Dec8n=yNTto6Uh0dN50fG_HbqHFQtmYOCchH7wsyJdm3_Gg@mail.gmail.com/

I believe that xdg-open issue has been fixed by you in
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5db61eb0f
org.el: Avoid xdg-open silent failure

So, it may be safe to re-apply that Carsten's commit now.

WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

On 12/02/2024 19:36, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> I believe, there are enough issues with mailcap implementation in Emacs,
>> but do we have some alternative?
[...]
> 
> I have been digging more on this issue recently, and I have found that
> Carsten once attempted to tweak this default to use xdg-mime:

Have you faced another issue with mailcap? Frankly speaking, I am unsure 
what particular issues we are going to address (file type, Emacs and Org 
versions, OS).

Did you intentionally mentioned "xdg-mime"? While xdg-open tries to find 
suitable .desktop file to open a file, xdg-mime is more close to the 
file(1) tool, but it consults signatures from shared-mime-info and 
site&user config files. In addition, unlike file(1), it uses file 
extensions to guess media type.

An advantage of mailcap.el is that it should allow users have 
Emacs-specific Emacs-wide configuration for viewers. In principle it 
should be more reliable when invoking emacs through xdg-open when Emacs 
is configured as a handler for some media types.

Is it necessary to modify Org? Maybe an alternative is to add "xdg-open 
%s" for */* to `mailcap-user-mime-data'.

> I believe that xdg-open issue has been fixed by you in
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5db61eb0f
> org.el: Avoid xdg-open silent failure

I am still not really comfortable due to the strategy to start external 
processes diverged from methods used in browse-url.el.

I have not had a look into xdg.el yet.

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> I have been digging more on this issue recently, and I have found that
>> Carsten once attempted to tweak this default to use xdg-mime:
>
> Have you faced another issue with mailcap? Frankly speaking, I am unsure 
> what particular issues we are going to address (file type, Emacs and Org 
> versions, OS).

I am mostly going to address user confusion about mailcap - (1) a number
of people are using DMs that rely on FreeDesktop and xdg-open; they get
surprised when .mailcap is used - I've seen a number of questions in
IRC/Matrix and on reddit related to how Org mode opens file links; (2)
Emacs' mailcap does not implement the specification fully - only a
subset of mailcap format is obeyed.

So, I do not feel that Emacs' implementation of mailcap support is a
good default for most users.

> Did you intentionally mentioned "xdg-mime"?

No. I meant xdg-open.

> ... While xdg-open tries to find 
> suitable .desktop file to open a file, xdg-mime is more close to the 
> file(1) tool, but it consults signatures from shared-mime-info and 
> site&user config files. In addition, unlike file(1), it uses file 
> extensions to guess media type.

That might be a good alternative to file indeed; although file is
probably more widely installed.

> An advantage of mailcap.el is that it should allow users have 
> Emacs-specific Emacs-wide configuration for viewers.

Yes, but mailcap is not documented in Emacs manual.
FYI, I first heard that .mailcap files are a thing when I encountered
org-file-apps.

> ... In principle it 
> should be more reliable when invoking emacs through xdg-open when Emacs 
> is configured as a handler for some media types.

I doubt so. There are problems with mailcap.el and there might be
problems with Emacs media type configuration. I prefer xdg.

> Is it necessary to modify Org? Maybe an alternative is to add "xdg-open 
> %s" for */* to `mailcap-user-mime-data'.

Do you mean changing the default value of `mailcap-mime-data'?

>> I believe that xdg-open issue has been fixed by you in
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5db61eb0f
>> org.el: Avoid xdg-open silent failure
>
> I am still not really comfortable due to the strategy to start external 
> processes diverged from methods used in browse-url.el.

Is it related to the problem at hand?

> I have not had a look into xdg.el yet.

AFAIK, xdg.el does not support xdg-open functionality. We cannot use it
here.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

On 13/02/2024 18:27, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
> I am mostly going to address user confusion about mailcap

XDG configuration and so xdg-open behavior is often confusing to users 
as well, especially in the cases of KDE and no DE. In GNOME it is 
alleviated by a step with the application chooser if default application 
has not been set yet.

> (1) a number
> of people are using DMs that rely on FreeDesktop and xdg-open;

DM is display manager. Even DE (desktop environment) is not necessary, 
some toolkits and applications have their own implementations of XDG specs.

> (2) Emacs' mailcap does not implement the specification fully - only a
> subset of mailcap format is obeyed.

Agree. On the other hand, mailcap.el is aware that Emacs can handle some 
files internally. This feature should be preserved in some way. In 
addition, mailcap has to be used if neither DISPLAY nor WAYLAND_DISPLAY 
is set for current terminal.

>> xdg-mime is more close to the file(1) tool
> 
> That might be a good alternative to file indeed; although file is
> probably more widely installed.

xdg-mime and xdg-open belong to the same package. In the case of no DE 
xdg-mime calls file(1) (if mimetype(1) is not installed).

> Yes, but mailcap is not documented in Emacs manual.

(info "emacs-mailcap") However it is not as useful as it should be.

>> Is it necessary to modify Org? Maybe an alternative is to add "xdg-open
>> %s" for */* to `mailcap-user-mime-data'.
> 
> Do you mean changing the default value of `mailcap-mime-data'?

I am unsure what is a proper way to do it. I am not confident enough 
with the code that selects handlers for files.

>>> I believe that xdg-open issue has been fixed by you in
>>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5db61eb0f
>>> org.el: Avoid xdg-open silent failure
>>
>> I am still not really comfortable due to the strategy to start external
>> processes diverged from methods used in browse-url.el.
> 
> Is it related to the problem at hand?

- Eli had objections, but did not provide any details.
- Error handling code was dropped for the sake of Emacs-25.
- In the case of no DE, quit from emacs means terminating of started 
applications.
- I hope a bug I faced is exotic enough to never happen in real life.

Unfortunately another approach is a kind of trade-off, not an 
unambiguous advantage.

> AFAIK, xdg.el does not support xdg-open functionality. We cannot use it
> here.

`xdg-mime-apps' looks like a building block. However it is not enough 
and perhaps has enough bugs.

Ideally Emacs should provide API that uses either xdg.el or mailcap.el 
as backend depending on runtime and user configuration. Org is not the 
only package that needs it.

P.S.
Ihor, perhaps your clock is several minutes ahead.

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

On 13/02/2024 22:44, Max Nikulin wrote:
> (info "emacs-mailcap")

(info "emacs-mime")

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 13/02/2024 18:27, Ihor Radchenko wrote:
>> Max Nikulin writes:
>> 
>> I am mostly going to address user confusion about mailcap
>
> XDG configuration and so xdg-open behavior is often confusing to users 
> as well, especially in the cases of KDE and no DE. In GNOME it is 
> alleviated by a step with the application chooser if default application 
> has not been set yet.

I'd say that it is less confusing.
Of course, if you know anything even less confusing than xdg-open,
please share.

>> (2) Emacs' mailcap does not implement the specification fully - only a
>> subset of mailcap format is obeyed.
>
> Agree. On the other hand, mailcap.el is aware that Emacs can handle some 
> files internally. This feature should be preserved in some way. In 
> addition, mailcap has to be used if neither DISPLAY nor WAYLAND_DISPLAY 
> is set for current terminal.

May you please explain more about this?
What happens if a mailcap entry, say, declares "less" as a mimetype
handler and Emacs is running in terminal?

Also, I tried to run xdg-open from terminal (not emulator - C-M-<F1> on
Linux) and it runs w3m for .png image vs. feh when running from terminal
emulator on X.

>>> xdg-mime is more close to the file(1) tool
>> 
>> That might be a good alternative to file indeed; although file is
>> probably more widely installed.
>
> xdg-mime and xdg-open belong to the same package. In the case of no DE 
> xdg-mime calls file(1) (if mimetype(1) is not installed).

So, we can try xdg-mime and fall back to file.

>> Yes, but mailcap is not documented in Emacs manual.
>
> (info "emacs-mailcap") However it is not as useful as it should be.

It is not expected to be read by Emacs users:

       This manual is directed at users who want to modify the behaviour of
    the MIME encoding/decoding process or want a more detailed picture of
    how the Emacs MIME library works, and people who want to write functions
    and commands that manipulate MIME elements.

So, I'd say that ordinary Emacs user probably has no idea that Emacs
consults .mailcap to open /files/. Maybe some users (who are already
familiar with .mailcap) expect .mailcap to be consulted when opening
email attachments. But I am not sure - even reading
https://man.archlinux.org/man/mailcap.5.en I cannot easily see why this
would be used to open files; not to render them inline.

>>> Is it necessary to modify Org? Maybe an alternative is to add "xdg-open
>>> %s" for */* to `mailcap-user-mime-data'.
>> 
>> Do you mean changing the default value of `mailcap-mime-data'?
>
> I am unsure what is a proper way to do it. I am not confident enough 
> with the code that selects handlers for files.

I feel that we are abusing the scope of mailcap when opening file links.

>>> I am still not really comfortable due to the strategy to start external
>>> processes diverged from methods used in browse-url.el.
>> 
>> Is it related to the problem at hand?
>
> - Eli had objections, but did not provide any details.
> - Error handling code was dropped for the sake of Emacs-25.
> - In the case of no DE, quit from emacs means terminating of started 
> applications.
> - I hope a bug I faced is exotic enough to never happen in real life.
>
> Unfortunately another approach is a kind of trade-off, not an 
> unambiguous advantage.

Sorry, but I have no idea what you are talking about. I assume that it
is some kind of Emacs bug you reported some time ago. I am not sure
which one. And I do not see how it is related to this discussion.

>> AFAIK, xdg.el does not support xdg-open functionality. We cannot use it
>> here.
>
> `xdg-mime-apps' looks like a building block. However it is not enough 
> and perhaps has enough bugs.
>
> Ideally Emacs should provide API that uses either xdg.el or mailcap.el 
> as backend depending on runtime and user configuration. Org is not the 
> only package that needs it.

Yes. But that's a lot of work; adding support of libmagic being just the
first step. Implementing something more reasonable for Org mode only is
a lower-handing fruit.

> P.S.
> Ihor, perhaps your clock is several minutes ahead.

I know.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION, default settings] Using mailcap as default handler for opening file links

[Max Nikulin]

Ihor,

To be clear, I am not against taking into account XDG media types 
associations. I just believe that emacs settings should have higher 
priorities. Another point is that xdg-open is not intelligent enough to 
be used unconditionally.

I have the following idea. If (mailcap-mime-info mime-type) returns a 
function then it is used to open the file. If it is a string (shell 
command) then depending on some user option and DISPLAY, WAYLAND_DISPLAY 
environment variables either xdg-open or mailcap shell command is executed.

Notice the following issue:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=12972
Move `org-open-file' and associated code out of Org mode

Several issues with running external handlers are highlighted in 
comments. The changes were committed before I discovered history of code 
around xdg-open in browse-url.el. It is unclear to me what Eli had in mind.

If some XDG-related code appears in Org then likely it will be demand to 
have it in Emacs core (dired already has some kludge). However my 
experience of communication with Emacs developers is not encouraging.

A side note. On Debian there is the "open" command that may be 
configured to either xdg-open or run-mailcap through alternatives.

Some notes are inline.

On 14/02/2024 21:51, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
> May you please explain more about this?
> What happens if a mailcap entry, say, declares "less" as a mimetype
> handler and Emacs is running in terminal?

An entry for "less" likely has terminal requirement in the "test" field 
and likely will be discarded by mailcap.el (I have not checked it).

> Also, I tried to run xdg-open from terminal (not emulator - C-M-<F1> on
> Linux) and it runs w3m for .png image vs. feh when running from terminal
> emulator on X.

xdg-open has a fallback to some hardcoded list of browsers. It is far 
from mailcap configuration. It does not use "see" and similar 
mailcap-related tools.

> So, I'd say that ordinary Emacs user probably has no idea that Emacs
> consults .mailcap to open /files/. Maybe some users (who are already
> familiar with .mailcap) expect .mailcap to be consulted when opening
> email attachments. But I am not sure - even reading
> https://man.archlinux.org/man/mailcap.5.en I cannot easily see why this
> would be used to open files; not to render them inline.

Users of mutt/pine/alpine have more chances to be familiar with mailcap.
Debian's version
https://manpages.debian.org/bookworm/mailcap/mailcap.5.en.html
is linked to more informative
https://manpages.debian.org/bookworm/mailcap/mailcap.order.5.en.html
https://manpages.debian.org/bookworm/mailcap/update-mime.8.en.html
though these tools are specific to Debian.

> I feel that we are abusing the scope of mailcap when opening file links.

 From my point of view, Org just uses facility that Emacs provides.

>> - Eli had objections, but did not provide any details.
>> - Error handling code was dropped for the sake of Emacs-25.
>> - In the case of no DE, quit from emacs means terminating of started
>> applications.
>> - I hope a bug I faced is exotic enough to never happen in real life.
> 
> Sorry, but I have no idea what you are talking about. I assume that it
> is some kind of Emacs bug you reported some time ago. I am not sure
> which one. And I do not see how it is related to this discussion.

See the link above.

>> Ideally Emacs should provide API that uses either xdg.el or mailcap.el
>> as backend depending on runtime and user configuration. Org is not the
>> only package that needs it.
> 
> Yes. But that's a lot of work; adding support of libmagic being just the
> first step. Implementing something more reasonable for Org mode only is
> a lower-handing fruit.

There are two orthogonal XDG specs:
- Lookup for a handler when media type is known
- Database of media types with file suffixes and signatures (magic 
sequences)

The former one is more important, unfortunately its implementation in 
Emacs has not nearly completed.