Adding text/org MIME type to jshttp/mime-db

Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Hello!

Based on this recent thread, the consensus seems to be that we'd like
the .org file extension to have a dedicated "text/org" MIME type:

https://yhetil.org/orgmode/6d94fff4-4d30-4121-bfd1-f267cb5b637c@gmail.com/

Until the Org syntax spec is complete and a proposal submitted to and
approved by IANA, we can negotiate with downstream MIME libraries to
override "applicaion/vnd.lotus-organizer" with "text/org" on a
case-by-case basis.  As Max Nikulin wrote in the thread linked above,

> An alternative is to negotiate with maintainers of mime.types in
> RedHat and Debian.

Support for "text/org" in <https://github.com/jshttp/mime-db> would let
the Agregore browser (<https://agregore.mauve.moe/>) render Org files.
Then hyperdrive.el users could share Org files to be viewed in Agregore.

John Kitchin opened a PR back in 2017, which was rejected:

https://github.com/jshttp/mime-db/pull/82

After I posted a comment on the thread, the maintainer responded:

> I would suggest opening a new PR.  If you could also include those
> references inline into the description.  Ideally if you could also get
> someone to describe what is implied by "defacto" that would be awesome.

I'd like to open a new PR.  Would you elaborate on "de facto mimetype"?

Thank you!

Joseph

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 02/08/2024 02:55, Joseph Turner wrote:
> Support for "text/org" in <https://github.com/jshttp/mime-db> would let
> the Agregore browser (<https://agregore.mauve.moe/>) render Org files.
> Then hyperdrive.el users could share Org files to be viewed in Agregore.
> 
> John Kitchin opened a PR back in 2017, which was rejected:
> 
> https://github.com/jshttp/mime-db/pull/82
[...]
>> I would suggest opening a new PR.
[...]
> I'd like to open a new PR.  Would you elaborate on "de facto mimetype"?

Nothing more than my opinion:

I think, if you are going to convince somebody outside of Org community 
then you should mention that the format is supported by GitHub and 
GitLab. Org files are rendered using org-ruby gem. Only a part of 
features is available, but still useful.
<https://github.com/github/markup/>

shared-mime-info used by Linux desktop applications has .org text/org 
association:
<https://gitlab.freedesktop.org/xdg/shared-mime-info/-/blob/master/data/freedesktop.org.xml.in>
(though text/x-org alias and -*- mode: org -*- magic are missed).
Thunderbird sends .org attachments as text/org without explicit 
configuration.

Till finalizing of https://orgmode.org/worg/org-syntax.html and writing 
spec (RFC) there is no point to send a request to IANA.

Emacs will continue to use text/x-org till somebody manages to convince 
developers that transition to text/org does not break backward compatibility
<https://debbugs.gnu.org/68687>
Actually Emacs will continue to use application/vnd.lotus-organizer in 
some cases since to /etc/mime.types has higher priority.

I am not familiar with Agregore, but my opinion that media type 
associations should be configurable in applications and either 
shared-mime-info or /etc/mime.types should has higher priority than any 
built-in mapping, perhaps besides some exceptions for media types 
directly related to the application.

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 02/08/2024 02:55, Joseph Turner wrote:
> Support for "text/org" in <https://github.com/jshttp/mime-db> would let
> the Agregore browser (<https://agregore.mauve.moe/>) render Org files.
> Then hyperdrive.el users could share Org files to be viewed in Agregore.
> 
> John Kitchin opened a PR back in 2017, which was rejected:
> 
> https://github.com/jshttp/mime-db/pull/82

I missed it at first, but it was *closed*, but not *rejected*.

<https://github.com/jshttp/mime-db/commit/5797174e5434a8d2978d9fdc13ec6a47ba20a02f>
2017-07-25T01:12:28Z
> Add text/x-org
> 
> closes #82

So you can use text/x-org.

> After I posted a comment on the thread, the maintainer responded:
> 
>> I would suggest opening a new PR.

I think, text/org may be added as well (without removing of text/x-org).

Perhaps a worg page may be created to clarify current state of media 
type for org files. The question is what is the appropriate directory.

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Max Nikulin <manikulin@gmail.com> writes:

> On 02/08/2024 02:55, Joseph Turner wrote:
>> Support for "text/org" in <https://github.com/jshttp/mime-db> would
>> let
>> the Agregore browser (<https://agregore.mauve.moe/>) render Org
>> files.
>> Then hyperdrive.el users could share Org files to be viewed in
>> Agregore.
>> John Kitchin opened a PR back in 2017, which was rejected:
>> https://github.com/jshttp/mime-db/pull/82
>
> I missed it at first, but it was *closed*, but not *rejected*.
>
> <https://github.com/jshttp/mime-db/commit/5797174e5434a8d2978d9fdc13ec6a47ba20a02f>
> 2017-07-25T01:12:28Z
>> Add text/x-org
>> closes #82

Thank you for catching my oversight!!

I dug a little deeper and discovered that the popular NPM packages that
make use of jshttp/mime-db rank the source of each MIME type and discard
duplicates which have lower "preference".  For example, the
jshttp/mime-types utility does this:

  // source preference (least -> most)
  var preference = ['nginx', 'apache', undefined, 'iana']

and then discards duplicate mappings which are of lower preference.

https://github.com/jshttp/mime-types/blob/eca390b71004a70f4551197638c077cdba8cf954/index.js#L155

I think https://github.com/broofa/mime does something similar.

The result is that the two popular NPM utilities for getting a MIME type
from a file extension (see <https://www.npmjs.com/search?q=mime>) ignore
the ".org" -> "text/x-org" mapping in favor of ".org" ->
"application/vnd.lotus-organizer".

> So you can use text/x-org.
>
>> After I posted a comment on the thread, the maintainer responded:
>> 
>>> I would suggest opening a new PR.
>
> I think, text/org may be added as well (without removing of
> text/x-org).

Shall we consider this this after we find a way for these low-preference
mappings to be useful?

> Perhaps a worg page may be created to clarify current state of media
> type for org files. The question is what is the appropriate directory.

I agree that it would be good to track this information in one place so
we don't repeat ourselves ;)

Thanks,

Joseph

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Joseph Turner <joseph@ushin.org> writes:

> I think https://github.com/broofa/mime does something similar.

Actually, broofa/mime relies on another package for ranking:

https://github.com/broofa/mime-score

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 04/08/2024 00:37, Joseph Turner wrote:
> jshttp/mime-types utility does this:
> 
>    // source preference (least -> most)
>    var preference = ['nginx', 'apache', undefined, 'iana']
> 
> and then discards duplicate mappings which are of lower preference.

I can not say that I have consistent vision which way media type 
database should be designed. It is rather close to off-topic on this 
list (though `org-file-types' is a kind of override map, built-in 
mailcap.el is far from being perfect, and Emacs ignores shared-mime-info 
completely).

- Both fallback and override entries should be implemented.
- It seems there are differences which way custom entries should be 
handled during compiling database and when an application uses third 
party mappings. The reason is that version of external database may be 
older or newer than version of an application. Some conflicts may cause 
fatal error during compiling of database while applications have to be 
more intelligent in respect to conflict resolution.
- Priorities of media type to suffix and suffix to media type may be 
different.
- My impression is that multiple types may have the same suffix, so 
there should be "get all media types for given suffix" method.
- It should be possible to specify priorities within the same source.

I am a bit surprised that jshttp/mime-types developers use added entries 
as fallback, not as overrides for IANA, but it is their design decision. 
Since it is a DB, not DB user, they can just drop conflicting entries. 
Perhaps they still want to preserve ambiguous mapping of multiple media 
types to the same suffix (otherwise text/x-org should be dropped due to 
a conflict).

> Max Nikulin writes:
>> I think, text/org may be added as well (without removing of
>> text/x-org).
> 
> Shall we consider this this after we find a way for these low-preference
> mappings to be useful?

I still believe that the following behavior is preferable

- applicaion/vnd.lotus-organizer -> .org
- text/x-org -> .org
- text/org -> .org
- .org -> text/org (preferably without relying on x- heuristics)

Unless a user explicitly configured local mapping .org to 
applicaion/vnd.lotus-organizer

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Max Nikulin <manikulin@gmail.com> writes:

> On 04/08/2024 00:37, Joseph Turner wrote:
>> jshttp/mime-types utility does this:
>>    // source preference (least -> most)
>>    var preference = ['nginx', 'apache', undefined, 'iana']
>> and then discards duplicate mappings which are of lower preference.
>
> I can not say that I have consistent vision which way media type
> database should be designed. It is rather close to off-topic on this
> list (though `org-file-types' is a kind of override map, built-in
> mailcap.el is far from being perfect, and Emacs ignores
> shared-mime-info completely).
>
> - Both fallback and override entries should be implemented.
> - It seems there are differences which way custom entries should be
>   handled during compiling database and when an application uses third
>   party mappings. The reason is that version of external database may
>   be older or newer than version of an application. Some conflicts may
>   cause fatal error during compiling of database while applications
>   have to be more intelligent in respect to conflict resolution.
> - Priorities of media type to suffix and suffix to media type may be
>   different.
> - My impression is that multiple types may have the same suffix, so
>   there should be "get all media types for given suffix" method.
> - It should be possible to specify priorities within the same source.
>
> I am a bit surprised that jshttp/mime-types developers use added
> entries as fallback, not as overrides for IANA, but it is their design
> decision. Since it is a DB, not DB user, they can just drop
> conflicting entries. Perhaps they still want to preserve ambiguous
> mapping of multiple media types to the same suffix (otherwise
> text/x-org should be dropped due to a conflict).

Yes, I am also surprised that jshttp/mime-types does not offer a way to
customize MIME type source preference.

However, on the topic of Org mode support in Agregore, I am not sure
that the ability to specify a different preference or even a "get all
media types for given suffix" function would help.  In this case, we
want the browser to treat .org files as text files and attempt to render
them.  Currently, .org files are treated as non-text files and the
browser attempts to download them when the user opens one.  This means
that our code to render Org files doesn't even get a chance to run.

For now, perhaps we can override the MIME type map in the extension:

https://github.com/AgregoreWeb/extension-agregore-renderer/issues/8#issuecomment-2267958828

>> Max Nikulin writes:
>>> I think, text/org may be added as well (without removing of
>>> text/x-org).
>> Shall we consider this this after we find a way for these
>> low-preference
>> mappings to be useful?
>
> I still believe that the following behavior is preferable
>
> - applicaion/vnd.lotus-organizer -> .org
> - text/x-org -> .org
> - text/org -> .org
> - .org -> text/org (preferably without relying on x- heuristics)
>
> Unless a user explicitly configured local mapping .org to
> applicaion/vnd.lotus-organizer

I agree with your preference!

Thanks,

Joseph

Re: Adding text/org MIME type to jshttp/mime-db

[Max Nikulin]

On 05/08/2024 10:19, Joseph Turner wrote:
> Max Nikulin writes:
>> - My impression is that multiple types may have the same suffix, so
>>    there should be "get all media types for given suffix" method.
>> - It should be possible to specify priorities within the same source.
[...]
> However, on the topic of Org mode support in Agregore, I am not sure
> that the ability to specify a different preference or even a "get all
> media types for given suffix" function would help.

My idea is that this method pushes developers toward more flexible 
design in respect to specifying priorities of records. With its presence 
it is more probable that users of the DB will be prepared to cases when 
2 step mappings media type -> suffix -> media type and suffix -> media 
type -> suffix give values that are different from the original ones.

> For now, perhaps we can override the MIME type map in the extension:
> 
> https://github.com/AgregoreWeb/extension-agregore-renderer/issues/8#issuecomment-2267958828

I have added a comment. The issue may be Content-Type header sent by 
server (or other side of p2p connection).

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Max Nikulin <manikulin@gmail.com> writes:

> On 05/08/2024 10:19, Joseph Turner wrote:
>> Max Nikulin writes:
>>> - My impression is that multiple types may have the same suffix, so
>>>    there should be "get all media types for given suffix" method.
>>> - It should be possible to specify priorities within the same source.
> [...]
>> However, on the topic of Org mode support in Agregore, I am not sure
>> that the ability to specify a different preference or even a "get all
>> media types for given suffix" function would help.
>
> My idea is that this method pushes developers toward more flexible
> design in respect to specifying priorities of records. With its
> presence it is more probable that users of the DB will be prepared to
> cases when 2 step mappings media type -> suffix -> media type and
> suffix -> media type -> suffix give values that are different from the
> original ones.

I think a more flexible design could help in this case.

>> For now, perhaps we can override the MIME type map in the extension:
>> https://github.com/AgregoreWeb/extension-agregore-renderer/issues/8#issuecomment-2267958828
>
> I have added a comment. The issue may be Content-Type header sent by
> server (or other side of p2p connection).

Thank you.  I've responded on that thread.

Re: Adding text/org MIME type to jshttp/mime-db

[Ihor Radchenko]

Joseph Turner <joseph@ushin.org> writes:

>> I think, text/org may be added as well (without removing of
>> text/x-org).
>
> Shall we consider this this after we find a way for these low-preference
> mappings to be useful?

If you mean "negotiate... to override "applicaion/vnd.lotus-organizer"
with "text/org"", I have no objections.

>> Perhaps a worg page may be created to clarify current state of media
>> type for org files. The question is what is the appropriate directory.
>
> I agree that it would be good to track this information in one place so
> we don't repeat ourselves ;)

We can put such a page alongside /org-syntax.org, maybe even linked from
there. Feel free to create such a page.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Joseph Turner <joseph@ushin.org> writes:
>
>>> I think, text/org may be added as well (without removing of
>>> text/x-org).
>>
>> Shall we consider this this after we find a way for these low-preference
>> mappings to be useful?
>
> If you mean "negotiate... to override "applicaion/vnd.lotus-organizer"
> with "text/org"", I have no objections.
>
>>> Perhaps a worg page may be created to clarify current state of media
>>> type for org files. The question is what is the appropriate directory.
>>
>> I agree that it would be good to track this information in one place so
>> we don't repeat ourselves ;)
>
> We can put such a page alongside /org-syntax.org, maybe even linked from
> there. Feel free to create such a page.

Would the following patch be appropriate?

Thanks!

Joseph


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-media-type.org-New-file-Describe-state-of-NodeJS.patch --]
[-- Type: text/x-diff, Size: 2218 bytes --]

From c898ba723faeb67929f52728c67e2b8858f99738 Mon Sep 17 00:00:00 2001
From: Joseph Turner <joseph@breatheoutbreathe.in>
Date: Fri, 9 Aug 2024 05:55:52 -0700
Subject: [PATCH] org-media-type.org: New file; Describe state of NodeJS MIME
 database

---
 org-media-type.org | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100644 org-media-type.org

diff --git a/org-media-type.org b/org-media-type.org
new file mode 100644
index 00000000..f5d69b7c
--- /dev/null
+++ b/org-media-type.org
@@ -0,0 +1,37 @@
+#+title: Org Media Type
+#+author: Joseph Turner
+#+options: toc:t ':t author:nil
+#+language: en
+#+category: worg
+#+bind: sentence-end-double-space t
+#+html_link_up:    index.html
+#+html_link_home:  https://orgmode.org/worg/
+
+#+begin_comment
+This file is released by its authors and contributors under the GNU
+Free Documentation license v1.3 or later, code examples are released
+under the GNU General Public License v3 or later.
+#+end_comment
+
+* Introduction
+
+This page tracks information about the ~.org~ extension media type.
+
+Based on [[https://yhetil.org/orgmode/6d94fff4-4d30-4121-bfd1-f267cb5b637c@gmail.com/][this thread on the Org mode mailing list]], the consensus is
+that the ~.org~ file extension should map to the ~text/org~ MIME type.
+
+Until the [[file:org-syntax.org][Org syntax spec]] is complete and a proposal submitted to and
+approved by IANA, we can ask downstream MIME libraries to override
+~applicaion/vnd.lotus-organizer~ with ~text/org~.
+
+* Downstream MIME Libraries
+
+** jshttp/mime-db - NodeJS
+
+[[https://github.com/jshttp/mime-db][jshttp/mime-db]] is the de facto standard MIME database for NodeJS.
+[[https://github.com/jshttp/mime-db/commit/5797174e5434a8d2978d9fdc13ec6a47ba20a02f][This commit]] adds a mapping from ~.org~ to ~text/x-org~, but the two
+popular NodeJS utilities for getting a MIME type from a file
+extension, [[https://github.com/jshttp/mime-types][jshttp/mime-types]] and [[https://github.com/broofa/mime][broofa/mime]], return
+~applicaion/vnd.lotus-organizer~.  Both libraries return only the
+highest preference media type, and IANA-sourced media types are
+preferred over other types.
-- 
2.41.0

Re: Adding text/org MIME type to jshttp/mime-db

[Ihor Radchenko]

Joseph Turner <joseph@ushin.org> writes:

>> We can put such a page alongside /org-syntax.org, maybe even linked from
>> there. Feel free to create such a page.
>
> Would the following patch be appropriate?

LGTM!
But we also need to link this new page from other places.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

[-- Attachment #1: Type: text/plain, Size: 373 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Joseph Turner <joseph@ushin.org> writes:
>
>>> We can put such a page alongside /org-syntax.org, maybe even linked from
>>> there. Feel free to create such a page.
>>
>> Would the following patch be appropriate?
>
> LGTM!
> But we also need to link this new page from other places.

How about this patch for orgweb?

Joseph


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-index.org-Link-to-new-worg-org-media-type-page.patch --]
[-- Type: text/x-diff, Size: 1116 bytes --]

From 5284e6cf66a8e4a8c2dcc0471e5285746c56a455 Mon Sep 17 00:00:00 2001
From: Joseph Turner <joseph@breatheoutbreathe.in>
Date: Sat, 10 Aug 2024 11:45:53 -0700
Subject: [PATCH] index.org: Link to new worg org-media-type page

---
 index.org | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/index.org b/index.org
index 025b4aa..6bfa6d0 100644
--- a/index.org
+++ b/index.org
@@ -119,9 +119,10 @@ * More Org
   :CUSTOM_ID: more
   :END:
 
-The /de facto/ mimetype for Org files is =text/org=. Org files use the
-~.org~ extension. Most operating systems don't know to open ~.org~ files
-in Emacs, but they can be configured to do so without much effort.
+The /de facto/ [[https://orgmode.org/worg/org-media-type.html][mimetype for Org files]] is =text/org=. Org files use the
+~.org~ extension. Most operating systems don't know to open ~.org~
+files in Emacs, but they can be configured to do so without much
+effort.
 
 While the reference implementation of Org in Emacs lisp is by far the
 most featureful, there are many additional [[file:tools.org][tools]] that work with Org.
-- 
2.41.0

Re: Adding text/org MIME type to jshttp/mime-db

[Ihor Radchenko]

Joseph Turner <joseph@ushin.org> writes:

>> But we also need to link this new page from other places.
>
> How about this patch for orgweb?

Thanks!
Applied, onto master for orgweb and master for worg.
https://git.sr.ht/~bzg/worg/commit/91954998
https://git.sr.ht/~bzg/orgweb/commit/d9222e889ed1f5fee0d328490ed731509fd5a398

I added the TINYCHANGE cookie to your orgweb patch, as orgweb is covered
under FSF copyright requirements.

Do note that your total contribution is ~15LOC at this point. You may
consider doing FSF paperwork to make more contributions to org-mode and
orgweb. worg is free from this requirement.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Ihor Radchenko <yantar92@posteo.net> writes:

> Joseph Turner <joseph@ushin.org> writes:
>
>>> But we also need to link this new page from other places.
>>
>> How about this patch for orgweb?
>
> Thanks!
> Applied, onto master for orgweb and master for worg.
> https://git.sr.ht/~bzg/worg/commit/91954998
> https://git.sr.ht/~bzg/orgweb/commit/d9222e889ed1f5fee0d328490ed731509fd5a398
>
> I added the TINYCHANGE cookie to your orgweb patch, as orgweb is covered
> under FSF copyright requirements.
>
> Do note that your total contribution is ~15LOC at this point. You may
> consider doing FSF paperwork to make more contributions to org-mode and
> orgweb. worg is free from this requirement.

Thanks!  I completed the FSF copyright assignment paperwork in May 2023.

Joseph

Re: Adding text/org MIME type to jshttp/mime-db

[Ihor Radchenko]

Joseph Turner <joseph@ushin.org> writes:

> Thanks!  I completed the FSF copyright assignment paperwork in May 2023.

Bastien, may you please confirm with FSF records?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding text/org MIME type to jshttp/mime-db

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Joseph Turner <joseph@ushin.org> writes:
>
>> Thanks!  I completed the FSF copyright assignment paperwork in May 2023.

I updated our records.
https://git.sr.ht/~bzg/worg/commit/29c58d61

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding text/org MIME type to jshttp/mime-db

[Joseph Turner]

Ihor Radchenko <yantar92@posteo.net> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Joseph Turner <joseph@ushin.org> writes:
>>
>>> Thanks!  I completed the FSF copyright assignment paperwork in May 2023.
>
> I updated our records.
> https://git.sr.ht/~bzg/worg/commit/29c58d61

Thank you!

Joseph