[RFC] MIME attachments for comint

[RFC] MIME attachments for comint

[Augusto Stoffel]

In due time, I would like to submit the following as an ELPA package.
For now, I'm writing to see if anyone would like to test or give
feedback.

    https://github.com/astoff/comint-mime

The main purpose of the package, for me at least, is to display graphics
directly in the Python shell.  But it can be extended to other comints
and other kinds of “MIME attachments”.

For instance, in Python it can pretty-print objects that define an HTML
representation.

In the regular shell, it provides a `mimecat' command, similar to the
`imgcat' of other terminals, but mimetic.

Here are some more assorted observations:

- Although the feature seems rather desirable to have even built into
  Emacs, I can't see how a sufficiently canonical implementation would
  be possible.  This package invents a little communication protocol
  between the inferior process and Emacs, which will obviously never be
  implemented by anyone else.

- To add support for a new comint, one just needs to supply some setup
  code to teach the inferior process to print the appropriate escape
  sequences.  Moreover, one needs a little bit of ELisp stuff to send
  the setup code to the inferior process (slightly tricky as usual).

- Concerning packaging, the idea at the moment would be to just have
  this one package adding mimetics to as many comints as possible.  (For
  now, I'm even including all ELisp code in one single file.)  But this
  would mean that the package has many soft (non-mandatory)
  dependencies.  Does that sound like a bad idea?

- The package depends on Emacs 28 (it uses OSC escape sequences).  I
  guess it would have to wait until Emacs 28 is released to be on ELPA?

- It seems shr doesn't support right-aligned text in tables.  Would this
  be hard to add?  This is kind of important to display numeric tables
  sensibly.

Re: [RFC] MIME attachments for comint

[Stefan Monnier]

Sounds like a cute package, thanks.

> Here are some more assorted observations:

Anything to say about security implications/measures?


        Stefan

Re: [RFC] MIME attachments for comint

[Augusto Stoffel]

On Mon, 27 Sep 2021 at 16:37, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Sounds like a cute package, thanks.
>
>> Here are some more assorted observations:
>
> Anything to say about security implications/measures?

Good question, I didn't think about this.  I guess it's safe to feed
`create-image', `svg-image' and shr with any kind of evil data, no?

But TeX markup could in principle execute arbitrary code.  I'm relying
on Org for TeX rendering, so I'd have to check how careful it is about
that.

>
>
>         Stefan

Re: [RFC] MIME attachments for comint

[Stefan Monnier]

Augusto Stoffel [2021-09-28 18:05:35] wrote:
> On Mon, 27 Sep 2021 at 16:37, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Sounds like a cute package, thanks.
>>> Here are some more assorted observations:
>> Anything to say about security implications/measures?
> Good question, I didn't think about this.

I think such a feature needs to be quite careful and proactively
defensive about that.

> I guess it's safe to feed `create-image', `svg-image' and shr with any
> kind of evil data, no?

What could go wrong, right?
I recommend you place strong restrictions on the formats supported so as
to stay within bounds which you positively know are safe (e.g. no worse
than what happens already with SHR rendering when viewing HTML email).


        Stefan

Re: [RFC] MIME attachments for comint

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But TeX markup could in principle execute arbitrary code.

I'm surprised and worried.  Can you show how that can happen?
 
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [RFC] MIME attachments for comint

[Augusto Stoffel]

On Thu, 30 Sep 2021 at 02:02, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > But TeX markup could in principle execute arbitrary code.
>
> I'm surprised and worried.  Can you show how that can happen?

From the tex manpage:

       -shell-escape
              Enable the \write18{command} construct.  The command can be  any
              shell  command.  This construct is normally disallowed for secu‐
              rity reasons.

On luatex, this switch also controls the availability of certain Lua
functions: os.execute(), os.exec(), os.spawn(), and io.popen().

This option is off by default, or at least should be in a sane OS --
that's why I said “could in principle”.  Also, I don't know exactly how
Org mode deals with this potential security issue in its LaTeX-preview
functionality, but I would expect this has been taken into
consideration.

In any case, I disabled TeX rendering in my little package for the time
being, until I'm confident it's safe.

Re: [RFC] MIME attachments for comint

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This option is off by default, or at least should be in a sane OS --
  > that's why I said “could in principle”.

I don't think we need to worry, in Emacs, that some system might have
enabled a big security hole.

What does Lua have to do with TeX?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [RFC] MIME attachments for comint

[Augusto Stoffel]

On Sat,  2 Oct 2021 at 19:17, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > This option is off by default, or at least should be in a sane OS --
>   > that's why I said “could in principle”.
>
> I don't think we need to worry, in Emacs, that some system might have
> enabled a big security hole.
>
> What does Lua have to do with TeX?

This is one of the big (not so) recent developments in TeX: there is now
a version of it with an embedded Lua interpreter [1].  As you know,
programming in TeX itself can be quite cumbersome, so this helps a lot
to extend TeX in new directions.

[1]: http://www.luatex.org/

Re: [RFC] MIME attachments for comint

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is one of the big (not so) recent developments in TeX: there is now
  > a version of it with an embedded Lua interpreter [1].  As you know,
  > programming in TeX itself can be quite cumbersome, so this helps a lot
  > to extend TeX in new directions.

Interesting.  Good that -shell-escape covers this too.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [RFC] MIME attachments for comint

[Gregory Heytings]

>> But TeX markup could in principle execute arbitrary code.
>
> I'm surprised and worried.  Can you show how that can happen?
>

You probably know that TeX has a \write18{} command with which it is 
possible to execute shell commands.  The behavior of that command is (in 
TeX Live) controlled by the shell_escape and shell_escape_commands 
configuration variables in texmf.cnf.  Their default values are:

shell_escape = p

which means that shell commands are allowed "partially", that is, that the 
only allowed commands are those that are listed in shell_escape_commands:

shell_escape_commands = bibtex,bibtex8,extractbb,gregorio,kpsewhich,makeindex,repstopdf,r-mpost,texosquery-jre8

But this restriction is easy to circumvent:

echo ls > bibtex
chmod +x bibtex
export PATH=.:$PATH
echo '\write18{bibtex}\bye' > test.tex
pdftex test.tex

In practice, this is not really a problem, because it requires to either 
change one of the programs listed in shell_escape_commands, or to change 
the PATH environment variable.  But it's fragile nonetheless.

Re: [RFC] MIME attachments for comint

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks.  I did not know about the write18 feature.

  > which means that shell commands are allowed "partially", that is, that the 
  > only allowed commands are those that are listed in shell_escape_commands:

  > shell_escape_commands = bibtex,bibtex8,extractbb,gregorio,kpsewhich,makeindex,repstopdf,r-mpost,texosquery-jre8

  > echo ls > bibtex
  > chmod +x bibtex
  > export PATH=.:$PATH
  > echo '\write18{bibtex}\bye' > test.tex
  > pdftex test.tex

  > In practice, this is not really a problem, because it requires to either 
  > change one of the programs listed in shell_escape_commands, or to change 
  > the PATH environment variable.  But it's fragile nonetheless.

I agree.

If someone is working on a feature in Emacs that will run TeX on files
you have not studied, I think we should make very sure there is no way for
that feature to use other than those default settings, unless the user
very directly insists.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [RFC] MIME attachments for comint

[Colin Baxter]

>>>>> Augusto Stoffel <arstoffel@gmail.com> writes:

    > On Mon, 27 Sep 2021 at 16:37, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
    >> Sounds like a cute package, thanks.
    >> 
    >>> Here are some more assorted observations:
    >> 
    >> Anything to say about security implications/measures?

    > Good question, I didn't think about this.  I guess it's safe to
    > feed `create-image', `svg-image' and shr with any kind of evil
    > data, no?

    > But TeX markup could in principle execute arbitrary code.  I'm

But only if you turn on full access to \write18, surely?

Best wishes

Re: [RFC] MIME attachments for comint

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> - Although the feature seems rather desirable to have even built into
>   Emacs, I can't see how a sufficiently canonical implementation would
>   be possible.  This package invents a little communication protocol
>   between the inferior process and Emacs, which will obviously never be
>   implemented by anyone else.

I think this is really neat functionality, but as you say -- it's rather
non-standard, and it's hard to see this ever becoming a standard thing.
(For instance, there's no imgcat in Debian/bullseye.)  So I'm afraid
adding it to Emacs core wouldn't be useful to that many people.

But I could see some people being interested in this, so putting it in
GNU ELPA would be fine by me.

> - It seems shr doesn't support right-aligned text in tables.  Would this
>   be hard to add?  This is kind of important to display numeric tables
>   sensibly.

Hm...  I think I vaguely remember adding that at some point, but I may
be wrong.  Do you have a test case that doesn't render correctly?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [RFC] MIME attachments for comint

[Augusto Stoffel]

On Tue, 28 Sep 2021 at 07:36, Lars Ingebrigtsen <larsi@gnus.org> wrote:

>> - It seems shr doesn't support right-aligned text in tables.  Would this
>>   be hard to add?  This is kind of important to display numeric tables
>>   sensibly.
>
> Hm...  I think I vaguely remember adding that at some point, but I may
> be wrong.  Do you have a test case that doesn't render correctly?

The following example

    <table style="text-align: right">
      <tr style="text-align: right"><td>A</td><td>1.0</td></tr>
      <tr><td style="text-align: right">AA</td><td>10.0</td></tr>
      <tr><td>AAA</td><td>100.0</td></tr>
    </table>

renders like this:

    A     1.0
    AA    10.0
    AAA   100.0

Is there any other syntax for text alignment in HTML?

Re: [RFC] MIME attachments for comint

[Robin Tarsiger]

Augusto Stoffel wrote:
> The following example
> 
>      <table style="text-align: right">
>        <tr style="text-align: right"><td>A</td><td>1.0</td></tr>
>        <tr><td style="text-align: right">AA</td><td>10.0</td></tr>
>        <tr><td>AAA</td><td>100.0</td></tr>
>      </table>
> 
> renders like this:
> 
>      A     1.0
>      AA    10.0
>      AAA   100.0
> 
> Is there any other syntax for text alignment in HTML?

There's the pre-CSS align="right" property style. Also valign for
vertical-align.

I notice that shr doesn't have anything close to a full CSS parser
to start with, which is understandable given the complexity of such
a thing. However, note that in a full document, it's also reasonable
in an HTML context to do something like

   <style>
   table.foo td.bar { text-align: right }
   table.foo td.baz { text-align: center }
   </style>
   <!-- ... -->
   <table class="foo">
     <tr><td class="bar">A</td><td class="baz">III</td><td class="baz">XII</td></tr>
     <tr><td class="bar">B</td><td class="baz">XIV</td><td class="baz">VII</td></tr>
   </table>

where the alignments cannot be inferred from a purely local
traversal. I don't know how many people do this, but I do when
I'm outputting HTML tables with known record formats, because
it makes it much easier to restyle the types/columns as needed
(that being the purpose of CSS in the first place).

I'm curious what your likely HTML sources are going to be for this,
since if you try to subset the above at all then what you need to
support becomes very dependent on what the generators are like.

-RTT

Re: [RFC] MIME attachments for comint

[Augusto Stoffel]

On Tue, 28 Sep 2021 at 11:29, Robin Tarsiger <rtt@dasyatidae.com> wrote:

> Augusto Stoffel wrote:
>> The following example
>>      <table style="text-align: right">
>>        <tr style="text-align: right"><td>A</td><td>1.0</td></tr>
>>        <tr><td style="text-align: right">AA</td><td>10.0</td></tr>
>>        <tr><td>AAA</td><td>100.0</td></tr>
>>      </table>
>> renders like this:
>>      A     1.0
>>      AA    10.0
>>      AAA   100.0
>> Is there any other syntax for text alignment in HTML?
>
> There's the pre-CSS align="right" property style. Also valign for
> vertical-align.

This also doesn't seem to work with shr.

> I notice that shr doesn't have anything close to a full CSS parser
> to start with, which is understandable given the complexity of such
> a thing. However, note that in a full document, it's also reasonable
> in an HTML context to do something like
>
>   <style>
>   table.foo td.bar { text-align: right }
>   table.foo td.baz { text-align: center }
>   </style>
>   <!-- ... -->
>   <table class="foo">
>     <tr><td class="bar">A</td><td class="baz">III</td><td class="baz">XII</td></tr>
>     <tr><td class="bar">B</td><td class="baz">XIV</td><td class="baz">VII</td></tr>
>   </table>
>
> where the alignments cannot be inferred from a purely local
> traversal. I don't know how many people do this, but I do when
> I'm outputting HTML tables with known record formats, because
> it makes it much easier to restyle the types/columns as needed
> (that being the purpose of CSS in the first place).
>
> I'm curious what your likely HTML sources are going to be for this,
> since if you try to subset the above at all then what you need to
> support becomes very dependent on what the generators are like.

Yes, the most obvious use-case here would be a Pandas dataframe (or the
equivalent from any other statistical package).  In the Pandas case, the
HTML looks pretty much like what you posted.

But the capability to right-align is a somewhat separate question from
the notation used to chose it.  If shr could render table with right
alignment, one could at least do some ad-hoc stuff for the main cases.

> -RTT

Re: [RFC] MIME attachments for comint

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> The following example
>
>     <table style="text-align: right">
>       <tr style="text-align: right"><td>A</td><td>1.0</td></tr>
>       <tr><td style="text-align: right">AA</td><td>10.0</td></tr>
>       <tr><td>AAA</td><td>100.0</td></tr>
>     </table>
>
> renders like this:
>
>     A     1.0
>     AA    10.0
>     AAA   100.0

Right.  I must be misremembering -- I can't find any code to control
alignment in shr.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [RFC] MIME attachments for comint

[Stefan Monnier]

> In due time, I would like to submit the following as an ELPA package.
> For now, I'm writing to see if anyone would like to test or give
> feedback.

Regarding GNU ELPA.  We can add it "any time" you want, so just let me
know when you want to do that.  Also, we can add it right away, the
"Version: 0" will prevent making an actual release tarball but it will
still make it available in GNU-devel.

You'll have to change the copyright notice on the third line of the
file, of course.


        Stefan

Re: [RFC] MIME attachments for comint

[Stefan Monnier]

> Regarding GNU ELPA.  We can add it "any time" you want, so just let me

[ Hmm... not sure what I was thinking when I typed those quotes, please
  disregard them.  ]


        Stefan

Re: [RFC] MIME attachments for comint

[Augusto Stoffel]

On Sun, 17 Oct 2021 at 12:20, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> In due time, I would like to submit the following as an ELPA package.
>> For now, I'm writing to see if anyone would like to test or give
>> feedback.
>
> Regarding GNU ELPA.  We can add it "any time" you want, so just let me
> know when you want to do that.  Also, we can add it right away, the
> "Version: 0" will prevent making an actual release tarball but it will
> still make it available in GNU-devel.
>
> You'll have to change the copyright notice on the third line of the
> file, of course.
>
>
>         Stefan


All right, let' add it "right now" then :-)

I've made the necessary changes, but I guess I'll wait until Emacs 28 is
out to make an official release, since the package depends on it.

PS: The loccur package is outdated on ELPA, could you pull the newest
version?  FWIW, I didn't get a reply from the maintainer when I asked if
he wanted to have auto-sync, but I can report that he was careful to
check that I have the CA papers before merging a patch from me.

comint-mime and loccur update (was: [RFC] MIME attachments for comint)

[Stefan Monnier]

Augusto Stoffel wrote:
> All right, let' add it "right now" then :-)

Done.

> I've made the necessary changes, but I guess I'll wait until Emacs 28 is
> out to make an official release, since the package depends on it.

Not in the mood for time travel, eh?

> PS: The loccur package is outdated on ELPA, could you pull the newest
> version?  FWIW, I didn't get a reply from the maintainer when I asked if
> he wanted to have auto-sync, but I can report that he was careful to
> check that I have the CA papers before merging a patch from me.

Any chance someone could update the upstream loccur?

Currently, the upstream code and the elpa.git code have diverged:

    Upstream of loccur has DIVERGED!
    
      Local changes:
    b232bc34d4  alexey.vereten..  * packages/loccur/loccur.el: Updated to version 1.2.4
    ca9c2159f3  monnier@iro.um..  * packages/loccur/loccur.el (loccur, loccur-current): Autoload.
    
      Upstream changes:
    01b7afa625  fourier@proton..  Updated function names
    2120345933  fourier@proton..  Updated version to 1.2.5
    9cd6c9eb17  fourier@proton..  Merge pull request #14 from astoff/master
    d4b286a0db  fourier@proton..  Updated contribution information and added license
    75bbd853d5  arstoffel@gmai..  Add loccur-isearch
    284d7bb285  fourier@proton..  Added melpa badge
    4934c0560d  alexey.vereten..  Updated version and comments
    099e8999b0  alexey.vereten..  Fixed issue #12
    194d70e6be  alexey.vereten..  Fixed issue #8
    5650277d58  alexey.vereten..  Implemented solution for the issue #9.
    650d91dda0  txm.fourier@gm..  Merge pull request #7 from syohex/fix-min-version
    0e3234586d  syohex@gmail.com  Set minimum Emacs version

So, we'd need someone with write access to the upstream repo to merge
the b232bc34d4 and ca9c2159f3 commits into it (I don't know if the code
changes are still needed, but we do need the metadata to be merged so
that we can bring the upstream into elpa.git via a fast-forward).


        Stefan

Re: comint-mime and loccur update (was: [RFC] MIME attachments for comint)

[Alexey Veretennikov]

Hey,

I'll update the upstream in the upcoming days.

BR,
/Alexey

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, October 18th, 2021 at 6:44 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Augusto Stoffel wrote:
>
> > All right, let' add it "right now" then :-)
>
> Done.
>
> > I've made the necessary changes, but I guess I'll wait until Emacs 28 is
> >
> > out to make an official release, since the package depends on it.
>
> Not in the mood for time travel, eh?
>
> > PS: The loccur package is outdated on ELPA, could you pull the newest
> >
> > version? FWIW, I didn't get a reply from the maintainer when I asked if
> >
> > he wanted to have auto-sync, but I can report that he was careful to
> >
> > check that I have the CA papers before merging a patch from me.
>
> Any chance someone could update the upstream loccur?
>
> Currently, the upstream code and the elpa.git code have diverged:
>
> Upstream of loccur has DIVERGED!
>
> Local changes:
>
> b232bc34d4 alexey.vereten.. * packages/loccur/loccur.el: Updated to version 1.2.4
>
> ca9c2159f3 monnier@iro.um.. * packages/loccur/loccur.el (loccur, loccur-current): Autoload.
>
> Upstream changes:
>
> 01b7afa625 fourier@proton.. Updated function names
>
> 2120345933 fourier@proton.. Updated version to 1.2.5
>
> 9cd6c9eb17 fourier@proton.. Merge pull request #14 from astoff/master
>
> d4b286a0db fourier@proton.. Updated contribution information and added license
>
> 75bbd853d5 arstoffel@gmai.. Add loccur-isearch
>
> 284d7bb285 fourier@proton.. Added melpa badge
>
> 4934c0560d alexey.vereten.. Updated version and comments
>
> 099e8999b0 alexey.vereten.. Fixed issue #12
>
> 194d70e6be alexey.vereten.. Fixed issue #8
>
> 5650277d58 alexey.vereten.. Implemented solution for the issue #9.
>
> 650d91dda0 txm.fourier@gm.. Merge pull request #7 from syohex/fix-min-version
>
> 0e3234586d syohex@gmail.com Set minimum Emacs version
>
> So, we'd need someone with write access to the upstream repo to merge
>
> the b232bc34d4 and ca9c2159f3 commits into it (I don't know if the code
>
> changes are still needed, but we do need the metadata to be merged so
>
> that we can bring the upstream into elpa.git via a fast-forward).
>
> Stefan