What is the most useful potential feature which Emacs lacks?

What is the most useful potential feature which Emacs lacks?

[ndame]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

There is a discussion on Reddit about sponsoring development of multithreading in Emacs, and people there say it's too hard, takes a lot of time and it doesn't even bring that much benefit to the user.

If this is the case (is it?) then what are those other features which could bring much more tangible benefits for the user and assuming somebody works on them full time sponsored by the community they can be implemented in, say,  a few months?

[-- Attachment #2: Type: text/html, Size: 490 bytes --]

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

ndame <ndame@protonmail.com> writes:

> There is a discussion on Reddit about sponsoring development of multithreading in Emacs, and people there say it's too hard, takes a lot of time and it doesn't even
> bring that much benefit to the user.
>
> If this is the case (is it?) then what are those other features which could bring much more tangible benefits for the user and assuming somebody works on them full time
> sponsored by the community they can be implemented in, say,  a few months?
Maybe a better support for graphics? Let people draw in Emacs window (on
GUi clients). It could benefit some org mode features, some poeple who
does uml via ditaa or similar, powerline users etc. Emacs could be used
in a similar fashion like Conky (system monitoring tool), for those that
use Emacs as a daemon, and could even make Emacs more of a office style
application for some people.

Could also benefit some UI stuff, for example drawing on screen could
benefit more mouse/touch interface so one day Emacs could get a "tablet
ui" or something.

Could be done by exposing underlaying OS/Widget Kit drawing (Cairo, X11,
Gdk, whatever Macs have) to draw directly on Emacs frame/either window
or buffer based. Could also be done by drawing to an "image" (in memory)
via say Cairo surface, XPixmap etc, and then displaying that image in
overlays as Emacs already do.

Could be nice with some task-based parallelism too, like Intel's TBB
does. TBB has Apache license, I am not so legal so I don't know if it is
compatible with GPL, so no idea if it is just opensource or free ...

Re: What is the most useful potential feature which Emacs lacks?

[ndame]

>
> Maybe a better support for graphics? Let people draw in Emacs window (on
> GUi clients).

Yes, it could be useful. One should come up with a concrete specification like what exactly should be implemented and an estimation of how much time it takes.

Somebody on Reddit mentioned that Clojure has a setup where the community sponsors the implementation of features. Emacs could also have something similar:

   In the Clojure community, there is something called ClojuristsTogether, where they
   fund several opensource projects for 3 months, compensated for $3k a month.

https://old.reddit.com/r/emacs/comments/ghq1yx/lets_get_real_multithreading_into_emacs_by_hiring/fqavqiy/

https://www.clojuriststogether.org/

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Kangas]

ndame <ndame@protonmail.com> writes:

> Somebody on Reddit mentioned that Clojure has a setup where the
> community sponsors the implementation of features. Emacs could also
> have something similar:
>
>    In the Clojure community, there is something called ClojuristsTogether, where they
>    fund several opensource projects for 3 months, compensated for $3k a month.
>
> https://old.reddit.com/r/emacs/comments/ghq1yx/lets_get_real_multithreading_into_emacs_by_hiring/fqavqiy/
>
> https://www.clojuriststogether.org/

If we asked our users, I think we would find that many people would be
willing to contribute financially.

Is there anything stopping us from doing something like this?

Best regards,
Stefan Kangas

Re: What is the most useful potential feature which Emacs lacks?

[ndame]

>
> If we asked our users, I think we would find that many people would be
> willing to contribute financially.
>

People are more likely to contribute if they get something sooner than otherwise.

So the questions is: is there someone who would'd love to work on his pet emacs feature full time instead of in his free time? Because if so and it's a popular feature (e.g. graphics canvas for emacs, vscode-like instant support for languages which sets up the lsp server, lsp client and completion automatically, etc.) then it's very likely the community would support that developer via crowdfunding for several months of full time emacs work in return of getting those features quickly in a few months.

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

ndame <ndame@protonmail.com> writes:

>>
>> If we asked our users, I think we would find that many people would be
>> willing to contribute financially.
>>
>
> People are more likely to contribute if they get something sooner than
> otherwise.

> So the questions is: is there someone who would'd love to work on his pet emacs
> feature full time instead of in his free time? Because if so and it's a popular
> feature (e.g. graphics canvas for emacs, vscode-like instant support for
> languages which sets up the lsp server, lsp client and completion automatically,
> etc.) then it's very likely the community would support that developer via
> crowdfunding for several months of full time emacs work in return of getting
> those features quickly in a few months.

Indeed. If you just ask people what they want for certain amount of $$$
you might end up building a death star :-)

https://www.bbc.com/news/world-us-canada-20997144

I can't find the link on U.S. goverment site any more.

It is probably, more constructive to offer a project, or two, to
contribute and help to.

Or maybe you could lift up and help some popular projects to integrate
themselves better into Emacs and develop maybe some more functionality
they might be missing, something like org, magit, projectile, etc.

Maybe you could offer a coaching/mentoring to someone who has a vision
to develop some new functionality that might benefit wider audience of
users?

Re: What is the most useful potential feature which Emacs lacks?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

I want to manage and process my pictures in emacs, like cropping.

I want to play video in emacs and control the playing with emacs style.

在 2020年5月12日 +0800 PM2:57，Arthur Miller <arthur.miller@live.com>，写道：
> ndame <ndame@protonmail.com> writes:
>
> > There is a discussion on Reddit about sponsoring development of multithreading in Emacs, and people there say it's too hard, takes a lot of time and it doesn't even
> > bring that much benefit to the user.
> >
> > If this is the case (is it?) then what are those other features which could bring much more tangible benefits for the user and assuming somebody works on them full time
> > sponsored by the community they can be implemented in, say, a few months?
> Maybe a better support for graphics? Let people draw in Emacs window (on
> GUi clients). It could benefit some org mode features, some poeple who
> does uml via ditaa or similar, powerline users etc. Emacs could be used
> in a similar fashion like Conky (system monitoring tool), for those that
> use Emacs as a daemon, and could even make Emacs more of a office style
> application for some people.
>
> Could also benefit some UI stuff, for example drawing on screen could
> benefit more mouse/touch interface so one day Emacs could get a "tablet
> ui" or something.
>
> Could be done by exposing underlaying OS/Widget Kit drawing (Cairo, X11,
> Gdk, whatever Macs have) to draw directly on Emacs frame/either window
> or buffer based. Could also be done by drawing to an "image" (in memory)
> via say Cairo surface, XPixmap etc, and then displaying that image in
> overlays as Emacs already do.
>
> Could be nice with some task-based parallelism too, like Intel's TBB
> does. TBB has Apache license, I am not so legal so I don't know if it is
> compatible with GPL, so no idea if it is just opensource or free ...
>

[-- Attachment #2: Type: text/html, Size: 2603 bytes --]

Re: What is the most useful potential feature which Emacs lacks?

[Eduardo Ochs]

Hi HaiJun,

several years ago - in 2012, I think - I tried the existing ways of
playing audio from Emacs and I found them hard to use and unreliable,
and I decided to implement a minimalistic way to play audios and
videos from Emacs by calling external programs (sort of) directly...
and I've been incredibly happy with it since then. It's very different
from what you're asking for, but it's very easy to set up and to hack.
If you want to take a look at it, it's here:

  http://angg.twu.net/eev-intros/find-audiovideo-intro.html
  http://angg.twu.net/eev-intros/find-audiovideo-intro.html#4

The link with the "#4" points to a demo. There's a good introduction
to eev here,

  http://angg.twu.net/emacsconf2019.html

and you can install it from ELPA.

  Cheers,
    Eduardo Ochs
    http://angg.twu.net/#eev


On Tue, 12 May 2020 at 21:56, HaiJun Zhang <netjune@outlook.com> wrote:
>
> I want to manage and process my pictures in emacs, like cropping.
>
> I want to play video in emacs and control the playing with emacs style.

Re: What is the most useful potential feature which Emacs lacks?

[Eduardo Ochs]

Oops, sorry, "#4" -> "#4.3"... the right link to the demo is:

  http://angg.twu.net/eev-intros/find-audiovideo-intro.html#4.3

On Tue, 12 May 2020 at 22:34, Eduardo Ochs <eduardoochs@gmail.com> wrote:
>
> Hi HaiJun,
>
> several years ago - in 2012, I think - I tried the existing ways of
> playing audio from Emacs and I found them hard to use and unreliable,
> and I decided to implement a minimalistic way to play audios and
> videos from Emacs by calling external programs (sort of) directly...
> and I've been incredibly happy with it since then. It's very different
> from what you're asking for, but it's very easy to set up and to hack.
> If you want to take a look at it, it's here:
>
>   http://angg.twu.net/eev-intros/find-audiovideo-intro.html
>   http://angg.twu.net/eev-intros/find-audiovideo-intro.html#4
>
> The link with the "#4" points to a demo. There's a good introduction
> to eev here,
>
>   http://angg.twu.net/emacsconf2019.html
>
> and you can install it from ELPA.
>
>   Cheers,
>     Eduardo Ochs
>     http://angg.twu.net/#eev
>
>
> On Tue, 12 May 2020 at 21:56, HaiJun Zhang <netjune@outlook.com> wrote:
> >
> > I want to manage and process my pictures in emacs, like cropping.
> >
> > I want to play video in emacs and control the playing with emacs style.

Re: What is the most useful potential feature which Emacs lacks?

[H. Dieter Wilhelm]

Some too personal ideas?

What I'm missing from stock Emacs:

- Synchronisation of BBDB and Calendar (or other Emacs' PIMs) with
  Nextcloud (or similar.  I read that Tramp connects now to Nextcloud
  :-).

- A tree view in dired (like ztree, for example).

- Starting executables from Dired under Windows (for example with S-RET,
  OK, there's this package dired-launch).

Where I often have to leave Emacs (mainly under Windows @work):

- Capturing screenshots and drawing stuff over them.  This is just an
  observation, I'm not asking to convert Emacs to the Gimp!

What I'm dreaming off:

- Operating a Tor browser (or others) as a true Emacs-Browser (I'm using
  EWW as often as possible, but when a web browser is already open, then
  it is hard to go back).

- Some Dictionary access, in the vain of ispell, when reading, writing
  and browsing.

- And maybe visual word and spelling suggestions like these Android
  keyboards (but I think some of you are working on such things...)

  Dieter


ndame <ndame@protonmail.com> writes:

> There is a discussion on Reddit about sponsoring development of
> multithreading in Emacs, and people there say it's too hard, takes a
> lot of time and it doesn't even bring that much benefit to the user.
>
> If this is the case (is it?) then what are those other features which
> could bring much more tangible benefits for the user and assuming
> somebody works on them full time sponsored by the community they can
> be implemented in, say, a few months?
>

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: What is the most useful potential feature which Emacs lacks?

[Michael Albinus]

dieter@duenenhof-wilhelm.de (H. Dieter Wilhelm) writes:

Hi Dieter,

> - Synchronisation of BBDB and Calendar (or other Emacs' PIMs) with
>   Nextcloud (or similar.  I read that Tramp connects now to Nextcloud
>   :-).

That is just for files. I don't know, whether such a synchronisation can
be implemented based on file copying.

And it needs GOA (Gnome Online Accounts).

>   Dieter

Best regards, Michael.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Connecting to Nextcloud is useful, but if you want your data to be
secure you shouldn't send it unencrypted to a company's server.
Does Tramp have the ability to encrypt the data before sending it to Nextcloud,
and decrypt it after fetching it from Nextcloud?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> Connecting to Nextcloud is useful, but if you want your data to be
> secure you shouldn't send it unencrypted to a company's server.
> Does Tramp have the ability to encrypt the data before sending it to Nextcloud,
> and decrypt it after fetching it from Nextcloud?

Tramp uses GVFS (Gnome Virtual FileSystem) as implementation. Under the
hood, the nextcloud server is connected via davs, that is the dav
protocol, ssl encrypted. Well, these days it is rather tls.

So yes, data are encrypted when Tramp accesses a nextcloud server.

Best regards, Michael.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So yes, data are encrypted when Tramp accesses a nextcloud server.

SSL encrypts the data for transport between Tramp and the nextcloud
server.  Whan I am talking about is to store the data in encrypted
form _in_ the nextcloud server, and not decrypt them until you fetch
them back again.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > So yes, data are encrypted when Tramp accesses a nextcloud server.
>
> SSL encrypts the data for transport between Tramp and the nextcloud
> server.  Whan I am talking about is to store the data in encrypted
> form _in_ the nextcloud server, and not decrypt them until you fetch
> them back again.

Tramp cannot run processes on a remote nextcloud server. Fortunately,
this isn't needed.

When you visit an encrypted file from the nextcloud server, let's call
it foo.gpg, Tramp creates a local copy of that file with the same
extension, like tmp.gpg. That file is visited then in the buffer, and
Emacs' mechanisms recognize it as encrypted, and ask you for the
passphrase to decrypt.

The reverse scenario works similar: If you want to save a buffer bound
to a file on the remote nextcloud server, let's call it foo.gpg again,
Tramp saves the buffer into a local tempoprary file with the same
extension, like tmp.gpg. This triggers encryption of the local
file. Afterwards, Tramp moves the local file to the nextcloud server
with the proper name.

For you (the user) everything looks like handling a local file.

Best regards, Michael.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > SSL encrypts the data for transport between Tramp and the nextcloud
  > > server.  Whan I am talking about is to store the data in encrypted
  > > form _in_ the nextcloud server, and not decrypt them until you fetch
  > > them back again.

  > Tramp cannot run processes on a remote nextcloud server. Fortunately,
  > this isn't needed.

We may be miscommunicating.

You should _only_ do encryption and decryption on your own machine,
using free software that you installed in the usual way.  You cannot
trust a server to do those operations for you.

So the way I have in mind is that Tramp would automatically encrypt
any file before sending it to Nextcloud, and decrypt after fetching
it.  So you would not have to choose your file names to specify "this
is encrypted data."  Tramp, when uploading, would put in something to
indicate that Tramp had encrypted the data, and that would tell Tramp
to decrypt it when fetching it back.

The file name you started with should be encrypted too, so that the 
nextcloud instance cannot see what it was.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> So the way I have in mind is that Tramp would automatically encrypt
> any file before sending it to Nextcloud, and decrypt after fetching
> it.  So you would not have to choose your file names to specify "this
> is encrypted data."  Tramp, when uploading, would put in something to
> indicate that Tramp had encrypted the data, and that would tell Tramp
> to decrypt it when fetching it back.
>
> The file name you started with should be encrypted too, so that the
> nextcloud instance cannot see what it was.

Such functionality does not exist in Tramp.

Remember, Tramp is just a stupid library, which offers alternative
implementations for primitives like write-region, copy-file, or
insert-file-contents. Before Tramp could implement automagic
encoding/decoding, we would need a design in Emacs how it shall look
like. Not only for Tramp, other file name handlers could profit from
this as well. And even files on mounted directories come to mind.

Best regards, Michael.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Before Tramp could implement automagic
  > encoding/decoding, we would need a design in Emacs how it shall look
  > like.

I would suggest that this be part of accessing files on cloudy services.
Anything going up gets encrypted first; anything coming down gets decrypted
on arrival.

However: when people use nextcloud, are they normally using their
own server instances?  Or normally using a server run by someone else?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[H. Dieter Wilhelm]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Before Tramp could implement automagic
>   > encoding/decoding, we would need a design in Emacs how it shall look
>   > like.
>
> I would suggest that this be part of accessing files on cloudy services.
> Anything going up gets encrypted first; anything coming down gets decrypted
> on arrival.
>
> However: when people use nextcloud, are they normally using their
> own server instances?  Or normally using a server run by someone else?

Sorry, I can't tell what the norm is.  Personally, I'm using a webhoster
for my Nextcloud instance.  And I've got a colleague which is running it
on his Raspberry Pi from home.  Of course: Both is possible.

   Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Sorry, I can't tell what the norm is.  Personally, I'm using a webhoster
  > for my Nextcloud instance.  And I've got a colleague which is running it
  > on his Raspberry Pi from home.  Of course: Both is possible.

Since we can't determine whether any given user will want the
auto-encrypt-decrypt, we should let users specify yes or no.
Eventually we might develop ideas for defaults more sophisticated
than just "yes" and "no".

The goal here is to be able to save and retrieve files on any server,
such that the server operator can't determine their contents or their
names.  Tramp would generate the remote file name to use, encrypt the
file contents, then write that into the generated remote file name on
the remote machine.

Tramp would have to maintain a local table mapping specified (local) names to
generated (remote) names.

The data that gets encrypted could contain first the specified file
name, then a delimiter, then the contents of the file.  That way, if
you lose the local table or it is lacking some files, but you have the
encryption key, you can decrypt each saved remote file and find out
what its original specified file name was.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Tramp and crypted files (was: What is the most useful potential feature which Emacs lacks?)

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

> Since we can't determine whether any given user will want the
> auto-encrypt-decrypt, we should let users specify yes or no.
> Eventually we might develop ideas for defaults more sophisticated
> than just "yes" and "no".

As I said the other thread, we could create a new connection method
"nextcloud-crypt" which does the job. A user could decide whether she
uses "nextcloud" or "nextcloud-crypt" when accessing a given file. This
could happen even in parallel for different files on the same server.

Another approach might be to create a new file name handler, which just
performs the encryption/decryption job, plus proper handling of file
names. This could be enabled for dedicated remote servers only, and
works in combination with Tramp. That new file name handlers performs the
encryption/deryption locally, and Tramp is responsible for copying
from/to the server.

> The goal here is to be able to save and retrieve files on any server,
> such that the server operator can't determine their contents or their
> names.  Tramp would generate the remote file name to use, encrypt the
> file contents, then write that into the generated remote file name on
> the remote machine.
>
> Tramp would have to maintain a local table mapping specified (local) names to
> generated (remote) names.
>
> The data that gets encrypted could contain first the specified file
> name, then a delimiter, then the contents of the file.  That way, if
> you lose the local table or it is lacking some files, but you have the
> encryption key, you can decrypt each saved remote file and find out
> what its original specified file name was.

That sounds unstable. I believe we shall find a way to encrypt/decrypt
the file name with the same passphrase as the contents of the file(s);
by this we wouldn't need to keep a local mapping file. The encrypted
file name could be adapted by base64 then, in order to make it fit to
the file system's naming conventions.

The advantage would be, that we would know the original file name w/o
copying and decrypting the whole file first.

A local mapping table would be in the way, if different people would
like to access a given file from their own Emacs stanzas. That's what
"cloudy servers" are good for.

Best regards, Michael.

Re: Tramp and crypted files (was: What is the most useful potential feature which Emacs lacks?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > As I said the other thread, we could create a new connection method
  > "nextcloud-crypt" which does the job. A user could decide whether she
  > uses "nextcloud" or "nextcloud-crypt" when accessing a given file. This
  > could happen even in parallel for different files on the same server.

That could work -- but it is limited to nextcloud only.  I am
thinking of this as an option to offer for any and all kinds of remote
machines.

  >  I believe we shall find a way to encrypt/decrypt
  > the file name with the same passphrase as the contents of the file(s);
  > by this we wouldn't need to keep a local mapping file. The encrypted
  > file name could be adapted by base64 then, in order to make it fit to
  > the file system's naming conventions.

I didn't think of that one, but it sounds good.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Tramp and crypted files

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > As I said the other thread, we could create a new connection method
>   > "nextcloud-crypt" which does the job. A user could decide whether she
>   > uses "nextcloud" or "nextcloud-crypt" when accessing a given file. This
>   > could happen even in parallel for different files on the same server.
>
> That could work -- but it is limited to nextcloud only.  I am
> thinking of this as an option to offer for any and all kinds of remote
> machines.

Agreed. Therefore I made the alternative proposal to add another file
name handler which is responsible for the en-/decryption part. Combined
with Tramp, it shall work for all remote connection types.

There is the proposal to use encfs for this purpose. I haven't played
with this yet, but it sounds interesting. Together with that guy, I will
try to do a proof-of-concept. Let's see where we end.

Best regards, Michael.

Re: Tramp and crypted files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Agreed. Therefore I made the alternative proposal to add another file
  > name handler which is responsible for the en-/decryption part. Combined
  > with Tramp, it shall work for all remote connection types.

Now that you understand the basic idea of my suggestion, you're the
best expert on what form it should take.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Tramp and crypted files

[Deus Max]

On Mon, May 18 2020, Michael Albinus wrote:

> Richard Stallman <rms@gnu.org> writes:
>
>> Since we can't determine whether any given user will want the
>> auto-encrypt-decrypt, we should let users specify yes or no.
>> Eventually we might develop ideas for defaults more sophisticated
>> than just "yes" and "no".
>
> As I said the other thread, we could create a new connection method
> "nextcloud-crypt" which does the job. A user could decide whether she
> uses "nextcloud" or "nextcloud-crypt" when accessing a given file. This
> could happen even in parallel for different files on the same server.
>
> Another approach might be to create a new file name handler, which just
> performs the encryption/decryption job, plus proper handling of file
> names. This could be enabled for dedicated remote servers only, and
> works in combination with Tramp. That new file name handlers performs the
> encryption/deryption locally, and Tramp is responsible for copying
> from/to the server.
>

Hi Michael,

I wrote a bash script that manages davfs2 and encfs for uploading the
data to the cloud with WebDAV. See for details
https://github.com/deusmax/cloud-davfs-encfs.

I have tested the package only with nextcloud, but the whole setup seems
to be generic webdav. Should work with other cloud providers that
support webdav.

Tramp uses the GOA library. I don't know if this is plus or a minus
compared to davfs2. I should look into supporting GOA, anyway.

I would be happy to port it to Tramp. You think it may work ?
Can someone give me a pointer to start ?

This work could result into creating connection methods like "davs-enc"
or "nextcloud-enc".

DeusMax

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

> Hi Michael,

Hi,

> I wrote a bash script that manages davfs2 and encfs for uploading the
> data to the cloud with WebDAV. See for details
> https://github.com/deusmax/cloud-davfs-encfs.
>
> I have tested the package only with nextcloud, but the whole setup seems
> to be generic webdav. Should work with other cloud providers that
> support webdav.
>
> Tramp uses the GOA library. I don't know if this is plus or a minus
> compared to davfs2. I should look into supporting GOA, anyway.
>
> I would be happy to port it to Tramp. You think it may work ?
> Can someone give me a pointer to start ?
>
> This work could result into creating connection methods like "davs-enc"
> or "nextcloud-enc".

Thanks for your proposal! I haven't tried encfs yet, but is sounds
promising.

As written in my other messages, I don't believe (anymore) we shall mix
the en-/decryption part with Tramp implementation. This shall be
implemented in another file name handler, working over local
files. Tramp with whatever backend would be responsible then for copying
the encrypted files from/to the remote side.

Best might be, if you start to read Emacs docs about file name handlers,
and how they are supposed to work. There's not too much in the doc; a
starting point would be (info "(elisp) Magic File Names")

In parallel, I will download your package, and play with it.

> DeusMax

Best regards, Michael.

Re: Tramp and crypted files

[Deus Max]

On Tue, May 19 2020, Michael Albinus wrote:

> Deus Max <deusmax@gmx.com> writes:
>
>
>> I wrote a bash script that manages davfs2 and encfs for uploading the
>> data to the cloud with WebDAV. See for details
>> https://github.com/deusmax/cloud-davfs-encfs.
>>
>> I have tested the package only with nextcloud, but the whole setup seems
>> to be generic webdav. Should work with other cloud providers that
>> support webdav.
>>
>> Tramp uses the GOA library. I don't know if this is plus or a minus
>> compared to davfs2. I should look into supporting GOA, anyway.
>>
>> I would be happy to port it to Tramp. You think it may work ?
>> Can someone give me a pointer to start ?
>>
>> This work could result into creating connection methods like "davs-enc"
>> or "nextcloud-enc".
>
> Thanks for your proposal! I haven't tried encfs yet, but is sounds
> promising.
>
> As written in my other messages, I don't believe (anymore) we shall mix
> the en-/decryption part with Tramp implementation. This shall be
> implemented in another file name handler, working over local
> files. Tramp with whatever backend would be responsible then for copying
> the encrypted files from/to the remote side.
>
Agree.
Encfs handles the encryption.
The actual files are encrypted, encfs defines a mount-point where the
files are displayed decrypted.

> Best might be, if you start to read Emacs docs about file name handlers,
> and how they are supposed to work. There's not too much in the doc; a
> starting point would be (info "(elisp) Magic File Names")
>
Great! thank you. I will look into it.

> In parallel, I will download your package, and play with it.
>
If you have any questions let me know.

Having an easy to use Tramp method for encrypting cloud data would be a
good plus for privacy.

DeusMax

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

Hi,

>> Best might be, if you start to read Emacs docs about file name handlers,
>> and how they are supposed to work. There's not too much in the doc; a
>> starting point would be (info "(elisp) Magic File Names")
>>
> Great! thank you. I will look into it.

It looks like you have signed FSF legal paper for PLANNER. At least
there is somebody listed with a similar email address as you have. If
you want to contribute to Emacs/Tramp (I suppose so), it might be good
to sign the paper also for these two projects.

If it isn't you, there are even more reasons to sign the paper :-)

Are you willing to do so?

> DeusMax

Best regards, Michael.

Re: Tramp and crypted files

[Deus Max]

On Wed, May 20 2020, Michael Albinus wrote:

> It looks like you have signed FSF legal paper for PLANNER. At least
> there is somebody listed with a similar email address as you have. If
> you want to contribute to Emacs/Tramp (I suppose so), it might be good
> to sign the paper also for these two projects.
>
> If it isn't you, there are even more reasons to sign the paper :-)
>
> Are you willing to do so?
>
> Best regards, Michael.

Yes, I will be willing to sign FSF papers.

That was some time ago!

Also some time ago, I had signed some papers FSF papers for php-mode
under a gmail account (deusmax@gmail.com) perhaps or badekas@aia.gr.

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

Hi,

>> It looks like you have signed FSF legal paper for PLANNER. At least
>> there is somebody listed with a similar email address as you have. If
>> you want to contribute to Emacs/Tramp (I suppose so), it might be good
>> to sign the paper also for these two projects.
>>
>> If it isn't you, there are even more reasons to sign the paper :-)
>>
>> Are you willing to do so?
>
> Yes, I will be willing to sign FSF papers.

Thanks, I'll send you the recent file off-list.

> Also some time ago, I had signed some papers FSF papers for php-mode
> under a gmail account (deusmax@gmail.com) perhaps or badekas@aia.gr.

That I couldn't find in copyright.list on fencepost. If it is important
for you, you might check with the FSF clerk about.

Best regards, Michael.

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

Hi,

>> As written in my other messages, I don't believe (anymore) we shall mix
>> the en-/decryption part with Tramp implementation. This shall be
>> implemented in another file name handler, working over local
>> files. Tramp with whatever backend would be responsible then for copying
>> the encrypted files from/to the remote side.
>
> Agree.
> Encfs handles the encryption.
> The actual files are encrypted, encfs defines a mount-point where the
> files are displayed decrypted.
>
> Having an easy to use Tramp method for encrypting cloud data would be a
> good plus for privacy.

I have played with encfs and your script as well as with first snippets
of a Tramp implementation. Just for discussion, here are my conclusions
so far:

- Encryption of files and file names shall be possible for *every*
  remote connection. This means, that the approach will be different
  from what you have done in your script (where you work over webdav
  based cloud servers).

- Encryption of files and file names shall be separated from vanilla
  Tramp. It is optional, and a user must enable it explicitly for a
  given remote directory. This is because of performance, and because of
  implementation simplicity. As a result, there shall be almost no
  change of existing Tramp; all encrytion functionality will be
  cumulated in a new tramp-crypt.el file.

  Of course, encryption can be activated for several remote directories
  in parallel. But they must not be subdirectories of each other.

- As a consequence, there will be an additional file name handler, which
  reacts on the same file name syntax as Tramp. It is arranged to be
  called before the vanilla Tramp file name handler. All of its
  functions will check, whether a user has activated encryption for a
  given remote directory. In that case, if an argument of a function is
  a file name which belongs to such a directory, that file name will be
  transformed into its crypted counterpart, and the native Tramp file
  name handler is activated for this function with encrypted file
  names. If the function returns file names, the reverse action is
  applied: if a file name is encrypted, the result will be adapted to
  contain the corresponding decrypted file name.

- For file copying, the file itself is either encrypted (when copying
  to remote) or decrypted (when copying from remote). Together with the
  encryption/decryption of the file name, the copy operation will be
  applied by vanilla Tramp operation.

- There will be *no* mounted encfs file system. File name
  encryption/decryption will be performed by "encfsctl encode ..." and
  "encfsctl decode ..." process calls. File encryption happens via
  "encfsctl cat ..." and "encfsctl cat --reverse ...".

- The local rootdir of a crypted remote directory will be created temporarily
  when needed. It is always rearrangeable via its config file
  .encfs6.xml, which contains the filesystem information. Only this
  config file will be kept persistently, one file per activated crypted
  remote directory, somewhere in ~/.emacs.d/. Optionally, it will be
  kept also in the crypted remote directory as well.

  With this, encrypted files from remote can be accessed by different
  Emacs sessions running from different host, by different users. All
  what they need to know is the remote directory name (in Tramp syntax),
  and the password the encryption/decryption is protected with. That's
  what "cloudy servers" are good for.

Comments?

> DeusMax

Best regards, Michael.

Re: Tramp and crypted files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > - Encryption of files and file names shall be separated from vanilla
  >   Tramp. It is optional, and a user must enable it explicitly for a
  >   given remote directory. This is because of performance, and because of
  >   implementation simplicity. As a result, there shall be almost no
  >   change of existing Tramp; all encrytion functionality will be
  >   cumulated in a new tramp-crypt.el file.

This seems like a good architecture for the design.

I suggest having a feature where the user can specify to always
use encryption for certain host names.  So if you specify a host
name which is on that list (of names, or perhaps regexps?), tramp
would modify the request so as to do the encryption.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Tramp and crypted files

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> I suggest having a feature where the user can specify to always
> use encryption for certain host names.  So if you specify a host
> name which is on that list (of names, or perhaps regexps?), tramp
> would modify the request so as to do the encryption.

You can add in your .emacs entries like "/nextcloud:user@host:/" to the
list of crypted remote directories. Several such entries would work like
your proposed list of host names.

It might be possible to use a regexp instead, but there are pitfalls: I
doubt, that adding "/ssh:user@host:/" is a good idea. I've added your
proposal to the TODO list.

Best regards, Michael.

Re: Tramp and crypted files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You can add in your .emacs entries like "/nextcloud:user@host:/" to the
  > list of crypted remote directories.

That sounds like the right thing.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Tramp and crypted files

[Deus Max]

On Mon, May 25 2020, Michael Albinus wrote:

> Deus Max <deusmax@gmx.com> writes:
>
> Hi,
>
>>> As written in my other messages, I don't believe (anymore) we shall mix
>>> the en-/decryption part with Tramp implementation. This shall be
>>> implemented in another file name handler, working over local
>>> files. Tramp with whatever backend would be responsible then for copying
>>> the encrypted files from/to the remote side.
>>
>> Agree.
>> Encfs handles the encryption.
>> The actual files are encrypted, encfs defines a mount-point where the
>> files are displayed decrypted.
>>
>> Having an easy to use Tramp method for encrypting cloud data would be a
>> good plus for privacy.
>
> I have played with encfs and your script as well as with first snippets
> of a Tramp implementation. Just for discussion, here are my conclusions
> so far:
>
> - Encryption of files and file names shall be possible for *every*
>   remote connection. This means, that the approach will be different
>   from what you have done in your script (where you work over webdav
>   based cloud servers).
>
OK.

> - Encryption of files and file names shall be separated from vanilla
>   Tramp. It is optional, and a user must enable it explicitly for a
>   given remote directory. This is because of performance, and because of
>   implementation simplicity. As a result, there shall be almost no
>   change of existing Tramp; all encrytion functionality will be
>   cumulated in a new tramp-crypt.el file.
>
Good.

>   Of course, encryption can be activated for several remote directories
>   in parallel. But they must not be subdirectories of each other.
>
> - As a consequence, there will be an additional file name handler, which
>   reacts on the same file name syntax as Tramp. It is arranged to be
>   called before the vanilla Tramp file name handler. All of its
>   functions will check, whether a user has activated encryption for a
>   given remote directory. In that case, if an argument of a function is
>   a file name which belongs to such a directory, that file name will be
>   transformed into its crypted counterpart, and the native Tramp file
>   name handler is activated for this function with encrypted file
>   names. If the function returns file names, the reverse action is
>   applied: if a file name is encrypted, the result will be adapted to
>   contain the corresponding decrypted file name.
>
> - For file copying, the file itself is either encrypted (when copying
>   to remote) or decrypted (when copying from remote). Together with the
>   encryption/decryption of the file name, the copy operation will be
>   applied by vanilla Tramp operation.
>
> - There will be *no* mounted encfs file system. File name
>   encryption/decryption will be performed by "encfsctl encode ..." and
>   "encfsctl decode ..." process calls. File encryption happens via
>   "encfsctl cat ..." and "encfsctl cat --reverse ...".
>
EncFS is not needed for the this setup. Vanilla gpg encryption could be
used for the above. Without hidding the filename (with .gpg extension).
only encrypt the content.

EncFs adds file name encryption and obsfucation, making in hard to guess
the encrypted file, even if you know the file name. So you have to
temporarily mount somewhere, in order to see the decrypted filenames.

> - The local rootdir of a crypted remote directory will be created temporarily
>   when needed. It is always rearrangeable via its config file
>   .encfs6.xml, which contains the filesystem information. Only this
>   config file will be kept persistently, one file per activated crypted
>   remote directory, somewhere in ~/.emacs.d/. Optionally, it will be
>   kept also in the crypted remote directory as well.
>
Yes, the .encfs6.xml is very importantf for EncFS.
I think encfs needs a temprorary mount point, to function. This can be a
weakness in a network situation, as any interruption could leave the
files open or in a strange state, inviting the case to be compromised.

>   With this, encrypted files from remote can be accessed by different
>   Emacs sessions running from different host, by different users. All
>   what they need to know is the remote directory name (in Tramp syntax),
>   and the password the encryption/decryption is protected with. That's
>   what "cloudy servers" are good for.
>
Correct me if I'm wrong, but I don't think the webdav protocol behaves
well for multi-user editing. It simple saves the last edit. without
comparing for merge conflicts. It is a last save takes all.
For access from different hosts, the user should take care to use strict
sequential access.

DeusMax

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

Hi,

> EncFs adds file name encryption and obsfucation, making in hard to guess
> the encrypted file, even if you know the file name. So you have to
> temporarily mount somewhere, in order to see the decrypted filenames.

No. The new tramp-crypt file name handler makes this transparent to
you. Given, you have declared "/nextcloud:host:/crypted/" as crypted
remote directory. If you call for example

(directory-file "/nextcloud:host:/crypted/subdir")

this file name handler will transform "/nextcloud:host:/crypted/subdir"
to "/nextcloud:host:/crypted/XXX", in case you have created a
subdirectory "subdir" and it has the name "XXX" on the nextcloud
server. Then the vanilla Tramp command is called as

(directory-file "/nextcloud:host:/crypted/XXX")

using the encrypted file name. It returns the list ("." ".." "YYY" "ZZZ"),
with "YYY" and "ZZZ" being encrypted file names on the server. This
result is received by the file name handler, and it transforms this list
to ("." ".." "foo" "bar"), with "foo" and "bar" being the plain text
file names of "YYY" and "ZZZ". So, finally you see

(directory-file "/nextcloud:host:/crypted/subdir")
=> ("." ".." "foo" "bar")

without even thinking about that this is a crypted remote
directory. Same scenario for all other magic primitives, which are
implemented by Tramp.

> Yes, the .encfs6.xml is very importantf for EncFS.
> I think encfs needs a temprorary mount point, to function. This can be a
> weakness in a network situation, as any interruption could leave the
> files open or in a strange state, inviting the case to be compromised.

No. An encfs mount point is needed only in case you create a new
.encfs6.xml file. Tramp would do this transparently by calling "encfs
tmpdir1 tmpdir2". Then it saves tmpdir1/.encfs6.xml, unmounts the encfs
mountpoint, and removes the temporary directories.

In ordinary use, if a file or file name needs to be encrypted or
decrypted, just the rootdir is necessary, no mountpoint. See this: I
have a root dir at /tmp/rootdir/. There is the crypted file
xyswI5g6Pf3R7qOMKy1jDA8m. And I can still do

--8<---------------cut here---------------start------------->8---
# mount | grep encfs

# ls -al /tmp/rootdir
total 8
drwxrwxr-x.   2 albinus albinus   80 May 29 10:48 .
drwxrwxrwt. 114 root    root    5960 May 29 10:45 ..
-rw-rw-r--.   1 albinus albinus 1297 May 29 10:32 .encfs6.xml
-rw-r--r--.   1 albinus albinus   26 May 29 10:48 xyswI5g6Pf3R7qOMKy1jDA8m

# encfsctl encode /tmp/rootdir foo
EncFS Password:
xyswI5g6Pf3R7qOMKy1jDA8m

# encfsctl decode /tmp/rootdir xyswI5g6Pf3R7qOMKy1jDA8m
EncFS Password:
foo

# encfsctl cat /tmp/rootdir xyswI5g6Pf3R7qOMKy1jDA8m
EncFS Password:
This is file foo.
--8<---------------cut here---------------end--------------->8---

Well, I must confess that I have trouble to make "encfsctl cat --reverse"
working. Will dig what's up.

>>   With this, encrypted files from remote can be accessed by different
>>   Emacs sessions running from different host, by different users. All
>>   what they need to know is the remote directory name (in Tramp syntax),
>>   and the password the encryption/decryption is protected with. That's
>>   what "cloudy servers" are good for.
>>
> Correct me if I'm wrong, but I don't think the webdav protocol behaves
> well for multi-user editing. It simple saves the last edit. without
> comparing for merge conflicts. It is a last save takes all.
> For access from different hosts, the user should take care to use strict
> sequential access.

Honestly, I don't care which Tramp method is used. Whether you use a
remote nextcloud server, or a remote ssh server, doesn't matter. The
user must decide what's best.

The same problem you mention happens for all remote files handled by
Tramp, also for not encrypted ones.

For my internal testing, I use as crypted remote directory "/ssh::/tmp/xxx/".
That's fast, and good enough.

> DeusMax

Best regards, Michael.

Re: Tramp and crypted files

[Deus Max]

On Fri, May 29 2020, Michael Albinus wrote:

> Deus Max <deusmax@gmx.com> writes:
>
>> EncFs adds file name encryption and obsfucation, making in hard to guess
>> the encrypted file, even if you know the file name. So you have to
>> temporarily mount somewhere, in order to see the decrypted filenames.
>
> No. The new tramp-crypt file name handler makes this transparent to
> you. Given, you have declared "/nextcloud:host:/crypted/" as crypted
> remote directory. If you call for example
>
> (directory-file "/nextcloud:host:/crypted/subdir")
>
> this file name handler will transform "/nextcloud:host:/crypted/subdir"
> to "/nextcloud:host:/crypted/XXX", in case you have created a
> subdirectory "subdir" and it has the name "XXX" on the nextcloud
> server. Then the vanilla Tramp command is called as
>
> (directory-file "/nextcloud:host:/crypted/XXX")
>
> using the encrypted file name. It returns the list ("." ".." "YYY" "ZZZ"),
> with "YYY" and "ZZZ" being encrypted file names on the server. This
> result is received by the file name handler, and it transforms this list
> to ("." ".." "foo" "bar"), with "foo" and "bar" being the plain text
> file names of "YYY" and "ZZZ". So, finally you see
>
> (directory-file "/nextcloud:host:/crypted/subdir")
> => ("." ".." "foo" "bar")
>
> without even thinking about that this is a crypted remote
> directory. Same scenario for all other magic primitives, which are
> implemented by Tramp.
>
Great! I  need to reed up on the new tramp-crypt handler.

>> Yes, the .encfs6.xml is very importantf for EncFS.
>> I think encfs needs a temprorary mount point, to function. This can be a
>> weakness in a network situation, as any interruption could leave the
>> files open or in a strange state, inviting the case to be compromised.
>
> No. An encfs mount point is needed only in case you create a new
> .encfs6.xml file. Tramp would do this transparently by calling "encfs
> tmpdir1 tmpdir2". Then it saves tmpdir1/.encfs6.xml, unmounts the encfs
> mountpoint, and removes the temporary directories.
>
True. Thanks for correcting that.

DeusM

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

Hi,

> Great! I  need to reed up on the new tramp-crypt handler.

Well, it took longer than expected due to other responsibilities. But
now I have pushed to master the very first version of tramp-crypt.el.

It isn't ready to be used in production, but it passes already the
majority of tests in tramp-tests.el. I haven't updated tramp.texi yet,
but the Commentary section in tramp-crypt.el should give you a first
howto for application.

Note that you can apply this for any remote directory. But for testing,
I have created a local directory "/tmp/crypt", and I have used the
remote directory "/ssh:hostnamegandalf:/tmp/crypt" - this makes it much
easier to test. Tests on my nextcloud server look also promising, but
this needs more polishing.

The Emacs Makefile doesn't seem to recreate tramp-loaddefs.el
automatically. In case you see compilation errors, try "make -C lisp
autoloads" first.

Comment welcome!

Best regards, Michael.

Re: Tramp and crypted files

[Deus Max]

On Sun, Jun 07 2020, Michael Albinus wrote:

> Deus Max <deusmax@gmx.com> writes:
>
> Hi,
>
>> Great! I  need to reed up on the new tramp-crypt handler.
>
> Well, it took longer than expected due to other responsibilities. But
> now I have pushed to master the very first version of tramp-crypt.el.
>
> It isn't ready to be used in production, but it passes already the
> majority of tests in tramp-tests.el. I haven't updated tramp.texi yet,
> but the Commentary section in tramp-crypt.el should give you a first
> howto for application.
>
> Note that you can apply this for any remote directory. But for testing,
> I have created a local directory "/tmp/crypt", and I have used the
> remote directory "/ssh:hostnamegandalf:/tmp/crypt" - this makes it much
> easier to test. Tests on my nextcloud server look also promising, but
> this needs more polishing.
>
> The Emacs Makefile doesn't seem to recreate tramp-loaddefs.el
> automatically. In case you see compilation errors, try "make -C lisp
> autoloads" first.
>
> Comment welcome!
>
> Best regards, Michael.

Hi Michael,

Sorry for the late reply, but other priorities have distracted me.
I have even fallen behind on reading the mailing list !

I would be happy to test and comment, but will need some time to settle
the other priorities.
I'll come back in a week or so.

DeusMax

Re: Tramp and crypted files

[Michael Albinus]

Deus Max <deusmax@gmx.com> writes:

> Hi Michael,

Hi,

> I would be happy to test and comment, but will need some time to settle
> the other priorities.
> I'll come back in a week or so.

No problem, take your time. In the meantime, I'll try to pass as many of
the tramp-tests.el tests as possible.

I'll plan to release Tramp 2.5.0 end of June; it would be great to know
whether tramp-crypt.el could be part of this.

> DeusMax

Best regards, Michael.

Re: Tramp and crypted files

[Deus Max]

On Fri, May 29 2020, Michael Albinus wrote:

>
> Well, I must confess that I have trouble to make "encfsctl cat --reverse"
> working. Will dig what's up.
>

The same here too. Keep getting:

   : The configuration loaded is not compatible with --reverse
   : Unable to initialize encrypted filesystem - check path.

Reading from
https://www.reddit.com/r/PlexACD/comments/5l0gqy/mount_in_reverse_mode_for_better_performance/,
found two hints:

  1. setup configuration with provide script
     : add in --reverse which mounts an encrypted (view) of a decrypted
     : folder, vs the current setup which is a decrypted view of the
     : encrypted folder. So encryption is only done upon read rather than
     : write.

  2. IV name chaining.
     : This is probably caused because you have IV name chaining on.
     : (This is set in your .encfs6.xml file). IV name chaining does
     : not work with reverse mounting.

Hope this helps. Of course, YMMV.

DeusM

Re: What is the most useful potential feature which Emacs lacks?

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > Before Tramp could implement automagic
>   > encoding/decoding, we would need a design in Emacs how it shall look
>   > like.
>
> I would suggest that this be part of accessing files on cloudy services.
> Anything going up gets encrypted first; anything coming down gets decrypted
> on arrival.

We cannot enable this by default. There are good reasons that people
bring their files on a "cloudy" server unencrypted, for example because
they want to share those files with other people, or they want to access
these files with other tools but Emacs. So it could be an option only,
like a new Tramp method "nextcloud-crypt" or whatever.

And it doesn't affect file copying only. You want to hide the file
names, which has effect for most of the file name operations Tramp
provides an own implementation.

> However: when people use nextcloud, are they normally using their
> own server instances?  Or normally using a server run by someone else?

Like Dieter said, both is happening. I, for example, use two local
nextcloud servers on my NAS devices in the basement, and I use also
public nextcloud servers somewhere else.

Best regards, Michael.

Re: What is the most useful potential feature which Emacs lacks?

[Eric S Fraga]

On Tuesday, 12 May 2020 at 12:00, H. Dieter Wilhelm wrote:
> I read that Tramp connects now to Nextcloud

Any tutorials on this?  I have tried, based on the small snippet in the
tramp info manual, and all I get is "There is no online account
...".  It would make some of the discussed features available easily
(relatively).

Thank you.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.6 on Debian bullseye/sid

Re: What is the most useful potential feature which Emacs lacks?

[Michael Albinus]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

Hi Eric,

>> I read that Tramp connects now to Nextcloud
>
> Any tutorials on this?  I have tried, based on the small snippet in the
> tramp info manual, and all I get is "There is no online account
> ...".  It would make some of the discussed features available easily
> (relatively).

"... your credentials must be populated in your ‘Online Accounts’
application outside Emacs."

From the Tramp manual, speaking about the nextcloud method.

> Thank you.

Best regards, Michael.

Re: What is the most useful potential feature which Emacs lacks?

[Eric S Fraga]

On Tuesday, 12 May 2020 at 17:34, Michael Albinus wrote:
> "... your credentials must be populated in your ‘Online Accounts’
> application outside Emacs."
>
> From the Tramp manual, speaking about the nextcloud method.

Thanks for this.  And, boy, did this ever send me down a rabbit
hole... anyway, finally managed to get GNOME running and created an
"online" account.  Now, back with my usual DE/WM and running Emacs
again, I get an error like this:

tramp-signal-hook-function:
   Host name ‘MY.NEXTCLOUD.SITE’ does not match
   ‘\`\(127\.0\.0\.1\|::1\|localhost6?\|t3610\)\'’

and I don't see why it should as that is the regexp for identifying a
local host, which is most definitely what I do not want given that I am
trying to connect to a nextcloud server...  hey hum.  Tramp info manual
not helpful in this regard unfortunately.

Anyway, apologies for hijacking the thread although, in my defence, the
desirable potential feature of being able to transparently access
resources on the web is what started me off...

Thanks again Michael.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.6 on Debian bullseye/sid

Tramp nextcloud (was: What is the most useful potential feature which Emacs lacks?)

[Michael Albinus]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

Hi Eric,

>> From the Tramp manual, speaking about the nextcloud method.
>
> Thanks for this.  And, boy, did this ever send me down a rabbit
> hole... anyway, finally managed to get GNOME running and created an
> "online" account.

Hmm. I'm not good in documentation. If you have proposals for the
manual, pls tell me.

> Now, back with my usual DE/WM and running Emacs
> again, I get an error like this:
>
> tramp-signal-hook-function:
>    Host name ‘MY.NEXTCLOUD.SITE’ does not match
>    ‘\`\(127\.0\.0\.1\|::1\|localhost6?\|t3610\)\'’
>
> and I don't see why it should as that is the regexp for identifying a
> local host, which is most definitely what I do not want given that I am
> trying to connect to a nextcloud server...  hey hum.  Tramp info manual
> not helpful in this regard unfortunately.

I have a nextcloud account on host "ecloud.global" with the user name
"michael.albinus@e.email". (Well, this belongs to the /e/ project, a
Google-less phone). So I open in Emacs

C-x C-f /nextcloud:michael.albinus@e.email@ecloud.global:/

Does this help? If not, what OS, Emacs version and Tramp version (if not
the Emacs built-in Tramp) are you using? Do you have enabled D-Bus in
your Emacs built?

> Anyway, apologies for hijacking the thread although, in my defence, the
> desirable potential feature of being able to transparently access
> resources on the web is what started me off...

I've changed the subject :-)

> Thanks again Michael.

Best regards, Michael.

Re: Tramp nextcloud

[H. Dieter Wilhelm]

Michael Albinus <michael.albinus@gmx.de> writes:
> ...
> I have a nextcloud account on host "ecloud.global" with the user name
> "michael.albinus@e.email". (Well, this belongs to the /e/ project, a
> Google-less phone). So I open in Emacs
>
> C-x C-f /nextcloud:michael.albinus@e.email@ecloud.global:/
>
> Does this help? If not, what OS, Emacs version and Tramp version (if not

Yes it helps here: Ubuntu 18.04, Emacs 28.0.50 :-)

Thank you Michael!

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Tramp nextcloud

[Eric S Fraga]

On Tuesday, 12 May 2020 at 19:39, Michael Albinus wrote:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>> Thanks for this.  And, boy, did this ever send me down a rabbit
>> hole... anyway, finally managed to get GNOME running and created an
>> "online" account.
>
> Hmm. I'm not good in documentation. If you have proposals for the
> manual, pls tell me.

Not your fault!  The rabbit hole was regarding gnome.  Obviously, it
would be best if tramp did not require external tools but that is
sometimes difficult.  I don't use gnome and the documentation for
gnome-online-accounts assumes that you are using gnome.

> I have a nextcloud account on host "ecloud.global" with the user name
> "michael.albinus@e.email". (Well, this belongs to the /e/ project, a
> Google-less phone). So I open in Emacs
>
> C-x C-f /nextcloud:michael.albinus@e.email@ecloud.global:/
>
> Does this help? 

It helps in that it is more clear as to what I should be specifying,
maybe.  That is, I'm not sure if my account on my server requires the
email address or only the user id.  I've tried both and I still get the
error about the server not matching the local host regexp in both
cases...  I don't think tramp gets as far as trying to authenticate with
the server.

> If not, what OS, Emacs version and Tramp version (if not the Emacs
> built-in Tramp) are you using? Do you have enabled D-Bus in your Emacs
> built?

I'm using Emacs from git (28.0.50) with the built-in tramp.  My Emacs
was last updated a week ago (but happy to update at any time).

The settings for my Emacs configuration are:

#define EMACS_CONFIG_FEATURES "XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND
 DBUS GSETTINGS GLIB NOTIFY INOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE
 HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS
 PDUMPER LCMS2 GMP"

so dbus does seem to be there.

Thanks again,
eric
-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.6 on Debian bullseye/sid

Re: Tramp nextcloud (SOLVED)

[Eric S Fraga]

Solved!  Although not sure how/why.

1. I updated Emacs from git/master just now and rebuilt.
2. I removed two tramp customization lines I had in my init files.[1]

I can now access nextcloud files using /nextcloud:userid@cloud.server:/

Thank you!

eric

Footnotes:
[1]  one of the problems with having an Emacs configuration that has
     evolved organically over 35+ years...

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.6 on Debian bullseye/sid

Re: Tramp nextcloud (SOLVED)

[Michael Albinus]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Solved!  Although not sure how/why.

Great!

> 1. I updated Emacs from git/master just now and rebuilt.
> 2. I removed two tramp customization lines I had in my init files.[1]
>
> I can now access nextcloud files using /nextcloud:userid@cloud.server:/

My fault. I should have asked you to try "emacs -Q" - the first request
I always send when seeing reports on Tramp problems :-)

> Thank you!
>
> eric

Best regards, Michael.

Re: Tramp nextcloud (SOLVED)

[Eric S Fraga]

On Wednesday, 13 May 2020 at 12:45, Michael Albinus wrote:
> My fault. I should have asked you to try "emacs -Q" - the first request
> I always send when seeing reports on Tramp problems :-)

Not at all.  I should have tried -Q first.

Thanks again,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.6 on Debian bullseye/sid

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

dieter@duenenhof-wilhelm.de (H. Dieter Wilhelm) writes:

> Some too personal ideas?
>
> What I'm missing from stock Emacs:
>
> - Synchronisation of BBDB and Calendar (or other Emacs' PIMs) with
>   Nextcloud (or similar.  I read that Tramp connects now to Nextcloud
>   :-).
>
> - A tree view in dired (like ztree, for example).
You can already have this with some third party pacakges

Checkout dired-hacks and dired-ranger. I think there were some other
packages offereing a tree view of directories, but I don't remember any
more I don't use dired that way myself.

> - Starting executables from Dired under Windows (for example with S-RET,
>   OK, there's this package dired-launch).
You can do this too, shouldn't be a problem with few shortcuts, or
dired-launch or open-with.
> Where I often have to leave Emacs (mainly under Windows @work):
>
> - Capturing screenshots and drawing stuff over them.  This is just an
>   observation, I'm not asking to convert Emacs to the Gimp!
Have you tried "ScreenShot":https://www.emacswiki.org/emacs/ScreenShot

I don't take many screenshots myself, so I didn't try it myself, just
remember it was there.

RE: What is the most useful potential feature which Emacs lacks?

[Drew Adams]

> - Starting executables from Dired under Windows (for example with S-
>   RET, OK, there's this package dired-launch).

Just use `!'.
Customize option `dired-guess-shell-alist-user' so
it associates the executable you want with the file
extension you want.  When you hit `!' on a file with
that extension you get the program you want as the
default - just hit `RET'.

You need (standard) library `dired-x.el' for this.

> - Some Dictionary access, in the vain of ispell, when reading, writing
>   and browsing.

This wiki page has lots of such (search for "dict"):

https://www.emacswiki.org/emacs/CategoryInterface

And this wiki page has thesauri (synonyms) libraries:

https://www.emacswiki.org/emacs/ThesauriAndSynonyms

Re: What is the most useful potential feature which Emacs lacks?

[Helmut Eller]

On Mon, May 11 2020, ndame wrote:

> There is a discussion on Reddit about sponsoring development of
> multithreading in Emacs, and people there say it's too hard, takes a
> lot of time and it doesn't even bring that much benefit to the user.
>
> If this is the case (is it?) then what are those other features which
> could bring much more tangible benefits for the user and assuming
> somebody works on them full time sponsored by the community they can
> be implemented in, say,  a few months?

1. Improving display of long lines.  Emacs gets very unresponsive when a
buffer contains long lines.  This is something I run into rather
frequently and it's very irritating.  Apparently this is a difficult
problem to fix because to the display code needs to "measure" how long
the line is in pixels and there is no other way to that than to iterate
over each character in a line.  But it may be possible reduce the number
of how often this measuring needs to done.  It seems to me that
web-browsers also need a long time to display long lines, but once the
line is drawn, scrolling works as quick as for short lines.

2. Being able to display HTML/CSS/SVG the way mainstream web-browsers
display would be nice.  But probably too big a project.  And in the long
run, probably a reason for Emacs to go extinct.

Helmut

Re: What is the most useful potential feature which Emacs lacks?

[Jose A. Ortega Ruiz]

On Tue, May 12 2020, Helmut Eller wrote:

[...]

> 1. Improving display of long lines.  Emacs gets very unresponsive when a
> buffer contains long lines.  This is something I run into rather
> frequently and it's very irritating.  Apparently this is a difficult
> problem to fix because to the display code needs to "measure" how long
> the line is in pixels and there is no other way to that than to iterate
> over each character in a line.  But it may be possible reduce the number
> of how often this measuring needs to done.  It seems to me that
> web-browsers also need a long time to display long lines, but once the
> line is drawn, scrolling works as quick as for short lines.
>
> 2. Being able to display HTML/CSS/SVG the way mainstream web-browsers
> display would be nice.  But probably too big a project.  And in the long
> run, probably a reason for Emacs to go extinct.

Just wanted to say, as someone who uses Emacs for literally everthing
except visiting a few web pages, that these two projects are exactly the
ones i'd need more.  In case someone's counting :)

Cheers,
jao, who wishes he had time to hack on this.
-- 
Experience is not what happens to you; it is what you do with what
happens to you.
 - Aldous Huxley

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Monnier]

> Just wanted to say, as someone who uses Emacs for literally everthing
> except visiting a few web pages, that these two projects are exactly the
> ones i'd need more.  In case someone's counting :)

For the "browser" part, have you tried the `xwidget` approach to the problem?
How 'bout EAF (emacs-application-framework)?


        Stefan

Re: What is the most useful potential feature which Emacs lacks?

[jao]

On Tue, May 12 2020, Stefan Monnier wrote:

>> Just wanted to say, as someone who uses Emacs for literally everthing
>> except visiting a few web pages, that these two projects are exactly the
>> ones i'd need more.  In case someone's counting :)
>
> For the "browser" part, have you tried the `xwidget` approach to the
> problem?

A few years ago: it was then in a very early stage and tended to kill
emacs.  And after a while i got the impression that it got out of steam
and further development stopped: but maybe that's not so?

> How 'bout EAF (emacs-application-framework)?

I haven't used it (yet), but its problem (IIUC) in my case is that it
wants to be my window manager (i'm a very happy user of exwm), pdf
viewer (ditto with pdf-tools), and so on, so it feels a bit too invasive
(i guess i also don't like the looks of qt apps, but that's just
prejudice).  Again, i might be wrong and the EAF be modular enough for
using only its browser.

I guess what i'd really like about a fully integrated browser would be
its being leaner than the webkit or quantum behemoths and fully
scriptable in elisp: if all we have is a widget which essentially embeds
one of those browsers and i have to access it using python, well,
there's not much difference with having a firefox process running in an
exwm workspace, as i do now, together with something like trydactil.

Something a bit better with graphical layout than emacs-w3m, which i
quite like already, would be almost enough for me, really.  But this is
most probably not a very reasonable expectation: people will want all
the HTML5 bells and wishes, i suppose...

Cheers,
jao
-- 
Omnia disce, videbus postea nihil esse superfluum - Hugh of St Victor

Re: What is the most useful potential feature which Emacs lacks?

[João Távora]

On Tue, May 12, 2020 at 1:30 PM Helmut Eller <eller.helmut@gmail.com> wrote:
> 2. Being able to display HTML/CSS/SVG the way mainstream web-browsers
> display would be nice.  But probably too big a project.  And in the long
> run, probably a reason for Emacs to go extinct.

That translates to "either Emacs becomes a Elisp-programmable
web-browser or goes extinct".  Indeed hard, but not impossible .

And Elisp can teach JS a thing or two. And what if Emacs could run JS?

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Kangas]

João Távora <joaotavora@gmail.com> writes:

> That translates to "either Emacs becomes a Elisp-programmable
> web-browser or goes extinct".  Indeed hard, but not impossible .

HTML rendering was mentioned in this talk at Emacsconf 2019 by
Perry E. Metzger:

   https://media.emacsconf.org/2019/26.html

   (Discussion on HTML starts around 00:30:00.)

He thinks it would have to be based on an external web rendering
toolkit (like WebKit).

Best regards,
Stefan Kangas

Re: What is the most useful potential feature which Emacs lacks?

[Helmut Eller]

On Wed, May 13 2020, Stefan Kangas wrote:

> João Távora <joaotavora@gmail.com> writes:
>
>> That translates to "either Emacs becomes a Elisp-programmable
>> web-browser or goes extinct".  Indeed hard, but not impossible .
>
> HTML rendering was mentioned in this talk at Emacsconf 2019 by
> Perry E. Metzger:
>
>    https://media.emacsconf.org/2019/26.html
>
>    (Discussion on HTML starts around 00:30:00.)
>
> He thinks it would have to be based on an external web rendering
> toolkit (like WebKit).

Or maybe Emacs could run inside a browser via WebAssembly or something
like that.

Helmut

Re: What is the most useful potential feature which Emacs lacks?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 250 bytes --]

On Thu, May 14, 2020 at 12:08:46PM +0200, Helmut Eller wrote:

[...]

> Or maybe Emacs could run inside a browser via WebAssembly or something
> like that.

I guess you're saying it tongue-in-cheek. But still I shudder at this
dystopia.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Thu, 14 May 2020 12:17:41 +0200, <tomas@tuxteam.de> said:

    nil> On Thu, May 14, 2020 at 12:08:46PM +0200, Helmut Eller wrote:
    nil> [...]

    >> Or maybe Emacs could run inside a browser via WebAssembly or something
    >> like that.

    nil> I guess you're saying it tongue-in-cheek. But still I shudder at this
    nil> dystopia.

Youʼre aware of <https://repl.it>, right?

Robert

Re: What is the most useful potential feature which Emacs lacks?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Thu, May 14, 2020 at 12:34:50PM +0200, Robert Pluim wrote:
> >>>>> On Thu, 14 May 2020 12:17:41 +0200, <tomas@tuxteam.de> said:
> 
>     nil> On Thu, May 14, 2020 at 12:08:46PM +0200, Helmut Eller wrote:
>     nil> [...]
> 
>     >> Or maybe Emacs could run inside a browser via WebAssembly or something
>     >> like that.
> 
>     nil> I guess you're saying it tongue-in-cheek. But still I shudder at this
>     nil> dystopia.
> 
> Youʼre aware of <https://repl.it>, right?

I'm not. I don't even want to follow that link -- I know it'll make me sad.
Don't get me wrong. I'm well aware of what WebAssembly /technically/ is, and
what's possible with it (including Spectre exploits).

It's rather watching the browser (the one piece of software in my environment
I most detest and mistrust) evolve into the indispensable operating system.

It's watching how a piece of formally free software becomes unfree because
of crushing complexity and decommoditiation of protocols

For one time I'm glad to be rather old and soon out of this game.

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We have a program, LibreJS, which blocks all webassebly code
as well as nonfree Javascript code.  We should try to teach people
NOT to run any webassebly.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Dmitry Gutov]

On 15.05.2020 06:25, Richard Stallman wrote:
> We have a program, LibreJS, which blocks all webassebly code
> as well as nonfree Javascript code.  We should try to teach people
> NOT to run any webassebly.

You're aware that webassembly doesn't always mean "proprietary", right?

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We have a program, LibreJS, which blocks all webassebly code
as well sa nonfree Javascript code.  We should try to teach people
NOT to run any webassebly.

I suspect that repl.it would be blocked by LibreJS.
Would someone like to tell me a 10-line summary of what it does?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Dmitry Gutov]

On 15.05.2020 06:25, Richard Stallman wrote:
> I suspect that repl.it would be blocked by LibreJS.

Maybe, maybe not. The landing page is mostly static HTML, with some 
embedded pictures and videos.

> Would someone like to tell me a 10-line summary of what it does?

A couple of marketing blurbs:

   Repl.it is a simple yet powerful online IDE, Editor, Compiler, 
Interpreter, and REPL. Code, compile, run, and host in 50+ programming 
languages...

   Code together, right from your browser. With Multiplayer, you can 
write, review and debug together, in real time. Share your entire Repl 
projects, or live Repl Embeds with the community.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    Code together, right from your browser. With Multiplayer, you can 
  > write, review and debug together, in real time. Share your entire Repl 
  > projects, or live Repl Embeds with the community.

Does this manner of operation involve using a server?  If so, whose server?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Dmitry Gutov]

On 16.05.2020 07:18, Richard Stallman wrote:
>    >    Code together, right from your browser. With Multiplayer, you can
>    > write, review and debug together, in real time. Share your entire Repl
>    > projects, or live Repl Embeds with the community.
> 
> Does this manner of operation involve using a server?  If so, whose server?

Quoting their docs again:

   Q. How does repl.it work?

   For most languages, code is sent to our servers where it will be
   executed in a secure sandbox. The result is then sent back and
   rendered on the website. However, some of our interpreters can run
   directly in your browser without going to the server.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > Q. How does repl.it work?

     > For most languages, code is sent to our servers where it will be
     > executed in a secure sandbox. The result is then sent back and
     > rendered on the website. However, some of our interpreters can run
     > directly in your browser without going to the server.

Thanks.  Now I see that repl.it operates in two ways, both of which
people ought to avoid on general principles.

So Emacs should not talk about repl.it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > That translates to "either Emacs becomes a Elisp-programmable
  > > web-browser or goes extinct".

HTML is a peripheral application area for Emacs.  We should not exaggerate
its importance.

When I want to look at HTML in an email, I use an external renderer:
lynx.

The built-in Emacs renderer gives ok results when the links don't matter.
For links, the way it tries to follow them is useless since it doesn't
go through Tor.  Anyway, I'd rather send an email to fetch the page contents.
The lynx output makes that easy to do.

The built-in Emacs renderer often takes a painfully long time.  I wish
I could easily tell it to give up without trying if the text is above
a certain size or complexity.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 14 May 2020 23:17:08 -0400
> Cc: eller.helmut@gmail.com, joaotavora@gmail.com, emacs-devel@gnu.org
> 
> The built-in Emacs renderer often takes a painfully long time.

Please try customizing shr-use-fonts to nil, and see if this makes
things less painful for you.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Please try customizing shr-use-fonts to nil, and see if this makes
  > things less painful for you.

Thanks.

I never invoke the HTML renderer explicitly.  It gets done automatically
by MIME display.  As a result, I don't know what it is called, so I don't
know where to look for customizations.

Would it be possible to change MIME display to show users, somehow,
what package it is using to render HTML?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: stefankangas@gmail.com, eller.helmut@gmail.com,
> 	joaotavora@gmail.com, emacs-devel@gnu.org
> Date: Sat, 16 May 2020 00:18:43 -0400
> 
>   > Please try customizing shr-use-fonts to nil, and see if this makes
>   > things less painful for you.
> 
> Thanks.
> 
> I never invoke the HTML renderer explicitly.  It gets done automatically
> by MIME display.  As a result, I don't know what it is called, so I don't
> know where to look for customizations.

Emacs has only one built-in HTML renderer, it is called shr.el.

> Would it be possible to change MIME display to show users, somehow,
> what package it is using to render HTML?

Ideas for how to do this are welcome.  Do we have any machinery in
Emacs to show similar information in other similar conditions?  For
example, visiting a file of some type automatically shows that file in
some format.  The way we should this is via the major mode name in the
mode line.  So perhaps we should mentions shr in the mode line as well
in these cases, as minor mode or somesuch?

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Monnier]

> mode line.  So perhaps we should mentions shr in the mode line as well
> in these cases, as minor mode or somesuch?

I'm not sure this info is important enough to warrant displaying it in
the mode-line, but `describe-mode` would be a "natural" place, indeed.


        Stefan

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Emacs has only one built-in HTML renderer, it is called shr.el.

What is the relation of eww to shr?

  > Ideas for how to do this are welcome.  Do we have any machinery in
  > Emacs to show similar information in other similar conditions? 

Perhaps display a header line saying

  Rendered by shr mode

and if you click on that you get the mode help for shr mode

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Andreas Schwab]

On Mai 16 2020, Richard Stallman wrote:

> What is the relation of eww to shr?

eww is a front end for shr, other users are rmail and gnus.  shr only
handles the rendering.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Why use the name shr for an HTML renderer?  I'd never guess that name,
and from the name I would never guess what it does.

How about renaming it to HTML Render?

A rendered HTML segment in a message can't really have its own major mode,
but if it sets up a text property keymap to get the commands of that mode,
it would be useful to offer a way to display the doc string of that mode.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> cc: emacs-devel@gnu.org
> Date: Sun, 17 May 2020 23:42:32 -0400
> 
> Why use the name shr for an HTML renderer?

As the file's header says, "shr" stands for "Simple HTML Renderer".

shr.el rename?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > As the file's header says, "shr" stands for "Simple HTML Renderer".

I think the name html-render.el would be much better.
Just seeing that in a directory listing, I would know what it does
and I'd remember the name.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: shr.el rename?

[Drew Adams]

> > As the file's header says, "shr" stands for "Simple HTML Renderer".
> 
> I think the name html-render.el would be much better.
> Just seeing that in a directory listing, I would know what it does
> and I'd remember the name.

+1.  (But it may be considered too late...)

In general, IMO it's fine for a library to have
a short prefix for the names it uses (at least
2 chars + hyphen).  But the prefix shouldn't be
confused with the library name.

That is, there's no reason a library name needs
to be so short that it becomes obscure.  I'm
guessing that's what happens sometimes: the
library and prefix are the same (for whatever
reason).

I've even noticed in one of the recent threads
that a discussion about too-short prefixes (e.g.
`s-', not to mention just `-') at some places
morphed temporarily into a discussion of the
corresponding library name, `s.el'.

Re: shr.el rename?

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > As the file's header says, "shr" stands for "Simple HTML Renderer".
>
> I think the name html-render.el would be much better.
> Just seeing that in a directory listing, I would know what it does
> and I'd remember the name.

Alternatively, you could just learn what "shr" means -- I think that'd
be less disruptive than changing the name of an eleven year old library
that's used all over the place.

Don't you?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: shr.el rename?

[Stefan Monnier]

>>   > As the file's header says, "shr" stands for "Simple HTML Renderer".
>> I think the name html-render.el would be much better.

Until someone wants to write a less-simple HTML renderer and ends up
having to call it `new-html-render.el`?


        Stefan

Re: shr.el rename?

[T.V Raman]

More interestingly, it should be possible in dired to show a quick
preview of .el files by popping up the first line in another temporary
buffer as you navigate through the dired buffer:-)
-- 

Re: shr.el rename?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Until someone wants to write a less-simple HTML renderer and ends up
  > having to call it `new-html-render.el`?

With a little imagination, perse could think of a name that refers
to some advantage or improvement in that renderer.  There will surely be one;
if we want to install that renderer, it will be for some reason.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: shr.el rename?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think the name html-render.el would be much better.
  > > Just seeing that in a directory listing, I would know what it does
  > > and I'd remember the name.

  > Alternatively, you could just learn what "shr" means -- I think that'd

I'm sorry if my words made it appear that I was concerned only with
whether I personally would remember the non-suggestive name "shr".
Renaming the package to a name that visibly fits what the package does
would be a cleanup that would make its meaning more obvious to
everyone.

Could we start renaming packages to meaningful, clear names when we
add them to Emacs or GNU ELPA?  We used to keep names short for MSDOS,
but that isn't necessary any more.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Filipp Gunbin]

On 16/05/2020 00:18 -0400, Richard Stallman wrote:

> Would it be possible to change MIME display to show users, somehow,
> what package it is using to render HTML?

Gnus uses the variable "mm-text-html-renderer" (also shr by default) to
select HTML renderer.  Maybe this variable can be reused in other
places.

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,
>   eller.helmut@gmail.com,  stefankangas@gmail.com,  joaotavora@gmail.com
> Date: Mon, 18 May 2020 18:20:18 +0300
> 
> Gnus uses the variable "mm-text-html-renderer" (also shr by default) to
> select HTML renderer.  Maybe this variable can be reused in other
> places.

Not surprisingly, Rmail has rmail-mime-render-html-function for the
same purpose.

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Thu, 14 May 2020 23:17:08 -0400, Richard Stallman <rms@gnu.org> said:

    Richard> The built-in Emacs renderer gives ok results when the links don't matter.
    Richard> For links, the way it tries to follow them is useless since it doesn't
    Richard> go through Tor.
   
The url library used by eww/shr supports SOCKS, and Tor can be run as
a SOCKS proxy.

Robert

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Fri, 15 May 2020 11:10:22 +0200
> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com,
>  Stefan Kangas <stefankangas@gmail.com>, joaotavora@gmail.com
> 
>     Richard> The built-in Emacs renderer gives ok results when the links don't matter.
>     Richard> For links, the way it tries to follow them is useless since it doesn't
>     Richard> go through Tor.
>    
> The url library used by eww/shr supports SOCKS, and Tor can be run as
> a SOCKS proxy.

It's a matter of some coding, I guess.  Richard added a defcustom to
use Tor in VC (it's already in Emacs 27), and I guess something
similar can be done for URL.

Any volunteers?

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Fri, 15 May 2020 13:21:06 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Date: Fri, 15 May 2020 11:10:22 +0200
    >> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com,
    >> Stefan Kangas <stefankangas@gmail.com>, joaotavora@gmail.com
    >> 
    Richard> The built-in Emacs renderer gives ok results when the links don't matter.
    Richard> For links, the way it tries to follow them is useless since it doesn't
    Richard> go through Tor.
    >> 
    >> The url library used by eww/shr supports SOCKS, and Tor can be run as
    >> a SOCKS proxy.

    Eli> It's a matter of some coding, I guess.  Richard added a defcustom to
    Eli> use Tor in VC (it's already in Emacs 27), and I guess something
    Eli> similar can be done for URL.

I donʼt think any coding is required:

You can point the url library at localhost:9050 (or whatever the Tor
default SOCKS port is) using:
    
    socks-server is a variable defined in `socks.el'.
    Its value is ("Default server" "socks" 1080 5)

and tell url to use SOCKS with:

    url-gateway-method is a variable defined in `url-vars.el'.
    Its value is `native'

      You can customize this variable.

    Documentation:
    The type of gateway support to use.
    Should be a symbol specifying how to get a connection from the local machine.

    Currently supported methods:
    `telnet': Run telnet in a subprocess to connect;
    `rlogin': Rlogin to another machine to connect;
    `socks': Connect through a socks server;
    `tls': Connect with TLS;
    `ssl': Connect with SSL (deprecated, use `tls' instead);
    `native': Connect directly.

unless youʼre talking about getting url to use 'torsocks' as a gateway
method, which would require some coding.

Robert

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: rms@gnu.org,  emacs-devel@gnu.org,  eller.helmut@gmail.com,
>   stefankangas@gmail.com,  joaotavora@gmail.com
> Date: Fri, 15 May 2020 13:07:08 +0200
> 
> unless youʼre talking about getting url to use 'torsocks' as a gateway
> method, which would require some coding.

I think I did mean 'torsocks', but I'm way out of my depth here, so
feel free to ignore me.

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Fri, 15 May 2020 14:28:50 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: rms@gnu.org,  emacs-devel@gnu.org,  eller.helmut@gmail.com,
    >> stefankangas@gmail.com,  joaotavora@gmail.com
    >> Date: Fri, 15 May 2020 13:07:08 +0200
    >> 
    >> unless youʼre talking about getting url to use 'torsocks' as a gateway
    >> method, which would require some coding.

    Eli> I think I did mean 'torsocks', but I'm way out of my depth here, so
    Eli> feel free to ignore me.

'torsocks' just does what I described up-thread: send network traffic
to the tor instance running on localhost:9050, which can already be
done using the emacs variables.

Robert

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Fri, 15 May 2020 13:49:23 +0200
> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com, stefankangas@gmail.com,
>  rms@gnu.org, joaotavora@gmail.com
> 
>     Eli> I think I did mean 'torsocks', but I'm way out of my depth here, so
>     Eli> feel free to ignore me.
> 
> 'torsocks' just does what I described up-thread: send network traffic
> to the tor instance running on localhost:9050, which can already be
> done using the emacs variables.

Is that easy enough for users?  Maybe we should have a defcustom that
would set that up?

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Fri, 15 May 2020 14:58:43 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Date: Fri, 15 May 2020 13:49:23 +0200
    >> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com, stefankangas@gmail.com,
    >> rms@gnu.org, joaotavora@gmail.com
    >> 
    Eli> I think I did mean 'torsocks', but I'm way out of my depth here, so
    Eli> feel free to ignore me.
    >> 
    >> 'torsocks' just does what I described up-thread: send network traffic
    >> to the tor instance running on localhost:9050, which can already be
    >> done using the emacs variables.

    Eli> Is that easy enough for users?  Maybe we should have a defcustom that
    Eli> would set that up?

To replace the two that we have already? People who want to do this
tend to be very technically capable, I donʼt see the need.

Robert

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org,  eller.helmut@gmail.com,  stefankangas@gmail.com,
>   rms@gnu.org,  joaotavora@gmail.com
> Date: Fri, 15 May 2020 14:14:02 +0200
> 
>     Eli> Is that easy enough for users?  Maybe we should have a defcustom that
>     Eli> would set that up?
> 
> To replace the two that we have already?

No, to arrange those two to be set to those particular values.

> People who want to do this tend to be very technically capable, I
> donʼt see the need.

Let's see what Richard has to say about this, he is the user who'd
need to make these customizations for his use case.

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Monnier]

> To replace the two that we have already? People who want to do this
> tend to be very technically capable, I donʼt see the need.

There's an argument to be made that "we" should encourage the use of Tor
(tho this "we" isn't specifically "we, the Free Software community" but it's
a community with a non-negligible overlap).


        Stefan

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Fri, 15 May 2020 11:22:23 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:

    >> To replace the two that we have already? People who want to do this
    >> tend to be very technically capable, I donʼt see the need.

    Stefan> There's an argument to be made that "we" should encourage the use of Tor
    Stefan> (tho this "we" isn't specifically "we, the Free Software community" but it's
    Stefan> a community with a non-negligible overlap).

Thatʼs a separate argument, but it pleads more for us to document how
to do this than to write code.

Robert

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > unless youʼre talking about getting url to use 'torsocks' as a gateway
  > method, which would require some coding.

If it supports 'socks' as a gateway method, supporting 'tor' as a
gateway method should be easy -- just translate it into 'socks' with the
suitable socks port.
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >     Richard> The built-in Emacs renderer gives ok results when the links don't matter.
  >     Richard> For links, the way it tries to follow them is useless since it doesn't
  >     Richard> go through Tor.
   
  > The url library used by eww/shr supports SOCKS, and Tor can be run as
  > a SOCKS proxy.

The issue is even more complex.  There are TWO ways I visit web pages
from Emacs, neither of which is eww.  I want to be able to choose
which way to use each time.

Doing that manually is easy enough -- if I can SEE the URL in the
buffer.  So I use lynx -dump to render the HTML, and it shows the URLs
of the links at the end.  Then I can do what I want to do.

The built-in Emacs renderer (is that eww?) doesn't show me the URLs,
which means I have no way to follow those links in either of the ways
I want to use.  Therefore, I need to hide the eww rendering and render
with lynx.

I would appreciate being able to specify "use lynx -dump to render HTML
from my incoming emails."  lynx -dump instead of eww that is.

Also helpful would be a way to customize how to follow a link in eww
rendering, which would let me write Lisp code to do one or the other
of the things I want to do.  Then maybe I would like eww as much as
lynx -dump.

But I would also want to be able to grab the URL into the kill buffer.
Could eww include those URLs in the rendered text?

I would guess lynx -dump is faster too, but I don't know for
certain.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Andreas Schwab]

On Mai 16 2020, Richard Stallman wrote:

> The built-in Emacs renderer (is that eww?) doesn't show me the URLs,

Yes, it does.  If you tab to an anchor, the URL is shown in the echo
area.  You can even edit the URL before following.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: What is the most useful potential feature which Emacs lacks?

[Yuri Khan]

On Sat, 16 May 2020 at 13:51, Andreas Schwab <schwab@linux-m68k.org> wrote:
> > The built-in Emacs renderer (is that eww?) doesn't show me the URLs,
>
> Yes, it does.  If you tab to an anchor, the URL is shown in the echo
> area.  You can even edit the URL before following.

Moreover, when point is on a link, pressing ‘u’ or ‘w’ in eww
(shr-maybe-probe-and-copy-url) will copy the URL to the kill ring.
Also, if you know that URL to be a redirect, pressing ‘u’ or ‘w’ again
will resolve it and copy the resolved destination URL.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, it does.  If you tab to an anchor, the URL is shown in the echo
  > area.  You can even edit the URL before following.

I never had any idea of that.  Perhaps I never tried using TAB in those
buffers.  Thanks.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Dmitry Gutov]

On 12.05.2020 15:30, Helmut Eller wrote:
> 2. Being able to display HTML/CSS/SVG the way mainstream web-browsers
> display would be nice.  But probably too big a project.  And in the long
> run, probably a reason for Emacs to go extinct.

Extinct, or just "not fully suitable as an OS replacement"? :)

Re: What is the most useful potential feature which Emacs lacks?

[Christopher Lemmer Webber]

ndame writes:

> There is a discussion on Reddit about sponsoring development of
> multithreading in Emacs, and people there say it's too hard, takes a
> lot of time and it doesn't even bring that much benefit to the user.
>
> If this is the case (is it?) then what are those other features which
> could bring much more tangible benefits for the user and assuming
> somebody works on them full time sponsored by the community they can
> be implemented in, say, a few months?

I guess there are really two potential directions to think about it:

 - What is the most potentially useful direction for newcomers who are
   familiar with other mainstream UI patterns?
 - What is the most potentially useful project for existing everyday
   emacs users?

I think these are two separate things to solve.  IMO the former is more
important than the latter right now because Emacs has a way of making
users capable of extending it... it is one of the most beautiful things
about the choice of lisp as its configuration system.

So the real question to me would be: how do we lower the barrier to
entry?  There's been a lot of discussion on this, and I think the
most promising direction to me so far has been seeing the wild success
of Spacemacs as a "starter pack" that comes preconfigured in a way that
it feels familiar to Vim users.

These days most programmers start off in neither Vi(m) nor Emacs, they
start out with UI patterns that solidified after those programs were
made.  My suspicion is that a Spacemacs-like "starter pack" would solve
a lot of this, but it's work that's hard to motivate... once most
developers are far enough down the Emacs rabbit hole, doing this work
really doesn't do much to scratch their own itch.

Thus if there's a space for paid work, I think it would be to do this
work which might not be as directly useful to the developer, but would
be directly useful to other newcomers.  It would be useful and important
IMO to directly test against newcomers' experiences and collect feedback.

(The more difficult question to me would be: do the training wheels ever
come off the bike?  Or is it just a "different UI pattern" at that point
that one comes to accept and use emacs in that way?)

Re: What is the most useful potential feature which Emacs lacks?

[Karl Fogel]

On 12 May 2020, Christopher Lemmer Webber wrote:
>ndame writes:
>
>> There is a discussion on Reddit about sponsoring development of
>> multithreading in Emacs, and people there say it's too hard, takes a
>> lot of time and it doesn't even bring that much benefit to the user.
>>
>> If this is the case (is it?) then what are those other features which
>> could bring much more tangible benefits for the user and assuming
>> somebody works on them full time sponsored by the community they can
>> be implemented in, say, a few months?
>
>I guess there are really two potential directions to think about it:
>
> - What is the most potentially useful direction for newcomers who are
>   familiar with other mainstream UI patterns?
> - What is the most potentially useful project for existing everyday
>   emacs users?
>
>I think these are two separate things to solve.  IMO the former is more
>important than the latter right now because Emacs has a way of making
>users capable of extending it... it is one of the most beautiful things
>about the choice of lisp as its configuration system.
>
>So the real question to me would be: how do we lower the barrier to
>entry?  [...]
>
>Thus if there's a space for paid work, I think it would be to do this
>work which might not be as directly useful to the developer, but would
>be directly useful to other newcomers.  It would be useful and important
>IMO to directly test against newcomers' experiences and collect feedback.

FWIW -- and perhaps to your surprise -- I would argue that this is *not* the important question for Emacs.

I've explained why in another thread:

  https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01855.html

  From: Karl Fogel <kfogel@red-bean.com>
  To: Emacs Devel
  Subject: Re: GNU Emacs raison d'etre
  Date: Wed, 13 May 2020 11:18:50 -0500
  Message-ID: <871rnnvmdx.fsf@red-bean.com>

Best regards,
-Karl

Re: What is the most useful potential feature which Emacs lacks?

[Christopher Lemmer Webber]

Karl Fogel writes:

> On 12 May 2020, Christopher Lemmer Webber wrote:
>
>>Thus if there's a space for paid work, I think it would be to do this
>>work which might not be as directly useful to the developer, but would
>>be directly useful to other newcomers.  It would be useful and important
>>IMO to directly test against newcomers' experiences and collect feedback.
>
> FWIW -- and perhaps to your surprise -- I would argue that this is *not* the important question for Emacs.
>
> I've explained why in another thread:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01855.html

Hi Karl, nice to hear you reply to me :)

I agree with a lot of your assessments, that the highest value in Emacs
is the willingness to invest in it.  A well customized emacs conforms to
the shape of one's body and mind.  And I agree then that this is a
selling point to advertise.

I still don't think that's in contradiction to the "conventional editor
starter pack" goal though.  I know people who are tantalized by the
*idea* of learning Emacs, but get an enormous amount of imposter
syndrome and feeling of being overwhelmed when dipping their toes in the
water... some have tried dipping their toes in the water a few times.

Me personally, once I decided to learn emacs I just jumped straight into
the deep end.  But that's not for everyone.  Sometimes it can be nice to
have a wading area in the swimming pool for some folks.

That's my feeling, anyway.
 - Chris

Re: What is the most useful potential feature which Emacs lacks?

[Karl Fogel]

On 13 May 2020, Christopher Lemmer Webber wrote:
>Hi Karl, nice to hear you reply to me :)

I don't (can't) read every post to this list, but I always read yours :-).

>I agree with a lot of your assessments, that the highest value in Emacs
>is the willingness to invest in it.  A well customized emacs conforms to
>the shape of one's body and mind.  And I agree then that this is a
>selling point to advertise.
>
>I still don't think that's in contradiction to the "conventional editor
>starter pack" goal though.  I know people who are tantalized by the
>*idea* of learning Emacs, but get an enormous amount of imposter
>syndrome and feeling of being overwhelmed when dipping their toes in the
>water... some have tried dipping their toes in the water a few times.
>
>Me personally, once I decided to learn emacs I just jumped straight into
>the deep end.  But that's not for everyone.  Sometimes it can be nice to
>have a wading area in the swimming pool for some folks.

I believe those goals *are* somewhat in tension, though.  I've taught Emacs to a fair number of people, sometimes successfully and sometimes not.  One thing that I recall every newcomer experiencing is, at least initially, the feeling that Emacs was constantly biting them -- constantly surprising them with unexpected and confusing behaviors that jump out from accidental keystrokes.  Two of the first things I always have to teach newcomers are `C-g' and `C-h l' :-).

Here's a concrete example I've seen over and over:

User does `C-x C-f' to find a file, but they hit Return at the wrong moment while typing the file path, causing a Dired buffer comes up visiting the file's directory.  The user is, of course, totally baffled by this result.  And yet it's obvious why this is a good default behavior for `find-file' -- for people who understand what's going on.

If the proposed starter pack is going to mitigate effects like that for newcomers, it can only do so by making the keybound functionality space sparser -- which of course then lowers the reward-for-investment rate as the user gains expertise.  How do you propose solving that?  Do we make an explicit "I'm ready to leave newcomer mode now" command?  But that requires the user to make a guess about the moment of their graduation from newcomer to non-newcomer -- and this moment is mythical, since the learning is a continuous process with no discrete boundary.

Changes that make Emacs better for newcomers *without* reducing the reward-for-investment rate are great, and I'm in favor of them like everyone else is.  No one would object to them; therefore they are not the subject of any disagreement.  I do think those changes are harder to find than you think they are.

Really I'm just suggesting a general framework for thinking about what Emacs's goals should be.  If we just say "Emacs should be easier for newcomers to learn", that's not a useful rallying cry IMHO.  If we say instead "Emacs should try to attract newcomers who have a higher-than-average probability of becoming high-investment users, and should explain early on to those newcomers what the road ahead looks like", *then* we have a high-level guiding principle we can actually use.

Best regards,
-Karl

Re: What is the most useful potential feature which Emacs lacks?

[Robert Pluim]

>>>>> On Wed, 13 May 2020 23:08:04 -0500, Karl Fogel <kfogel@red-bean.com> said:
    Karl> Here's a concrete example I've seen over and over:

    Karl> User does `C-x C-f' to find a file, but they hit Return at the wrong
    Karl> moment while typing the file path, causing a Dired buffer comes up
    Karl> visiting the file's directory.  The user is, of course, totally
    Karl> baffled by this result.  And yet it's obvious why this is a good
    Karl> default behavior for `find-file' -- for people who understand what's
    Karl> going on.

Are you proposing "you appear to be visiting a directory, would you
like to enable dired? [yes/no/once]"? That would be massively
annoying, but only once per feature, so I could live with it. Or we
could add an 'emacs-expert-user' variable (or should that be
'expert-emacs-user'? :-) ).

    Karl> If the proposed starter pack is going to mitigate effects like that
    Karl> for newcomers, it can only do so by making the keybound functionality
    Karl> space sparser -- which of course then lowers the reward-for-investment
    Karl> rate as the user gains expertise.  How do you propose solving that?
    Karl> Do we make an explicit "I'm ready to leave newcomer mode now" command?
    Karl> But that requires the user to make a guess about the moment of their
    Karl> graduation from newcomer to non-newcomer -- and this moment is
    Karl> mythical, since the learning is a continuous process with no discrete
    Karl> boundary.

disabled commands already offer this, no?

Robert

Re: What is the most useful potential feature which Emacs lacks?

[Christopher Lemmer Webber]

Karl Fogel writes:

> If the proposed starter pack is going to mitigate effects like that
> for newcomers, it can only do so by making the keybound functionality
> space sparser -- which of course then lowers the reward-for-investment
> rate as the user gains expertise.  How do you propose solving that?
> Do we make an explicit "I'm ready to leave newcomer mode now" command?
> But that requires the user to make a guess about the moment of their
> graduation from newcomer to non-newcomer -- and this moment is
> mythical, since the learning is a continuous process with no discrete
> boundary.

I've talked about this with a friend who is a prominent Blender user;
the program Blender has a similar issue where it has a perceived amount
of UI complexity; a lot of it also because things were foreign.  (It
also has not quite as much but a similar-ish level of extensibility.)

Users for years complained that the program is inaccessible.  Apparently
they did go through many UI improvements and eventually the UI did get
much easier to use (last time I opened it up though, since my experience
was from much older versions, *I* was confused... eg they switched the
order of right and left mouse buttons to be what other programs do).

For years, many of the suggestions for improving it though were around
changing or removing features.  It seems that recently they have made
many changes and managed to not remove features... I wonder how they did
it?  New users seem to be increasingly happy with the changes.  But of
course, some still complain that it's "too complicated", but a certain
amount of this is a domain problem: Blender chooses to be a powerful 3d
tool, and to a certain degree there will always be a certain amount of
overwhelmingness to it that's unavoidable.  Notably, this is also true
with the command line, which is also something that many have to become
convinced to become unafraid of.

Here are two lessons that I've thought about since those Blender
conversations though:

 - One thought is: you can change the defaults to be not what the
   current group is familiar with, but what matches the
   mainstream... and do compatibility in the *opposite* direction.  What
   if instead of turning on cua-mode, you turn on legacy-mode?

   I know this won't be a popular choice here, but it's worth proposing.
   When I opened Blender again, I reversed the mouse buttons because it
   was more "what I was used to from the old days" since Blender (like
   Emacs) made some convention decisions before there was a "mainstream
   choice".  But I could set it back.

   Notably this happened in a small way in emacs already: the scrollbars
   moved from the left to the right by default.  Well!  I rebound them
   back.  But I'm glad they're in the right by default now: that's the
   right decision.

   Still, this is maybe a harder sell for eg many of Emacs' keybindings,
   and the history list is very long on things like "kill"... I suspect
   the current community would be up in arms too much to accept such
   change.  But it's worth proposing the complete alternative.

   Let's not forget: some ideas in emacs are the way they are because
   they're more powerful.  Some of them are the way they are because
   Emacs predates modern UIs.  Maybe the result for what we choose to
   (not) do is the same.  But we shouldn't forget that these are
   (sometimes, not always) two different things.

 - An alternate line of thinking mentioned by my friend, frustrated by
   the calls to "streamline Blender", was to bring up a talk or paper or
   something they heard about Super Mario Bros being an excellent first
   UI: the game has a significant amount of complexity, but that
   complexity is *gradually introduced to the player*, and in an
   intuitive way.

   I don't know what talk or paper or article they were talking about,
   but this article has some degree of summary:

     https://medium.com/swlh/the-perfect-game-tutorial-analyzing-super-marios-level-design-92f08c28bdf7

   You enter the world, you find out you can move to the right.  But you
   keep moving to the right, you run into your first enemy and die.

   You discover jump.  You try to jump, you accidentally hit this block
   above you, it does something interesting.

   You get far enough along in the level, you've absorbed those patterns
   already... so now you get to encounter new challenges.

   Eventually new game elements being introduced becomes more rare, and
   instead you reach the level of combinatorial introduction of elements
   together.

What I like about the latter approach is it doesn't suggest needing to
switch between a binary transition point of "beginner mode" and "expert
mode".  Instead, you have a gradual "just in time" introduction of
complexity.

Now, how to map that to Emacs?  I'm not sure yet... but maybe it's a
perspective worth starting from that allows for easing in newcomers
without throwing out expressive power.

Or maybe I just play too many video games, and that's why I appreciate
the analogy. :)

 - Chris

Re: What is the most useful potential feature which Emacs lacks?

[Dmitry Gutov]

On 14.05.2020 19:35, Christopher Lemmer Webber wrote:
>   - An alternate line of thinking mentioned by my friend, frustrated by
>     the calls to "streamline Blender", was to bring up a talk or paper or
>     something they heard about Super Mario Bros being an excellent first
>     UI: the game has a significant amount of complexity, but that
>     complexity is*gradually introduced to the player*, and in an
>     intuitive way.

I don't know if we can honestly say that Super Mario Bros is a complex 
game, but the principle can probably be applied everywhere to some degree.

For Emacs, it could probably be used to create some tutorial with 
gradually ramping up the complexity of tasks.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For Emacs, it could probably be used to create some tutorial with 
  > gradually ramping up the complexity of tasks.

If someone gets an inspiration for this, please give it a try.

Perhaps a game in which you try to correct a text
faster than a robotic ooponent can mess it up?
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > User does `C-x C-f' to find a file, but they hit Return at the
  > wrong moment while typing the file path, causing a Dired buffer
  > comes up visiting the file's directory.  The user is, of course,
  > totally baffled by this result.  And yet it's obvious why this is
  > a good default behavior for `find-file' -- for people who
  > understand what's going on.

This is a useful observation.  How can we arrange to inform users
about this at the right time?

One idea: the first time a user tries to specify a directory in find-file,
display a help screen to explain what that means, and describe a few
usual ways out.

Do you think this would ameliorate the problem?

Any other ideas?

    > If we just say "Emacs should be easier for newcomers to learn",
    > that's not a useful rallying cry IMHO.

I think it can have a practical benefit -- if it motivates people to
observe beginners' stumbling blocks as you have done, and then to study
how to help beginners cope with them.

What other frequent points of confusion have you observed?

It could be useful to make a list of those, and we could
try to work on each of them over time.

  > If we say instead "Emacs should try to attract newcomers who have
  > a higher-than-average probability of becoming high-investment
  > users, and should explain early on to those newcomers what the
  > road ahead looks like"

We should do that, but not by exaggerating.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[Karl Fogel]

On 14 May 2020, Richard Stallman wrote:
>  > User does `C-x C-f' to find a file, but they hit Return at the
>  > wrong moment while typing the file path, causing a Dired buffer
>  > comes up visiting the file's directory.  The user is, of course,
>  > totally baffled by this result.  And yet it's obvious why this is
>  > a good default behavior for `find-file' -- for people who
>  > understand what's going on.
>
>This is a useful observation.  How can we arrange to inform users
>about this at the right time?
>
>One idea: the first time a user tries to specify a directory in find-file,
>display a help screen to explain what that means, and describe a few
>usual ways out.
>
>Do you think this would ameliorate the problem?

My guess is it would ameliorate the problem for some users, and scare off other ones (ones who are less patient, or, put another way, who are less willing to invest time in reading an explanatory screen like that to figure out what just happened).

>    > If we just say "Emacs should be easier for newcomers to learn",
>    > that's not a useful rallying cry IMHO.
>
>I think it can have a practical benefit -- if it motivates people to
>observe beginners' stumbling blocks as you have done, and then to study
>how to help beginners cope with them.

The danger I am worried about is a change that simultaneously makes Emacs *more* approachable for the type of newcomer who isn't inclined toward investment but *less* rewarding to the type of user who is.  When we face that tradeoff, let's optimize for the second kind of user.

Obviously, changes that give the former benefit while not incurring the latter cost are non-controversial and we should all be in favor of them.  (How hard they may be to implement is a separate question.)

>What other frequent points of confusion have you observed?

Well, there are a bunch; the next time I'm helping a newbie I'll try to watch and keep a list.  I think I've posted the main ones that I can easily recall.

None of this is a criticism of Emacs, by the way.  The Atom editor, for example, seems to be a lot friendlier for newcomers, at least when I watch them use Atom.  On the other hand, I have yet to see anyone -- newcomer or experienced user -- using Atom to do text manipulation anywhere near as fast and fluently as an experienced Emacs user does.

One major source of these observations for me has been the regular Tuesday night in-person gatherings in Chicago at https://chihacknight.org/.  Unfortunately, those are not happening in person right now, of course, and no one knows when they will resume.  For the same reason, it will probably be a long time before I have a chance again to observe newcomers editing (in any editor) in person.

>It could be useful to make a list of those, and we could
>try to work on each of them over time.

Making such a list is not easy, but if I get a chance to do it I will.

>  > If we say instead "Emacs should try to attract newcomers who have
>  > a higher-than-average probability of becoming high-investment
>  > users, and should explain early on to those newcomers what the
>  > road ahead looks like"
>
>We should do that, but not by exaggerating.

As I reiterated in some other posts: I wasn't exaggerating.  Emacs is more daunting for newcomers than other editors, and its dauntingness is connected inextricably to its power -- to the path by which it rewards investment.

I believe that for most users, Emacs really only starts to pay off after more than a year of regular use.  By "pay off", I mean something quite specific: I mean reaching the point where one can, on average, accomplish one's tasks noticeably faster in Emacs than one could have in some other editor had one spent that year learning that other editor instead.  

This was my own experience, and it's what I've seen happen with others.  Now, I'm sure there are exceptions: if someone spends a few months *intensely* learning Emacs, especially with in-person help, they'll surely learn as much as some other person might have learned in a year or even more.  If I were to quit my job and practice piano all day long, I could get pretty good pretty fast too :-).  But I'm talking about the common case: of users who today are fluent Emacs users, I think (though based on anecdata) that most of them took more than a year to get to the point described above.

(I'm assuming vim is not the "other editor" in the above scenario, since vim too is extensible and very much oriented toward rewarding sustained investment -- and, no coincidence, fluent vim users do things at least as quickly as fluent Emacs users do, as far as I can tell.  I've sometimes heard that Microsoft Visual Studio Code can be the same way, but I've rarely had chances to observe experienced people using it, so I can't say.)

Best regards,
-Karl

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Monnier]

>>One idea: the first time a user tries to specify a directory in find-file,
>>display a help screen to explain what that means, and describe a few
>>usual ways out.
>>Do you think this would ameliorate the problem?
> My guess is it would ameliorate the problem for some users, and scare off
> other ones (ones who are less patient, or, put another way, who are less
> willing to invest time in reading an explanatory screen like that to figure
> out what just happened).

Maybe we could simply use the "confirm" mechanism.  I.e. don't put any
explanatory text, but just emit "[confirm]" and wait for a second RET,
just like we do when `C-x C-f` specifies a non-existing file.


        Stefan

Re: What is the most useful potential feature which Emacs lacks?

[Karl Fogel]

On 28 May 2020, Stefan Monnier wrote:
>>>One idea: the first time a user tries to specify a directory in find-file,
>>>display a help screen to explain what that means, and describe a few
>>>usual ways out.
>>>Do you think this would ameliorate the problem?
>> My guess is it would ameliorate the problem for some users, and scare off
>> other ones (ones who are less patient, or, put another way, who are less
>> willing to invest time in reading an explanatory screen like that to figure
>> out what just happened).
>
>Maybe we could simply use the "confirm" mechanism.  I.e. don't put any
>explanatory text, but just emit "[confirm]" and wait for a second RET,
>just like we do when `C-x C-f` specifies a non-existing file.

It would only do this the first time a user does `find-file' on a directory?  Or would it be every time?

If it's every time, that punishes experienced users, who do `find-file' on directories deliberately and don't want an extra keystroke to get in the way.

If it's only the first time, then I think it has other disadvantages:

- It doesn't really help the new user know what's going on or what's about to happen.

- It mis-trains them to think that when they do `find-file' on a directory they'll be asked for confirmation, when actually that's only going to happen this one time.  (Or maybe they have to set some variable to get what we currently consider the "normal" behavior?  But that's adding even complexity.)

I admit that I'm basically using this as Yet Another Example of how newcomer-friendliness is inherently in tension with rewards-investors :-).  It's hard to make find-file-on-a-directory less surprising for newcomers without either reducing the feature's investability (that is, without reducing a persistent new user's ability to learn about it by digesting a surprise and comprehending it) or reducing the feature's utility to the experienced users who are already familiar with it.

So, IMHO, we should change nothing about Emacs's behavior here: I think `find-file' does the best thing it can do right now.

(However, maybe it would be good to cover this behavior, or at least warn about it, in the tutorial or in other new-user help documentation -- I haven't looked at that documentation in a long time, so I can't say for sure.)

Best regards,
-Karl

RE: What is the most useful potential feature which Emacs lacks?

[Drew Adams]

> If it's every time, that punishes experienced users, who do `find-file'
> on directories deliberately and don't want an extra keystroke to get in
> the way.
> 
> If it's only the first time, then I think it has other disadvantages:
> 
> - It doesn't really help the new user know what's going on or what's
> about to happen.
> 
> - It mis-trains them to think that when they do `find-file' on a
> directory they'll be asked for confirmation, when actually that's only
> going to happen this one time.  (Or maybe they have to set some
> variable to get what we currently consider the "normal" behavior?  But
> that's adding even complexity.)
> 
> I admit that I'm basically using this as Yet Another Example of how
> newcomer-friendliness is inherently in tension with rewards-investors
> :-).  It's hard to make find-file-on-a-directory less surprising for
> newcomers without either reducing the feature's investability (that is,
> without reducing a persistent new user's ability to learn about it by
> digesting a surprise and comprehending it) or reducing the feature's
> utility to the experienced users who are already familiar with it.
> 
> So, IMHO, we should change nothing about Emacs's behavior here: I think
> `find-file' does the best thing it can do right now.
> 
> (However, maybe it would be good to cover this behavior, or at least
> warn about it, in the tutorial or in other new-user help documentation
> -- I haven't looked at that documentation in a long time, so I can't
> say for sure.)

+1, for everything.

Re: What is the most useful potential feature which Emacs lacks?

[Dmitry Gutov]

On 28.05.2020 20:19, Karl Fogel wrote:
> It would only do this the first time a user does `find-file' on a directory?  Or would it be every time?

Someday, we could also turn on fido-mode by default. There, RET means 
enter a directory. And/or pick the first suggestion.

It's a decent behavior for long-times users as well, since C-j provides 
an override.

Re: What is the most useful potential feature which Emacs lacks?

[Alan Third]

On Thu, May 28, 2020 at 09:45:28PM +0300, Dmitry Gutov wrote:
> On 28.05.2020 20:19, Karl Fogel wrote:
> > It would only do this the first time a user does `find-file' on a directory?  Or would it be every time?
> 
> Someday, we could also turn on fido-mode by default. There, RET means enter
> a directory. And/or pick the first suggestion.
> 
> It's a decent behavior for long-times users as well, since C-j provides an
> override.

This seems like a reasonable solution to me. Alternatively perhaps we
just need to sell C-x C-f as "open a file or directory" rather than
"find a file"?
-- 
Alan Third

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Monnier]

> This seems like a reasonable solution to me. Alternatively perhaps we
> just need to sell C-x C-f as "open a file or directory" rather than
> "find a file"?

A directory *is* a file.


        Stefan

Re: What is the most useful potential feature which Emacs lacks?

[Alan Third]

On Thu, May 28, 2020 at 05:02:33PM -0400, Stefan Monnier wrote:
> > This seems like a reasonable solution to me. Alternatively perhaps we
> > just need to sell C-x C-f as "open a file or directory" rather than
> > "find a file"?
> 
> A directory *is* a file.

Well why are these damn newbies getting confused when they open a
directory? ;)
-- 
Alan Third

Re: What is the most useful potential feature which Emacs lacks?

[andres.ramirez]

Hi.

>>>>> "Alan" == Alan Third <alan@idiocy.org> writes:

    Alan> On Thu, May 28, 2020 at 05:02:33PM -0400, Stefan Monnier
    Alan> wrote:
    >> > This seems like a reasonable solution to me. Alternatively
    >> perhaps we > just need to sell C-x C-f as "open a file or
    >> directory" rather than > "find a file"?
    >> 
    >> A directory *is* a file.

    Alan> Well why are these damn newbies getting confused when they
    Alan> open a directory? ;) -- Alan Third

Because on mainstream products (There is a distinction about file and
folder. But on Unix everything is a file).

Best Regards

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Monnier]

>> A directory *is* a file.
> Well why are these damn newbies getting confused when they open a
> directory? ;)

;-)


        Stefan

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

Alan Third <alan@idiocy.org> writes:

> On Thu, May 28, 2020 at 05:02:33PM -0400, Stefan Monnier wrote:
>> > This seems like a reasonable solution to me. Alternatively perhaps we
>> > just need to sell C-x C-f as "open a file or directory" rather than
>> > "find a file"?
>> 
>> A directory *is* a file.
>
> Well why are these damn newbies getting confused when they open a
> directory? ;)
n00bs! :-)

Re: What is the most useful potential feature which Emacs lacks?

[Joost Kremers]

On Thu, May 28 2020, Stefan Monnier wrote:
>> This seems like a reasonable solution to me. Alternatively 
>> perhaps we
>> just need to sell C-x C-f as "open a file or directory" rather 
>> than
>> "find a file"?
>
> A directory *is* a file.

And a smart phone is a computer, but that's not how most people 
think about it. :-)

-- 
Joost Kremers
Life has its moments

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

Joost Kremers <joostkremers@fastmail.fm> writes:

> On Thu, May 28 2020, Stefan Monnier wrote:
>>> This seems like a reasonable solution to me. Alternatively perhaps we
>>> just need to sell C-x C-f as "open a file or directory" rather than
>>> "find a file"?
>>
>> A directory *is* a file.
>
> And a smart phone is a computer, but that's not how most people think about it.
> :-)

And Dired is "directory editor", wich suggest a directory is a  file,
but even I use to speak of dired (and emacs) as a file manager, and
other file managers are not called directory editors, despite of file
managing being directories editing. Just another example of how we
humnas build abstractions on top of each other to simplify our
reasoning. Those abstractions then get their own life.

Re: What is the most useful potential feature which Emacs lacks?

[Karl Fogel]

On 28 May 2020, Stefan Monnier wrote:
>> This seems like a reasonable solution to me. Alternatively perhaps we
>> just need to sell C-x C-f as "open a file or directory" rather than
>> "find a file"?
>
>A directory *is* a file.

Not in a sense that's meaningful to most non-programmers.  Even many programmers don't naturally conceptualize directories as files -- especially those who have learned programming in the WWW era, since they haven't had to do much programming in which file<->directory unity matters.

If explanations of Emacs features start to depend on people having filesystem programming experience, that will narrow Emacs' audience quite a bit more than necessary :-).

Best regards,
-Karl

RE: What is the most useful potential feature which Emacs lacks?

[Drew Adams]

> >A directory *is* a file.
> 
> Not in a sense that's meaningful to most non-programmers.  Even many
> programmers don't naturally conceptualize directories as files --
> especially those who have learned programming in the WWW era, since
> they haven't had to do much programming in which file<->directory unity
> matters.

Yes.

A directory is often _implemented_ using a file,
and files and dirs can be manipulated in many of
the same ways.

But it helps to let a user know when some operation
works for either one, as opposed to just one or the
other.  Creating a new directory isn't the same
thing as creating a new file.

And Emacs reflects this in its prompts and doc
strings.  Dired is just one example ("visit this
file or directory in another window").

And for users who are used to thinking in terms of
files and "folders" there's no conception at all
of a folder being a file, regardless of the
underlying implementation.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This seems like a reasonable solution to me. Alternatively perhaps we
  > just need to sell C-x C-f as "open a file or directory" rather than
  > "find a file"?

That would make our initial explanations more complex, and that might
lead to more confusion than clarity.

I think it is better to explain this wrinkle when the user encounters it,
not before.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: What is the most useful potential feature which Emacs lacks?

[Drew Adams]

>> This seems like a reasonable solution to me. Alternatively perhaps
>> we just need to sell C-x C-f as "open a file or directory" rather
>> than "find a file"?
> 
> That would make our initial explanations more complex, and that might
> lead to more confusion than clarity.

I disagree.  It's clearer to state, when it's the case,
that a given operation acts on a file or a dir, or on
files and dirs.

We do that pretty consistently in Dired and some other
areas of Emacs.  We should do it wherever it makes
sense, and it makes sense wherever someone might not
understand that the operation applies to either/both.

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > This seems like a reasonable solution to me. Alternatively perhaps we
>   > just need to sell C-x C-f as "open a file or directory" rather than
>   > "find a file"?
>
> That would make our initial explanations more complex, and that might
> lead to more confusion than clarity.
>
> I think it is better to explain this wrinkle when the user encounters it,
> not before.

Aren't users encountering that wrinkle first time they open a file?

Observe there are even more wrinkles there to explain: if file does not
exist Emacs creates a new buffer, and if user ment a directory, the
buffer will still be just a plain file not a dir. And what about if
there are some non-existent directories on the way? Emacs asks if  user
wants them to be created ... so there are quite a few wrinkles in that
one, not the simplest behaviour to explain anyway :-).

I don't know how you reasoned back in days when find file was
introduced/created, but I guess, it was more Unixy, a dir is just a file
(everything in Unix is file), so it was assumed that dirs are just
files? I don't think millenials see dirs as just files, but I don't
know, I might be a bit prejudiced (right word?) here.

Re: What is the most useful potential feature which Emacs lacks?

[Thibaut Verron]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

Le ven. 29 mai 2020 à 15:20, Arthur Miller <arthur.miller@live.com> a
écrit :

> Richard Stallman <rms@gnu.org> writes:
>
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >
> >   > This seems like a reasonable solution to me. Alternatively perhaps we
> >   > just need to sell C-x C-f as "open a file or directory" rather than
> >   > "find a file"?
> >
> > That would make our initial explanations more complex, and that might
> > lead to more confusion than clarity.
> >
> > I think it is better to explain this wrinkle when the user encounters it,
> > not before.
>
> Aren't users encountering that wrinkle first time they open a file?
>
> Observe there are even more wrinkles there to explain: if file does not
> exist Emacs creates a new buffer, and if user ment a directory, the
> buffer will still be just a plain file not a dir. And what about if
> there are some non-existent directories on the way? Emacs asks if  user
> wants them to be created ... so there are quite a few wrinkles in that
> one, not the simplest behaviour to explain anyway :-).
>

But those are consistent with how other software behaves. The user won't
think of a buffer as much as of a not-yet-saved file, but that's also
consistent with what emacs does.

Other software would force the directory to be created before opening the
file, but that's minor I believe.

Being able to open a directory just like a file, on the other hand, is not
usual. Web browsers can do it, but that's because their "files" are almost
like directories.

I personally like the ido approach of having different keys for accessing
files and directories. Simple operations like listing files or creating a
new one are done directly in the ido buffer, no need for dired.

Thibaut

[-- Attachment #2: Type: text/html, Size: 2689 bytes --]

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

Alan Third <alan@idiocy.org> writes:

> On Thu, May 28, 2020 at 09:45:28PM +0300, Dmitry Gutov wrote:
>> On 28.05.2020 20:19, Karl Fogel wrote:
>> > It would only do this the first time a user does `find-file' on a directory?  Or would it be every time?
>> 
>> Someday, we could also turn on fido-mode by default. There, RET means enter
>> a directory. And/or pick the first suggestion.
>> 
>> It's a decent behavior for long-times users as well, since C-j provides an
>> override.
>
> This seems like a reasonable solution to me. Alternatively perhaps we
> just need to sell C-x C-f as "open a file or directory" rather than
> "find a file"?
I think you are spot on there. I don't think many people today think of
directories as just "files", so directory part of "find file" gets lost
for some. BTW, how about selling it as "Find file or directory"?

Re: What is the most useful potential feature which Emacs lacks?

[ndame]

> what are those other features which could bring much more tangible benefits for the user

BTW, somebody on Reddit said "I would easily pay $10 a year to have someone dedicated to work on Emacs. Many of us have employers that I guess would be willing to pay a dime as well." and others agreed.

Emacs Reddit has 40k members and it may be goal with which many of its members can sympahtize.

Do you think it could help Emacs if it had someone working on it full time, checking bug reports, taking care of all the usual things, fixing bugs, etc. thereby lightening the load on maintainers and core developers, so they have more time for developing features?

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

ndame <ndame@protonmail.com> writes:

>> what are those other features which could bring much more tangible benefits for the user
>
> BTW, somebody on Reddit said "I would easily pay $10 a year to have someone
> dedicated to work on Emacs. Many of us have employers that I guess would be
> willing to pay a dime as well." and others agreed.
>
> Emacs Reddit has 40k members and it may be goal with which many of its members can sympahtize.
>
> Do you think it could help Emacs if it had someone working on it full time,
> checking bug reports, taking care of all the usual things, fixing bugs, etc.
> thereby lightening the load on maintainers and core developers, so they have
> more time for developing features?
I can't even imagine how much Emacs would evolve if Eli got full-time
payed jobb on Emacs .... 

Re: What is the most useful potential feature which Emacs lacks?

[ndame]

>
> I can't even imagine how much Emacs would evolve if Eli got full-time
> payed jobb on Emacs ....

Well, let's ask him if he's interested. Eli?

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

ndame <ndame@protonmail.com> writes:

>>
>> I can't even imagine how much Emacs would evolve if Eli got full-time
>> payed jobb on Emacs ....
>
> Well, let's ask him if he's interested. Eli?
I would definitely pay $10 a year or halv if Eli would go full time on
Emacs :-).

Re: What is the most useful potential feature which Emacs lacks?

[Eli Zaretskii]

> Date: Thu, 14 May 2020 04:10:50 +0000
> From: ndame <ndame@protonmail.com>
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> >
> > I can't even imagine how much Emacs would evolve if Eli got full-time
> > payed jobb on Emacs ....
> 
> Well, let's ask him if he's interested. Eli?

It is not a question of interest, not at my age and point in life.

Anyway, the above is unlikely to happen.

Re: What is the most useful potential feature which Emacs lacks?

[ndame]

>
> It is not a question of interest, not at my age and point in life.
>
> Anyway, the above is unlikely to happen.

If by chance you know somebody who is qualified and could be interested in the job then please let him or her know about this possibility. Provided you think a full time developer could help Emacs development.  The core developers are in a better position to judge this.

Emacs development has an untapped potential in users who don't have the time or inclination to work on Emacs code, but they use Emacs a lot, so they are willing to donate for Emacs development. Most likely the vast majority of Emacs users are in this position, so the untapped potential is huge.

Re: What is the most useful potential feature which Emacs lacks?

[Stefan Kangas]

ndame <ndame@protonmail.com> writes:

> If by chance you know somebody who is qualified and could be
> interested in the job then please let him or her know about this
> possibility. Provided you think a full time developer could help Emacs
> development.  The core developers are in a better position to judge
> this.

Maybe this is starting in the wrong end.

There is some serious work required to even get something like this
going.  Perhaps one should look for a candidate to do that work first.

Not that I have any clout to say much about anything, but here's what I
think:

Whoever is prepared to organize this will work on a volunteer basis.
This would be a task of great responsibility, so it has to be a
trustworthy person, possibly already on emacs-devel or prepared to
establish and maintain that communication.

Such a volunteer would start with writing up a serious proposal for how
to organize it.  This should involve some specifics about financing (at
least: how to collect donations), recruitment of candidates,
advertising, deliverables, time-frame, etc.  Probably many other things.

Next, you would need to be able to work patiently in discussion to
convince emacs-devel, Eli, RMS and others that this is a good idea.
Part of that work is convincing people that you are the candidate to see
this through.  Maybe the FSF needs to be discussed with and convinced
too.

And, finally, the correct candidate for that work would be prepared to
accept that, even having invested the above time, maybe it's not
feasible to get all the moving parts working together to realize this.

If someone is interested in doing that work, maybe this has a chance.
I don't see that it's a hard task, just one that would take some time,
dedication and organizing.

Just my two cents.

Best regards,
Stefan Kangas

PS. BTW, is there any reason Emacs is not already part of GSOC?

Re: What is the most useful potential feature which Emacs lacks?

[Arthur Miller]

Stefan Kangas <stefankangas@gmail.com> writes:

> ndame <ndame@protonmail.com> writes:
>
>> If by chance you know somebody who is qualified and could be
>> interested in the job then please let him or her know about this
>> possibility. Provided you think a full time developer could help Emacs
>> development.  The core developers are in a better position to judge
>> this.
>
> Maybe this is starting in the wrong end.
>
> There is some serious work required to even get something like this
> going.  Perhaps one should look for a candidate to do that work first.
>
> Not that I have any clout to say much about anything, but here's what I
> think:
>
> Whoever is prepared to organize this will work on a volunteer basis.
> This would be a task of great responsibility, so it has to be a
> trustworthy person, possibly already on emacs-devel or prepared to
> establish and maintain that communication.
>
> Such a volunteer would start with writing up a serious proposal for how
> to organize it.  This should involve some specifics about financing (at
> least: how to collect donations), recruitment of candidates,
> advertising, deliverables, time-frame, etc.  Probably many other things.
>
> Next, you would need to be able to work patiently in discussion to
> convince emacs-devel, Eli, RMS and others that this is a good idea.
> Part of that work is convincing people that you are the candidate to see
> this through.  Maybe the FSF needs to be discussed with and convinced
> too.
>
> And, finally, the correct candidate for that work would be prepared to
> accept that, even having invested the above time, maybe it's not
> feasible to get all the moving parts working together to realize this.
>
> If someone is interested in doing that work, maybe this has a chance.
> I don't see that it's a hard task, just one that would take some time,
> dedication and organizing.
>
> Just my two cents.
>
> Best regards,
> Stefan Kangas
>
> PS. BTW, is there any reason Emacs is not already part of GSOC?

Could FSF control a crowd-sourcing fond that goes to development?

They could have something like a list of bugs/features, maybe have a
poll where public can vote for priority on that list, and announce a
prize money to those who turn in accepted patch. Does not even need to
be an Emacs specific fond, could be a list of bugs/features/improvements
in all GNU software. It could be open to anyone who is willing to send
in technically and legaly (FSF paperwork, licence and so on) acceptable
patch.

Re: What is the most useful potential feature which Emacs lacks?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The FSF could set up a special fund for Emacs development.
It does that for other programs.

However, experience with other packages says that it probably
won't bring in enough money to pay a developer.  The other programs
use their funds for other, smaller expenses.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What is the most useful potential feature which Emacs lacks?

[chad]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

On Sun, May 17, 2020 at 5:43 AM Stefan Kangas <stefankangas@gmail.com>
wrote:

> PS. BTW, is there any reason Emacs is not already part of GSOC?
>

GSOC projects to improve Emacs have happened in the past. IIUC, the main
hurdles are finding interesting projects that fit in the short timeline,
and finding mentors for such projects. For example, several years back
there were a couple GSOC projects to push guile&emacs integration:
https://www.gnu.org/software/guile/news/gsoc-guile-emacs-and-emacsy.html

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 986 bytes --]

What is the most useful potential feature which Emacs lacks?

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

Namespaces for me.  The kind where I can write a new package without worrying
about prefixing anything, declare a public interface and later import the
package with an appropriate namespace prefix.  No more unnecessary namespace
pollution or pondering about the tradeoff between ergonomics and potential
namespace clashes.  Experimentation gets easier, say you'd want to have list
processing functions à la <insert favorite Lisp>.  You can now design such a
thing without forcing a clunky or misleading namespace prefix on everyone.  Or
imagine you'd want to replace or prototype a built-in facility.  Or you'd like
to use just one thing from a package.  All these things are now possible, at
the cost of `grep` becoming a less powerful tool.

Vasilij

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: What is the most useful potential feature which Emacs lacks?

[Björn Lindqvist]

I've long wanted GMail integration in Gnus. The IMAP interface just
doesn't work very well. I know it will never happen in a million years
because GMail is non-free software but one can wish. :)

Den mån 11 maj 2020 kl 22:10 skrev ndame <ndame@protonmail.com>:
>
> There is a discussion on Reddit about sponsoring development of multithreading in Emacs, and people there say it's too hard, takes a lot of time and it doesn't even bring that much benefit to the user.
>
> If this is the case (is it?) then what are those other features which could bring much more tangible benefits for the user and assuming somebody works on them full time sponsored by the community they can be implemented in, say,  a few months?



-- 
mvh/best regards Björn Lindqvist

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Konstantin Kharlamov]

Hello, my 50 cents. The most missing feature, which has almost every
text editor out there except Emacs, is no doubt sane autocompletion.

"But there are autocompletion plugins!" someone might say. Okay,
here's the problem: due to lack of multithreading, they usually set a
timeout before autocompletion appears. This makes plugins limited to
just one usecase: when a user doesn't know what to type next (like,
doesn't remember exact method name, etc.). But most often people know
what they want, and they type very fast. In this case what
autocompletion should provide is prediction. And Emacs autocompletion
plugins are failing bad at this: if you type fast, this means timeout
is never expired, and completion never appears.

I could set timeout to 0 (which I used to do some years ago), but this
often results in terrible lags.

And it gets worse: even if timeout expires, depending on completion
backend you can get a freeze. For example, how about completion from
tags backend, where the TAGS file is 183MB in size! This is the actual
TAGS file I got from LibreOffice project.

Lack of such simple but immensely useful feature is so disappointing
that some years ago I was trying to migrate over to some other
editor. And when I was asked "is it worth learning Emacs", I was
recommending Vim. Because, while I use Emacs with Evil mode, and I
admit this is much more powerful than Vim with plugins (which I use
too for quick edits in terminal), but sane autocompletion is something
anyone would expect to work out of the box. And the fact it not only
doesn't work OOTB, but there's no way to make it even work, will sure
turn away users quite quickly.

On Mon, 2020-05-11 at 20:09 +0000, ndame wrote:
> There is a discussion on Reddit about sponsoring development of
> multithreading in Emacs, and people there say it's too hard, takes a
> lot of time and it doesn't even bring that much benefit to the user.
>
> If this is the case (is it?) then what are those other features which
> could bring much more tangible benefits for the user and assuming
> somebody works on them full time sponsored by the community they can
> be implemented in, say,  a few months?

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Dmitry Gutov]

On 03.06.2020 14:39, Konstantin Kharlamov wrote:
> Lack of such simple but immensely useful feature is so disappointing
> that some years ago I was trying to migrate over to some other
> editor.

What simple feature? Prediction? Is that like mind reading?

As far a multithreading goes, try some backend that uses an external 
program (either of the LSP clients, or irony, rtags, etc). That's a 
basic kind of concurrency already available in Emacs.

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Konstantin Kharlamov]

On Wed, 2020-06-03 at 15:36 +0300, Dmitry Gutov wrote:
> On 03.06.2020 14:39, Konstantin Kharlamov wrote:
> > Lack of such simple but immensely useful feature is so
> > disappointing
> > that some years ago I was trying to migrate over to some other
> > editor.
> 
> What simple feature? Prediction? Is that like mind reading?

You misunderstand what I say. In technical terms "prediction" here
means "the timeout set to 0".

> As far a multithreading goes, try some backend that uses an external 
> program (either of the LSP clients, or irony, rtags, etc). That's a 
> basic kind of concurrency already available in Emacs.


Just to make sure: and it not gonna lag if I set timeout to 0? If yes,
then great to know, maybe I fell behind recent developments, I should
try it then.

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Dmitry Gutov]

On 03.06.2020 15:50, Konstantin Kharlamov wrote:
> On Wed, 2020-06-03 at 15:36 +0300, Dmitry Gutov wrote:
>> On 03.06.2020 14:39, Konstantin Kharlamov wrote:
>>> Lack of such simple but immensely useful feature is so
>>> disappointing
>>> that some years ago I was trying to migrate over to some other
>>> editor.
>>
>> What simple feature? Prediction? Is that like mind reading?
> 
> You misunderstand what I say. In technical terms "prediction" here
> means "the timeout set to 0".

Timeout set to 0 means firing completion requests right after every user 
input. If completion is synchronous, that will of course lead to a 
slowdown (depending on how slow the backend is).

For asynchronous ones, it could be fine, if the backing process can 
handle requests at such frequency.

In general, the value of 0 seems wasteful to me, but it should work.

>> As far a multithreading goes, try some backend that uses an external
>> program (either of the LSP clients, or irony, rtags, etc). That's a
>> basic kind of concurrency already available in Emacs.
> 
> 
> Just to make sure: and it not gonna lag if I set timeout to 0? If yes,
> then great to know, maybe I fell behind recent developments, I should
> try it then.

To be 100% sure, you should try it yourself (I don't do C/C++). Maybe 
someone else here can testify, though.

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Konstantin Kharlamov]

On Wed, 2020-06-03 at 16:10 +0300, Dmitry Gutov wrote:
> On 03.06.2020 15:50, Konstantin Kharlamov wrote:
> > On Wed, 2020-06-03 at 15:36 +0300, Dmitry Gutov wrote:
> > > On 03.06.2020 14:39, Konstantin Kharlamov wrote:
> > > > Lack of such simple but immensely useful feature is so
> > > > disappointing
> > > > that some years ago I was trying to migrate over to some other
> > > > editor.
> > >
> > > What simple feature? Prediction? Is that like mind reading?
> >
> > You misunderstand what I say. In technical terms "prediction" here
> > means "the timeout set to 0".
>
> Timeout set to 0 means firing completion requests right after every
> user
> input. If completion is synchronous, that will of course lead to a
> slowdown (depending on how slow the backend is).
>
> For asynchronous ones, it could be fine, if the backing process can
> handle requests at such frequency.
>
> In general, the value of 0 seems wasteful to me, but it should work.

I am not sure why you say it seems wasteful. Do you mean perhaps, as
opposed to setting, say, 100ms? 100ms I think is the top limit this
timeout should be set. I just tested how quickly I can type a string
"prog". I fired up `libinput debug-events` and tried to type
"prog". The letter "g" says "+0.256s", i.e. it took 256ms. This means
even if I set to 100ms, there's high chance I won't get any
autocompletion.

> > > As far a multithreading goes, try some backend that uses an
> > > external
> > > program (either of the LSP clients, or irony, rtags, etc). That's
> > > a
> > > basic kind of concurrency already available in Emacs.
> >
> > Just to make sure: and it not gonna lag if I set timeout to 0? If
> > yes,
> > then great to know, maybe I fell behind recent developments, I
> > should
> > try it then.
>
> To be 100% sure, you should try it yourself (I don't do C/C++).
> Maybe
> someone else here can testify, though.

Thank you! Indeed, I confirm this does seem to work with an async
backend. I tested it as follows:

1. Opened a test.cpp file, enabled company-irony, checked that
   completion works.
2. I set `(setq company-idle-delay 0)`
3. I paused the irony-server process with `kill -SIGSTOP $(pidof
   irony-server)`
4. I tried typing a gibberish to see if I get any delay in rendering a
   text.

I don't see any lags, so I assume using an async backend with the
timeout set to 0 should work fine. This is great news! I wonder if
company mode should default to zero or so timeout, and print a warning
if somebody tries to connect a non-async backend?

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Dmitry Gutov]

On 03.06.2020 16:59, Konstantin Kharlamov wrote:

> I am not sure why you say it seems wasteful. Do you mean perhaps, as
> opposed to setting, say, 100ms?

Or 50ms, maybe.

Wasteful of CPU cycles and laptop battery, I'd say. Of course, the exact 
impact depends on the size of your project.

> 100ms I think is the top limit this
> timeout should be set.

The defaults are not set in stone, we can discuss changing them on the 
company issue tracker. But it's a balancing act. Set the value too small 
-- and the result is more likely to annoy people who don't rely on the 
popup as much.

> I just tested how quickly I can type a string
> "prog". I fired up `libinput debug-events` and tried to type
> "prog". The letter "g" says "+0.256s", i.e. it took 256ms. This means
> even if I set to 100ms, there's high chance I won't get any
> autocompletion.

Should I take this to mean the completion request itself took 156ms to 
finish?

>> To be 100% sure, you should try it yourself (I don't do C/C++).
>> Maybe
>> someone else here can testify, though.
> 
> Thank you! Indeed, I confirm this does seem to work with an async
> backend. I tested it as follows:
> 
> 1. Opened a test.cpp file, enabled company-irony, checked that
>     completion works.
> 2. I set `(setq company-idle-delay 0)`
> 3. I paused the irony-server process with `kill -SIGSTOP $(pidof
>     irony-server)`
> 4. I tried typing a gibberish to see if I get any delay in rendering a
>     text.
> 
> I don't see any lags, so I assume using an async backend with the
> timeout set to 0 should work fine.

Indeed, that's what asynchronous means. But the quality of the user 
experience also depends on how quickly the backing server can handle 
those requests.

> This is great news! I wonder if
> company mode should default to zero or so timeout, and print a warning
> if somebody tries to connect a non-async backend?

The majority of backends are synchronous. And the "standard" completion 
API for Emacs (which we want to integrate with) still only supports the 
synchronous convention.

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Konstantin Kharlamov]

On Wed, 2020-06-03 at 17:21 +0300, Dmitry Gutov wrote:
> On 03.06.2020 16:59, Konstantin Kharlamov wrote:
> 
> > I am not sure why you say it seems wasteful. Do you mean perhaps,
> > as
> > opposed to setting, say, 100ms?
> 
> Or 50ms, maybe.
> 
> Wasteful of CPU cycles and laptop battery, I'd say. Of course, the
> exact 
> impact depends on the size of your project.
> 
> > 100ms I think is the top limit this
> > timeout should be set.
> 
> The defaults are not set in stone, we can discuss changing them on
> the 
> company issue tracker. But it's a balancing act. Set the value too
> small 
> -- and the result is more likely to annoy people who don't rely on
> the 
> popup as much.
> 
> > I just tested how quickly I can type a string
> > "prog". I fired up `libinput debug-events` and tried to type
> > "prog". The letter "g" says "+0.256s", i.e. it took 256ms. This
> > means
> > even if I set to 100ms, there's high chance I won't get any
> > autocompletion.
> 
> Should I take this to mean the completion request itself took 156ms
> to 
> finish?

I mean, I was typing outside Emacs, in the same terminal where
`libinput debug-events` was running. I was just testing how fast I can
type, to figure out how small the timeout should be set if I want a
completion to pop up.

> > > To be 100% sure, you should try it yourself (I don't do C/C++).
> > > Maybe
> > > someone else here can testify, though.
> > 
> > Thank you! Indeed, I confirm this does seem to work with an async
> > backend. I tested it as follows:
> > 
> > 1. Opened a test.cpp file, enabled company-irony, checked that
> >     completion works.
> > 2. I set `(setq company-idle-delay 0)`
> > 3. I paused the irony-server process with `kill -SIGSTOP $(pidof
> >     irony-server)`
> > 4. I tried typing a gibberish to see if I get any delay in
> > rendering a
> >     text.
> > 
> > I don't see any lags, so I assume using an async backend with the
> > timeout set to 0 should work fine.
> 
> Indeed, that's what asynchronous means. But the quality of the user 
> experience also depends on how quickly the backing server can handle 
> those requests.

Well, I paused the backing server in the steps above, so server
couldn't answer. This was emulating a behaviour where a user works with
a project too big for backend to return a completion immediately.

Expected behaviour was that I should not get any lags while typing, and
there were no lags.

> > This is great news! I wonder if
> > company mode should default to zero or so timeout, and print a
> > warning
> > if somebody tries to connect a non-async backend?
> 
> The majority of backends are synchronous. And the "standard"
> completion 
> API for Emacs (which we want to integrate with) still only supports
> the 
> synchronous convention.

I am a bit confused by the last sentence. What's the relation between
the Emacs API and already working company-mode? Did you mean, company-
mode is trying to be compatible with backends for some standard Emacs
API, and those can't be async?

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Dmitry Gutov]

On 03.06.2020 17:40, Konstantin Kharlamov wrote:

> I mean, I was typing outside Emacs, in the same terminal where
> `libinput debug-events` was running. I was just testing how fast I can
> type, to figure out how small the timeout should be set if I want a
> completion to pop up.

Not sure how to interpret its output. Here's what I was getting while 
typing 'prog':

  event15  KEYBOARD_KEY     +21.57s	*** (-1) pressed
  event15  KEYBOARD_KEY     +21.66s	*** (-1) pressed
  event15  KEYBOARD_KEY     +21.68s	*** (-1) released
  event15  KEYBOARD_KEY     +21.74s	*** (-1) pressed
  event15  KEYBOARD_KEY     +21.76s	*** (-1) released
  event15  KEYBOARD_KEY     +21.84s	*** (-1) pressed
  event15  KEYBOARD_KEY     +21.86s	*** (-1) released
  event15  KEYBOARD_KEY     +22.00s	*** (-1) released

That seems like I hit a key per ~100ms, but handling the releases adds 
its latency.

>>> I don't see any lags, so I assume using an async backend with the
>>> timeout set to 0 should work fine.
>>
>> Indeed, that's what asynchronous means. But the quality of the user
>> experience also depends on how quickly the backing server can handle
>> those requests.
> 
> Well, I paused the backing server in the steps above, so server
> couldn't answer. This was emulating a behaviour where a user works with
> a project too big for backend to return a completion immediately.

That tests the best case (which is, it assumes that when you turn the 
server back, it can handle the load). No typing latency is good (of 
course), but if you don't see completion suggestions because the backing 
process is overloaded, that's not so great either.

I'm not saying it's going to be a problem with Irony or LSP necessarily, 
but I've seen this reported in other circumstances. Just something to 
pay attention to.

>> The majority of backends are synchronous. And the "standard"
>> completion
>> API for Emacs (which we want to integrate with) still only supports
>> the
>> synchronous convention.
> 
> I am a bit confused by the last sentence. What's the relation between
> the Emacs API and already working company-mode? Did you mean, company-
> mode is trying to be compatible with backends for some standard Emacs
> API, and those can't be async?

Pretty much.

Eglot does include a hack to minimize typing latency as well, but it's 
an ad-hoc solution, and there's no way Company would be able to detect 
this kind of approach being used.

RE: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Drew Adams]

> Lack of such simple but immensely useful feature
> is so disappointing

FWIW -

Icicles lets you control what you call autocompletion,
and what Icicles calls incremental completion, in a
few ways.

This page explains it:

https://www.emacswiki.org/emacs/Icicles_-_Icompletion#IncrementalCompletion

Option `icicle-incremental-completion' is the main
on/off control.

Options `icicle-incremental-completion-delay',
`icicle-incremental-completion-threshold', and
`icicle-show-completions-initially' determine
when it kicks in.

But one size doesn't fit all.  Different sets of
completion candidates (e.g. domain size), and
different kinds of completion, imply different
fits, in terms of such option values.

A given command that uses completion knows its
candidates domain and the kind(s) of completion.
It can bind such options to values that provide
appropriate default behavior for that command.

But just as importantly, you can change the
option values interactively, while completing:
increase or decrease the delay or threshold,
for example, or change the general on/off
behavior.

To me, easy user control is the most important
thing for such a feature.  A command knows what
kind of completion it's dealing with, but only
a user knows just what s?he wants/needs, during
any given invocation of a command that uses
completion.

Re: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Konstantin Kharlamov]

On Wed, 2020-06-03 at 07:49 -0700, Drew Adams wrote:
> > Lack of such simple but immensely useful feature
> > is so disappointing
> 
> FWIW -
> 
> Icicles lets you control what you call autocompletion,
> and what Icicles calls incremental completion, in a
> few ways.
> 
> This page explains it:
> 
> https://www.emacswiki.org/emacs/Icicles_-_Icompletion#IncrementalCompletion
> 
> Option `icicle-incremental-completion' is the main
> on/off control.

I don't know if I'm misreading it, but from what I read about
this mode, it seems to only provide completions for minibuffer
commands. Undoubtedly useful feature, and for this usecase I already
use ido-mode with fuzzy matches. But in my original text I was rather
referring to autocompletion when you write code/text, i.e. in buffers
different from the minibuffer.

RE: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Drew Adams]

> I don't know if I'm misreading it, but from what I read about
> this mode, it seems to only provide completions for minibuffer
> commands. Undoubtedly useful feature, and for this usecase I already
> use ido-mode with fuzzy matches. But in my original text I was rather
> referring to autocompletion when you write code/text, i.e. in buffers
> different from the minibuffer.

You did not misread it.  I should have made that
clear.  But I skipped reading the thread, so I
didn't realize that's all the question is about.

Icicles does support buffer-text completion here
and there, but as you say, most of what's involved
is about completion using the minibuffer -
`completing-read', `read-file-name', etc.

Even the (limited) Icicles support for buffer-text
completion uses the minibuffer as long as there
are multiple matching candidates, however.  So in
those cases you can take advantage of the usual
Icicles features.

You can think of such cases as popping up a list
of autocomplete choices.  The list is shown in
`*Completions*' though, not in your usual menu
form.

This the doc topic that talks about Icicles
buffer-text completion:

https://www.emacswiki.org/emacs/Icicles_-_Completion_in_Other_Buffers

Yes, this is a major lacuna.  I haven't bothered
to provide a general Icicles solution to buffer-text
completion based on, say, `completion-at-point'.

But if I did get around to doing that I'd no doubt
tie in the usual Icicles completion behavior, as
I did for those few special cases, but in a more
general way.

IOW, you'd still use the minibuffer when multiple
candidates match both (a) the buffer text to be
completed and (b) any minibuffer input.  You'd
still be able to progressively complete with
multiple input patterns.  You'd still be able to
sort and change sort orders on the fly.  And so on.

RE: What is the most useful potential feature which Emacs lacks? A: Autocompletion

[Drew Adams]

To be more clear:

My purpose in mentioning what Icicles does wrt
incremental completion ("autocompletion") was
to suggest features that can be useful in this
context - food for thought - not to solicit
use of Icicles.

IOW, vanilla Emacs might also consider such
features, or similar.  And none of what I
described requires the use of the minibuffer,
even if that's what Icicles uses.