Auto-installing packages?

Auto-installing packages?

[Stefan Monnier]

Every time I see someone asking for or posting a little Elisp snippet
that automatically installs packages at Emacs startup, I'm wondering how
to answer that need: OT1H I find such requests fundamentally flawed in
terms of security, privacy, and reliability, but OTOH they correspond to
a real desire and lots of our users seem to not only be willing to
forego their security, privacy and reliability for that feature, but
they actually do so.

So, I'm wondering how we could better serve our users by providing them
with an alternative solution to their underlying need, which hopefully
suffers less from those problems.

Maybe some way to have something along the line of autoloads but that
instead of loading the Elisp code from a locally installed package, they
prompt the user explaining that this functionality requires installing
package FOO, etc?


        Stefan

Re: Auto-installing packages?

[Phil Sainty]

On 2018-09-14 13:51, Stefan Monnier wrote:
> Maybe some way to have something along the line of autoloads but that
> instead of loading the Elisp code from a locally installed package, 
> they
> prompt the user explaining that this functionality requires installing
> package FOO, etc?

On the one hand I quite like that idea, and my first thought is that if
Emacs by default knew about commands from GNU ELPA packages, then maybe
more authors would actually be inspired to put their packages into the
GNU ELPA package archive, which would be good.

On the other hand I see the significant caveat that most of the packages
in GNU ELPA are probably of no interest to any given user, and a large
quantity of autoload declarations for irrelevant things clogging up the
function namespace seems like it could very well be annoying.

(Arguably no different to most of the regular autoloads, but I think
that ELPA has the potential to become a *lot* larger.)

We could always include a user option to say whether or not the GNU ELPA
autoloads should be included in completions?  If that was non-nil by
default, new users get the benefits you're looking for, but people can
disable the behaviour if they wish.


-Phil

Re: Auto-installing packages?

[T.V Raman]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

autoload-remote?


So one could potentially say
(autoload-remote "fn" "pkg" t)

in a module that needs "pkg" and have the magic happen
> Every time I see someone asking for or posting a little Elisp snippet
> that automatically installs packages at Emacs startup, I'm wondering how
> to answer that need: OT1H I find such requests fundamentally flawed in
> terms of security, privacy, and reliability, but OTOH they correspond to
> a real desire and lots of our users seem to not only be willing to
> forego their security, privacy and reliability for that feature, but
> they actually do so.
>
> So, I'm wondering how we could better serve our users by providing them
> with an alternative solution to their underlying need, which hopefully
> suffers less from those problems.
>
> Maybe some way to have something along the line of autoloads but that
> instead of loading the Elisp code from a locally installed package, they
> prompt the user explaining that this functionality requires installing
> package FOO, etc?
>
>
>         Stefan
>

-- 

Re: Auto-installing packages?

[Phillip Lord]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> Every time I see someone asking for or posting a little Elisp snippet
> that automatically installs packages at Emacs startup, I'm wondering how
> to answer that need: OT1H I find such requests fundamentally flawed in
> terms of security, privacy, and reliability, but OTOH they correspond to
> a real desire and lots of our users seem to not only be willing to
> forego their security, privacy and reliability for that feature, but
> they actually do so.
>
> So, I'm wondering how we could better serve our users by providing them
> with an alternative solution to their underlying need, which hopefully
> suffers less from those problems.
>
> Maybe some way to have something along the line of autoloads but that
> instead of loading the Elisp code from a locally installed package, they
> prompt the user explaining that this functionality requires installing
> package FOO, etc?


I use `use-package' to "automatically" install packages for me. I use
several machines and sync my configuration between them. The packages I
want get auto-installed this way. Also, when I update Emacs, I
re-install all my packages.

The auto-installing, autoload would be nice. It would answer the
long-standing desire to incorporate ELPA packages more deeply into
Emacs. However, to make it work well, I think, ELPA needs updating to
support multiple versions of packages for multiple versions of Emacs.

Phil

Re: Auto-installing packages?

[Perry E. Metzger]

On Thu, 13 Sep 2018 21:51:09 -0400 Stefan Monnier
<monnier@IRO.UMontreal.CA> wrote:
> So, I'm wondering how we could better serve our users by providing
> them with an alternative solution to their underlying need, which
> hopefully suffers less from those problems.
> 
> Maybe some way to have something along the line of autoloads but
> that instead of loading the Elisp code from a locally installed
> package, they prompt the user explaining that this functionality
> requires installing package FOO, etc?

That's not too bad. A couple of flavors of Linux have a feature at
the command line where if you run a common program that isn't
installed, it tells you what package will install it. This would be
comparable.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Auto-installing packages?

[Stefan Monnier]

> I use `use-package' to "automatically" install packages for me.

How does it work?
Doesn't this silently install at startup the packages you specified?
I really dislike the "silently" part (and am not too happy about doing
all of it at startup either).

Another related issue is the "auto-upgrade" part.  This one is worse
because it seems to inevitably involve some kind of "polling" of
a central server, which I find to be a serious privacy issue.

> The auto-installing, autoload would be nice. It would answer the
> long-standing desire to incorporate ELPA packages more deeply into
> Emacs.

Indeed, there's the problem of installing those auto-install-autoloads,
and admittedly, for GNU ELPA packages we could imagine pre-installing
them for all GNU ELPA packages, but I'd like for it to be easy enough to
setup to convince most users to use the same mechanism for
MELPA packages.

Also, pre-installing them for GNU ELPA packages obviously will only
handle those packages that are part of GNU ELPA when Emacs is released.

[ And, it's not necessarily all clear how to setup
  auto-install-autoloads for some packages, such as ada-ref-man.  ]

> However, to make it work well, I think, ELPA needs updating to
> support multiple versions of packages for multiple versions of Emacs.

Likely, yes.


        Stefan

Re: Auto-installing packages?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The difference between Emacs and packages available over the net
is not merely technical.  It affects legal issues, and it affects
communication of our message,

So we should not try to minimize its visibility.  Just the contrary,
we need to make it the difference visible, something that users will
think about.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Auto-installing packages?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That's not too bad. A couple of flavors of Linux have a feature at
  > the command line where

Linux does not implement a command line, so I have a feeling you're
probably talking about the GNU/Linux system.  We, the GNU Project,
started the development of that system, years before Torvalds got
involved.

"Flavors of Linux" attributes our work to him and gives us non of the
credit.  Would you please call them "flavors of GNU/Linux," so as to
give us equal mention?

See https://gnu.org/gnu/linux-and-gnu.html and
https://gnu.org/gnu/gnu-linux-faq.html, plus the history in
https://gnu.org/gnu/the-gnu-project.html.


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Auto-installing packages?

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I use `use-package' to "automatically" install packages for me.
>
> How does it work?
> Doesn't this silently install at startup the packages you specified?
> I really dislike the "silently" part (and am not too happy about doing
> all of it at startup either).

Yep. It runs "package-install". It's not silent, since you get all the
install messages. And, yes, doing it as part of startup is very slow
iff none of the packages are installed. Also, it breaks startup if a
package installation fails.

Still this only happens when you use Emacs on a new machine or in my
case, when I change Emacs version.


> Another related issue is the "auto-upgrade" part.  This one is worse
> because it seems to inevitably involve some kind of "polling" of
> a central server, which I find to be a serious privacy issue.

It doesn't autoupdate, although it's simple to make it do so, but
automating the "package-refresh, update-all" cycle. A privacy issue,
yes, maybe, although it's not massively different from running this
manually.


>> The auto-installing, autoload would be nice. It would answer the
>> long-standing desire to incorporate ELPA packages more deeply into
>> Emacs.
>
> Indeed, there's the problem of installing those auto-install-autoloads,

There is a practical issue with handling it *just like* autoloads. At
the moment, autoloads happens in the command-loop. So, from nearly
anywhere. That's okay, because if an autoloaded package cannot be
autoloaded, then Emacs has some pretty major issues.

Installing a package over a network is a much more failable process.

> and admittedly, for GNU ELPA packages we could imagine pre-installing
> them for all GNU ELPA packages, but I'd like for it to be easy enough to
> setup to convince most users to use the same mechanism for
> MELPA packages.
>
> Also, pre-installing them for GNU ELPA packages obviously will only
> handle those packages that are part of GNU ELPA when Emacs is released.

No, you misunderstand me. You don't need to pre-install this way. You
release "Emacs without ELPA", but include
"auto-install-autoloads". Emacs will get bigger this way over time.


Phil

Re: Auto-installing packages?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> The difference between Emacs and packages available over the net
> is not merely technical.  It affects legal issues, and it affects
> communication of our message,
>
> So we should not try to minimize its visibility.  Just the contrary,
> we need to make it the difference visible, something that users will
> think about.


And, yet, the mantra is "ELPA is Emacs". It requires copyright
assignment (with all the intense pain that this entails), the files say
"part of Emacs" and so forth.

So, for the case of Emacs and ELPA installing over the network vs
pre-packaged is, I think, merely technical.

Phil

Re: Auto-installing packages?

[Stefan Monnier]

>>> I use `use-package' to "automatically" install packages for me.
>> How does it work?
>> Doesn't this silently install at startup the packages you specified?
>> I really dislike the "silently" part (and am not too happy about doing
>> all of it at startup either).
> Yep. It runs "package-install". It's not silent, since you get all the

By "silent" I meant that it does it without asking for
permission/confirmation first.  I understand that it's a small matter of
programming, but from the UI-side I consider this unacceptable.

> install messages. And, yes, doing it as part of startup is very slow
> iff none of the packages are installed. Also, it breaks startup if a
> package installation fails.
> Still this only happens when you use Emacs on a new machine or in my
> case, when I change Emacs version.

Which all argues in favor of prompting.

>> Another related issue is the "auto-upgrade" part.  This one is worse
>> because it seems to inevitably involve some kind of "polling" of
>> a central server, which I find to be a serious privacy issue.
> It doesn't autoupdate, although it's simple to make it do so, but
> automating the "package-refresh, update-all" cycle. A privacy issue,
> yes, maybe, although it's not massively different from running this
> manually.

I find a constant "call home" done automatically qualitatively very
different from running it manually (e.g. you can make sure it's always
done from your home, for example).

But I wouldn't oppose an option to automatically check for updates in
the background (I'd also be OK with an option to auto-download them, but
I'd oppose auto-installing them updates without an explicit user
interaction).

>> Indeed, there's the problem of installing those auto-install-autoloads,
> There is a practical issue with handling it *just like* autoloads.
> At the moment, autoloads happens in the command-loop.

Actually, it happens at a much lower-level within `funcall`.

> So, from nearly anywhere.

Oh, yes.

> That's okay, because if an autoloaded package cannot be autoloaded,
> then Emacs has some pretty major issues.
> Installing a package over a network is a much more failable process.

No doubt.  Maybe rather than "prompt the user, install, and retry the
call", we'll need to "abort the current command, then prompt, ...".

> No, you misunderstand me. You don't need to pre-install this way. You
> release "Emacs without ELPA", but include
> "auto-install-autoloads". Emacs will get bigger this way over time.

Ah, make the auto-installs into an upgradable package, yes that makes
a lot of sense.


        Stefan

Re: Auto-installing packages?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And, yet, the mantra is "ELPA is Emacs". It requires copyright
  > assignment (with all the intense pain that this entails), the files say
  > "part of Emacs" and so forth.

  > So, for the case of Emacs and ELPA installing over the network vs
  > pre-packaged is, I think, merely technical.

That is a valid point, in the case of GNU ELPA.

But would that facility be limited to GNU ELPA?
Or would it also be used with other package archives
that are not part of Emacs?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Auto-installing packages?

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> I use `use-package' to "automatically" install packages for me.
>>> How does it work?
>>> Doesn't this silently install at startup the packages you specified?
>>> I really dislike the "silently" part (and am not too happy about doing
>>> all of it at startup either).
>> Yep. It runs "package-install". It's not silent, since you get all the
>
> By "silent" I meant that it does it without asking for
> permission/confirmation first.  I understand that it's a small matter of
> programming, but from the UI-side I consider this unacceptable.

Yes. Although, in this case, I have put the packages I want to install
into my .emacs. So not "silent" at that point. I am doing something
concrete.



>> install messages. And, yes, doing it as part of startup is very slow
>> iff none of the packages are installed. Also, it breaks startup if a
>> package installation fails.
>> Still this only happens when you use Emacs on a new machine or in my
>> case, when I change Emacs version.
>
> Which all argues in favor of prompting.


It's easy to add, although I suspect answering "Y" would become part of
the muscule memory.


> I find a constant "call home" done automatically qualitatively very
> different from running it manually (e.g. you can make sure it's always
> done from your home, for example).
>
> But I wouldn't oppose an option to automatically check for updates in
> the background (I'd also be OK with an option to auto-download them, but
> I'd oppose auto-installing them updates without an explicit user
> interaction).

I forget how package.el works, but it inherits behaviour from that. So,
it will update the package-list (i.e. call home) if your .emacs.d/elpa
is empty. But I don't think it refreshes it to look for updates.



> Actually, it happens at a much lower-level within `funcall`.
>
>> So, from nearly anywhere.
>
> Oh, yes.
>
>> That's okay, because if an autoloaded package cannot be autoloaded,
>> then Emacs has some pretty major issues.
>> Installing a package over a network is a much more failable process.
>
> No doubt.  Maybe rather than "prompt the user, install, and retry the
> call", we'll need to "abort the current command, then prompt, ...".


Yes.


>> No, you misunderstand me. You don't need to pre-install this way. You
>> release "Emacs without ELPA", but include
>> "auto-install-autoloads". Emacs will get bigger this way over time.
>
> Ah, make the auto-installs into an upgradable package, yes that makes
> a lot of sense.


Just so. What this means is someone installing Emacs one year after a
release would (likely) get updated packages, compared to release
date. Whether packages would update themselves after the first start
would be a different question. They wouldn't need to.

Phil

Re: Auto-installing packages?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > And, yet, the mantra is "ELPA is Emacs". It requires copyright
>   > assignment (with all the intense pain that this entails), the files say
>   > "part of Emacs" and so forth.
>
>   > So, for the case of Emacs and ELPA installing over the network vs
>   > pre-packaged is, I think, merely technical.
>
> That is a valid point, in the case of GNU ELPA.
>
> But would that facility be limited to GNU ELPA?

If we wanted to release a stripped-down Emacs, which auto-installed a
pre-configured set of packages from ELPA, then I assume that you would
insist that it would be, for reasons of copyright if nothing else.

> Or would it also be used with other package archives
> that are not part of Emacs?

Emacs already has the ability to do this, but it is up to the user to
choose to do this.

Phil

Re: Auto-installing packages?

[Stefan Monnier]

> Yes. Although, in this case, I have put the packages I want to install
> into my .emacs.  So not "silent" at that point. I am doing something
> concrete.

Not "am doing": "have done".  It's quite different.
A big part of the prompt is to ask whether we should do it *now*.

> It's easy to add, although I suspect answering "Y" would become part of
> the muscule memory.

I doubt it (it's supposed to be something that occurs rarely, so muscle
memory probably wouldn't come into play), but in any case I don't think
it's a problem if it builds into someone's muscle memory.

> I forget how package.el works, but it inherits behaviour from that. So,
> it will update the package-list (i.e. call home) if your .emacs.d/elpa
> is empty.

It doesn't do asynchronously, currently.
It only does it in response to something like `M-x package-list-packages`.
[ And IIUC it currently doesn't do it in response to `M-x
  package-install` even it probably should.  ]


        Stefan

Re: Auto-installing packages?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

In response to

  > >   > So, for the case of Emacs and ELPA installing over the network vs
  > >   > pre-packaged is, I think, merely technical.

I wrote this:

  > > That is a valid point, in the case of GNU ELPA.
  > >
  > > But would that facility be limited to GNU ELPA?

  > > Or would it also be used with other package archives
  > > that are not part of Emacs?

You responded

  > Emacs already has the ability to do this, but it is up to the user to
  > choose to do this.

That doesn't address the issue I am raising.  It is a tangent.

Emacs has the ability to do any computation at all, if a user programs
it.  The issue here is not what Emacs "has the ability to do", but
what it offers/encourages/suggests/makes convenient.

If we are going to add a convenient facility to load external
packages, it should be for ELPA (which is more or less a part of
Emacs), not for other package archives (which are NOT part of Emacs).

In general, treating all package archives alike is something to avoid.
It is a pitfall.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Auto-installing packages?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

>   > > That is a valid point, in the case of GNU ELPA.
>   > >
>   > > But would that facility be limited to GNU ELPA?
>
>   > > Or would it also be used with other package archives
>   > > that are not part of Emacs?
>
> You responded
>
>   > Emacs already has the ability to do this, but it is up to the user to
>   > choose to do this.
>
> That doesn't address the issue I am raising.  It is a tangent.
>
> Emacs has the ability to do any computation at all, if a user programs
> it.  The issue here is not what Emacs "has the ability to do", but
> what it offers/encourages/suggests/makes convenient.
>
> If we are going to add a convenient facility to load external
> packages, it should be for ELPA (which is more or less a part of
> Emacs), not for other package archives (which are NOT part of Emacs).
>
> In general, treating all package archives alike is something to avoid.
> It is a pitfall.


I don't think it's a tangent. Emacs out of the box already favours ELPA,
and does not enable other archives. The default for additional mechanism
for loading external packages would be at least as restrictive --
i.e. out of the box, Emacs would only use this additional mechanism for
ELPA.

I would assume that anything else we added would also be usable with
other package archives. So, the GNU distribution of Emacs might, for
example, not ship with org-mode, but auto-install and then autoload it
on first usage. The same facility could be used by, for example,
spacemacs to auto-install and then auto-load any of the packages that
spacemacs provides.

It would be possible to make the use of an auto-install functionality
work straight-forwardly only with ELPA, but I can't see why you would.

Phil

Re: Auto-installing packages?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't think it's a tangent. Emacs out of the box already favours ELPA,
  > and does not enable other archives. The default for additional mechanism
  > for loading external packages would be at least as restrictive --
  > i.e. out of the box, Emacs would only use this additional mechanism for
  > ELPA.

I'm concerned that external packages, or other versions, would set a
variable to enable other package archives "by default".  This seems
like replacing a plaster wall that has a door, with a curtain made of
strings of beads.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)