Recursion problems with package management

Recursion problems with package management

[Rusi]

Collecting some thoughts and threads on package management here.
tl;dr at bottom

On Friday, August 14, 2015 at 2:01:08 AM UTC+5:30, Stefan Monnier wrote:
> On Thursday, August 13, 2015 at 12:55:26 PM UTC+5:30, Tomás wrote:
> > FWIW, and based on a recent experience of mine, yes, I think both ways
> > are needed/useful and complement each other. Recently, I installed some
> > package from ELPA (magit) and it failed to byte-compile. I've yet to
> > investigate what went wrong (perhaps my Emacs version is too new/old,
> > what have you), but I now find myself wrangling with the complexities
> > of the package itself *and* those of the packaging system.
> > 
> > So some "wholly integrated solution" makes life easier only when things
> > work out (duh ;-) Otherwise it makes life harder, and what's more important
> > in my view -- it tends to make a stronger separation of "outer circle"
> > and "inner circle", making it more difficult to get involved.

> My crystal ball tells me that you upgraded Magit from an Emacs where the
> previous Magit was already loaded, and the old (loaded) version
> interfered while byte-compiling the new version.
> [ The typical interference, looks like this: Emacs tries to compile
>   foo-1.el which does (require 'foo-2) to get some macros, but (the old)
>   `foo-2' is already loaded so we end up using the old version of the
>   macros (or the macros may simply not be defined at all).  ]

Here that we are at the root of the problem I think -- emacs package-managing 
emacs is a recursion problem.

In short, if emacs package-manager is weak enough it will work...  at
the cost of being 1/2-assed. If it is full-featured enough to be a basic,
all-round useful building block, recursion/meta-circularity needs to
be a (significant) factor in its design.

Obvious instance of the above: Imagine async.el is a package.
How to install async asyncly without async?

Here are some random (non emacs) instances of the above issue

1. apt-get upgrade apt-get works
What about 'apt-get install apt-get' 

2. Linux needs a shell, including to startup
So there is dash in addition to bash

3. Everything runs on top (transitively) the kernel.
But the kernel is a package that can be upgraded.
kernel install requires restart.
Which needs help of apt/grub etc.

4. The general idea, need, issues for bootloading-OS, bootstrapping-compiler

5. firefox plugin-install requires restart firefox. Why?


Coming to emacs package management

Edward's req-package usage-pattern has exemplified use thus
[see https://github.com/edvorg/emacs-configs ]
There is an init.el containing
(add-hook 'after-init-hook (lambda () (load "~/.emacs.d/init-real.el")))
And the real stuff is in init-real.el.

Related to the formidable list shown by Robert Thorpe
[thread "How can I change the default loading buffer"]

On Sunday, August 2, 2015 at 9:26:13 PM UTC+5:30, Robert Thorpe wrote:
> Emanuel Berg writes:
> > OK, so what is the method to find out (or rule of
> > thumb) what you shouldn't put directly into .emacs but
> > instead into `emacs-startup-hook'?
> 
> Have a look at (info "(elisp)Startup Summary") and see if any of it
> applies to you.  Check from the bottom of the list up.
> 
> You can only delay things as far at step 24, so there's little
> use in worrying much about the earlier steps.  For people who use the
> session manager and --daemon they could cause problems though.
> 
> This is how it is for me, going backwards, last step first:
> 
> * 24.  `emacs-startup-hook' and `term-setup-hook' are run.
> * 23.  initial-buffer-choice.  Because the initial-buffer-choice is set
> here, what happens in emacs-startup-hook can change it.  What happens in
> the init file can't though.  I use this feature.  I setup
> initial-buffer-choice to my ToDo file.  In emacs-startup-hook I look for
> a bookmark into my init file, if it exists then I jump to it.
> * 22.  Exit point for --batch.  If you use Emacs a lot in batch mode
> it's worth doing a lot of visual stuff in emacs-startup-hook because
> that way it won't slow down your batch programs.
> * 21.  Other command line options.  I think this includes files
> specified on the command line.
> * 20.  Initial echo area message.  Not very important.
> * 19.  Terminal specific lisp library.  This can be important if you use
> terminal emulation and need one of these.  I've never had to worry about
> this.
> * 18.  Set the mode of *scratch*.
> * 17.  Run after-init-hook.
> * 16.  Set after-init-time.  I think this is used in measuring how fast
> Emacs init is.  If you use after-init-hook or emacs-startup-hook then
> the stuff in those isn't measured.  In different situations you may want
> that or not want it.  You can always put everything into the init file
> temporarily just to see how long the full init takes.
> * 15.  Initialize packages.  This is important, packages aren't
> initialized in the init file.  There has been some discussion about
> changing this.  Many people change it by putting package-initialize in
> their init file.
> * 14.  Load abbrevs.  Abbrevs are rarely used programmatically so I
> don't think this matters much.
> * 13.  Load "default" library.  I've never used that.
> * 12.  Loading of the init file.

--------------------------------------------------------
tl;dr
Emacs package-managing emacs is thhe usual lure of Turing complete language can 
do anything does not mean doing it is appropriate.

If we must use elisp for it, it would be good to have a separate
'binding-time'. Just as in the C world compile time and runtime are
rigorously separated, it would be neat IMHO if elisp running in
package-mgmt mode did it in batch-mode and required restart of emacs.

Perhaps req-package is in the earlier binding time
And use-package the later one?

Re: Recursion problems with package management

[Ian Zimmerman]

On 2015-08-14 19:30 -0700, Rusi wrote:

> tl;dr
> Emacs package-managing emacs is thhe usual lure of Turing complete language can 
> do anything does not mean doing it is appropriate.
> 
> If we must use elisp for it, it would be good to have a separate
> 'binding-time'. Just as in the C world compile time and runtime are
> rigorously separated, it would be neat IMHO if elisp running in
> package-mgmt mode did it in batch-mode and required restart of emacs.
> 
> Perhaps req-package is in the earlier binding time
> And use-package the later one?

**WARNING** **WARNING** **WARNING**
Major snark mode on!

The answer is obvious: let distros do what they're supposed to do,
which is, um, packaging.  Doh!

Debian, for instance, already has the /etc/emacs/site-start.d directory
to do load ordering.  Now, there are three things I hear you screaming
right now:

1. The mechanism is hardly used right now, with everthing loading at the
same level (50).  That's just an opportunity to do much better!

2. The mechanism is pathetically weak, with all the problems of SysV
init on which it seems to be based.  Again, so replace it - but with
another (better) OS level thing, not with an Emacs thing.

3. Clearly we don't want maintainers to do separate packaging work for
each distro.  So, this is something that needs to be standardized among
the distros.  They _are_ willing to do that, when there's compelling
technical case for it.

In fact, I'm sending links to these threads to Rob Browning (Debian
Emacs maintainer) right now.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

Federated package management [was Recursion problems with package management]

[Rusi]

On Saturday, August 15, 2015 at 8:53:39 AM UTC+5:30, Ian Zimmerman wrote:
> On 2015-08-14 19:30 -0700, Rusi wrote:
> 
> > tl;dr
> > Emacs package-managing emacs is thhe usual lure of Turing complete language can 
> > do anything does not mean doing it is appropriate.
> > 
> > If we must use elisp for it, it would be good to have a separate
> > 'binding-time'. Just as in the C world compile time and runtime are
> > rigorously separated, it would be neat IMHO if elisp running in
> > package-mgmt mode did it in batch-mode and required restart of emacs.
> > 
> > Perhaps req-package is in the earlier binding time
> > And use-package the later one?
> 
> **WARNING** **WARNING** **WARNING**
> Major snark mode on!
> 
> The answer is obvious: let distros do what they're supposed to do,
> which is, um, packaging.  Doh!
> 
> Debian, for instance, already has the /etc/emacs/site-start.d directory
> to do load ordering.  Now, there are three things I hear you screaming
> right now:
> 
> 1. The mechanism is hardly used right now, with everthing loading at the
> same level (50).  That's just an opportunity to do much better!
> 
> 2. The mechanism is pathetically weak, with all the problems of SysV
> init on which it seems to be based.  Again, so replace it - but with
> another (better) OS level thing, not with an Emacs thing.
> 
> 3. Clearly we don't want maintainers to do separate packaging work for
> each distro.  So, this is something that needs to be standardized among
> the distros.  They _are_ willing to do that, when there's compelling
> technical case for it.
> 
> In fact, I'm sending links to these threads to Rob Browning (Debian
> Emacs maintainer) right now.

We are talking of something interesting and useful but another thing now
-- federated package management.

[Just trying to keep the threads distinct by subject]

About need for FPM:
Lets say (for discussion not trolling!) that the debian apt system is the
best package mgmt tool.  However it is by definition *downstream* and therefore
behind the cutting/bleeding edge.

Conversely every non-trivial large system has evolved its own package management
subsystem.
egs. Haskell-hackage, Python-Pypy, Ruby-gems, Tex-texmgr, firefox-plugins
Likewise emacs.

Most of these are woefully inadequate compared to apt

So just as ONE package-mgmt system makes no sense unless there are a plurality
of packages, I'd like to suggest that DIFFERENT package management systems would
be more useful if they cooperated rather than competed.

Something along the lines that one should be able to do
(along the lines of git submodule)

# apt-get submodule emacs install magit

and apt-get should chain-out to the emacs package system

Re: Federated package management [was Recursion problems with package management]

[Rusi]

On Saturday, August 15, 2015 at 8:53:39 AM UTC+5:30, Ian Zimmerman wrote:
> 3. Clearly we don't want maintainers to do separate packaging work for
> each distro.  So, this is something that needs to be standardized among
> the distros.  They _are_ willing to do that, when there's compelling
> technical case for it.

And what is the fate of emacs users on package-manager-less OSes?

[Yeah snarky comments about Windows are appropriate here.
Except for the fact that well-known distros like slackware would also belong!
]