Dealing with a system Emacs properly

Dealing with a system Emacs properly

[David Masterson]

I recently upgraded Linux on my Chromebook to Debian 12.5 (bookworm),
This included upgrading Emacs from 27.1 to 28.2 in the installation of
Debian.  I have been installing packages for Emacs from (M)Elpa into my
~/.emacs.d/elpa directory.  Recently, I noticed that Org was at v9.5.5
whereas the latest one was 9.7.6 in list-packages.  I also noticed that
there were several org-*.signed files in my elpa dir suggesting that
auto-update had tried to update Org and failed.  I also noticed that Org
v9.5.5 is installed in /usr/share/emacs/28.2/lisp/org (ie. not a package
versioned directory).

Is there a recommended method of installing Elpa packages (like Org)
into my home directory such that they properly override the Debian
system package? Am I supposed to remove by hand the system package?

-- 
David Masterson

Re: Dealing with a system Emacs properly

[Joel Reicher]

David Masterson <dsmasterson@gmail.com> writes:

> I recently upgraded Linux on my Chromebook to Debian 12.5 
> (bookworm), This included upgrading Emacs from 27.1 to 28.2 in 
> the installation of Debian.  I have been installing packages for 
> Emacs from (M)Elpa into my ~/.emacs.d/elpa directory.  Recently, 
> I noticed that Org was at v9.5.5 whereas the latest one was 
> 9.7.6 in list-packages.

Are you saying a "personal" installation of Org was 
discarded/downgraded/ignored when you upgraded Emacs, or are you 
saying that an upgrade using, say, "package-upgrade-all" did not 
upgrade Org?

The latter is normal behaviour for builtin packages...

       By contrast, ‘package-upgrade’ and ‘package-upgrade-all’ 
       never
       consider built-in packages.  If you want to use these 
       commands for
       upgrading some built-in packages, you need to upgrade each 
       of those
       packages, once, either via ‘C-u M-x package-install <RET>’, 
       or by
       customizing ‘package-install-upgrade-built-in’ to a 
       non-‘nil’ value, and
       then upgrading the package once via the package menu or by
       ‘package-install’.

But if it's the former, do you have some aspect of your package 
management customised? A package quickstart or load list?

> Is there a recommended method of installing Elpa packages (like 
> Org) into my home directory such that they properly override the 
> Debian system package? Am I supposed to remove by hand the 
> system package?

No, that should not be necessary. A vanilla package install should 
shadow a builtin.

Regards,

        - Joel

Re: Dealing with a system Emacs properly

[Michel Verdier]

On 2024-07-12, David Masterson wrote:

> I recently upgraded Linux on my Chromebook to Debian 12.5 (bookworm),
> This included upgrading Emacs from 27.1 to 28.2 in the installation of
> Debian. I have been installing packages for Emacs from (M)Elpa into my
> ~/.emacs.d/elpa directory.  Recently, I noticed that Org was at v9.5.5
> whereas the latest one was 9.7.6 in list-packages.  I also noticed that

You could use bookworm-backports to get 29.3 installed. I also get org
built-in at 9.6.15. If you want more on debian you have to use testing
version (trixie).

Re: Dealing with a system Emacs properly

[David Masterson]

Joel Reicher <joel.reicher@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> I recently upgraded Linux on my Chromebook to Debian 12.5
>> (bookworm), This included upgrading Emacs from 27.1 to 28.2 in the
>> installation of Debian.  I have been installing packages for Emacs
>> from (M)Elpa into my ~/.emacs.d/elpa directory.  Recently, I noticed
>> that Org was at v9.5.5 whereas the latest one was 9.7.6 in
>> list-packages.
>
> Are you saying a "personal" installation of Org was
> discarded/downgraded/ignored when you upgraded Emacs, or are you
> saying that an upgrade using, say, "package-upgrade-all" did not
> upgrade Org?

I have the package auto-package-update which regularly checks my
packages for updates on [M]Elpa. I had package-install(ed) Org ~v8.0 and
this package had been regularly updating it.  Along the way, I found my
local Org had a conflict with the system installed Org and I think I
removed the system Org (in Emacs 27.1) to get around that.  When I
upgraded Debian to get Emacs 28.2, I think something similar happened.
I didn't notice until I saw a few "org-*.signed" files in my elpa dir
with no corresponding dir suggesting that auto-package-update had tried
and failed to update Org, but, if there were warnings/errors, I ignored
them among other package updates because it was usually ok to do
that. Perhaps package

> The latter is normal behaviour for builtin packages...

Latter?  I don't see package-upgrade-all in describe-function.  

>       By contrast, ‘package-upgrade’ and ‘package-upgrade-all’ never
>       consider built-in packages.  If you want to use these commands
>       for upgrading some built-in packages, you need to upgrade each
>       of those packages, once, either via ‘C-u M-x package-install
>       <RET>’, or by customizing ‘package-install-upgrade-built-in’ to
>       a non-‘nil’ value, and then upgrading the package once via the
>       package menu or by ‘package-install’.

Hmm. I don't think that will work because the system package is not in a
versioned directory.  Also, you'd have to run Emacs under sudo.

> But if it's the former, do you have some aspect of your package
> management customised? A package quickstart or load list?

I use use-package to load all the interesting packages.  I'm trying to
bulletproof my .emacs so that I can use it on multiple systems and have
all the packages loaded as necessary, WIP

>> Is there a recommended method of installing Elpa packages (like Org)
>> into my home directory such that they properly override the Debian
>> system package? Am I supposed to remove by hand the system package?
>
> No, that should not be necessary. A vanilla package install should
> shadow a builtin.

Okay, by shadow you mean cover.  Most of the time, that's true.  Once in
awhile, there's some problem.

-- 
David Masterson

Re: Dealing with a system Emacs properly

[David Masterson]

Michel Verdier <mv524@free.fr> writes:

> On 2024-07-12, David Masterson wrote:
>
>> I recently upgraded Linux on my Chromebook to Debian 12.5 (bookworm),
>> This included upgrading Emacs from 27.1 to 28.2 in the installation of
>> Debian. I have been installing packages for Emacs from (M)Elpa into my
>> ~/.emacs.d/elpa directory.  Recently, I noticed that Org was at v9.5.5
>> whereas the latest one was 9.7.6 in list-packages.  I also noticed that
>
> You could use bookworm-backports to get 29.3 installed. I also get org
> built-in at 9.6.15. If you want more on debian you have to use testing
> version (trixie).

Can you explain 'backports' (or point me to a reference)?

-- 
David Masterson

Re: Dealing with a system Emacs properly

[Joel Reicher]

David Masterson <dsmasterson@gmail.com> writes:

> Joel Reicher <joel.reicher@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>> I recently upgraded Linux on my Chromebook to Debian 12.5 
>>> (bookworm), This included upgrading Emacs from 27.1 to 28.2 in 
>>> the installation of Debian.  I have been installing packages 
>>> for Emacs from (M)Elpa into my ~/.emacs.d/elpa directory. 
>>> Recently, I noticed that Org was at v9.5.5 whereas the latest 
>>> one was 9.7.6 in list-packages.
>>
>> Are you saying a "personal" installation of Org was 
>> discarded/downgraded/ignored when you upgraded Emacs, or are 
>> you saying that an upgrade using, say, "package-upgrade-all" 
>> did not upgrade Org?
>
> I have the package auto-package-update which regularly checks my 
> packages for updates on [M]Elpa.

I don't use that anymore, but I can see it draws on both 
package-alist and package--builtins so I'd expect it to work for 
upgrading builtins also. Can you check the values of those 
variables if the issue is still happening or happens again?

> I had package-install(ed) Org ~v8.0 and this package had been 
> regularly updating it.  Along the way, I found my local Org had 
> a conflict with the system installed Org and I think I removed 
> the system Org (in Emacs 27.1) to get around that.  When I 
> upgraded Debian to get Emacs 28.2, I think something similar 
> happened.  I didn't notice until I saw a few "org-*.signed" 
> files in my elpa dir with no corresponding dir suggesting that 
> auto-package-update had tried and failed to update Org, but, if 
> there were warnings/errors, I ignored them among other package 
> updates because it was usually ok to do that. Perhaps package

I think it should have been possible to do whatever needed doing 
without removing or altering files from the "system paths", but 
it's hard for me to guess what might have happened.

>> The latter is normal behaviour for builtin packages...
>
> Latter?  I don't see package-upgrade-all in describe-function.

Possibly it's not in the version you have?

https://www.gnu.org/software/emacs/manual/html_node/emacs/Package-Installation.html#index-package_002dupgrade

>>> Is there a recommended method of installing Elpa packages 
>>> (like Org) into my home directory such that they properly 
>>> override the Debian system package? Am I supposed to remove by 
>>> hand the system package?
>>
>> No, that should not be necessary. A vanilla package install 
>> should shadow a builtin.
>
> Okay, by shadow you mean cover.  Most of the time, that's true. 
> Once in awhile, there's some problem.

Could you provide specifics? I believe redefinition and overriding 
(of anything!) should always be possible in Emacs.

Regards,

        - Joel

Re: Dealing with a system Emacs properly

[Michel Verdier]

On 2024-07-13, David Masterson wrote:

>> You could use bookworm-backports to get 29.3 installed. I also get org
>> built-in at 9.6.15. If you want more on debian you have to use testing
>> version (trixie).
>
> Can you explain 'backports' (or point me to a reference)?

bookworm-backports contains packages from testing (trixie) but compiled
with stable (bookworm) librairies. So you can install those packages on a
stable debian and get newer versions. With this you have the best: stable
and (almost) up-to-date. This does not address the built-in problem but
perhaps mitigate it.

https://wiki.debian.org/Backports
(contains install informations)

Re: Dealing with a system Emacs properly

[David Masterson]

Joel Reicher <joel.reicher@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Joel Reicher <joel.reicher@gmail.com> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>> I have the package auto-package-update which regularly checks my
>> packages for updates on [M]Elpa.
>
> I don't use that anymore, but I can see it draws on both package-alist
> and package--builtins so I'd expect it to work for upgrading builtins
> also. Can you check the values of those variables if the issue is
> still happening or happens again?

Hmm. I see Org-9.5.5 in package--builtins even after I removed it from
/usr/share/emacs/elpa (well, hid it in /usr/share/emacs) to try to see
to see if I could local install Org-9.7.6 (which seemed to work).  How
did package--builtins find 9.5.5 ?!?

>>> The latter is normal behaviour for builtin packages...
>>
>> Latter?  I don't see package-upgrade-all in describe-function.
>
> Possibly it's not in the version you have?
>
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Package-Installation.html#index-package_002dupgrade

Ah, yeah.  The manual your pointing to is for Emacs 29.2 while I'm at
28.2.  I'll have to look at Debian backports to see if I can install an
Emacs v29 in bookworm.  Building Emacs from scratch myself winds up
loading too many dependent system software which overloads my small
diskspace on my Chromebook.

>> Okay, by shadow you mean cover.  Most of the time, that's true. Once
>> in awhile, there's some problem.
>
> Could you provide specifics? I believe redefinition and overriding (of
> anything!) should always be possible in Emacs.

Don't remember the specifics, but I believe it had something to with
org-compat.el and that the builtin org-compat was being loaded before
the new Org was loaded and, therefore, interfered with the load of the
new Org.  Org-compat is supposed to avoid this, but an update to Org
broke it.  My hack was to remove the old Org instead of waiting for the
fix. 

-- 
David Masterson

Re: Dealing with a system Emacs properly

[David Masterson]

Michel Verdier <mv524@free.fr> writes:

> On 2024-07-13, David Masterson wrote:
>
>>> You could use bookworm-backports to get 29.3 installed. I also get org
>>> built-in at 9.6.15. If you want more on debian you have to use testing
>>> version (trixie).
>>
>> Can you explain 'backports' (or point me to a reference)?
>
> bookworm-backports contains packages from testing (trixie) but compiled
> with stable (bookworm) librairies. So you can install those packages on a
> stable debian and get newer versions. With this you have the best: stable
> and (almost) up-to-date. This does not address the built-in problem but
> perhaps mitigate it.
>
> https://wiki.debian.org/Backports
> (contains install informations)

Thanks!

-- 
David Masterson