Replace 'pp' with 'prin1' to generate ELPA archive-contents

Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]


I've just tried replacing the 'pp' call with 'prin1' in elpa-admin's
`elpaa-update-archive-contents', and on my checkout the size dropped
from ~130 kb to 100 kb.  It is not much, but I don't understand why the
archive contents should be pretty printed, and even a few kilobyte less
can make a difference for some people.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-elpa-admin.el-elpaa-update-archive-contents-Use-prin.patch --]
[-- Type: text/x-patch, Size: 847 bytes --]

From 592e208beae6f921772967ab2b800606a647899b Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Sat, 22 Oct 2022 12:49:49 +0200
Subject: [PATCH] * elpa-admin.el (elpaa--update-archive-contents): Use 'prin1'

---
 elpa-admin.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/elpa-admin.el b/elpa-admin.el
index 00c3393499..f358fe0d68 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -167,7 +167,7 @@ Delete backup files also."
                            (string-lessp (symbol-name (car x)) (symbol-name (car y))))))
     (elpaa--message "new AC: %S" ac)
     (with-temp-buffer
-      (pp ac (current-buffer))
+      (prin1 ac (current-buffer))
       (write-region nil nil filename)
       (let ((default-directory (expand-file-name dir)))
         (elpaa--html-make-index (cdr ac))))))
-- 
2.38.0

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> I've just tried replacing the 'pp' call with 'prin1' in elpa-admin's
> `elpaa-update-archive-contents', and on my checkout the size dropped
> from ~130 kb to 100 kb.  It is not much, but I don't understand why the
> archive contents should be pretty printed, and even a few kilobyte less
> can make a difference for some people.

If we care about 30kb, how adding gzip support instead?

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> I've just tried replacing the 'pp' call with 'prin1' in elpa-admin's
>> `elpaa-update-archive-contents', and on my checkout the size dropped
>> from ~130 kb to 100 kb.  It is not much, but I don't understand why the
>> archive contents should be pretty printed, and even a few kilobyte less
>> can make a difference for some people.
>
> If we care about 30kb, how adding gzip support instead?

That is a good point, I have tried out a few different compression
algorithms on the archive contents generated with `prin1', and it seems
that bz2 would give the best results -- though in the end there is just
a difference of ±2kb:

  -rw-r--r-- 1 philip philip   100950 Oct 22 12:49 archive-contents
  -rw-r--r-- 1 philip philip    23952 Oct 22 12:49 archive-contents.bz2
  -rw-r--r-- 1 philip philip    28537 Oct 22 12:49 archive-contents.gz
  -rw-r--r-- 1 philip philip    25080 Oct 22 12:49 archive-contents.xz
  -rw-r--r-- 1 philip philip    26997 Oct 22 12:49 archive-contents.zst

I don't know if it makes sense to provide all of these, in case the
client doesn't have bz2 installed or if the overhead of trying out
multiple HTTP requests would outweigh the advantage of compressing in
the first place.

Another thing I just thought of is that HTTP can serve gzip'ed content,
but unless I am mistaken the elpa.gnu.org server doesn't make use of
this capability, even though url.el appears to send the right header
(Accept-Encoding: gzip).  My guess would be that enabling this would
improve the situation immediately more than any hacking on
elpa-admin.el/package.el.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

Stefan Kangas [2022-10-22 04:56:47] wrote:
> Philip Kaludercic <philipk@posteo.net> writes:
>> I've just tried replacing the 'pp' call with 'prin1' in elpa-admin's
>> `elpaa-update-archive-contents', and on my checkout the size dropped
>> from ~130 kb to 100 kb.  It is not much, but I don't understand why the
>> archive contents should be pretty printed, and even a few kilobyte less
>> can make a difference for some people.
> If we care about 30kb, how adding gzip support instead?

Gzippping this file and the tarballs would be a very welcome
change, yes.  The main issue is to make sure all the expected clients
can decompress those files.

IOW we need to add support for that to `package.el`.
The sooner the better.

AFAIK it's pretty easy to add support to Apache to auto-generate the
`.gz` files on the fly.  I tried setting up the server to do the
opposite (automatically serving the uncompressed version of the file
when the file name without `.gz` is requested) but couldn't get it
to work.


        Stefan

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> I have tried out a few different compression algorithms on the archive
> contents generated with `prin1', and it seems that bz2 would give the
> best results -- though in the end there is just a difference of ±2kb:

The main benefit of using gzip is that it's more portable, as it's
almost always installed on any given system.  Given the small file
sizes, that consideration is probably more important than the exact
compression ratio.

> Another thing I just thought of is that HTTP can serve gzip'ed
> content, but unless I am mistaken the elpa.gnu.org server doesn't make
> use of this capability, even though url.el appears to send the right
> header (Accept-Encoding: gzip).  My guess would be that enabling this
> would improve the situation immediately more than any hacking on
> elpa-admin.el/package.el.

Makes sense.  Did you check what melpa.org does about that, BTW?

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Stefan Kangas [2022-10-22 04:56:47] wrote:
>> Philip Kaludercic <philipk@posteo.net> writes:
>>> I've just tried replacing the 'pp' call with 'prin1' in elpa-admin's
>>> `elpaa-update-archive-contents', and on my checkout the size dropped
>>> from ~130 kb to 100 kb.  It is not much, but I don't understand why the
>>> archive contents should be pretty printed, and even a few kilobyte less
>>> can make a difference for some people.
>> If we care about 30kb, how adding gzip support instead?
>
> Gzippping this file and the tarballs would be a very welcome
> change, yes.  The main issue is to make sure all the expected clients
> can decompress those files.
>
> IOW we need to add support for that to `package.el`.
> The sooner the better.
>
> AFAIK it's pretty easy to add support to Apache to auto-generate the
> `.gz` files on the fly.  I tried setting up the server to do the
> opposite (automatically serving the uncompressed version of the file
> when the file name without `.gz` is requested) but couldn't get it
> to work.

I am not an Apache expert, but from what I read the "SetOutputFilter
DEFLATE" directive should enable gzip compression for requests where the
client supports it.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> I have tried out a few different compression algorithms on the archive
>> contents generated with `prin1', and it seems that bz2 would give the
>> best results -- though in the end there is just a difference of ±2kb:
>
> The main benefit of using gzip is that it's more portable, as it's
> almost always installed on any given system.  Given the small file
> sizes, that consideration is probably more important than the exact
> compression ratio.

Yes, right.

>> Another thing I just thought of is that HTTP can serve gzip'ed
>> content, but unless I am mistaken the elpa.gnu.org server doesn't make
>> use of this capability, even though url.el appears to send the right
>> header (Accept-Encoding: gzip).  My guess would be that enabling this
>> would improve the situation immediately more than any hacking on
>> elpa-admin.el/package.el.
>
> Makes sense.  Did you check what melpa.org does about that, BTW?

I hadn't previously, but they appear to do so.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

>> AFAIK it's pretty easy to add support to Apache to auto-generate the
>> `.gz` files on the fly.  I tried setting up the server to do the
>> opposite (automatically serving the uncompressed version of the file
>> when the file name without `.gz` is requested) but couldn't get it
>> to work.
>
> I am not an Apache expert, but from what I read the "SetOutputFilter
> DEFLATE" directive should enable gzip compression for requests where the
> client supports it.

We already did that for `.tar` files (and a few others).
I have now changed the config to also apply compression to
`archive-contents` (and to the new `elpa-packages.eld`).


        Stefan

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> AFAIK it's pretty easy to add support to Apache to auto-generate the
>>> `.gz` files on the fly.  I tried setting up the server to do the
>>> opposite (automatically serving the uncompressed version of the file
>>> when the file name without `.gz` is requested) but couldn't get it
>>> to work.
>>
>> I am not an Apache expert, but from what I read the "SetOutputFilter
>> DEFLATE" directive should enable gzip compression for requests where the
>> client supports it.
>
> We already did that for `.tar` files (and a few others).

Aren't the .tar files already compressed?

> I have now changed the config to also apply compression to
> `archive-contents` (and to the new `elpa-packages.eld`).

Great!

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
> Date: Wed, 26 Oct 2022 06:43:20 +0000
> 
> Aren't the .tar files already compressed?

No, they are basically just a concatenation of the original files'
contents, with some headers that describe the files' names and
attributes.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
>> Date: Wed, 26 Oct 2022 06:43:20 +0000
>> 
>> Aren't the .tar files already compressed?
>
> No, they are basically just a concatenation of the original files'
> contents, with some headers that describe the files' names and
> attributes.

Of course, I meant to ask whether elpa.gnu.org was serving .tar.gz (or
some other format) files or just plain tarballs.  It turns out that this
is not the case, which means that the HTTP compression is not
pointless.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

>> We already did that for `.tar` files (and a few others).
>
> Aren't the .tar files already compressed?

No, otherwise they'd be `.tar.gz`, `.tar.lz`, etc...
We don't compress them so that you can install them even from Emacsen
that don't have access to a `.gz` decompressor.

And yes, it would be nice to change this so that `elpa.gnu.org` could
store the tarballs compressed and have Apache decompress them on the
rare occasion where the client doesn't have a `.gz` decompressor.
But I couldn't figure out how to convince Apache to do that (see
https://stackoverflow.com/questions/26895894/get-apache-to-auto-decompress-a-file-but-only-if-needed
for what happened last time I tried).


        Stefan

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
> Date: Wed, 26 Oct 2022 08:10:13 -0400
> 
> And yes, it would be nice to change this so that `elpa.gnu.org` could
> store the tarballs compressed and have Apache decompress them on the
> rare occasion where the client doesn't have a `.gz` decompressor.

How about making zlib a required library for Emacs?  I think most
systems have that nowadays, and we could provide our copy of the
library as part of our sources for those who might not have it.
Several projects I regularly build already do that, AFAICT.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Po Lu]

On October 26, 2022 8:18:20 PM GMT+08:00, Eli Zaretskii <eliz@gnu.org> wrote:

>How about making zlib a required library for Emacs?

This would be a little drastic, but what about not supporting package installation from ELPA if zlib isn't installed?

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> No, otherwise they'd be `.tar.gz`, `.tar.lz`, etc...
> We don't compress them so that you can install them even from Emacsen
> that don't have access to a `.gz` decompressor.

Could we provide both the .tar and .tar.gz, and leave it up to
package.el to download whichever one is supported?

Or do we have very limited disk space?

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

Po Lu [2022-10-26 22:11:38] wrote:
> On October 26, 2022 8:18:20 PM GMT+08:00, Eli Zaretskii <eliz@gnu.org> wrote:
>>How about making zlib a required library for Emacs?
> This would be a little drastic,

zlib is ON by default, is required internally by gnutls which is also ON
by default, and is a requirement for the native compiler as well (which
is not ON by default), so it's not *that* drastic.

> but what about not supporting package installation from ELPA if zlib
> isn't installed?

Maybe if we relax this to "either compiled with zlib or can find `gzip`
in `exec-path`", it might be tolerable.


        Stefan

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

Stefan Kangas [2022-10-26 07:33:56] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> No, otherwise they'd be `.tar.gz`, `.tar.lz`, etc...
>> We don't compress them so that you can install them even from Emacsen
>> that don't have access to a `.gz` decompressor.
> Could we provide both the .tar and .tar.gz, and leave it up to
> package.el to download whichever one is supported?

I'd rather not.

> Or do we have very limited disk space?

There's that as well.


        Stefan

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Robert Pluim]

>>>>> On Wed, 26 Oct 2022 11:02:00 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:

    Stefan> Po Lu [2022-10-26 22:11:38] wrote:
    >> On October 26, 2022 8:18:20 PM GMT+08:00, Eli Zaretskii <eliz@gnu.org> wrote:
    >>> How about making zlib a required library for Emacs?
    >> This would be a little drastic,

    Stefan> zlib is ON by default, is required internally by gnutls which is also ON
    Stefan> by default, and is a requirement for the native compiler as well (which
    Stefan> is not ON by default), so it's not *that* drastic.

And letʼs not forget that the default value of `package-archives'
contains https URLs, so anyone building Emacs without GnuTLS is
already causing themselves problems with package downloads.

    >> but what about not supporting package installation from ELPA if zlib
    >> isn't installed?

    Stefan> Maybe if we relax this to "either compiled with zlib or can find `gzip`
    Stefan> in `exec-path`", it might be tolerable.

The latter would imply http downloads only for packages, which is not
something we should promote.

Robert
-- 

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

Robert Pluim [2022-10-26 17:18:49] wrote:

>>>>>> On Wed, 26 Oct 2022 11:02:00 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:
>
>     Stefan> Po Lu [2022-10-26 22:11:38] wrote:
>     >> On October 26, 2022 8:18:20 PM GMT+08:00, Eli Zaretskii <eliz@gnu.org> wrote:
>     >>> How about making zlib a required library for Emacs?
>     >> This would be a little drastic,
>
>     Stefan> zlib is ON by default, is required internally by gnutls which is also ON
>     Stefan> by default, and is a requirement for the native compiler as well (which
>     Stefan> is not ON by default), so it's not *that* drastic.
>
> And letʼs not forget that the default value of `package-archives'
> contains https URLs, so anyone building Emacs without GnuTLS is
> already causing themselves problems with package downloads.

Not really:

(defcustom package-archives `(("gnu" .
                               ,(format "http%s://elpa.gnu.org/packages/"
                                        (if (gnutls-available-p) "s" "")))
                              ("nongnu" .
                               ,(format "http%s://elpa.nongnu.org/nongnu/"
                                        (if (gnutls-available-p) "s" ""))))

>     Stefan> Maybe if we relax this to "either compiled with zlib or can find `gzip`
>     Stefan> in `exec-path`", it might be tolerable.
>
> The latter would imply http downloads only for packages, which is not
> something we should promote.

We're not promoting them, we're not just doing all we can to
prevent them.


        Stefan

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Robert Pluim]

>>>>> On Wed, 26 Oct 2022 11:25:16 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:

    >> And letʼs not forget that the default value of `package-archives'
    >> contains https URLs, so anyone building Emacs without GnuTLS is
    >> already causing themselves problems with package downloads.

    Stefan> Not really:

    Stefan> (defcustom package-archives `(("gnu" .
    Stefan>                                ,(format "http%s://elpa.gnu.org/packages/"
    Stefan>                                         (if (gnutls-available-p) "s" "")))
    Stefan>                               ("nongnu" .
    Stefan>                                ,(format "http%s://elpa.nongnu.org/nongnu/"
    Stefan>                                         (if (gnutls-available-p) "s" ""))))

Oh, Iʼd looked at the result, not the source. But I guess itʼs kind of
a moot point, as I doubt there are any distributions that donʼt build
Emacs with GnuTLS. [1]

Robert

Footnotes:
[1]  A dozen will now be pointed out to me :-)

-- 

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> zlib is ON by default, is required internally by gnutls which is also ON
> by default, and is a requirement for the native compiler as well (which
> is not ON by default), so it's not *that* drastic.

Neither of those three features are supported on MS-DOS, along with
networking in general, so I doubt the package system works there in the
first place.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> Oh, Iʼd looked at the result, not the source. But I guess itʼs kind of
> a moot point, as I doubt there are any distributions that donʼt build
> Emacs with GnuTLS.

FWIW, the old GNU/Linux distributions (Fedora 10, 9, 7) that I use to
test backwards compatibility in the X build ship a GnuTLS that doesn't
work with modern Emacs.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Robert Pluim]

>>>>> On Thu, 27 Oct 2022 08:50:38 +0800, Po Lu <luangruo@yahoo.com> said:

    Po Lu> Robert Pluim <rpluim@gmail.com> writes:
    >> Oh, Iʼd looked at the result, not the source. But I guess itʼs kind of
    >> a moot point, as I doubt there are any distributions that donʼt build
    >> Emacs with GnuTLS.

    Po Lu> FWIW, the old GNU/Linux distributions (Fedora 10, 9, 7) that I use to
    Po Lu> test backwards compatibility in the X build ship a GnuTLS that doesn't
    Po Lu> work with modern Emacs.

The newest of those is from 2008. I highly doubt anybody still running
such a release wants a modern Emacs.

Robert
-- 

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> The newest of those is from 2008. I highly doubt anybody still running
> such a release wants a modern Emacs.

I do.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Robert Pluim]

>>>>> On Thu, 27 Oct 2022 18:38:23 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Robert Pluim <rpluim@gmail.com> writes:
    >> The newest of those is from 2008. I highly doubt anybody still running
    >> such a release wants a modern Emacs.

    Po> I do.

To do actual work? OK, but you know how to work around the consequences.

Robert
-- 

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> To do actual work?

If making sure Emacs builds with old versions of X libraries counts,
yes.

Re: Replace 'pp' with 'prin1' to generate ELPA archive-contents

[Stefan Monnier]

Po Lu [2022-10-27 08:49:03] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> zlib is ON by default, is required internally by gnutls which is also ON
>> by default, and is a requirement for the native compiler as well (which
>> is not ON by default), so it's not *that* drastic.
> Neither of those three features are supported on MS-DOS, along with
> networking in general,

Good: it means that the use of zlib would have no effect on MS-DOS.

> so I doubt the package system works there in the first place.

I expect that `list-packages` doesn't work, nor does `package-install`,
but you should still be able to use `package-install-(file|from-buffer)`.


        Stefan