Re: Preferred approach to inclusion of data in ELPA package

[Hugo Thunnissen <devel@hugot.nl>]

Philip Kaludercic <philipk@posteo.net> writes:

> Hugo Thunnissen <devel@hugot.nl> writes:
>
>> On 8/19/23 17:51, Philip Kaludercic wrote:
>>> Hugo Thunnissen <devel@hugot.nl> writes:
>>>
>>>> On 8/17/23 23:14, Philip Kaludercic wrote:
>>>>> Another idea is to have a Makefile generate the file, like the one you
>>>>> describe in option 2., that is generate whenever the package is built
>>>>> and bundled into a tarball for distribution.  That way you don't have to
>>>>> store a binary blob in your repository, and you can avoid burdening the
>>>>> user with additional computations at either compile or runtime.
>>>>>
>>>>> Does the generation require any special functionality/tools/code to be
>>>>> provided on the device the index is generated on?
>>>> The php function/class stubs are generated with a php script, but I'm
>>>> checking the resulting stubs file into git. The index itself can be
>>>> built with just my package based on the stubs file.
>>> I saw that, and the commit did not look that nice, but I cannot say that
>>> I have looked into the issue in sufficient detail to say with certainty
>>> or not that there is no better solution.
>>>
>> There are alternatives or improvements to this approach, which I
>> mentioned in my response to sbaug
>> (https://lists.gnu.org/archive/html/emacs-devel/2023-08/msg00748.html)
>> . I don't think I'll be able to get around having to distribute-, or
>> having the user download/generate some kind of index though.
>
> If PHP were available in the ELPA build environment, do you think that
> would change anything?
>

Theoretically, yes, as I would not have to check the stubs into version
control. Is this a realistic scenario though? To generate stubs, I
currently require a PHP installation with a whole slew of extensions
installed to be able to generate stubs for them. Currently, I only do
this for PHP 8.2, but I will probably end up having to do this for more
versions of PHP. That is a whole lot of packages that would have to be
present in ELPA's build environment. And the list of extensions is not
guaranteed to be static either.

My current idea is to create container images for each version of PHP
and generate stub files by executing the generation scripts in these
containers. To give you an idea of the list of required packages, this
is an example of the installation step in a Dockerfile I'm using:

RUN apt-get update && apt-get -y install \
    php8.2-memcached \
    php-redis \
    php8.2-bcmath \
    php8.2-bz2 \
    php8.2-cli \
    php8.2-common \
    php8.2-curl \
    php8.2-gmp \
    php8.2-intl \
    php-json \
    php8.2-mbstring \
    php8.2-mysql \
    php8.2-odbc \
    php8.2-opcache \
    php8.2-pgsql \
    php8.2-readline \
    php8.2-tidy \
    php8.2-xml \
    php8.2-xsl \
    php8.2-zip \
    php8.2-gd \
    php-bcmath \
    php-apcu \
    php-cli \
    php-imagick \
    php-intl \
    php-xdebug \
    php-amqp

>>>> Also: If the former is the case, is the reduction in load time that
>>>> this brings even significant enough to be worth the bother or should I
>>>> just hold off on this while I look for a more efficient solution?
>>> I'd say it would be worth it, if the resulting package would be smaller
>>> and would load quicker.  After all, the performance on your laptop might
>>> not be that significant of a difference, while for someone else with an
>>> older or slower device, a 30%-speedup is pretty significant.
>>
>> You're right, on less performant systems it could make a more
>> significant difference. And good news, after a few improvements the
>> load time is now down to ~150ms on my laptop. I also chose to load the
>> data when the mode is first initialized, so simply loading my package
>> won't cause the index to be loaded with it. The dumping of the index
>> is done automatically when it is not present, so it would technically
>> be fine to just distribute the PHP stubs with the package instead of
>> the .eld index file. This would just make the user wait a little
>> longer the first time they use the mode.
>
> Would it be possible to load the information when it is required
> (e.g. necessary for completion)?

This is a good idea. Classes within the project's scope are already
parsed/indexed on demand like that, I should apply this to stub classes
as well.

Global/native functions are another story though. Since they're not
namespaced, it's hard to break them into smaller loadable sets. Aside
from that, function completion generally requires the list of functions
to be loaded indiscriminately.  Luckily, the functions only make up a
little over 1/4th of the stubs (~2000 out of ~7300 lines).