Re: Preferred approach to inclusion of data in ELPA package

[Philip Kaludercic <philipk@posteo.net>]

Hugo Thunnissen <devel@hugot.nl> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Hugo Thunnissen <devel@hugot.nl> writes:
>>
>>> On 8/19/23 17:51, Philip Kaludercic wrote:
>>>> Hugo Thunnissen <devel@hugot.nl> writes:
>>>>
>>>>> On 8/17/23 23:14, Philip Kaludercic wrote:
>>>>>> Another idea is to have a Makefile generate the file, like the one you
>>>>>> describe in option 2., that is generate whenever the package is built
>>>>>> and bundled into a tarball for distribution.  That way you don't have to
>>>>>> store a binary blob in your repository, and you can avoid burdening the
>>>>>> user with additional computations at either compile or runtime.
>>>>>>
>>>>>> Does the generation require any special functionality/tools/code to be
>>>>>> provided on the device the index is generated on?
>>>>> The php function/class stubs are generated with a php script, but I'm
>>>>> checking the resulting stubs file into git. The index itself can be
>>>>> built with just my package based on the stubs file.
>>>> I saw that, and the commit did not look that nice, but I cannot say that
>>>> I have looked into the issue in sufficient detail to say with certainty
>>>> or not that there is no better solution.
>>>>
>>> There are alternatives or improvements to this approach, which I
>>> mentioned in my response to sbaug
>>> (https://lists.gnu.org/archive/html/emacs-devel/2023-08/msg00748.html)
>>> . I don't think I'll be able to get around having to distribute-, or
>>> having the user download/generate some kind of index though.
>>
>> If PHP were available in the ELPA build environment, do you think that
>> would change anything?
>>
>
> Theoretically, yes, as I would not have to check the stubs into version
> control. Is this a realistic scenario though? To generate stubs, I
> currently require a PHP installation with a whole slew of extensions
> installed to be able to generate stubs for them. Currently, I only do
> this for PHP 8.2, but I will probably end up having to do this for more
> versions of PHP. That is a whole lot of packages that would have to be
> present in ELPA's build environment. And the list of extensions is not
> guaranteed to be static either.

In that case I had underestimated the effort.  While not perfect, it
seems the best approach for now might well be to track the generated
file in the Git repository :/

> My current idea is to create container images for each version of PHP
> and generate stub files by executing the generation scripts in these
> containers. To give you an idea of the list of required packages, this
> is an example of the installation step in a Dockerfile I'm using:
>
> RUN apt-get update && apt-get -y install \
>     php8.2-memcached \
>     php-redis \
>     php8.2-bcmath \
>     php8.2-bz2 \
>     php8.2-cli \
>     php8.2-common \
>     php8.2-curl \
>     php8.2-gmp \
>     php8.2-intl \
>     php-json \
>     php8.2-mbstring \
>     php8.2-mysql \
>     php8.2-odbc \
>     php8.2-opcache \
>     php8.2-pgsql \
>     php8.2-readline \
>     php8.2-tidy \
>     php8.2-xml \
>     php8.2-xsl \
>     php8.2-zip \
>     php8.2-gd \
>     php-bcmath \
>     php-apcu \
>     php-cli \
>     php-imagick \
>     php-intl \
>     php-xdebug \
>     php-amqp
>
>>>>> Also: If the former is the case, is the reduction in load time that
>>>>> this brings even significant enough to be worth the bother or should I
>>>>> just hold off on this while I look for a more efficient solution?
>>>> I'd say it would be worth it, if the resulting package would be smaller
>>>> and would load quicker.  After all, the performance on your laptop might
>>>> not be that significant of a difference, while for someone else with an
>>>> older or slower device, a 30%-speedup is pretty significant.
>>>
>>> You're right, on less performant systems it could make a more
>>> significant difference. And good news, after a few improvements the
>>> load time is now down to ~150ms on my laptop. I also chose to load the
>>> data when the mode is first initialized, so simply loading my package
>>> won't cause the index to be loaded with it. The dumping of the index
>>> is done automatically when it is not present, so it would technically
>>> be fine to just distribute the PHP stubs with the package instead of
>>> the .eld index file. This would just make the user wait a little
>>> longer the first time they use the mode.
>>
>> Would it be possible to load the information when it is required
>> (e.g. necessary for completion)?
>
> This is a good idea. Classes within the project's scope are already
> parsed/indexed on demand like that, I should apply this to stub classes
> as well.
>
> Global/native functions are another story though. Since they're not
> namespaced, it's hard to break them into smaller loadable sets. Aside
> from that, function completion generally requires the list of functions
> to be loaded indiscriminately.  Luckily, the functions only make up a
> little over 1/4th of the stubs (~2000 out of ~7300 lines).