Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Stefan Monnier]

Sean Whitton [2022-12-18 16:36:56] wrote:
>     * elpa-packages (notmuch): New package.

FWIW, I think this is a mistake  because:

> + (notmuch               :url "https://git.notmuchmail.org/git/notmuch"

This is not a repository containing the Emacs Notmuch package but
a repository containing the Notmuch command line MUA tools, and within
which they decided to also keep their Emacs front end.
[ The ELisp code is about 7% of the total in there :-(  ]

The (Non)GNU ELPA infrastructure is ill-equipped to handle such setups
(for that reason we don't have an `erlang-mode` package).

> +  :ignored-files
> +  ("emacs/make-deps.el" "emacs/Makefile*" "emacs/*.tmpl" "emacs/rstdoc.*"
> +   "emacs/notmuch-emacs-mua*" "bindings" "compat" "completion" "contrib"
> +   "debian" "devel" "doc" "lib" "packaging" "parse-time-string" "COPYING"
> +   "performance-test" "test" "util" "vim" "*.c" "*.h" "configure"
> +   "INSTALL" "Makefile*" "README.rst" "notmuch-git.py" "version.txt"
> +   ".travis.yml" "sh.config" "sphinx.config" ".mailmap" "COPYING-GPL-3")

Reads hellish to me.  This is crap *we* have to maintain in response to
whatever happens upstream.

> +  :make ("emacs/notmuch-pkg.el" "emacs/notmuch-version.el")

(Non)GNU ELPA really wants to create the <foo>-pkg.el from the headers
in the <foo>.el main file.  The above may sometimes work but will bite
us sooner or later.

> +  ;; FIXME: notmuch-pkg.el is not committed to Git.

Please don't fix this.  If it's in the Git it gets worse (tends to
create spurious conflicts while building the package.  Our scripts
*should* be resilient to such problems, but based on past experience,
I doubt they are yet in all cases).

> +  ;; Upstream are amenable to making changes in this area, but they have a
> +  ;; number of build targets that involve generated snapshot version numbers,
> +  ;; and we would need to confirm that committing notmuch-pkg.el doesn't
> +  ;; interfere too much with any of those.

That's in large part due to the repository not being for Emacs's package
for something else (related, admittedly, yet different and hence faced
with lots of other constraints).

The fix should start by adding a `Version:` to the `notmuch.el` file.
There's no deep reason why the Emacs UI's version should evolve in
lockstep with the Notmuch MUA itself.

But in any case, a `grep 0.37` in the repository shows that they already
have two copies of the version number (one in `version.txt` and one in
`bindings/python/notmuch/version.py`), so adding a third shouldn't
be a big deal.

> +  ;; Alternatively, could we have elpa-admin run 'make emacs/notmuch-pkg.el'
> +  ;; before looking for a version?
> +  :version-map ((nil "0.37" "0.37")))
                               ^^^^^^
This 3rd element should be a Git revision (IOW a commit id).

BTW, the build of the tarball currently fails in the `:make` command
because it looks for Git info that's not available in the sandbox
(probably to build a "short Git revision name" but since we don't have
the tags of notmuch.git, that won't do what's expected even if we fix
make the sandbox more lax) and also because it tries to compile a C file
(probably to extract some info from something like `notmuch.h`) but
`elpa.gnu.org` doesn't have a C compiler.  There might be further
problems along the way.


        Stefan

git: vs https: (was: [nongnu] main d1d82e3c06 1/2: * Makefile (admin): Use https:// instead of git:// for the URI)

[Stefan Monnier]

>     * Makefile (admin): Use https:// instead of git:// for the URI.

I used `git:` because fetching a single branch over `https:` was *much*
slower.  AFAIK, this was because `git.sv.gnu.org` didn't run Git's
"smart HTTP" on its https server.

I just tried it again, it seems that `https:` is now as fast as `git:`, tho.
Was I just lucky, or has the problem really been fixed?


        Stefan

Re: git: vs https:

[Sean Whitton]

Hello,

On Tue 20 Dec 2022 at 02:56PM -05, Stefan Monnier wrote:

>>     * Makefile (admin): Use https:// instead of git:// for the URI.
>
> I used `git:` because fetching a single branch over `https:` was *much*
> slower.  AFAIK, this was because `git.sv.gnu.org` didn't run Git's
> "smart HTTP" on its https server.

There was no way I was going to run any code fetched over git:// :)

> I just tried it again, it seems that `https:` is now as fast as `git:`, tho.
> Was I just lucky, or has the problem really been fixed?

I just tested, and it does indeed seem to be fixed.

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello,

Sorry I didn't catch the build failure before pushing.  I'll write a
separate message about that.

On Tue 20 Dec 2022 at 02:44PM -05, Stefan Monnier wrote:

> Sean Whitton [2022-12-18 16:36:56] wrote:
>>     * elpa-packages (notmuch): New package.
>
> FWIW, I think this is a mistake  because:
>
>> + (notmuch               :url "https://git.notmuchmail.org/git/notmuch"
>
> This is not a repository containing the Emacs Notmuch package but
> a repository containing the Notmuch command line MUA tools, and within
> which they decided to also keep their Emacs front end.
> [ The ELisp code is about 7% of the total in there :-(  ]
>
> The (Non)GNU ELPA infrastructure is ill-equipped to handle such setups
> (for that reason we don't have an `erlang-mode` package).
>
>> +  :ignored-files
>> +  ("emacs/make-deps.el" "emacs/Makefile*" "emacs/*.tmpl" "emacs/rstdoc.*"
>> +   "emacs/notmuch-emacs-mua*" "bindings" "compat" "completion" "contrib"
>> +   "debian" "devel" "doc" "lib" "packaging" "parse-time-string" "COPYING"
>> +   "performance-test" "test" "util" "vim" "*.c" "*.h" "configure"
>> +   "INSTALL" "Makefile*" "README.rst" "notmuch-git.py" "version.txt"
>> +   ".travis.yml" "sh.config" "sphinx.config" ".mailmap" "COPYING-GPL-3")
>
> Reads hellish to me.  This is crap *we* have to maintain in response to
> whatever happens upstream.

I think that maintaining notmuch.el inside the rest of notmuch.git is a
reasonable choice by notmuch upstream; I do the same in two of my
projects where the non-Elisp heavily outweighs the Elisp.

Perhaps we could overcome this particular limitation in the scripts.
How about adding :files or :include, or similar, where if that's
present, then only files matching the globs listed are taken up, and
everything else is ignored?

For background: currently notmuch.el is published to MELPA and
apparently a lot of users like to install it that way.  But one
consequence is that users often end up with a newer version of
notmuch.el than the version of the command line tools, and this causes
spurious bug reports, etc..  So it would be good for upstream, and
users, to have a version on NonGNU ELPA which matched upstream's stable
releases.

(Additionally and less importantly, I would like to add mailscripts.el
to NonGNU ELPA, as you know, and mailscripts.el depends on notmuch.el.)

>> +  ;; Alternatively, could we have elpa-admin run 'make emacs/notmuch-pkg.el'
>> +  ;; before looking for a version?
>> +  :version-map ((nil "0.37" "0.37")))
>                                ^^^^^^
> This 3rd element should be a Git revision (IOW a commit id).

0.37 is a committish,[1] if not a commit; isn't that okay?

[1]  See gitglossary(7).

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello notmuch maintainers,

As discussed on IRC, I'm seeing if I can add notmuch.el to NonGNU ELPA.

Currently the build is failing and Stefan Monnier, who designed the
packaging scripts, suggests adding a Version: header to notmuch.el:

On Tue 20 Dec 2022 at 02:44PM -05, Stefan Monnier wrote:

>> +  :make ("emacs/notmuch-pkg.el" "emacs/notmuch-version.el")
>
> (Non)GNU ELPA really wants to create the <foo>-pkg.el from the headers
> in the <foo>.el main file.  The above may sometimes work but will bite
> us sooner or later.
>
>> +  ;; FIXME: notmuch-pkg.el is not committed to Git.
>
> Please don't fix this.  If it's in the Git it gets worse (tends to
> create spurious conflicts while building the package.  Our scripts
> *should* be resilient to such problems, but based on past experience,
> I doubt they are yet in all cases).
> [...]
> The fix should start by adding a `Version:` to the `notmuch.el` file.
> [...]
> But in any case, a `grep 0.37` in the repository shows that they already
> have two copies of the version number (one in `version.txt` and one in
> `bindings/python/notmuch/version.py`), so adding a third shouldn't
> be a big deal.

I know that you don't want to commit notmuch-pkg.el to git, and that's
in fact undesirable from a NonGNU ELPA point of view, it turns out.  But
given how there is already version.txt and version.py, could we have a
header in notmuch.el too, do you think?

Thanks.

-- 
Sean Whitton

Re: git: vs https:

[Stefan Monnier]

>>>     * Makefile (admin): Use https:// instead of git:// for the URI.
>> I used `git:` because fetching a single branch over `https:` was *much*
>> slower.  AFAIK, this was because `git.sv.gnu.org` didn't run Git's
>> "smart HTTP" on its https server.
> There was no way I was going to run any code fetched over git:// :)

The difference was large enough back when I wrote this code that it was
tempting to think the "git fetch" was just frozen (IIRC it took several
minutes to fetch this small `elpa-admin` branch, downloading a large
fraction of the whole elpa.git repository).

>> I just tried it again, it seems that `https:` is now as fast as `git:`, tho.
>> Was I just lucky, or has the problem really been fixed?
> I just tested, and it does indeed seem to be fixed.

OK, I just checked with

    GIT_CURL_VERBOSE=1 GIT_TRACE=1                    \
        git clone --single-branch --verbose --depth 1 \
            https://git.sv.gnu.org/git/emacs/elpa.git

and indeed, it seems to be using the "smart" protocol, yay!


        Stefan

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Stefan Monnier]

>> Reads hellish to me.  This is crap *we* have to maintain in response to
>> whatever happens upstream.
>
> I think that maintaining notmuch.el inside the rest of notmuch.git is a
> reasonable choice by notmuch upstream; I do the same in two of my
> projects where the non-Elisp heavily outweighs the Elisp.

I'm not blaming them.  I do think nowadays their users would be better
served by decoupling the two, tho.  And indeed, the Debian package
`elpa-notmuch` does not depend on a specific version of the `notmuch`
tools, so in practice the two seem to be sufficiently decoupled that
they can evolve separately.

> Perhaps we could overcome this particular limitation in the scripts.
> How about adding :files or :include, or similar, where if that's
> present, then only files matching the globs listed are taken up, and
> everything else is ignored?

The problem is not with the tarball, but with the Git side.  (Non)GNU
ELPA is not just an archive of tarballs, but it's built around Git
repositories hosting the relevant code and its full history.

While I hope Git grows support for "partial clones" (or more likely is
superseded by something else which does support such things), the
current situation is that you just can't do that.  Which means
`elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
part of Notmuch, for example, and `package-vc` will still have to
download the 10MB of Notmuch's repository just to get at the relevant
0.5MB.

The closest we can get, AFAIK, is to run an interposer repository which
fetches from https://git.notmuchmail.org/git/notmuch and filters the
`emacs` subtree into a new branch, and then add *that* branch to
`elpa-packages`.  Of course, that creates a whole parallel history, so
it may end up biting us down the line, but I think given the current
situation it'd the "least wrong" way to do it (The Right Way being if
they do that split upstream and then use the new branch as the official
upstream).

> (Additionally and less importantly, I would like to add mailscripts.el
> to NonGNU ELPA, as you know, and mailscripts.el depends on notmuch.el.)

I understand the desire to add Notmuch, yes.
I'd be happy to see it in (Non)GNU ELPA as well.

>>> +  ;; Alternatively, could we have elpa-admin run 'make emacs/notmuch-pkg.el'
>>> +  ;; before looking for a version?
>>> +  :version-map ((nil "0.37" "0.37")))
>> This 3rd element should be a Git revision (IOW a commit id).
> 0.37 is a committish,[1] if not a commit;

"0.37" of what?  Remember we're talking about `nongnu.git` which contains
*all* the NonGNU ELPA packages.  So, upstream tags don't make much sense
there (and for that reason also they're not pulled from upstream).

> isn't that okay?

[ And hence: no, it's not okay, I'm afraid.  ]

It would be good to add optional support for using Git tags to label
versions (probably via renaming the tags from `<foo>` to
`upstream/<pkg>/<foo>` or something like that), but I'm still waiting for
Someone™ to write that code.


        Stefan

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello,

On Wed 21 Dec 2022 at 12:01AM -05, Stefan Monnier wrote:

>> Perhaps we could overcome this particular limitation in the scripts.
>> How about adding :files or :include, or similar, where if that's
>> present, then only files matching the globs listed are taken up, and
>> everything else is ignored?
>
> The problem is not with the tarball, but with the Git side.  (Non)GNU
> ELPA is not just an archive of tarballs, but it's built around Git
> repositories hosting the relevant code and its full history.
>
> While I hope Git grows support for "partial clones" (or more likely is
> superseded by something else which does support such things), the
> current situation is that you just can't do that.  Which means
> `elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
> part of Notmuch, for example, and `package-vc` will still have to
> download the 10MB of Notmuch's repository just to get at the relevant
> 0.5MB.

Couldn't we filter out mail to elpa-diffs where the diffs only touch
files that aren't included using :files/:include?

Your point about package-vc identifies a disadvantage of using
upstream's history, to be sure, but I think it's outweighed by the value
of distributing upstream's actual history (arguably it's part of the
preferred form for modification for the code).

> The closest we can get, AFAIK, is to run an interposer repository
> which fetches from https://git.notmuchmail.org/git/notmuch and filters
> the `emacs` subtree into a new branch, and then add *that* branch to
> `elpa-packages`.  Of course, that creates a whole parallel history, so
> it may end up biting us down the line, but I think given the current
> situation it'd the "least wrong" way to do it (The Right Way being if
> they do that split upstream and then use the new branch as the
> official upstream).

Is it a hard requirement that the updates be automatic?  Couldn't we
just manually import the emacs/ subdirectory to the branch whenever
upstream make a release?  It doesn't happen every week, and it could be
scripted.

> "0.37" of what?  Remember we're talking about `nongnu.git` which contains
> *all* the NonGNU ELPA packages.  So, upstream tags don't make much sense
> there (and for that reason also they're not pulled from upstream).
>
>> isn't that okay?
>
> [ And hence: no, it's not okay, I'm afraid.  ]

Ah right, thank you.

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Stefan Monnier]

>> While I hope Git grows support for "partial clones" (or more likely is
>> superseded by something else which does support such things), the
>> current situation is that you just can't do that.  Which means
>> `elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
>> part of Notmuch, for example, and `package-vc` will still have to
>> download the 10MB of Notmuch's repository just to get at the relevant
>> 0.5MB.
> Couldn't we filter out mail to elpa-diffs where the diffs only touch
> files that aren't included using :files/:include?

The elpa-diffs email messages are generated on Savannah's side for all
pushes, so we really don't have very much control over them :-(

> Your point about package-vc identifies a disadvantage of using
> upstream's history, to be sure, but I think it's outweighed by the value
> of distributing upstream's actual history (arguably it's part of the
> preferred form for modification for the code).

I fully agree that using upstream's history verbatim is
a significant advantage.  Which is why ideally, they'd split their ELisp
code into a separate branch/repository.  Short of that we have to choose
between various sets of ills :-(

>> The closest we can get, AFAIK, is to run an interposer repository
>> which fetches from https://git.notmuchmail.org/git/notmuch and filters
>> the `emacs` subtree into a new branch, and then add *that* branch to
>> `elpa-packages`.  Of course, that creates a whole parallel history, so
>> it may end up biting us down the line, but I think given the current
>> situation it'd the "least wrong" way to do it (The Right Way being if
>> they do that split upstream and then use the new branch as the
>> official upstream).
> Is it a hard requirement that the updates be automatic?

No it doesn't have to be automatic.  It should definitely be scripted
enough, otherwise I expect it'll get messy (with errors in the parallel
history, etc...).  But if it's not fully automatic, I expect it will
be forgotten.

> Couldn't we just manually import the emacs/ subdirectory to the branch
> whenever upstream make a release?

Not sure what you mean by that, but if it doesn't preserve the Git
history of the ELisp files, then it's definitely worse than `git
subtree` (or `git filter-branch/repo`).

> It doesn't happen every week, and it could be scripted.

This is not the first time the need arises, so such a script would be
a welcome addition (and we could run it on elpa.gnu.org to make the
workaround reasonably transparent).


        Stefan

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Sean Whitton <spwhitton@spwhitton.name> writes:

> For background: currently notmuch.el is published to MELPA and
> apparently a lot of users like to install it that way.  But one
> consequence is that users often end up with a newer version of
> notmuch.el than the version of the command line tools, and this causes
> spurious bug reports, etc..  So it would be good for upstream, and
> users, to have a version on NonGNU ELPA which matched upstream's stable
> releases.

Isn't notmuch one of those programs that usually distribute the Emacs
Lisp code along with the package (or as a dependency) that you would
install using your system file manager?  Or in other words, how
interoperable is notmuch.el with different versions of notmuch?  Does
the current build from NonGNU ELPA work even if I have a old release?

Re: git: vs https:

[Andreas Schwab]

On Dez 20 2022, Sean Whitton wrote:

> There was no way I was going to run any code fetched over git:// :)

It doesn't make any difference with https.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> While I hope Git grows support for "partial clones" (or more likely is
> superseded by something else which does support such things), the
> current situation is that you just can't do that.  Which means
> `elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
> part of Notmuch, for example, and `package-vc` will still have to
> download the 10MB of Notmuch's repository just to get at the relevant
> 0.5MB.

Do you know if this is being worked on?  Does it even fit into Git's
internal model?

(This might be too hacking, but I recently found out that GitHub
supports checking out repositories using Subversion.  I guess they use
some bridge to translate the requests.  I don't know if it is possible
to make use of this and a second SVN->Git bridge to check out partial
repositories as SVN does)

> The closest we can get, AFAIK, is to run an interposer repository which
> fetches from https://git.notmuchmail.org/git/notmuch and filters the
> `emacs` subtree into a new branch, and then add *that* branch to
> `elpa-packages`.  Of course, that creates a whole parallel history, so
> it may end up biting us down the line, but I think given the current
> situation it'd the "least wrong" way to do it (The Right Way being if
> they do that split upstream and then use the new branch as the official
> upstream).

This was discussed back when I wanted to add a few modes developed in
mono-repos (erlang was one example, cmake is another).  I had managed to
make this work using filter-branch IIRC, but we never set up a mirror.
Perhaps we should tackle that again, even if the solution is suboptimal
and host the filtered repositories on savannah.nongnu.org.  I could set
up a similar structure to nongnu.git and have each repository we are
interested in mirrored on their own branches that
nongnu.git:elpa-packages would target.

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[David Bremner]

Sean Whitton <spwhitton-PEZ64Ft4C9UnzZ6mRAm98g@public.gmane.org> writes:

>
> I know that you don't want to commit notmuch-pkg.el to git, and that's
> in fact undesirable from a NonGNU ELPA point of view, it turns out.  But
> given how there is already version.txt and version.py, could we have a
> header in notmuch.el too, do you think?

The existence of version.py is a bug that we hope to fix soon. It is
used only by the obsolete and deprecated python bindings. Current
bindings (under bindings/python-cffi) use version.txt.

Adding a version header to notmuch.el and maintaining it
semi-automatically in the way version.py is seems more fragile (sed'ing
a file also edited by humans) and more work (the code to generate
notmuch-pkg.el already exists).

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Stefan Monnier]

>> While I hope Git grows support for "partial clones" (or more likely is
>> superseded by something else which does support such things), the
>> current situation is that you just can't do that.  Which means
>> `elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
>> part of Notmuch, for example, and `package-vc` will still have to
>> download the 10MB of Notmuch's repository just to get at the relevant
>> 0.5MB.
>
> Do you know if this is being worked on?  Does it even fit into Git's
> internal model?

I don't think so, no.

> (This might be too hacking, but I recently found out that GitHub
> supports checking out repositories using Subversion.  I guess they use
> some bridge to translate the requests.  I don't know if it is possible
> to make use of this and a second SVN->Git bridge to check out partial
> repositories as SVN does)

It's no better than using `git subtree`.

Maybe something like Pijul (which supports partial trees really well)
will grow to be a worthy competitor, but we're pretty far from
that, AFAIK.

>> The closest we can get, AFAIK, is to run an interposer repository which
>> fetches from https://git.notmuchmail.org/git/notmuch and filters the
>> `emacs` subtree into a new branch, and then add *that* branch to
>> `elpa-packages`.  Of course, that creates a whole parallel history, so
>> it may end up biting us down the line, but I think given the current
>> situation it'd the "least wrong" way to do it (The Right Way being if
>> they do that split upstream and then use the new branch as the official
>> upstream).
>
> This was discussed back when I wanted to add a few modes developed in
> mono-repos (erlang was one example, cmake is another).  I had managed to
> make this work using filter-branch IIRC, but we never set up a mirror.
> Perhaps we should tackle that again, even if the solution is suboptimal

I think we should yes.

> and host the filtered repositories on savannah.nongnu.org.

Or the filtered result could just live in nongnu.git/elpa.git.

> I could set up a similar structure to nongnu.git and have each
> repository we are interested in mirrored on their own branches that
> nongnu.git:elpa-packages would target.

I was thinking of doing the work within `elpaa--fetch`, actually.


        Stefan

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello,

On Wed 21 Dec 2022 at 11:03PM -05, Stefan Monnier wrote:

>>
>> This was discussed back when I wanted to add a few modes developed in
>> mono-repos (erlang was one example, cmake is another).  I had managed to
>> make this work using filter-branch IIRC, but we never set up a mirror.
>> Perhaps we should tackle that again, even if the solution is suboptimal
>
> I think we should yes.
>
>> and host the filtered repositories on savannah.nongnu.org.
>
> Or the filtered result could just live in nongnu.git/elpa.git.

Right, the external/foo branch can just be the new history.

>> I could set up a similar structure to nongnu.git and have each
>> repository we are interested in mirrored on their own branches that
>> nongnu.git:elpa-packages would target.
>
> I was thinking of doing the work within `elpaa--fetch`, actually.

I'd like to work on this.  How far did you get, Philip?  How can I help?

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello,

On Wed 21 Dec 2022 at 08:45AM GMT, Philip Kaludercic wrote:

> Isn't notmuch one of those programs that usually distribute the Emacs
> Lisp code along with the package (or as a dependency) that you would
> install using your system file manager?  Or in other words, how
> interoperable is notmuch.el with different versions of notmuch?  Does
> the current build from NonGNU ELPA work even if I have a old release?

Somewhat interoperable.  It will often work fine, but not always.

I would never install notmuch.el using package.el, instead obtaining
matching versions of notmuch.el and notmuch(1) from the Debian archive,
or using upstream's Makefile targets which install notmuch.el.  However,
it seems that a lot of users like to install notmuch.el using package.el
no matter how they are installing notmuch(1).

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello,

On Wed 21 Dec 2022 at 07:53AM -04, David Bremner wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>>
>> I know that you don't want to commit notmuch-pkg.el to git, and that's
>> in fact undesirable from a NonGNU ELPA point of view, it turns out.  But
>> given how there is already version.txt and version.py, could we have a
>> header in notmuch.el too, do you think?
>
> The existence of version.py is a bug that we hope to fix soon. It is
> used only by the obsolete and deprecated python bindings. Current
> bindings (under bindings/python-cffi) use version.txt.
>
> Adding a version header to notmuch.el and maintaining it
> semi-automatically in the way version.py is seems more fragile (sed'ing
> a file also edited by humans) and more work (the code to generate
> notmuch-pkg.el already exists).

Okay, thanks.  We can investigate whether we can get the
branch-filtering script to substitute in the version.

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Dmitry Gutov]

On 22/12/2022 06:03, Stefan Monnier wrote:
>>> While I hope Git grows support for "partial clones" (or more likely is
>>> superseded by something else which does support such things), the
>>> current situation is that you just can't do that.  Which means
>>> `elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
>>> part of Notmuch, for example, and `package-vc` will still have to
>>> download the 10MB of Notmuch's repository just to get at the relevant
>>> 0.5MB.
>> Do you know if this is being worked on?  Does it even fit into Git's
>> internal model?
> I don't think so, no.
> 

Could this help?

https://www.git-scm.com/docs/git-sparse-checkout

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Stefan Monnier]

>>>> While I hope Git grows support for "partial clones" (or more likely is
>>>> superseded by something else which does support such things), the
>>>> current situation is that you just can't do that.  Which means
>>>> `elpa-diffs` will be spammed with irrelevant commit to the non-Emacs
>>>> part of Notmuch, for example, and `package-vc` will still have to
>>>> download the 10MB of Notmuch's repository just to get at the relevant
>>>> 0.5MB.
>>> Do you know if this is being worked on?  Does it even fit into Git's
>>> internal model?
>> I don't think so, no.
>> 
>
> Could this help?
>
> https://www.git-scm.com/docs/git-sparse-checkout

I don't think so: we need a "sparse clone" rather than a "sparse
checkout".


        Stefan

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Andreas Schwab]

On Dez 22 2022, Stefan Monnier wrote:

> I don't think so: we need a "sparse clone" rather than a "sparse
> checkout".

You can get a shallow clone.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Sean Whitton <spwhitton@spwhitton.name> writes:

> Hello,
>
> On Wed 21 Dec 2022 at 11:03PM -05, Stefan Monnier wrote:
>
>>>
>>> This was discussed back when I wanted to add a few modes developed in
>>> mono-repos (erlang was one example, cmake is another).  I had managed to
>>> make this work using filter-branch IIRC, but we never set up a mirror.
>>> Perhaps we should tackle that again, even if the solution is suboptimal
>>
>> I think we should yes.
>>
>>> and host the filtered repositories on savannah.nongnu.org.
>>
>> Or the filtered result could just live in nongnu.git/elpa.git.

Oh, this is even better.  I recall there being some issue with this
approach, but if you think it is ok, we can add support right into
elpa-admin.el.

> Right, the external/foo branch can just be the new history.
>
>>> I could set up a similar structure to nongnu.git and have each
>>> repository we are interested in mirrored on their own branches that
>>> nongnu.git:elpa-packages would target.
>>
>> I was thinking of doing the work within `elpaa--fetch`, actually.
>
> I'd like to work on this.  How far did you get, Philip?  How can I help?

Not particularly far, I had a little script I don't think I'd be able to
find again.  I believe this SO answer was my main tip:

     https://stackoverflow.com/a/52269934

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Dez 22 2022, Stefan Monnier wrote:
>
>> I don't think so: we need a "sparse clone" rather than a "sparse
>> checkout".
>
> You can get a shallow clone.

A shallow clone would loose all the history?  We would loose all the
history, which would be against the point.

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Andreas Schwab]

On Dez 24 2022, Philip Kaludercic wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> On Dez 22 2022, Stefan Monnier wrote:
>>
>>> I don't think so: we need a "sparse clone" rather than a "sparse
>>> checkout".
>>
>> You can get a shallow clone.
>
> A shallow clone would loose all the history?

No.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Dez 24 2022, Philip Kaludercic wrote:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> On Dez 22 2022, Stefan Monnier wrote:
>>>
>>>> I don't think so: we need a "sparse clone" rather than a "sparse
>>>> checkout".
>>>
>>> You can get a shallow clone.
>>
>> A shallow clone would loose all the history?
>
> No.

In that case I must be misunderstanding something.  How would you
proceed to create a shallow clone of erlang-mode from the otp
repository:

  https://github.com/erlang/otp/tree/master/lib/tools/emacs

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Andreas Schwab]

Just use git clone --depth, as explained in the manual page.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Just use git clone --depth, as explained in the manual page.

From git-clone(1):

  --depth <depth>
      Create a shallow clone with a history truncated to the specified
      number of commits. Implies --single-branch unless
      --no-single-branch is given to fetch the histories near the tips of
      all branches. If you want to clone submodules shallowly, also pass
      --shallow-submodules.

Yes, but that looses the Git history, beyond the depth you indicate
using the argument.  Furthermore, you don't get a subtree but still
clone all the unrelated files.  Naturally the revision hash also
changes.  This is a perpendicular solution to what we want.

We want to strip away all files and commits in a monorepo that have no
relation to an emacs package, located in some sub-directory.

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Andreas Schwab]

On Dez 24 2022, Philip Kaludercic wrote:

> We want to strip away all files and commits in a monorepo that have no
> relation to an emacs package, located in some sub-directory.

If you want to rebuild history, use git filter-repo.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Dez 24 2022, Philip Kaludercic wrote:
>
>> We want to strip away all files and commits in a monorepo that have no
>> relation to an emacs package, located in some sub-directory.
>
> If you want to rebuild history, use git filter-repo.

"filter-branch"[0] or "filter-repo".  The former is exactly what I had
been using.  The latter, I have never heard of before but it seems to be
based on "filter-branch".

[0] https://github.com/newren/git-filter-repo

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Sean Whitton]

Hello,

On Sat 24 Dec 2022 at 11:09AM GMT, Philip Kaludercic wrote:

> Not particularly far, I had a little script I don't think I'd be able to
> find again.  I believe this SO answer was my main tip:
>
>      https://stackoverflow.com/a/52269934

Cool, would you like to work on this imminently or shall I?  Let's avoid
duplicating effort.

-- 
Sean Whitton

Re: [nongnu] main 385623dca6 2/2: * elpa-packages (notmuch): New package.

[Philip Kaludercic]

I do not think that I'll be having the time to read into the details again, at least for the next few days, but I'd certainly be interested in folowing the development.