RFE: default hooks for git repositories

RFE: default hooks for git repositories

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

Hello:

Since V2 creates git shards automatically, I suggest there should be a 
way to tell it to install some set of default hooks in all new git repos 
it creates. The easiest would be by using a skel dir and specifying it 
in PI_CONFIG:

[publicinbox]
  githooksskel=/usr/share/public-inbox/hooks

We currently do it via a cronjob, but unless we run it very frequently, 
there's a delay between when a new shard is created and when we add the 
hooks to it.

Best,
-K

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: RFE: default hooks for git repositories

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> Since V2 creates git shards automatically, I suggest there should be a way
> to tell it to install some set of default hooks in all new git repos it
> creates. The easiest would be by using a skel dir and specifying it in
> PI_CONFIG:
> 
> [publicinbox]
>  githooksskel=/usr/share/public-inbox/hooks

Seems reasonable.  Though "gitInitTemplateDir" would probably be
easier-to-implement and cover hooks.  This should let us avoid
creating needless "branches" directories and "info/exclude"
files, as well.

Curious, what hooks do you use?  I've never used any.

> We currently do it via a cronjob, but unless we run it very frequently,
> there's a delay between when a new shard is created and when we add the
> hooks to it.

I've also been thinking of a way to get mirrors to automatically
know about new epochs, too.  Maybe submodules can be used...


Note: I prefer the "epoch" term for the $N.git dirs over "shard"
      to disambiguate it from the Xapian partitioning we do.

Re: RFE: default hooks for git repositories

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > Hello:
> > 
> > Since V2 creates git shards automatically, I suggest there should be a way
> > to tell it to install some set of default hooks in all new git repos it
> > creates. The easiest would be by using a skel dir and specifying it in
> > PI_CONFIG:
> > 
> > [publicinbox]
> >  githooksskel=/usr/share/public-inbox/hooks
> 
> Seems reasonable.  Though "gitInitTemplateDir" would probably be
> easier-to-implement and cover hooks.  This should let us avoid
> creating needless "branches" directories and "info/exclude"
> files, as well.
> 
> Curious, what hooks do you use?  I've never used any.

Actually, is there a reason the git-native "init.templateDir"
configuration variable isn't sufficient?

i.e. something about public-inbox git repos should be set/unset
which is fundamentally incompatible with code-oriented git repos?

I've been trying to use normal git defaults and directives as
possible to minimize the learning curve.


I've started working on a dirty patch to implement this feature;
but it's rather intrusive and I'm not sure I like it...

Re: RFE: default hooks for git repositories

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Actually, is there a reason the git-native "init.templateDir"
> configuration variable isn't sufficient?

Ping?

> i.e. something about public-inbox git repos should be set/unset
> which is fundamentally incompatible with code-oriented git repos?
> 
> I've been trying to use normal git defaults and directives as
> possible to minimize the learning curve.

Re: RFE: default hooks for git repositories

[Konstantin Ryabitsev]

On Mon, Mar 18, 2019 at 11:11:52PM +0000, Eric Wong wrote:
>Eric Wong <e@80x24.org> wrote:
>> Actually, is there a reason the git-native "init.templateDir"
>> configuration variable isn't sufficient?
>
>Ping?

Sorry, Eric, I'm close to declaring email bankruptcy. :) I am okay with 
any solution that lets us add hooks. To explain why we need hooks in 
public-inbox repos, it's mostly for the ease of mirroring -- I want a 
post-update hook to update the grokmirror [1] manifest so that our 
mirroring script knows when there are updates and also when there are 
new archives or repository epochs added.

So, I defer to you entirely when it comes to how to do it. :)


[1] https://github.com/mricon/grokmirror/

-K

Re: RFE: default hooks for git repositories

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Mar 18, 2019 at 11:11:52PM +0000, Eric Wong wrote:
> > Eric Wong <e@80x24.org> wrote:
> > > Actually, is there a reason the git-native "init.templateDir"
> > > configuration variable isn't sufficient?
> > 
> > Ping?
> 
> Sorry, Eric, I'm close to declaring email bankruptcy. :) I am okay with any
> solution that lets us add hooks. To explain why we need hooks in
> public-inbox repos, it's mostly for the ease of mirroring -- I want a
> post-update hook to update the grokmirror [1] manifest so that our mirroring
> script knows when there are updates and also when there are new archives or
> repository epochs added.
> 
> So, I defer to you entirely when it comes to how to do it. :)

Yes, putting the hook in the init.templateDir should get you
what you want.  It would work the same for newly-created code
repositories, too.

> [1] https://github.com/mricon/grokmirror/

I should look into that for my own mirrors, even :)

Re: RFE: default hooks for git repositories

[Konstantin Ryabitsev]

On Mon, Apr 01, 2019 at 06:04:03PM +0000, Eric Wong wrote:
>> Sorry, Eric, I'm close to declaring email bankruptcy. :) I am okay 
>> with any
>> solution that lets us add hooks. To explain why we need hooks in
>> public-inbox repos, it's mostly for the ease of mirroring -- I want a
>> post-update hook to update the grokmirror [1] manifest so that our mirroring
>> script knows when there are updates and also when there are new archives or
>> repository epochs added.
>>
>> So, I defer to you entirely when it comes to how to do it. :)
>
>Yes, putting the hook in the init.templateDir should get you
>what you want.  It would work the same for newly-created code
>repositories, too.

So, I just realized that, at least when mail is delivered via 
public-inbox-mda, none of the hooks are ever triggered. I've tried with 
post-update, post-receive, post-commit, pre-commit, etc. Am I doing 
something wrong, or is this because the operation is being done on a low 
enough level that the hooks aren't triggered as they would with a 
regular git operation?

-K

Re: RFE: default hooks for git repositories

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> So, I just realized that, at least when mail is delivered via
> public-inbox-mda, none of the hooks are ever triggered. I've tried with
> post-update, post-receive, post-commit, pre-commit, etc. Am I doing
> something wrong, or is this because the operation is being done on a low
> enough level that the hooks aren't triggered as they would with a regular
> git operation?

Hmm... Perhaps git-fast-import does not call hooks?
That might need to be brought up with the git list.

(only online for a few minutes, gotta run out again)

Re: RFE: default hooks for git repositories

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Hmm... Perhaps git-fast-import does not call hooks?

Yeah, I remember; low-level git stuff doesn't do hooks.

So we might want to implement our own...
(won't be online much the next few days)

[PATCH] TODO: add an item for hooks support

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Yeah, I remember; low-level git stuff doesn't do hooks.
> 
> So we might want to implement our own...

-------8<------
Subject: [PATCH] TODO: add an item for hooks support

and add a note for grokmirror support/integration, too
---
 TODO | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/TODO b/TODO
index ef77d1c..7a3bb6b 100644
--- a/TODO
+++ b/TODO
@@ -98,3 +98,8 @@ all need to be considered for everything we introduce)
 
 * imperfect scraper importers for obfuscated list archives
   (e.g. obfuscated Mailman stuff, Google Groups, etc...)
+
+* support hooks, since low-level git-fast-import does not run them
+  https://public-inbox.org/meta/20190405174329.GA21472@chatter.qube.local/
+
+* investigate native grokmirror support/integration
-- 
EW

update on grokmirror integration [was: [PATCH] TODO: add an item for hooks support]

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> +* investigate native grokmirror support/integration

GitPython in Debian 9.x is busted with sorted packed-refs;
but the GitPython in Debian testing is fine and the next stable
release is just around the corner.

Hoping the Debian packager can put in an updated grokmirror, soon:

	https://bugs.debian.org/927271

But I think existing PSGI/Plack with Plack::Middleware::ConditionalGET
static file support is sufficient on the HTTP-side, and
our Import.pm should probably call $GIT_DIR/hooks/post-update...