Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Stefan Beller]

> BTW in light of the discussion we are having elsewhere I just need to
> point out that it was *dramatically* faster for me to edit run-command.c,
> find "hooks/" and adjust the code manually than it would have been to save
> the diff and apply it.
>
> That's because I do not have advanced tooling on top of email (and I also
> could not handle mutt, so I am stuck with a not-really-scriptable email
> client).
>
> Just sayin'.

I ran into the same problem, just for a larger patch, so I figured I can
download that from the public inbox and git-am it locally.
So I maneuvered to the cover letter of the patch series I am interested in[1]
and downloaded the series as a mbox.gz[2].

However git-am choked on the cover-letter with:
> Patch is empty. Was it split wrong?
> When you have resolved this problem, run "git am --continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".

The way forward from here was to `git am --skip` the first mail (the
cover letter)
and the rest got applied cleanly.

So as a discussion starter:
* Should git am skip a patch 00/XX automatically ?
  It is obviously a cover letter, which may be text only or
  has an intra diff to a prior version. Neither is what we want for now.
  Although there is this other discussion of storing the cover letter,
  so maybe an empty patch that is numbered 0 is fine to skip for now?
  Once the discussion settles whether we want to store it in
  branch.<name>.description or as an empty commit at the end or at the beginning

* Should the public-inbox offer another link to patches 1-n, without
  the cover letter? Or should it add instructions:

        If this is a patch series you can apply it locally as:
        curl <link> >tmpXXX
        git am tmpXXX && git am --skip && git am --continue

I tend to favor the first option of Git learning how to process the
cover letter more easily.

Thanks,
Stefan


[1] https://public-inbox.org/git/20160815230702.30817-1-jacob.e.keller@intel.com/
[2] https://public-inbox.org/git/20160815230702.30817-1-jacob.e.keller@intel.com/t.mbox.gz
as found in
> Thread overview: 4+ messages in thread (expand / mbox.gz / Atom feed / [top])
> 2016-08-15 23:06 Jacob Keller [this message]
> 2016-08-15 23:07 ` [PATCH v6 1/3] diff.c: remove output_prefix_length field Jacob Keller
> 2016-08-15 23:07 ` [PATCH v6 2/3] graph: add support for --line-prefix on all graph-aware output Jacob Keller
> 2016-08-15 23:07 ` [PATCH v6 3/3] diff: add SUBMODULE_DIFF format to display submodule diff Jacob Keller

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> So as a discussion starter:
> * Should git am skip a patch 00/XX automatically ?

No.  My preference is to add "--initial-skip=<N>", though.

When I receive a patch series to reroll another series, I somehow
know and verify that earlier N patches have not changed, I detach
the HEAD at the last unchanged commit from the previous round and
apply the remainder of the new series, so that I can preserve the
author timestamps of earlier steps from the previous series.  By
the time I "know and verify" where the first step that was updated,
I have a full series in a single mbox; having "--initial-skip=<N>"
would help with that use case, too, and "skipping the first" is a
narrow special case of giving N=1.

> * Should the public-inbox offer another link to patches 1-n, without
>   the cover letter? Or should it add instructions:
>
>         If this is a patch series you can apply it locally as:
>         curl <link> >tmpXXX
>         git am tmpXXX && git am --skip && git am --continue

I do not think it is sensible for "cover-letter" specific
instructions.  However, I do not think it is unreasonable to either
add another mbox.gz link or replace the behaviour of mbox.gz link so
that you can grab a mbox that contains "this message and everything
after it in the thread".  That way, I could open the first message,
see something like this I found in your message:

>> Thread overview: 4+ messages in thread (expand / mbox.gz / Atom feed / [top])
>> 2016-08-15 23:06 Jacob Keller [this message]
>> 2016-08-15 23:07 ` [PATCH v6 1/3] diff.c: remove output_prefix_length field Jacob Keller
>> 2016-08-15 23:07 ` [PATCH v6 2/3] graph: add support for --line-prefix on all graph-aware output Jacob Keller
>> 2016-08-15 23:07 ` [PATCH v6 3/3] diff: add SUBMODULE_DIFF format to display submodule diff Jacob Keller

and then go to 1/3 and click that "this and everything that
follows".

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jeff King]

On Tue, Aug 16, 2016 at 10:10:42AM -0700, Junio C Hamano wrote:

> Stefan Beller <sbeller@google.com> writes:
> 
> > So as a discussion starter:
> > * Should git am skip a patch 00/XX automatically ?
> 
> No.  My preference is to add "--initial-skip=<N>", though.
> 
> When I receive a patch series to reroll another series, I somehow
> know and verify that earlier N patches have not changed, I detach
> the HEAD at the last unchanged commit from the previous round and
> apply the remainder of the new series, so that I can preserve the
> author timestamps of earlier steps from the previous series.  By
> the time I "know and verify" where the first step that was updated,
> I have a full series in a single mbox; having "--initial-skip=<N>"
> would help with that use case, too, and "skipping the first" is a
> narrow special case of giving N=1.

For my workflow, it is not about "initial skip", but rather just "skip
emails that don't have patches in them at all". My MUA makes it easy to
tag a whole thread (or subthread), cover letter and discussion included,
and then dump it all to git-am.

And I think that would be the same for a public-inbox workflow (if it
learns to grab sub-threads; otherwise you end up with earlier iterations
of the series attached to the same thread).

That is solving a different problem than you, though, where you want to
skip actual patches because you know they are unchanged.

-Peff

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> For my workflow, it is not about "initial skip", but rather just "skip
> emails that don't have patches in them at all".

OK.  That is different from "the subject line says 0/N so let's
skip".

If we can safely determine that there is no patch in a message,
skipping it may feel sensible.  Historically, we try to err on the
safe side by stopping when we do not understand an input and that is
where "Patch is empty" message came from, because skipping and
continuing, even with a warning, while applying tons of patches is
not good enough to get user's attention.

An "ignore empty input" option (and eventually with a configuration)
may not be a bad idea.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Stefan Beller]

On Tue, Aug 16, 2016 at 10:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> So as a discussion starter:
>> * Should git am skip a patch 00/XX automatically ?
>
> No.  My preference is to add "--initial-skip=<N>", though.
>
> When I receive a patch series to reroll another series, I somehow
> know and verify that earlier N patches have not changed, I detach
> the HEAD at the last unchanged commit from the previous round and
> apply the remainder of the new series, so that I can preserve the
> author timestamps of earlier steps from the previous series.  By
> the time I "know and verify" where the first step that was updated,
> I have a full series in a single mbox; having "--initial-skip=<N>"
> would help with that use case, too, and "skipping the first" is a
> narrow special case of giving N=1.

In your work flow, how do you respect the cover letter?
e.g. in 3787e3c16ced:

    Merge branch 'ew/http-backend-batch-headers'

    The http-backend (the server-side component of smart-http
    transport) used to trickle the HTTP header one at a time.  Now
    these write(2)s are batched.

    * ew/http-backend-batch-headers:
      http-backend: buffer headers before sending

Is the text from the original author (and if so from which version
of the cover letter) or is it your work?

>
>> * Should the public-inbox offer another link to patches 1-n, without
>>   the cover letter? Or should it add instructions:
>>
>>         If this is a patch series you can apply it locally as:
>>         curl <link> >tmpXXX
>>         git am tmpXXX && git am --skip && git am --continue
>
> I do not think it is sensible for "cover-letter" specific
> instructions.  However, I do not think it is unreasonable to either
> add another mbox.gz link or replace the behaviour of mbox.gz link so
> that you can grab a mbox that contains "this message and everything
> after it in the thread".  That way, I could open the first message,
> see something like this I found in your message:
>
>>> Thread overview: 4+ messages in thread (expand / mbox.gz / Atom feed / [top])
>>> 2016-08-15 23:06 Jacob Keller [this message]
>>> 2016-08-15 23:07 ` [PATCH v6 1/3] diff.c: remove output_prefix_length field Jacob Keller
>>> 2016-08-15 23:07 ` [PATCH v6 2/3] graph: add support for --line-prefix on all graph-aware output Jacob Keller
>>> 2016-08-15 23:07 ` [PATCH v6 3/3] diff: add SUBMODULE_DIFF format to display submodule diff Jacob Keller
>
> and then go to 1/3 and click that "this and everything that
> follows".

Both thoughts are sensible; However the --initial-skip=<n>
doesn't address the special case of storing the cover letter
(which we eventually want to do?)

I thought of it as the following with room for improvement:

diff --git a/builtin/am.c b/builtin/am.c (shite space broken):
index 739b34d..5f08b61 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1246,6 +1246,7 @@ static int parse_mail(struct am_state *state,
const char *mail)
        FILE *fp;
        struct strbuf sb = STRBUF_INIT;
        struct strbuf msg = STRBUF_INIT;
+       struct strbuf subject = STRBUF_INIT;
        struct strbuf author_name = STRBUF_INIT;
        struct strbuf author_date = STRBUF_INIT;
        struct strbuf author_email = STRBUF_INIT;
@@ -1309,6 +1310,7 @@ static int parse_mail(struct am_state *state,
const char *mail)
                        if (msg.len)
                                strbuf_addch(&msg, '\n');
                        strbuf_addstr(&msg, x);
+                       strbuf_addstr(&subject, x);
                } else if (skip_prefix(sb.buf, "Author: ", &x))
                        strbuf_addstr(&author_name, x);
                else if (skip_prefix(sb.buf, "Email: ", &x))
@@ -1325,8 +1327,17 @@ static int parse_mail(struct am_state *state,
const char *mail)
        }

        if (is_empty_file(am_path(state, "patch"))) {
-               printf_ln(_("Patch is empty. Was it split wrong?"));
-               die_user_resolve(state);
+               if (indicates_coverletter(&subject))
+                       /*
+                        * TODO: store the cover letter as the first or last
+                        * commit or as branch.<name>.description
+                        */
+                       ret = 1;
+                       goto finish;
+               else {
+                       printf_ln(_("Patch is empty. Was it split wrong?"));
+                       die_user_resolve(state);
+               }
        }

        strbuf_addstr(&msg, "\n\n");

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> In your work flow, how do you respect the cover letter?
> e.g. in 3787e3c16ced:
>
>     Merge branch 'ew/http-backend-batch-headers'
>
>     The http-backend (the server-side component of smart-http
>     transport) used to trickle the HTTP header one at a time.  Now
>     these write(2)s are batched.
>
>     * ew/http-backend-batch-headers:
>       http-backend: buffer headers before sending
>
> Is the text from the original author (and if so from which version
> of the cover letter) or is it your work?

The source of truth in the merge log message is the "What's cooking"
report.  I really prefer to write these in my own words, as that is
a good yardstick to measure how much/little I understand the topic.
If I cannot describe it concisely, in a way suitable as an entry in
the release notes, that means I am merging a topic I do not have a
good idea about, which is quite irresponsive.  Forcing me to write
these myself keeps me honest.

Of course, if a cover letter describes the topic well, it would help
me write the entry in the "What's cooking" report.

It is a bit tricky to aim for the automation, though.  The cover is
an overview of the proposed log messages and typically tells a story
"I do this, and then this, and finally that", plus a reroll-specific
commentary like "what changed since the last round".  On the other
hand, the entries in the release notes gives a description of what
happened from a third-party's point of view.  They are told in
different voice for different target audience.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
> > * Should the public-inbox offer another link to patches 1-n, without
> >   the cover letter? Or should it add instructions:
> >
> >         If this is a patch series you can apply it locally as:
> >         curl <link> >tmpXXX
> >         git am tmpXXX && git am --skip && git am --continue

Currently for web users, I suggest:

	curl $URL >tmpXXX

	# open tmpXXXX and tag+copy to patchesXXX using MUA of choice:
	# (also seems to be what Jeff describes):
	mutt -f tmpXXX

	git am patchesXXXX

> I do not think it is sensible for "cover-letter" specific
> instructions.  However, I do not think it is unreasonable to either
> add another mbox.gz link or replace the behaviour of mbox.gz link so
> that you can grab a mbox that contains "this message and everything
> after it in the thread".  That way, I could open the first message,
> see something like this I found in your message:
> 
> >> Thread overview: 4+ messages in thread (expand / mbox.gz / Atom feed / [top])
> >> 2016-08-15 23:06 Jacob Keller [this message]
> >> 2016-08-15 23:07 ` [PATCH v6 1/3] diff.c: remove output_prefix_length field Jacob Keller
> >> 2016-08-15 23:07 ` [PATCH v6 2/3] graph: add support for --line-prefix on all graph-aware output Jacob Keller
> >> 2016-08-15 23:07 ` [PATCH v6 3/3] diff: add SUBMODULE_DIFF format to display submodule diff Jacob Keller
> 
> and then go to 1/3 and click that "this and everything that
> follows".

Adding more links might still fall down in cases where
fixup/squash patches are sent for specific patches in a series;
or when a v{N+1} series is posted in-reply-to an existing
series.

Perhaps adding checkbox next to each item might work as a
select-to-include-in-mbox download form.  However, I'm already
finding the lack of horizontal space disconcerting.

Maybe the YYYY-MM-DD could be shortened to YYYYMMDD.  It would
be closer to the date searching syntax used by mairix, as well
as the search enhancement I started working on earlier today:

  https://public-inbox.org/meta/20160816084926.29394-1-e@80x24.org/T/
  (still will deploy soonish)

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Currently for web users, I suggest:
> 
> 	curl $URL >tmpXXX
> 
> 	# open tmpXXXX and tag+copy to patchesXXX using MUA of choice:
> 	# (also seems to be what Jeff describes):
> 	mutt -f tmpXXX
> 
> 	git am patchesXXXX

I should add this is also a better match to an "offline first"
workflow for disconnected use.  My Internet connections drop
all the time :<

I don't mind people over-downloading at all, and stuffing an
extra cover + comments is likely more efficient with gzip than
going back online to fetch separately.

> Perhaps adding checkbox next to each item might work as a
> select-to-include-in-mbox download form.

...So I'm not sure if I want to invest time into this idea
since either the server or user could be offline by the
time messages are selected.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Stefan,

On Tue, 16 Aug 2016, Stefan Beller wrote:

> > BTW in light of the discussion we are having elsewhere I just need to
> > point out that it was *dramatically* faster for me to edit run-command.c,
> > find "hooks/" and adjust the code manually than it would have been to save
> > the diff and apply it.
> >
> > That's because I do not have advanced tooling on top of email (and I also
> > could not handle mutt, so I am stuck with a not-really-scriptable email
> > client).
> >
> > Just sayin'.
> 
> I ran into the same problem, just for a larger patch, so I figured I can
> download that from the public inbox and git-am it locally.
> So I maneuvered to the cover letter of the patch series I am interested in[1]
> and downloaded the series as a mbox.gz[2].

Maybe you can adapt the script I had written to perform that magic for
gmane?

https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh

The relevant part is the one between the lines 48--72, where it detects
0/N mails and then looks for the first children containing k/N for k=1..N.

BTW I take this thread as yet another proof that people are unhappy with
mail list-based review: if you have to build *that much* tooling around it
(and Peff & Junio certainly have a megaton of advanced and sophisticated
tooling around it, holy cow!) it is really incorrect to state that the
mail list-driven approach works for you. It is much closer to the truth to
say that the mail-list-plus-loads-of-custom-tools-driven approach works
for you.

I am really not a fan of this.

The theory "it's inclusive because everyone has access to mail" falls on
its face, badly, when even old timers have to build entire infrastructures
around it just to begin to be able to use it efficiently.

It reminds me of an old software developer I met long ago, who claimed CVS
works for him. He had written tens of thousands of lines of shell scripts,
is what allowed "CVS" to work for him.

Same here. Old dogs claim the mail list-approach works for them. Nope.
Doesn't. Else you would not have written all those custom scripts.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> BTW I take this thread as yet another proof that people are unhappy with
> mail list-based review: if you have to build *that much* tooling around it
> (and Peff & Junio certainly have a megaton of advanced and sophisticated
> tooling around it, holy cow!) it is really incorrect to state that the
> mail list-driven approach works for you. It is much closer to the truth to
> say that the mail-list-plus-loads-of-custom-tools-driven approach works
> for you.
> 
> I am really not a fan of this.
> 
> The theory "it's inclusive because everyone has access to mail" falls on
> its face, badly, when even old timers have to build entire infrastructures
> around it just to begin to be able to use it efficiently.
> 
> It reminds me of an old software developer I met long ago, who claimed CVS
> works for him. He had written tens of thousands of lines of shell scripts,
> is what allowed "CVS" to work for him.
> 
> Same here. Old dogs claim the mail list-approach works for them. Nope.
> Doesn't. Else you would not have written all those custom scripts.

git and cogito started as a bunch of custom scripts, too.
IMHO, it's what makes Free Software (and *nix) great:
users have control to customize and improve things.
With scripts, they don't even need to learn a build process
to do so.

I see a choice of mail client as no different than a choice of
text editor.  Neither my mail client or text editor is heavily
customized.  The key feature I rely on from both tools is piping
data to external commands.


OTOH, today, I see people using git aliases all the time which
look more like ASM instructions than user commands.

Is the widespread use of these aliases a deficiency to git?
Maybe, I don't know.

Normally, I do not care about aliases: it's a private thing;
but it also makes it incredibly difficult for me to help
users when they're exposed in public.


Users ought to be able to pick, choose, and replace tools as
they wish as long as an interchange format remains stable
and widely-supported.

Fwiw, I still use patch(1) pretty often, even on patches
generated with git.  I see nothing wrong with that; patch is
lenient in ways git-apply was explicitly designed not to be.
And I don't always use git send-email or my normal MUA for
sending; or use git for generating diffs.  I do:

	diff -u a b | mail -s WIP-blah-blah $SOMEONE


While you and I are long-time git hackers, I don't think it's
reasonable for everyone to use git or git-specific tools;
even to git.git for one-offs like portability or doc fixes.

Even today, at least one Linux kernel hacker still uses quilt to
generate patches: http://ozlabs.org/~akpm/mmotm/

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Junio C Hamano]

Eric Wong <e@80x24.org> writes:

> I see a choice of mail client as no different than a choice of
> text editor.  Neither my mail client or text editor is heavily
> customized.  The key feature I rely on from both tools is piping
> data to external commands.

FWIW, that applies to me exactly, too.

> unsubscribe: meta+unsubscribe@public-inbox.org

Did you mean this, really?

> archive: https://public-inbox.org/meta/

This I understand, though.

Ahh, that's coming from meta@public-inbox.org?

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi,

On Thu, 18 Aug 2016, Junio C Hamano wrote:

> Eric Wong <e@80x24.org> writes:
> 
> > unsubscribe: meta+unsubscribe@public-inbox.org
> 
> Did you mean this, really?

FWIW I do not see this line in my original mail from Eric.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Eric,

On Thu, 18 Aug 2016, Eric Wong wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>
> > Old dogs claim the mail list-approach works for them. Nope. Doesn't.
> > Else you would not have written all those custom scripts.
> 
> git and cogito started as a bunch of custom scripts, too.

The difference is that neither git nor cogito were opinionated. Those
custom scripts are. They are for one particular workflow, with one
particular mail client, with a strong bias to a Unix-y environment.

I work really hard to make Git for Windows as easy and fun to use as
possible. I just wish that we were working together to make it as easy and
fun to contribute to Git, too.

> I see a choice of mail client as no different than a choice of
> text editor.  Neither my mail client or text editor is heavily
> customized.  The key feature I rely on from both tools is piping
> data to external commands.

There you go. That key feature seems to be unavailable in the most
wide-spread email client: Outlook. So by not restricting the choice you
should make it possible to use that mail client, too, right?

We do not even have a section on Outlook in our SubmittingPatches.

Okay, if not the most popular mail client, then web mail? Nope, nope,
nope. No piping *at all* to external commands from there.

So you basically slam the door shut on the vast majority of email users.

That is not leaving much choice to the users in my book.

> OTOH, today, I see people using git aliases all the time which
> look more like ASM instructions than user commands.

I see this as a completely different beast. Aliases help users accelerate
their personal workflow. Whereas anybody who is already willing to
contribute to Git *must* go through that non-personal workflow we impose:
paste the diff in a very specific format into the mail, and don't you dare
use a mail client that mangles whitespace (which is, like, pretty much
every single popular mail client out there).

So *allowing* users to configure their own aliases, and *forcing* them to
figure out how to transport patches through a medium hostile to patches,
is pretty much two diametrically opposed things.

> Users ought to be able to pick, choose, and replace tools as
> they wish as long as an interchange format remains stable
> and widely-supported.

Right. Let's talk about the interchange format called mails, for the data
called patches. Is it stable and widely-supported?

Can users really pick and choose the tools they like most to send patches
to the Git project? Like, the Outlook client? Or the GMail client?

> Even today, at least one Linux kernel hacker still uses quilt to
> generate patches: http://ozlabs.org/~akpm/mmotm/

Andrew does not count, he lives in his own universe.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Stefan Beller]

It was not my intend to start this discussion again with my initial email.
I rather wanted to point out how I make progress in doing my own
tooling.

I mean if email works well for Junio (both as a maintainer as
well as a contributor) and Jeff as a contributor, then I can adapt
my workflow to that, because these two have brought in
8300 of 33000 non merge patches. (i.e. they had 25% of the
patches over the life time of the project and are with the project
longer than others IIUC). So why would I demand they change
their style just to accommodate one newcomer like me?

>
>> I see a choice of mail client as no different than a choice of
>> text editor.  Neither my mail client or text editor is heavily
>> customized.  The key feature I rely on from both tools is piping
>> data to external commands.
>
> There you go. That key feature seems to be unavailable in the most
> wide-spread email client: Outlook. So by not restricting the choice you
> should make it possible to use that mail client, too, right?

Well I think this data piping is essential to any workflow. Even if were to
abandon email completely and roll our own communications protocol,
one of the first things added would be an API to "use your own text editor".

In my case git-send-email works well for the sending direction without a lot
of custom tooling (read: none apart from the initial configuration).

>
> We do not even have a section on Outlook in our SubmittingPatches.

"You can write one? Pretty please?"
would be the canonical answer. ;)

>
> Okay, if not the most popular mail client, then web mail? Nope, nope,
> nope. No piping *at all* to external commands from there.
>
> So you basically slam the door shut on the vast majority of email users.
>
> That is not leaving much choice to the users in my book.
>
>> OTOH, today, I see people using git aliases all the time which
>> look more like ASM instructions than user commands.
>
> I see this as a completely different beast. Aliases help users accelerate
> their personal workflow. Whereas anybody who is already willing to
> contribute to Git *must* go through that non-personal workflow we impose:
> paste the diff in a very specific format into the mail, and don't you dare
> use a mail client that mangles whitespace (which is, like, pretty much
> every single popular mail client out there).

Maybe we should invent a patch format that copes with broken whitespace?
(git-format-patch --allow-broken-whitespace), e.g. replace a tab by
"-_______" (not exactly 8 chars, but stopping at the columns 8, 16, 24 etc)
git am/apply would need to know about the unbreaking the white space, too.

>
> So *allowing* users to configure their own aliases, and *forcing* them to
> figure out how to transport patches through a medium hostile to patches,
> is pretty much two diametrically opposed things.

I think the point was that people carelessly expose their aliases in e.g. their
blogs. (e.g. "to do <foo> you just need to git a <file> &&git ci &&
git rr", which
leaves others just guessing? It certainly feels similar to not knowing how to
configure your email client?)

>
>> Users ought to be able to pick, choose, and replace tools as
>> they wish as long as an interchange format remains stable
>> and widely-supported.
>
> Right. Let's talk about the interchange format called mails, for the data
> called patches. Is it stable and widely-supported?

It is stable as it has been around for years and you can choose whether you
use git apply or the patch utility. It is widely supported as it is raw text so
it can be used across different platforms. However it doesn't cope well with
email, as email modifies text sometimes such as mangling white spaces.

>
> Can users really pick and choose the tools they like most to send patches
> to the Git project? Like, the Outlook client? Or the GMail client?

Of course, see[1] ;)
[1] https://public-inbox.org/git/CA+55aFy2AEe7ew5Px=2Uit6hraGV9zFr=JZ57rSYXWMQ4nMjeg@mail.gmail.com/

Thanks,
Stefan

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Stefan Beller <sbeller@google.com> wrote:
> Maybe we should invent a patch format that copes with broken whitespace?

No redundant new formats, please.  MIME attachments are already
widely-supported and fine by me.  But it's not my call for git.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Stefan,

On Fri, 19 Aug 2016, Stefan Beller wrote:

> >> I see a choice of mail client as no different than a choice of text
> >> editor.  Neither my mail client or text editor is heavily customized.
> >> The key feature I rely on from both tools is piping data to external
> >> commands.
> >
> > There you go. That key feature seems to be unavailable in the most
> > wide-spread email client: Outlook. So by not restricting the choice
> > you should make it possible to use that mail client, too, right?
> 
> Well I think this data piping is essential to any workflow.

Data piping can go through Git, with convenient commands.

> Even if were to abandon email completely and roll our own communications
> protocol, one of the first things added would be an API to "use your own
> text editor".
> 
> In my case git-send-email works well for the sending direction without a
> lot of custom tooling (read: none apart from the initial configuration).

Sending those mails is but the tiny, first step of contributing patches.
You know as well as I do that many a times contributors have to work
through many iterations to get their work accepted.

So while send-email helps with one direction, everything after that is
hard, manual work.

> > We do not even have a section on Outlook in our SubmittingPatches.
> 
> "You can write one? Pretty please?" would be the canonical answer. ;)

Sure. And my answer to that is: I cannot write it. Why? Because I cannot
get it the heck to work. Because Outlook supports writing mails, i.e.
messages from humans for humans. You can change the font, insert a nice
photo from your vacations, left-justify, right-justify, center the text.
You can do all kinds of nice things that you need to do when talking to
humans.

You can even paste a diff, for a human to read. Humans are very good at
not even seeing that there is no space at the beginning of the line.
Humans are also very good at understanding that those 8 spaces are the
same as the tab in the source code.

Outlook can also keep excellent track of who was Cc:ed, of mail threads,
filtering mails based on a plethora of criteria, integrating with a
calendar, etc.

So Outlook does really an excellent job.

What it does *not* to well is something mail was not designed for.

> Maybe we should invent a patch format that copes with broken whitespace?

Or maybe we do not even have to go that far, but maybe we can teach `git
apply` a mode where it is much smarter about whitespace changes and
wrapped text in the patch it receives?

That would probably go a long way further to making the patch submission
process we use more friendly to human beings.

It still would not make it easy to go from replies containing suggestions
how to improve the code to the corresponding file/revision.

> >> Users ought to be able to pick, choose, and replace tools as they
> >> wish as long as an interchange format remains stable and
> >> widely-supported.
> >
> > Right. Let's talk about the interchange format called mails, for the
> > data called patches. Is it stable and widely-supported?
> 
> It is stable as it has been around for years and you can choose whether
> you use git apply or the patch utility.

You seem to assume that mail clients have an easy time with this supposed
"stable" format.

They don't.

> It is widely supported as it is raw text so it can be used across
> different platforms. However it doesn't cope well with email, as email
> modifies text sometimes such as mangling white spaces.

I "mangle" whitespace all the time when I respond to mails. You will note
that I re-wrap quoted text to 76 columns/row.

So I am as guilty as any mail client of your charge.

Sorry.

> > Can users really pick and choose the tools they like most to send patches
> > to the Git project? Like, the Outlook client? Or the GMail client?
> 
> Of course, see[1] ;)
> [1] https://public-inbox.org/git/CA+55aFy2AEe7ew5Px=2Uit6hraGV9zFr=JZ57rSYXWMQ4nMjeg@mail.gmail.com/

You speak in riddles. That link leads to Linus' mail talking about
committerdates and generation numbers.

It does not help me, not in the slightest, to send a patch via Outlook or
the web interface of GoogleMail without risking to get yelled at for
corrupting the patch.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jeff King]

On Fri, Aug 19, 2016 at 09:55:54AM -0700, Stefan Beller wrote:

> It was not my intend to start this discussion again with my initial email.
> I rather wanted to point out how I make progress in doing my own
> tooling.
> 
> I mean if email works well for Junio (both as a maintainer as
> well as a contributor) and Jeff as a contributor, then I can adapt
> my workflow to that, because these two have brought in
> 8300 of 33000 non merge patches. (i.e. they had 25% of the
> patches over the life time of the project and are with the project
> longer than others IIUC). So why would I demand they change
> their style just to accommodate one newcomer like me?

Even though I do really like the mail-based workflow, I think this is a
dangerous line of reasoning. If the project were just me and Junio,
working as efficiently as two people possibly can, then sure, asking us
to change from what works for us would be silly.

But it's not. We have to make sure that the project community thrives.
That includes catering to some degree to occasional contributors, and
doing things to attract new members to the community as old ones drift
away.

You'll notice that I hand-waved away "to some degree" there. There is
definitely a balance to be found in managing the time of the maintainer
and the reviewers, versus making things easier for new contributors. As
a reductio ad absurdum, the simplest thing for contributors would be to
make a vague bug report and have the maintainer produce a polished
patch. That obviously does not scale. :)

Likewise, it is not just a matter of time spent, but workflows impact
_who_ will join[1]. Contributing to git is very friendly to a certain
niche of Unix die-hards, and that impacts who bothers to do so, and
consequently, what contributions we see (both to code and to
discussion). There's value in diversity of opinions[2], and we should be
wary of becoming an obsolete and out-of-touch mono-culture.

So I say "dangerous" because that is one way that open source projects
can die: the number of contributors dwindles, development slows, there
are no new ideas in the community, etc.

I don't think git would ever die off completely; there are too many
users. But there have been projects that seem to ossify for many years,
and are rejuvenated only when they shake up some elements of the
community or workflow (e.g., mutt is recently seeing such a resurgence;
sometimes it even takes the form of a follow-on project, like CVS->SVN,
with new people).

I don't think we're at that point with git. But it is something to be
mindful of. It's not clear to me if mutt-loving luddites like myself are
the last of a dying breed, or if there will always be enough of us to
churn out contributions to projects like git.

-Peff

[1] I think Dscho feels this much more acutely on Git for Windows than
    we do on the regular git mailing list, because the "who" audience
    for GfW is much different than the Unix world.

[2] I also think there's such as a thing as "too many opinions" in a
    project. If we started rewriting bits of git in Haskell (just to
    pick on a random pretty-far-from-C language), things would get very
    complex very quickly. So again, it's about finding a balance.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Thu, 18 Aug 2016, Eric Wong wrote:
> > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> >
> > > Old dogs claim the mail list-approach works for them. Nope. Doesn't.
> > > Else you would not have written all those custom scripts.
> > 
> > git and cogito started as a bunch of custom scripts, too.
> 
> The difference is that neither git nor cogito were opinionated. Those
> custom scripts are. They are for one particular workflow, with one
> particular mail client, with a strong bias to a Unix-y environment.
> 
> I work really hard to make Git for Windows as easy and fun to use as
> possible. I just wish that we were working together to make it as easy and
> fun to contribute to Git, too.

I guess this is a fundamental difference between *nix and
Windows culture.

I enjoy using and contributing to git because it interacts well
with generic tools.  *nix kernels are optimized for this with
decent (not great)[*] process spawning and IPC performance.

I know Windows users have major performance problems with
shell scripts; but they are also largely helpless to improve
Windows kernel performance.

So, I guess monolithic tools became popular on Windows, instead.

> > I see a choice of mail client as no different than a choice of
> > text editor.  Neither my mail client or text editor is heavily
> > customized.  The key feature I rely on from both tools is piping
> > data to external commands.
> 
> There you go. That key feature seems to be unavailable in the most
> wide-spread email client: Outlook. So by not restricting the choice you
> should make it possible to use that mail client, too, right?
> 
> We do not even have a section on Outlook in our SubmittingPatches.
> 
> Okay, if not the most popular mail client, then web mail? Nope, nope,
> nope. No piping *at all* to external commands from there.
> 
> So you basically slam the door shut on the vast majority of email users.

Users have a choice to use a more scriptable mail client
(but I guess the OS nudges users towards monolithic tools)

It's unfortunate the world is so full of proprietary things;
but I think it's our responsibility as Free Software developers
to encourage the use of Free (or "Open Source") tools which
users can control.

> That is not leaving much choice to the users in my book.

Users of alpine, gnus, mutt, sylpheed, thunderbird, kmail,
roundcube, squirelmail, etc. can all download the source, hack,
fix and customize things.  It's easier with smaller software,
of course:  git-send-email does not even require learning
the build process or separate download.

<snip stuff Stefan already covered>

> > Users ought to be able to pick, choose, and replace tools as
> > they wish as long as an interchange format remains stable
> > and widely-supported.
> 
> Right. Let's talk about the interchange format called mails, for the data
> called patches. Is it stable and widely-supported?
> 
> Can users really pick and choose the tools they like most to send patches
> to the Git project? Like, the Outlook client? Or the GMail client?

Personally, I don't mind patches as MIME attachments if that
avoids corruption, MIME seems well-supported at this point.
It's not my call, though.  But as Stephan pointed, Linus
does it, too.



[*] Guess what: I have performance problems with fork/execve on
    Linux, too.  However, Linux developers already provide
    mechanisms to improve spawn performance (CLONE_VFORK and
    vfork(2)); so the next step is to get userspace like dash,
    make, perl, etc to support these.

    glibc 2.24 was just released with an improved posix_spawn
    for Linux (using CLONE_VFORK), so that's a step forward and
    might make sharing code with Windows easier, too.

    It's not a high priority for me at the moment, but I intend
    to get everything in my toolset which relies on fork+execve
    to use posix_spawn or vfork+execve instead.  I have the
    source to all of these, so at least I can do something
    about it.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Eric,

On Fri, 19 Aug 2016, Eric Wong wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > On Thu, 18 Aug 2016, Eric Wong wrote:
> > > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > >
> > > > Old dogs claim the mail list-approach works for them. Nope.
> > > > Doesn't.  Else you would not have written all those custom
> > > > scripts.
> > > 
> > > git and cogito started as a bunch of custom scripts, too.
> > 
> > The difference is that neither git nor cogito were opinionated. Those
> > custom scripts are. They are for one particular workflow, with one
> > particular mail client, with a strong bias to a Unix-y environment.
> > 
> > I work really hard to make Git for Windows as easy and fun to use as
> > possible. I just wish that we were working together to make it as easy
> > and fun to contribute to Git, too.
> 
> I guess this is a fundamental difference between *nix and Windows
> culture.

I do not understand how you get from "I wish to make it fun to contribute
to Git" to "there is a fundamental difference between *nix and Windows
culture".

> I know Windows users have major performance problems with
> shell scripts;

That's because shell scripting is not native to Windows. I wish Linux had
a Powershell, allowing for decent scripting that does not try to smoosh
everything into a line-based text format. (Of course, since last week,
Linux does have a Powershell.)

Powershell is blazing fast, by the way, and not as ridiculously limited in
its expressibility as shell scripting.

But all of this is digressing from the original topic. I do not think this
is a productive.

> > We do not even have a section on Outlook in our SubmittingPatches.
> > 
> > Okay, if not the most popular mail client, then web mail? Nope, nope,
> > nope. No piping *at all* to external commands from there.
> > 
> > So you basically slam the door shut on the vast majority of email users.
> 
> Users have a choice to use a more scriptable mail client
> (but I guess the OS nudges users towards monolithic tools)

You call that choice. Are you serious?

> > That is not leaving much choice to the users in my book.
> 
> Users of alpine, gnus, mutt, sylpheed, thunderbird, kmail,
> roundcube, squirelmail, etc. can all download the source, hack,
> fix and customize things.  It's easier with smaller software,
> of course:  git-send-email does not even require learning
> the build process or separate download.

Now I am getting upset. This is a BS argument. Sure, I can hack the source
of these tools.

But why on earth do I *have* to? Why can't we use or create an open
contribution process *that works without having to work so hard to be able
to contribute*?

So unfortunately this thread has devolved. Which is sad. Because all I
wanted is to have a change in Git's submission process that would not
exclude *so many* developers. That is really all I care about. Not about
tools. Not about open vs proprietary, or standards.

I just want developers who are already familiar with Git, and come up with
an improvement to Git itself, to be able to contribute it without having
to pull out their hair in despair.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jakub Narębski]

W dniu 22.08.2016 o 15:18, Johannes Schindelin pisze:

> So unfortunately this thread has devolved. Which is sad. Because all I
> wanted is to have a change in Git's submission process that would not
> exclude *so many* developers. That is really all I care about. Not about
> tools. Not about open vs proprietary, or standards.
> 
> I just want developers who are already familiar with Git, and come up with
> an improvement to Git itself, to be able to contribute it without having
> to pull out their hair in despair.

What is lacking in using submitGit tool for those that have problems
with sending patches via email?

Submitting changes in Git comes in three phases:
 - submit email with patches
 - review and discuss patch
 - apply patches from email

Pull request via GitHub / Bitbucket / GitLab is easier than sending
patches via email (pity that GitHub et al. do not have such submitGit-like
automation built-in).  But I think email, with threaded view, careful
trimming of quoted contents, multi-level quotes is superior to exiting
web-based solutions.

Regards,
-- 
Jakub Narębski

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 3351 bytes --]

Hi Kuba,

On Mon, 22 Aug 2016, Jakub Narębski wrote:

> W dniu 22.08.2016 o 15:18, Johannes Schindelin pisze:
> 
> > So unfortunately this thread has devolved. Which is sad. Because all I
> > wanted is to have a change in Git's submission process that would not
> > exclude *so many* developers. That is really all I care about. Not about
> > tools. Not about open vs proprietary, or standards.
> > 
> > I just want developers who are already familiar with Git, and come up with
> > an improvement to Git itself, to be able to contribute it without having
> > to pull out their hair in despair.
> 
> What is lacking in using submitGit tool for those that have problems
> with sending patches via email?

Where do I start? And where do I stop? Here is a *very* brief list of
issues from the top of my head (and the history of my web browser):

- You cannot open a PR on GitHub and include the PR's cover letter as
  cover letter: https://github.com/rtyley/submitgit/issues/9

- You cannot Cc: people explicitly:
  https://github.com/rtyley/submitgit/issues/31

- submitGit does not include any interdiff

- it is really hard to get back from mails to the corresponding commits

- you have to register with yet another service to send mails on your
  behalf. Would be nicer if the mails could be sent from a submitGit
  address (moderated, of course) and did not need a separate registration
  step with some scary permission granting.

- submitGit requires you to go to a separate website to interact with the
  submitGit web app. Would be so much nicer if it were a bot operating on
  PRs.

- comments sent as replies have no connection to the PR *nor* the commits
  they refer to (making submitGit basically a pimped up git-send-email,
  nothing more).

- submitGit would require a substantial effort from me to learn how to
  extend/fix it, to run the web app locally and run its tests. That is a
  rather steep hurdle.

This is an incomplete list, of course.

> Submitting changes in Git comes in three phases:
>  - submit email with patches
>  - review and discuss patch
>  - apply patches from email

You forgot a really crucial step. Maybe you did not go through dozens of
iterations in your patch series as I regularly do, or something, so it is
probably easy for you to forget:

  - find the commit in question, run rebase -i and patch it as suggested

This is something that costs me quite some time to do. It is easily the
most annoying aspect of the mail list-based approach for me.

> Pull request via GitHub / Bitbucket / GitLab is easier than sending
> patches via email (pity that GitHub et al. do not have such submitGit-like
> automation built-in).  But I think email, with threaded view, careful
> trimming of quoted contents, multi-level quotes is superior to exiting
> web-based solutions.

They are not exiting, but I know what you meant.

The thing is: GitHub does not need such an automation. Because most
projects are pretty happy with the process centered around the web app.

It is only projects such as Linux, Cygwin and Git itself who refuse to
allow for tools that would let the majority of potential contributors
stick with their favorite way to read and write mails (I am talking about
users of GMail and Outlook, of course).

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jakub Narębski]

Hello Johannes,

W dniu 25.08.2016 o 15:21, Johannes Schindelin pisze:
> On Mon, 22 Aug 2016, Jakub Narębski wrote:
>> W dniu 22.08.2016 o 15:18, Johannes Schindelin pisze:
>>
>>> So unfortunately this thread has devolved. Which is sad. Because all I
>>> wanted is to have a change in Git's submission process that would not
>>> exclude *so many* developers. That is really all I care about. Not about
>>> tools. Not about open vs proprietary, or standards.
>>>
>>> I just want developers who are already familiar with Git, and come up with
>>> an improvement to Git itself, to be able to contribute it without having
>>> to pull out their hair in despair.
>>
>> What is lacking in using submitGit tool for those that have problems
>> with sending patches via email?
> 
> Where do I start? And where do I stop? Here is a *very* brief list of
> issues from the top of my head (and the history of my web browser):

Let me reorder those issues and group them into specific categories.
 
> - You cannot open a PR on GitHub and include the PR's cover letter as
>   cover letter: https://github.com/rtyley/submitgit/issues/9
> 
> - you have to register with yet another service to send mails on your
>   behalf. Would be nicer if the mails could be sent from a submitGit
>   address (moderated, of course) and did not need a separate registration
>   step with some scary permission granting.
> 
> - submitGit requires you to go to a separate website to interact with the
>   submitGit web app. Would be so much nicer if it were a bot operating on
>   PRs.

Those are issues about lack of integration (or need for better integration)
with GitHub pull requests.  I don't know if tighter integration would be
possible (without going the route of web browser plugin, like ZenHub) with
current API.  Note that many integrations require you to go and use their
separate site, like e.g. HuBoard.

Also, for some people registering on GitHub is "yet another service"... ;-)

> - You cannot Cc: people explicitly:
>   https://github.com/rtyley/submitgit/issues/31
> 
> - submitGit does not include any interdiff

These are, I think, mainly related to lack of support for series *iteration*,
for sending new version of patch series / of pull request.

I don't know how well GitHub pull requests deal with iteration and
refinement, and what is available as API to apps such as submitGit.

> 
> - it is really hard to get back from mails to the corresponding commits
> 
> - comments sent as replies have no connection to the PR *nor* the commits
>   they refer to (making submitGit basically a pimped up git-send-email,
>   nothing more).

I guess that turning submitGit into two-way bridge between email and PR's
would be quite difficult, if at all possible.  For one, it would have
to parse emails to turn response as email into comment on diff in PR.
For second, I'm not sure if all features of email workflow are possible
to represent in PR's: cover letter, comments for individual commits,
commits about commit messages, commits about changes, free-form comments.
But maybe they are possible; the trouble would be in translating back
and forth.

> - submitGit would require a substantial effort from me to learn how to
>   extend/fix it, to run the web app locally and run its tests. That is a
>   rather steep hurdle.

Well, you cannot extend/fix GitHub at all yourself, isn't it? ;-P

> 
> This is an incomplete list, of course.

Right.

>> Submitting changes in Git comes in three phases:
>>  - submit email with patches
>>  - review and discuss patch
>>  - apply patches from email
> 
> You forgot a really crucial step. Maybe you did not go through dozens of
> iterations in your patch series as I regularly do, or something, so it is
> probably easy for you to forget:
> 
>   - find the commit in question, run rebase -i and patch it as suggested
> 
> This is something that costs me quite some time to do. It is easily the
> most annoying aspect of the mail list-based approach for me.

I probably don't have as many topics in flight, and maybe the number of
iterations is smaller, but I don't remember having troubles with that.

Well, when I was most active doung Git development I were using StGit
patch management interface instead of rebase -i; I think it makes things
easier.  Perhaps git-series could replace it, or augment it.

>> Pull request via GitHub / Bitbucket / GitLab is easier than sending
>> patches via email (pity that GitHub et al. do not have such submitGit-like
>> automation built-in).  But I think email, with threaded view, careful
>> trimming of quoted contents, multi-level quotes is superior to exiting
>> web-based solutions.
> 
> They are not exiting, but I know what you meant.

That was a typo: s/exiting/existing/
 
> The thing is: GitHub does not need such an automation. Because most
> projects are pretty happy with the process centered around the web app.

It might be that they don't need features not available in a web app.
It might be that it is only way they know.  Not all projects are happy
to being on GitHub even if they are happy with using hosted git.

And they were probably created after GitHub / GitLab / Bitbucket were
here. Also - citation needed.

> It is only projects such as Linux, Cygwin and Git itself who refuse to
> allow for tools that would let the majority of potential contributors
> stick with their favorite way to read and write mails (I am talking about
> users of GMail and Outlook, of course).

Those are projects that started before GitHub (for obvious reasons), and
which created a good enough workflow based on email.  It might be that
they are ossified relics, or it might be that they don't want to trade
advantages of email based workflow for advantages of web app based workflow.


First, email clients and Usenet news readers support threading.  I haven't
found a good web app that supports threading well (though it might be
a matter of small set of such apps examined by me).  They allow marking
and labeling posts to return back later.

Second, email allows offline work, and handle well sporadic Internet
connection.  As far as I know web apps do not handle breaks in net
access well, but I might be mistaken.  Hopefully this problem would
vanish with improving broadband... though there always would be places
without constant always-on Internet connection.

Third, email (or rather conventions around email) allows to provide
 - description of the whole series (in cover letter)
 - comments for each commit individually (outside of commit message)
 - make commit or series be a reply to discussion (useful with WIPs)
For reviewer it allows to
 - comment on the whole series in response to cover letter
 - comment on individual commit
 - comment on the commit message
 - comment on the description of commit
 - comment on changes
 - start a discussion based on a commit
 - propose improvements as patches

Are all of them possible in PR's?


To reiterate what I have said elsewhere in this thread: while it is
a good idea to lower barriers to contribution, especially for new classes
of users (like MS Windows users, without a way to easily install and
configure MTA so that git-send-email and/or git-imap-send just works,
or with default email client without support for preserving whitespace
and monospace fonts), it is reviewers and maintainers that are sparse
resource.

Best,
-- 
Jakub Narębski

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 8273 bytes --]

Hi Kuba,

On Sun, 28 Aug 2016, Jakub Narębski wrote:

> W dniu 25.08.2016 o 15:21, Johannes Schindelin pisze:
> > On Mon, 22 Aug 2016, Jakub Narębski wrote:
> >> W dniu 22.08.2016 o 15:18, Johannes Schindelin pisze:
> >>
> >>> So unfortunately this thread has devolved. Which is sad. Because all
> >>> I wanted is to have a change in Git's submission process that would
> >>> not exclude *so many* developers. That is really all I care about.
> >>> Not about tools. Not about open vs proprietary, or standards.
> >>>
> >>> I just want developers who are already familiar with Git, and come
> >>> up with an improvement to Git itself, to be able to contribute it
> >>> without having to pull out their hair in despair.
> >>
> >> What is lacking in using submitGit tool for those that have problems
> >> with sending patches via email?
> > 
> > Where do I start? And where do I stop? Here is a *very* brief list of
> > issues from the top of my head (and the history of my web browser):
> 
> Let me reorder those issues and group them into specific categories.

I am afraid that this is not the direction I was interested in.

You asked how submitGit fell short of my goal to provide a more convenient
and more efficient submission process, and I listed a couple of reasons.

I am really more interested in a better process, than in a point-by-point
discussion of the reasons I listed.

And some of those discussions really only distract. This comment, for
example, seems to be designed to veer off in a direction that, quite
frankly, does not matter for what I *really* wanted to discuss:

> > - submitGit requires you to go to a separate website to interact with the
> >   submitGit web app. Would be so much nicer if it were a bot operating on
> >   PRs.
> 
> [...]
> 
> Also, for some people registering on GitHub is "yet another service"... ;-)

I am really curious how this is supposed to keep the discussion rational.

*Of course* I implied that it is bad enough to require contributors to
register with one web service, and that requiring to register with *yet
another service* is just excessive.

Sheesh.

> > - You cannot Cc: people explicitly:
> >   https://github.com/rtyley/submitgit/issues/31
> > 
> > - submitGit does not include any interdiff
> 
> These are, I think, mainly related to lack of support for series *iteration*,
> for sending new version of patch series / of pull request.

Yes, of course. Because that is what makes the process so particularly
tedious, thankyouverymuch.

> I don't know how well GitHub pull requests deal with iteration and
> refinement, and what is available as API to apps such as submitGit.

Hmm. Then it may be a good idea if I let you find out before we continue
to discuss these bullet points that I never wanted to discuss.

> > - submitGit would require a substantial effort from me to learn how to
> >   extend/fix it, to run the web app locally and run its tests. That is a
> >   rather steep hurdle.
> 
> Well, you cannot extend/fix GitHub at all yourself, isn't it? ;-P

Sure you can. There is an API and you can register hooks. You can do even
more advanced stuff [*1*].

> >> Submitting changes in Git comes in three phases:
> >>  - submit email with patches
> >>  - review and discuss patch
> >>  - apply patches from email
> > 
> > You forgot a really crucial step. Maybe you did not go through dozens of
> > iterations in your patch series as I regularly do, or something, so it is
> > probably easy for you to forget:
> > 
> >   - find the commit in question, run rebase -i and patch it as suggested
> > 
> > This is something that costs me quite some time to do. It is easily the
> > most annoying aspect of the mail list-based approach for me.
> 
> I probably don't have as many topics in flight, and maybe the number of
> iterations is smaller, but I don't remember having troubles with that.

Well, aren't you lucky.

I bet you did not intend to sound condescending there, even.

> > It is only projects such as Linux, Cygwin and Git itself who refuse to
> > allow for tools that would let the majority of potential contributors
> > stick with their favorite way to read and write mails (I am talking
> > about users of GMail and Outlook, of course).
> 
> Those are projects that started before GitHub (for obvious reasons), and
> which created a good enough workflow based on email.  It might be that
> they are ossified relics, or it might be that they don't want to trade
> advantages of email based workflow for advantages of web app based
> workflow.

Those are projects that started before Git was invented. Yet all three
projects traded the advantages of patches and tarballs for advantages of
using Git.

> First, email clients and Usenet news readers support threading.  I
> haven't found a good web app that supports threading well (though it
> might be a matter of small set of such apps examined by me).  They allow
> marking and labeling posts to return back later.
> 
> Second, email allows offline work, and handle well sporadic Internet
> connection.  As far as I know web apps do not handle breaks in net
> access well, but I might be mistaken.  Hopefully this problem would
> vanish with improving broadband... though there always would be places
> without constant always-on Internet connection.
> 
> Third, email (or rather conventions around email) allows to provide
>  - description of the whole series (in cover letter)
>  - comments for each commit individually (outside of commit message)
>  - make commit or series be a reply to discussion (useful with WIPs)
> For reviewer it allows to
>  - comment on the whole series in response to cover letter
>  - comment on individual commit
>  - comment on the commit message
>  - comment on the description of commit
>  - comment on changes
>  - start a discussion based on a commit
>  - propose improvements as patches

I find these reasons to be more a defense of "ossified practices" than
based in practical considerations.

For example, it is delusional to think that you can do any proper review
of any moderately complex patch without having access to a worktree
reflecting the revision after the patch.

So this entire idea that review is inherently something you would want to
be able to do without having access to a Git repository reflecting the
patches in question is an idea I reject flat out.

For the record, I have used GitHub Pull Requests *extensively*, as
contributor, as reviewer and as maintainer.

The Pull Request web interface has changed over time, but one thing
remained constant: it was always much more efficient to interact with than
going through the mail list.

The commit diff is naturally linked to *the commit*.

If I reply to notification emails when somebody left a comment, it is
automatically turned into a reply on the web.

If I do not want to reply via email, or if I need to see a bigger context,
the notification email has a link which opens the web interface, where I
can easily not only increase the diff context but also see other files in
the same revision, e.g. to check a signature in a .h file.

These are just a few things that are essential to efficient code review,
and that are simply impossible with a mailing list approach, unless it is
opinionated by requiring specific tooling to deal with the fact that code
is transported through a code-hostile medium.

So please understand that I am really interested in more efficient code
review instead of a lengthy philosophical discussions, and that I am
really saddened that I could not gain anything from your mail in that
respect.

I really hope that something constructive comes out of this whole
discussion, because if we simply stick with this unwieldy patch submission
process that costs so much of my time, then I really only lost time for
nothing in return.

Ciao,
Johannes

Footnote *1*: I wrote a little GreaseMonkey script to add a TOC button to
the GitHub wiki editor: https://tomancaklab.github.io/gfm-add-toc.user.js
Similar extensions can be implemented to augment the PR workflow to your
liking, with the advantage of being opt-in.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Fri, 19 Aug 2016, Eric Wong wrote:
> > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > On Thu, 18 Aug 2016, Eric Wong wrote:
> > > > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > >
> > > > > Old dogs claim the mail list-approach works for them. Nope.
> > > > > Doesn't.  Else you would not have written all those custom
> > > > > scripts.
> > > > 
> > > > git and cogito started as a bunch of custom scripts, too.
> > > 
> > > The difference is that neither git nor cogito were opinionated. Those
> > > custom scripts are. They are for one particular workflow, with one
> > > particular mail client, with a strong bias to a Unix-y environment.

<snip 3 lines I was not responding to>

> > I guess this is a fundamental difference between *nix and Windows
> > culture.
> 
> I do not understand how you get from "I wish to make it fun to contribute
> to Git" to "there is a fundamental difference between *nix and Windows
> culture".

Sorry, I over-quoted by 3 lines.

<snip more digression..>

> > > We do not even have a section on Outlook in our SubmittingPatches.
> > > 
> > > Okay, if not the most popular mail client, then web mail? Nope, nope,
> > > nope. No piping *at all* to external commands from there.
> > > 
> > > So you basically slam the door shut on the vast majority of email users.
> > 
> > Users have a choice to use a more scriptable mail client
> > (but I guess the OS nudges users towards monolithic tools)
> 
> You call that choice. Are you serious?
> 
> > > That is not leaving much choice to the users in my book.
> > 
> > Users of alpine, gnus, mutt, sylpheed, thunderbird, kmail,
> > roundcube, squirelmail, etc. can all download the source, hack,
> > fix and customize things.  It's easier with smaller software,
> > of course:  git-send-email does not even require learning
> > the build process or separate download.
> 
> Now I am getting upset. This is a BS argument. Sure, I can hack the source
> of these tools.
>
> But why on earth do I *have* to? Why can't we use or create an open
> contribution process *that works without having to work so hard to be able
> to contribute*?

The process we have is already open.  It may be *nix-centric,
and *nix may be picky about it's friends, but it is open:

	Anybody can still contribute today without any sort of
	registration, credentialism, or terms-of-service(*).

I am looking beyond git.

I hate signing up for websites.  For many years, I have used
Debian as a proxy for other projects with less open contribution
processes:

	apt-get source ...; <hack>; reportbug ...

Of course, going through Debian maintainers is not always
reliable or efficient.

I foolishly hoped git-svn would put an end to all the
registration-required bug tracker instances so I could just
send my changes directly to upstream maintainers without any
sort of registration.  Did not happen :<

> So unfortunately this thread has devolved. Which is sad. Because all I
> wanted is to have a change in Git's submission process that would not
> exclude *so many* developers. That is really all I care about. Not about
> tools. Not about open vs proprietary, or standards.
> 
> I just want developers who are already familiar with Git, and come up with
> an improvement to Git itself, to be able to contribute it without having
> to pull out their hair in despair.

We want the same thing.  I just want to go farther and get
people familiar with (federated|decentralized) tools instead of
proprietary and centralized ones.



(*) I wish git could get rid of the DCO, even.  But at least
    it's far better than the "papers, please" policy for some
    GNU projects.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Eric,

On Mon, 22 Aug 2016, Eric Wong wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>
> > I just want developers who are already familiar with Git, and come up with
> > an improvement to Git itself, to be able to contribute it without having
> > to pull out their hair in despair.
> 
> We want the same thing.  I just want to go farther and get
> people familiar with (federated|decentralized) tools instead of
> proprietary and centralized ones.

Why require users to get familiar with (federated|decentralized) tools
*unless* they make things provably more convenient? So far, I only see
that this would add to the hurdle, not improve things.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jakub Narębski]

W dniu 25.08.2016 o 14:58, Johannes Schindelin pisze:
> On Mon, 22 Aug 2016, Eric Wong wrote:
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>>
>>> I just want developers who are already familiar with Git, and come up with
>>> an improvement to Git itself, to be able to contribute it without having
>>> to pull out their hair in despair.
>>
>> We want the same thing.  I just want to go farther and get
>> people familiar with (federated|decentralized) tools instead of
>> proprietary and centralized ones.
> 
> Why require users to get familiar with (federated|decentralized) tools
> *unless* they make things provably more convenient? So far, I only see
> that this would add to the hurdle, not improve things.

Arguably for some federated/decentralized tools are preferred
(for philosophical reasons), even if they do not achieve even feature
parity with centralized tools (c.f. FSF).  Though Git is not there
to improve the world, just to be better...

On the other hand some may say that centralized tools (such as GitHub
and its pull requests) do not achieve feature parity with email based
workflow... though the basics are here.

Best regards,
-- 
Jakub Narębski

Re: Working with public-inbox.org

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

Hi Kuba,

On Sun, 28 Aug 2016, Jakub Narębski wrote:

> W dniu 25.08.2016 o 14:58, Johannes Schindelin pisze:
> > On Mon, 22 Aug 2016, Eric Wong wrote:
> >> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> >>
> >>> I just want developers who are already familiar with Git, and come up with
> >>> an improvement to Git itself, to be able to contribute it without having
> >>> to pull out their hair in despair.
> >>
> >> We want the same thing.  I just want to go farther and get
> >> people familiar with (federated|decentralized) tools instead of
> >> proprietary and centralized ones.
> > 
> > Why require users to get familiar with (federated|decentralized) tools
> > *unless* they make things provably more convenient? So far, I only see
> > that this would add to the hurdle, not improve things.
> 
> Arguably for some federated/decentralized tools are preferred
> (for philosophical reasons), even if they do not achieve even feature
> parity with centralized tools (c.f. FSF).

This statement is false. If you had talked about *your* preference, it
would be true. But to state that federated/decentralized tools are
universally preferred is nonsense.

You know as well as I do that most users/contributors go for convenience.

And if you require an inconvenient step (e.g. subscribing to a
federated/decentralized philosophy), most potential contributors simply
stop being potential contributors. End of story.

Ciao,
Johannes

Re: Working with public-inbox.org

[Jakub Narębski]

Hello Johannes,

W dniu 28.08.2016 o 10:36, Johannes Schindelin pisze:
> On Sun, 28 Aug 2016, Jakub Narębski wrote:
>> W dniu 25.08.2016 o 14:58, Johannes Schindelin pisze:
>>> On Mon, 22 Aug 2016, Eric Wong wrote:
>>>> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>>>>
>>>>> I just want developers who are already familiar with Git, and come up with
>>>>> an improvement to Git itself, to be able to contribute it without having
>>>>> to pull out their hair in despair.
>>>>
>>>> We want the same thing.  I just want to go farther and get
>>>> people familiar with (federated|decentralized) tools instead of
>>>> proprietary and centralized ones.
>>>
>>> Why require users to get familiar with (federated|decentralized) tools
>>> *unless* they make things provably more convenient? So far, I only see
>>> that this would add to the hurdle, not improve things.
>>
>> Arguably for some federated/decentralized tools are preferred
>> (for philosophical reasons), even if they do not achieve even feature
>> parity with centralized tools (c.f. FSF).

Philosophical and ideological reasons.

> This statement is false. If you had talked about *your* preference, it
> would be true. But to state that federated/decentralized tools are
> universally preferred is nonsense.

The key word is "some" (or, if you prefer, "a few").

> You know as well as I do that most users/contributors go for convenience.
> 
> And if you require an inconvenient step (e.g. subscribing to a
> federated/decentralized philosophy), most potential contributors simply
> stop being potential contributors. End of story.

For some people registering on GitHub and using web interface (though
I think there is also command line interface, don't know if it covers
PRs) would be the inconvenient step.  Just saying.


What I would like to see is bidirectional bridge between email and
GitHub (and possibly other hosting sites), so that everybody could use
their favorite interface, be it Git + email client, or Git + web browser
(or desktop client for GitHub).  Just like thanks to Gmane and Gmane-like
public-inbox I can use either email client or news reader, and it is
[nearly] transparent regarding to which one I use.

Even better if hosting site implementation (of pull requests, etc.)
were based on federated systems (like email, or Usenet, or IRC, or XMPP),
and not closed in walled gardens; with hosting site just being one
particular implementation (like e.g. Ghost blog engine, and Ghost Pro
hosting site).

I agree that lowering barriers to contribution for new class of users
(like MS Windows users; setting up MTA on Linux etc. for git-send-email
or git-imap-send is not very complicated), if it would be possible
without inconveniencing existing reviewers... which are even more
important, judging by the amount of "needs review" messages in
"What's cooking...".

Best,
-- 
Jakub Narębski

Re: Working with public-inbox.org

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

Hi Kuba,

On Sun, 28 Aug 2016, Jakub Narębski wrote:

> W dniu 28.08.2016 o 10:36, Johannes Schindelin pisze:
> > On Sun, 28 Aug 2016, Jakub Narębski wrote:
> >
> >> Arguably for some federated/decentralized tools are preferred
> >> (for philosophical reasons), even if they do not achieve even feature
> >> parity with centralized tools (c.f. FSF).
> 
> Philosophical and ideological reasons.

My mistake. In my mind, I corrected the grammar to skip the "for", not to
insert "developers" after "some".

Still. Based on my experience in the workforce, it appears to me that you
still confuse ease with convenience: many Windows developers I worked with
are quite capable of doing much more complicated things than configuring
imap-send/send-email. But why would they?

Coming back to your "philosophical and ideological reasons". IMO it is not
very wise to go down that path. Much wiser would it be to make provable
time savings a guiding principle.

Ciao,
Johannes

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jeff King]

On Thu, Aug 18, 2016 at 02:42:34PM +0200, Johannes Schindelin wrote:

> BTW I take this thread as yet another proof that people are unhappy with
> mail list-based review: if you have to build *that much* tooling around it
> (and Peff & Junio certainly have a megaton of advanced and sophisticated
> tooling around it, holy cow!) it is really incorrect to state that the
> mail list-driven approach works for you. It is much closer to the truth to
> say that the mail-list-plus-loads-of-custom-tools-driven approach works
> for you.
> 
> I am really not a fan of this.
> 
> The theory "it's inclusive because everyone has access to mail" falls on
> its face, badly, when even old timers have to build entire infrastructures
> around it just to begin to be able to use it efficiently.
> 
> It reminds me of an old software developer I met long ago, who claimed CVS
> works for him. He had written tens of thousands of lines of shell scripts,
> is what allowed "CVS" to work for him.
> 
> Same here. Old dogs claim the mail list-approach works for them. Nope.
> Doesn't. Else you would not have written all those custom scripts.

I read this over, didn't agree, waited a whole day for perspective, and
still just can't agree. So now I'm responding. :)

There is nothing wrong with building tooling around your workflow. If we
had a GitHub-based workflow, I'd build tooling around that, too. One of
the things I _like_ about a mail-based workflow is how easy it is to
build that tooling, and to get it to integrate with other existing
tools. It's the major reason I'm resistant to moving development to
GitHub. Not only would I have to throw away all of my tools, but I'm not
sure I could easily build equivalent ones.

Now, I am perfectly open to the idea that more of the tooling should be
standardized, so people do not have to build their own. But the major
problem there is that so much of what I've built is about gluing things
together for the rest of _my_ tools. I've shared my techniques and
scripts for using mutt several times, but they don't do somebody a lick
of good if they are using gmail or thunderbird.

So I don't really think I have a megaton of tooling. I just have a
little bit of glue to existing tools I was using anyway. And I think
that is where the disconnect is. If you are not using mutt already, then
it sure seems like a big jump to change your MUA. And I'm sympathetic to
that. But I don't think that means that the mailing-list approach is not
working for me, as you claim in the last paragraph.

-Peff

PS There _are_ some open questions in our workflow that are not really
   mailing list specific. E.g., the fact that it is hard to see whether
   and if your patch was picked up by Junio, what changes were made,
   tracking multiple versions, etc. I _don't_ have good tooling for
   that, but it's something that could be made generally available, as
   it's unrelated to the MUA. It's also not necessarily specific to
   mailing list development (e.g., a push-based workflow that
   aggressively rebases runs into the same versioning questions).

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jakub Narębski]

W dniu 19.08.2016 o 17:03, Jeff King pisze:
[...]
 
> There is nothing wrong with building tooling around your workflow. If we
> had a GitHub-based workflow, I'd build tooling around that, too. One of
> the things I _like_ about a mail-based workflow is how easy it is to
> build that tooling, and to get it to integrate with other existing
> tools. It's the major reason I'm resistant to moving development to
> GitHub. Not only would I have to throw away all of my tools, but I'm not
> sure I could easily build equivalent ones.

Also, you would have the same problem with tooling around specific Git
hosting site as there is with tooling around specific email client.
Tool that works with GitHub (like submitGit) won't work with Bitbucket
or GitLab.

Well, unless such tooling was built in in all hosting sites.  They have
support for sending mails anyway, isn't it?

But perhaps the problem is current lack of tooling in the opposite direction,
namely getting patches from mailing list and applying them to GitHub repo,
or Bitbucket, or GitLab.  Though with working Git, it is something easier
than sending patches via email; it is enough that email client can save
email to a file (or better, whole sub-thread to file or files).

Also, there is lack of tools that convert inline (in-diff) comments on
GitHub (or Bitbucket, or GitLab) to email with review...


There is also distrust for centralized solutions (email is, in principle,
decentralized).  See projects like https://solid.mit.edu, etc.

> Now, I am perfectly open to the idea that more of the tooling should be
> standardized, so people do not have to build their own. But the major
> problem there is that so much of what I've built is about gluing things
> together for the rest of _my_ tools. I've shared my techniques and
> scripts for using mutt several times, but they don't do somebody a lick
> of good if they are using gmail or thunderbird.

It would be nice to have links to those tools in SubmittingPatches and/or
the Git homepage.

[...] 
> PS There _are_ some open questions in our workflow that are not really
>    mailing list specific. E.g., the fact that it is hard to see whether
>    and if your patch was picked up by Junio, what changes were made,
>    tracking multiple versions, etc. I _don't_ have good tooling for
>    that, but it's something that could be made generally available, as
>    it's unrelated to the MUA. It's also not necessarily specific to
>    mailing list development (e.g., a push-based workflow that
>    aggressively rebases runs into the same versioning questions).

Well, "What's cooking ..." and 'pu' branch helps, I think.

-- 
Jakub Narębski

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 08/20/2016 03:57 PM, Jakub Narębski wrote:

> But perhaps the problem is current lack of tooling in the opposite direction,
> namely getting patches from mailing list and applying them to GitHub repo,
> or Bitbucket, or GitLab.  Though with working Git, it is something easier
> than sending patches via email; it is enough that email client can save
> email to a file (or better, whole sub-thread to file or files).

Given that public-inbox provides an NNTP interface, couldn't the ARTICLE 
<message-id> NNTP command be used to easily retrieve the messages in a 
given patch series (at least compared to POP or IMAP).  Perhaps 
git-send-email could be modified to include the message-id value of each 
patch in the series that it sends to the mailing list and include it in 
the cover letter.

Then a script could be written (i.e., git-download-patch) which could 
parse the cover letter message (specified using its message-id), and 
download all the patches in series, which can then be applied using 
git-am.  This would in fact take the email client out of the equation in 
terms of saving patches.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

Hi Arif,

On Tue, 23 Aug 2016, Arif Khokar wrote:

> On 08/20/2016 03:57 PM, Jakub Narębski wrote:
> 
> > But perhaps the problem is current lack of tooling in the opposite
> > direction, namely getting patches from mailing list and applying them
> > to GitHub repo, or Bitbucket, or GitLab.  Though with working Git, it
> > is something easier than sending patches via email; it is enough that
> > email client can save email to a file (or better, whole sub-thread to
> > file or files).
> 
> Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
> <message-id> NNTP command be used to easily retrieve the messages in a
> given patch series (at least compared to POP or IMAP).  Perhaps
> git-send-email could be modified to include the message-id value of each
> patch in the series that it sends to the mailing list and include it in
> the cover letter.

I am no expert in the NNTP protocol (I abandoned News long ago), but if
you go from HTML, you can automate the process without requiring changes
in format-patch.

> Then a script could be written (i.e., git-download-patch) which could
> parse the cover letter message (specified using its message-id), and
> download all the patches in series, which can then be applied using
> git-am.  This would in fact take the email client out of the equation in
> terms of saving patches.

I recently adapted an old script I had to apply an entire patch series
given the GMane link to its cover letter:

https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh

Maybe you find it in you to adapt that to work with public-inbox.org?

Ciao,
Johannes

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi Arif,
> 
> On Tue, 23 Aug 2016, Arif Khokar wrote:
> 
> > On 08/20/2016 03:57 PM, Jakub Narębski wrote:
> > 
> > > But perhaps the problem is current lack of tooling in the opposite
> > > direction, namely getting patches from mailing list and applying them
> > > to GitHub repo, or Bitbucket, or GitLab.  Though with working Git, it
> > > is something easier than sending patches via email; it is enough that
> > > email client can save email to a file (or better, whole sub-thread to
> > > file or files).
> > 
> > Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
> > <message-id> NNTP command be used to easily retrieve the messages in a
> > given patch series (at least compared to POP or IMAP).  Perhaps
> > git-send-email could be modified to include the message-id value of each
> > patch in the series that it sends to the mailing list and include it in
> > the cover letter.

I think that makes sense; perhaps an X-Git-Followups: header
from send-email which lists the child Message-IDs the same way
References: does for ancestors.  (perhaps there's already a
standardized header for listing children)

I thought about allowing a giant MIME message with all the
patches attached, too but that won't work for a large patch
series due to size limits along various SMTP hops.
Compression might make spam filters unhappy, too.

> I am no expert in the NNTP protocol (I abandoned News long ago), but if
> you go from HTML, you can automate the process without requiring changes
> in format-patch.
> 
> > Then a script could be written (i.e., git-download-patch) which could
> > parse the cover letter message (specified using its message-id), and
> > download all the patches in series, which can then be applied using
> > git-am.  This would in fact take the email client out of the equation in
> > terms of saving patches.

	w3m -dump -dump_source nntp://<NNTP-server>/<Message-ID>

ought to already work for news.gmane.org and news.public-inbox.org

The Net::NNTP Perl module is a standard part of the Perl distro
for many years, now (along with Net::SMTP), so that would not
be a roadblock for implementing a custom downloader distributed
with git.

> I recently adapted an old script I had to apply an entire patch series
> given the GMane link to its cover letter:
> 
> https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh
> 
> Maybe you find it in you to adapt that to work with public-inbox.org?

I would be hesitant to depend too much on public-inbox.org until
more mirrors appear.  Even then, NNTP is a better-established
protocol and a fallback to news.gmane still works.
(public-inbox.org is powered by hamsters running on wheels,
 sometimes I let them rest :)

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jeff King]

On Wed, Aug 24, 2016 at 06:49:38PM +0000, Eric Wong wrote:

> > > Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
> > > <message-id> NNTP command be used to easily retrieve the messages in a
> > > given patch series (at least compared to POP or IMAP).  Perhaps
> > > git-send-email could be modified to include the message-id value of each
> > > patch in the series that it sends to the mailing list and include it in
> > > the cover letter.
> 
> I think that makes sense; perhaps an X-Git-Followups: header
> from send-email which lists the child Message-IDs the same way
> References: does for ancestors.  (perhaps there's already a
> standardized header for listing children)

I think that's harder to adapt to some workflows, since it implies
generating all of the message-ids ahead of time (whereas if you are
feeding the messages into an existing MUA, it may generate them on the
fly as it sends).

> I thought about allowing a giant MIME message with all the
> patches attached, too but that won't work for a large patch
> series due to size limits along various SMTP hops.
> Compression might make spam filters unhappy, too.

This was a problem faced by binary groups on Usenet, which had to split
large files across many messages.

It has been a long time since I've dealt with those, but I think the
state of the art involved using "1/20", "2/20", etc in the subjects to
piece together the original. There may also have been header or body
content that included a unique id, so you always knew which messages
were part of a set.

They also used things like forward error correction to handle dropped
messages, but I don't think we need to go that far.

So parsing the "PATCH 1/20" headers sounds hacky, but I think it has
worked for years in other communities.

-Peff

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Jeff King <peff@peff.net> wrote:
> On Wed, Aug 24, 2016 at 06:49:38PM +0000, Eric Wong wrote:
> > > > Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
> > > > <message-id> NNTP command be used to easily retrieve the messages in a
> > > > given patch series (at least compared to POP or IMAP).  Perhaps
> > > > git-send-email could be modified to include the message-id value of each
> > > > patch in the series that it sends to the mailing list and include it in
> > > > the cover letter.
> > 
> > I think that makes sense; perhaps an X-Git-Followups: header
> > from send-email which lists the child Message-IDs the same way
> > References: does for ancestors.  (perhaps there's already a
> > standardized header for listing children)
> 
> I think that's harder to adapt to some workflows, since it implies
> generating all of the message-ids ahead of time (whereas if you are
> feeding the messages into an existing MUA, it may generate them on the
> fly as it sends).

Yeah, it would be limited to git send-email users, only :<

> > I thought about allowing a giant MIME message with all the
> > patches attached, too but that won't work for a large patch
> > series due to size limits along various SMTP hops.
> > Compression might make spam filters unhappy, too.
> 
> This was a problem faced by binary groups on Usenet, which had to split
> large files across many messages.
> 
> It has been a long time since I've dealt with those, but I think the
> state of the art involved using "1/20", "2/20", etc in the subjects to
> piece together the original. There may also have been header or body
> content that included a unique id, so you always knew which messages
> were part of a set.
> 
> They also used things like forward error correction to handle dropped
> messages, but I don't think we need to go that far.
> 
> So parsing the "PATCH 1/20" headers sounds hacky, but I think it has
> worked for years in other communities.

nzb (an XML format) seems to be the thing for Usenet binaries,
nowadays.  Maybe it's workable for git, maybe it's overkill or
not worth it for the two (non-.onion) NNTP servers we have.

nzb seems widely supported enough (on a Debian jessie system):

$ apt-cache search nzb
sabnzbdplus - web-based binary newsgrabber with nzb support
sabnzbdplus-theme-classic - classic interface templates for the SABnzbd+ binary newsgrabber
sabnzbdplus-theme-iphone - transitional package for migration to sabnzbdplus-theme-mobile
sabnzbdplus-theme-mobile - mobile interface templates for the SABnzbd+ binary newsgrabber
sabnzbdplus-theme-plush - plush interface templates for the SABnzbd+ binary newsgrabber
sabnzbdplus-theme-smpl - smpl interface templates for the SABnzbd+ binary newsgrabber
libnzb-dev - An nzb based Usenet binary grabber (development files)
libnzb0c2a - An nzb based Usenet binary grabber (runtime library)
lottanzb - simple and automated Usenet downloader for Newzbin (NZB) files
nzb - Usenet binary grabber
nzbget - command-line based binary newsgrabber for nzb files
python-pynzb - unified API for parsing NZB files from NNTP (Usenet) servers
spotweb - web interface to search and filter Usenet spots

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 08/24/2016 02:49 PM, Eric Wong wrote:
>> On Tue, 23 Aug 2016, Arif Khokar wrote:

>>> Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
>>> <message-id> NNTP command be used to easily retrieve the messages in a
>>> given patch series (at least compared to POP or IMAP).  Perhaps
>>> git-send-email could be modified to include the message-id value of each
>>> patch in the series that it sends to the mailing list and include it in
>>> the cover letter.
>
> I think that makes sense; perhaps an X-Git-Followups: header
> from send-email which lists the child Message-IDs the same way
> References: does for ancestors.

That sounds like a better idea compared to what I came up with 
originally and it would be much easier to parse out of the downloaded 
cover letter message headers as opposed to looking for 
delimiters/keywords in the message body.

> (perhaps there's already a standardized header for listing children)

I don't recall ever seeing anything like that in any RFC or message 
header I've read through.  It's an interesting idea though.

> I thought about allowing a giant MIME message with all the
> patches attached, too but that won't work for a large patch
> series due to size limits along various SMTP hops.

I think the vast majority of SMTP servers allow several megabyte 
messages though their configured policy.  I can't speak for those who 
use their own SMTP servers though.  NNTP servers may limit an individual 
message to a megabyte or less though.

> Compression might make spam filters unhappy, too.

Perhaps, but there should be more reliance on IP blacklists and DMARC as 
a first line of defense against SPAM.

>>> Then a script could be written (i.e., git-download-patch) which could
>>> parse the cover letter message (specified using its message-id), and
>>> download all the patches in series, which can then be applied using
>>> git-am.  This would in fact take the email client out of the equation in
>>> terms of saving patches.
>
> 	w3m -dump -dump_source nntp://<NNTP-server>/<Message-ID>
>
> ought to already work for news.gmane.org and news.public-inbox.org

That's interesting.  I didn't know w3m was capable of that.  But, given 
the way you specified it, it doesn't show the article headers. 
Replacing the -dump and -dump_source options with -dump_both would 
display the headers as well.

> The Net::NNTP Perl module is a standard part of the Perl distro
> for many years, now (along with Net::SMTP), so that would not
> be a roadblock for implementing a custom downloader distributed
> with git.

I wrote a prototype one and included it in my response to Johannes in 
<DM5PR17MB1353B99EBD5F4FD23360DD41D3ED0@DM5PR17MB1353.namprd17.prod.outlook.com>. 
  It could be used as a starting point in terms of making it easier to 
download patches to apply with git-am (without having to rely on one's 
MUA or the tooling around it).

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 08/24/2016 11:34 AM, Johannes Schindelin wrote:
> Hi Arif,

Hello Johannes,

> On Tue, 23 Aug 2016, Arif Khokar wrote:
>
>> Given that public-inbox provides an NNTP interface, couldn't the ARTICLE
>> <message-id> NNTP command be used to easily retrieve the messages in a
>> given patch series (at least compared to POP or IMAP).  Perhaps
>> git-send-email could be modified to include the message-id value of each
>> patch in the series that it sends to the mailing list and include it in
>> the cover letter.
>
> I am no expert in the NNTP protocol (I abandoned News long ago), but if
> you go from HTML, you can automate the process without requiring changes
> in format-patch.

Could you elaborate further on what you mean by using HTML in this context?

> I recently adapted an old script I had to apply an entire patch series
> given the GMane link to its cover letter:
>
> https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh
>
> Maybe you find it in you to adapt that to work with public-inbox.org?

I was thinking more along the lines of using NNTP to retrieve the 
patches and save them to disk (rather than HTTP).  For example, a perl 
script like the following could retrieve the article directly over NNTP. 
  I haven't tested whether the resulting file would work with git-am 
though (since it may not meet the criteria of a mbox file).

You can invoke it as follows:

> perl download_patch.pl "<520a941f7472ac1cb4fa41e6bba33a0afc2f5999.1471264971.git.johannes.schindelin@gmx.de>"


* Forgive any formatting issues resulting from my use of Thunderbird as 
a MUA.

use strict;
use warnings;

use Net::NNTP;

# Assume $ARGV[0] is the message id
my $message_id = $ARGV[0];

my $gmane_nntp = Net::NNTP->new('news.gmane.org');
my $article_ref = $gmane_nntp->article($message_id);

# Make a filename like git-format-patch would
my $counter = 1;
my $subject_line = (grep /^Subject: /, @$article_ref)[0];
$subject_line =~ s/^Subject:[^]]+] //;
$subject_line =~ s/ /-/g;
my $filename = sprintf "%04i-%s", $counter, $subject_line;

print "Filename:\n";
print $filename;
print "\n";

print "Article\n";
my $article_str = join "", @$article_ref;

print $article_str;

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 2311 bytes --]

Hi Arif,

On Wed, 24 Aug 2016, Johannes Schindelin wrote:

> On Tue, 23 Aug 2016, Arif Khokar wrote:
> 
> > On 08/20/2016 03:57 PM, Jakub Narębski wrote:
> > 
> > > But perhaps the problem is current lack of tooling in the opposite
> > > direction, namely getting patches from mailing list and applying
> > > them to GitHub repo, or Bitbucket, or GitLab.  Though with working
> > > Git, it is something easier than sending patches via email; it is
> > > enough that email client can save email to a file (or better, whole
> > > sub-thread to file or files).
> > 
> > Given that public-inbox provides an NNTP interface, couldn't the
> > ARTICLE <message-id> NNTP command be used to easily retrieve the
> > messages in a given patch series (at least compared to POP or IMAP).
> > Perhaps git-send-email could be modified to include the message-id
> > value of each patch in the series that it sends to the mailing list
> > and include it in the cover letter.
> 
> I am no expert in the NNTP protocol (I abandoned News long ago), but if
> you go from HTML, you can automate the process without requiring changes
> in format-patch.
> 
> > Then a script could be written (i.e., git-download-patch) which could
> > parse the cover letter message (specified using its message-id), and
> > download all the patches in series, which can then be applied using
> > git-am.  This would in fact take the email client out of the equation
> > in terms of saving patches.
> 
> I recently adapted an old script I had to apply an entire patch series
> given the GMane link to its cover letter:
> 
> https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh
> 
> Maybe you find it in you to adapt that to work with public-inbox.org?

Oh well. That would have been too easy a task, right?

As it happens, I needed this functionality myself (when reworking my
git-path-in-subdir patch to include Mike Rappazzo's previous patch series
that tried to fix the same bug).

I copy-edited the script to work with public-inbox.org, it accepts a
Message-ID or URL or GMane URL and will try to apply the patch (or patch
series) on top of the current revision:

https://github.com/git-for-windows/build-extra/blob/2268850552c7/apply-from-public-inbox.sh

Ciao,
Johannes

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 02/10/2017 11:10 AM, Johannes Schindelin wrote:
> Hi Arif,
>
> On Wed, 24 Aug 2016, Johannes Schindelin wrote:

>> I recently adapted an old script I had to apply an entire patch series
>> given the GMane link to its cover letter:
>>
>> https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh
>>
>> Maybe you find it in you to adapt that to work with public-inbox.org?
>
> Oh well. That would have been too easy a task, right?
>
> As it happens, I needed this functionality myself (when reworking my
> git-path-in-subdir patch to include Mike Rappazzo's previous patch series
> that tried to fix the same bug).
>
> I copy-edited the script to work with public-inbox.org, it accepts a
> Message-ID or URL or GMane URL and will try to apply the patch (or patch
> series) on top of the current revision:
>
> https://github.com/git-for-windows/build-extra/blob/2268850552c7/apply-from-public-inbox.sh

Thanks for the link.  One thing that comes to mind that is that it may 
be better to just download the patches and then manually apply them 
afterwords rather than doing it in the script itself.  Or at least add 
an option to the script to not automatically invoke git am.

Getting back to the point I made when this thread was still active, I 
still think it would be better to be able to list the message-id values 
in the header or body of the cover letter message of a patch series 
(preferably the former) in order to facilitate downloading the patches 
via NNTP from gmane or public-inbox.org.  That would make it easier 
compared to the different, ad-hoc, methods that exist for each email client.

Alternatively, or perhaps in addition to the list of message-ids, a list 
of URLs to public-inbox.org or gmane messages could also be provided for 
those who prefer to download patches via HTTP.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Arif,

On Mon, 13 Feb 2017, Arif Khokar wrote:

> On 02/10/2017 11:10 AM, Johannes Schindelin wrote:
> >
> > On Wed, 24 Aug 2016, Johannes Schindelin wrote:
> 
> > > I recently adapted an old script I had to apply an entire patch
> > > series given the GMane link to its cover letter:
> > >
> > > https://github.com/git-for-windows/build-extra/blob/master/apply-from-gmane.sh
> > >
> > > Maybe you find it in you to adapt that to work with
> > > public-inbox.org?
> >
> > Oh well. That would have been too easy a task, right?
> >
> > As it happens, I needed this functionality myself (when reworking my
> > git-path-in-subdir patch to include Mike Rappazzo's previous patch
> > series that tried to fix the same bug).
> >
> > I copy-edited the script to work with public-inbox.org, it accepts a
> > Message-ID or URL or GMane URL and will try to apply the patch (or
> > patch series) on top of the current revision:
> >
> > https://github.com/git-for-windows/build-extra/blob/2268850552c7/apply-from-public-inbox.sh
> 
> Thanks for the link.  One thing that comes to mind that is that it may
> be better to just download the patches and then manually apply them
> afterwords rather than doing it in the script itself.  Or at least add
> an option to the script to not automatically invoke git am.

I actually had expected *you* to put in a little bit of an effort, too. In
fact, I was very disappointed that you did not even look into porting that
script to use public-inbox instead of GMane.

> Getting back to the point I made when this thread was still active, I
> still think it would be better to be able to list the message-id values
> in the header or body of the cover letter message of a patch series
> (preferably the former) in order to facilitate downloading the patches
> via NNTP from gmane or public-inbox.org.  That would make it easier
> compared to the different, ad-hoc, methods that exist for each email
> client.

You can always do that yourself: you can modify your cover letter to
include that information.

Note that doing this automatically in format-patch may not be appropriate,
as 1) the Message-ID could be modified depending on the mail client used
to send the mails, and 2) it is not unheard of that a developer
finds a bug in the middle of sending a patch series, fixes that bug, and
regenerates the remainder of the patch series, completely rewriting those
Message-IDs.

> Alternatively, or perhaps in addition to the list of message-ids, a list
> of URLs to public-inbox.org or gmane messages could also be provided for
> those who prefer to download patches via HTTP.

At this point, I am a little disinterested in a discussion without code. I
brought some code to the table, after all.

Ciao,
Johannes

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 02/13/2017 09:37 AM, Johannes Schindelin wrote:
> Hi Arif,
>
> On Mon, 13 Feb 2017, Arif Khokar wrote:

>> Thanks for the link.  One thing that comes to mind that is that it may
>> be better to just download the patches and then manually apply them
>> afterwords rather than doing it in the script itself.  Or at least add
>> an option to the script to not automatically invoke git am.
>
> I actually had expected *you* to put in a little bit of an effort, too. In
> fact, I was very disappointed that you did not even look into porting that
> script to use public-inbox instead of GMane.

I wasn't aware of that expectation.  My idea was to use NNTP as a way to 
facilitate the development of a new git utility that would serve as the 
inverse of git-send-email (sort of like the relationship between git 
format-patch and git am), rather than using a

IIRC, I had posted some proof-of-concept Perl code to do so back in 
August in 
<DM5PR17MB1353B99EBD5F4FD23360DD41D3ED0@DM5PR17MB1353.namprd17.prod.outlook.com>

Looking at public-inbox now at the archives of this group, it appears 
that several of the messages I sent weren't archived for some reason 
(and I didn't see any more responses to what I posted at the time).  The 
messages are accessible via NNTP when connecting to gmane though.

Also, looking at the source of the message I referenced, it appears that 
my MUA decided to base64 encode the message for some reason (which may 
have resulted in it getting filtered by those who I sent the message to).

I will look into this more now (given yours and Junio's responses).

>> Getting back to the point I made when this thread was still active, I
>> still think it would be better to be able to list the message-id values
>> in the header or body of the cover letter message of a patch series
>> (preferably the former) in order to facilitate downloading the patches
>> via NNTP from gmane or public-inbox.org.  That would make it easier
>> compared to the different, ad-hoc, methods that exist for each email
>> client.
>
> You can always do that yourself: you can modify your cover letter to
> include that information.

Certainly, but it would be nice to be able to have it done automatically 
by git format-patch (which I'll look into).

> Note that doing this automatically in format-patch may not be appropriate,
> as 1) the Message-ID could be modified depending on the mail client used
> to send the mails

I think the best approach would be not to make including the message-id 
values the default behavior.  Specifying a new command-line option to 
enable that behavior should address those concerns I think.

> and 2) it is not unheard of that a developer
> finds a bug in the middle of sending a patch series, fixes that bug, and
> regenerates the remainder of the patch series, completely rewriting those
> Message-IDs.

Perhaps, but should something like that not warrant a re-roll of sorts. 
That is, one should reply to the partial patch series stating that there 
is a bug that renders this particular patch (series) un-usable and the 
re-roll could be posted as a reply to the original cover letter?

>> Alternatively, or perhaps in addition to the list of message-ids, a list
>> of URLs to public-inbox.org or gmane messages could also be provided for
>> those who prefer to download patches via HTTP.
>
> At this point, I am a little disinterested in a discussion without code. I
> brought some code to the table, after all.

If you have the time, please take a look at the message-id I referenced. 
  If you need, I can re-post the proof-of-concept code.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 02/13/2017 10:56 PM, Arif Khokar wrote:

> I wasn't aware of that expectation.  My idea was to use NNTP as a way to
> facilitate the development of a new git utility that would serve as the
> inverse of git-send-email (sort of like the relationship between git
> format-patch and git am), rather than using a

...custom script that's tightly coupled to gmane and public-inbox.org

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Arif Khokar <arif.i.khokar@gmail.com> wrote:
> On 02/13/2017 09:37 AM, Johannes Schindelin wrote:
> >I actually had expected *you* to put in a little bit of an effort, too. In
> >fact, I was very disappointed that you did not even look into porting that
> >script to use public-inbox instead of GMane.
> 
> I wasn't aware of that expectation.  My idea was to use NNTP as a way to
> facilitate the development of a new git utility that would serve as the
> inverse of git-send-email (sort of like the relationship between git
> format-patch and git am), rather than using a

Speaking for myself, I usually don't expect much, especially
from newcomers.  So I am disappointed to see Dscho's disappointment
aimed at you, Arif.  Especially since you're not a regular and
we have no idea how much free time, attention span, or familiarity
with Bourne shell you have.

> IIRC, I had posted some proof-of-concept Perl code to do so back in August
> in <DM5PR17MB1353B99EBD5F4FD23360DD41D3ED0@DM5PR17MB1353.namprd17.prod.outlook.com>
> 
> Looking at public-inbox now at the archives of this group, it appears that
> several of the messages I sent weren't archived for some reason (and I
> didn't see any more responses to what I posted at the time).  The messages
> are accessible via NNTP when connecting to gmane though.

It looks like it went to gmane via the meta@public-inbox.org to
gmane.mail.public-inbox.general mirror, not via the git@vger mirror.
I can't find it on git@vger's mail-archive.com mirror, either:

https://mail-archive.com/search?q=Arif+Khokar&l=git%40vger.kernel.org

> Also, looking at the source of the message I referenced, it appears that my
> MUA decided to base64 encode the message for some reason (which may have
> resulted in it getting filtered by those who I sent the message to).

It probably wasn't base64, but maybe it was one of these:
http://vger.kernel.org/majordomo-taboos.txt

Or it was the SPF softfail which you can see in the headers on both
gmane and public-inbox.
It might even be the '_' (underscore) in your other address.
But even Junio gets dropped by vger sometimes:
https://public-inbox.org/git/20170127035753.GA2604@dcvr/

But if I had to guess, vger gets hit by truckloads of spam and
the the backscatter volume could become unimaginable, so perhaps
it has good reason to discard silently.



Anyways, the eventual goal of public-inbox is to flip the
mailing list model backwards into "archives first" mode,
so a message needs to make it into public archives before
it goes out to subscribers.  That might prevent or avoid
such problems... *shrug*

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Junio C Hamano]

Arif Khokar <arif.i.khokar@gmail.com> writes:

> ... I
> still think it would be better to be able to list the message-id
> values in the header or body of the cover letter message of a patch
> series (preferably the former) in order to facilitate downloading the
> patches via NNTP from gmane or public-inbox.org.

You are looking at builtin/log.c::make_cover_letter()?  Patches
welcome, but I think you'd need a bit of preparatory refactoring
to allow gen_message_id() be called for all messages _before_ this
function is called, as currently we generate them as we emit each
patch.

> Alternatively, or perhaps in addition to the list of message-ids, a
> list of URLs to public-inbox.org or gmane messages could also be
> provided for those who prefer to download patches via HTTP.

Many people around here do not want to repeat the mistake of relying
too much on one provider.  Listing Message-IDs may be a good idea,
listing URLs that are tied to one provider is less so.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 02/13/2017 02:21 PM, Junio C Hamano wrote:
> Arif Khokar <arif.i.khokar@gmail.com> writes:
>
>> ... I
>> still think it would be better to be able to list the message-id
>> values in the header or body of the cover letter message of a patch
>> series (preferably the former) in order to facilitate downloading the
>> patches via NNTP from gmane or public-inbox.org.
>
> You are looking at builtin/log.c::make_cover_letter()?  Patches
> welcome, but I think you'd need a bit of preparatory refactoring
> to allow gen_message_id() be called for all messages _before_ this
> function is called, as currently we generate them as we emit each
> patch.

Thank you for the advice; I'll look into it.

One concern I have regarding this idea is whether or not SMTP servers 
typically replace a Message-Id header set by the client.  If that's the 
case, then this approach may unexpectedly fail for some people depending 
on what email provider they use (if they don't maintain their own MTA 
setup).

>> Alternatively, or perhaps in addition to the list of message-ids, a
>> list of URLs to public-inbox.org or gmane messages could also be
>> provided for those who prefer to download patches via HTTP.
>
> Many people around here do not want to repeat the mistake of relying
> too much on one provider.  Listing Message-IDs may be a good idea,
> listing URLs that are tied to one provider is less so.

I agree.  I've actually thought that it would be useful to mirror a 
read-only copy of the mailing list on a public newsgroup that could be 
accessed through a free provider such as eternal-september.org or the 
Google groups interface.  It certainly would reduce the potential losing 
easy access to the archives of this email list if gmane and 
public-inbox.org fail.  But I suspect doing something like that would 
potentially increase the spam volume substantially for regular 
participants/contributors.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Junio C Hamano]

Arif Khokar <arif.i.khokar@gmail.com> writes:

> One concern I have regarding this idea is whether or not SMTP servers
> typically replace a Message-Id header set by the client.

The clients are supposed to give Message-IDs, but because some
clients fail to do so, SMTP server implementations are allowed to
add an ID to avoid leaving a message nameless (IIRC, 6.3 in
RFC2821).  So "replace" would be in violation.

But some parts of the world ignore RFCs, so...

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 02/13/2017 11:41 PM, Junio C Hamano wrote:
> Arif Khokar <arif.i.khokar@gmail.com> writes:
>
>> One concern I have regarding this idea is whether or not SMTP servers
>> typically replace a Message-Id header set by the client.
>
> The clients are supposed to give Message-IDs, but because some
> clients fail to do so, SMTP server implementations are allowed to
> add an ID to avoid leaving a message nameless (IIRC, 6.3 in
> RFC2821).  So "replace" would be in violation.
>
> But some parts of the world ignore RFCs, so...

Based on my testing, gmail and comcast (and my work email) will preserve 
the Message-Id header set by the client, but 
hotmail.com/live.com/outlook.com will replace it with their generated value.

Based on a small sample of email addresses of those who post to this 
list, it appears that most people are using their own MTA to send email, 
or are using gmail, so they probably wouldn't be affected by the issue.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jeff King]

On Mon, Feb 13, 2017 at 08:41:51PM -0800, Junio C Hamano wrote:

> Arif Khokar <arif.i.khokar@gmail.com> writes:
> 
> > One concern I have regarding this idea is whether or not SMTP servers
> > typically replace a Message-Id header set by the client.
> 
> The clients are supposed to give Message-IDs, but because some
> clients fail to do so, SMTP server implementations are allowed to
> add an ID to avoid leaving a message nameless (IIRC, 6.3 in
> RFC2821).  So "replace" would be in violation.
> 
> But some parts of the world ignore RFCs, so...

I know there are some terrible servers out there, but I think we can
discount any such server as horribly broken. Rewriting message-ids would
cause threading problems any time the sender referred to their own
messages. So "format-patch --thread" would fail to work, and even
replying to your own message from your "sent" folder would fail.

-Peff

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Peff,

On Fri, 19 Aug 2016, Jeff King wrote:

> On Thu, Aug 18, 2016 at 02:42:34PM +0200, Johannes Schindelin wrote:
> 
> > BTW I take this thread as yet another proof that people are unhappy
> > with mail list-based review: if you have to build *that much* tooling
> > around it (and Peff & Junio certainly have a megaton of advanced and
> > sophisticated tooling around it, holy cow!) it is really incorrect to
> > state that the mail list-driven approach works for you. It is much
> > closer to the truth to say that the
> > mail-list-plus-loads-of-custom-tools-driven approach works for you.
> > 
> > I am really not a fan of this.
> > 
> > The theory "it's inclusive because everyone has access to mail" falls
> > on its face, badly, when even old timers have to build entire
> > infrastructures around it just to begin to be able to use it
> > efficiently.
> > 
> > It reminds me of an old software developer I met long ago, who claimed
> > CVS works for him. He had written tens of thousands of lines of shell
> > scripts, is what allowed "CVS" to work for him.
> > 
> > Same here. Old dogs claim the mail list-approach works for them. Nope.
> > Doesn't. Else you would not have written all those custom scripts.
> 
> I read this over, didn't agree, waited a whole day for perspective, and
> still just can't agree. So now I'm responding. :)

Thank you for your constructive feedback.

Obviously I just can't agree with what you wrote, at least not completely.
So after waiting an entire weekend for perspective, here are my thoughts
on your comments:

> There is nothing wrong with building tooling around your workflow. If we
> had a GitHub-based workflow, I'd build tooling around that, too.

Sure. You, Junio and myself seem to be kings of scripting. Automating
common tasks through scripting is something that allows me to not go
completely bunkers with the workload I have. I imagine it is similar on
your side.

> One of the things I _like_ about a mail-based workflow is how easy it is
> to build that tooling, and to get it to integrate with other existing
> tools.

Here I disagree violently. What I would have agreed to would be a
statement similar to "It is easy to integrate scripting with mutt".

To be quite frank: we are talking about very different things when we talk
about mail-based workflows. Heck, we even talk about gazillions of
different things when we talk about an email! Just think about this here
email: you might read it written in a font that makes it easy to discern
"1" from "l" from "I". For a vast number of people, this is not even true!

So what we are talking about here are apples and oranges and apple cider
vinegar. Email clients are *so vastly different* from one another, it
would be ludicrious to assume that they have anything in common when it
comes to "mail-based workflows".

As a matter of fact, the thing that you pointed out as the most important
("how easy it is [...] to integrate with other existing tools") does *not*
apply for the *vast* majority of email clients, most prominently Outlook,
GMail, Apple Mail and Thunderbird.

And that mere fact is very, very important to keep in mind. We build Git,
which is very, very successful because it fills a need of developers using
means that are accessible to many, directly or indirectly (the
command-line).

Yet, when it comes to contributing to Git's source code, we deviate from
the common path and require a means that is *not* accessible to many. We
require them to use something different than their regular email client.

> It's the major reason I'm resistant to moving development to GitHub. Not
> only would I have to throw away all of my tools, but I'm not sure I
> could easily build equivalent ones.

I am sympathetic to your reasoning (even if I vividly disagree with your
assessment that it would be difficult to build tools around the GitHub
API, I made quite a couple of such tools myself, and it is quite easy, you
can even script it on the command-line using cURL).

However, I have to point out that the Git project is really uninviting to
contributors, and that this resistance is part of what makes it so.

> Now, I am perfectly open to the idea that more of the tooling should be
> standardized, so people do not have to build their own. But the major
> problem there is that so much of what I've built is about gluing things
> together for the rest of _my_ tools. I've shared my techniques and
> scripts for using mutt several times, but they don't do somebody a lick
> of good if they are using gmail or thunderbird.

It is nice of you to share those tools, of course, and you are correct
that the specificity of your tools limits their being useful. I, for one,
cannot use your mutt-based tools.

(I gladly use git-jump now, though, but it is still very limited by its
being specific to vim. In other words, both your mutt-based scripts and
your git-jump script are limited in their audience just by being so
opinionated. This is distinctly different from Git's being unopinionated.)

> So I don't really think I have a megaton of tooling. I just have a
> little bit of glue to existing tools I was using anyway.

Those existing tools are part of that megaton. For the purpose of my
argument, they contribute as much to the barrier of entry to contribute
easily to Git's source code as anything else in the mail-based workflow.

> And I think that is where the disconnect is. If you are not using mutt
> already, then it sure seems like a big jump to change your MUA. And I'm
> sympathetic to that. But I don't think that means that the mailing-list
> approach is not working for me, as you claim in the last paragraph.

Okay, let's call it the mailing-list-plus-mutt-plus-glue approach, then.

My point stands. We are way more uninviting to contributors than
necessary. And a huge part of the problem is that we require contributors
to send their patches inlined into whitespace-preserving mails.

Personally, I think we can do much better than that.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Duy Nguyen]

On Mon, Aug 22, 2016 at 8:06 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> My point stands. We are way more uninviting to contributors than
> necessary. And a huge part of the problem is that we require contributors
> to send their patches inlined into whitespace-preserving mails.

We probably can settle this in the next git survey with a new
question: what's stopping you from contributing to git?
-- 
Duy

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Philip Oakley]

From: "Duy Nguyen" <pclouds@gmail.com>
> On Mon, Aug 22, 2016 at 8:06 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> My point stands. We are way more uninviting to contributors than
>> necessary. And a huge part of the problem is that we require contributors
>> to send their patches inlined into whitespace-preserving mails.
>
> We probably can settle this in the next git survey with a new
> question: what's stopping you from contributing to git?
> -- 
One has to be careful that some of this is preaching to the choir.

Those who have difficulty won't be anywhere near the survey. Those that are 
near the survey will have enough nouce to contribute to the level they 
manage.

Also the argument here has started to be at cross purposes about the issues 
vs the solutions.

I do note that dscho's patches now have the extra footer (below the three 
dashes) e.g.

Published-As: https://github.com/dscho/git/releases/tag/cat-file-filters-v1
Fetch-It-Via: git fetch https://github.com/dscho/git cat-file-filters-v1

If say I used that, and sent my patch series via Outlook Express (<sigh>), 
with it's white space damage, would those footers help once the content has 
been reviewed (rather than white spacing style) in the applying the patch?

Even, Is there a way of accepting email that has HTML embedded by the 
client, rather than [vger, etc.] simply deleting the email as 'not 
acceptable'. Unless users can get past these sort of (to them) petty and 
awkward restrictions, it's going to continue to be difficult to encourage 
participation.

--

Philip

PS Sudden though about the 'Published-As: Fetch-It-Via: ' stuff. I don't 
think we have a method of 'Fetch-As-A-Series' so that what is received from 
the server (e.g. GitHub, as a git command equivalent) is just those 
patches/commits in the series, and the recipient can then see if they apply 
cleanly at the point they want to apply them? This would almost be a 'pack 
file' that is all commit deltas, that can be given to say 'rebase' (or 
'apply'). It could also be expanded back into an mbox format if required...

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Philip,

On Mon, 22 Aug 2016, Philip Oakley wrote:

> From: "Duy Nguyen" <pclouds@gmail.com>
> > On Mon, Aug 22, 2016 at 8:06 PM, Johannes Schindelin
> > <Johannes.Schindelin@gmx.de> wrote:
> > > My point stands. We are way more uninviting to contributors than
> > > necessary. And a huge part of the problem is that we require contributors
> > > to send their patches inlined into whitespace-preserving mails.
> >
> > We probably can settle this in the next git survey with a new
> > question: what's stopping you from contributing to git?
> > -- 
> One has to be careful that some of this is preaching to the choir.

Exactly.

> I do note that dscho's patches now have the extra footer (below the three
> dashes) e.g.
> 
> Published-As: https://github.com/dscho/git/releases/tag/cat-file-filters-v1
> Fetch-It-Via: git fetch https://github.com/dscho/git cat-file-filters-v1
> 
> If say I used that, and sent my patch series via Outlook Express (<sigh>),
> with it's white space damage, would those footers help once the content has
> been reviewed (rather than white spacing style) in the applying the patch?

I considered recommending this as some way to improve the review process.
The problem, of course, is that it is very easy to craft an email with an
innocuous patch and then push some malicious patch to the linked
repository.

Now, with somebody like me who would lose a lot when destroying trust, it
is highly unlikely. But it is possible that in between the hundreds of
sincere contributors a bad apple tries to sneak in bad stuff.

Therefore, if we were to support a Git-driven contribution process that
*also* sends mail, that mail needs to be generated by a trusted source, to
ensure that the content of the mail is identical to the original Git
commits.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Eric Wong]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Mon, 22 Aug 2016, Philip Oakley wrote:
> > I do note that dscho's patches now have the extra footer (below the three
> > dashes) e.g.
> > 
> > Published-As: https://github.com/dscho/git/releases/tag/cat-file-filters-v1
> > Fetch-It-Via: git fetch https://github.com/dscho/git cat-file-filters-v1
> > 
> > If say I used that, and sent my patch series via Outlook Express (<sigh>),
> > with it's white space damage, would those footers help once the content has
> > been reviewed (rather than white spacing style) in the applying the patch?
> 
> I considered recommending this as some way to improve the review process.
> The problem, of course, is that it is very easy to craft an email with an
> innocuous patch and then push some malicious patch to the linked
> repository.

Perhaps an automated checker of some sort packaged with git
would help.
(And perhaps combinable with the downloader Arif proposed)

> Now, with somebody like me who would lose a lot when destroying trust, it
> is highly unlikely. But it is possible that in between the hundreds of
> sincere contributors a bad apple tries to sneak in bad stuff.

Yes, I would never mix reviews + patch applications of emails vs
git-fetched data.  Having a sender providing both is good; but
the recipient needs to pick one or the other to use exclusively
for that series.

Either look exclusively at what is fetched and respond to that;
or look exclusively at emails and ignore data from git fetch.

However, ensuring the emails and the contents of the git fetch
could be done optionally to ensure there's no tampering or
accidents for other reviewers.

> Therefore, if we were to support a Git-driven contribution process that
> *also* sends mail, that mail needs to be generated by a trusted source, to
> ensure that the content of the mail is identical to the original Git
> commits.

For decentralized systems, independent reproducibilility is
needed.  Rather than trusting one source, I'd rather have some
sort of downloading + checking tool which checks multiple
mirrors (git protocols and NNTP).  That would allow users to
independently verify the veracity of what they got emailed vs
what is fetched.

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Eric,

On Wed, 24 Aug 2016, Eric Wong wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> 
> > Now, with somebody like me who would lose a lot when destroying trust,
> > it is highly unlikely. But it is possible that in between the hundreds
> > of sincere contributors a bad apple tries to sneak in bad stuff.
> 
> Yes, I would never mix reviews + patch applications of emails vs
> git-fetched data.

Well, such a categorical statement seems to exclude all convenience I had
in mind.

My idea was that, say, a web service running on a trusted server with a
trusted code base could send mails that would be trusted to contain
correct SHA-1 information, allowing for a review in the mail, but still
use the much, much more convenient Git tools to actually work on the code.

I really hope that not everybody is so categorically against introducing
much needed convenience.

Ciao,
Dscho

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 08/24/2016 09:04 AM, Johannes Schindelin wrote:
> Hi Philip,
>
> On Mon, 22 Aug 2016, Philip Oakley wrote:

>> I do note that dscho's patches now have the extra footer (below the three
>> dashes) e.g.
>>
>> Published-As: https://github.com/dscho/git/releases/tag/cat-file-filters-v1
>> Fetch-It-Via: git fetch https://github.com/dscho/git cat-file-filters-v1

<snip>

> I considered recommending this as some way to improve the review process.
> The problem, of course, is that it is very easy to craft an email with an
> innocuous patch and then push some malicious patch to the linked
> repository.

It should be possible to verify the SHA1 of the blob before and after 
the patch is applied given the values listed near the beginning of the 
git diff output.  So, for instance, if I apply the malicious patch to my 
local repository, the SHA1 of the resulting blob would not match what 
was listed in at least one of the diffs.

But whether that is sufficient for verification depends on the work 
flow.  Are patches typically applied on the blob that corresponds to the 
first SHA1 value listed in the diff for that file?

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Arif,

On Thu, 25 Aug 2016, Arif Khokar wrote:

> On 08/24/2016 09:04 AM, Johannes Schindelin wrote:
> >
> > On Mon, 22 Aug 2016, Philip Oakley wrote:
> 
> >> I do note that dscho's patches now have the extra footer (below the
> >> three dashes) e.g.
> >>
> >> Published-As: https://github.com/dscho/git/releases/tag/cat-file-filters-v1
> >> Fetch-It-Via: git fetch https://github.com/dscho/git cat-file-filters-v1
> 
> <snip>
> 
> > I considered recommending this as some way to improve the review process.
> > The problem, of course, is that it is very easy to craft an email with an
> > innocuous patch and then push some malicious patch to the linked
> > repository.
> 
> It should be possible to verify the SHA1 of the blob before and after 
> the patch is applied given the values listed near the beginning of the 
> git diff output.

There is no guarantee that the SHA-1 has not been tampered with.

Ciao,
Johannes

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Arif Khokar]

On 08/25/2016 09:01 AM, Johannes Schindelin wrote:
> Hi Arif,
>
> On Thu, 25 Aug 2016, Arif Khokar wrote:

>>> I considered recommending this as some way to improve the review process.
>>> The problem, of course, is that it is very easy to craft an email with an
>>> innocuous patch and then push some malicious patch to the linked
>>> repository.
>>
>> It should be possible to verify the SHA1 of the blob before and after
>> the patch is applied given the values listed near the beginning of the
>> git diff output.
>
> There is no guarantee that the SHA-1 has not been tampered with.

I was implying that the resulting SHA1 of the blob after the malicious 
patch was applied would differ compared to the resulting blob after 
applying the innocuous patch.  Even if you alter the SHA1 value within 
the patch itself, it doesn't change the SHA1 of the result (unless 
you're able to get a hash collision).

But, if you want to guarantee that the SHA1 hasn't been tampered in the 
email, you could sign it with your private GPG key and others could 
verify the signature with your public key (assuming the web-of-trust 
applies).

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

Hi Arif,

On Thu, 25 Aug 2016, Arif Khokar wrote:

> On 08/25/2016 09:01 AM, Johannes Schindelin wrote:
> >
> > On Thu, 25 Aug 2016, Arif Khokar wrote:
> 
> >>> I considered recommending this as some way to improve the review
> >>> process.  The problem, of course, is that it is very easy to craft
> >>> an email with an innocuous patch and then push some malicious patch
> >>> to the linked repository.
> >>
> >> It should be possible to verify the SHA1 of the blob before and after
> >> the patch is applied given the values listed near the beginning of
> >> the git diff output.
> >
> > There is no guarantee that the SHA-1 has not been tampered with.
> 
> I was implying that the resulting SHA1 of the blob after the malicious
> patch was applied would differ compared to the resulting blob after
> applying the innocuous patch.  Even if you alter the SHA1 value within
> the patch itself, it doesn't change the SHA1 of the result (unless
> you're able to get a hash collision).
> 
> But, if you want to guarantee that the SHA1 hasn't been tampered in the
> email, you could sign it with your private GPG key and others could
> verify the signature with your public key (assuming the web-of-trust
> applies).

Given that I try to convince my fellow core Git developers to adopt an
*easier* patch submission process, that wastes less of contributors' time,
I would be strongly opposed to requiring a web of trust and GPG signatures
just to be able to submit patches to git.git.

Ciao,
Johannes

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Jakub Narębski]

W dniu 22.08.2016 o 15:15, Duy Nguyen pisze:
> On Mon, Aug 22, 2016 at 8:06 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>>
>> My point stands. We are way more uninviting to contributors than
>> necessary. And a huge part of the problem is that we require contributors
>> to send their patches inlined into whitespace-preserving mails.
> 
> We probably can settle this in the next git survey with a new
> question: what's stopping you from contributing to git?

Added to second take on proposed questions for Git User's Survey 2016
https://public-inbox.org/git/ae804c55-d145-fc90-e1a9-6ebd6c60453a@gmail.com/T/#u
'[RFCv2] Proposed questions for "Git User's Survey 2016", take two'

Message-ID: <91a2ffbe-a73b-fbb9-96da-9aea4d439e5a@gmail.com>
 
-- 
Jakub Narębski

Re: Working with public-inbox.org [Was: [PATCH] rev-parse: respect core.hooksPath in --git-path]

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Hi Kuba & Duy,

On Sun, 28 Aug 2016, Jakub Narębski wrote:

> W dniu 22.08.2016 o 15:15, Duy Nguyen pisze:
> > On Mon, Aug 22, 2016 at 8:06 PM, Johannes Schindelin
> > <Johannes.Schindelin@gmx.de> wrote:
> >>
> >> My point stands. We are way more uninviting to contributors than
> >> necessary. And a huge part of the problem is that we require contributors
> >> to send their patches inlined into whitespace-preserving mails.
> > 
> > We probably can settle this in the next git survey with a new
> > question: what's stopping you from contributing to git?
> 
> Added to second take on proposed questions for Git User's Survey 2016
> https://public-inbox.org/git/ae804c55-d145-fc90-e1a9-6ebd6c60453a@gmail.com/T/#u
> '[RFCv2] Proposed questions for "Git User's Survey 2016", take two'
> 
> Message-ID: <91a2ffbe-a73b-fbb9-96da-9aea4d439e5a@gmail.com>

I would like to strongly caution against putting too much stock into this
users' survey. It is the best we have, granted. Yet I have not heard from
anybody that they participated in the survey, unless they were also
subscribed to the Git mailing list.

Ciao,
Johannes

Announcing Git User's Survey 2016 [was: Working with public-inbox.org]

[Jakub Narębski]

Hello Johannes,

W dniu 28.08.2016 o 10:38, Johannes Schindelin pisze:
> On Sun, 28 Aug 2016, Jakub Narębski wrote:
>> W dniu 22.08.2016 o 15:15, Duy Nguyen pisze:
>>> On Mon, Aug 22, 2016 at 8:06 PM, Johannes Schindelin
>>> <Johannes.Schindelin@gmx.de> wrote:
>>>>
>>>> My point stands. We are way more uninviting to contributors than
>>>> necessary. And a huge part of the problem is that we require contributors
>>>> to send their patches inlined into whitespace-preserving mails.
>>>
>>> We probably can settle this in the next git survey with a new
>>> question: what's stopping you from contributing to git?
>>
>> Added to second take on proposed questions for Git User's Survey 2016
>> https://public-inbox.org/git/ae804c55-d145-fc90-e1a9-6ebd6c60453a@gmail.com/T/#u
>> '[RFCv2] Proposed questions for "Git User's Survey 2016", take two'
>>
>> Message-ID: <91a2ffbe-a73b-fbb9-96da-9aea4d439e5a@gmail.com>
> 
> I would like to strongly caution against putting too much stock into this
> users' survey. It is the best we have, granted. Yet I have not heard from
> anybody that they participated in the survey, unless they were also
> subscribed to the Git mailing list.

I tried in past and will try for this year Git User's Survey to be
announced more widely than just Git mailing list (git@vger.kernel.org).


And I think it was successful; we had 6385 responses in the survey
from 2012 (the one without analysis on Git Wiki) - the last one,
the maximum of responses we had was 11498 responses in 2011.
This is almost an order of magnitude more than number of past and
current contributors ("git shortlog -s | wc -l"), and I think
much more than all current readers and watchers of Git mailing list.

In 2011 there was "How did you hear about this Git User's Survey?"
question at the end (with 7022 responses out of 11498); most popular
answers were
 - news web site or social news site (e.g. Digg, Reddit)  1592 / 22.7%
 - git hosting site                                       1315 / 18.7%
 - other - please specify                                 1287 / 18.3%
   (which was not git mailing list)
 ...
 - git mailing list                                        230 /  3.3%	 
 - git-related mailing list (msysGit, ...)                  53 /  0.8%
https://git.wiki.kernel.org/index.php/GitSurvey2011#35._How_did_you_hear_about_this_Git_User.27s_Survey.3F

In 2012 I have used different channels for different announcements;
examining channel list shows that Git Homepage one was most used.
This does not mean that people who selected those answers, or used
that channels do not read git mailing list; they might just find
the news somewhere else first.

Also, "Which communication channel(s) do you use?" in 2011, there were
only 387 responders who selected git@vger.kernel.org out of 11498
in total (and out of 1018 people who selected at least one answer
in this question).  Though some people might though lurking is not use...
https://git.wiki.kernel.org/index.php/GitSurvey2011#32._Which_communication_channel.28s.29_do_you_use.3F



I am not saying that there would be no bias in Git User's Survey
2016 responses.  I agree with you that we should take caution
when analyzing results of "What's stopping you from contributing
to Git?" question.  But it might be nor as bad as you fear.


That said, where should Git User's Survey 2016 be announced?
The ones that should be easy are:
 - Git mailing list, and related lists
 - #git IRC channel, and its description (and other IRC channels)
 - Git page on Google+
 - Git Homepage
 - Git Wiki
 - Git Rev News

The ones that would be harder are git hosting sites, among others
the GitHub blog.

The ones that would be quite difficult are news sites and news
aggregation sites, such as LWN.net or HackerNews.

Where else?

Thanks in advance,
-- 
Jakub Narębski

Re: Announcing Git User's Survey 2016 [was: Working with public-inbox.org]

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

Hi Kuba,

On Sun, 28 Aug 2016, Jakub Narębski wrote:

> W dniu 28.08.2016 o 10:38, Johannes Schindelin pisze:
>
> > I would like to strongly caution against putting too much stock into
> > this users' survey. It is the best we have, granted. Yet I have not
> > heard from anybody that they participated in the survey, unless they
> > were also subscribed to the Git mailing list.
> 
> I tried in past and will try for this year Git User's Survey to be
> announced more widely than just Git mailing list (git@vger.kernel.org).

I did not mean to criticise you. I think you are doing the best you can,
and it is valuable.

My point is: many professional developers use Git not necessarily because
they want to, but because their day job requires them to do so.

I *highly* doubt that we reach a notable fraction of those developers,
even if they are arguably power users of any development tool, including
Git.

The question is not so much how to advertise the survey. I skip almost all
surveys I am asked to participate in, because I am just a little bit busy
all the time. I feel that my colleagues do the same. Unless forced to take
a survey, they skip it.

Ciao,
Dscho

Re: Announcing Git User's Survey 2016 [was: Working with public-inbox.org]

[Jakub Narębski]

Hello Johannes,

W dniu 09.09.2016 o 15:06, Johannes Schindelin napisał:
> On Sun, 28 Aug 2016, Jakub Narębski wrote:
>> W dniu 28.08.2016 o 10:38, Johannes Schindelin pisze:
>>
>>> I would like to strongly caution against putting too much stock into
>>> this users' survey. It is the best we have, granted. Yet I have not
>>> heard from anybody that they participated in the survey, unless they
>>> were also subscribed to the Git mailing list.
>>
>> I tried in past and will try for this year Git User's Survey to be
>> announced more widely than just Git mailing list (git@vger.kernel.org).
> 
> I did not mean to criticise you. I think you are doing the best you can,
> and it is valuable.
[...]
> 
> The question is not so much how to advertise the survey. I skip almost all
> surveys I am asked to participate in, because I am just a little bit busy
> all the time. I feel that my colleagues do the same. Unless forced to take
> a survey, they skip it.

Right, that's a problem.  Thanks for reminding me.

I hope that the fact that by default (via the use of cookies) you can
return to Survs.com survey at later time (assuming that you do it from the
same computer and the same web browser), and continue responding.  Taking
30 minutes or more at once may be a problem, taking 10 x 3 minutes may
not be.

But I won't have too much hope...
-- 
Jakub Narębski