Hero culture among Guix maintainers

Hero culture among Guix maintainers

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 4666 bytes --]

Hey Guix. There's a specific thing I'm motivated to address after the recent security incident with the "cosmetic" patches & all the fallout of that.

In one of the comments that lead off that thread, Mark asked "does anyone else find it worrisome that Raghav has commit access?" I speculate this comment may have predisposed some people to read subsequent comments from Mark as harsh or suspicious.

I distinctly remember reading that message and thinking "oh Mark is not amused." I'd write a message like that if I'd already tried privately to resolve a person's practices and they'd been resistant, causing me to give up on them and leaving me feeling little choice but assuming bad faith. I don't know what history the participants in that incident had with one another, I don't follow the list or know everybody well enough to have any knowledge or good guess.

One way or another, part of the subtext I got from that thread is that Mark, an established and respected senior contributor here, believes making an error like the one Léo and Raghav made is beneath the level of somebody who's entrusted with membership in the committer group. That reminds me of a common attitude I've seen in operations at a lot of companies, that ops are heroes who take their duties very seriously and feel an extreme responsibility to follow procedures and avoid using their power destructively.

That attitude is a liability at any organization, because we're all fallible and guaranteed to fault on occasions, but I think especially so in a high-trust inclusive atmosphere like what Guix has been building. I noticed that Léo joined, got really engaged with improving the software, and was quickly accepted as a committer, which I thought was really cool. I haven't applied for commit access myself yet, both because I have anxiety about acting out of line and thus want more time to learn the norms of the community, and also because I feel reasonably at ease with the tools and processes for non-committers to contribute. But I saw that and thought it was a great sign that a committed contributor like Léo was able to gain the group's trust so quickly. It's a strength and would be a shame to lose that.

But if everyone who's entrusted with commit access is also expected to live up to the heroic responsibilities of the classic ops/sysadmin role, then I think we're setting people up for failure. Ops at the best companies are guaranteed to make mistakes, and they have the cultural training to be Very Sorry about it and Learn Their Lesson and so on. But how do we expect every committer to a volunteer open source project to behave that way? Blaming a volunteer for a bad commit, calling them out on the mat to fess up and say they're sorry, is big "blame the intern" energy and it's ultimately not conducive to our security as an organization. I think that's still true even if you assume good faith and use only factual statements throughout.

It felt to me like Mark was expecting (demanding?) a certain cultural performance from Léo: acknowledgement of what was done wrong, contrition for his part in it, and a promise not to repeat it. This is typical in ops organizations, but it's not necessarily humane, and I don't think it's a reasonable expectation in a volunteer project. A reexamination of the hero culture among the Guix developers might be in order to avoid similar confrontations in the future.

What does it look like to step back from hero culture? One tool I've used working for paranoid security organizations is to require at least two signatures/sign-offs on any merge to a "protected" branch (like master or core-updates,) one of whom has to be part of a "security ops" subgroup who take on responsibility of this extra review. This pratice works on the simple acknowledgement that any given committer is fallible and two heads are better than one. It's impossible to know for sure, but I imagine this practice would have caught the mistake in Léo's commit to core-updates, and might have avoided a lot of anxiety. Some of Christopher Baines recent work on visualizing the impact of commits could also help aid this review task.

I'd also be interested to see a mechanism for marking commits in the "protected" branches as vulnerable, such that "pull" and "time-machine" can give a warning (or refuse to use those commits.) This might make occasional bad commits less catastrophic and thus reduce anxiety, allowing us to maintain a safer git tree without having to rewrite history or maintain heroically high standards of judgment about every commit.

Cheers, and an honest thank you for everybody's thoughtful messages this past week!
Ryan

[-- Attachment #2: Type: text/html, Size: 4951 bytes --]

Re: Hero culture among Guix maintainers

[Jelle Licht]

Hello Ryan,

tl;dr: (!= 'accountability 'blame), but accountability is essential to
any social endeavour.

Ryan Prior <rprior@protonmail.com> writes:

> Hey Guix. There's a specific thing I'm motivated to address after the
> recent security incident with the "cosmetic" patches & all the fallout
> of that.

Some proposals and ideas have been put forward on how to prevent
situations such as the recent one from getting to the ugly point they
did. I do understand the values of the proposed ideas, yet I think they
are an orthogonal concern to what actually went on here, and that is
vagueness on the nature of accountability (and trust, to a lesser
extent).

> One way or another, part of the subtext I got from that thread is that
> Mark, an established and respected senior contributor here, believes
> making an error like the one Léo and Raghav made is beneath the level
> of somebody who's entrusted with membership in the committer
> group. That reminds me of a common attitude I've seen in operations at
> a lot of companies, that ops are heroes who take their duties very
> seriously and feel an extreme responsibility to follow procedures and
> avoid using their power destructively.

If you can indulge me a bit and allow me to make a snide remark: I
sincerely hope we hold folks to the standard of not using their power
destructively, in the general case.

What do you think trust is, essentially? To me, it means exactly what
you describe as being a hero; being expected to be accountable for your
actions and commitments. 

That is not to say that anyone has the expectation you won't be making
mistakes; you will, inevitably. But it is your own responsibility to
either decline the accountability, or to live up to it. Is that scary?
Yes, but so is driving a car the first time, or caring for a newborn, or
operating heavy machinery etc etc.

> That attitude is a liability at any organization, because we're all
> fallible and guaranteed to fault on occasions, but I think especially
> so in a high-trust inclusive atmosphere like what Guix has been
> building. I noticed that Léo joined, got really engaged with improving
> the software, and was quickly accepted as a committer, which I thought
> was really cool. I haven't applied for commit access myself yet, both
> because I have anxiety about acting out of line and thus want more
> time to learn the norms of the community, and also because I feel
> reasonably at ease with the tools and processes for non-committers to
> contribute. But I saw that and thought it was a great sign that a
> committed contributor like Léo was able to gain the group's trust so
> quickly. It's a strength and would be a shame to lose that.

What you describe as hero culture comes across as an uncharitable
interpretation of accountability in general, IMHO. Things only get iffy
if accountability is seen as a zero sum game; 'someone else's mistakes
are my Golden Ticket to a promotion'. Guix does not seem to be such a
project to me.

> But if everyone who's entrusted with commit access is also expected to
> live up to the heroic responsibilities of the classic ops/sysadmin
> role, then I think we're setting people up for failure. Ops at the
> best companies are guaranteed to make mistakes, and they have the
> cultural training to be Very Sorry about it and Learn Their Lesson and
> so on. But how do we expect every committer to a volunteer open source
> project to behave that way? Blaming a volunteer for a bad commit,
> calling them out on the mat to fess up and say they're sorry, is big
> "blame the intern" energy and it's ultimately not conducive to our
> security as an organization. I think that's still true even if you
> assume good faith and use only factual statements throughout.

Because of language barriers this might not be a clear distinction (at
least in my mother tongue), but accountability in a functional
organisation is not at all about blame. It is about being criticial,
reflective and open minded on your role and how one can contribute to
reach a desired outcome. Very often, that involves stating "I forked up"
in case you made an honest mistake. From my point of view, folks are
given the benefit of the doubt in most cases in the Guix community.

> It felt to me like Mark was expecting (demanding?) a certain cultural
> performance from Léo: acknowledgement of what was done wrong,
> contrition for his part in it, and a promise not to repeat it. This is
> typical in ops organizations, but it's not necessarily humane, and I
> don't think it's a reasonable expectation in a volunteer project. A
> reexamination of the hero culture among the Guix developers might be
> in order to avoid similar confrontations in the future.

What you refer to as a cultural performance, might be just that, but
it does serve a set of important functions;

1 (short) confirmation that there was no act of bad faith (e.g. "Oops,
  my mistake")

2 clarify continued dedication to the goals of the organisation/group

3 actionable steps to possibly prevent the mistake from happening again.

Only then is it appropriate to start listing mitigating
circumstances. There are situations where one is accountable, but could
not have prevented a problem from occurring in the first place, but only
after the previous three steps is the 'right' time to push one's side of
the narrative. Without that, you are not perceived to be taking your
accountability seriously. (Mind you, I do this wrong 90% of the time,
I'm just describing the asymptotic 'perfect').

Technology-minded folks often want to skip steps 1 and 2, which leads to
frustration and the breaking down of trust, at least in the short term,
which is shame. One could ask, "why don't we just skip steps 1 and 2,
and assume that those are implied?", but that is a recipe for disaster;
Only the entire thing together shows that somebody is still committed to
being held accountable. Leave out any of the steps, and
misunderstandings can and will arise. A valid and praise-worthy way to
resolve such a situation is to reconsider being accountable, and step
down from a certain role/position.

> What does it look like to step back from hero culture? One tool I've
> used working for paranoid security organizations is to require at
> least two signatures/sign-offs on any merge to a "protected" branch
> (like master or core-updates,) one of whom has to be part of a
> "security ops" subgroup who take on responsibility of this extra
> review. This pratice works on the simple acknowledgement that any
> given committer is fallible and two heads are better than one. It's
> impossible to know for sure, but I imagine this practice would have
> caught the mistake in Léo's commit to core-updates, and might have
> avoided a lot of anxiety. Some of Christopher Baines recent work on
> visualizing the impact of commits could also help aid this review
> task.

Léo (or you, or me, or anyone) will find new, more interesting ways to
make mistakes. Note that I am not stating we should not prevent
preventable mistakes! I am simply stating that your proposal would have
simply delayed the issue at hand, not prevented it entire. 

I'd rather (additionaly) work towards a more sustainable solution, which
I think needs to be an agreement on what accountability (in the context
of Guix) is (or should be!).

> I'd also be interested to see a mechanism for marking commits in the
> "protected" branches as vulnerable, such that "pull" and
> "time-machine" can give a warning (or refuse to use those commits.)
> This might make occasional bad commits less catastrophic and thus
> reduce anxiety, allowing us to maintain a safer git tree without
> having to rewrite history or maintain heroically high standards of
> judgment about every commit.

My first practical contribution in this reply: this seems like a cool
workflow enhancement to have, indeed!

> Cheers, and an honest thank you for everybody's thoughtful messages this past week!
> Ryan

Thank you for your very well formulated message!
 - Jelle

Re: Hero culture among Guix maintainers

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

Hi,

> tl;dr: (!= 'accountability 'blame), but accountability is essential to
> any social endeavour.

I agree with Jelle and I find it essential to adopt this stance within
the Guix community.

I don't think it's made explicit in writing anywhere, maybe we should
update the Code of Conduct with a paragraph on the matter?

> From my point of view, folks are
> given the benefit of the doubt in most cases in the Guix community.

I've witnessed cases where it didn't happen; maybe it should be
encouraged a bit further.
Not sure how though.  Publish a blog article maybe?

> I'd rather (additionaly) work towards a more sustainable solution, which
> I think needs to be an agreement on what accountability (in the context
> of Guix) is (or should be!).

Good idea!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]