Re: Guix Survey (follow up on "How can we decrease the cognitive overhead for contributors?")

[Katherine Cox-Buday <cox.katherine.e@gmail.com>]

On 10/2/23 5:24 AM, Wilko Meyer wrote:

>> - Where solicitations to complete the survey are broadcast is very
>>    important. E.g. if we only send it to guix-dev, this skews the
>>    responses to questions like "where do you talk about Guix".
> 
> Definitely, I'm not entirely sure on how to solve this; publishing
> surveys on as many channels that seem fitting could maybe mitigate this?
> Then again the selection of communication channels is highly subjective
> as well.

I don't think we have to be very selective about where to broadcast the 
survey. I think the answer is: anywhere Guix people hang out or anyone 
feels it might be useful to do so.

I think the only danger here is missing some popular hangout spots due 
to ignorance of their existence. We should enumerate them to ensure we 
catch them all.

>> - When soliciting responses to the survey, it's very important to set
>>    expectations about the survey in the solicitation. It is important
>>    to briefly describe what the survey is like and how long the survey
>>    will take. Without this, some people will have uncertainty about
>>    what they're committing to and not even try.
>>
>> - The survey should endeavor to remain on the shorter end; many will
>>    not complete longer surveys.
> 
> This is another good reason to start with a narrow focus on questions
> regarding contributions instead of a general survey.
> 
>> - Does the survey need translation to eliminate language barriers?
> 
> Most FLOSS surveys I've looked at were english only; which comes with a
> certain language bias. Realistically I'd say that, given a survey may
> contain free form questions, translation also seems to be a resource
> issue when it comes to analyzing the results.

Does the GNU project have a "general translation" team?

Maybe some of our Guix community members who speak multiple languages 
would be willing to translate the survey into their primary language?

>> - The survey should use a uniform measurement system throughout. Don't
>>    use scales with different magnitudes in different questions, and
>>    don't suddenly invert whether higher is better or worse.
> 
> Good point, this also means that questions should be asked in a way,
> that they can be answered using the same metrics/scale?

I think that's the idea and should be the rule for which there are 
exceptions.

>> - As you've already mentioned, free-form questions are very difficult
>>    to quantify, and I think we should use them with caution.
>>    Communities rooted in philosophical values, as Guix is, have
>>    impassioned people and resolving a large number of free-form
>>    responses to a quantitative statement may be difficult.
> 
> My approach to free form questions is to, on one hand try to quantify
> trends (things that are mentioned often, key topics that are mentioned),
> on the other try to derrive actionable items/issues from them that can
> be worked on. Quantifiying qualitative responses is cumbersome, and as
> you've also pointed out, quite difficult; but identifying trends/key
> topics and maybe actionable items/issues from those could work. WDYT?

My opinion is that we should not do free form questions for this first 
time. We're new at this, we have enough topics to cover, and the topics 
we are covering seem to cause a lot of discussion (that's good) which 
could lead to a lot of text to read through.

>> - Up front, it may be difficult to identify all the root-causes of
>>    something the project wants to know about. Instead of trying to
>>    infer these, ask the questions directly. E.g. instead of questions
>>    about liking crunchy vegetables, orange vegetables, and root
>>    vegetables, ask whether they like carrots.
>>
>>    However, if you think you have some idea of the root-causes, you can
>>    ask those as well to see if the correlation you think exists does.
> 
> If we've a first draft of a prospective, narrowed-down on contributions,
> survey, the questions should probably be benchmarked against these
> criteria. I revisted my loose collection of survey questions I posted
> earlier on here and realized that I probably have to rephrase a vast
> majority of those, to be consistent with this.
> 
>> - You may want to ensure the survey has "marker questions" which
>>    clearly categorize your responder for you to make it easier to make
>>    the statements you'd like to make. E.g. if you're interested in
>>    analyzing what vegetarians vs omnivores think of carrots, ask that
>>    so you don't have to try and infer it later.
> 
> I'll revisit the original thread on how to decrease cognitive overhead
> for contributors to see what good markers could be. With a grain of
> salt, I think we should figure out ways to identify participants that:
> 
> - were contributors to guix before but stopped contributing
Maybe:

      (1) Have you made contributions to Guix in the past?

This is our marker question.

combined with:

      (2) How many contributions to Guix per month would you estimate
      you've made in the past year? (identify ranges we're interested in)

This is a dimension that would be useful to have for analyzing other 
responses. We can infer that

> - want to contribute but experienced blockers that stopped them from
>    participating

      (3) On a scale of 1-5, how much do you agree with the statement, "I
      would like to contribute to Guix."

combined with

      (4) On a scale of 1-5, how much do you agree with the statement, "I
      think it is easy to contribute to Guix."

We can then infer that people with higher scores to (3), lower scores to 
(4), and low contribution ranges from (2) are experiencing blockers that 
stop them from participating.

> as well as a way to map out e.g. the frequency of contributions?

(covered above)

>> And finally, I'd like to suggest:
>>
>> I think since this is the result of a discussion about how to lower
>> the cognitive overhead of contributing, the goal of this initial
>> survey should be:
>>
>> 1. To quantify how easy it is to contribute to Guix.
>> 2. To quantify how easy it is to maintain Guix.
>> 3. To correlate (1) and (2) with people's opinion of using email for
>> contributions.
>> 4. To correlate (1) and (2) with people's opinion of using a forge for
>> contributions.
>> 5. To correlate (1) and (2) with people's opinion on only improving tooling.
>> 6. To be able to do trend-analysis year-over-year on these issues.
>>
>> I would suggest adding these questions to a survey exploring the
>> contribution process:
>>
>>      On a scale of 1-5, 1 being "Strongly Disagree" and 5 being
>>      "Strongly Agree", how much do you agree with the statements:
>>
>>      "I think it is easy to contribute to Guix."
>>
>>      "I think contributions to Guix will be reviewed in a timely
>>      manner."
>>
>>      "I think email is the best way to manage introducing code to
>>      Guix."
>>
>>      "I think a web-forge is the best way to manage introducing code to
>>      Guix."
>>
>>      "I think working on tools made specifically for Guix is the best
>>      way to improve the contribution process."
> 
> These questions are excellent to address large parts of the issues being
> discussed in the original thread on this list.
> 
>> I am very interested in the usage patterns of Guix, and I firmly
>> believe some survey should explore this.
>> I'm not sure if we should combine the two; does it make it too long of
>> a survey?
> 
> I'd say, that it could be beneficial to ask for usage as long as it
> helps to map out the background of guixes users. I wouldn't go too much
> into detail, but this subset of questions on general usage I shared
> before (and maybe a question on familiarity with Guile/Scheme) would
> still provide value:
> 
>>> - Why do you use Guix? (freeform)
>>> - Where/on what platform do you use Guix? (Guix System, on top of other
>>>    distributions etc.)
>>> - How many years have you been using Guix?
>>> - In what context do you use Guix? (academic, work, private etc.)
>>> - What do you use Guix for? (packaging, systemconf, reproducible
>>>    research and so on)

This might be too broad a question since Guix is an operating system.

>>> - Have you ever considered to stop using Guix, if so why? (freeform)
>>> - Which features keep you using Guix? (should be a list with optional
>>>    freeform)

This might also be too broad a question since people's motivations are 
likely to be varied.

>>> - To extend guix packages/work on new packages, you...
>>>    ...upstream to guix proper
>>>    ...maintain a fork of guix proper
>>>    ...maintain your own guix channel
>>>    ...provide a guix.scm for the respective projects
> 
> to assess the questionees background better/be able to give more
> context. WDYT?

As discussed above, my opinion is that we should drop the free-form 
questions. Other than that and my comments, I'd be OK with including 
these style of questions in this survey.

But these are my opinions. I'm open to discussion!

Is this rough list of questions our first pass at what the survey should 
contain?

Have you done any more research on what other communities are doing?

What are next steps?