Re: Guix Survey (follow up on "How can we decrease the cognitive overhead for contributors?")

[Wilko Meyer <w@wmeyer.eu>]

Hi Katherine,

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> This is awesome, thanks Wilko!
>
> I've been talking with my wife who is the field of psychology. As part
> of her degree, and as part of her ongoing education, she's studied how
> to design studies/surveys so that they're not biased and don't produce
> contaminated data. She's not an expert at this, but she knows more
> about it than me :)

This is super valuable! Thanks for taking the time to share so many
detailed and elobarate considerations on how to design surveys in
general and a survey for guix specifically. This was super interesting
to read and to think about!

> The things we could come up with which we thought were important to
> consider are:
>
> - You must first define your goals for the survey. Is it meant to see
>   who is using Guix? Who is contributing? How they find the
>   contribution process? How they find using Guix? There are many
>   dimensions, and we may need to create more than one survey.

Given the original thread on this list, I'd say the focus should be on the
contribution process, secondarily on who is contributing and maybe the
areas Guix is used in? As much as I'd love to see an extensive survey
that also covers a more detailed picture on Guix usage/e.g. focusses on
particular technical aspects of Guix; I'd have to agree that a narrower
scope that provides data for the issues that have been discusses in the
original thread is probably better (especially considering that the
results have to be analyzed/interpreted and time tends to be a rather
limited resource).

> - The medium of the survey is very important. E.g. some people won't
>   reply to a survey served something that uses JavaScript. Some people
>   may not be able to reply to a survey unless accessibility concerns
>   are met. Some people won't overcome the barrier of having to log in
>   to respond.

I'm by no means an expert on accessibility; AAA-level contrast,
screenreader compatibility, and using a dyslexic friendly typeface by
default should be things to consider as a default. It'd be interesting
to see how other surveys (e.g. Nix, Emacs, Haskell and so on) are
handling these things.

> - Where solicitations to complete the survey are broadcast is very
>   important. E.g. if we only send it to guix-dev, this skews the
>   responses to questions like "where do you talk about Guix".

Definitely, I'm not entirely sure on how to solve this; publishing
surveys on as many channels that seem fitting could maybe mitigate this?
Then again the selection of communication channels is highly subjective
as well.

> - When the solicitations are made is very important. Some religions do
>   not allow use of electronics on certain days, or times of the year.
>   Some people are away on holiday during parts of the year. Some
>   people are trying to meet deadlines during fiscal quarters. It may
>   be impossible to accommodate everyone, but giving a little
>   consideration to the issue and a sufficient window of time may cover
>   most cases.

This is also something where input from other survey creators could
help; I would guess that having a relatively broad participation window
could be part of a solution.

> - When soliciting responses to the survey, it's very important to set
>   expectations about the survey in the solicitation. It is important
>   to briefly describe what the survey is like and how long the survey
>   will take. Without this, some people will have uncertainty about
>   what they're committing to and not even try.
>
> - The survey should endeavor to remain on the shorter end; many will
>   not complete longer surveys.

This is another good reason to start with a narrow focus on questions
regarding contributions instead of a general survey.

> - Does the survey need translation to eliminate language barriers?

Most FLOSS surveys I've looked at were english only; which comes with a
certain language bias. Realistically I'd say that, given a survey may
contain free form questions, translation also seems to be a resource
issue when it comes to analyzing the results.

> - The survey should use a uniform measurement system throughout. Don't
>   use scales with different magnitudes in different questions, and
>   don't suddenly invert whether higher is better or worse.

Good point, this also means that questions should be asked in a way,
that they can be answered using the same metrics/scale?

> - As you've already mentioned, free-form questions are very difficult
>   to quantify, and I think we should use them with caution.
>   Communities rooted in philosophical values, as Guix is, have
>   impassioned people and resolving a large number of free-form
>   responses to a quantitative statement may be difficult.

My approach to free form questions is to, on one hand try to quantify
trends (things that are mentioned often, key topics that are mentioned),
on the other try to derrive actionable items/issues from them that can
be worked on. Quantifiying qualitative responses is cumbersome, and as
you've also pointed out, quite difficult; but identifying trends/key
topics and maybe actionable items/issues from those could work. WDYT?

> - Questions which are intended to solicit a agree/disagree should be
>   phrased as "I" questions, e.g.:
>
> 	On a scale of 1-5, how much do you agree with the phrase "I
> 	like carrots"?
>
> - Questions should not be leading, and be biased towards the positive.
>   E.g., with the carrots example, don't do this:
>
>        Carrots are disgusting. How much do you agree with this?
>
>   and don't do this:
>
>         On a scale of 1-5, how much do you agree with the phrase "I
>         think carrots are disgusting!"
>
> - Up front, it may be difficult to identify all the root-causes of
>   something the project wants to know about. Instead of trying to
>   infer these, ask the questions directly. E.g. instead of questions
>   about liking crunchy vegetables, orange vegetables, and root
>   vegetables, ask whether they like carrots.
>
>   However, if you think you have some idea of the root-causes, you can
>   ask those as well to see if the correlation you think exists does.

If we've a first draft of a prospective, narrowed-down on contributions,
survey, the questions should probably be benchmarked against these
criteria. I revisted my loose collection of survey questions I posted
earlier on here and realized that I probably have to rephrase a vast
majority of those, to be consistent with this.

> - You may want to ensure the survey has "marker questions" which
>   clearly categorize your responder for you to make it easier to make
>   the statements you'd like to make. E.g. if you're interested in
>   analyzing what vegetarians vs omnivores think of carrots, ask that
>   so you don't have to try and infer it later.

I'll revisit the original thread on how to decrease cognitive overhead
for contributors to see what good markers could be. With a grain of
salt, I think we should figure out ways to identify participants that:

- were contributors to guix before but stopped contributing
- want to contribute but experienced blockers that stopped them from
  participating

as well as a way to map out e.g. the frequency of contributions?

> - We were unable to resolve the question of astroturfing wherein one
>   malicious party responds many times to skew the data. This might be
>   difficult to address without relying on a vendor who has solved this
>   concern somehow, but requires logins, JavaScript, or something else
>   people won't use.

The emacs survey doesn't require javascript; they've build a
self-written survey framework in julia iirc to address issues revolving
around javascript and finding ways to do input validation. I don't know
if they've explicitly tried to implement measures against astroturfing
or how their survey framework works in general; but it may be worth
considering. 

> And finally, I'd like to suggest:
>
> I think since this is the result of a discussion about how to lower
> the cognitive overhead of contributing, the goal of this initial
> survey should be:
>
> 1. To quantify how easy it is to contribute to Guix.
> 2. To quantify how easy it is to maintain Guix.
> 3. To correlate (1) and (2) with people's opinion of using email for
> contributions.
> 4. To correlate (1) and (2) with people's opinion of using a forge for
> contributions.
> 5. To correlate (1) and (2) with people's opinion on only improving tooling.
> 6. To be able to do trend-analysis year-over-year on these issues.
>
> I would suggest adding these questions to a survey exploring the
> contribution process:
>
>     On a scale of 1-5, 1 being "Strongly Disagree" and 5 being
>     "Strongly Agree", how much do you agree with the statements:
>
>     "I think it is easy to contribute to Guix."
>
>     "I think contributions to Guix will be reviewed in a timely
>     manner."
>
>     "I think email is the best way to manage introducing code to
>     Guix."
>
>     "I think a web-forge is the best way to manage introducing code to
>     Guix."
>
>     "I think working on tools made specifically for Guix is the best
>     way to improve the contribution process."

These questions are excellent to address large parts of the issues being
discussed in the original thread on this list.

> I am very interested in the usage patterns of Guix, and I firmly
> believe some survey should explore this.
> I'm not sure if we should combine the two; does it make it too long of
> a survey?

I'd say, that it could be beneficial to ask for usage as long as it
helps to map out the background of guixes users. I wouldn't go too much
into detail, but this subset of questions on general usage I shared
before (and maybe a question on familiarity with Guile/Scheme) would
still provide value:

>> - Why do you use Guix? (freeform)
>> - Where/on what platform do you use Guix? (Guix System, on top of other
>>   distributions etc.)
>> - How many years have you been using Guix?
>> - In what context do you use Guix? (academic, work, private etc.)
>> - What do you use Guix for? (packaging, systemconf, reproducible
>>   research and so on)
>> - Have you ever considered to stop using Guix, if so why? (freeform)
>> - Which features keep you using Guix? (should be a list with optional
>>   freeform)
>> - To extend guix packages/work on new packages, you...
>>   ...upstream to guix proper
>>   ...maintain a fork of guix proper
>>   ...maintain your own guix channel
>>   ...provide a guix.scm for the respective projects

to assess the questionees background better/be able to give more
context. WDYT?

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu