Re: Guix Survey (follow up on "How can we decrease the cognitive overhead for contributors?")

[Katherine Cox-Buday <cox.katherine.e@gmail.com>]

This is awesome, thanks Wilko!

I've been talking with my wife who is the field of psychology. As part 
of her degree, and as part of her ongoing education, she's studied how 
to design studies/surveys so that they're not biased and don't produce 
contaminated data. She's not an expert at this, but she knows more about 
it than me :)

The things we could come up with which we thought were important to 
consider are:

- You must first define your goals for the survey. Is it meant to see
   who is using Guix? Who is contributing? How they find the
   contribution process? How they find using Guix? There are many
   dimensions, and we may need to create more than one survey.

- The medium of the survey is very important. E.g. some people won't
   reply to a survey served something that uses JavaScript. Some people
   may not be able to reply to a survey unless accessibility concerns
   are met. Some people won't overcome the barrier of having to log in
   to respond.

- Where solicitations to complete the survey are broadcast is very
   important. E.g. if we only send it to guix-dev, this skews the
   responses to questions like "where do you talk about Guix".

- When the solicitations are made is very important. Some religions do
   not allow use of electronics on certain days, or times of the year.
   Some people are away on holiday during parts of the year. Some
   people are trying to meet deadlines during fiscal quarters. It may
   be impossible to accommodate everyone, but giving a little
   consideration to the issue and a sufficient window of time may cover
   most cases.

- When soliciting responses to the survey, it's very important to set
   expectations about the survey in the solicitation. It is important
   to briefly describe what the survey is like and how long the survey
   will take. Without this, some people will have uncertainty about
   what they're committing to and not even try.

- The survey should endeavor to remain on the shorter end; many will
   not complete longer surveys.

- Does the survey need translation to eliminate language barriers?

- The survey should use a uniform measurement system throughout. Don't
   use scales with different magnitudes in different questions, and
   don't suddenly invert whether higher is better or worse.

- As you've already mentioned, free-form questions are very difficult
   to quantify, and I think we should use them with caution.
   Communities rooted in philosophical values, as Guix is, have
   impassioned people and resolving a large number of free-form
   responses to a quantitative statement may be difficult.

- Questions which are intended to solicit a agree/disagree should be
   phrased as "I" questions, e.g.:

	On a scale of 1-5, how much do you agree with the phrase "I
	like carrots"?

- Questions should not be leading, and be biased towards the positive.
   E.g., with the carrots example, don't do this:

        Carrots are disgusting. How much do you agree with this?

   and don't do this:

         On a scale of 1-5, how much do you agree with the phrase "I
         think carrots are disgusting!"

- Up front, it may be difficult to identify all the root-causes of
   something the project wants to know about. Instead of trying to
   infer these, ask the questions directly. E.g. instead of questions
   about liking crunchy vegetables, orange vegetables, and root
   vegetables, ask whether they like carrots.

   However, if you think you have some idea of the root-causes, you can
   ask those as well to see if the correlation you think exists does.

- You may want to ensure the survey has "marker questions" which
   clearly categorize your responder for you to make it easier to make
   the statements you'd like to make. E.g. if you're interested in
   analyzing what vegetarians vs omnivores think of carrots, ask that
   so you don't have to try and infer it later.

- We were unable to resolve the question of astroturfing wherein one
   malicious party responds many times to skew the data. This might be
   difficult to address without relying on a vendor who has solved this
   concern somehow, but requires logins, JavaScript, or something else
   people won't use.

And finally, I'd like to suggest:

I think since this is the result of a discussion about how to lower the 
cognitive overhead of contributing, the goal of this initial survey 
should be:

1. To quantify how easy it is to contribute to Guix.
2. To quantify how easy it is to maintain Guix.
3. To correlate (1) and (2) with people's opinion of using email for 
contributions.
4. To correlate (1) and (2) with people's opinion of using a forge for 
contributions.
5. To correlate (1) and (2) with people's opinion on only improving tooling.
6. To be able to do trend-analysis year-over-year on these issues.

I would suggest adding these questions to a survey exploring the 
contribution process:

     On a scale of 1-5, 1 being "Strongly Disagree" and 5 being
     "Strongly Agree", how much do you agree with the statements:

     "I think it is easy to contribute to Guix."

     "I think contributions to Guix will be reviewed in a timely
     manner."

     "I think email is the best way to manage introducing code to
     Guix."

     "I think a web-forge is the best way to manage introducing code to
     Guix."

     "I think working on tools made specifically for Guix is the best
     way to improve the contribution process."

I am very interested in the usage patterns of Guix, and I firmly believe 
some survey should explore this. I'm not sure if we should combine the 
two; does it make it too long of a survey?