Running a Guix User and Contributor survey

Running a Guix User and Contributor survey

[Steve George]

Hi,

I've designed a 'Guix User and Contributor Survey' which I'd like to run to 
explore what users love about Guix, and how contributors interact 
with the project. I would love feedback on it.

The rationale for a survey is that it's a way to build up an understanding of 
our users and contributors which can inform our thoughts and decisions. As a 
project that's focused on user-freedom we don't use telemetry or have the 
resources to run professional user research. A survey provides a broad picture 
across our community, it standardises the questions that are asked and can 
show trends across our community. The proposal links to surveys by projects 
like Nix and Fedora which I've used for inspiration.

Informed by those surveys, and my own interations with Guix users I've selected 
questions that delve into how people are adopting and using Guix. The second 
part explores how people contribute to Guix and their interactions with the 
project. Please see the design document for the various considerations and 
constraints.

Some specific ways you can help are:

* The tricky part of running a survey is keeping it to a reasonable length, you 
quickly find there's lots you'd love to know, but the balancing factor is the 
more questions someone has to answer the lower the completion rate! Ideas on 
how to shorten the design while keeping the same areas of enquiry would be 
welcome.

* I'm sure there are alternative word options that might be clearer, along with 
the inevitable typos: either a PR (it's on Codeberg where anyone can create an 
account) or a patch would be welcome.

* If anyone has skills/experience in analysing results and would like to get 
involved I would love help.

Depending on the response/feedback I'll then move into implementing and running it. All the details here:

https://codeberg.org/futurile/guix-survey/src/branch/main/guix-user-contributor-survey-proposal.md

Thanks,

Steve / Futurile

Re: Running a Guix User and Contributor survey

[Luis Felipe]

[-- Attachment #1.1.1: Type: text/plain, Size: 2617 bytes --]

Hi Steve,

On 30/10/24 12:03, Steve George wrote:
> Hi,
>
> I've designed a 'Guix User and Contributor Survey' which I'd like to run to
> explore what users love about Guix, and how contributors interact
> with the project. I would love feedback on it.
>
> The rationale for a survey is that it's a way to build up an understanding of
> our users and contributors which can inform our thoughts and decisions. As a
> project that's focused on user-freedom we don't use telemetry or have the
> resources to run professional user research. A survey provides a broad picture
> across our community, it standardises the questions that are asked and can
> show trends across our community. The proposal links to surveys by projects
> like Nix and Fedora which I've used for inspiration.

I gave it a first look, and I want to see the results already :)


> Informed by those surveys, and my own interations with Guix users I've selected
> questions that delve into how people are adopting and using Guix. The second
> part explores how people contribute to Guix and their interactions with the
> project. Please see the design document for the various considerations and
> constraints.
>
> Some specific ways you can help are:
>
> * The tricky part of running a survey is keeping it to a reasonable length, you
> quickly find there's lots you'd love to know, but the balancing factor is the
> more questions someone has to answer the lower the completion rate! Ideas on
> how to shorten the design while keeping the same areas of enquiry would be
> welcome.

Personally, I think it is a bit long. Would it make sense to make it two 
surveys instead, one for users and one for contributors?

I'm thinking that if it is perceived as lengthy, people might omit the 
free-form questions at the end or be too tired to express themselves as 
they'd like to.


> * I'm sure there are alternative word options that might be clearer, along with
> the inevitable typos: either a PR (it's on Codeberg where anyone can create an
> account) or a patch would be welcome.

I only noticed one typo:

+ decison making → decision making

And two trifles⸮:

1. "Guixers" used to refer to "Guix denizens". I would omit it because, 
to my knowledge, the community hasn't decided on an English demonym yet 
(/me runs and takes cover).
2. Use of "Linux" in the first and other questions to refer to GNU/Linux 
distributions in general. I'd use the latter, to follow the same 
convention used in Guix documentation.

That's all for now.

Looking forward to survey publication :)


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 2881 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Running a Guix User and Contributor survey

[Steve George]

On 30 Oct, Luis Felipe wrote:
(...)
> On 30/10/24 12:03, Steve George wrote:
(...) 
> I gave it a first look, and I want to see the results already :)

Thank-you, appreciate you going through it.

(...)
> > * The tricky part of running a survey is keeping it to a reasonable length, you
> > quickly find there's lots you'd love to know, but the balancing factor is the
> > more questions someone has to answer the lower the completion rate! Ideas on
> > how to shorten the design while keeping the same areas of enquiry would be
> > welcome.
> 
> Personally, I think it is a bit long. Would it make sense to make it two
> surveys instead, one for users and one for contributors?
> 
> I'm thinking that if it is perceived as lengthy, people might omit the
> free-form questions at the end or be too tired to express themselves as
> they'd like to.
(...)

Yeah, I have the same concern. I'm going to have a look at it when it's
laid-up as a questionnaire and see how it seems for filling it out. I'm not
sure about splitting it up - I'll give it some serious thought though as your
reaction is what I was worrying about.

> > * I'm sure there are alternative word options that might be clearer, along with
> > the inevitable typos: either a PR (it's on Codeberg where anyone can create an
> > account) or a patch would be welcome.
> 
> I only noticed one typo:
> 
> + decison making → decision making
> 
> And two trifles⸮:
> 
> 1. "Guixers" used to refer to "Guix denizens". I would omit it because, to
> my knowledge, the community hasn't decided on an English demonym yet (/me
> runs and takes cover).
> 2. Use of "Linux" in the first and other questions to refer to GNU/Linux
> distributions in general. I'd use the latter, to follow the same convention
> used in Guix documentation.

Thanks for these.

Futurile / Steve

Re: Running a Guix User and Contributor survey

[Ekaitz Zarraga]

Hi Steve,

I don't think the survey is very long.

We could try to answer and then share it with a note on the top that 
says how long it takes, so people can decide when to fill it.

Also I like when people have space to answer, because that motivates 
insightful responses. It probably goes against the theory on how to make 
a good survey but I believe it could be interesting to leave the space 
for complex thoughts. I think complexity of thoughts is often ignored 
and it's important to take good decisions.

Some specifics:

To the question about why Guix (Q5) I would add some flavor of "Free 
Software".

About if people is paid for working on Guix (Q22), I think there's 
something lost there even if the second answer can make it fit: people 
that use Guix and they are not paid for contributing in Guix but 
fixing/improving Guix sometimes is some consequence of their job. I 
mean, they are paid for it, but their job is not to improve Guix, but 
something else. It's tangential.

About the financial support (Q29) maybe it's possible to ask how much 
people is open to give to Guix per month.


Those are my two cents. In general, I like the survey.

It's probably ok just to launch it than to try to make it perfect the 
first time. We could run one per year and improve them according to the 
answers we get. Those that don't really give us interesting information 
we could just ignore.

Have you thought about what platform use?

I think we could do with Framasoft:
https://framasoft.frama.io/framaforms/en/

Thanks for taking the time to do this, Steve.

Cheers!
Ekaitz

Re: Running a Guix User and Contributor survey

[Wilko Meyer]

Hi Steve,

Steve George <steve@futurile.net> writes:

> I've designed a 'Guix User and Contributor Survey' which I'd like to run to 
> explore what users love about Guix, and how contributors interact 
> with the project. I would love feedback on it.

Thanks for putting work into this, that's awesome! In terms of general
feedback: There was a general discussion on surveys[0] a year ago, I'd
like to highlight this mail[1] by Katherine in particular, as it
contains excellent points on survey/questionnaire design and good
practices.

Regarding your proposed questionnaire: I think that it would benefit
from asking about the level of Guile proficiency/if there's
prior-experience with lisp family languages/if people learned Guile
because of Guix.

> * The tricky part of running a survey is keeping it to a reasonable length, you 
> quickly find there's lots you'd love to know, but the balancing factor is the 
> more questions someone has to answer the lower the completion rate! Ideas on 
> how to shorten the design while keeping the same areas of enquiry would be 
> welcome.

It could be feasible to categorize questions into two boxes:

- Questions we want to track over time as we want to track how responses
  change over the years, that have to be iterated yearly.
- Questions we want to ask once/every few years, as we want to work with
  a snapshot in time instead of longterm tracking answers in short
  iterations.

and then have a annual/biennial survey, always asking all questions from
the first box, and then split-up questions from the second box
throughout the years. 

> * If anyone has skills/experience in analysing results and would like to get 
> involved I would love help.

Don't know yet what my timebudget will look like when the results are
in, but generally I'd be available for this (if time permits)!

[0]: https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00402.html
[1]: https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00481.html

-- 
Kind regards,
Wilko Meyer

Re: Running a Guix User and Contributor survey

[indieterminacy]

On 2024-11-01 08:59, Wilko Meyer wrote:
> Hi Steve,
> 
> Steve George <steve@futurile.net> writes:
> 
>> I've designed a 'Guix User and Contributor Survey' which I'd like to 
>> run to
>> explore what users love about Guix, and how contributors interact
>> with the project. I would love feedback on it.
> 
> Thanks for putting work into this, that's awesome! In terms of general
> feedback: There was a general discussion on surveys[0] a year ago, I'd
> like to highlight this mail[1] by Katherine in particular, as it
> contains excellent points on survey/questionnaire design and good
> practices.
> 
> Regarding your proposed questionnaire: I think that it would benefit
> from asking about the level of Guile proficiency/if there's
> prior-experience with lisp family languages/if people learned Guile
> because of Guix.
> 
>> * The tricky part of running a survey is keeping it to a reasonable 
>> length, you
>> quickly find there's lots you'd love to know, but the balancing factor 
>> is the
>> more questions someone has to answer the lower the completion rate! 
>> Ideas on
>> how to shorten the design while keeping the same areas of enquiry 
>> would be
>> welcome.
> 
> It could be feasible to categorize questions into two boxes:
> 
> - Questions we want to track over time as we want to track how 
> responses
>   change over the years, that have to be iterated yearly.
> - Questions we want to ask once/every few years, as we want to work 
> with
>   a snapshot in time instead of longterm tracking answers in short
>   iterations.
> 
> and then have a annual/biennial survey, always asking all questions 
> from
> the first box, and then split-up questions from the second box
> throughout the years.
> 

If it is to be a biennial survey it may be useful to allow people to 
upload previous questionaires,
so that they can be parsed for populating new survey responses.

I can provide a parser prior to the subsequent polling in TXR if that 
helps.

>> * If anyone has skills/experience in analysing results and would like 
>> to get
>> involved I would love help.
> 
> Don't know yet what my timebudget will look like when the results are
> in, but generally I'd be available for this (if time permits)!
> 
> [0]: 
> https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00402.html
> [1]: 
> https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00481.html

Re: Running a Guix User and Contributor survey

[Steve George]

Hi Ekaitz,

Thanks for going through it!

On 31 Oct, Ekaitz Zarraga wrote:
> Hi Steve,
> 
> I don't think the survey is very long.
> 
> We could try to answer and then share it with a note on the top that says
> how long it takes, so people can decide when to fill it.
(...)

Agreed, I've added some intro text. And when I've set it up I'll run through it and make sure there's text saying how long it should take.

> Also I like when people have space to answer, because that motivates
> insightful responses. It probably goes against the theory on how to make a
> good survey but I believe it could be interesting to leave the space for
> complex thoughts. I think complexity of thoughts is often ignored and it's
> important to take good decisions.
> 
> Some specifics:
> 
> To the question about why Guix (Q5) I would add some flavor of "Free
> Software".
(...)

Added, thanks for suggesting it.

> About if people is paid for working on Guix (Q22), I think there's something
> lost there even if the second answer can make it fit: people that use Guix
> and they are not paid for contributing in Guix but fixing/improving Guix
> sometimes is some consequence of their job. I mean, they are paid for it,
> but their job is not to improve Guix, but something else. It's tangential.
(...)

Agreed, that we're not fully exploring this for now. For this time I only want 
to give it one question. Lets see what we get back and we can decide if it
should have further depth in future.

> About the financial support (Q29) maybe it's possible to ask how much people
> is open to give to Guix per month.
(...)

Honestly, I think the way to find this out is to "do it". Something to explore
as a follow-up to the other thread maybe!

> Those are my two cents. In general, I like the survey.
> 
> It's probably ok just to launch it than to try to make it perfect the first
> time. We could run one per year and improve them according to the answers we
> get. Those that don't really give us interesting information we could just
> ignore.
(...)

Agreed, it's better to do something, lets see if it gets a response and people fill it in.

> Have you thought about what platform use?
> 
> I think we could do with Framasoft:
> https://framasoft.frama.io/framaforms/en/
(...)

I wasn't aware of this platform, I'll add it to the list.

Thanks for the input!

Steve / Futurile

Re: Running a Guix User and Contributor survey

[Steve George]

Hi Wilko,

Thanks for going through it.

On  1 Nov, Wilko Meyer wrote:
> Hi Steve,
> 
> Steve George <steve@futurile.net> writes:
> 
> > I've designed a 'Guix User and Contributor Survey' which I'd like to run to 
> > explore what users love about Guix, and how contributors interact 
> > with the project. I would love feedback on it.
> 
> Thanks for putting work into this, that's awesome! In terms of general
> feedback: There was a general discussion on surveys[0] a year ago, I'd
> like to highlight this mail[1] by Katherine in particular, as it
> contains excellent points on survey/questionnaire design and good
> practices.
(...)

I remembered that thread when I started my design document, and then promptly forgot! Thanks for the reminder I'll go through it. 

> Regarding your proposed questionnaire: I think that it would benefit
> from asking about the level of Guile proficiency/if there's
> prior-experience with lisp family languages/if people learned Guile
> because of Guix.
(...)

Did you see that I put in Guile/Scheme/Lisp as both motivators for new users, but also asking whether they were challenges for learning. Although, we haven't directly asked 'do you have prior experience in Lisp' we should get good insight into the overall situation.

The second part is separate I think: are you interested in whether Guile is seing new users because of Guix?

> > * The tricky part of running a survey is keeping it to a reasonable length, you 
> > quickly find there's lots you'd love to know, but the balancing factor is the 
> > more questions someone has to answer the lower the completion rate! Ideas on 
> > how to shorten the design while keeping the same areas of enquiry would be 
> > welcome.
> 
> It could be feasible to categorize questions into two boxes:
> 
> - Questions we want to track over time as we want to track how responses
>   change over the years, that have to be iterated yearly.
> - Questions we want to ask once/every few years, as we want to work with
>   a snapshot in time instead of longterm tracking answers in short
>   iterations.
> 
> and then have a annual/biennial survey, always asking all questions from
> the first box, and then split-up questions from the second box
> throughout the years. 
(...)

Yes understood. For the first survey or so, we should expect some changes as trends and areas of concern become clear. 

As we've already seen it's generally true that additional questions are easy to come up with, but removing them is a lot harder.

> > * If anyone has skills/experience in analysing results and would like to get 
> > involved I would love help.
> 
> Don't know yet what my timebudget will look like when the results are
> in, but generally I'd be available for this (if time permits)!
> 

Understood, I'll ping you when the time comes to see if you have any time.

Steve / Futurile