From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id UDEQMil8DGX2GwAA9RJhRA:P1 (envelope-from ) for ; Thu, 21 Sep 2023 19:23:53 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id UDEQMil8DGX2GwAA9RJhRA (envelope-from ) for ; Thu, 21 Sep 2023 19:23:53 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5EA8B4D2DD for ; Thu, 21 Sep 2023 19:23:53 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=A+fmNvws; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1695317033; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=v9rvDJTj7jkKD2WnIXl+TMprEUQuM2dkTs2tT3lBB9Q=; b=lKjHs1gG5WOofOj2Y/8GFfjgAPkLRogaoFwWqVMyVcNJnNK5bpN0c/u4H6uXREMqQuJKJA KAiJfmSVqIQGePCC0Lcp7UJ4ohcVnDO79Otvl5agAfNmRgH/HoAupY4eTt2EB2vuPmVrsD 6v1XIiK7DU1OplSh8tmzn9sIJig1pYaFNU/lhn/eRSuLAjG0T8e9WeX1E2EXqnOp9tTVf3 hnh1PKY+bh3movIvh74OAWKcc/pnFIuWBcDeiSFYPuHR8a/kYS5dXkfANS/kxkn8/vly/d t7xYAXMlp2TtcuNFtdqFmv2zNLZ5LiVD0xxim3mM+f+cSIPMqess0dmZzYk6Cw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=A+fmNvws; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1695317033; a=rsa-sha256; cv=none; b=Dhoin1b9Pa3j04QaMSBLquo/JH7n/EaYJ8gGrpQ4eGJh6dOBlJBXwPBj7FJqXq8oHc6x+W 1O8pN7SwwfH7zORmV1bbnCLiZS4sE4zF1fIOIUu7dTSZ4+0VbOMtdlpBZQzn0lf621R0/u PlRQJeN0DA5h2ToToZKXS3ThXXYCQeu+mnvKXtz6LBfotAWEE/zDzGSTEGQgQ24uF4nwSj 2p3QB0OyT+sHt0sAcwUWTEFH/RvRWo2f25N/no0Fjiwj9dEPO2E3ooM9GjVRsyC6k8OS3j ZvoEhFI/fEla/e2LGMR+yNGTjiZIYYVHXYKDwgM82Pr8baxkTMuEbnOURqI2Ww== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qjNOQ-0004rB-Dp; Thu, 21 Sep 2023 13:23:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qjNOP-0004r3-31 for guix-devel@gnu.org; Thu, 21 Sep 2023 13:23:09 -0400 Received: from mail-io1-xd2c.google.com ([2607:f8b0:4864:20::d2c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qjNON-0004kY-9x for guix-devel@gnu.org; Thu, 21 Sep 2023 13:23:08 -0400 Received: by mail-io1-xd2c.google.com with SMTP id ca18e2360f4ac-79a7dc97aa2so49409339f.2 for ; Thu, 21 Sep 2023 10:23:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1695316986; x=1695921786; darn=gnu.org; h=content-transfer-encoding:in-reply-to:from:references:newsgroups:cc :to:content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=v9rvDJTj7jkKD2WnIXl+TMprEUQuM2dkTs2tT3lBB9Q=; b=A+fmNvws0MheuF7jmyUJrSvFonDUddjn1E1+JNOTctzYzoWVcaFUBPs2lb5ArUgoJV Ye0i7TANCbBt8rsmHvsuGiRXsc9emXF+meLEtpi3qJw5gb9MU7cK+u+kV9rL7jKb6yZY f3K2X3EGB1wIE8NNaTJgQiMVGOoOqhyvMLvgYhNNZKWPaahN31+7+pFnUCeaFMAt6Imt C93KF6OPpFNwGGtZcULSQGJdz/9f++snxGZvZEGCTQjLnS06rbMDKsnvOKHd+rOY9dQi iN3HcmVvu5A8ZYIQDiAv4w3SDlVMYDLslCmLYdTkynUKjdE5XZ+yh5astSvaTqqzEJn3 nt/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695316986; x=1695921786; h=content-transfer-encoding:in-reply-to:from:references:newsgroups:cc :to:content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v9rvDJTj7jkKD2WnIXl+TMprEUQuM2dkTs2tT3lBB9Q=; b=QTOHfrGJOB8FwaX2Fp52ezEjVnDVujujgboxH+7nNODy8g6vgMPnGVU8G+1GFH9KfN 8l9j7aN5K+3m7IpXwXD0n21WCcHv0VZ0E3R5k4VehGXyx/xDq6a5KemGp1qG1tKcjgrs DYdApLRKKSOdIaJ+u440U6Vdjya4bkWzDIMx9EbUC0ALMl9bSH6J0bQ42idy+/owgF3S mb+WkAy9AGBh7hHzmG1ZfD/ev8nmrhfhjASeMtQejFYzXn3XRGtHT/AQ9pManu/7mh/i gRTSd+NkUMdasxiDF94BwqCdSIhZp86mqZRaRsD3fHTdGXACD+66gKZlNzK1dgAeHZiG GVfA== X-Gm-Message-State: AOJu0Yy+7Qm3vH7iNp75uAgbLEaHcBboLCWoidoK5PElYZxarKiROJ/g mcp7dTSIkI9S3J79c9EGCDI= X-Google-Smtp-Source: AGHT+IGuirEZj0D+xUs93E0MzJ2f8bTOLuxR4oA+F9Y4hxbx2x6XfMQTbIPlBm+1Ar+71SvF7GXgAg== X-Received: by 2002:a5d:9c01:0:b0:791:385c:f8b0 with SMTP id 1-20020a5d9c01000000b00791385cf8b0mr6408887ioe.3.1695316986100; Thu, 21 Sep 2023 10:23:06 -0700 (PDT) Received: from [10.0.2.153] (c-174-51-218-141.hsd1.co.comcast.net. [174.51.218.141]) by smtp.gmail.com with ESMTPSA id q19-20020a5d87d3000000b0079216d6f219sm477956ios.14.2023.09.21.10.23.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 21 Sep 2023 10:23:05 -0700 (PDT) Message-ID: <20f3daf8-19ee-87b6-c403-6ff0cac130f2@gmail.com> Date: Thu, 21 Sep 2023 11:23:04 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: Guix Survey (follow up on "How can we decrease the cognitive overhead for contributors?") Content-Language: en-US To: Wilko Meyer , Simon Tournier Cc: guix-devel@gnu.org Newsgroups: gmane.comp.gnu.guix.devel References: <87y1h6qjxl.fsf@wmeyer.eu> <875y459ocy.fsf@gmail.com> <87il847azc.fsf@wmeyer.eu> From: Katherine Cox-Buday In-Reply-To: <87il847azc.fsf@wmeyer.eu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=2607:f8b0:4864:20::d2c; envelope-from=cox.katherine.e@gmail.com; helo=mail-io1-xd2c.google.com X-Spam_score_int: -35 X-Spam_score: -3.6 X-Spam_bar: --- X-Spam_report: (-3.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-1.473, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx1.migadu.com X-Migadu-Spam-Score: -9.61 X-Spam-Score: -9.61 X-Migadu-Queue-Id: 5EA8B4D2DD X-TUID: pfVjgQcUKry2 This is awesome, thanks Wilko! I've been talking with my wife who is the field of psychology. As part of her degree, and as part of her ongoing education, she's studied how to design studies/surveys so that they're not biased and don't produce contaminated data. She's not an expert at this, but she knows more about it than me :) The things we could come up with which we thought were important to consider are: - You must first define your goals for the survey. Is it meant to see who is using Guix? Who is contributing? How they find the contribution process? How they find using Guix? There are many dimensions, and we may need to create more than one survey. - The medium of the survey is very important. E.g. some people won't reply to a survey served something that uses JavaScript. Some people may not be able to reply to a survey unless accessibility concerns are met. Some people won't overcome the barrier of having to log in to respond. - Where solicitations to complete the survey are broadcast is very important. E.g. if we only send it to guix-dev, this skews the responses to questions like "where do you talk about Guix". - When the solicitations are made is very important. Some religions do not allow use of electronics on certain days, or times of the year. Some people are away on holiday during parts of the year. Some people are trying to meet deadlines during fiscal quarters. It may be impossible to accommodate everyone, but giving a little consideration to the issue and a sufficient window of time may cover most cases. - When soliciting responses to the survey, it's very important to set expectations about the survey in the solicitation. It is important to briefly describe what the survey is like and how long the survey will take. Without this, some people will have uncertainty about what they're committing to and not even try. - The survey should endeavor to remain on the shorter end; many will not complete longer surveys. - Does the survey need translation to eliminate language barriers? - The survey should use a uniform measurement system throughout. Don't use scales with different magnitudes in different questions, and don't suddenly invert whether higher is better or worse. - As you've already mentioned, free-form questions are very difficult to quantify, and I think we should use them with caution. Communities rooted in philosophical values, as Guix is, have impassioned people and resolving a large number of free-form responses to a quantitative statement may be difficult. - Questions which are intended to solicit a agree/disagree should be phrased as "I" questions, e.g.: On a scale of 1-5, how much do you agree with the phrase "I like carrots"? - Questions should not be leading, and be biased towards the positive. E.g., with the carrots example, don't do this: Carrots are disgusting. How much do you agree with this? and don't do this: On a scale of 1-5, how much do you agree with the phrase "I think carrots are disgusting!" - Up front, it may be difficult to identify all the root-causes of something the project wants to know about. Instead of trying to infer these, ask the questions directly. E.g. instead of questions about liking crunchy vegetables, orange vegetables, and root vegetables, ask whether they like carrots. However, if you think you have some idea of the root-causes, you can ask those as well to see if the correlation you think exists does. - You may want to ensure the survey has "marker questions" which clearly categorize your responder for you to make it easier to make the statements you'd like to make. E.g. if you're interested in analyzing what vegetarians vs omnivores think of carrots, ask that so you don't have to try and infer it later. - We were unable to resolve the question of astroturfing wherein one malicious party responds many times to skew the data. This might be difficult to address without relying on a vendor who has solved this concern somehow, but requires logins, JavaScript, or something else people won't use. And finally, I'd like to suggest: I think since this is the result of a discussion about how to lower the cognitive overhead of contributing, the goal of this initial survey should be: 1. To quantify how easy it is to contribute to Guix. 2. To quantify how easy it is to maintain Guix. 3. To correlate (1) and (2) with people's opinion of using email for contributions. 4. To correlate (1) and (2) with people's opinion of using a forge for contributions. 5. To correlate (1) and (2) with people's opinion on only improving tooling. 6. To be able to do trend-analysis year-over-year on these issues. I would suggest adding these questions to a survey exploring the contribution process: On a scale of 1-5, 1 being "Strongly Disagree" and 5 being "Strongly Agree", how much do you agree with the statements: "I think it is easy to contribute to Guix." "I think contributions to Guix will be reviewed in a timely manner." "I think email is the best way to manage introducing code to Guix." "I think a web-forge is the best way to manage introducing code to Guix." "I think working on tools made specifically for Guix is the best way to improve the contribution process." I am very interested in the usage patterns of Guix, and I firmly believe some survey should explore this. I'm not sure if we should combine the two; does it make it too long of a survey?