From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id +I7XCs1GJGWMQQAAG6o9tA:P1 (envelope-from ) for ; Mon, 09 Oct 2023 20:30:37 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id +I7XCs1GJGWMQQAAG6o9tA (envelope-from ) for ; Mon, 09 Oct 2023 20:30:37 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D79FD69C98 for ; Mon, 9 Oct 2023 20:30:36 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FHrfKbo6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1696876237; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=6v3332Rg3O6c/Uq414bu5mHjtdCdXCVTdJOUFSFbxgc=; b=JtIjnI7TfvdzUMwgznafJbf4Hoc7zh+8ZkFhDWEIaPdKfTpEJkIt3c38VIzNNiyuFtoQlQ sWKKvqxydrYbQ5UCOUWX0QCrjCkLDaMzN9I/3sFtFzDr/AwrsimXbLRbkG3dWXJ1HHLUX9 AuV3I8pfiLk69554JbpUvhCmPvrXQmDrsPJS2tR0BSRfS/pV7ekj38pbn32fKYsWs73pom +sxsYzoODnXdgHOzzCbhM1nrZScs1sL68KENiDmWMVT8Oqohaa3VxB8F1O6bavstQwwlmm J3ErynWtRWqJnsE/jDsqToNdXkHEQCGtCEkZmXVSnhZVb6hwKt9/39jEhRR55g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1696876237; a=rsa-sha256; cv=none; b=OXI+sciUfDmdceNxHu4voD/KixCs38eMyGZTsabSHHq7JdbBDpNp7olXoftdnI9SYjXQKH vGLP/HypOFNBahHO/xyVTvkuwOJ3yEoUUK+euD2oMk7n9+lMw+GD/dwnnxi23jdg7RqgHj SJ8tcSHO0DiRtfUiZKdGnUi7UDa7Mp7r2jjOtrMj1U6WpT6J3OH7cyb+WfsFkmiAsmELZx oOyZ3idTMjoWOZ2U5pHKbbd3TdQnx9QvqMEg5iyCGKqEaWqHOGNuUGhvxsefZkMkaNvcat 5xXHrqVadp8IqX9/EQWR4ADB7keuOxsDtLNJziFwG8ZFfhVIIeRNDlRMW/iJyw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FHrfKbo6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qpv0z-0004Js-RO; Mon, 09 Oct 2023 14:30:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qpv0y-0004J4-P5 for guix-devel@gnu.org; Mon, 09 Oct 2023 14:30:00 -0400 Received: from mail-io1-xd2a.google.com ([2607:f8b0:4864:20::d2a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qpv0w-0006pC-NA for guix-devel@gnu.org; Mon, 09 Oct 2023 14:30:00 -0400 Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-79faba5fe12so179949339f.3 for ; Mon, 09 Oct 2023 11:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696876197; x=1697480997; darn=gnu.org; h=content-transfer-encoding:in-reply-to:from:references:newsgroups:cc :to:content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=6v3332Rg3O6c/Uq414bu5mHjtdCdXCVTdJOUFSFbxgc=; b=FHrfKbo6dW8BM4mH62Y4QyfbWAkG8zR57gsDzIhOZVuLnl4odDu5QjglAAqK0rQLeF XolOzJxwBeYx4XSMTUy0iGKEDQMAvbolu9WSJuep48peHqBRPf+z0+aPY899Q5XcAyY9 toSi8/VR+EHPt5c9D0EHa8w/wRixE357L/OU7bm5JSdnD5S7XHcLIiMJy6IUz1k0KJ8s soiDdPfO1EIrGK+9ZCf41cLnYzksJCIyZKVJlV2XU3sdu/efE1eit/nnQotDMRBstExs KfiTOTUKN9mhHy/pzJpLT7IhmuooWn24TROYmBRhQdXGj4N1BkC2VJ1j4ODX+Frazw7r XzGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696876197; x=1697480997; h=content-transfer-encoding:in-reply-to:from:references:newsgroups:cc :to:content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6v3332Rg3O6c/Uq414bu5mHjtdCdXCVTdJOUFSFbxgc=; b=wWQd4touwCAdsFWNN+ILLExaoSPmf3lNV4kb+98OGwNERKHhsTO3xQNRP+3u4ObF5e 3xhEXY+DF7J7gKVl8IZpUzaIYugw7KwNblAB3wiXGQ+DDbhvCReMAIetw9CvcmtbMpYh xKaonrZbQ21W28Gbeh0bbkL35YTgEUUZTKTe5De8sAXSpPmbk5ZHd0ofljNaqWPslJ03 vXmDtU8zBwVNrXCRvwkKzMH9URedcm6Z2JZX8vrXh/kKqitM0+Ibi0P052xYshKCFYY9 6lTuk2Fp9Bx97MlfNutrubdAZcdGqii3PMNpPoeOSkdmxvwncUawNQ55EdkB5+TBpDwa lhsg== X-Gm-Message-State: AOJu0YyUZ0v5m/FnTSGYMNvfMutuilzWeabJAF4o+V982s2ZjAS2SdPR PwzETOd+h5Qv00bC9FtkEFB9qiQvkoY= X-Google-Smtp-Source: AGHT+IH9XdB+tF1UQ/y2ZgyJciuQj+Eeva/cZkav6yqvw60qUyRoXtQnwaroETP7uWq88T7wb9dtOQ== X-Received: by 2002:a05:6602:2191:b0:795:8805:1378 with SMTP id b17-20020a056602219100b0079588051378mr19471974iob.9.1696876197027; Mon, 09 Oct 2023 11:29:57 -0700 (PDT) Received: from [10.0.2.153] (c-174-51-218-141.hsd1.co.comcast.net. [174.51.218.141]) by smtp.gmail.com with ESMTPSA id m7-20020a02cdc7000000b0043a21abd491sm2235121jap.120.2023.10.09.11.29.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 09 Oct 2023 11:29:56 -0700 (PDT) Message-ID: <2ae35de1-0e77-7652-43df-b8ff3622eb00@gmail.com> Date: Mon, 9 Oct 2023 12:29:54 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: Guix Survey (follow up on "How can we decrease the cognitive overhead for contributors?") Content-Language: en-US To: Wilko Meyer Cc: guix-devel@gnu.org Newsgroups: gmane.comp.gnu.guix.devel References: <87y1h6qjxl.fsf@wmeyer.eu> <875y459ocy.fsf@gmail.com> <87il847azc.fsf@wmeyer.eu> <20f3daf8-19ee-87b6-c403-6ff0cac130f2@gmail.com> <87h6n9mcan.fsf@wmeyer.eu> From: Katherine Cox-Buday In-Reply-To: <87h6n9mcan.fsf@wmeyer.eu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=2607:f8b0:4864:20::d2a; envelope-from=cox.katherine.e@gmail.com; helo=mail-io1-xd2a.google.com X-Spam_score_int: -53 X-Spam_score: -5.4 X-Spam_bar: ----- X-Spam_report: (-5.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-3.339, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx0.migadu.com X-Migadu-Spam-Score: -6.53 X-Spam-Score: -6.53 X-Migadu-Queue-Id: D79FD69C98 X-TUID: UIQTS11X0WT3 On 10/2/23 5:24 AM, Wilko Meyer wrote: >> - Where solicitations to complete the survey are broadcast is very >> important. E.g. if we only send it to guix-dev, this skews the >> responses to questions like "where do you talk about Guix". > > Definitely, I'm not entirely sure on how to solve this; publishing > surveys on as many channels that seem fitting could maybe mitigate this? > Then again the selection of communication channels is highly subjective > as well. I don't think we have to be very selective about where to broadcast the survey. I think the answer is: anywhere Guix people hang out or anyone feels it might be useful to do so. I think the only danger here is missing some popular hangout spots due to ignorance of their existence. We should enumerate them to ensure we catch them all. >> - When soliciting responses to the survey, it's very important to set >> expectations about the survey in the solicitation. It is important >> to briefly describe what the survey is like and how long the survey >> will take. Without this, some people will have uncertainty about >> what they're committing to and not even try. >> >> - The survey should endeavor to remain on the shorter end; many will >> not complete longer surveys. > > This is another good reason to start with a narrow focus on questions > regarding contributions instead of a general survey. > >> - Does the survey need translation to eliminate language barriers? > > Most FLOSS surveys I've looked at were english only; which comes with a > certain language bias. Realistically I'd say that, given a survey may > contain free form questions, translation also seems to be a resource > issue when it comes to analyzing the results. Does the GNU project have a "general translation" team? Maybe some of our Guix community members who speak multiple languages would be willing to translate the survey into their primary language? >> - The survey should use a uniform measurement system throughout. Don't >> use scales with different magnitudes in different questions, and >> don't suddenly invert whether higher is better or worse. > > Good point, this also means that questions should be asked in a way, > that they can be answered using the same metrics/scale? I think that's the idea and should be the rule for which there are exceptions. >> - As you've already mentioned, free-form questions are very difficult >> to quantify, and I think we should use them with caution. >> Communities rooted in philosophical values, as Guix is, have >> impassioned people and resolving a large number of free-form >> responses to a quantitative statement may be difficult. > > My approach to free form questions is to, on one hand try to quantify > trends (things that are mentioned often, key topics that are mentioned), > on the other try to derrive actionable items/issues from them that can > be worked on. Quantifiying qualitative responses is cumbersome, and as > you've also pointed out, quite difficult; but identifying trends/key > topics and maybe actionable items/issues from those could work. WDYT? My opinion is that we should not do free form questions for this first time. We're new at this, we have enough topics to cover, and the topics we are covering seem to cause a lot of discussion (that's good) which could lead to a lot of text to read through. >> - Up front, it may be difficult to identify all the root-causes of >> something the project wants to know about. Instead of trying to >> infer these, ask the questions directly. E.g. instead of questions >> about liking crunchy vegetables, orange vegetables, and root >> vegetables, ask whether they like carrots. >> >> However, if you think you have some idea of the root-causes, you can >> ask those as well to see if the correlation you think exists does. > > If we've a first draft of a prospective, narrowed-down on contributions, > survey, the questions should probably be benchmarked against these > criteria. I revisted my loose collection of survey questions I posted > earlier on here and realized that I probably have to rephrase a vast > majority of those, to be consistent with this. > >> - You may want to ensure the survey has "marker questions" which >> clearly categorize your responder for you to make it easier to make >> the statements you'd like to make. E.g. if you're interested in >> analyzing what vegetarians vs omnivores think of carrots, ask that >> so you don't have to try and infer it later. > > I'll revisit the original thread on how to decrease cognitive overhead > for contributors to see what good markers could be. With a grain of > salt, I think we should figure out ways to identify participants that: > > - were contributors to guix before but stopped contributing Maybe: (1) Have you made contributions to Guix in the past? This is our marker question. combined with: (2) How many contributions to Guix per month would you estimate you've made in the past year? (identify ranges we're interested in) This is a dimension that would be useful to have for analyzing other responses. We can infer that > - want to contribute but experienced blockers that stopped them from > participating (3) On a scale of 1-5, how much do you agree with the statement, "I would like to contribute to Guix." combined with (4) On a scale of 1-5, how much do you agree with the statement, "I think it is easy to contribute to Guix." We can then infer that people with higher scores to (3), lower scores to (4), and low contribution ranges from (2) are experiencing blockers that stop them from participating. > as well as a way to map out e.g. the frequency of contributions? (covered above) >> And finally, I'd like to suggest: >> >> I think since this is the result of a discussion about how to lower >> the cognitive overhead of contributing, the goal of this initial >> survey should be: >> >> 1. To quantify how easy it is to contribute to Guix. >> 2. To quantify how easy it is to maintain Guix. >> 3. To correlate (1) and (2) with people's opinion of using email for >> contributions. >> 4. To correlate (1) and (2) with people's opinion of using a forge for >> contributions. >> 5. To correlate (1) and (2) with people's opinion on only improving tooling. >> 6. To be able to do trend-analysis year-over-year on these issues. >> >> I would suggest adding these questions to a survey exploring the >> contribution process: >> >> On a scale of 1-5, 1 being "Strongly Disagree" and 5 being >> "Strongly Agree", how much do you agree with the statements: >> >> "I think it is easy to contribute to Guix." >> >> "I think contributions to Guix will be reviewed in a timely >> manner." >> >> "I think email is the best way to manage introducing code to >> Guix." >> >> "I think a web-forge is the best way to manage introducing code to >> Guix." >> >> "I think working on tools made specifically for Guix is the best >> way to improve the contribution process." > > These questions are excellent to address large parts of the issues being > discussed in the original thread on this list. > >> I am very interested in the usage patterns of Guix, and I firmly >> believe some survey should explore this. >> I'm not sure if we should combine the two; does it make it too long of >> a survey? > > I'd say, that it could be beneficial to ask for usage as long as it > helps to map out the background of guixes users. I wouldn't go too much > into detail, but this subset of questions on general usage I shared > before (and maybe a question on familiarity with Guile/Scheme) would > still provide value: > >>> - Why do you use Guix? (freeform) >>> - Where/on what platform do you use Guix? (Guix System, on top of other >>> distributions etc.) >>> - How many years have you been using Guix? >>> - In what context do you use Guix? (academic, work, private etc.) >>> - What do you use Guix for? (packaging, systemconf, reproducible >>> research and so on) This might be too broad a question since Guix is an operating system. >>> - Have you ever considered to stop using Guix, if so why? (freeform) >>> - Which features keep you using Guix? (should be a list with optional >>> freeform) This might also be too broad a question since people's motivations are likely to be varied. >>> - To extend guix packages/work on new packages, you... >>> ...upstream to guix proper >>> ...maintain a fork of guix proper >>> ...maintain your own guix channel >>> ...provide a guix.scm for the respective projects > > to assess the questionees background better/be able to give more > context. WDYT? As discussed above, my opinion is that we should drop the free-form questions. Other than that and my comments, I'd be OK with including these style of questions in this survey. But these are my opinions. I'm open to discussion! Is this rough list of questions our first pass at what the survey should contain? Have you done any more research on what other communities are doing? What are next steps?