[-- Attachment #1: Type: text/plain, Size: 1592 bytes --] Hello! There is two ways to ship security fixes to packages: 1. Update to a patched version if upstream provides one 2. Apply or backport individual patches to fix the issues in the shipped version Grafts are most reliable for 2. but there's cases where using 2. is lots of work and we can't afford that right now. An example is ImageMagick where not all security issues get a CVE so essentially the only way of getting security fixes is to fetch master or get the latest release. There's also some types of packages where we are not sure whether we can use grafting or not, such as Python ones. For these reasons, I would like to propose a new branch called security-updates that would be based on master where we queue security fixes that introduce any arbitrary number of rebuilds without using grafts. We would merge the security-updates branch as soon as there is complete substitute availability for the branch and it's future merged version within master. The downsides of this approach are that: 1. Substitutes availability does not mean we can ship the updates quickly because this might mean hundreds of megabytes if not gigabytes of new substitutes to fetch to actually get the update. 2. Users that don't use substitutes will suffer big rebuilds on each security update shipped through this branch. For these reasons, grafting should still be preferred when possible, but there are cases where it cannot be used for technical reasons or lack of resources reasons but we still must provide a fix quickly. What do you think? Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #1: Type: text/plain, Size: 2623 bytes --] Léo Le Bouter <lle-bout@zaclys.net> writes: > There is two ways to ship security fixes to packages: > > 1. Update to a patched version if upstream provides one > 2. Apply or backport individual patches to fix the issues in the > shipped version > > Grafts are most reliable for 2. but there's cases where using 2. is > lots of work and we can't afford that right now. An example is > ImageMagick where not all security issues get a CVE so essentially the > only way of getting security fixes is to fetch master or get the latest > release. > > There's also some types of packages where we are not sure whether we > can use grafting or not, such as Python ones. > > For these reasons, I would like to propose a new branch called > security-updates that would be based on master where we queue security > fixes that introduce any arbitrary number of rebuilds without using > grafts. > > We would merge the security-updates branch as soon as there is complete > substitute availability for the branch and it's future merged version > within master. > > The downsides of this approach are that: > > 1. Substitutes availability does not mean we can ship the updates > quickly because this might mean hundreds of megabytes if not gigabytes > of new substitutes to fetch to actually get the update. > 2. Users that don't use substitutes will suffer big rebuilds on each > security update shipped through this branch. > > For these reasons, grafting should still be preferred when possible, > but there are cases where it cannot be used for technical reasons or > lack of resources reasons but we still must provide a fix quickly. > > What do you think? Can you clarify what specific problem or problems you're proposing this security-updates branch to address? You mention applying and backporting patches is lots of work, and uncertainty around whether grafts work in particular situations. Personally I think staging and core-updates are quite a bit of work, and adding more complexity to the process involves more work in my mind. Additionally, this isn't going to provide more information about areas where grafts can't be used (if those exist). Now the software involved is getting better at rapidly building things for substitutes, personally I see a way forward through trying to measure and potentially increase the rate of change for outputs in general. Going faster also involves more work probably, but in terms of the process, that might just mean that updates to more packages can be merged to master directly, without sitting on a non-master branch. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 987 bytes --]
[-- Attachment #1: Type: text/plain, Size: 2122 bytes --] On Fri, 2021-03-26 at 22:13 +0000, Christopher Baines wrote: > Can you clarify what specific problem or problems you're proposing > this > security-updates branch to address? Substitute availability of security updates when they are released, without causing big rebuilds on master for users before the build farm had time to produce substitutes. > You mention applying and backporting patches is lots of work, and > uncertainty around whether grafts work in particular > situations. Also that some times backporting is just not possible because security fixes are not properly labeled upstream as security-relevant and manual review of each and every commit is just not viable. > Personally I think staging and core-updates are quite a bit of work, > and > adding more complexity to the process involves more work in my > mind. Additionally, this isn't going to provide more information > about > areas where grafts can't be used (if those exist). I understand. There's lot of uncertainty on how grafts work exactly for me and in what situations they work and what situations they do not. The only way I am certain some security fix is correctly applied in GNU Guix is when the vulnerable version of the package is not packaged at all anymore in GNU Guix. > Now the software involved is getting better at rapidly building > things > for substitutes, personally I see a way forward through trying to > measure and potentially increase the rate of change for outputs in > general. Going faster also involves more work probably, but in terms > of > the process, that might just mean that updates to more packages can > be > merged to master directly, without sitting on a non-master branch. I would like this, merging things to master directly when we feel it is the right thing would be what I would like to do. Even if it causes big world rebuilds, when we can't graft. There's another thing I saw that was ongoing but can't remember where, that 'guix pull' could hold off updating to newer revisions unless substitutes are available. Thanks for your input. Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #1: Type: text/plain, Size: 2836 bytes --] Léo Le Bouter <lle-bout@zaclys.net> writes: > On Fri, 2021-03-26 at 22:13 +0000, Christopher Baines wrote: >> Can you clarify what specific problem or problems you're proposing >> this >> security-updates branch to address? > > Substitute availability of security updates when they are released, > without causing big rebuilds on master for users before the build farm > had time to produce substitutes. Ok. >> You mention applying and backporting patches is lots of work, and >> uncertainty around whether grafts work in particular >> situations. > > Also that some times backporting is just not possible because security > fixes are not properly labeled upstream as security-relevant and manual > review of each and every commit is just not viable. > >> Personally I think staging and core-updates are quite a bit of work, >> and >> adding more complexity to the process involves more work in my >> mind. Additionally, this isn't going to provide more information >> about >> areas where grafts can't be used (if those exist). > > I understand. There's lot of uncertainty on how grafts work exactly for > me and in what situations they work and what situations they do not. > The only way I am certain some security fix is correctly applied in GNU > Guix is when the vulnerable version of the package is not packaged at > all anymore in GNU Guix. > >> Now the software involved is getting better at rapidly building >> things >> for substitutes, personally I see a way forward through trying to >> measure and potentially increase the rate of change for outputs in >> general. Going faster also involves more work probably, but in terms >> of >> the process, that might just mean that updates to more packages can >> be >> merged to master directly, without sitting on a non-master branch. > > I would like this, merging things to master directly when we feel it is > the right thing would be what I would like to do. Even if it causes big > world rebuilds, when we can't graft. > > There's another thing I saw that was ongoing but can't remember where, > that 'guix pull' could hold off updating to newer revisions unless > substitutes are available. I think approaching the substitute availability problem this way, and supporting users delaying updates if they want to is the way to go, at least to avoid process on the side of making changes. I'm hoping things will get better through a combination of these factors: - Measuring the "churn" in Guix, which will hopefully allow for intentionally increaseing this - Keeping packages more up to date generally - Allowing users to choose whether they want the latest packages, or want to avoid building things (this means important security updates that cause rebuilds can be merged to master) [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 987 bytes --]
[-- Attachment #1: Type: text/plain, Size: 3823 bytes --] Hi Léo, On Fri, 26 Mar 2021 at 21:10, Léo Le Bouter <lle-bout@zaclys.net> wrote: > For these reasons, I would like to propose a new branch called > security-updates that would be based on master where we queue security > fixes that introduce any arbitrary number of rebuilds without using > grafts. > > We would merge the security-updates branch as soon as there is complete > substitute availability for the branch and it's future merged version > within master. Basically, what you are proposing is more or less staging. The issue is that the cycles are sometimes longer than expected. The cycle is defined as 6 weeks for staging. Is it really 6 weeks? Well, It could be shorter if there is more man power *and* if the build farm is working better. About the man power, it is already a good job. Let plot roughly the number of days between 2 merges between staging and master. --8<---------------cut here---------------start------------->8--- $ git log --format="%cd|%s" --date=format:"%Y-%m-%d" --after=2019-01-01 \ | grep Merge | grep staging | grep master \ | cut -d'|' -f1 \ > /tmp/staging.txt --8<---------------cut here---------------end--------------->8--- then this tiny R script: --8<---------------cut here---------------start------------->8--- $ cat /tmp/dates.R library("ggplot2") raw <- read.table("/tmp/staging.txt") dat <- raw$V1 dates <- data.frame(start=strptime(dat[1:length(dat)-1], format="%Y-%m-%d"), end=strptime(dat[2:length(dat)], format="%Y-%m-%d")) dates$diff <- difftime(dates$start, dates$end, units="days") p <- ggplot(dates, aes(x=start, y=diff)) + geom_point(color='red') + geom_smooth() ggsave("/tmp/number-of-days-staging-merges.pdf") --8<---------------cut here---------------end--------------->8--- shows the attached graph. Well, it is variable and the variance and standard deviation should be considered. The mean is less then 8 days. Obviously, I can have done a mistake and this should be taken with grain of salt. My point is: we should root the practise on what we are already doing and see what are the incremental changes to improve. For example, staging is not merged since a couple of months. Why? Because none of us is taking the time to make it. Hard to be close to the oven and in the same to the mill. :-) About the build farm, it is doing better and better. Mathieu introduced some plots [1] and again I think we should explore a bit the data to see what is the average time to build staging, how many packages are built on average, etc. Moreover, priorities have recently been introduced. I do know if we have enough time elapse to see their impacts. They are per branch, IIRC, and maybe we could imagine a priority ’properties’ applied for a package because it is a security update. IIUC, it is not implemented yet. And as I said elsewhere, “to me, security is important. But it's no less important than everything *else* that is also important!“, so personally I am not convinced that security updates deserve a special treatment compared to a regular update. That’s my opinion. :-) That’s said, from my point of view, staging or security-updates are only names but somehow they cover the same idea: update a package with many dependants and provide the substitutes as soon as it is available. The key for a better scaling is, IMHO, to tweak what is going to master, staging and core-updates; as I wrote in [2]. But the real hard part is to collectively make this tough work of merging branches on the long run. Cheers, simon 1: <https://ci.guix.gnu.org/metrics> 2: <https://yhetil.org/guix/86mtv29erk.fsf@gmail.com> [-- Attachment #2: graph.pdf --] [-- Type: application/pdf, Size: 11423 bytes --]
[-- Attachment #1: Type: text/plain, Size: 613 bytes --] Thanks for your feedback. On Sat, 2021-03-27 at 13:29 +0100, zimoun wrote: > And as I said elsewhere, “to me, security is important. But it's > no less important than everything *else* that is also important!“, so > personally I am not convinced that security updates deserve a special > treatment compared to a regular update. That’s my opinion. :-) You can't think this, security updates have prioritized channel of distribution in every other GNU/Linux distribution, we in GNU Guix created grafts for it, it's not possible to not ship security updates promptly, it puts all users at risk. [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
On Sat, 27 Mar 2021 at 13:42, Léo Le Bouter <lle-bout@zaclys.net> wrote: > On Sat, 2021-03-27 at 13:29 +0100, zimoun wrote: >> And as I said elsewhere, “to me, security is important. But it's >> no less important than everything *else* that is also important!“, so >> personally I am not convinced that security updates deserve a special >> treatment compared to a regular update. That’s my opinion. :-) > > You can't think this, security updates have prioritized channel of > distribution in every other GNU/Linux distribution, we in GNU Guix > created grafts for it, it's not possible to not ship security updates > promptly, it puts all users at risk. Oh, I am a big boy and I can think whatever I want! :-) Kidding aside. First, what does it mean «risk»? How do you evaluate it? Is it a relative evaluation or an absolute one? Second, I am not arguing that security is not important. I am saying that security is important, as important as everything else that is also important. What does it mean «important»? How do you evaluate it? Is it a relative evaluation or an absolute one? Third, I am aligned with Leo’s words [1]. And probably with yours too. :-) To me, a better security is not implied by special treatments for security fixes but instead a better treatment for the updates in general. You are proposing a new branch and Chris and I are saying that this branch already exists and is staging. The real question is to know how staging currently behaves: how many time between 2 merges? how many time to rebuild? how many packages are rebuilt between 2 merges? etc. Is it enough? If not, what could be done to improve? etc. 1: <https://yhetil.org/guix/YFEDt%2FPUd2ZeC6%2FF@jasmine.lan/> Cheers, simon
[-- Attachment #1: Type: text/plain, Size: 1837 bytes --] On Sat, 2021-03-27 at 14:56 +0100, zimoun wrote: > Oh, I am a big boy and I can think whatever I want! :-) > > Kidding aside. ... > > First, what does it mean «risk»? How do you evaluate it? Is it a > relative evaluation or an absolute one? Most if not all users do not want their machines to be compromised or any side-effects of that. > Second, I am not arguing that security is not important. I am saying > that security is important, as important as everything else that is > also > important. What does it mean «important»? How do you evaluate > it? Is it a > relative evaluation or an absolute one? Having security-updates branch or any other mechanism to ship security updates promptly does not mean that the rest is not important. > Third, I am aligned with Leo’s words [1]. And probably with yours > too. :-) To me, a better security is not implied by special > treatments for security fixes but instead a better treatment for the > updates in general. Security updates *need* special treatment. We already specially treat them with grafts because it's an absolute necessity. We already have private disclosure mailing lists in GNU Guix because security updates need special treatment. > > You are proposing a new branch and Chris and I are saying that this > branch already exists and is staging. The real question is to know > how > staging currently behaves: how many time between 2 merges? how many > time to rebuild? how many packages are rebuilt between 2 > merges? etc. > Is it enough? If not, what could be done to improve? etc. The question whether this is solved by a branch or by making pushing to master directly big rebuilds more viable, that I do not know, but you cannot put forward the arguments you've made, they do not work. Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
On Sat, 27 Mar 2021 at 15:14, Léo Le Bouter <lle-bout@zaclys.net> wrote:
> but you
> cannot put forward the arguments you've made, they do not work.
Ahah, I am happy to know it. I hope it is because a “miscommunication»
and not because you do not carefully read or because maybe you only see
through the tiny lens of known security vulnerabilities. From my
opinion, your point of view to tackle the issue is wrong. That’s said.
Best regards,
simon
[-- Attachment #1: Type: text/plain, Size: 929 bytes --] On Tue, 2021-03-30 at 13:48 +0200, zimoun wrote: > Ahah, I am happy to know it. I hope it is because a > “miscommunication» > and not because you do not carefully read or because maybe you only > see > through the tiny lens of known security vulnerabilities. From my > opinion, your point of view to tackle the issue is wrong. That’s > said. I feel harassed by your comments because you obsessed on this zstd issue and try to make it the cause of some other problems you saw without any evidence. I don't think I look through the "tiny lens of known security vulnerabilities", every distro has prioritization for security updates, I think it is the right thing to do, that's how I work with it and I think it is right. If you don't agree then OK but I wont stop thinking it's the right way to do things and that not prioritizing security issues does not work. > > Best regards, > simon Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
Léo,
Léo Le Bouter <lle-bout@zaclys.net> skribis:
> I feel harassed by your comments because you obsessed on this zstd
> issue and try to make it the cause of some other problems you saw
> without any evidence.
It’s unacceptable to call someone “obsessed” just because you disagree
and calling Simon’s comments “harassment” is equally inappropriate.
We’re all passionate about the project, and each one of us looks at it
from a different angle.
You’re new to the project. I think we’re all glad you joined; that has
already boosted security work and the POWER9 port. But we also have
expectations: that you follow written rules (the code of conduct, the
“Commit Access” section of the manual), and the unwritten rule that, as
a newcomer, you would humbly listen to suggestions made by more
experienced contributors.
The unresolved issues Simon points out are not just technical problems;
they’re a hindrance to our cooperation and to our well-being as a group.
It’s completely okay to make mistakes, but it’s not okay to break the
community rules and to dismiss the opinion of others as unimportant.
Please adjust accordingly.
Thanks,
Ludo’.
[-- Attachment #1: Type: text/plain, Size: 2644 bytes --] Hello Ludo, On Wed, 2021-03-31 at 23:29 +0200, Ludovic Courtès wrote: > It’s unacceptable to call someone “obsessed” just because you > disagree > and calling Simon’s comments “harassment” is equally inappropriate. I really do feel harassed by their comments, it's not just because I disagree, it's that I feel they have been following me around in several of my contributions repeating the same issues, I have heard their criticism and I do disagree but I don't feel like bringing it up over and over and over following me around is a great thing to do at all! I did not try to go and yell all around after for example 'guix pull' was broken after some changes to substitutes handling that were pushed for example. I just trust people to do the best. I know people want the best for GNU Guix. I also want the best for GNU Guix. I am not trying to somehow deliberately break GNU Guix. > We’re all passionate about the project, and each one of us looks at > it > from a different angle. > > You’re new to the project. I think we’re all glad you joined; that > has > already boosted security work and the POWER9 port. But we also have > expectations: that you follow written rules (the code of conduct, the > “Commit Access” section of the manual), and the unwritten rule that, > as > a newcomer, you would humbly listen to suggestions made by more > experienced contributors. I try to listen, and I think I have listened to criticism from many different people. I think we've reached a point where I could not listen anymore from Zimoun because I felt we were going in circles. There's different opinions from different places it seems, where some people agree with my way of doing and some others do not, and I felt like I had more approval from other people and that while Zimoun's criticism has been heard I have not acted upon all of their criticism. Is it possible to do that? > The unresolved issues Simon points out are not just technical > problems; > they’re a hindrance to our cooperation and to our well-being as a > group. > It’s completely okay to make mistakes, but it’s not okay to break the > community rules and to dismiss the opinion of others as unimportant. I do not dismiss other's opinion as not important, I disagree, what do you want me to do? We have different way of doing things, if I do them I do them my way, if they do them they do it their way, what else can we do here? > > Please adjust accordingly. I don't feel like that's entirely fair but I always strive to self- improve. > Thanks, > Ludo’. Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
Hi Léo, > Hello Ludo, > > On Wed, 2021-03-31 at 23:29 +0200, Ludovic Courtès wrote: >> It’s unacceptable to call someone “obsessed” just because you >> disagree >> and calling Simon’s comments “harassment” is equally inappropriate. > > I really do feel harassed by their comments, it's not just because I > disagree, it's that I feel they have been following me around in > several of my contributions repeating the same issues, I have heard > their criticism and I do disagree but I don't feel like bringing it up > over and over and over following me around is a great thing to do at > all! I don’t doubt it feels this way for you, but I seriously doubt Simon follows you around and responds to your emails to cause a stink. Since a couple of months Simon has been doing invaluable work to tie up loose ends in bug reports and mailing list discussions; thanks to this tireless work many stalled projects have been able to move forward. Applying the same strategy to finding consensus and establishing the lack of consensus in fresh discussions is very valuable, in my opinion. You started quite a few discussions with lots of recommendations and some controversial points; I think it is obvious that this leads to a discussion of these points, including the unavoidable discussion of misunderstandings (which can be pretty frustrating for all involved). For what it’s worth, I agree with the points made by Leo Famulari, Ludovic, and Simon. I don’t think we should have a security-updates branch, because the role of that branch is effectively taken by staging. >> We’re all passionate about the project, and each one of us looks at >> it >> from a different angle. >> >> You’re new to the project. I think we’re all glad you joined; that >> has >> already boosted security work and the POWER9 port. But we also have >> expectations: that you follow written rules (the code of conduct, the >> “Commit Access” section of the manual), and the unwritten rule that, >> as >> a newcomer, you would humbly listen to suggestions made by more >> experienced contributors. > > I try to listen, and I think I have listened to criticism from many > different people. I think we've reached a point where I could not > listen anymore from Zimoun because I felt we were going in circles. That’s fine. We have no deadlines, so stepping back from what feels like a heated discussion for a while and revisiting the points later comes at very little cost. Obviously, you don’t *have* to accept other people’s criticism. But we collectively aim to act in a collegial fashion, finding consensus before forging ahead with changes to processes. I have not been convinced by your arguments and your appeals in this discussion. I can’t speak for others, but I would consider it very unwise to ignore the lack of consensus and just start a new branch when that isn’t what the collective of long-term contributors agrees to do. If that’s not what you’re planning to do then my comments carry no weight. I do want to stress, though, that hurry is usually misplaced in the decision-making process of this project. There are only few exceptions to this and none warrant precipitating a falling-out. -- Ricardo
[-- Attachment #1: Type: text/plain, Size: 2264 bytes --] On Thu, 2021-04-01 at 16:58 +0200, Ricardo Wurmus wrote: > Hi Léo, > [...] > That’s fine. We have no deadlines, so stepping back from what feels > like a heated discussion for a while and revisiting the points later > comes at very little cost. > > Obviously, you don’t *have* to accept other people’s criticism. But > we > collectively aim to act in a collegial fashion, finding consensus > before > forging ahead with changes to processes. I have not been convinced > by > your arguments and your appeals in this discussion. I can’t speak > for > others, but I would consider it very unwise to ignore the lack of > consensus and just start a new branch when that isn’t what the > collective of long-term contributors agrees to do. > > If that’s not what you’re planning to do then my comments carry no > weight. I do want to stress, though, that hurry is usually misplaced > in > the decision-making process of this project. There are only few > exceptions to this and none warrant precipitating a falling-out. > I never planned to go on and create that security-updates branch, it is clear that there's no consensus for that, also it would require other tools we don't have. However what I disagree with Simon is that they think security updates should not be prioritized, which I don't agree with very strongly. They have repeated that many times in various ways and it annoys me that these arguments come up again and again in response to my messages or contributions even though Simon knows I disagree. There's no other problem to me. There's a thing about my personality and I am sorry to be that way, it's that I am ready to question myself but at a point when the demand to question becomes confrontational way rather than in a friendly way I stop being able to. I feel like I have taken into account many criticism and changed my workflow due to that. I am glad people help me improve the contributions I make to GNU Guix. This thread is a proposal, like all others were, and on some comments I feel attacked personally for even mentionning things that are only proposals to me (I don't even think they're good solutions in the absolute sense, I just propose them). Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #1: Type: text/plain, Size: 735 bytes --] Sorry for duplicated email, On Thu, 2021-04-01 at 16:58 +0200, Ricardo Wurmus wrote: > I don’t think we should have a security-updates > branch, because the role of that branch is effectively taken by > staging. I don't think that's the case because staging is documented for things that do not make too many rebuilds (in which case they'd go to core- updates), and some times security updates do cause rebuilds in the 1800+ and they could not go through staging. The proposed security- updates branch would not have that limitation. We could of course also revisit the policy for staging and prioritize security updates through that branch while also committing to actually merging that branch on schedule. Léo [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --]