From mboxrd@z Thu Jan 1 00:00:00 1970 From: zimoun Subject: Guix mirror: sourceware discussion report Date: Fri, 13 Dec 2019 18:46:54 +0100 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:58783) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ifp1w-00063d-JZ for guix-devel@gnu.org; Fri, 13 Dec 2019 12:47:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ifp1u-000419-Jr for guix-devel@gnu.org; Fri, 13 Dec 2019 12:47:08 -0500 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]:46279) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ifp1u-0003zZ-8l for guix-devel@gnu.org; Fri, 13 Dec 2019 12:47:06 -0500 Received: by mail-qk1-x732.google.com with SMTP id r14so215857qke.13 for ; Fri, 13 Dec 2019 09:47:06 -0800 (PST) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Guix Devel Dear, Currently "guix pull" from Savannah and issues can arise. As we recently experimented. Tobias and Ricardo recently discussed how to mirror the repo. IMHO, it is a good idea to mirror but not a good idea to locate it on Ricardo infrastructure, again. :-) Well, I propose to see if we can mirror on sourceware.org which are GNU friends. ;-) I took the liberty to contact them on IRC. See below. What I see is: we still "guix pull" from Savannah as usual. Then if Savannah is down, we catch the error and we try (transparently for the user) the sourceware url. As a first contact, they agree on the principle. Even they ask numbers about traffic etc. :-) What do you think? All the best, simon -- [Fri Dec 13 2019] I am working with Guix and we discussed mirroring the GNU Guix repo on sourceware. [17:56] The best before doing plan is to ask if it should be possible, on the principle. [17:57] is this a source code repo or somethign else? (sorry, not familiar) Also, what would be the reason for a mirror on sourceware if you already have a repo? [17:58] To be more concrete, and if you are not familiar with Guix, there is command named "guix pull" which basically "git pull". Currently, the main repo is on Savannah. And everything is fine with Savannah. :-) However, when Savannah is down; which happens, really rarely, but happens, then all Guix is kind-of broken. [17:59] so how big is this repo, what form is it, ? [18:00] Broken means that the users cannot update their system. And it is not related to development but related on normal usage. Well, "guix pull" is kind-of equivalent to "apt-get update". accessed via git or what? we're not set up to host large binaries e.g. [18:01] Yes, via Git. And it is only for source. Maybe hold off on this until new sourceware is online? The mirror of https://git.savannah.gnu.org/cgit/guix.git/ And if it is too big, maybe even only the master branch [18:02] hm, I can't git-clone that bad boy here ah never mind starting [18:03] What I have in mind, and I speak for myself and not on the behalf of the Guix project. not that big apparently, some 170MB of content, that's not a problem sorry, my browser returns me the wrong URL [18:04] https://git.savannah.gnu.org/cgit/guix.git/ https://git.savannah.gnu.org/git/guix.git yeah do you have a personal sourceware presence already? Just to be on the same wavelength, currently, each time an user run "guix pull" then they "git pull" from https://git.savannah.gnu.org/git/guix.git. However, when Savannah is down which rarely happens, everybody is annoyed. So I would like that "guix pull" catch the error and then try to use a mirror. Which means, that sourceware.org/your-convention-guix will be the default mirror. So the traffic pull will be very low (when Savannah is [18:11] down or unreachable) and the traffic push a bit more. guix is awesome! But savannah often is overloaded :{ No I do not have a personnal sourceware presence already. [18:12] * fche has no objection how do you plan to make the local git repo a fresh enough mirror? Then, I can ask to Savannah admin to have a hook to automatically mirror to sourceware or you can run a cron job on sourceware to auto git-pull periodically [18:13] yes, I have not think too much about the details. Because I would like first if it was even possible. :-) do you expect the repo to seriously grow over time ? [18:14] No. But I can do some stats with Git to answer you more precisely. Currently, Guix is around 13000 commits per years. However, I do not know how many addition lines it represents [18:16] It guix.git is ~124MB currently. It normally sees multiple commits a day, but most of them are fairly small (a couple of lines to a handful of files). [18:18] I mean it's not going to grow from 120MB to 1GB or 10G in the forseeable future? those are the levels at which we might get concerned Who knows ;-) But I do not think so. And in case, you can say: stop, it is too much for our infrastructure. [18:20] Well, if we have a first agreement on an hostage (mirror), can I save this conversation and post it on the mailing list guix-devel. Collect feedback; especially from the maintainers. Then come back to you with more concrete details. [18:21] 'forseeable' doesn't ask for infinite prophecy powers :) :-) yeah, I think we can do it. thank you for your help. but how much load on sourceware will this cause? do you want to be CC of the email? guix is very much pure GNU, and very much sources should produce reproducible builds. You can expect it to stay text based forever and never see any binary blobs. segher, it's a git repo similar in size to valgrind [18:22] Almost no load on sourceware. As I tried to explain above. It would be a feedback when Savannah is down -- which happens rarely from my experience. fche: no... how many pulls per seconds are asked from it [18:23] zimoun: that doesn't change the worst case at all segher: I have no idea. I think it is hard to count. segher, I expect nothing noticeable, but zimoun will let us know if there's reason to worry [18:24] ie a few downloads a day is nothing; a hundred git updates an hour is nothing and a simple gcc pull is almost 20m zimoun, I can also provide a backup mirror. Not as fancy as sourceware, but if you need something to fallback on. [18:25] segher: yes, it is just a first contact to see if we can agree. I totally understand your limitation. a fresh clone, i mean the gcc git repo is weird/svny and much larger yes my point is that we already are overloaded [18:26] not with git clones the server, in general zimoun, it would go under https://code.wildebeest.org/git/mirror/ (I cannot guarantee an SLA though, but I think neither can sourceware) and definitely not with git updates if it is actually light weight, then who cares ;-) [18:27] overloaded sourceware is why I suggested waiting until the new server comes online. right There's also the issue of outdated git. [18:28] it already is super annoying (to GCC) that half of the time (or whatever; it feels very often) the problem is caused by cygwin, not by us Probably more like 70% of the time. :-) [18:29] mjw: thank you. And thank you for the hard work you did on the gnu list moderation. cgf: it feels that way, yes zimoun, haha, yeah about that... o boy. I may be wrong but I don't think there are any other projects like cygwin on sourceware. cygwin is a source code repository (lightweight) and a binary distribution mechanism (imo, heavy weight) [18:30] but I digress... mjw: as you are also on #guix maybe you can suggest the idea of wildebeest. Did you subscribe to the guix-devel mailing list? [18:31] cgf: right, and sharing resources with a project with different usage patterns of those resources is a bit problematic segher, let us know of problems when they appear, and we'll keep tweaking things [18:32] there are a lot of knobs to turn fche: i can tell you things were bad yesterday [18:33] not too useful, eh :-) My point coming here and asking is: establish a first contact to see if we can go further together. Or if it is already a no go. Because I understand you need numbers and details. And I do not have them, right now. But I would like to open the discussion on this topic of mirroring. segher, exactly :) And on Monday gcc will switch to git too! [18:34] zimoun, we can put something like that up in a matter of minutes (gcc bugzilla gave ISEs most of the time) well as a mirror and then Dec 31! finally git all the way! mjw, that should help our loads, svn is much less efficient than git fche: thank you for the offer but I am not maintainer and I speak on my own voice. I would like to hear the civodul/Ludo's opinion :-) [18:35] civodul did suggest sourceware too, I did tell him to join here. But does he listen to me... [18:36] mjw: yeah! I know... that's why I am doing ;-) but you know, french diplomacy :-) [18:37]