Github mirror and archive

Github mirror and archive

[Pjotr Prins]

A bit of a long post, but I would like some feedback:



** Going off github

Because of the discussion around JOSS I did a thought experiment last
night and it scares me: what if I closed my Google account, what if I
closed Facebook and what if I closed github. I'll turn this into a
BLOG entry, most likely.

Facebook would be the easy one (if they allow you - I think you can't
remove it). I use Facebook about weekly to check on some friends and
read a cartoon. Closing Facebook would have no impact on my life
though changing accounts would be a hassle because I would need to
re-register the interesting friends.

Google is a bit harder. I use gmail to monitor the JOSS mailing list
and a few others. I run my own mail server so I can switch Google
accounts and easily re-register. You can't use groups without a Google
account, but if I closed today I could be back when I wanted with a
different account. No real harm done. I also need it for the Google
Summer of Code (GSoC).  Google is famous for tracking and when you are
logged in (as we normally are) it can track the sites you visit. It is
enlightening to visit

 https://myactivity.google.com/myactivity?utm_source=my-account&utm_medium=&utm_campaign=my-acct-promo

For example I visited the Dlang website yesterday which apparently is
tracked via facebook AND google. Google knows where I am and that I
rented a car and my current interest is supercomputing. I have decided
to start switching Google accounts annually. Fortunately Google caters
for that - so I am not the only one. Even GSoC requires you to
re-register every year.

Now Github. I am one of the early github adopters and created my first
repository November 2008. Being a Ruby guy I was aware of the garage
startup and very much in favour of what they were attempting. It was
(and perhaps is) a cool distributed company. But now it is very
critical to my work flow. Too critical, I think, because even the
thought experiment of removing my account scares me. First of all
github provides me an internet personae. Anyone who wants to assess my
work can visit github and get a clear picture of what I am working on.
This is a valuable resource, but I can probably move it elsewhere
without too much loss. Github has a nice way of showing work, though,
and it is easy to see what organizations I am contributing to and what
type of comments I post. I always tell students that the github track
record is more important (to people like me) than the publishing
record in scientific journal. Code is much clearer to me about what
people (can) contribute. I therefore use github to assess and monitor
other peoples work. Hosting your code elsewhere, as long as it has a
decent interface, is good enough for me - I'll change that
recommendation to something more generic. Commenting on multiple sites
will need multiple accounts.

So I need an account for github hosted repositories, but I can switch
accounts annually without too many concerns, though it will be a
hassle.

There are a few projects and organizations, however, which make it
very hard to leave my account and one of them is my work environment
(https://github.com/genenetwork) and the other is the Journal of Open
Source Software (JOSS). They require an account and they, implicitely,
require me to use the same account over time. These two environments
(and a few others) depend on all members in the organization to be
stable. It would not do if everyone switched accounts every year.

GeneNetwork could move elsewhere, but it would make it much harder to
collaborate. People already have github accounts which makes it easy
to contribute. Being on a different site requires people logging in
separately and would impact contributions (including bug reports) for
sure. I am certain this will have an impact: you want to make
contributing as easy as possible (especially in Science where people
don't have patience to juggle accounts).

The key thing here is the issue tracker. Github made it so that you
can not easily move the issue tracker to a different provider. Much of
the value of github is in using the issue trackers.

This cluster effect is much more visible on JOSS. In fact the journal
can not exist without github because the software stack is (1) built
on the issue tracker and (2) attracts authors and reviewers through
its network. Removing my account would invalidate the reviews I am
involved in and moving JOSS elsewhere would loose the cluster effect.

In all, I have come to depend on github, critically because
organizations I am involved in would be severely impacted when we
moved elsewhere. Even for GNU Guix - which pointedly does not use
github - would be severely hampered if something happened to
github. So many projects are hosted on github that distribution and
deployment of free software would be severely broken if github went
down. In other words we have built a monster.

People wrote me that they are concerned about other trends. For
example the Julia language is basing its package infrastructure on
github and therefore compromises robustness for convenience. The only
setup that is robust is a distributed setup - i.e., to not depend on
one party.

If github changed its ways (e.g., add advertising, a new charging
model, sell user behavior to 3rd parties or government) or its
policies (i.e., repository incompatibility with GPL - they are border
line already) and we would want to move out, it would be incredibly
hard, not least because some of our software is built on top of
github's interface and API. And because github's software is non-FOSS
we'd be stuck somewhere in a dark land.

This is not a situation I want to be in. I don't like to have all my
eggs in one basket. It is why I run my own mail server and web
server. It is why I don't like Google groups and github.

I hereby decide I am going to (gradually) move all my projects off
github and remove my main account. This won't happen within a year, I
expect, but it is clear to me that I need to do this. I will still
have github accounts, but plan to change them on a regular basis.

For projects that depend on github (such as JOSS) we ought to build a
mirror of the issue tracker. There already is some software for this
(hosted on, yes, github), so it should be reasonably trivial. Also
software releases distributed by github should be mirrored elsewhere.
GNU Guix, for example, should mirror and fully archive the source
tarballs that github provides.

Pj.

-- 

Re: Github mirror and archive

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

2017-04-06 11:53 GMT+02:00 Pjotr Prins <pjotr.public12@thebird.nl>:


In all, I have come to depend on github, critically because
> organizations I am involved in would be severely impacted when we
> moved elsewhere. Even for GNU Guix - which pointedly does not use
> github - would be severely hampered if something happened to
> github. So many projects are hosted on github that distribution and
> deployment of free software would be severely broken if github went
> down. In other words we have built a monster.
>

[...]



> Also
> software releases distributed by github should be mirrored elsewhere.
> GNU Guix, for example, should mirror and fully archive the source
> tarballs that github provides.
>

A side note: If and when Guix will have GNUnet, it could use it to
distribute non only binaries but also tarballs

[-- Attachment #2: Type: text/html, Size: 1338 bytes --]

Re: Github mirror and archive

[Pjotr Prins]

On Thu, Apr 06, 2017 at 09:53:02AM +0000, Pjotr Prins wrote:
> A bit of a long post, but I would like some feedback:

I guess most of you agree ;). Preaching to the converted.

Pj.

Re: Github mirror and archive

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> First of all
> github provides me an internet personae. Anyone who wants to assess my
> work can visit github and get a clear picture of what I am working on.
> This is a valuable resource, but I can probably move it elsewhere
> without too much loss. Github has a nice way of showing work, though,
> and it is easy to see what organizations I am contributing to and what
> type of comments I post. I always tell students that the github track
> record is more important (to people like me) than the publishing
> record in scientific journal. […] I therefore use github to assess and monitor
> other peoples work. Hosting your code elsewhere, as long as it has a
> decent interface, is good enough for me - I'll change that
> recommendation to something more generic. Commenting on multiple sites
> will need multiple accounts.

This is what I find scary.  I’ve received job offers from people who
apparantly judged my skill based solely on what I published on Github.
Ironically this happened right after I moved most of my projects to my
private server, so there wasn’t really much to look at anyway.

There also appear to be a number of researchers who think they can
derive trends and representative statements about free software from a
survey of what’s on Github.

> People wrote me that they are concerned about other trends. For
> example the Julia language is basing its package infrastructure on
> github and therefore compromises robustness for convenience. The only
> setup that is robust is a distributed setup - i.e., to not depend on
> one party.

The “devtools” package for R is also a popular way for people to install
software directly from Github, giving others an incentive to use and
publish their code on Github.

Does Julia really *depend* on Github, or is this a more generic
dependency on any git repository?

> If github changed its ways (e.g., add advertising, a new charging
> model, sell user behavior to 3rd parties or government) or its
> policies (i.e., repository incompatibility with GPL - they are border
> line already) and we would want to move out, it would be incredibly
> hard, not least because some of our software is built on top of
> github's interface and API. And because github's software is non-FOSS
> we'd be stuck somewhere in a dark land.

Things like that have happened to hosters in the past.  It’s not just a
“what if” scenario.  People are forgetful.

> I hereby decide I am going to (gradually) move all my projects off
> github and remove my main account. This won't happen within a year, I
> expect, but it is clear to me that I need to do this. I will still
> have github accounts, but plan to change them on a regular basis.

I applaud this decision.  I have done something similar by removing
*most* of my personal projects from there, except for those that I’m
required to work on at my job and those where I still haven’t taken the
time to work off the issue tracker…

Changing accounts regularly seems like a good idea, too.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Github mirror and archive

[Marco van Zwetselaar]

On 06/04/17 12:53, Pjotr Prins wrote:
> ** Going off github

I've been running GitLab self-hosted since December and find it a fine
alternative.

Their installation instructions try hard to scare you away from
self-hosting and installation from source, but I think that's more to
keep non-sysadmins from shooting themselves in the foot than it is to do
with inherent complexity.

A guix package[1] of course would be the bee's knees. :)

GitLab's OAuth2 authentication and GitHub (and other) import facilities
ease integration with those clinging to their GitHub or other 'social'
accounts.

Zwets

[1] I've been dying to make one as I'm stuck at GitLab 8.14 due to
version dependencies in 8.15+ that aren't in Ubuntu 16.04 LTS.  My
servers run Ubuntu LTS, with Guix for out-of-distro packages.  I lack
the time and guixperience to do this right now but will gladly share
notes and help out if someone takes this on.

Re: Github mirror and archive

[Marco van Zwetselaar]

Forgot to mention the obvious: the rationale for the below is my desire
to unchain from GitHub.


On 07/04/17 10:26, Marco van Zwetselaar wrote:
> On 06/04/17 12:53, Pjotr Prins wrote:
>> ** Going off github
> I've been running GitLab self-hosted since December and find it a fine
> alternative.
>
> Their installation instructions try hard to scare you away from
> self-hosting and installation from source, but I think that's more to
> keep non-sysadmins from shooting themselves in the foot than it is to do
> with inherent complexity.
>
> A guix package[1] of course would be the bee's knees. :)
>
> GitLab's OAuth2 authentication and GitHub (and other) import facilities
> ease integration with those clinging to their GitHub or other 'social'
> accounts.
>
> Zwets
>
> [1] I've been dying to make one as I'm stuck at GitLab 8.14 due to
> version dependencies in 8.15+ that aren't in Ubuntu 16.04 LTS.  My
> servers run Ubuntu LTS, with Guix for out-of-distro packages.  I lack
> the time and guixperience to do this right now but will gladly share
> notes and help out if someone takes this on.
>
>

Re: Github mirror and archive

[ng0]

Marco van Zwetselaar transcribed 0.9K bytes:
> On 06/04/17 12:53, Pjotr Prins wrote:
> > ** Going off github
> 
> I've been running GitLab self-hosted since December and find it a fine
> alternative.
> 
> Their installation instructions try hard to scare you away from
> self-hosting and installation from source, but I think that's more to
> keep non-sysadmins from shooting themselves in the foot than it is to do
> with inherent complexity.
> 
> A guix package[1] of course would be the bee's knees. :)
> 
> GitLab's OAuth2 authentication and GitHub (and other) import facilities
> ease integration with those clinging to their GitHub or other 'social'
> accounts.
> 
> Zwets
> 
> [1] I've been dying to make one as I'm stuck at GitLab 8.14 due to
> version dependencies in 8.15+ that aren't in Ubuntu 16.04 LTS.  My
> servers run Ubuntu LTS, with Guix for out-of-distro packages.  I lack
> the time and guixperience to do this right now but will gladly share
> notes and help out if someone takes this on.
> 
> 
> 

GitLab is super resource consuming. Okay, the more users, the more
resources are used up anywhere, but we are glad when we can switch from
Gitlab to Pagure on our new server.
In resources it won't make much of a difference, but the specs of the
server are far better.

I seem to recall looking into packaging gitlab a while back. I don't
count it as a priority for me. But just as a note, it rarely ever works
out to wait for someone else to take on the job.
I have an almost finished 0ad for months now. No one picks up the
package in the middle which needs to be fixed for 0ad to build. If you
can work on it and maybe need assistance and it can take a long time
anyway, it's a good task to work on, even if it lies around, collects
digital dust and serves as a starting point for someone else :) That's
how rustc came to be, someone did it, left it, someone else did it, left
it, I did it and left it, and 3 or 4 more people worked on it until it
finally was ready.

Re: Github mirror and archive

[Pjotr Prins]

On Fri, Apr 07, 2017 at 03:58:13PM +0000, ng0 wrote:
> digital dust and serves as a starting point for someone else :) That's
> how rustc came to be, someone did it, left it, someone else did it, left
> it, I did it and left it, and 3 or 4 more people worked on it until it
> finally was ready.

I think that is fine advice. Best is to start.

Pj.
-- 

Re: Github mirror and archive

[Ludovic Courtès]

Hello!

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> ** Going off github

Congrats, sounds like the right decision to me!

In addition to its most obvious problems¹ (including privacy—“if the
service is free, then *you* are the product”), GitHub has become a
single point of failure for a large part of the free software commons.
It has managed to build a centralized platform around a distributed VCS,
to the point that many Git users seem to not realize that Git itself is
distribute, or in some cases, that Git can be used without GitHub.

Similar to Marco’s experience, the institute I work for, which was
running a FusionForge instance and losing users to github.com, has now
deployed a self-hosted GitLab-free-software instance, which I think is a
great improvement.

Kudos!  :-)

Ludo’.

¹ https://www.gnu.org/software/repo-criteria-evaluation.html