Limited-history local archives

[Konstantin Ryabitsev <konstantin@linuxfoundation.org>]

Hi, all:

I wonder if it would be useful to have a feature allowing someone to run 
a limited-history local copy of a larger remote archive -- for example 
if someone only wanted a 3-month copy of LKML instead of the whole 
20-year enchilada.

It's possible to accomplish this with git already [^1], e.g. you can use 
the following to grab a copy of LKML starting with December 2019:

  $ git clone --bare --shallow-since 2019-12-01 https://lore.kernel.org/lkml/git/7 lkml-since-dec.git
  $ cd lkml-since-dec.git
  $ git config --add remote.origin.fetch '+refs/heads/master:refs/heads/master'

You can now run "git fetch" as usual and perform all the normal 
operations, such as "git show {rev}:m" to get the message contents.  
Obviously, if we try to get a revision from before December 1, the 
operation fails:

  $ git show dae740ca679710fbe8b97b3e704d63e3e7883fd9:m
  fatal: Path 'm' does not exist in 'dae740ca679710fbe8b97b3e704d63e3e7883fd9'

If we enable uploadpack.allowAnySHA1InWant on the server, we can then 
fetch this object directly:

  $ git fetch --depth 1 origin dae740ca679710fbe8b97b3e704d63e3e7883fd9
  remote: Counting objects: 3, done.
  remote: Compressing objects: 100% (2/2), done.
  remote: Total 3 (delta 0), reused 3 (delta 0)
  Unpacking objects: 100% (3/3), done.
  From https://lore.kernel.org/lkml/git/7
   * branch              dae740ca679710fbe8b97b3e704d63e3e7883fd9 -> FETCH_HEAD

Now this succeeds:

  $ git show dae740ca679710fbe8b97b3e704d63e3e7883fd9:m

We can then periodically reshallow the archive (e.g. once a day) in 
order to get rid of older objects:

  $ git fetch --shallow-since 2019-12-15 --update-shallow origin master
  $ git gc --prune=now

There isn't really an RFC or anything associated with this -- I just 
wanted to share this idea as a possibly useful way of reducing local 
storage requirements while still being able to operate directly on 
public-inbox git repositories -- e.g. with a tool like l2md 
(https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/).

-K

[^1]: Theoretically, this will become even easier in the future with 
      partial-clone functionality, though I believe that's mostly 
      written to support fetching large blobs from CDNs and wouldn't be 
      as useful for very linear public-inbox repositories.