[PATCH 0/2] doc updates for -clone + -cindex

[PATCH 0/2] doc updates for -clone + -cindex

[Eric Wong]

cindex still needs work (both doc and implementation)...

Eric Wong (2):
  doc: add manpage for -cindex
  doc: clone: add a note about git 2.41+ and fetch.hideRefs

 Documentation/public-inbox-cindex.pod | 141 ++++++++++++++++++++++++++
 Documentation/public-inbox-clone.pod  |   6 +-
 MANIFEST                              |   1 +
 3 files changed, 147 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/public-inbox-cindex.pod

[PATCH 1/2] doc: add manpage for -cindex

[Eric Wong]

It's similar to a combination of -index and -extindex but
perhaps more refined this time around...
---
 Documentation/public-inbox-cindex.pod | 141 ++++++++++++++++++++++++++
 MANIFEST                              |   1 +
 2 files changed, 142 insertions(+)
 create mode 100644 Documentation/public-inbox-cindex.pod

diff --git a/Documentation/public-inbox-cindex.pod b/Documentation/public-inbox-cindex.pod
new file mode 100644
index 00000000..93149e62
--- /dev/null
+++ b/Documentation/public-inbox-cindex.pod
@@ -0,0 +1,141 @@
+=head1 NAME
+
+public-inbox-index - create and update search for code repositories
+
+=head1 SYNOPSIS
+
+public-inbox-cindex [OPTIONS] GIT_DIR...
+
+public-inbox-cindex [OPTIONS] --update
+
+=head1 DESCRIPTION
+
+public-inbox-cindex creates and updates the Xapian search index for
+git code repository (C<coderepo>) search.  It can also associate
+(fuzzy join) coderepos with Xapian-indexed inboxes.  It only indexes
+commit messages and diffs as they would show up in a an email.  It
+does not index the contents of blobs directly.
+
+Like inbox indices, coderepo indices can either be internal or external
+to a coderepo.  Either way, they're both created and updated through
+public-inbox-cindex.
+
+Once the initial indices are created by public-inbox-cindex,
+the L</--update> switch will incrementally update them.
+
+=head1 OPTIONS
+
+=over
+
+=item -d EXTDIR
+
+Use the given directory as an external index.  External indices are
+generally recommended to internal indices since they do not need
+write access to any code repositories themselves.  They are highly
+recommended when many repositories share a common history or if
+there is an M:N relationship between inboxes and coderepos.
+
+=item -j JOBS
+
+=item --jobs=JOBS
+
+Influences the number of Xapian indexing shards.
+
+If the repo has not been indexed or initialized, C<JOBS - 1>
+shards will be created.
+
+Default: the number of existing Xapian shards
+
+=item --reindex
+
+Forces a re-index of all commits.  This can be used for in-place
+upgrades and bugfixes while read-only processes are utilizing the index.
+
+=item --update
+
+=item -u
+
+Incrementally index all previously-indexed coderepos.
+
+=item --prune
+
+Removes commits which are no longer accessible via git.
+Use this after L<git-gc(1)> (or L<git-prune(1)>).
+
+=item --no-fsync
+
+Disables L<fsync(2)> and L<fdatasync(2)> operations on SQLite
+and Xapian.  This is only effective with Xapian 1.4+.  This is
+primarily intended for systems with low RAM and the small
+(default) C<--batch-size=1m>.  Users of large C<--batch-size>
+may even find disabling L<fdatasync(2)> causes too much dirty
+data to accumulate, resulting on latency spikes from writeback.
+
+=item --max-size SIZE
+
+=item --batch-size SIZE
+
+These affect the coderepo index the same way they affect
+inbox indices.  See L<public-inbox-index(1)>.
+
+=back
+
+=head1 FILES
+
+For internal indices, the Xapian DB is stored in
+C<$GIT_DIR/public-inbox-cindex>.
+
+External indices are stored wherever L</-d> EXTDIR points.
+
+=head1 CONFIGURATION
+
+=over 8
+
+=item publicinbox.indexMaxSize
+
+=item publicinbox.indexBatchSize
+
+These configuration knobs affect the coderepo index the same way
+they affect inbox indices.  See L<public-inbox-index(1)>.
+
+=back
+
+=head1 ENVIRONMENT
+
+=over 8
+
+=item PI_CONFIG
+
+Used to override the default "~/.public-inbox/config" value.
+
+=item XAPIAN_FLUSH_THRESHOLD
+
+The number of documents to update before committing changes to
+disk.  This environment is handled directly by Xapian, refer to
+Xapian API documentation for more details.
+
+Use C<publicinbox.indexBatchSize> instead.
+
+=back
+
+=head1 UPGRADING
+
+Occasionally, public-inbox will update it's schema version and
+require a full reindex by running this command with L</--reindex>.
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<public-inbox-index(1)>
diff --git a/MANIFEST b/MANIFEST
index 44eaa497..1001ca08 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -67,6 +67,7 @@ Documentation/lei.pod
 Documentation/lei_design_notes.txt
 Documentation/marketing.txt
 Documentation/mknews.perl
+Documentation/public-inbox-cindex.pod
 Documentation/public-inbox-clone.pod
 Documentation/public-inbox-compact.pod
 Documentation/public-inbox-config.pod

Re: [PATCH 1/2] doc: add manpage for -cindex

[Kyle Meyer]

Eric Wong writes:

> +=head1 NAME
> +
> +public-inbox-index - create and update search for code repositories

s/-index/-cindex/

> +(fuzzy join) coderepos with Xapian-indexed inboxes.  It only indexes
> +commit messages and diffs as they would show up in a an email.  It

s/a an/an/

> +=head1 UPGRADING
> +
> +Occasionally, public-inbox will update it's schema version and

s/it's/its/

(This typo is shared with the docs of public-inbox-extindex and
public-inbox-index.)

Re: [PATCH 1/2] doc: add manpage for -cindex

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> 
> > +=head1 NAME
> > +
> > +public-inbox-index - create and update search for code repositories
> 
> s/-index/-cindex/
> 
> > +(fuzzy join) coderepos with Xapian-indexed inboxes.  It only indexes
> > +commit messages and diffs as they would show up in a an email.  It
> 
> s/a an/an/

Thanks, will push with this squashed in:

diff --git a/Documentation/public-inbox-cindex.pod b/Documentation/public-inbox-cindex.pod
index 93149e62..d2d5dadc 100644
--- a/Documentation/public-inbox-cindex.pod
+++ b/Documentation/public-inbox-cindex.pod
@@ -1,6 +1,6 @@
 =head1 NAME
 
-public-inbox-index - create and update search for code repositories
+public-inbox-cindex - create and update search for code repositories
 
 =head1 SYNOPSIS
 
@@ -13,7 +13,7 @@ public-inbox-cindex [OPTIONS] --update
 public-inbox-cindex creates and updates the Xapian search index for
 git code repository (C<coderepo>) search.  It can also associate
 (fuzzy join) coderepos with Xapian-indexed inboxes.  It only indexes
-commit messages and diffs as they would show up in a an email.  It
+commit messages and diffs as they would show up in an email.  It
 does not index the contents of blobs directly.
 
 Like inbox indices, coderepo indices can either be internal or external
@@ -120,7 +120,7 @@ Use C<publicinbox.indexBatchSize> instead.
 
 =head1 UPGRADING
 
-Occasionally, public-inbox will update it's schema version and
+Occasionally, public-inbox will update its schema version and
 require a full reindex by running this command with L</--reindex>.
 
 =head1 CONTACT

[PATCH 3/2] doc: *index: fix misuse of apostrophe in `its'

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> > +=head1 UPGRADING
> > +
> > +Occasionally, public-inbox will update it's schema version and
> 
> s/it's/its/
> 
> (This typo is shared with the docs of public-inbox-extindex and
> public-inbox-index.)

Thanks.  I think I remember learning this many decades ago :x

---------8<-----------
Subject: [PATCH] doc: *index: fix misuse of apostrophe in `its'

Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87leeovmig.fsf@kyleam.com/
---
 Documentation/public-inbox-extindex.pod | 2 +-
 Documentation/public-inbox-index.pod    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/public-inbox-extindex.pod b/Documentation/public-inbox-extindex.pod
index 8370b905..66e04060 100644
--- a/Documentation/public-inbox-extindex.pod
+++ b/Documentation/public-inbox-extindex.pod
@@ -121,7 +121,7 @@ Default: none, uses C<publicinbox.indexBatchSize>
 
 =head1 UPGRADING
 
-Occasionally, public-inbox will update it's schema version and
+Occasionally, public-inbox will update its schema version and
 require a full index by running this command.
 
 =head1 CONTACT
diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod
index 011ade3c..11108f78 100644
--- a/Documentation/public-inbox-index.pod
+++ b/Documentation/public-inbox-index.pod
@@ -319,7 +319,7 @@ Default: none, uses C<publicinbox.indexBatchSize>
 
 =head1 UPGRADING
 
-Occasionally, public-inbox will update it's schema version and
+Occasionally, public-inbox will update its schema version and
 require a full index by running this command.
 
 =head1 CONTACT

[PATCH 2/2] doc: clone: add a note about git 2.41+ and fetch.hideRefs

[Eric Wong]

It's unusable for large mirrors, otherwise.
---
 Documentation/public-inbox-clone.pod | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Documentation/public-inbox-clone.pod b/Documentation/public-inbox-clone.pod
index 152fed73..c3254fa0 100644
--- a/Documentation/public-inbox-clone.pod
+++ b/Documentation/public-inbox-clone.pod
@@ -127,7 +127,11 @@ as relative to the C<DESTINATION> directory.  If only C<--manifest=>
 is specified where C<FILE> is an empty string (C<"">), then C<manifest.js.gz>
 (C<$DESTINATION/manifest.js.gz>) is the implied value of C<FILE>.
 
-This is a new option in public-inbox 2.0+
+When updating manifests with many forks using the same objstore,
+git 2.41+ is highly recommended for performance as we automatically
+use the C<fetch.hideRefs> feature to speed up negotiation.
+
+C<--manifest=> is a new option in public-inbox 2.0+
 
 =item --remote-manifest=URL|RELATIVE_PATH