unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* WIP: searching all of lore
@ 2020-11-26 19:45 Eric Wong
  2020-11-28 22:34 ` Eric Wong
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Eric Wong @ 2020-11-26 19:45 UTC (permalink / raw)
  To: workflows; +Cc: meta

Requires Tor, for now:

http://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion/all/
http://lore.czquwvybam4bgbro.onion/all/

It seems v3 onions with longer URLs are more secure, these days;
but requires newer Tor.

On Debian or RH-based systems, it's as easy as:
	<apt|yum> install tor torsocks

	# assuming w3m is installed:
	torsocks w3m http://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion/all/

Other browsers can configure SOCKS5 proxies to 127.0.0.1:9050
or be wrapped via torsocks (which uses LD_PRELOAD)

Disclaimers:

  I don't know much about Tor security (or security in general).
  I see Tor as an alternative to paying corrupt organizations
  (ICANN) and lets me self-host without a static IP address.

  I've also had numerous ISP and power outages this year,
  probably because more neighbors are home due to the pandemic,
  so don't expect 99.999% uptime, either.  And I'm a klutz who
  always trips over cables on (relatively) good days, and I've
  only had bad days since March :<

How to replicate
----------------

	# I'm using the following to update mirror and lore.kernel.org
	# (see grokmirror docs) for more.  Old command:
	grok-pull -v -c repos.conf
	public-inbox-index --all # update per-inbox indices

	# add "-L basic" or "-L medium" to reduce space requirements
	# to either -extindex or -index commands

	# The new command, not finalized yet:
	public-inbox-extindex --all -v /path/to/ALL


The following changes in a otherwise boring ~/.public-inbox/config
(or whatever $PI_CONFIG is set to)

; not yet stable or finalized, yet:
; this section allows, "all" is a special case, currently
[extindex "all"]
	topdir = /path/to/ALL

; these are already documented in public-inbox-config(5)
[publicinbox]
	; 'all' ignores domain name matching,
	; useful for inboxes served via multiple domains
	wwwlisting = all
	grokManifest = all

	; users with larger machines may want to bump this,
	; the default is for machines w/ 256MB RAM
	; (which I still use, sometimes)
	indexBatchSize = 100m

# I'm using the following to update from lore.kernel.org
# (see grokmirror docs) for more
grok-pull -v -c repos.conf
public-inbox-index --all # update per-inbox indices
public-inbox-extindex --all -v /path/to/ALL # index [extindex "all"]

This is running commit 95cb3e48fc5c4e847cdc111c2c8c9f0b70bdea56
git clone https://public-inbox.org/public-inbox.git

More changes coming (JMAP, speedups), and there's probably still
lots of stuff broken and need fixing (including my brain :<)

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2021-03-17 18:37 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-26 19:45 WIP: searching all of lore Eric Wong
2020-11-28 22:34 ` Eric Wong
2020-12-05 20:07   ` Eric Wong
2020-12-08 14:01     ` Konstantin Ryabitsev
2020-12-08 18:02       ` Eric Wong
2020-12-08 18:11         ` Konstantin Ryabitsev
2020-12-01 14:00 ` Konstantin Ryabitsev
2020-12-01 18:48   ` Eric Wong
2021-03-17  7:11 ` Eric Wong
2021-03-17 13:27   ` Konstantin Ryabitsev
2021-03-17 18:18     ` Eric Wong
2021-03-17 18:37       ` Konstantin Ryabitsev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).