Re: Showcasing lei at Linux Plumbers

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Eric:
> 
> I am getting ready for my presentation to the Linux Plumbers (happening in a
> few weeks, eek), which is based around lore, lei (I see what you did there)
> and search-based subscriptions. I want to make it hands-on with practical
> examples, which is what developers would appreciate more than just dry
> manpages.
> 
> I am in the process of wrapping my head around lei tooling, but I may have
> some questions in the process, so I wanted to start this thread as a record of
> my poking at it. :)

Yeah, I'm still trying to figure out how some things are
supposed to work myself...

> What I currently have:
> 
> - an imap mailbox 
> - lei configured and installed locally (in a debian container)

Fwiw, most of the functionality works much better with Maildir
because of potential password prompts needed for IMAP and
interactivity required.

> The goal is to illustrate how to use this to start "receiving" mail for a
> subsystem without subscribing to any of the lists. The example I have in mind
> is the LANDLOCK subsystem, and the reason I picked it is because it already
> has a well-defined set of search criteria we can use:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS#n10462
> 
>     LANDLOCK SECURITY MODULE
>     ...
>     F:  Documentation/security/landlock.rst
>     F:  Documentation/userspace-api/landlock.rst
>     F:  include/uapi/linux/landlock.h
>     F:  samples/landlock/
>     F:  security/landlock/
>     F:  tools/testing/selftests/landlock/
>     K:  landlock
>     K:  LANDLOCK
> 
> This means we want to configure lei to grab any mail from lore.kernel.org/all/
> that matches this query:
> 
>     dfn:Documentation/security/landlock.rst OR
>     dfn:Documentation/userspace-api/landlock.rst OR
>     dfn:include/uapi/linux/landlock.h OR
>     dfn:samples/landlock/ OR
>     dfn:security/landlock/ OR
>     dfn:tools/testing/selftests/landlock/ OR
>     dfhh:landlock
> 
> https://lore.kernel.org/all/?q=dfn%3ADocumentation%2Fsecurity%2Flandlock.rst+OR+dfn%3ADocumentation%2Fuserspace-api%2Flandlock.rst+OR+dfn%3Ainclude%2Fuapi%2Flinux%2Flandlock.h+OR+dfn%3Asamples%2Flandlock%2F+OR+dfn%3Asecurity%2Flandlock%2F+OR+dfn%3Atools%2Ftesting%2Fselftests%2Flandlock%2F+OR+dfhh%3Alandlock

For HTTP(S)-based queries, I would add rt: (received-time)
around the whole thing and maybe use "lei edit-search" to tweak
for subsequent runs.  Not sure if the rt: handling should be
automatic for HTTP(S) (local Xapian searches track max docid, instead)

> I'll want to retrieve any threads and follow-ups and upload them to my imap
> landlock folder -- and then run in the background and just continuously update
> things as more mail comes in, so I don't have to remember to run anything
> manually.
> 
> What succession of lei commands would accomplish this?

OK, there's two main commands, "lei q" and "lei up".
Both of which may prompt for passwords depending on how
git-credential is set up:

	# the destination, could be Maildir
	MFOLDER=imaps://user@example.com/INBOX.landlock

	# initial search:
	lei q -o $MFOLDER -t -I https://lore.kernel.org/all/ --stdin <<EOF
	(
		dfn:Documentation/security/landlock.rst OR
		dfn:Documentation/userspace-api/landlock.rst OR
		dfn:include/uapi/linux/landlock.h OR
		dfn:samples/landlock/ OR
		dfn:security/landlock/ OR
		dfn:tools/testing/selftests/landlock/ OR
		dfhh:landlock
	) AND rt:2.months.ago..
	EOF

	# update whenever, may prompt for IMAP password, but could be
	# cron-ed or similar if passwords are cached
	lei up $MFOLDER

	# Optional: tweaking the search parameters can be done via
	lei edit-search $MFOLDER

For Maildirs, "lei up --all=local" works as it should.

"lei up --all" and "lei up --all=remote" don't work, yet,
because prompting for multiple IMAP folders (with potentially
different accounts) can get a bit complicated.  But
"lei up $ONE_IMAP_FOLDER" already works.

> Thanks for your continued help.

No problem, thanks for your patience since everything seems
overwhelming :<