Using lei with podman + toolbox

Using lei with podman + toolbox

[Konstantin Ryabitsev]

Hi, all:

These are my quickie instructions for how to use lei in a toolbox environment
if you are running a distribution like Fedora and don't want to install a lot
of perl dependencies into your main OS.

1. Grab the dockerfile:
   https://gist.github.com/mricon/046ba7c8b03bd92176dbe83e04f2466c

   Right now, it's as below, though it may change in the future:
   --- start: public-inbox.dockerfile ---
   # Podman/Toolbox container for public-inbox
   FROM docker.io/library/debian

   LABEL com.github.containers.toolbox="true" \
         com.github.debarshiray.toolbox="true"

   RUN apt-get update && \
       apt-get -y install sudo libcap2-bin locales vim \
                          git liburi-perl libemail-mime-perl libplack-perl libtimedate-perl \
                          libdbd-sqlite3-perl libsearch-xapian-perl libnet-server-perl \
                          libinline-c-perl libemail-address-xs-perl libparse-recdescent-perl \
                          xapian-tools libencode-perl libdbi-perl liblinux-inotify2-perl \
                          libio-compress-perl curl libmail-imapclient-perl libsocket-msghdr-perl \
                          sqlite3 libgit2-dev make eatmydata man-db pkg-config

   # Change this to your locale, if you're not en_CA
   RUN echo "en_CA.UTF-8 UTF-8" >> /etc/locale.gen && \
       locale-gen && \
       sed -i -e 's/ ALL$/ NOPASSWD:ALL/' /etc/sudoers && \
       touch /etc/localtime && \
       echo VARIANT_ID=container >> /etc/os-release

   RUN git clone https://public-inbox.org /usr/local/public-inbox && \
       cd /usr/local/public-inbox && \
       perl Makefile.PL && \
       make && \
       make install && \
       make clean

   CMD /bin/bash
   --- end: public-inbox.dockerfile ---

2. podman build -t public-inbox -f public-inbox.dockerfile
3. toolbox create --image localhost/public-inbox:latest lei
4. toolbox enter lei

That should let you use "lei" commands right after entering the container. To
update public-inbox, just run "git pull" in /usr/local/public-inbox and build
it again.

-K

Re: Using lei with podman + toolbox

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hi, all:
> 
> These are my quickie instructions for how to use lei in a toolbox environment
> if you are running a distribution like Fedora and don't want to install a lot
> of perl dependencies into your main OS.

Off the top of my head, I think Search::Xapian // Xapian.pm was
the main thing that was missing from CentOS 7.  Does Fedora have
that?

(disclaimer: I don't care for Docker, seems like a giant waste
of space and bandwidth compared to just using the distro)

> 1. Grab the dockerfile:
>    https://gist.github.com/mricon/046ba7c8b03bd92176dbe83e04f2466c
> 
>    Right now, it's as below, though it may change in the future:
>    --- start: public-inbox.dockerfile ---
>    # Podman/Toolbox container for public-inbox
>    FROM docker.io/library/debian
> 
>    LABEL com.github.containers.toolbox="true" \
>          com.github.debarshiray.toolbox="true"
> 
>    RUN apt-get update && \
>        apt-get -y install sudo libcap2-bin locales vim \
>                           git liburi-perl libemail-mime-perl libplack-perl libtimedate-perl \

Email::MIME isn't used at all outside of tests (but it's widely packaged).
No idea why libcap2-bin and vim are explicit dependencies (any
editor will do).  Don't need Plack for lei, either.

>                           libdbd-sqlite3-perl libsearch-xapian-perl libnet-server-perl \
>                           libinline-c-perl libemail-address-xs-perl libparse-recdescent-perl \

No need for Net::Server nor Parse::RecDescent for lei.  I don't
use Net::Server at all outside of tests, since I use systemd.

Email::Address::XS and TimeDate can be useful for messed up
messages, but low importance (I think they're widely packaged).
E:A:X and P:RD are required for -imapd but nothing else.

>                           xapian-tools libencode-perl libdbi-perl liblinux-inotify2-perl \
>                           libio-compress-perl curl libmail-imapclient-perl libsocket-msghdr-perl \

Socket::Msghdr makes lei a teeny bit faster, but I don't think
it's worth using another distro or running a compiler to get
since Inline::C is already available in all distros.  Everything
else should be in Fedora...

>                           sqlite3 libgit2-dev make eatmydata man-db pkg-config

eatmydata shouldn't be useful outside of development, and
libgit2+pkg-config isn't used by lei, yet
(it is for -httpd/-imapd/-nntpd)

native C++ Xapian wrapper [was: Using lei with podman + toolbox]

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Off the top of my head, I think Search::Xapian // Xapian.pm was
> the main thing that was missing from CentOS 7.  Does Fedora have
> that?

Btw, I've been considering a Just-Ahead-Of-Time C++ wrapper for
Xapian.  SWIG-or-not, bindings always seem behind and limited in
functionality compared to the native Xapian API.  I still need
to learn the "++" parts of C++...

Re: Using lei with podman + toolbox

[Konstantin Ryabitsev]

On Thu, Sep 09, 2021 at 11:36:14PM +0000, Eric Wong wrote:
> > These are my quickie instructions for how to use lei in a toolbox environment
> > if you are running a distribution like Fedora and don't want to install a lot
> > of perl dependencies into your main OS.
> 
> Off the top of my head, I think Search::Xapian // Xapian.pm was
> the main thing that was missing from CentOS 7.  Does Fedora have
> that?

Seems to have it as perl-Search-Xapian, which provides Xapian.pm.

> (disclaimer: I don't care for Docker, seems like a giant waste
> of space and bandwidth compared to just using the distro)

Well, this is for toolbox which uses podman, not docker. Toolbox is actually
the preferred mechanism in Fedora for setting up quickie work environments,
especially on something like Fedora Silverblue with its immutable root
partition.

I don't intend these instructions as the preferred mechanism for getting lei
up and running, just to be clear. Eventually, it will be packaged for most
distros -- but for now it's a convenient way to get the latest version on the
platform most likely to be most tested (Debian).

> >    RUN apt-get update && \
> >        apt-get -y install sudo libcap2-bin locales vim \
> >                           git liburi-perl libemail-mime-perl libplack-perl libtimedate-perl \
> 
> Email::MIME isn't used at all outside of tests (but it's widely packaged).
> No idea why libcap2-bin and vim are explicit dependencies (any
> editor will do).  Don't need Plack for lei, either.

Yeah, some of these were mostly for "make test" runs and others are adding
basic packages to the container image to make it slightly more usable (I don't
like nano or whatever is the default editor in the container image).

> No need for Net::Server nor Parse::RecDescent for lei.  I don't
> use Net::Server at all outside of tests, since I use systemd.
> 
> Email::Address::XS and TimeDate can be useful for messed up
> messages, but low importance (I think they're widely packaged).
> E:A:X and P:RD are required for -imapd but nothing else.
> 
> Socket::Msghdr makes lei a teeny bit faster, but I don't think
> it's worth using another distro or running a compiler to get
> since Inline::C is already available in all distros.  Everything
> else should be in Fedora...
> 
> >                           sqlite3 libgit2-dev make eatmydata man-db pkg-config
> 
> eatmydata shouldn't be useful outside of development, and
> libgit2+pkg-config isn't used by lei, yet
> (it is for -httpd/-imapd/-nntpd)

All noted -- I may be the one who packages things for Fedora at some point, so
this is useful info.

-K

Re: Using lei with podman + toolbox

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, Sep 09, 2021 at 11:36:14PM +0000, Eric Wong wrote:
> > > These are my quickie instructions for how to use lei in a toolbox environment
> > > if you are running a distribution like Fedora and don't want to install a lot
> > > of perl dependencies into your main OS.
> > 
> > Off the top of my head, I think Search::Xapian // Xapian.pm was
> > the main thing that was missing from CentOS 7.  Does Fedora have
> > that?
> 
> Seems to have it as perl-Search-Xapian, which provides Xapian.pm.

Oh I meant the newer SWIG Xapian.pm.  Search::Xapian is
Search/Xapian.pm which uses XS and isn't getting new features.

The XS version is far better-tested, but the SWIG version gives
access to some newer APIs.  AFAIK neither lets us do custom
query parsing like notmuch does in C++ (our approxidate
rt:/dt:/d: handling is a huge hack)

> > (disclaimer: I don't care for Docker, seems like a giant waste
> > of space and bandwidth compared to just using the distro)
> 
> Well, this is for toolbox which uses podman, not docker. Toolbox is actually
> the preferred mechanism in Fedora for setting up quickie work environments,
> especially on something like Fedora Silverblue with its immutable root
> partition.

Ah, still seems like a waste of space and bandwidth :>
(It just took me several hours to download upgrades from
 buster => bullseye on a small dev VM)

> I don't intend these instructions as the preferred mechanism for getting lei
> up and running, just to be clear. Eventually, it will be packaged for most
> distros -- but for now it's a convenient way to get the latest version on the
> platform most likely to be most tested (Debian).

I actually test everything on FreeBSD to force myself into doing
things portably.  Probably 95% of my "git push" is done on
FreeBSD (IOW, whenever I have connectivity to that VM).

> All noted -- I may be the one who packages things for Fedora at some point, so
> this is useful info.

Cool.  I'm wondering how to better arrange INSTALL to suit
different usages (lei-only vs daemons vs mda/watch-only).

Re: Using lei with podman + toolbox

[Konstantin Ryabitsev]

On Fri, Sep 10, 2021 at 01:56:53PM +0000, Eric Wong wrote:
> > Seems to have it as perl-Search-Xapian, which provides Xapian.pm.
> 
> Oh I meant the newer SWIG Xapian.pm.  Search::Xapian is
> Search/Xapian.pm which uses XS and isn't getting new features.

Ah, no, doesn't look like xapian-bindings-perl is built.
I may open an RFE with the packager for this once I get around to that.

> > All noted -- I may be the one who packages things for Fedora at some point, so
> > this is useful info.
> 
> Cool.  I'm wondering how to better arrange INSTALL to suit
> different usages (lei-only vs daemons vs mda/watch-only).

I mean, it's not like it pulls in huge dependency trees. Perl cpan packages
are a few KB in size, so pulling in some extras isn't going to inconvenience a
lot of people or anything. :)

-K