Remote xapian database

Remote xapian database

[admin]

Hello,

Xapian have remote database mode:
http://xapian.org/docs/remote.html

Do you plan use it in future? I have more then 8 million letters and my
xapian db use 50 gb store :)

If I can use remote database, I will move database to external server
from my work PC (with Pentium 4) it will be very fast! :)

Thank you!

-- 
Best regards,
Roman Kravets

Re: Remote xapian database

[Mark Walters]

On Thu, 26 Jun 2014, admin@softded.net wrote:
> Hello,
>
> Xapian have remote database mode:
> http://xapian.org/docs/remote.html
>
> Do you plan use it in future? I have more then 8 million letters and my
> xapian db use 50 gb store :)

I don't think we have any plans to do that. However, assuming you use
emacs frontend (and/or the command line) then notmuch works very well
over ssh. See http://notmuchmail.org/remoteusage/

Several of the notmuch developers (including me) use this so although it
is not guaranteed to work correctly, in practice it seems to work pretty
well.

Best wishes

Mark

>
> If I can use remote database, I will move database to external server
> from my work PC (with Pentium 4) it will be very fast! :)
>
> Thank you!
>
> -- 
> Best regards,
> Roman Kravets
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: Remote xapian database

[David Bremner]

admin@softded.net writes:

> Hello,
>
> Xapian have remote database mode:
> http://xapian.org/docs/remote.html
>
> Do you plan use it in future? I have more then 8 million letters and my
> xapian db use 50 gb store :)
>
> If I can use remote database, I will move database to external server
> from my work PC (with Pentium 4) it will be very fast! :)

One thing to keep in mind is that there are several places in the code
where mail files are opened directly, so just having remote access to
the xapian database is not going to be enough, unless you want to have a
local copy of the messages, which is almost surely larger than the
database.

Re: Remote xapian database

[Roman Kravets]

Mark,

In this case, the file with the letters must be on the remote server?


On Thu, Jun 26, 2014 at 04:25:17PM +0100, Mark Walters wrote:
> 
> On Thu, 26 Jun 2014, admin@softded.net wrote:
> > Hello,
> >
> > Xapian have remote database mode:
> > http://xapian.org/docs/remote.html
> >
> > Do you plan use it in future? I have more then 8 million letters and my
> > xapian db use 50 gb store :)
> 
> I don't think we have any plans to do that. However, assuming you use
> emacs frontend (and/or the command line) then notmuch works very well
> over ssh. See http://notmuchmail.org/remoteusage/
> 
> Several of the notmuch developers (including me) use this so although it
> is not guaranteed to work correctly, in practice it seems to work pretty
> well.
> 
> Best wishes
> 
> Mark
> 
> >
> > If I can use remote database, I will move database to external server
> > from my work PC (with Pentium 4) it will be very fast! :)
> >
> > Thank you!
> >
> > -- 
> > Best regards,
> > Roman Kravets
> > _______________________________________________
> > notmuch mailing list
> > notmuch@notmuchmail.org
> > http://notmuchmail.org/mailman/listinfo/notmuch

Re: Remote xapian database

[Roman Kravets]

Dear David,

It is not problem. Storage - it low cost resources.
I have problem with CPU utilization.
When I use "notmuch new" my CPU used 100% :(
If can I move database to another server, it remove load from my slow computer.

On Thu, Jun 26, 2014 at 02:50:15PM -0300, David Bremner wrote:
> admin@softded.net writes:
> 
> > Hello,
> >
> > Xapian have remote database mode:
> > http://xapian.org/docs/remote.html
> >
> > Do you plan use it in future? I have more then 8 million letters and my
> > xapian db use 50 gb store :)
> >
> > If I can use remote database, I will move database to external server
> > from my work PC (with Pentium 4) it will be very fast! :)
> 
> One thing to keep in mind is that there are several places in the code
> where mail files are opened directly, so just having remote access to
> the xapian database is not going to be enough, unless you want to have a
> local copy of the messages, which is almost surely larger than the
> database.

Re: Remote xapian database

[David Bremner]

Roman Kravets <admin@softded.net> writes:

> It is not problem. Storage - it low cost resources.  I have problem
> with CPU utilization.  When I use "notmuch new" my CPU used 100% :( If
> can I move database to another server, it remove load from my slow
> computer.

It's hard for me to judge how many people would be interested in such a
"half distributed" solution, but if someone wants to investigate, the
pieces I can see right away are

1) figuring out the configuration changes needed to separate the
location of the database from the location of the mail files. This is
something we've been wanting to do anyway, just so people can use
different disks, so you could probably get help with that.


2) Figure out how to nicely encode the remote versus local
distinction without adding heavy dependencies to notmuch. There is some
related discussion at

        id:1340656899-5644-1-git-send-email-ethan@betacantrips.com

2) The actual remote database open would be done in
notmuch_database_open, in lib/database.cc