* [PATCH RFC?] Compactification support @ 2012-08-20 15:31 Ben Gamari 2012-08-20 15:31 ` [PATCH 1/3] Add notmuch_database_close_compact Ben Gamari ` (3 more replies) 0 siblings, 4 replies; 12+ messages in thread From: Ben Gamari @ 2012-08-20 15:31 UTC (permalink / raw) To: notmuch Here is a small patchset adding support for database compaction when supported by Xapian. Here I add a function which both compacts and closes the database. This is due to it being not entirely clear whether one is allowed to compact the database while it is being held open for read/write. Since it is nice to be able to hold the read/write lock while compacting, I close the database only after the compaction has finished which Xapian seems to be fine with. That being said, the database is moved after compaction, so it seems appropriate to lump compaction in with closing the database. Lastly, I wasn't entirely sure whether/how Xapian's status messages should be presented to the user. At the moment I simply spit them out on stderr. Perhaps I should just pass a callback to notmuch_database_close_compact and leave this to the caller? Cheers, - Ben ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/3] Add notmuch_database_close_compact 2012-08-20 15:31 [PATCH RFC?] Compactification support Ben Gamari @ 2012-08-20 15:31 ` Ben Gamari 2012-08-21 7:35 ` Tomi Ollila 2012-08-20 15:31 ` [PATCH 2/3] Produce status messages during compacting Ben Gamari ` (2 subsequent siblings) 3 siblings, 1 reply; 12+ messages in thread From: Ben Gamari @ 2012-08-20 15:31 UTC (permalink / raw) To: notmuch --- configure | 25 ++++++++++++++++++++++++- lib/database.cc | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ lib/notmuch.h | 14 ++++++++++++++ 3 files changed, 92 insertions(+), 1 deletion(-) diff --git a/configure b/configure index dc0dba4..370fedd 100755 --- a/configure +++ b/configure @@ -270,7 +270,8 @@ printf "Checking for Xapian development files... " have_xapian=0 for xapian_config in ${XAPIAN_CONFIG}; do if ${xapian_config} --version > /dev/null 2>&1; then - printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //') + xapian_version=$(${xapian_config} --version | sed -e 's/.* //') + printf "Yes (%s).\n" ${xapian_version} have_xapian=1 xapian_cxxflags=$(${xapian_config} --cxxflags) xapian_ldflags=$(${xapian_config} --libs) @@ -282,6 +283,24 @@ if [ ${have_xapian} = "0" ]; then errors=$((errors + 1)) fi +have_xapian_compact=0 +if [ ${have_xapian} = "1" ]; then + printf "Checking for Xapian compact support... " + IFS='.' + old_args=$@ + set -- ${xapian_version} + xapian_major_version=$1 + xapian_minor_version=$2 + xapian_subminor_version=$3 + set -- ${old_args} + if [ "${xapian_major_version}" -gt 1 ] || [ ${xapian_minor_version} -gt 2 ] || [ ${xapian_subminor_version} -ge 6 ]; then + printf "Yes.\n" + have_xapian_compact=1 + else + printf "No.\n" + fi +fi + printf "Checking for GMime development files... " have_gmime=0 IFS=';' @@ -675,6 +694,9 @@ LINKER_RESOLVES_LIBRARY_DEPENDENCIES = ${linker_resolves_library_dependencies} XAPIAN_CXXFLAGS = ${xapian_cxxflags} XAPIAN_LDFLAGS = ${xapian_ldflags} +# Whether compact is supported by this version of Xapian +HAVE_XAPIAN_COMPACT = ${have_xapian_compact} + # Flags needed to compile and link against GMime-2.4 GMIME_CFLAGS = ${gmime_cflags} GMIME_LDFLAGS = ${gmime_ldflags} @@ -711,6 +733,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\ \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\ \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\ + -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) \\ -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS) EOF diff --git a/lib/database.cc b/lib/database.cc index 761dc1a..6e83a61 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -781,6 +781,60 @@ notmuch_database_close (notmuch_database_t *notmuch) } void +notmuch_database_close_compact (notmuch_database_t *notmuch) +{ + void *local = talloc_new (NULL); + Xapian::Compactor compactor; + char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path; + +#if HAVE_XAPIAN_COMPACT + if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, ".notmuch"))) { + fprintf (stderr, "Out of memory\n"); + goto DONE; + } + + if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) { + fprintf (stderr, "Out of memory\n"); + goto DONE; + } + + if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", xapian_path))) { + fprintf (stderr, "Out of memory\n"); + goto DONE; + } + + if (! (old_xapian_path = talloc_asprintf (local, "%s.old", xapian_path))) { + fprintf (stderr, "Out of memory\n"); + goto DONE; + } + + try { + compactor.set_renumber(false); + compactor.add_source(xapian_path); + compactor.set_destdir(compact_xapian_path); + compactor.compact(); + + if (rename(xapian_path, old_xapian_path)) { + fprintf (stderr, "Error moving old database out of the way\n"); + goto DONE; + } + + if (rename(compact_xapian_path, xapian_path)) { + fprintf (stderr, "Error moving compacted database\n"); + goto DONE; + } + } catch (Xapian::InvalidArgumentError e) { + fprintf (stderr, "Error while compacting: %s", e.get_msg().c_str()); + } + +#endif + + notmuch_database_close(notmuch); +DONE: + talloc_free(local); +} + +void notmuch_database_destroy (notmuch_database_t *notmuch) { notmuch_database_close (notmuch); diff --git a/lib/notmuch.h b/lib/notmuch.h index 3633bed..50babfb 100644 --- a/lib/notmuch.h +++ b/lib/notmuch.h @@ -215,6 +215,20 @@ notmuch_database_open (const char *path, void notmuch_database_close (notmuch_database_t *database); +/* Close the given notmuch database and then compact it. + * + * After notmuch_database_close_compact has been called, calls to + * other functions on objects derived from this database may either + * behave as if the database had not been closed (e.g., if the + * required data has been cached) or may fail with a + * NOTMUCH_STATUS_XAPIAN_EXCEPTION. + * + * notmuch_database_close_compact can be called multiple times. Later + * calls have no effect. + */ +void +notmuch_database_close_compact (notmuch_database_t *notmuch); + /* Destroy the notmuch database, closing it if necessary and freeing * all associated resources. */ void -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] Add notmuch_database_close_compact 2012-08-20 15:31 ` [PATCH 1/3] Add notmuch_database_close_compact Ben Gamari @ 2012-08-21 7:35 ` Tomi Ollila 2012-08-21 14:49 ` Ben Gamari 0 siblings, 1 reply; 12+ messages in thread From: Tomi Ollila @ 2012-08-21 7:35 UTC (permalink / raw) To: Ben Gamari, notmuch On Mon, Aug 20 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: > --- > configure | 25 ++++++++++++++++++++++++- > lib/database.cc | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ > lib/notmuch.h | 14 ++++++++++++++ > 3 files changed, 92 insertions(+), 1 deletion(-) > > diff --git a/configure b/configure > index dc0dba4..370fedd 100755 > --- a/configure > +++ b/configure > @@ -270,7 +270,8 @@ printf "Checking for Xapian development files... " > have_xapian=0 > for xapian_config in ${XAPIAN_CONFIG}; do > if ${xapian_config} --version > /dev/null 2>&1; then > - printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //') > + xapian_version=$(${xapian_config} --version | sed -e 's/.* //') > + printf "Yes (%s).\n" ${xapian_version} > have_xapian=1 > xapian_cxxflags=$(${xapian_config} --cxxflags) > xapian_ldflags=$(${xapian_config} --libs) > @@ -282,6 +283,24 @@ if [ ${have_xapian} = "0" ]; then > errors=$((errors + 1)) > fi > > +have_xapian_compact=0 > +if [ ${have_xapian} = "1" ]; then > + printf "Checking for Xapian compact support... " > + IFS='.' > + old_args=$@ > + set -- ${xapian_version} > + xapian_major_version=$1 > + xapian_minor_version=$2 > + xapian_subminor_version=$3 > + set -- ${old_args} The part above breaks the argument vector in case there are spaces in args (I though $IFS chars, but try the script), execute: $ echo '#!/bin/bash IFS=. for x in "$@"; do echo $x; done; echo foo=$@ for x in $foo; do echo $x; done; echo set -- $foo for x in "$@"; do echo $x; done; echo ' > foo.bash $ bash foo.bash a "b c" d Also, after processing, IFS is not restored (to $DEFAULT_IFS) an alternative is to put the code in function like the following way: set_xapian_version () { xapian_major_version=$1 xapian_minor_version=$2 xapian_subminor_version=$3 } have_xapian_compact=0 if [ ${have_xapian} = "1" ]; then printf "Checking for Xapian compact support... " IFS=. set_xapian_version ${xapian_version} IFS=$DEFAULT_IFS if [ "${xapian_major_version}" -gt 1 ] || [ ${xapian_minor_version} -gt 2 ] || [ ${xapian_subminor_version} -ge 6 ]; then printf "Yes.\n" have_xapian_compact=1 else printf "No.\n" fi fi Hmm, I guess the check above is to determine whether xapian version is 1.2.6 or newer, but is there xapian version 1.1.6 or 1.0.6 or 0.3.0 or so ? I am not qualified to comment about compaction itself :) In the last patch you give copyright to Carl (which is OK). However I'd debate whether it is good practise to declare Carl as author; I'd say that is OK if he agrees to claim authorship to your potentially shi^H^H^Hperfect code ;) There are at least a few style issues below: e.g. no space between function name and opening parenthesis. Tomi > + if [ "${xapian_major_version}" -gt 1 ] || [ ${xapian_minor_version} -gt 2 ] || [ ${xapian_subminor_version} -ge 6 ]; then > + printf "Yes.\n" > + have_xapian_compact=1 > + else > + printf "No.\n" > + fi > +fi > + > printf "Checking for GMime development files... " > have_gmime=0 > IFS=';' > @@ -675,6 +694,9 @@ LINKER_RESOLVES_LIBRARY_DEPENDENCIES = ${linker_resolves_library_dependencies} > XAPIAN_CXXFLAGS = ${xapian_cxxflags} > XAPIAN_LDFLAGS = ${xapian_ldflags} > > +# Whether compact is supported by this version of Xapian > +HAVE_XAPIAN_COMPACT = ${have_xapian_compact} > + > # Flags needed to compile and link against GMime-2.4 > GMIME_CFLAGS = ${gmime_cflags} > GMIME_LDFLAGS = ${gmime_ldflags} > @@ -711,6 +733,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\ > CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\ > \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\ > \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\ > + -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) \\ > -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) > CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS) > EOF > diff --git a/lib/database.cc b/lib/database.cc > index 761dc1a..6e83a61 100644 > --- a/lib/database.cc > +++ b/lib/database.cc > @@ -781,6 +781,60 @@ notmuch_database_close (notmuch_database_t *notmuch) > } > > void > +notmuch_database_close_compact (notmuch_database_t *notmuch) > +{ > + void *local = talloc_new (NULL); > + Xapian::Compactor compactor; > + char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path; > + > +#if HAVE_XAPIAN_COMPACT > + if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, ".notmuch"))) { > + fprintf (stderr, "Out of memory\n"); > + goto DONE; > + } > + > + if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) { > + fprintf (stderr, "Out of memory\n"); > + goto DONE; > + } > + > + if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", xapian_path))) { > + fprintf (stderr, "Out of memory\n"); > + goto DONE; > + } > + > + if (! (old_xapian_path = talloc_asprintf (local, "%s.old", xapian_path))) { > + fprintf (stderr, "Out of memory\n"); > + goto DONE; > + } > + > + try { > + compactor.set_renumber(false); > + compactor.add_source(xapian_path); > + compactor.set_destdir(compact_xapian_path); > + compactor.compact(); > + > + if (rename(xapian_path, old_xapian_path)) { > + fprintf (stderr, "Error moving old database out of the way\n"); > + goto DONE; > + } > + > + if (rename(compact_xapian_path, xapian_path)) { > + fprintf (stderr, "Error moving compacted database\n"); > + goto DONE; > + } > + } catch (Xapian::InvalidArgumentError e) { > + fprintf (stderr, "Error while compacting: %s", e.get_msg().c_str()); > + } > + > +#endif > + > + notmuch_database_close(notmuch); > +DONE: > + talloc_free(local); > +} > + > +void > notmuch_database_destroy (notmuch_database_t *notmuch) > { > notmuch_database_close (notmuch); > diff --git a/lib/notmuch.h b/lib/notmuch.h > index 3633bed..50babfb 100644 > --- a/lib/notmuch.h > +++ b/lib/notmuch.h > @@ -215,6 +215,20 @@ notmuch_database_open (const char *path, > void > notmuch_database_close (notmuch_database_t *database); > > +/* Close the given notmuch database and then compact it. > + * > + * After notmuch_database_close_compact has been called, calls to > + * other functions on objects derived from this database may either > + * behave as if the database had not been closed (e.g., if the > + * required data has been cached) or may fail with a > + * NOTMUCH_STATUS_XAPIAN_EXCEPTION. > + * > + * notmuch_database_close_compact can be called multiple times. Later > + * calls have no effect. > + */ > +void > +notmuch_database_close_compact (notmuch_database_t *notmuch); > + > /* Destroy the notmuch database, closing it if necessary and freeing > * all associated resources. */ > void > -- > 1.7.9.5 > > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] Add notmuch_database_close_compact 2012-08-21 7:35 ` Tomi Ollila @ 2012-08-21 14:49 ` Ben Gamari 2012-08-22 6:07 ` Tomi Ollila 0 siblings, 1 reply; 12+ messages in thread From: Ben Gamari @ 2012-08-21 14:49 UTC (permalink / raw) To: Tomi Ollila, notmuch Tomi Ollila <tomi.ollila@iki.fi> writes: > On Mon, Aug 20 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: > >> --- >> configure | 25 ++++++++++++++++++++++++- >> lib/database.cc | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> lib/notmuch.h | 14 ++++++++++++++ >> 3 files changed, 92 insertions(+), 1 deletion(-) >> >> diff --git a/configure b/configure >> index dc0dba4..370fedd 100755 >> --- a/configure >> +++ b/configure >> @@ -270,7 +270,8 @@ printf "Checking for Xapian development files... " >> have_xapian=0 >> for xapian_config in ${XAPIAN_CONFIG}; do >> if ${xapian_config} --version > /dev/null 2>&1; then >> - printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //') >> + xapian_version=$(${xapian_config} --version | sed -e 's/.* //') >> + printf "Yes (%s).\n" ${xapian_version} >> have_xapian=1 >> xapian_cxxflags=$(${xapian_config} --cxxflags) >> xapian_ldflags=$(${xapian_config} --libs) >> @@ -282,6 +283,24 @@ if [ ${have_xapian} = "0" ]; then >> errors=$((errors + 1)) >> fi >> >> +have_xapian_compact=0 >> +if [ ${have_xapian} = "1" ]; then >> + printf "Checking for Xapian compact support... " >> + IFS='.' >> + old_args=$@ >> + set -- ${xapian_version} >> + xapian_major_version=$1 >> + xapian_minor_version=$2 >> + xapian_subminor_version=$3 >> + set -- ${old_args} > > The part above breaks the argument vector in case there are spaces in > args (I though $IFS chars, but try the script), execute: > Hmmm, I suppose so. > $ echo '#!/bin/bash > IFS=. > for x in "$@"; do echo $x; done; echo > foo=$@ > for x in $foo; do echo $x; done; echo > set -- $foo > for x in "$@"; do echo $x; done; echo > ' > foo.bash > > $ bash foo.bash a "b c" d > > Also, after processing, IFS is not restored (to $DEFAULT_IFS) > I assumed this would be alright since IFS is set in the next > an alternative is to put the code in function like the following > way: > Sounds good to me. > Hmm, I guess the check above is to determine whether xapian version is > 1.2.6 or newer, but is there xapian version 1.1.6 or 1.0.6 or 0.3.0 or so ? > Eh? 1.2.6 is the first Xapian release to have Compactor exposed in the public API. > I am not qualified to comment about compaction itself :) > Nor am I really. I just noticed that this functionality was blocking on library support which is now in place. It seemed that a pretty straightforward thing to implement and it hasn't broken my index yet. > In the last patch you give copyright to Carl (which is OK). However I'd > debate whether it is good practise to declare Carl as author; I'd say that > is OK if he agrees to claim authorship to your potentially shi^H^H^Hperfect > code ;) > Yikes. That certainly wasn't intentional. I'll fix this in the next iteration. > There are at least a few style issues below: e.g. no space between function > name and opening parenthesis. > Duly noted. Thanks! Cheers, - Ben ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] Add notmuch_database_close_compact 2012-08-21 14:49 ` Ben Gamari @ 2012-08-22 6:07 ` Tomi Ollila 2012-08-23 5:56 ` Kim Minh Kaplan 0 siblings, 1 reply; 12+ messages in thread From: Tomi Ollila @ 2012-08-22 6:07 UTC (permalink / raw) To: Ben Gamari, notmuch On Tue, Aug 21 2012, Ben Gamari wrote: > Tomi Ollila <tomi.ollila@iki.fi> writes: > >> On Mon, Aug 20 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: >> > >> Hmm, I guess the check above is to determine whether xapian version is >> 1.2.6 or newer, but is there xapian version 1.1.6 or 1.0.6 or 0.3.0 or so ? >> > Eh? 1.2.6 is the first Xapian release to have Compactor exposed in the > public API. The comparison code >>> if [ "${xapian_major_version}" -gt 1 ] || >>> [ ${xapian_minor_version} -gt 2 ] || >>> [ ${xapian_subminor_version} -ge 6 ]; then would match e.g. 1.1.6, 1.0.6 and 0.3.0 Presuming that those variables are always numeric the comparison could be: if [ ${xapian_major_version} -gt 1 ] || [ ${xapian_major_version} -eq 1 -a ${xapian_minor_version} -gt 2 ] || [ ${xapian_major_version} -eq 1 -a ${xapian_minor_version} -eq 2 -a \ ${xapian_subminor_version} -ge 6 ]; then (I could not figure out anything shorter and/or cleaner) > > Thanks! > > Cheers, > > - Ben Tomi ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] Add notmuch_database_close_compact 2012-08-22 6:07 ` Tomi Ollila @ 2012-08-23 5:56 ` Kim Minh Kaplan 2012-08-23 6:26 ` Tomi Ollila 0 siblings, 1 reply; 12+ messages in thread From: Kim Minh Kaplan @ 2012-08-23 5:56 UTC (permalink / raw) To: Tomi Ollila; +Cc: notmuch Tomi Ollila : > On Tue, Aug 21 2012, Ben Gamari wrote: > >> Eh? 1.2.6 is the first Xapian release to have Compactor exposed in the >> public API. > > Presuming that those variables are always numeric the comparison could be: > > if [ ${xapian_major_version} -gt 1 ] || > [ ${xapian_major_version} -eq 1 -a ${xapian_minor_version} -gt 2 ] || > [ ${xapian_major_version} -eq 1 -a ${xapian_minor_version} -eq 2 -a \ > ${xapian_subminor_version} -ge 6 ]; then > > (I could not figure out anything shorter and/or cleaner) Try: case "${xapian_version}" in 0.*|1.[01].*|1.2.[0-5]) ;; *) ... ;; esac Kim Minh. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] Add notmuch_database_close_compact 2012-08-23 5:56 ` Kim Minh Kaplan @ 2012-08-23 6:26 ` Tomi Ollila 0 siblings, 0 replies; 12+ messages in thread From: Tomi Ollila @ 2012-08-23 6:26 UTC (permalink / raw) To: Kim Minh Kaplan; +Cc: notmuch On Thu, Aug 23 2012, Kim Minh Kaplan <kimminh.kaplan+nomuch@afnic.fr> wrote: > Tomi Ollila : > >> On Tue, Aug 21 2012, Ben Gamari wrote: >> >>> Eh? 1.2.6 is the first Xapian release to have Compactor exposed in the >>> public API. >> >> Presuming that those variables are always numeric the comparison could be: >> >> if [ ${xapian_major_version} -gt 1 ] || >> [ ${xapian_major_version} -eq 1 -a ${xapian_minor_version} -gt 2 ] || >> [ ${xapian_major_version} -eq 1 -a ${xapian_minor_version} -eq 2 -a \ >> ${xapian_subminor_version} -ge 6 ]; then >> >> (I could not figure out anything shorter and/or cleaner) > > Try: > > case "${xapian_version}" in > 0.*|1.[01].*|1.2.[0-5]) ;; > *) ... ;; > esac That sure is shorter -- and splitting xapian_version is not required... .. also that would take care the (improbable?) case that `${xapian_config} -- version` outputs something else than #.#.# in the future. On the other hand, the above doesn't catch junk, so maybe: case ${xapian_version} in 0.*|1.[01].*|1.2.[0-5]) handle no case ;; [1-9]*.[0-9]*.[0-9]*) handle yes case -- approximated test ;; *) failure ;; esac (and we (approximately) expect #.#.#) In any case, excellent idea ! > Kim Minh. Thanks, Tomi ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 2/3] Produce status messages during compacting 2012-08-20 15:31 [PATCH RFC?] Compactification support Ben Gamari 2012-08-20 15:31 ` [PATCH 1/3] Add notmuch_database_close_compact Ben Gamari @ 2012-08-20 15:31 ` Ben Gamari 2012-08-20 15:31 ` [PATCH 3/3] Add notmuch compact command Ben Gamari 2012-08-21 8:07 ` [PATCH RFC?] Compactification support Jani Nikula 3 siblings, 0 replies; 12+ messages in thread From: Ben Gamari @ 2012-08-20 15:31 UTC (permalink / raw) To: notmuch --- lib/database.cc | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/lib/database.cc b/lib/database.cc index 6e83a61..49aa36d 100644 --- a/lib/database.cc +++ b/lib/database.cc @@ -780,11 +780,24 @@ notmuch_database_close (notmuch_database_t *notmuch) notmuch->value_range_processor = NULL; } +class NotmuchCompactor : public Xapian::Compactor +{ +public: + virtual void + set_status (const std::string &table, const std::string &status) + { + if (status.length() == 0) + fprintf (stderr, "compacting table %s:\n", table.c_str()); + else + fprintf (stderr, " %s\n", status.c_str()); + } +}; + void notmuch_database_close_compact (notmuch_database_t *notmuch) { void *local = talloc_new (NULL); - Xapian::Compactor compactor; + NotmuchCompactor compactor; char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path; #if HAVE_XAPIAN_COMPACT -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/3] Add notmuch compact command 2012-08-20 15:31 [PATCH RFC?] Compactification support Ben Gamari 2012-08-20 15:31 ` [PATCH 1/3] Add notmuch_database_close_compact Ben Gamari 2012-08-20 15:31 ` [PATCH 2/3] Produce status messages during compacting Ben Gamari @ 2012-08-20 15:31 ` Ben Gamari 2012-08-21 8:07 ` [PATCH RFC?] Compactification support Jani Nikula 3 siblings, 0 replies; 12+ messages in thread From: Ben Gamari @ 2012-08-20 15:31 UTC (permalink / raw) To: notmuch --- Makefile.local | 1 + notmuch-client.h | 3 +++ notmuch-compact.c | 43 +++++++++++++++++++++++++++++++++++++++++++ notmuch.c | 3 +++ 4 files changed, 50 insertions(+) create mode 100644 notmuch-compact.c diff --git a/Makefile.local b/Makefile.local index de984ab..848605d 100644 --- a/Makefile.local +++ b/Makefile.local @@ -278,6 +278,7 @@ notmuch_client_srcs = \ gmime-filter-headers.c \ hooks.c \ notmuch.c \ + notmuch-compact.c \ notmuch-config.c \ notmuch-count.c \ notmuch-dump.c \ diff --git a/notmuch-client.h b/notmuch-client.h index ae9344b..a6c624b 100644 --- a/notmuch-client.h +++ b/notmuch-client.h @@ -157,6 +157,9 @@ int notmuch_cat_command (void *ctx, int argc, char *argv[]); int +notmuch_compact_command (void *ctx, int argc, char *argv[]); + +int notmuch_config_command (void *ctx, int argc, char *argv[]); const char * diff --git a/notmuch-compact.c b/notmuch-compact.c new file mode 100644 index 0000000..6deb2ec --- /dev/null +++ b/notmuch-compact.c @@ -0,0 +1,43 @@ +/* notmuch - Not much of an email program, (just index and search) + * + * Copyright © 2009 Carl Worth + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see http://www.gnu.org/licenses/ . + * + * Author: Carl Worth <cworth@cworth.org> + */ + +#include "notmuch-client.h" + +int +notmuch_compact_command (void *ctx, unused (int argc), unused (char *argv[])) +{ + notmuch_config_t *config; + notmuch_database_t *notmuch; + + config = notmuch_config_open (ctx, NULL, NULL); + if (config == NULL) + return 1; + + if (notmuch_database_open (notmuch_config_get_database_path (config), + NOTMUCH_DATABASE_MODE_READ_WRITE, ¬much)) + return 1; + + printf ("Compacting database... "); + notmuch_database_close_compact (notmuch); + printf ("Done.\n"); + notmuch_database_destroy (notmuch); + + return 0; +} diff --git a/notmuch.c b/notmuch.c index 477a09c..7b6c5ae 100644 --- a/notmuch.c +++ b/notmuch.c @@ -74,6 +74,9 @@ static command_t commands[] = { { "restore", notmuch_restore_command, "[--accumulate] [<filename>]", "Restore the tags from the given dump file (see 'dump')." }, + { "compact", notmuch_compact_command, + NULL, + "Compacts the database." }, { "config", notmuch_config_command, "[get|set] <section>.<item> [value ...]", "Get or set settings in the notmuch configuration file." }, -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH RFC?] Compactification support 2012-08-20 15:31 [PATCH RFC?] Compactification support Ben Gamari ` (2 preceding siblings ...) 2012-08-20 15:31 ` [PATCH 3/3] Add notmuch compact command Ben Gamari @ 2012-08-21 8:07 ` Jani Nikula 2012-08-21 14:48 ` Ben Gamari 3 siblings, 1 reply; 12+ messages in thread From: Jani Nikula @ 2012-08-21 8:07 UTC (permalink / raw) To: Ben Gamari, notmuch On Mon, 20 Aug 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: > Here is a small patchset adding support for database compaction when > supported by Xapian. Here I add a function which both compacts and > closes the database. What are the benefits of this compared to xapian-compact [1]? Jani. [1] http://xapian.org/docs/admin_notes.html#compacting-a-database ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC?] Compactification support 2012-08-21 8:07 ` [PATCH RFC?] Compactification support Jani Nikula @ 2012-08-21 14:48 ` Ben Gamari 2012-08-21 15:25 ` Tomi Ollila 0 siblings, 1 reply; 12+ messages in thread From: Ben Gamari @ 2012-08-21 14:48 UTC (permalink / raw) To: Jani Nikula, notmuch Jani Nikula <jani@nikula.org> writes: > On Mon, 20 Aug 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: >> Here is a small patchset adding support for database compaction when >> supported by Xapian. Here I add a function which both compacts and >> closes the database. > > What are the benefits of this compared to xapian-compact [1]? > I think it's a bit of a shame you are currently forced to interact with Xapian directly (an implementation detail) to compact the database. This set seeks to abstract this out. This is especially important since it's not too difficult to get the invocation of xapian-compact wrong (e.g. forget --no-renumber). Also, in the past Carl (id:"87ocl1lut1.fsf@yoom.home.cworth.org") has mentioned it would be nice if compact would be run after the initial run of "notmuch new". This patchset sets the groundwork to enable this. Cheers, - Ben ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC?] Compactification support 2012-08-21 14:48 ` Ben Gamari @ 2012-08-21 15:25 ` Tomi Ollila 0 siblings, 0 replies; 12+ messages in thread From: Tomi Ollila @ 2012-08-21 15:25 UTC (permalink / raw) To: Ben Gamari, Jani Nikula, notmuch On Tue, Aug 21 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: > Jani Nikula <jani@nikula.org> writes: > >> On Mon, 20 Aug 2012, Ben Gamari <bgamari.foss@gmail.com> wrote: >>> Here is a small patchset adding support for database compaction when >>> supported by Xapian. Here I add a function which both compacts and >>> closes the database. >> >> What are the benefits of this compared to xapian-compact [1]? >> > I think it's a bit of a shame you are currently forced to interact with > Xapian directly (an implementation detail) to compact the database. This > set seeks to abstract this out. This is especially important since it's > not too difficult to get the invocation of xapian-compact wrong > (e.g. forget --no-renumber). Better reason would be that xapian-compact(1) may not be available although the library is -- if the command is available it is easy to execve(2) it with correct arguments... (?) Tomi > Also, in the past Carl (id:"87ocl1lut1.fsf@yoom.home.cworth.org") has > mentioned it would be nice if compact would be run after the initial run > of "notmuch new". This patchset sets the groundwork to enable this. > > Cheers, > > - Ben ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2012-08-23 6:26 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-08-20 15:31 [PATCH RFC?] Compactification support Ben Gamari 2012-08-20 15:31 ` [PATCH 1/3] Add notmuch_database_close_compact Ben Gamari 2012-08-21 7:35 ` Tomi Ollila 2012-08-21 14:49 ` Ben Gamari 2012-08-22 6:07 ` Tomi Ollila 2012-08-23 5:56 ` Kim Minh Kaplan 2012-08-23 6:26 ` Tomi Ollila 2012-08-20 15:31 ` [PATCH 2/3] Produce status messages during compacting Ben Gamari 2012-08-20 15:31 ` [PATCH 3/3] Add notmuch compact command Ben Gamari 2012-08-21 8:07 ` [PATCH RFC?] Compactification support Jani Nikula 2012-08-21 14:48 ` Ben Gamari 2012-08-21 15:25 ` Tomi Ollila
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).