* [PATCH] fix file descriptor leak with compressed email
@ 2019-09-22 18:48 James Troup
2019-09-22 22:43 ` fix file descriptor leak David Bremner
0 siblings, 1 reply; 5+ messages in thread
From: James Troup @ 2019-09-22 18:48 UTC (permalink / raw
To: notmuch
Hi,
I'm trying notmuch for the first time using notmuch 0.29.1-2ubuntu1
backported to Ubuntu 19.04. My email is stored in Gnus nnml format
and is approximately 410K mails with 60% compressed. While trying to
run 'notmuch new' I very quickly ran into issues with 'Too many open
files', even after adjusting the maximum number of open file
descriptor up to 65K.
The patch below solved the problem for me.
diff -ur orig/notmuch-0.29.1/util/gmime-extra.c notmuch-0.29.1/util/gmime-extra.c
--- orig/notmuch-0.29.1/util/gmime-extra.c 2019-06-11 23:16:48.000000000 +0000
+++ notmuch-0.29.1/util/gmime-extra.c 2019-09-22 18:25:28.629546076 +0000
@@ -28,6 +28,8 @@
/* ignore filter id */
(void)g_mime_stream_filter_add ((GMimeStreamFilter *)gzstream, gzfilter);
+ g_object_unref(gzfilter);
+ g_object_unref(file_stream);
return gzstream;
} else {
return file_stream;
--
James
^ permalink raw reply [flat|nested] 5+ messages in thread
* fix file descriptor leak
2019-09-22 18:48 [PATCH] fix file descriptor leak with compressed email James Troup
@ 2019-09-22 22:43 ` David Bremner
2019-09-22 22:44 ` [PATCH 1/2] test: known broken test file descriptor leak in gzip file open David Bremner
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: David Bremner @ 2019-09-22 22:43 UTC (permalink / raw
To: James Troup, notmuch
I'm think this probably deserves a point release. Please test/review.
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] test: known broken test file descriptor leak in gzip file open
2019-09-22 22:43 ` fix file descriptor leak David Bremner
@ 2019-09-22 22:44 ` David Bremner
2019-09-22 22:44 ` [PATCH 2/2] util: unreference objects referenced by the returned stream obj David Bremner
2019-10-13 12:31 ` fix file descriptor leak David Bremner
2 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2019-09-22 22:44 UTC (permalink / raw
To: James Troup, notmuch
James Troup reported this bug in id:87pnjsf9q5.fsf@canonical.com
---
test/T750-gzip.sh | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/test/T750-gzip.sh b/test/T750-gzip.sh
index 5b678fa1..f4e0c42a 100755
--- a/test/T750-gzip.sh
+++ b/test/T750-gzip.sh
@@ -167,4 +167,12 @@ This is just a test message (#7)
EOF
test_expect_equal_file EXPECTED OUTPUT
+# there are more than 200 messages in this corpus
+add_email_corpus lkml
+test_begin_subtest "new doesn't run out of file descriptors with many gzipped files"
+test_subtest_known_broken
+ulimit -n 200
+gzip --recursive ${MAIL_DIR}
+test_expect_success "notmuch new"
+
test_done
--
2.23.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] util: unreference objects referenced by the returned stream obj
2019-09-22 22:43 ` fix file descriptor leak David Bremner
2019-09-22 22:44 ` [PATCH 1/2] test: known broken test file descriptor leak in gzip file open David Bremner
@ 2019-09-22 22:44 ` David Bremner
2019-10-13 12:31 ` fix file descriptor leak David Bremner
2 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2019-09-22 22:44 UTC (permalink / raw
To: James Troup, notmuch
We want freeing the returned stream to also free these underlying
objects. Compare tests/test-filters.c in the gmime 3.2.x source, which
uses this same idiom.
Thanks to James Troup for the report and the fix.
---
test/T750-gzip.sh | 1 -
| 2 ++
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/test/T750-gzip.sh b/test/T750-gzip.sh
index f4e0c42a..fac41d39 100755
--- a/test/T750-gzip.sh
+++ b/test/T750-gzip.sh
@@ -170,7 +170,6 @@ test_expect_equal_file EXPECTED OUTPUT
# there are more than 200 messages in this corpus
add_email_corpus lkml
test_begin_subtest "new doesn't run out of file descriptors with many gzipped files"
-test_subtest_known_broken
ulimit -n 200
gzip --recursive ${MAIL_DIR}
test_expect_success "notmuch new"
--git a/util/gmime-extra.c b/util/gmime-extra.c
index 7562d906..4a591cae 100644
--- a/util/gmime-extra.c
+++ b/util/gmime-extra.c
@@ -28,6 +28,8 @@ _gzfile_maybe_filter (GMimeStream *file_stream) {
/* ignore filter id */
(void)g_mime_stream_filter_add ((GMimeStreamFilter *)gzstream, gzfilter);
+ g_object_unref(gzfilter);
+ g_object_unref(file_stream);
return gzstream;
} else {
return file_stream;
--
2.23.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: fix file descriptor leak
2019-09-22 22:43 ` fix file descriptor leak David Bremner
2019-09-22 22:44 ` [PATCH 1/2] test: known broken test file descriptor leak in gzip file open David Bremner
2019-09-22 22:44 ` [PATCH 2/2] util: unreference objects referenced by the returned stream obj David Bremner
@ 2019-10-13 12:31 ` David Bremner
2 siblings, 0 replies; 5+ messages in thread
From: David Bremner @ 2019-10-13 12:31 UTC (permalink / raw
To: notmuch
David Bremner <david@tethera.net> writes:
> I'm think this probably deserves a point release. Please test/review.
I've applied these changes to release and master. I'll hold off a few
days before doing a point release, please let me know if anything
catches fire.
d
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-10-13 12:33 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-09-22 18:48 [PATCH] fix file descriptor leak with compressed email James Troup
2019-09-22 22:43 ` fix file descriptor leak David Bremner
2019-09-22 22:44 ` [PATCH 1/2] test: known broken test file descriptor leak in gzip file open David Bremner
2019-09-22 22:44 ` [PATCH 2/2] util: unreference objects referenced by the returned stream obj David Bremner
2019-10-13 12:31 ` fix file descriptor leak David Bremner
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).