From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: ludo@gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Newsgroups: gmane.lisp.guile.devel Subject: Reducing iconv-induced memory usage Date: Tue, 26 Apr 2011 23:10:32 +0200 Message-ID: <87r58oyb07.fsf@gnu.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: dough.gmane.org 1303852256 26395 80.91.229.12 (26 Apr 2011 21:10:56 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Tue, 26 Apr 2011 21:10:56 +0000 (UTC) To: guile-devel@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Tue Apr 26 23:10:50 2011 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1QEpX0-0006li-Dh for guile-devel@m.gmane.org; Tue, 26 Apr 2011 23:10:50 +0200 Original-Received: from localhost ([::1]:54433 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QEpWz-0005vJ-TY for guile-devel@m.gmane.org; Tue, 26 Apr 2011 17:10:49 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:50989) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QEpWw-0005vB-1B for guile-devel@gnu.org; Tue, 26 Apr 2011 17:10:46 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QEpWv-0002nv-1P for guile-devel@gnu.org; Tue, 26 Apr 2011 17:10:45 -0400 Original-Received: from lo.gmane.org ([80.91.229.12]:36236) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QEpWu-0002nq-L7 for guile-devel@gnu.org; Tue, 26 Apr 2011 17:10:44 -0400 Original-Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1QEpWt-0006j0-9Y for guile-devel@gnu.org; Tue, 26 Apr 2011 23:10:43 +0200 Original-Received: from reverse-83.fdn.fr ([80.67.176.83]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 26 Apr 2011 23:10:43 +0200 Original-Received: from ludo by reverse-83.fdn.fr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 26 Apr 2011 23:10:43 +0200 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 68 Original-X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: reverse-83.fdn.fr X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 =?iso-8859-1?Q?Flor=E9al?= an 219 de la =?iso-8859-1?Q?R=E9volution?= X-PGP-Key-ID: 0xEA52ECF4 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 83C4 F8E5 10A3 3B4C 5BEA D15D 77DD 95E2 EA52 ECF4 X-OS: x86_64-unknown-linux-gnu User-Agent: Gnus/5.110015 (No Gnus v0.15) Emacs/23.3 (gnu/linux) Cancel-Lock: sha1:094rmNkCEhGBU86AoU8Rc0d0U+c= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 80.91.229.12 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:12355 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Hello! As Andy noted in the past, iconv conversion descriptors associated with ports take up a lot of malloc’d memory, that only gets freed when finalizers are run. On GNU/Linux, a UTF-8 → UTF-8 C.D., which does nothing, mallocs 180 KiB (!), according to the program attached. So the problem is acute. So I think we should special-case UTF-8 I/O to not use iconv at all. For output, it’s easy since we already do the conversion to UTF-8 in ‘display_string’. For input, it’s a bit more work because input byte streams have to be checked for invalid sequences. I’m working on a patch but I’d like to get initial feedback and also about whether it should wait until after 2.0.1 or not. Thanks, Ludo’. --=-=-= Content-Type: text/x-csrc Content-Disposition: inline; filename=iconv.c Content-Description: the program #include #include static size_t total; static void * (*prev_hook) (size_t, const void *); static void * m (size_t s, const void *c) { __malloc_hook = prev_hook; printf ("alloc %zi\n", s); void *r = malloc (s); total += s; __malloc_hook = &m; return r; } static void my_init_hook (void) { prev_hook = __malloc_hook; __malloc_hook = &m; } void (*__malloc_initialize_hook) (void) = my_init_hook; int main (int argc, char *argv[]) { total = 0; iconv_open ("UTF-8", "UTF-8"); printf ("allocated %zi B\n", total); return 0; } --=-=-=--