Update on the Emacs release schedule?

Update on the Emacs release schedule?

[Lars Magne Ingebrigtsen]

Are we getting much closer to the Emacs 24.1 release date?  I think that
Emacs is looking pretty solid, and I'm kinda wondering what we're
waiting for.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Update on the Emacs release schedule?

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Are we getting much closer to the Emacs 24.1 release date?  I think that
> Emacs is looking pretty solid, and I'm kinda wondering what we're
> waiting for.  :-)

We are getting closer indeed, but the documentation updates are not
done.  We're due for another pretest, but there are a couple more
off-list issues I'd like to wrap up first.

As for the code, there are still a number of issues that need more
attention, most prominently the mysterious memory leak(s) that may or
may not involve Gnus and/or GnuTLS and/or Mac OS X.

By the by, could someone please take a look at this Viper bug?

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9146

Re: Update on the Emacs release schedule?

[Carsten Mattner]

On Sat, Jan 7, 2012 at 8:10 AM, Chong Yidong <cyd@gnu.org> wrote:
> As for the code, there are still a number of issues that need more
> attention, most prominently the mysterious memory leak(s) that may or
> may not involve Gnus and/or GnuTLS and/or Mac OS X.

For the record, I use a --without-gnutls Emacs while also not
using Gnus, and am part of the users seeing the "leaks".

Ready to test any patch or memory statistics collector background
scripts for analysis.

Still running debug builds of Emacs just in case there's another crash.
No other crash seen so far, but I've enabled Ido just recently, and
have to use it more extensively before I can be reasonably sure it
might be solved.

Reading Tom Tromey's recent blog post, I'd say any improvements
to the allocator and/or collector will be a huge NEWS entry for 24.

fixing memory leaks before the pretest (was: Update on the Emacs release schedule?)

[Ted Zlatanov]

On Sat, 07 Jan 2012 15:10:38 +0800 Chong Yidong <cyd@gnu.org> wrote: 

CY> As for the code, there are still a number of issues that need more
CY> attention, most prominently the mysterious memory leak(s) that may or
CY> may not involve Gnus and/or GnuTLS and/or Mac OS X.

I am seeing memory growth on GNU/Linux with Gnus and GnuTLS that I don't
see otherwise without Gnus, so it's faintly possible GnuTLS is not the
determining factor.  I have gone over the gnutls.c code and don't see
where the GnuTLS glue could be leaking.  If it is, I'll need a tool like
Valgrind to help me, and last time I tried that, the reports were not
helpful to me (too much data, not enough leading back to GnuTLS).  I
spent 2 days on this last week and meant to bring it up this week,
actually (the discussion about GnuTLS on W32 sort of distracted me :)

Maybe someone who actually knows how to use Valgrind could help me or
try to find the leaks themselves?

Ted

Re: Update on the Emacs release schedule?

[Ted Zlatanov]

On Sat, 7 Jan 2012 12:02:59 +0100 Carsten Mattner <carstenmattner@googlemail.com> wrote: 

CM> Reading Tom Tromey's recent blog post, I'd say any improvements
CM> to the allocator and/or collector will be a huge NEWS entry for 24.

The post can be found at http://tromey.com/blog/?p=709 and is worth a read.

Ted

Re: fixing memory leaks before the pretest

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I am seeing memory growth on GNU/Linux with Gnus and GnuTLS that I
> don't see otherwise without Gnus, so it's faintly possible GnuTLS is
> not the determining factor.  I have gone over the gnutls.c code and
> don't see where the GnuTLS glue could be leaking.

Do you have a test case for, e.g. creating and closing a few thousand
GnuTLS connections to a localhost running apache with https and seeing
if there is any memory impact?  That would be the first thing I would
try, but I haven't had the time to look into this.

Re: fixing memory leaks before the pretest (was: Update on the Emacs release schedule?)

[Carsten Mattner]

2012/1/7 Ted Zlatanov <tzz@lifelogs.com>:
> On Sat, 07 Jan 2012 15:10:38 +0800 Chong Yidong <cyd@gnu.org> wrote:
>
> CY> As for the code, there are still a number of issues that need more
> CY> attention, most prominently the mysterious memory leak(s) that may or
> CY> may not involve Gnus and/or GnuTLS and/or Mac OS X.
>
> I am seeing memory growth on GNU/Linux with Gnus and GnuTLS that I don't
> see otherwise without Gnus, so it's faintly possible GnuTLS is not the
> determining factor.  I have gone over the gnutls.c code and don't see
> where the GnuTLS glue could be leaking.  If it is, I'll need a tool like
> Valgrind to help me, and last time I tried that, the reports were not
> helpful to me (too much data, not enough leading back to GnuTLS).  I
> spent 2 days on this last week and meant to bring it up this week,
> actually (the discussion about GnuTLS on W32 sort of distracted me :)
>
> Maybe someone who actually knows how to use Valgrind could help me or
> try to find the leaks themselves?

Tried LLVM AddressSanitizer? It's supposed to be a low-impact compile flag.
http://clang.llvm.org/docs/AddressSanitizer.html
From the documentation I'm not sure it's useful for finding leaks.

Re: Update on the Emacs release schedule?

[Carsten Mattner]

2012/1/7 Ted Zlatanov <tzz@lifelogs.com>:
> On Sat, 7 Jan 2012 12:02:59 +0100 Carsten Mattner <carstenmattner@googlemail.com> wrote:
>
> CM> Reading Tom Tromey's recent blog post, I'd say any improvements
> CM> to the allocator and/or collector will be a huge NEWS entry for 24.
>
> The post can be found at http://tromey.com/blog/?p=709 and is worth a read.

Some of the comments on tromey.com and at
http://news.ycombinator.net/item?id=3433424
plus
http://www.reddit.com/r/emacs/comments/o5yx9/
seem to not know the current state of Guile's elisp branch and even more
some seem to dismiss Guile as inferior to cmucl/sbcl while explaining the
non-adoption of Common Lisp in Emacs with a missing "good" GNU Common Lisp
implementation. I don't know, but that sounds like uninformed ranting to me.

Re: Update on the Emacs release schedule?

[Stefan Monnier]

> For the record, I use a --without-gnutls Emacs while also not
> using Gnus, and am part of the users seeing the "leaks".

AFAIK you're not seeing leaks, but only excessive memory use (and
failure to return memory to the system early enough for your taste),


        Stefan

Re: Update on the Emacs release schedule?

[Carsten Mattner]

On Sat, Jan 7, 2012 at 7:11 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> For the record, I use a --without-gnutls Emacs while also not
>> using Gnus, and am part of the users seeing the "leaks".
>
> AFAIK you're not seeing leaks, but only excessive memory use (and
> failure to return memory to the system early enough for your taste),

Maybe :). How do we define "early enough"? Hours or days?
I have let it run for hours a couple times without any memory
being released.
I don't want to criticize, but believe it's best to report what I see,
which is apparently similar to what others see.
This is all unscientific and I'd be happy to be shown I was wrong.

How about we agree on a common set of probes and statistic
gathering methods and only report the same set of measurements?
To make it comparable.

I started to wonder about emacs memory usage when on Linux it
consumed 90 to 100 megs. I could hardly believe my eyes, having just
edited a couple small source files.

Re: Update on the Emacs release schedule?

[Jan Djärv]

Hello.

7 jan 2012 kl. 19:32 skrev Carsten Mattner <carstenmattner@googlemail.com>:

> On Sat, Jan 7, 2012 at 7:11 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> For the record, I use a --without-gnutls Emacs while also not
>>> using Gnus, and am part of the users seeing the "leaks".
>> 
>> AFAIK you're not seeing leaks, but only excessive memory use (and
>> failure to return memory to the system early enough for your taste),
> 
> Maybe :). How do we define "early enough"? Hours or days?

Never. AFAIK Osx never gives back memory allocated to the system.

That is why  changing malloc implementation might be the only solution. 

    Jan D.

Re: fixing memory leaks before the pretest (was: Update on the Emacs release schedule?)

[YAMAMOTO Mitsuharu]

>>>>> On Sat, 7 Jan 2012 17:57:38 +0100, Carsten Mattner <carstenmattner@googlemail.com> said:

> 2012/1/7 Ted Zlatanov <tzz@lifelogs.com>:
>> On Sat, 07 Jan 2012 15:10:38 +0800 Chong Yidong <cyd@gnu.org>
>> wrote:
>> 
CY> As for the code, there are still a number of issues that need more
CY> attention, most prominently the mysterious memory leak(s) that may
CY> or may not involve Gnus and/or GnuTLS and/or Mac OS X.
>> 
>> I am seeing memory growth on GNU/Linux with Gnus and GnuTLS that I
>> don't see otherwise without Gnus, so it's faintly possible GnuTLS
>> is not the determining factor.  I have gone over the gnutls.c code
>> and don't see where the GnuTLS glue could be leaking.  If it is,
>> I'll need a tool like Valgrind to help me, and last time I tried
>> that, the reports were not helpful to me (too much data, not enough
>> leading back to GnuTLS).  I spent 2 days on this last week and
>> meant to bring it up this week, actually (the discussion about
>> GnuTLS on W32 sort of distracted me :)
>> 
>> Maybe someone who actually knows how to use Valgrind could help me
>> or try to find the leaks themselves?

> Tried LLVM AddressSanitizer? It's supposed to be a low-impact
> compile flag.  http://clang.llvm.org/docs/AddressSanitizer.html
> From the documentation I'm not sure it's useful for finding leaks.

I suggested a possible way to analyze heap usage on Mac OS X only
using some commands that bundled with the standard developer tools:

  http://lists.gnu.org/archive/html/emacs-devel/2011-12/msg00552.html .

Has anybody who are seeing memory issues on Mac OS X tried this?  At
least we can make sure whether there are some bogus root references or
freed memory is not returned to the system.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: Update on the Emacs release schedule?

[Stefan Monnier]

>>> For the record, I use a --without-gnutls Emacs while also not
>>> using Gnus, and am part of the users seeing the "leaks".
>> AFAIK you're not seeing leaks, but only excessive memory use (and
>> failure to return memory to the system early enough for your taste),
> Maybe :). How do we define "early enough"? Hours or days?

Never.  It's still not necessarily a leak.
A leak is when memory is lost and can never be reused again (short of
restarting Emacs).  Whereas IIUC your case is a situation where the
memory is not lost: it will never be returned to the OS but it will be
reused by Emacs itself if/when the occasion shows up.

A leak manifests itself by a memory footprint that keeps on growing
indefinitely even when the usage pattern does not justify any
such increase.


        Stefan

Re: fixing memory leaks before the pretest

[Ted Zlatanov]

On Sat, 07 Jan 2012 23:49:01 +0800 Chong Yidong <cyd@gnu.org> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I am seeing memory growth on GNU/Linux with Gnus and GnuTLS that I
>> don't see otherwise without Gnus, so it's faintly possible GnuTLS is
>> not the determining factor.  I have gone over the gnutls.c code and
>> don't see where the GnuTLS glue could be leaking.

CY> Do you have a test case for, e.g. creating and closing a few thousand
CY> GnuTLS connections to a localhost running apache with https and seeing
CY> if there is any memory impact?  That would be the first thing I would
CY> try, but I haven't had the time to look into this.

I did, to a Dovecot IMAP server:

#+begin_src lisp
(require 'gnutls)
(setq gnutls-log-level 10)
(dotimes (n 10000)
  (setq buf (open-gnutls-stream "tls" "tls-buffer" "myserver" "imaps"))
  (delete-process buf))
#+end_src

It took a while, but throughout the memory usage remained consistently
at 331 MB virtual, 53 MB resident, 15 MB shared.  Once the loop ended
and Emacs calmed down, virtual was the same but resident dropped to 39
MB and shared was 4 MB.  By comparison a new Emacs without any GnuTLS
initialization is 320 MB virtual, 45 MB resident, and 14 MB shared.

Based on that, I don't think there are obvious GnuTLS leaks in the
connection open/close sequence.  There may be leaks in the packet
handling, though.  I tried:

#+begin_src lisp
(dotimes (n 10000)
  (setq buf (open-gnutls-stream "tls" "tls-buffer" "myserver" "imaps"))
  (process-send-string buf "hello there")
  (sit-for 1)
  (message (with-current-buffer (process-buffer buf) (buffer-string)))
  (delete-process buf))
#+end_src

and did not see a difference in the memory usage either (331/53/10).
The IMAP server's greetings definitely came through, so the test was
exercising the connection.  This does not necessarily mean there are no
leaks, but if they exist, they are probably not large or don't happen
with the first few packets exchanged.

I can look at this further... let me know what tests I should do.  I'd
like to prove there are leaks at all.

Thanks
Ted