PURESIZE increased (again)

PURESIZE increased (again)

[Eli Zaretskii]

Romain, could you please tell why you increased BASE_PURESIZE by
another 10,000?  I don't see any relevant discussions in emacs-devel
archives lately.

Judging by "cvs log", BASE_PURESIZE went up by 100,000 since last
October, whereas before that it kept its value for 2 years without any
changes.  Perhaps we should consider each such change more carefully.

FWIW, I've just built a w32 port, and it told me that it used up
1216152 pure bytes, so BASE_PURESIZE of 1210000 sounds _way_ too much
(since src/s/ms-w32.h defines 137500 bytes of SYSTEM_PURESIZE_EXTRA).

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Sun, 16 Apr 2006 10:07:59 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Judging by "cvs log", BASE_PURESIZE went up by 100,000 since last
> October, whereas before that it kept its value for 2 years without any
> changes.  Perhaps we should consider each such change more carefully.

For that matter, could someone (Richard?) tell what are the thumb
rules for estimating the required value of SYSTEM_PURESIZE_EXTRA
(other than by trial and error)?  Will the combined size of
system-specific *.el/*.elc files loaded by loadup.el be a good start,
or are there any additional bits to take into consideration?

(I'm asking because I suspect that the value of SYSTEM_PURESIZE_EXTRA
could be exaggerated as well, at least on Windows.)

Re: PURESIZE increased (again)

[Romain Francoise]

Eli Zaretskii <eliz@gnu.org> writes:

> Romain, could you please tell why you increased BASE_PURESIZE by
> another 10,000?

Because I built Emacs and there wasn't enough pure space available...
The log indicated 1200696 bytes used, so I increased the base size
another notch to 1210000 (on GNU/Linux, SYSTEM_PURESIZE_EXTRA is zero).
I grant you that a 10,000 bytes increase for an overflow of 696 bytes
might be too much, but since the size keeps going up it's just preparing
for future changes...

As you may or may not know, since August 2005 I've been packaging weekly
snapshots of Emacs CVS for the Debian GNU/Linux operating system;
this gives me the opportunity to build Emacs automatically with a
variety of different configuration options and compiler flags every
week, so I catch these kinds of problems pretty quickly.  I think the
overflow is caused this week by Bill's latest batch of changes to
Customize.

> Judging by "cvs log", BASE_PURESIZE went up by 100,000 since last
> October, whereas before that it kept its value for 2 years without any
> changes.

Since last October, image, international/fontset, dnd, x-dnd, mwheel,
tool-bar, emacs-lisp/syntax, font-lock, jit-lock, rfn-eshadow and fringe
have been added to the list of preloaded libraries.  And other loaded
libraries have grown as well.

-- 
Romain Francoise <romain@orebokech.com> | The sea! the sea! the open
it's a miracle -- http://orebokech.com/ | sea! The blue, the fresh, the
                                        | ever free! --Bryan W. Procter

Re: PURESIZE increased (again)

[Andreas Schwab]

Romain Francoise <romain@orebokech.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Romain, could you please tell why you increased BASE_PURESIZE by
>> another 10,000?
>
> Because I built Emacs and there wasn't enough pure space available...
> The log indicated 1200696 bytes used,

Where do you see that?  I see only 1198536 bytes (1877040 on 64bit).

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Sun, 16 Apr 2006 14:13:56 +0200
> 
> Romain Francoise <romain@orebokech.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Romain, could you please tell why you increased BASE_PURESIZE by
> >> another 10,000?
> >
> > Because I built Emacs and there wasn't enough pure space available...
> > The log indicated 1200696 bytes used,
> 
> Where do you see that?

I second the question.  How was that build configured, for what
variant of GNU/Linux, and which CPU?

> I see only 1198536 bytes (1877040 on 64bit).

And I see 1102088 in a GNU/Linux build without X.  I doubt that adding
X could bump it by 100K bytes.

I also doubt very much that the Windows build is so different from a
32-build on GNU/Linux.

So your results, Romain, look quite strange.  I think we need to
understand them before we decide to increase PURESIZE.

Re: PURESIZE increased (again)

[Reiner Steib]

On Sun, Apr 16 2006, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@suse.de>
>> Romain Francoise <romain@orebokech.com> writes:
>> > Because I built Emacs and there wasn't enough pure space available...
>> > The log indicated 1200696 bytes used,
>> 
>> Where do you see that?
>
> I second the question.  How was that build configured, for what
> variant of GNU/Linux, and which CPU?
>
>> I see only 1198536 bytes (1877040 on 64bit).
>
> And I see 1102088 in a GNU/Linux build without X.  I doubt that adding
> X could bump it by 100K bytes.

FWIW, I saw the following message when building with the current
sources (some minor local changes) today on 64bit ...

,----
| Dumping under names emacs and emacs-22.0.50.1
| emacs:0:Pure Lisp storage overflow (approx. 2155304 bytes needed)
| 5504 pure bytes used
| 
| "GNU Emacs 22.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.4.9)
|  of 2006-04-20 on bridgekeeper"
| 
| system-configuration-options
| "'--prefix=/import/xtra/emacs/HEAD' '--with-gtk' '--exec-prefix=/import/xtra/emacs/HEAD-x86_64'"
`----

... and last Tuesday on 32bit:

,----
| Dumping under names emacs and emacs-22.0.50.4
| emacs:0:Pure Lisp storage overflow (approx. 1342376 bytes needed)
| 2832 pure bytes used
| [...]
| 
| GNU Emacs 22.0.50.4 (i686-pc-linux-gnu, GTK+ Version 2.4.9) of
| 2006-04-18 on rabbit
| 
| system-configuration-options:
| "'--prefix=/import/xtra/emacs/HEAD' '--with-gtk' '--exec-prefix=/import/xtra/emacs/HEAD-i686'"
`----

With today's sources, I get "75244 pure bytes used" on 32bit (same
options).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: romain@orebokech.com, emacs-devel@gnu.org
> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Date: Thu, 20 Apr 2006 18:51:38 +0200
> 
> >> I see only 1198536 bytes (1877040 on 64bit).
> >
> > And I see 1102088 in a GNU/Linux build without X.  I doubt that adding
> > X could bump it by 100K bytes.
> 
> FWIW, I saw the following message when building with the current
> sources (some minor local changes) today on 64bit ...
> 
> ,----
> | Dumping under names emacs and emacs-22.0.50.1
> | emacs:0:Pure Lisp storage overflow (approx. 2155304 bytes needed)
> | 5504 pure bytes used

This has nothing to do with PURESIZE, it's the ridiculously low number
5504 that is the problem.  (Note that, on a 64-bit host, 2155304 is
well within the limits of BASE_PURESIZE enlarged by PURESIZE_RATIO.)

How come you only have 5504 bytes of space in the pure[] array?  Is
this during the first stage of bootstrap perhaps?  If so, please look
at the last "Dumping under names ..." message near the end of the
build, it's that one that's important.  (I imagine you already know
that.)

Re: PURESIZE increased (again)

[Reiner Steib]

On Thu, Apr 20 2006, Eli Zaretskii wrote:

>> From: Reiner Steib <reinersteib+gmane@imap.cc>
[...]
>> ,----
>> | Dumping under names emacs and emacs-22.0.50.1
>> | emacs:0:Pure Lisp storage overflow (approx. 2155304 bytes needed)
>> | 5504 pure bytes used
>
> This has nothing to do with PURESIZE, it's the ridiculously low number
> 5504 that is the problem.  (Note that, on a 64-bit host, 2155304 is
> well within the limits of BASE_PURESIZE enlarged by PURESIZE_RATIO.)
>
> How come you only have 5504 bytes of space in the pure[] array?  Is
> this during the first stage of bootstrap perhaps?  

It was in "make bootstrap".

> If so, please look at the last "Dumping under names ..." message
> near the end of the build, it's that one that's important.

Now I've built again (without a bootstrap) and I got (there is only
one "Dumping under names" in the logs):

,----[ 64 bit ]
| Loading tooltip...
| nil
| Finding pointers to doc strings...
| Finding pointers to doc strings...done
| Dumping under names emacs and emacs-22.0.50.3
| emacs:0:Pure Lisp storage overflow (approx. 2155304 bytes needed)
| 5504 pure bytes used
| ./emacs -q -batch -f list-load-path-shadows
`----

I also get the "Pure space overflow" warning from
`fancy-splash-screens-1':

,----[ <f1> v pure-space-overflow RET ]
| pure-space-overflow is a variable defined in `startup.el'.
| Its value is t
`----

,----[ <f1> v pure-bytes-used RET ]
| pure-bytes-used is a variable defined in `src/alloc.c'.
| Its value is 5504
| 
| Documentation:
| Number of bytes of sharable Lisp data allocated so far.
`----

On a 32 bit host (same source tree) I get:

,----[ 32 bit ]
| Dumping under names emacs and emacs-22.0.50.2
| emacs:0:Pure Lisp storage overflow (approx. 1343128 bytes needed)
| 3616 pure bytes used
`----

> (I imagine you already know that.)

No, I didn't know.  And I don't know why these numbers are so low on
my machines.  How can I investigate?  Will it cause problems?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Stefan Monnier]

>> (I imagine you already know that.)

> No, I didn't know.  And I don't know why these numbers are so low on
> my machines.  How can I investigate?  Will it cause problems?

AFAIK these low numbers have no meaning: if the pure size overflows, the
number printed is always some ridiculously low number, unrelated to the
actual size of the pure space (or more specifically: probably somehow
related, but not equal to).


        Stefan

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: romain@orebokech.com,  emacs-devel@gnu.org
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 20 Apr 2006 17:37:59 -0400
> 
> AFAIK these low numbers have no meaning: if the pure size overflows, the
> number printed is always some ridiculously low number, unrelated to the
> actual size of the pure space

Yes, you are right: the backup strategy zeroes out pure_bytes_used, so
what gets printed is just the portion that didn't fit into pure[].

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: romain@orebokech.com, emacs-devel@gnu.org
> Date: Thu, 20 Apr 2006 23:03:03 +0200
> 
> How can I investigate?

Run temacs under GDB, put a breakpoint on Fload, and each time it
breaks print the value of pure_bytes_used.  Then post here the
results; perhaps PURESIZE needs to be bumped up after all.

> Will it cause problems?

Not grave problems (we have plan B for such situations), but it
shouldn't happen.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Fri, 21 Apr 2006 10:48:04 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
> > From: Reiner Steib <reinersteib+gmane@imap.cc>
> > Cc: romain@orebokech.com, emacs-devel@gnu.org
> > Date: Thu, 20 Apr 2006 23:03:03 +0200
> > 
> > How can I investigate?
> 
> Run temacs under GDB, put a breakpoint on Fload, and each time it
> breaks print the value of pure_bytes_used.  Then post here the
> results; perhaps PURESIZE needs to be bumped up after all.

Here are my results (on a 32-bit MS-Windows box).  Each line shows the
loaded file (or, in some cases where something other than loading a
file caused a significant increase in pure storage use, the
description of what caused it), the used pure size _after_ that file
was loaded, and how many pure bytes were used up by loading the file.

When I compare this with the results from a week ago, when PURESIZE
was set to its current value, I see an increase of 408 bytes.

<before loadup.el>      76659  76659
"loadup.el"		76684  25
"emacs-lisp/byte-run"	79800  3116
"emacs-lisp/backquote"	81780  1980
"subr"			127984  46204
"version.el"		129952  1968
"widget"		130407  455
"custom"		147120  16713
"emacs-lisp/map-ynp"	150897  3777
"env"			153364  2467
"cus-start"		155309  1945
"international/mule"	189020  33711
"international/mule-conf.el"	196213  7193
"format"		209256  13043
"bindings"		229803  20547
"files"			311179  81376
"cus-face"		321585  10406
"faces"			363714  42129
"loaddefs.el"		495541  131827
"simple"		570743  75202
"help"			592364  21621
"international/mule-cmds"	635436  43072
"case-table"		638260  2824
"international/utf-8"	649290  11030
"international/utf-16"	670822  21532
"international/characters"	672790  1968
"international/latin-1"	672790  0
"international/latin-2"	672816  26
"international/latin-3"	672840  24
"international/latin-4"	672864  24
"international/latin-5"	672888  24
"international/latin-8"	672912  24
"international/latin-9"	672936  24
"language/chinese"	679235  6299
"language/cyrillic"	686488  7253
"language/indian"	689400  2912
"language/devanagari"	689672  272
"language/malayalam"	689816  144
"language/tamil"	689944  128
"language/kannada"	690196  252
"language/english"	690310  114
"language/ethiopic"	691479  1169
"language/european"	705081  13602
"language/czech"	705294  213
"language/slovak"	705503  209
"language/romanian"	705729  226
"language/greek"	706406  677
"language/hebrew"	707543  1137
"language/japanese"	710441  2898
"language/korean"	712023  1582
"language/lao"		712528  505
"language/thai"		713741  1213
"language/tibetan"	741920  28179
"language/vietnamese"	745414  3494
"language/misc-lang"	745514  100
"language/utf-8-lang"	745931  417
"language/georgian"	746145  214
"international/ucs-tables"	751691  5546
"indent"		758344  6653
"window"		770722  12378
"frame"			788783  18061
"term/tty-colors"	792672  3889
"font-core"		796850  4178
"facemenu"		806421  9571
"emacs-lisp/syntax"	809140  2719
"font-lock"		838384  29244
"jit-lock"		845168  6784
"mouse"			872031  26863
"scroll-bar"		878109  6078
"select"		884743  6634
"emacs-lisp/timer"	893244  8501
"isearch"		924600  31356
"rfn-eshadow"		929220  4620
"menu-bar"		952370  23150
"paths.el"		955068  2698
"startup"		988802  33734
"emacs-lisp/lisp"	996917  8115
"textmodes/page"	998229  1312
"register"		1003850  5621
"textmodes/paragraphs"	1009099  5249
"emacs-lisp/lisp-mode"	1023012  13913
"textmodes/text-mode"	1025226  2214
"textmodes/fill"	1041697  16471
"replace"		1065984  24287
"abbrev"		1071468  5484
"buff-menu"		1082602  11134
"fringe"		1086492  3890
"image"			1092752  6260
"international/fontset"	1105035  12283
"dnd"			1107598  2563
"mwheel"		1112327  4729
"tool-bar"		1116677  4350
"ls-lisp"		1125033  8356
"disp-table"		1128403  3370
"dos-w32"		1133480  5077
"w32-vars"		1134365  885
"w32-fns"		1140600  6235
"emacs-lisp/float-sup"	1140952  352
"vc-hooks"		1154753  13801
"jka-cmpr-hook"		1160230  5477
"ediff-hook"		1160359  129
"tooltip"		1165552  5193
<purecopy load-history>	1216528  50976
"lisp/subdirs.el"	1216528  0
"code-pages"		1216528  0
"encoded-kb"		1216528  0

Re: PURESIZE increased (again)

[Reiner Steib]

On Fri, Apr 21 2006, Eli Zaretskii wrote:

>> From: Reiner Steib <reinersteib+gmane@imap.cc>
>> Cc: romain@orebokech.com, emacs-devel@gnu.org
>> Date: Thu, 20 Apr 2006 23:03:03 +0200
>> 
>> How can I investigate?
>
> Run temacs under GDB, put a breakpoint on Fload, and each time it
> breaks print the value of pure_bytes_used.  Then post here the
> results; perhaps PURESIZE needs to be bumped up after all.

Sorry for the late answer.  I still get an overflow on 64 bit with
today's sources (I can do the same on 32 bit if it's useful):

--8<---------------cut here---------------start------------->8---
$ cd [...]/emacs/cvs-HEAD/emacs/src$ grep define.BASE_PURESIZE puresize.h
#define BASE_PURESIZE (1205000 + SYSTEM_PURESIZE_EXTRA + SITELOAD_PURESIZE_EXTRA)
$ cd [...]/emacs/cvs-HEAD/x86_64/src/; gdb ./temacs
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
[...]
Breakpoint 1 at 0x556da0: file [...]/emacs/src/sysdep.c, line 1373.
(gdb) break lread.c:717
Breakpoint 2 at 0x5e79dc: file [...]/emacs/src/lread.c, line 717.
(gdb) run -batch -l loadup dump
Starting program: [...]/emacs/cvs-HEAD/x86_64/src/temacs -batch -l loadup dump
[Thread debugging using libthread_db enabled]
[New Thread 182940272320 (LWP 30412)]
[Switching to Thread 182940272320 (LWP 30412)]

Breakpoint 2, Fload (file=11077635, noerror=10658513, nomessage=10658513, nosuffix=10658513, must_suffix=10658513)
    at [...]/emacs/src/lread.c:718
718       register int fd = -1;
(gdb) print pure_bytes_used

[ `print pure_bytes_used' and `continue';
  only  lines  matching `Loading\|^\$' shown ]

$1 = 108747
Loading loadup.el (source)...
$1 = 108747
Loading loadup.el (source)...
$2 = 108788
Loading emacs-lisp/byte-run...
$3 = 114536
Loading emacs-lisp/backquote...
$4 = 118000
Loading subr...
$5 = 205328
Loading version.el (source)...
$6 = 208625
Loading widget...
$7 = 209471
Loading custom...
$8 = 241496
Loading emacs-lisp/map-ynp...
$9 = 248473
Loading env...
$10 = 253068
Loading cus-start...
$11 = 255909
Loading international/mule...
$12 = 317596
Loading international/mule-conf.el (source)...
$13 = 327085
Loading format...
$14 = 350864
Loading bindings...
$15 = 388019
Loading files...
$16 = 537811
Loading cus-face...
$17 = 556913
Loading faces...
$18 = 638354
Loading loaddefs.el (source)...
$19 = 831317
Loading simple...
$20 = 973359
Loading help...
$21 = 1011360
Loading international/mule-cmds...
$22 = 1089544
Loading case-table...
$23 = 1095208
Loading international/utf-8...
$24 = 1120978
Loading international/utf-16...
$25 = 1173558
Loading international/characters...
$26 = 1176406
Loading international/latin-1 (source)...
$27 = 1176406
Loading international/latin-2 (source)...
$28 = 1176448
Loading international/latin-3 (source)...
$29 = 1176488
Loading international/latin-4 (source)...
$30 = 1176528
Loading international/latin-5 (source)...
$31 = 1176568
Loading international/latin-8 (source)...
$32 = 1176608
Loading international/latin-9 (source)...
$33 = 1176648
Loading language/chinese...
$34 = 1188035
Loading language/cyrillic...
$35 = 1200328
Loading language/indian...
$36 = 1205072
Loading language/devanagari (source)...
$37 = 1205496
Loading language/malayalam (source)...
$38 = 1205728
Loading language/tamil (source)...
$39 = 1205944
Loading language/kannada (source)...
$40 = 1206348
Loading language/english (source)...
$41 = 1206518
Loading language/ethiopic...
$42 = 1208167
Loading language/european...
$43 = 1227713
Loading language/czech (source)...
$44 = 1228054
Loading language/slovak (source)...
$45 = 1228391
Loading language/romanian (source)...
$46 = 1228745
Loading language/greek (source)...
$47 = 1229686
Loading language/hebrew (source)...
$48 = 1231287
Loading language/japanese (source)...
$49 = 1235385
Loading language/korean (source)...
$50 = 1237607
Loading language/lao (source)...
$51 = 1238336
Loading language/thai (source)...
$52 = 1240061
Loading language/tibetan...
$53 = 1287208
Loading language/vietnamese...
$54 = 1292846
Loading language/misc-lang (source)...
$55 = 1293018
Loading language/utf-8-lang (source)...
$56 = 1293803
Loading language/georgian (source)...
$57 = 1294121
Loading international/ucs-tables...
$58 = 1303955
Loading indent...
$59 = 1317264
Loading window...
$60 = 1342250
Loading frame...
$61 = 1377063
Loading term/tty-colors...
$62 = 1384552
Loading font-core...
$63 = 1392250
Loading facemenu...
$64 = 1410597
Loading emacs-lisp/syntax...
$65 = 1415424
Loading font-lock...
$66 = 1466936
Loading jit-lock...
$67 = 1480512
Loading mouse...
$68 = 1532487
Loading scroll-bar...
$69 = 1544909
Loading select...
$70 = 1555983
Loading emacs-lisp/timer...
$71 = 1572540
Loading isearch...
$72 = 1633608
Loading rfn-eshadow...
$73 = 1641876
Loading menu-bar...
$74 = 1682850
Loading paths.el (source)...
$75 = 1686340
Loading startup...
$76 = 1746594
Loading emacs-lisp/lisp...
$77 = 1762973
Loading textmodes/page...
$78 = 1765605
Loading register...
$79 = 1776122
Loading textmodes/paragraphs...
$80 = 1785931
Loading emacs-lisp/lisp-mode...
$81 = 1812188
Loading textmodes/text-mode...
$82 = 1816570
Loading textmodes/fill...
$83 = 1848273
Loading replace...
$84 = 1894000
Loading abbrev...
$85 = 1904972
Loading buff-menu...
$86 = 1926450
Loading fringe...
$87 = 1933480
Loading image...
$88 = 1945720
Loading international/fontset...
$89 = 1968027
Loading dnd...
$90 = 1973102
Loading mwheel...
$91 = 1982031
Loading tool-bar...
$92 = 1991013
Loading x-dnd...
$93 = 3206
Loading emacs-lisp/float-sup...
$94 = 3864
Loading vc-hooks...
$95 = 785
Loading jka-cmpr-hook...
$96 = 774
Loading ediff-hook...
$97 = 1055
Loading tooltip...
$98 = 944
$99 = 986

(gdb) continue 
Continuing.
Dumping under names emacs and emacs-22.0.50.7
emacs:0:Pure Lisp storage overflow (approx. 2157456 bytes needed)
9296 pure bytes used

Program exited normally.
(gdb)
--8<---------------cut here---------------end--------------->8---

Maybe you could run this output through your awk script.

>> Will it cause problems?
>
> Not grave problems (we have plan B for such situations), but it
> shouldn't happen.

I also noticed an enormous memory consumption as pointed out in the
thread "mem leak" (started by Miles):
http://thread.gmane.org/gmane.emacs.devel/53300/focus=53311

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Wed, 26 Apr 2006 15:50:12 +0200
> 
> Sorry for the late answer.  I still get an overflow on 64 bit with
> today's sources (I can do the same on 32 bit if it's useful):
> 
> --8<---------------cut here---------------start------------->8---
> $ cd [...]/emacs/cvs-HEAD/emacs/src$ grep define.BASE_PURESIZE puresize.h
> #define BASE_PURESIZE (1205000 + SYSTEM_PURESIZE_EXTRA + SITELOAD_PURESIZE_EXTRA)
> $ cd [...]/emacs/cvs-HEAD/x86_64/src/; gdb ./temacs
> GNU gdb 6.2.1
> Copyright 2004 Free Software Foundation, Inc.
> [...]
> Breakpoint 1 at 0x556da0: file [...]/emacs/src/sysdep.c, line 1373.
> (gdb) break lread.c:717
> Breakpoint 2 at 0x5e79dc: file [...]/emacs/src/lread.c, line 717.
> (gdb) run -batch -l loadup dump
> Starting program: [...]/emacs/cvs-HEAD/x86_64/src/temacs -batch -l loadup dump
> [Thread debugging using libthread_db enabled]
> [New Thread 182940272320 (LWP 30412)]
> [Switching to Thread 182940272320 (LWP 30412)]
> [...]
> Loading tool-bar...
> $92 = 1991013
> Loading x-dnd...
> $93 = 3206
> Loading emacs-lisp/float-sup...
> $94 = 3864
> Loading vc-hooks...
> $95 = 785
> Loading jka-cmpr-hook...
> $96 = 774
> Loading ediff-hook...
> $97 = 1055
> Loading tooltip...
> $98 = 944
> $99 = 986
> 
> (gdb) continue 
> Continuing.
> Dumping under names emacs and emacs-22.0.50.7
> emacs:0:Pure Lisp storage overflow (approx. 2157456 bytes needed)

These results are very strange indeed.  I've built today's CVS on an
x86_64 box (Red Hat GNU/Linux) in 3 different ways: with GTK, with
Motif and with Lucid, and they all needed only 1881740 bytes, give or
take a few dozen bytes.  How come your build requires a whopping 275KB
more?

Comparison of your GDB session with mine shows that each time a .el
file is loaded, it uses up the exact same amount of pure storage in
your build as in mine.  But every .elc file takes more pure storage on
your machine, sometimes only by 1KB, sometimes by as much as 20KB.

Do you have some local changes on your system, or is this a plain
"make bootstrap" of the CVS checkout, with all the defaults wrt
compiler switches, libraries, etc.?  (Not that I see how local changes
to anything but the Lisp files themselves could produce such bloat.)

Does anyone have ideas as to what could cause such a significant
difference in pure storage use on two identical architectures?

I'm stumped.

Re: PURESIZE increased (again)

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> Do you have some local changes on your system, or is this a plain
> "make bootstrap" of the CVS checkout, with all the defaults wrt
> compiler switches, libraries, etc.?  (Not that I see how local changes
> to anything but the Lisp files themselves could produce such bloat.)
>
> Does anyone have ideas as to what could cause such a significant
> difference in pure storage use on two identical architectures?
>
> I'm stumped.

Different data types (UNION_something or what it was?)?  Different
alignment?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: David Kastrup <dak@gnu.org>
> Date: Thu, 27 Apr 2006 22:52:10 +0200
> 
> Different data types (UNION_something or what it was?)?

Should it matter?  Looking at the code, it sound like in both cases,
Lisp_Object should take the same amount of storage.  (I don't have
access to a 64-bit machine where I'm typing this, so I cannot verify
this by compiling.)

> Different alignment?

The alignment should be the same since it's the same architecture and
the same compiler.

Re: PURESIZE increased (again)

[Luc Teirlinck]

Eli Zaretskii wrote:

   Comparison of your GDB session with mine shows that each time a .el
   file is loaded, it uses up the exact same amount of pure storage in
   your build as in mine.  But every .elc file takes more pure storage on
   your machine, sometimes only by 1KB, sometimes by as much as 20KB.

Of course, using different compilers or different C Libraries,
_including_ different version numbers for the same compiler or library
will not have any effect on the .el file, but it can definitely have
an effect on the .elc files.  Different OS or different versions of
the same OS of course also can make a difference.  As I already
mentioned, talking about KB is meaningless in this context, this is
about average percentages.  Nearly 13 percent seems indeed to be
somewhat to the large side.  On the other hand a difference of less
than one percent, like the difference between our two numbers for 32
bit (which we discussed before) seems to be well within the bounds of
what is to be expected and hence completely meaningless.  But maybe
variations tend to be larger for 64 bit than for 32?

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Thu, 27 Apr 2006 16:19:01 -0500 (CDT)
> From: Luc Teirlinck <teirllm@dms.auburn.edu>
> CC: emacs-devel@gnu.org
> 
> Of course, using different compilers or different C Libraries,
> _including_ different version numbers for the same compiler or library
> will not have any effect on the .el file, but it can definitely have
> an effect on the .elc files.  Different OS or different versions of
> the same OS of course also can make a difference.

As you wrote later, since pure storage comes out of a static array, it
is hard to believe the OS or the compiler can make the difference.
But for the record, the compiler I used was gcc 3.4.5.

> As I already mentioned, talking about KB is meaningless in this
> context, this is about average percentages.

I don't know whether average is the right measure here (we won't know
for sure until we discover the reason(s) for the discrepancy), but
how's this: loading subr.elc took 14KB more in Rainer's build than in
mine, bindings.elc took 23KB more, utf-16.elc took 15KB more, etc.  I
think these are very large differences.

> But maybe variations tend to be larger for 64 bit than for 32?

AFAIU the byte compiling, there should be no variation at all on the
same architecture and OS, as long as the same files are loaded.

Re: PURESIZE increased (again)

[Stefan Monnier]

>> But maybe variations tend to be larger for 64 bit than for 32?

> AFAIU the byte compiling, there should be no variation at all on the
> same architecture and OS, as long as the same files are loaded.

I believe that some filenames end up in purespace, so the size of purespace
might change depending on the compilation location or the
installation location.  This may explain small differences, but not the
major one we're seeing here,


        Stefan

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: Luc Teirlinck <teirllm@dms.auburn.edu>,  emacs-devel@gnu.org
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 28 Apr 2006 12:09:44 -0400
> 
> I believe that some filenames end up in purespace, so the size of purespace
> might change depending on the compilation location or the
> installation location.

Are you talking about load-history?  That one is copied to pure space
at a specific location during loadup (after loading "site-init"), so
any differences due to that ought to appear only at that spot.

By contrast, what I see is a steady deviation each time some .elc file
is loaded.

Re: PURESIZE increased (again)

[Reiner Steib]

On Thu, Apr 27 2006, Eli Zaretskii wrote:

> These results are very strange indeed.  I've built today's CVS on an
> x86_64 box (Red Hat GNU/Linux) in 3 different ways: with GTK, with
> Motif and with Lucid, and they all needed only 1881740 bytes, give or
> take a few dozen bytes.

My build was with GTK (on SUSE 9.2 GNU/Linux).

> How come your build requires a whopping 275KB more?
>
> Comparison of your GDB session with mine shows that each time a .el
> file is loaded, it uses up the exact same amount of pure storage in
> your build as in mine.  But every .elc file takes more pure storage on
> your machine, sometimes only by 1KB, sometimes by as much as 20KB.

Thanks for your investigations.

> Do you have some local changes on your system, or is this a plain
> "make bootstrap" of the CVS checkout, with all the defaults wrt
> compiler switches, libraries, etc.?

It was without "bootstrap".  I've set
MYCPPFLAGS='-DENABLE_CHECKING=1'.  I use the same source tree for
x86_64 and i686 with different exec-prefix:

  configure --prefix=[...]/HEAD --with-gtk \
    --exec-prefix=[...]/HEAD-`uname -m`

> (Not that I see how local changes to anything but the Lisp files
> themselves could produce such bloat.)
>
> Does anyone have ideas as to what could cause such a significant
> difference in pure storage use on two identical architectures?

I have some (minor, IMHO) local changes.  I will try tomorrow without
any local changes and report back again.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Thu, 27 Apr 2006 23:56:12 +0200
> 
> I've set MYCPPFLAGS='-DENABLE_CHECKING=1'.

I don't think this should matter, as it only activates some assertions
in the C code.

> I use the same source tree for x86_64 and i686 with different
> exec-prefix:
> 
>   configure --prefix=[...]/HEAD --with-gtk \
>     --exec-prefix=[...]/HEAD-`uname -m`

But you do compile in two different directories for each architecture,
right?  That is, the source tree is common, but the build directories
are different, yes?  Or do you use the same directory, but clean it up
between any two builds?  If the latter, how do you clean it?

> I have some (minor, IMHO) local changes.  I will try tomorrow without
> any local changes and report back again.

Thanks.

Re: PURESIZE increased (again)

[Reiner Steib]

On Fri, Apr 28 2006, Eli Zaretskii wrote:

>> From: Reiner Steib <reinersteib+gmane@imap.cc>
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 27 Apr 2006 23:56:12 +0200
>> 
>> I've set MYCPPFLAGS='-DENABLE_CHECKING=1'.
>
> I don't think this should matter, as it only activates some assertions
> in the C code.
>
>> I use the same source tree for x86_64 and i686 with different
>> exec-prefix:
>> 
>>   configure --prefix=[...]/HEAD --with-gtk \
>>     --exec-prefix=[...]/HEAD-`uname -m`
>
> But you do compile in two different directories for each architecture,
> right?  That is, the source tree is common, but the build directories
> are different, yes?  

Yes, I use a common source tree and two separate build directories.
The source tree is in .../cvs-HEAD/emacs and the build directories are
.../cvs-HEAD/`uname -m` (-> i686 and x86_64).

> Or do you use the same directory, but clean it up between any two
> builds?  If the latter, how do you clean it?
>
>> I have some (minor, IMHO) local changes.  I will try tomorrow without
>> any local changes and report back again.

Okay, the following results are with an unmodified, clean source
tree[1] with configure ...  && make bootstrap.  I've put the complete
logs for each arch (i686 and x86_64) on the web [2] in the files
emacs-build-`date -I`-`uname -m`.log.gz (*.gz; 39K / 54K).

During "make bootstrap", I get:

,----[ x86_64 ]
| Finding pointers to doc strings...done
| Dumping under names emacs and emacs-22.0.50
| 108788 pure bytes used
| mv -f emacs bootstrap-emacs
| [...]
| Finding pointers to doc strings...done
| Dumping under names emacs and emacs-22.0.50.1
| emacs:0:Pure Lisp storage overflow (approx. 2159640 bytes needed)
| 1536 pure bytes used
| ./emacs -q -batch -f list-load-path-shadows
`----

,----[ i686 ]
| Finding pointers to doc strings...done
| Dumping under names emacs and emacs-22.0.50
| 75284 pure bytes used
| mv -f emacs bootstrap-emacs
| [...]
| Finding pointers to doc strings...done
| Dumping under names emacs and emacs-22.0.50.1
| emacs:0:Pure Lisp storage overflow (approx. 1345256 bytes needed)
| 608 pure bytes used
| ./emacs -q -batch -f list-load-path-shadows
`----

Here are the results for pure_bytes_used from running temacs under gdb
("run -batch -l loadup dump" with breakpoint in Fload) for both x86_64
and i686.  The full gdb logs are in temacs-gdb-`date -I`-`uname
-m`.log.gz on [2].  temacs-gdb-`date -I`-`uname -m`.pure.log.gz
contain the relevant parts [3] which I include here in this message:

--8<---------------cut here---------------start------------->8---
~/src/links/emacs/cvs-HEAD/x86_64/src$ gdb ./temacs 
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-suse-linux"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

DISPLAY = localhost:11.0
TERM = xterm
Breakpoint 1 at 0x556da0: file /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/sysdep.c, line 1373.
(gdb) break lread.c:717
Breakpoint 2 at 0x5e79dc: file /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/lread.c, line 717.
(gdb) run -batch -l loadup dump
Starting program: /import/Archive/Groups/Productivity/Editors/Emacs/emacs/cvs-HEAD/x86_64/src/temacs -batch -l loadup dump
[Thread debugging using libthread_db enabled]
[New Thread 182940272320 (LWP 28159)]
[Switching to Thread 182940272320 (LWP 28159)]

Breakpoint 2, Fload (file=11077635, noerror=10658513, nomessage=10658513, nosuffix=10658513, must_suffix=10658513)
    at /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/lread.c:718
718       register int fd = -1;
(gdb) print pure_bytes_used
$1 = 108747
$1 = 108747
(gdb) continue
Continuing.
Loading loadup.el (source)...
Loading loadup.el (source)...
Using load-path (/home/dept/ste/src/links/emacs/cvs-HEAD/emacs/lisp)

Breakpoint 2, Fload (file=11080355, noerror=10658513, nomessage=10658513, nosuffix=10658513, must_suffix=10658513)
    at /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/lread.c:718
718       register int fd = -1;
(gdb) print pure_bytes_used
$2 = 108788
$2 = 108788
Loading emacs-lisp/byte-run...
$3 = 114536
Loading emacs-lisp/backquote...
$4 = 118000
Loading subr...
$5 = 205328
Loading version.el (source)...
$6 = 208625
Loading widget...
$7 = 209471
Loading custom...
$8 = 241496
Loading emacs-lisp/map-ynp...
$9 = 248473
Loading env...
$10 = 253068
Loading cus-start...
$11 = 255909
Loading international/mule...
$12 = 317596
Loading international/mule-conf.el (source)...
$13 = 327085
Loading format...
$14 = 350864
Loading bindings...
$15 = 388019
Loading files...
$16 = 537811
Loading cus-face...
$17 = 556913
Loading faces...
$18 = 638354
Loading loaddefs.el (source)...
$19 = 833869
Loading simple...
$20 = 975911
Loading help...
$21 = 1013912
Loading international/mule-cmds...
$22 = 1092096
Loading case-table...
$23 = 1097760
Loading international/utf-8...
$24 = 1123530
Loading international/utf-16...
$25 = 1176110
Loading international/characters...
$26 = 1178958
Loading international/latin-1 (source)...
$27 = 1178958
Loading international/latin-2 (source)...
$28 = 1179000
Loading international/latin-3 (source)...
$29 = 1179040
Loading international/latin-4 (source)...
$30 = 1179080
Loading international/latin-5 (source)...
$31 = 1179120
Loading international/latin-8 (source)...
$32 = 1179160
Loading international/latin-9 (source)...
$33 = 1179200
Loading language/chinese...
$34 = 1190587
Loading language/cyrillic...
$35 = 1202880
Loading language/indian...
$36 = 1207624
Loading language/devanagari (source)...
$37 = 1208048
Loading language/malayalam (source)...
$38 = 1208280
Loading language/tamil (source)...
$39 = 1208496
Loading language/kannada (source)...
$40 = 1208900
Loading language/english (source)...
$41 = 1209070
Loading language/ethiopic...
$42 = 1210719
Loading language/european...
$43 = 1230265
Loading language/czech (source)...
$44 = 1230606
Loading language/slovak (source)...
$45 = 1230943
Loading language/romanian (source)...
$46 = 1231297
Loading language/greek (source)...
$47 = 1232238
Loading language/hebrew (source)...
$48 = 1233839
Loading language/japanese (source)...
$49 = 1237937
Loading language/korean (source)...
$50 = 1240159
Loading language/lao (source)...
$51 = 1240888
Loading language/thai (source)...
$52 = 1242613
Loading language/tibetan...
$53 = 1289760
Loading language/vietnamese...
$54 = 1295398
Loading language/misc-lang (source)...
$55 = 1295570
Loading language/utf-8-lang (source)...
$56 = 1296355
Loading language/georgian (source)...
$57 = 1296673
Loading international/ucs-tables...
$58 = 1306507
Loading indent...
$59 = 1319816
Loading window...
$60 = 1344802
Loading frame...
$61 = 1379615
Loading term/tty-colors...
$62 = 1387104
Loading font-core...
$63 = 1394802
Loading facemenu...
$64 = 1413149
Loading emacs-lisp/syntax...
$65 = 1417976
Loading font-lock...
$66 = 1469488
Loading jit-lock...
$67 = 1483064
Loading mouse...
$68 = 1534775
Loading scroll-bar...
$69 = 1547197
Loading select...
$70 = 1558271
Loading emacs-lisp/timer...
$71 = 1574828
Loading isearch...
$72 = 1635896
Loading rfn-eshadow...
$73 = 1644164
Loading menu-bar...
$74 = 1685082
Loading paths.el (source)...
$75 = 1688572
Loading startup...
$76 = 1748922
Loading emacs-lisp/lisp...
$77 = 1765301
Loading textmodes/page...
$78 = 1767933
Loading register...
$79 = 1778450
Loading textmodes/paragraphs...
$80 = 1788259
Loading emacs-lisp/lisp-mode...
$81 = 1814516
Loading textmodes/text-mode...
$82 = 1818898
Loading textmodes/fill...
$83 = 1850601
Loading replace...
$84 = 1896328
Loading abbrev...
$85 = 1907316
Loading buff-menu...
$86 = 1928794
Loading fringe...
$87 = 1935824
Loading image...
$88 = 1948064
Loading international/fontset...
$89 = 1970371
Loading dnd...
$90 = 1975446
Loading mwheel...
$91 = 1984375
Loading tool-bar...
$92 = 1993357
Loading x-dnd...
$93 = 5574
Loading emacs-lisp/float-sup...
$94 = 6232
Loading vc-hooks...
$95 = 3081
Loading jka-cmpr-hook...
$96 = 3062
Loading ediff-hook...
$97 = 3343
Loading tooltip...
$98 = 3192
$99 = 3234
(gdb) continue
Continuing.
Dumping under names emacs and emacs-22.0.50.2
emacs:0:Pure Lisp storage overflow (approx. 2159640 bytes needed)
1536 pure bytes used

Program exited normally.
(gdb) 
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
~/src/links/emacs/cvs-HEAD/i686/src$ gdb ./temacs 
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

DISPLAY = localhost:12.0
TERM = xterm
Breakpoint 1 at 0x8185989: file /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/sysdep.c, line 1393.
(gdb) break lread.c:717
Breakpoint 2 at 0x821014c: file /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/lread.c, line 717.
(gdb) run -batch -l loadup dump
Starting program: /import/Archive/Groups/Productivity/Editors/Emacs/emacs/cvs-HEAD/i686/src/temacs -batch -l loadup dump
[Thread debugging using libthread_db enabled]
[New Thread 1083362592 (LWP 12468)]
[Switching to Thread 1083362592 (LWP 12468)]

Breakpoint 2, Fload (file=138854739, noerror=138630009, nomessage=138630009, nosuffix=138630009, must_suffix=138630009)
    at /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/lread.c:718
718       register int fd = -1;
(gdb) print pure_bytes_used
$1 = 75259
$1 = 75259
(gdb) continue
Continuing.
Loading loadup.el (source)...
Loading loadup.el (source)...
Using load-path (/home/dept/ste/src/links/emacs/cvs-HEAD/emacs/lisp)

Breakpoint 2, Fload (file=138858059, noerror=138630009, nomessage=138630009, nosuffix=138630009, must_suffix=138630009)
    at /home/dept/ste/src/links/emacs/cvs-HEAD/emacs/src/lread.c:718
718       register int fd = -1;
(gdb) print pure_bytes_used
$2 = 75284
$2 = 75284
Loading emacs-lisp/byte-run...
$3 = 78752
Loading emacs-lisp/backquote...
$4 = 80996
Loading subr...
$5 = 134560
Loading version.el (source)...
$6 = 136543
Loading widget...
$7 = 137055
Loading custom...
$8 = 156712
Loading emacs-lisp/map-ynp...
$9 = 161145
Loading env...
$10 = 164028
Loading cus-start...
$11 = 165893
Loading international/mule...
$12 = 205060
Loading international/mule-conf.el (source)...
$13 = 212149
Loading format...
$14 = 226984
Loading bindings...
$15 = 250075
Loading files...
$16 = 343443
Loading cus-face...
$17 = 354265
Loading faces...
$18 = 402666
Loading loaddefs.el (source)...
$19 = 536045
Loading simple...
$20 = 623551
Loading help...
$21 = 648524
Loading international/mule-cmds...
$22 = 696932
Loading case-table...
$23 = 700388
Loading international/utf-8...
$24 = 715498
Loading international/utf-16...
$25 = 744694
Loading international/characters...
$26 = 746662
Loading international/latin-1 (source)...
$27 = 746662
Loading international/latin-2 (source)...
$28 = 746688
Loading international/latin-3 (source)...
$29 = 746712
Loading international/latin-4 (source)...
$30 = 746736
Loading international/latin-5 (source)...
$31 = 746760
Loading international/latin-8 (source)...
$32 = 746784
Loading international/latin-9 (source)...
$33 = 746808
Loading language/chinese...
$34 = 754027
Loading language/cyrillic...
$35 = 762104
Loading language/indian...
$36 = 765160
Loading language/devanagari (source)...
$37 = 765432
Loading language/malayalam (source)...
$38 = 765576
Loading language/tamil (source)...
$39 = 765704
Loading language/kannada (source)...
$40 = 765956
Loading language/english (source)...
$41 = 766070
Loading language/ethiopic...
$42 = 767351
Loading language/european...
$43 = 781289
Loading language/czech (source)...
$44 = 781502
Loading language/slovak (source)...
$45 = 781711
Loading language/romanian (source)...
$46 = 781937
Loading language/greek (source)...
$47 = 782614
Loading language/hebrew (source)...
$48 = 783751
Loading language/japanese (source)...
$49 = 786649
Loading language/korean (source)...
$50 = 788231
Loading language/lao (source)...
$51 = 788736
Loading language/thai (source)...
$52 = 789949
Loading language/tibetan...
$53 = 818144
Loading language/vietnamese...
$54 = 821998
Loading language/misc-lang (source)...
$55 = 822098
Loading language/utf-8-lang (source)...
$56 = 822555
Loading language/georgian (source)...
$57 = 822769
Loading international/ucs-tables...
$58 = 829083
Loading indent...
$59 = 837056
Loading window...
$60 = 852034
Loading frame...
$61 = 873295
Loading term/tty-colors...
$62 = 877872
Loading font-core...
$63 = 882682
Loading facemenu...
$64 = 893621
Loading emacs-lisp/syntax...
$65 = 896756
Loading font-lock...
$66 = 928936
Loading jit-lock...
$67 = 937152
Loading mouse...
$68 = 969623
Loading scroll-bar...
$69 = 977069
Loading select...
$70 = 983919
Loading emacs-lisp/timer...
$71 = 994052
Loading isearch...
$72 = 1031464
Loading rfn-eshadow...
$73 = 1036460
Loading menu-bar...
$74 = 1062514
Loading paths.el (source)...
$75 = 1065212
Loading startup...
$76 = 1103578
Loading emacs-lisp/lisp...
$77 = 1113445
Loading textmodes/page...
$78 = 1115053
Loading register...
$79 = 1121610
Loading textmodes/paragraphs...
$80 = 1127723
Loading emacs-lisp/lisp-mode...
$81 = 1144212
Loading textmodes/text-mode...
$82 = 1146906
Loading textmodes/fill...
$83 = 1166169
Loading replace...
$84 = 1194136
Loading abbrev...
$85 = 1200676
Loading buff-menu...
$86 = 8962
Loading fringe...
$87 = 3236
Loading image...
$88 = 728
Loading international/fontset...
$89 = 4771
Loading dnd...
$90 = 7838
Loading mwheel...
$91 = 3199
Loading tool-bar...
$92 = 8501
Loading x-dnd...
$93 = 1606
Loading emacs-lisp/float-sup...
$94 = 2008
Loading vc-hooks...
$95 = 8281
Loading jka-cmpr-hook...
$96 = 4374
Loading ediff-hook...
$97 = 4535
Loading tooltip...
$98 = 600
$99 = 626
(gdb) continue
Continuing.
Dumping under names emacs and emacs-22.0.50.2
emacs:0:Pure Lisp storage overflow (approx. 1345256 bytes needed)
608 pure bytes used

Program exited normally.
(gdb)
--8<---------------cut here---------------end--------------->8---

Finally, in footnotes [4] and [5] there is some information about the
machines (CPU, gcc, gdb).  Both machine are running GNU/Linux (SuSE
9.2), 32 and 64 version respectively with all current security patches
available from SUSE.  glibc is glibc-2.3.3-118 on both machines.

Someone pointed out that the filenames (or the directory) might be
stored.  The real directory name is quite long, see [1].  Let me know
if other information might be useful.

Bye, Reiner.

[1]

$ cd ~/src/links/emacs/cvs-HEAD
$ /bin/pwd
/import/Archive/Groups/Productivity/Editors/Emacs/emacs/cvs-HEAD
$ cd emacs
$ cvs update

(on 2006-04-28 approx. at 12:43 CEST)

$ cvs diff -u  > ../rs-`date -I`.patch
$ patch -R -p0 < ../rs-`date -I`.patch
$ cd ..
$ find i686/ x86_64/ -name '*.o'|xargs -r rm
$ find emacs/ -name '*.elc'|xargs -r rm

On each host I run a shell script which basically does... (all details
in emacs-build-`date -I`-`uname -m`.log.gz on [2]

$ export MYCPPFLAGS='-DENABLE_CHECKING=1'
$ cd `uname -m`
$ ../emacs/configure --prefix=/import/xtra/emacs/HEAD --with-gtk \
  --exec-prefix=/import/xtra/emacs/HEAD-`uname -m` &&
  make bootstrap

[2] <URL:http://theotp1.physik.uni-ulm.de/~ste/tmp/emacs/>

[3]
$ sed -ne '1,35p;/^\$[0-9]* = /p;/^Loading/p;919,1000p' \
  < temacs-gdb-`date -I`-`uname -m`.log \
  > temacs-gdb-`date -I`-`uname -m`.pure.log

[4] The x86_64 machine has an AMD Athlon CPU:

$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 31
model name      : AMD Athlon(tm) 64 Processor 3500+
stepping        : 0
cpu MHz         : 2202.859
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 pni syscall nx mmxext lm 3dnowext 3dnow ts
bogomips        : 4308.99
TLB size        : 1088 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

$ gcc -v
Reading specs from /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.4/specs
Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada --disable-checking --libdir=/usr/lib64 --enable-libgcj --with-gxx-include-dir=/usr/include/g++ --with-slibdir=/lib64 --with-system-zlib --enable-shared --enable-__cxa_atexit x86_64-suse-linux
Thread model: posix
gcc version 3.3.4 (pre 3.3.5 20040809)

$ gdb -v
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".

[5] The "i686" machine has two Intel Xeon CPUs:

$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.40GHz
stepping        : 9
cpu MHz         : 2400.431
cache size      : 512 KB
physical id     : 0
siblings        : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips        : 4751.36

processor       : 1
[...]
processor       : 2
[...]
processor       : 3
[...]

$ gcc -v
Reading specs from /usr/lib/gcc-lib/i586-suse-linux/3.3.4/specs
Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada --disable-checking --libdir=/usr/lib --enable-libgcj --with-gxx-include-dir=/usr/include/g++ --with-slibdir=/lib --with-system-zlib --enable-shared --enable-__cxa_atexit i586-suse-linux
Thread model: posix
gcc version 3.3.4 (pre 3.3.5 20040809)
$ gdb -v
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".

Re: PURESIZE increased (again)

[Andreas Schwab]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> Someone pointed out that the filenames (or the directory) might be
> stored.

There are no absolute file names in the pure space, except for a few
system directories.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Apr 2006 15:11:52 +0200
> 
> Okay, the following results are with an unmodified, clean source
> tree[1] with configure ...  && make bootstrap.  I've put the complete
> logs for each arch (i686 and x86_64) on the web [2] in the files
> emacs-build-`date -I`-`uname -m`.log.gz (*.gz; 39K / 54K).
> 
> During "make bootstrap", I get:
> 
> ,----[ x86_64 ]
> | Finding pointers to doc strings...done
> | Dumping under names emacs and emacs-22.0.50
> | 108788 pure bytes used
> | mv -f emacs bootstrap-emacs
> | [...]
> | Finding pointers to doc strings...done
> | Dumping under names emacs and emacs-22.0.50.1
> | emacs:0:Pure Lisp storage overflow (approx. 2159640 bytes needed)
> | 1536 pure bytes used
> | ./emacs -q -batch -f list-load-path-shadows
> `----

Thanks.  So the mystery is still here, since this is the same value
you reported before.

I think it's time to see whether the *.elc files we use are identical.
Can you upload a couple of worst offenders, say, files.elc and
simple.elc?  I'd like to compare them to mine.

TIA

Re: PURESIZE increased (again)

[Reiner Steib]

On Fri, Apr 28 2006, Eli Zaretskii wrote:

> I think it's time to see whether the *.elc files we use are identical.
> Can you upload a couple of worst offenders, say, files.elc and
> simple.elc?  I'd like to compare them to mine.

http://theotp1.physik.uni-ulm.de/~ste/tmp/emacs/lisp/

(The *.elc files were compile on i686.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Fri, 28 Apr 2006 19:15:53 +0200
> 
> On Fri, Apr 28 2006, Eli Zaretskii wrote:
> 
> > I think it's time to see whether the *.elc files we use are identical.
> > Can you upload a couple of worst offenders, say, files.elc and
> > simple.elc?  I'd like to compare them to mine.
> 
> http://theotp1.physik.uni-ulm.de/~ste/tmp/emacs/lisp/
> 
> (The *.elc files were compile on i686.)

Thanks.

I compared these with the versions compiled on a 32-bit Windows host,
and they seem to be identical, except for 2 aspects:

  . Source files, whose absolute file names appear in a comment at the
    beginning of the .elc files.  These are in comments, so they are
    obviously not the reason for the differences in pure space usage.

  . Minor differences in the defvar's and custom forms, like these:

    -(defvar backup-inhibited nil (#$ . 2763))
    +(defvar backup-inhibited nil (#$ . 2801))

    -(custom-declare-variable 'backup-by-copying 'nil '(#$ . -3035) :type 'boolean :group 'backup)
    +(custom-declare-variable 'backup-by-copying 'nil '(#$ . -3073) :type 'boolean :group 'backup)

I think these differences are immaterial, but just so we don't miss
something of importance: could someone who knows more than myself
about the byte-compiled code please tell what are those numbers that
differ between the two systems?

Meanwhile, I will try to prepare a GDB session that we could use to
compare the pure space allocation during loadup.

Thanks again for working on this.

Re: PURESIZE increased (again)

[Stefan Monnier]

>     -(defvar backup-inhibited nil (#$ . 2763))
>     +(defvar backup-inhibited nil (#$ . 2801))

The number is the position in the file (in bytes) of the
corresponding docstring.  It's presumably different because the source file
name in comment earlier in the file.


        Stefan

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 29 Apr 2006 11:27:42 -0400
> 
> >     -(defvar backup-inhibited nil (#$ . 2763))
> >     +(defvar backup-inhibited nil (#$ . 2801))
> 
> The number is the position in the file (in bytes) of the
> corresponding docstring.  It's presumably different because the source file
> name in comment earlier in the file.

Thanks.  That was my guess, but I wanted to be sure.

Re: PURESIZE increased (again)

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>   . Minor differences in the defvar's and custom forms, like these:
>
>     -(defvar backup-inhibited nil (#$ . 2763))
>     +(defvar backup-inhibited nil (#$ . 2801))
>
>     -(custom-declare-variable 'backup-by-copying 'nil '(#$ . -3035) :type 'boolean :group 'backup)
>     +(custom-declare-variable 'backup-by-copying 'nil '(#$ . -3073) :type 'boolean :group 'backup)
>
> I think these differences are immaterial, but just so we don't miss
> something of importance: could someone who knows more than myself
> about the byte-compiled code please tell what are those numbers that
> differ between the two systems?

These are pointers to the doc string, offsets into the file (the sign is
the user-variable-p flag).  Since the comment at the start of the files
differ in length the offsets will differ too.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Reiner Steib]

On Sat, Apr 29 2006, Eli Zaretskii wrote:

>> From: Reiner Steib <reinersteib+gmane@imap.cc>
>> On Fri, Apr 28 2006, Eli Zaretskii wrote:
>> 
>> > I think it's time to see whether the *.elc files we use are identical.
>> > Can you upload a couple of worst offenders, say, files.elc and
>> > simple.elc?  I'd like to compare them to mine.
[...]
> I compared these with the versions compiled on a 32-bit Windows host,
> and they seem to be identical, except for 2 aspects:
>
>   . Source files, whose absolute file names appear in a comment at the
>     beginning of the .elc files.  These are in comments, so they are
>     obviously not the reason for the differences in pure space usage.
>
>   . Minor differences in the defvar's and custom forms, like these:
>
>     -(defvar backup-inhibited nil (#$ . 2763))
>     +(defvar backup-inhibited nil (#$ . 2801))
[...]
>
> Meanwhile, I will try to prepare a GDB session that we could use to
> compare the pure space allocation during loadup.

I just wanted to state that I still need to use
"-DSITELOAD_PURESIZE_EXTRA=100000" to avoid an overflow on x86_64
(GNU/Linux) even after the recent increase of BASE_PURESIZE[1].  90000
is not sufficient.  On i686 (GNU/Linux) I need 140000 (130000 is not
sufficient).

Bye, Reiner.

[1]
,----
| 2006-05-24  Luc Teirlinck  <teirllm@auburn.edu>
| 
| 	* puresize.h (BASE_PURESIZE): Increase to 1210000.
`----
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Tue, 30 May 2006 21:40:30 +0200
> 
> > Meanwhile, I will try to prepare a GDB session that we could use to
> > compare the pure space allocation during loadup.
> 
> I just wanted to state that I still need to use
> "-DSITELOAD_PURESIZE_EXTRA=100000" to avoid an overflow on x86_64

Yes, I didn't yet have time to prepare a GDB session I promised.
Hopefully, I will have that time soon.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Tue, 30 May 2006 21:40:30 +0200
> 
> > Meanwhile, I will try to prepare a GDB session that we could use to
> > compare the pure space allocation during loadup.
> 
> I just wanted to state that I still need to use
> "-DSITELOAD_PURESIZE_EXTRA=100000" to avoid an overflow on x86_64
> (GNU/Linux) even after the recent increase of BASE_PURESIZE[1].  90000
> is not sufficient.  On i686 (GNU/Linux) I need 140000 (130000 is not
> sufficient).

I've looked around some more with GDB, and here's what I'd like you to
do.

Run temacs under GDB, and type these commands:

    (gdb) b Fload
    (gdb) commands
    >p file
    >xstring
    >end
    (gdb) r -batch -l loadup

Then, whenever Fload is called, GDB will display the name of the file
that is about to be loaded.  When it shows "emacs-lisp/byte-run", type
the following commands:

    (gdb) set height 0
    (gdb) b alloc.c:4718
    (gdb) commands
    >bt 2
    >xbacktrace
    >continue
    >end
    (gdb) c

This will run for a while until it finishes loading byte-run, and stop
on the next call to Fload.  At that time, please copy everything that
GDB displayed until that point, and put it somewhere on the Web where
I can read it (or post here).  I will then compare with what I see on
my system.  (If possible, please do this with a 32-bit build, as my
access top a 64-bit host is sporadic.)

TIA

Re: PURESIZE increased (again)

[Reiner Steib]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On Sat, Jun 03 2006, Eli Zaretskii wrote:

> Run temacs under GDB, and type these commands:
>
>     (gdb) b Fload
>     (gdb) commands
>     >p file
>     >xstring
>     >end
>     (gdb) r -batch -l loadup
>
> Then, whenever Fload is called, GDB will display the name of the file
> that is about to be loaded.  When it shows "emacs-lisp/byte-run", type
> the following commands:
>
>     (gdb) set height 0
>     (gdb) b alloc.c:4718
>     (gdb) commands
>     >bt 2
>     >xbacktrace
>     >continue
>     >end
>     (gdb) c
>
> This will run for a while until it finishes loading byte-run, and stop
> on the next call to Fload.  At that time, please copy everything that
> GDB displayed until that point, and put it somewhere on the Web where
> I can read it (or post here).  I will then compare with what I see on
> my system.  (If possible, please do this with a 32-bit build, as my
> access top a 64-bit host is sporadic.)

Attached you'll find the gzipped gdb log (from a fresh bootstrapped
build on 32-bit GNU/Linux).

BTW, the above is on SUSE 9.2.  On SUSE 10.0 I don't observe an
overflow.  I don't have gcc version of SUSE 10.0 here, but I can
provide it later if it's of interest.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

[-- Attachment #2: pure-space-debug-2006-06-09.log.gz --]
[-- Type: application/x-gzip, Size: 3931 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Reiner Steib <reinersteib+gmane@imap.cc>
> Cc: emacs-devel@gnu.org
> Date: Fri, 09 Jun 2006 17:33:24 +0200
> 
> Attached you'll find the gzipped gdb log (from a fresh bootstrapped
> build on 32-bit GNU/Linux).

Thanks.  Will look at it today or tomorrow.

> BTW, the above is on SUSE 9.2.  On SUSE 10.0 I don't observe an
> overflow.  I don't have gcc version of SUSE 10.0 here, but I can
> provide it later if it's of interest.

Please see whether the GCC versions are different (although I cannot
yet see how a different version could explain such large
differences).  What is the GCC version on your SUSE 9.2?

Can people who know about SUSE think of any differences between these
two versions that could explain the difference in pure space usage?

Re: PURESIZE increased (again)

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Please see whether the GCC versions are different (although I cannot
> yet see how a different version could explain such large
> differences).  What is the GCC version on your SUSE 9.2?

9.2 has gcc 3.3.4, whereas 10.0 has 4.0.2.

> Can people who know about SUSE think of any differences between these
> two versions that could explain the difference in pure space usage?

I could think of data alignment being an issue, but otherwise the
behaviour of the generated code should be identical (modulo bugs).

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: emacs-devel@gnu.org
> Date: Fri, 09 Jun 2006 18:57:41 +0200
> 
> 9.2 has gcc 3.3.4, whereas 10.0 has 4.0.2.

Thanks.

> > Can people who know about SUSE think of any differences between these
> > two versions that could explain the difference in pure space usage?
> 
> I could think of data alignment being an issue

The overflow happens on 9.2, but not on 10.0.  Is it plausible that
GCC 3.3.4 on SuSe 9.2 produces code with higher alignment requirements
than GCC 4.0.2 on SuSe 10.0?

Re: PURESIZE increased (again)

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> The overflow happens on 9.2, but not on 10.0.  Is it plausible that
> GCC 3.3.4 on SuSe 9.2 produces code with higher alignment requirements
> than GCC 4.0.2 on SuSe 10.0?

I don't think so.  And I cannot reproduce the problem: there are only
1207936 pure bytes used when building on 9.2, exactly the same as on 10.0.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 10 Jun 2006 00:33:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The overflow happens on 9.2, but not on 10.0.  Is it plausible that
> > GCC 3.3.4 on SuSe 9.2 produces code with higher alignment requirements
> > than GCC 4.0.2 on SuSe 10.0?
> 
> I don't think so.  And I cannot reproduce the problem: there are only
> 1207936 pure bytes used when building on 9.2, exactly the same as on 10.0.

So it's not the OS and not GCC.  What else can possibly affect this?
Would a list of installed packages from Reiner help get a clue?

Re: PURESIZE increased (again)

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> So it's not the OS and not GCC.  What else can possibly affect this?
> Would a list of installed packages from Reiner help get a clue?

I don't see how that can have any influence.  The only hook that can
increase the amount of preloaded libraries is site-load.el, but then you
are supposed to define SITELOAD_PURESIZE_EXTRA appropriately.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Reiner Steib]

On Sat, Jun 10 2006, Andreas Schwab wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> So it's not the OS and not GCC.  What else can possibly affect this?
>> Would a list of installed packages from Reiner help get a clue?
>
> I don't see how that can have any influence.  The only hook that can
> increase the amount of preloaded libraries is site-load.el, but then you
> are supposed to define SITELOAD_PURESIZE_EXTRA appropriately.

I don't have a `site-load.el' file in my source and build directories:

$ find . -name 'site-*'
$ 

I only have the one from the distribution and some `site-load.el' in
XEmacs directories:

$ rpm -qf /usr/share/emacs/21.3/lisp/site-load.el 
emacs-21.3-193.4

Here are the compiler versions:

$ gcc --version
gcc (GCC) 3.3.4 (pre 3.3.5 20040809)
[...]
$ cat /etc/SuSE-release 
SuSE Linux 9.2 (i586)
VERSION = 9.2

I forgot to mention that the i586 machine is a two processor machine,
but that makes no difference: I get an overflow on a single processor
machine as well (SuSE 9.2).

$ cat /etc/SuSE-release 
SUSE LINUX 10.0 (i586) OSS
VERSION = 10.0
$ gcc --version
gcc (GCC) 4.0.2 20050901 (prerelease) (SUSE Linux)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Luc Teirlinck]

>From my previous message:

   Of course, using different compilers or different C Libraries,
   _including_ different version numbers for the same compiler or library
   will not have any effect on the .el file, but it can definitely have
   an effect on the .elc files

On second thought, this seems less immediately obvious, from a purely
"logical" point of view.  But apparently, from the empirical evidence,
differences in compiler, library or OS do seem to matter much more
compiled Lisp files.  Probably, the alignment issues are more complex
for them.

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Ken Raeburn]

On Apr 27, 2006, at 18:12, Luc Teirlinck wrote:
>> From my previous message:
>
>    Of course, using different compilers or different C Libraries,
>    _including_ different version numbers for the same compiler or  
> library
>    will not have any effect on the .el file, but it can definitely  
> have
>    an effect on the .elc files
>
> On second thought, this seems less immediately obvious, from a purely
> "logical" point of view.  But apparently, from the empirical evidence,
> differences in compiler, library or OS do seem to matter much more
> compiled Lisp files.  Probably, the alignment issues are more complex
> for them.

 From what I've seen of the Lisp structures and allocation code, I'd  
be very surprised if this were true.  You *may* get different  
compilers imposing different alignment/padding constraints for  
structures or unions, but such cases should be very rare and arguably  
reported as bugs (it'll lead to interoperability problems if they're  
used on the same system), and I doubt any of the types Emacs Lisp  
uses get any significant padding.  And I don't think there are many  
data types allocated in loading a .elc file that are not used when  
loading a .el file.

Ken

Re: PURESIZE increased (again)

[Luc Teirlinck]

I may be wrong.  I got the impression that differences in, for
instance, the C library and even the actual version of, say glibc, did
matter, from, for instance, the following comment from alloc.c, which
seems to say that different versions of glibc waste different amounts
of memory on alignment.  But maybe I misunderstood the comment.

  /* Padding to leave at the end of a malloc'd block.  This is to give
     malloc a chance to minimize the amount of memory wasted to alignment.
     It should be tuned to the particular malloc library used.
     On glibc-2.3.2, malloc never tries to align, so a padding of 0 is best.
     posix_memalign on the other hand would ideally prefer a value of 4
     because otherwise, there's 1020 bytes wasted between each ablocks.
     In Emacs, testing shows that those 1020 can most of the time be
     efficiently used by malloc to place other objects, so a value of 0 can
     still preferable unless you have a lot of aligned blocks and virtually
     nothing else.  */
  #define BLOCK_PADDING 0

Re: PURESIZE increased (again)

[Ken Raeburn]

On Apr 27, 2006, at 18:53, Luc Teirlinck wrote:
> I may be wrong.  I got the impression that differences in, for
> instance, the C library and even the actual version of, say glibc, did
> matter, from, for instance, the following comment from alloc.c, which
> seems to say that different versions of glibc waste different amounts
> of memory on alignment.  But maybe I misunderstood the comment.

No, I think you got that right ... the runtime process size and  
efficiency of heap allocation can vary a lot depending on the  
libraries.  But the pure storage in Emacs doesn't get allocated that  
way; it comes out of a statically allocated array named pure[] in  
alloc.c, which has its own allocation routines (pure_alloc and friends).

Ken

Re: PURESIZE increased (again)

[Andreas Schwab]

Ken Raeburn <raeburn@gnu.org> writes:

> No, I think you got that right ... the runtime process size and
> efficiency of heap allocation can vary a lot depending on the  libraries.
> But the pure storage in Emacs doesn't get allocated that  way; it comes
> out of a statically allocated array named pure[] in  alloc.c, which has
> its own allocation routines (pure_alloc and friends).

The padding used by pure_alloc depends on USE_LSB_TAG, which may explain
the difference.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Luc Teirlinck <teirllm@dms.auburn.edu>, eliz@gnu.org,
> 	emacs-devel@gnu.org
> Date: Fri, 28 Apr 2006 16:18:22 +0200
> 
> The padding used by pure_alloc depends on USE_LSB_TAG, which may explain
> the difference.

Thanks.

But this will only explain the difference if Reiner says that he
disabled USE_LSB_TAG (which AFAIK is on by default on x86_64 GNU/Linux
systems).  I simply built the checked-out tree, so my value of
USE_LSB_TAG should be the default.

Reiner?

Re: PURESIZE increased (again)

[Reiner Steib]

On Fri, Apr 28 2006, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@suse.de>
[...]
>> The padding used by pure_alloc depends on USE_LSB_TAG, which may explain
>> the difference.
>
> Thanks.
>
> But this will only explain the difference if Reiner says that he
> disabled USE_LSB_TAG (which AFAIK is on by default on x86_64 GNU/Linux
> systems).  I simply built the checked-out tree, so my value of
> USE_LSB_TAG should be the default.

I did the same.  gdb_use_lsb is 1 on both archs when printed in gdb:

(gdb) p gdb_use_lsb
$1 = 1

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: PURESIZE increased (again)

[Luc Teirlinck]

My previous reply was due to confusion.  The comment I quoted seems
irrelevant for objects allocated in pure space.

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Ken Raeburn]

On Apr 27, 2006, at 16:38, Eli Zaretskii wrote:
> Comparison of your GDB session with mine shows that each time a .el
> file is loaded, it uses up the exact same amount of pure storage in
> your build as in mine.  But every .elc file takes more pure storage on
> your machine, sometimes only by 1KB, sometimes by as much as 20KB.

That is weird.  Perhaps output from (garbage-collect) before and  
after loading the individual .elc files would show something useful?   
The byte and object counts *should* be the same (uh, unless the  
pathnames to the elc files are stored somewhere but el file pathnames  
are not).  It might also be useful to check that the .elc files you  
two are getting (you've both done "make bootstrap", right?) are  
actually similar.

Ken

Re: PURESIZE increased (again)

[David Kastrup]

Ken Raeburn <raeburn@raeburn.org> writes:

> On Apr 27, 2006, at 16:38, Eli Zaretskii wrote:
>> Comparison of your GDB session with mine shows that each time a .el
>> file is loaded, it uses up the exact same amount of pure storage in
>> your build as in mine.  But every .elc file takes more pure storage on
>> your machine, sometimes only by 1KB, sometimes by as much as 20KB.
>
> That is weird.  Perhaps output from (garbage-collect) before and
> after loading the individual .elc files would show something useful?
> The byte and object counts *should* be the same (uh, unless the
> pathnames to the elc files are stored somewhere but el file pathnames
> are not).

Not necessarily.  Most filenames are written to accommodate 8+3. so if
we have 12345678.el\0, this is twelve bytes.  The same with .elc might
require sixteen bytes after alignment.

But it is more likely that some byte code internals cause trouble.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: PURESIZE increased (again)

[Ken Raeburn]

On Apr 27, 2006, at 18:38, David Kastrup wrote:
> Not necessarily.  Most filenames are written to accommodate 8+3. so if
> we have 12345678.el\0, this is twelve bytes.  The same with .elc might
> require sixteen bytes after alignment.

I don't think that should be varying between compilers though.   
Unless the x86_64 ABI (implementation) isn't stable yet, which would  
be bad news...

Ken

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> From: David Kastrup <dak@gnu.org>
> Date: Fri, 28 Apr 2006 00:38:34 +0200
> 
> Not necessarily.  Most filenames are written to accommodate 8+3. so if
> we have 12345678.el\0, this is twelve bytes.  The same with .elc might
> require sixteen bytes after alignment.

As I wrote elsewhere, we need to explain several KB per .elc file, so
a few bytes of difference won't do, even if the file names are stored.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 27 Apr 2006 18:24:55 -0400
> 
> The byte and object counts *should* be the same (uh, unless the  
> pathnames to the elc files are stored somewhere but el file pathnames  
> are not).

Even if this is true (which I don't think it is), how can a stored
name explain 20KB of difference?

> It might also be useful to check that the .elc files you two are
> getting (you've both done "make bootstrap", right?) are actually
> similar.

That's the point: how _could_ they be different?

Re: PURESIZE increased (again)

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-devel@gnu.org
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Thu, 27 Apr 2006 18:24:55 -0400
>> 
>> The byte and object counts *should* be the same (uh, unless the  
>> pathnames to the elc files are stored somewhere but el file pathnames  
>> are not).
>
> Even if this is true (which I don't think it is), how can a stored
> name explain 20KB of difference?

Maybe the principal data structure for a byte code passage takes a
different size because of alignment or data type issues?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: PURESIZE increased (again)

[Ken Raeburn]

On Apr 28, 2006, at 01:29, Eli Zaretskii wrote:
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Thu, 27 Apr 2006 18:24:55 -0400
>>
>> The byte and object counts *should* be the same (uh, unless the
>> pathnames to the elc files are stored somewhere but el file pathnames
>> are not).

(Actually, I should've said the increases in the byte and object  
counts, when each .elc file is loaded.)

> Even if this is true (which I don't think it is), how can a stored
> name explain 20KB of difference?

It wouldn't, but if it does happen (and I don't think it does,  
either, but I don't recall for sure) it would be a reason for the  
numbers not to be *exactly* the same, so then a very small  
discrepancy wouldn't be a big problem.  But like you say, I don't  
think it happens.

>> It might also be useful to check that the .elc files you two are
>> getting (you've both done "make bootstrap", right?) are actually
>> similar.
>
> That's the point: how _could_ they be different?

Barring the obvious, like local hacks affecting the byte-code  
optimizer, or some local bug causing character encoding conversions  
to be applied to byte-code strings, I have no idea.  But since I have  
no other good idea how the 20K difference came up loading a .elc  
file, I figure breaking the problem down might help.  For example:  
First, confirm that some file foo.elc to be loaded is (functionally)  
the same, and that it consume different amounts of storage, on the  
two systems.  Then split it apart (binary search, one S-expression at  
a time, whatever) and see if there's some particular kind of  
expression in the .elc file that consumes different amounts of  
storage on the two systems.  If we know what it is, perhaps we can  
figure out why it's handled differently.  But if the files are  
different, then the problem isn't differing storage consumed by  
(identical) loaded objects, and we go off in a very different  
direction....

If people want to expend that much effort on it, of course.

Ken

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Fri, 28 Apr 2006 03:07:35 -0400
> 
> > That's the point: how _could_ they be different?
> 
> Barring the obvious, like local hacks affecting the byte-code  
> optimizer, or some local bug causing character encoding conversions  
> to be applied to byte-code strings, I have no idea.  But since I have  
> no other good idea how the 20K difference came up loading a .elc  
> file, I figure breaking the problem down might help.  For example:  
> First, confirm that some file foo.elc to be loaded is (functionally)  
> the same, and that it consume different amounts of storage, on the  
> two systems.

Yes, if we find no other explanation, comparing .elc files would be
the way to go.

> Then split it apart (binary search, one S-expression at  
> a time, whatever) and see if there's some particular kind of  
> expression in the .elc file that consumes different amounts of  
> storage on the two systems.

No need for binary search, I think: since we are debugging pure
storage use, it's better to put a breakpoint on the functions that
allocate space off pure[], and then xbacktrace will show what
expression is being evaluated, while the C code will show how much
pure space is being allocated.

> If people want to expend that much effort on it, of course.

275KB of memory sounds like a good reason to me, but that's me.

Re: PURESIZE increased (again)

[Luc Teirlinck]

Reiner Steib wrote:

   I also get the "Pure space overflow" warning from
   `fancy-splash-screens-1':

I also get the "Pure space overflow" warning on 32 bit GNU/Linux, with
freshly downloaded CVS.  Any reason not to increase PURESIZE to get
rid of this problem?

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Fri, 21 Apr 2006 18:10:03 -0500 (CDT)
> From: Luc Teirlinck <teirllm@dms.auburn.edu>
> CC: eliz@gnu.org, romain@orebokech.com, emacs-devel@gnu.org
> 
> Reiner Steib wrote:
> 
>    I also get the "Pure space overflow" warning from
>    `fancy-splash-screens-1':
> 
> I also get the "Pure space overflow" warning on 32 bit GNU/Linux, with
> freshly downloaded CVS.  Any reason not to increase PURESIZE to get
> rid of this problem?

I increased it by 5K, but could you please post your data of pure[]
usage similar to what I posted here yesterday?  I think we simply
_must_ understand why on similar systems the numbers are so different
(with today's CVS, I'm still 7KB short of the limit before the 5K
increase).  I don't like solutions to problems we don't understand
well enough, it looks like kludging around the problem instead of
solving it.

TIA

Re: PURESIZE increased (again)

[Miles Bader]

On 4/22/06, Eli Zaretskii <eliz@gnu.org> wrote:
> I think we simply
> _must_ understand why on similar systems the numbers are so different

Why?  What's the _downside_ of adding a fudge factor to puresize?  Is
it worth the time to debug?  [On a modern system.]

It's nice to understand every detail, but sometimes it's not worth the effort.

-Miles
--
Do not taunt Happy Fun Ball.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Sat, 22 Apr 2006 20:35:05 +0900
> From: "Miles Bader" <miles@gnu.org>
> Cc: "Luc Teirlinck" <teirllm@dms.auburn.edu>, romain@orebokech.com, 
> 	Reiner.Steib@gmx.de, emacs-devel@gnu.org
> 
> On 4/22/06, Eli Zaretskii <eliz@gnu.org> wrote:
> > I think we simply
> > _must_ understand why on similar systems the numbers are so different
> 
> Why?  What's the _downside_ of adding a fudge factor to puresize?

It makes the memory footprint larger.

> It's nice to understand every detail,

If you don't understand the problem, how do you know you indeed fixed
it?  How do you know there isn't some other factor at work here?

> but sometimes it's not worth the effort.

What effort?  It took me 30 seconds to run temacs under GDB to produce
the data, and another 5 minutes to write and run an Awk one-liner to
process the data into the table I posted.  I wish all problems I ever
have to explore will take this little effort.

Re: PURESIZE increased (again)

[Luc Teirlinck]

Eli Zaretskii wrote"

   > Why?  What's the _downside_ of adding a fudge factor to puresize?

   It makes the memory footprint larger.

By a completely negligible percentage (a fraction of a percent),
obviously not enough to worry about or waste any time on trying to
reduce it further.  Certainly not in the CVS version, where
BASE_PURESIZE needs to be increased all the time and the only
non-negligible effects of making the fudge factor even tinier are to
increase the frequency of the required increases and to force most CVS
user to increase the value locally if they have a very few small
uninstalled changes.

Comparing my present pure-bytes-used of 1200904 with the 1036280 from
an old CVS version of 2005-02-07, suggests that pure-bytes-used is
currently growing faster than 13 percent a year (between the the
reference points I used it grew at an annualized rate of 13.7
percent).  I do not know whether such a rate of growth constitutes a
problem.  I have no real figures, but I would guess that the amount of
memory available on the average computer grows even faster than 14
percent a year, in which case the growth of pure-bytes-used does not
represent a real problem.

Anyway, whether the rate of growth of pure-bytes-used represents a
problem or not, reducing the fudge-factor from a fraction of one
percent to an even tinier fraction of one percent is completely
meaningless.  As long as that fudge factor stays within one percent,
it stays negligible and does not contribute to the rate of growth.

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Sat, 22 Apr 2006 20:59:51 -0500 (CDT)
> From: Luc Teirlinck <teirllm@dms.auburn.edu>
> CC: miles@gnu.org, romain@orebokech.com, Reiner.Steib@gmx.de,
>         emacs-devel@gnu.org
> 
> Eli Zaretskii wrote"
> 
>    > Why?  What's the _downside_ of adding a fudge factor to puresize?
> 
>    It makes the memory footprint larger.
> 
> By a completely negligible percentage (a fraction of a percent),
> obviously not enough to worry about or waste any time on trying to
> reduce it further.

I measure memory footprint in bytes, not in percents.  10KB is not
negligible, IMHO, even if taken in isolation.

> Comparing my present pure-bytes-used of 1200904 with the 1036280 from
> an old CVS version of 2005-02-07, suggests that pure-bytes-used is
> currently growing faster than 13 percent a year

Again, this is 170KB growth, certainly not a negligible amount of
memory.

Re: PURESIZE increased (again)

[Nick Roberts]

 > > Comparing my present pure-bytes-used of 1200904 with the 1036280 from
 > > an old CVS version of 2005-02-07, suggests that pure-bytes-used is
 > > currently growing faster than 13 percent a year
 > 
 > Again, this is 170KB growth, certainly not a negligible amount of
 > memory.

But given the RAM doubles every 18-24 months, increasing by only 13 percent a
year means Emacs is becoming progressively easier to run.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

RE: PURESIZE increased (again)

[Drew Adams]

    given that RAM doubles every 18-24 months,...

by the time Emacs 22 releases, available RAM will be Omega.

And Emacs RAM usage doesn't grow that fast, so you will by then be able to
run Omega Emacs sessions simultaneously.

;-) Just kidding, folks - had to get in one last release joke before it
actually happens!

Re: PURESIZE increased (again)

[Alan Shutko]

Nick Roberts <nickrob@snap.net.nz> writes:

> But given the RAM doubles every 18-24 months, increasing by only 13 percent a
> year means Emacs is becoming progressively easier to run.

Emacs already has already about the lowest memory requirements of the
applications I use to do real work.  My eclipse uses about 10x as much
memory... 

-- 
Alan Shutko <ats@acm.org> - I am the rocks.
Killed a lawyer, put spikes on the back of my ambulance!

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Nick Roberts <nickrob@snap.net.nz>
> Date: Sun, 23 Apr 2006 15:46:03 +1200
> Cc: Luc Teirlinck <teirllm@dms.auburn.edu>, romain@orebokech.com,
> 	Reiner.Steib@gmx.de, emacs-devel@gnu.org
> 
>  > Again, this is 170KB growth, certainly not a negligible amount of
>  > memory.
> 
> But given the RAM doubles every 18-24 months, increasing by only 13 percent a
> year means Emacs is becoming progressively easier to run.

Maybe _you_ buy a new machine or upgrade its memory once every 18-24
months.  I don't.

Re: PURESIZE increased (again)

[Richard Stallman]

    But given the RAM doubles every 18-24 months, increasing by only 13 percent a
    year means Emacs is becoming progressively easier to run.

Indeed, Emacs is now not particularly large, nor particularly slow to
start up.  A program using a mere Eight Megabytes is no longer
Constantly Swapping.

Re: PURESIZE increased (again)

[Nick Roberts]

 >     But given the RAM doubles every 18-24 months, increasing by only 13
 >     percent a year means Emacs is becoming progressively easier to run.
 > 
 > Indeed, Emacs is now not particularly large, nor particularly slow to
 > start up.  A program using a mere Eight Megabytes is no longer
 > Constantly Swapping.

That quote is from another (more witty) Nick Roberts, although I wish I
could take credit for it.  But you're right, it's no longer appropriate
and a new one is needed.  Now if can only be the one to think of it...


-- 
Nick                                           http://www.inet.net.nz/~nickrob

Re: PURESIZE increased (again)

[Bill Wohler]

Eli Zaretskii <eliz@gnu.org> writes:

> Again, this is 170KB growth, certainly not a negligible amount of
> memory.

That growth is .35% of my 48 MB Emacs process, so for what it's worth,
*I* think it's negligible ;-).

-- 
Bill Wohler <wohler@newt.com>  http://www.newt.com/wohler/  GnuPG ID:610BD9AD
Maintainer of comp.mail.mh FAQ and MH-E. Vote Libertarian!
If you're passed on the right, you're in the wrong lane.

Re: PURESIZE increased (again)

[Luc Teirlinck]

Bill Wohler wrote:

   Eli Zaretskii <eliz@gnu.org> writes:

   > Again, this is 170KB growth, certainly not a negligible amount of
   > memory.

   That growth is .35% of my 48 MB Emacs process, so for what it's worth,
   *I* think it's negligible ;-).

When I suggested that it even _might_ be a problem, I somehow saw a
zero too many, I thought it was 1.7M (I should have looked more
carefully), in which case it could have been something that _might_ be
worth worrying about.

Given that it only is 170K, I agree that it is obvious that this
increase represents no problem whatsoever and that we should not worry
about it.

When I start Emacs with `emacs -q -nbc', the original memory usage is
9856K.  But when I start actually using it, the Megs start growing
immediately.  For instance, if all I do in my freshly launched Emacs,
is `M-x customize-browse' and open all top level groups to get a basic
overview (by clicking on the `+' next to it and then closing it back
by clicking on the `-'), the memory used goes up to 19568K.  (That is
opening _only_ the top level groups, no subgroups at all.)  And then
something apparently added an extra 8K to it quite a while after I
stopped using the Emacs (probably font-lock or redisplay).

If you have so little memory that 170K is worth worrying about, you
quite simply have not enough memory to run Emacs (and _definitely_ not
enough memory to run things like Gnome, KDE or common web browsers
like Mozilla).  If you have so little memory that even 10K is
non-negligible, I have no idea what you could run.  Not even vi, which
takes exactly 1 Meg.  Since I doubt that vi really requires _exactly_ 1M,
even vi, which is especially designed to work on systems with very
little memory, does not seem to care about small fudge factors like 10K.

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Bill Wohler]

Luc Teirlinck <teirllm@dms.auburn.edu> wrote:

> If you have so little memory that 170K is worth worrying about, you
> quite simply have not enough memory to run Emacs (and _definitely_ not
> enough memory to run things like Gnome, KDE or common web browsers
> like Mozilla).  If you have so little memory that even 10K is
> non-negligible, I have no idea what you could run.

You bring up a good point. In what sort of environments do people run
Emacs?

If they are running under X or under Windows, Emacs certainly won't be
the largest process. If they are running in a console on a PC or
mainframe, 170 kB is probably nothing to worry about either.

However, what if folks are running Emacs on embedded devices? Or PDAs?
It's probably not worth our time to worry about these devices in the
general case (since the number is probably zero given the constraints),
but rather provide a stripped-down version of Emacs that pretty much
just provides basic editing, plus anything else that space permits. That
would then allow Emacs to run on these small devices. For comparison,
most of the applications on my Treo are less than 100 kB, although a
couple have pushed over the 1 MB threshold.

Someday, when I have Linux running on my smartphone, I would like to be
able to edit files in directories in Emacs instead of editing
"categorized memos" as I do today.

-- 
Bill Wohler <wohler@newt.com>  http://www.newt.com/wohler/  GnuPG ID:610BD9AD
Maintainer of comp.mail.mh FAQ and MH-E. Vote Libertarian!
If you're passed on the right, you're in the wrong lane.

Re: PURESIZE increased (again)

[David Kastrup]

Bill Wohler <wohler@newt.com> writes:

> Someday, when I have Linux running on my smartphone, I would like to
> be able to edit files in directories in Emacs instead of editing
> "categorized memos" as I do today.

I think you'd better invest your energy into a cable or bluetooth link
and make do with tramp.  Emacs is geared for getting work done, and
smartphone keyboards aren't.

I tend to write SMS or Phonebook entries for my phone on my laptop and
upload them.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: PURESIZE increased (again)

[Robert J. Chassell]

   ... Emacs to run on these small devices. 

Back in the 1980s, I stripped an Emacs 18 down to below 300 kilobytes;
it had the common editing commands and probably C mode, but very
little else.  It took up less space than VI ...

    Someday, when I have Linux running on my smartphone, I would like
    to be able to edit files in directories in Emacs instead of
    editing "categorized memos" as I do today.

Yes, that is fully possible, even taking 10, 20, or more times as much
space as the 1980s instance.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: PURESIZE increased (again)

[Richard Stallman]

    However, what if folks are running Emacs on embedded devices? Or PDAs?

They are rather bad platforms for GNU Emacs, as well as unlikely.  I
do not want to put any effort into making GNU Emacs work better on
them--it would distract attention from things that really matter.

There are (or at least were) other Emacses designed for smaller
platforms.  So please let's forget about those platforms here.

    Someday, when I have Linux running on my smartphone, I would like to be

If it can run GNU Emacs, it will surely be a GNU/Linux system, not
just Linux.  So please say "when I have GNU/Linux running on my
smartphone".  Please don't give the credit for our work to someone
else.  See http://www.gnu.org/gnu/gnu-linux-faq.html.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Sun, 23 Apr 2006 12:29:07 -0500 (CDT)
> From: Luc Teirlinck <teirllm@dms.auburn.edu>
> Cc: emacs-devel@gnu.org
> 
> If you have so little memory that 170K is worth worrying about, you
> quite simply have not enough memory to run Emacs (and _definitely_ not
> enough memory to run things like Gnome, KDE or common web browsers
> like Mozilla).  If you have so little memory that even 10K is
> non-negligible, I have no idea what you could run.  Not even vi, which
> takes exactly 1 Meg.  Since I doubt that vi really requires _exactly_ 1M,
> even vi, which is especially designed to work on systems with very
> little memory, does not seem to care about small fudge factors like 10K.

Luc, you simply misunderstand what I said, and so your arguments
_completely_ miss the point, so much so that they are almost absurd.

My point was twofold:

 . 10KB of memory well used is nothing to worry about.  However, 10KB
   of _wasted_ memory is something I don't dismiss too easily, because
   there are other programs running on the same machine, and while
   10KB for Emacs is a negligible amount, it is certainly _not_ so for
   a program with a 50KB footprint that needs to run at the same time.

 . My original motivation for insisting to understand the growth was
   that there could be some other factor at work here (a.k.a. ``bug'').

Now please let's stop this thread because it threatens to deteriorate
into mocking the subject, and the original problem was resolved
already.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Bill Wohler <wohler@newt.com>
> Date: Sun, 23 Apr 2006 08:54:59 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Again, this is 170KB growth, certainly not a negligible amount of
> > memory.
> 
> That growth is .35% of my 48 MB Emacs process, so for what it's worth,
> *I* think it's negligible ;-).

It's not negligible if you have a program other than Emacs that needs
to run and starts paging because it's 150KB short of its memory usage.

Re: PURESIZE increased (again)

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > Date: Sat, 22 Apr 2006 20:59:51 -0500 (CDT)
  > > From: Luc Teirlinck <teirllm@dms.auburn.edu>
  > > CC: miles@gnu.org, romain@orebokech.com, Reiner.Steib@gmx.de,
  > >         emacs-devel@gnu.org
  > > 
  > > Comparing my present pure-bytes-used of 1200904 with the 1036280 from
  > > an old CVS version of 2005-02-07, suggests that pure-bytes-used is
  > > currently growing faster than 13 percent a year
  >
  > Again, this is 170KB growth, certainly not a negligible amount of
  > memory.


Since 2005-02-07 the following have been added to loadup.el:

+ (load "facemenu")
+ (load "emacs-lisp/syntax")
+ (load "font-lock")
+ (load "jit-lock")
+ (load "rfn-eshadow")
+ (if (fboundp 'x-create-frame)
+     (progn
+       (load "fringe")
+       (load "image")
+       (load "international/fontset")
+       (load "dnd")
+       (load "mwheel")
+       (load "tool-bar")))
+ (if (featurep 'x)
+     (load "x-dnd"))
+ (load "jka-cmpr-hook")
+ (if (fboundp 'x-show-tip) (load "tooltip"))

After deleting these the pure size used goes from 1200912 to 1086896
on my system (x86 Fedora Core 5). The bulk of it is due to the
font-lock related files.

Given that the above additions to loadup.el have added some desirable
functionality to the default emacs build, the increase in pure memory
use should be acceptable.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: Luc Teirlinck <teirllm@dms.auburn.edu>, romain@orebokech.com,
>         Reiner.Steib@gmx.de, emacs-devel@gnu.org
> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Sun, 23 Apr 2006 09:23:43 -0700
> 
> Given that the above additions to loadup.el have added some desirable
> functionality to the default emacs build, the increase in pure memory
> use should be acceptable.

I didn't say the additions were useless, just that the added footprint
wasn't negligible.

Re: PURESIZE increased (again)

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > Cc: Luc Teirlinck <teirllm@dms.auburn.edu>, romain@orebokech.com,
  > >         Reiner.Steib@gmx.de, emacs-devel@gnu.org
  > > From: Dan Nicolaescu <dann@ics.uci.edu>
  > > Date: Sun, 23 Apr 2006 09:23:43 -0700
  > > 
  > > Given that the above additions to loadup.el have added some desirable
  > > functionality to the default emacs build, the increase in pure memory
  > > use should be acceptable.
  > 
  > I didn't say the additions were useless, just that the added footprint
  > wasn't negligible.

I had no intention of implying that. Just that the additional space is
well explained by the added functionality, so there's little reason to
worry that is due to hidden bug/problem.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Sun, 23 Apr 2006 11:48:05 -0700
> 
>   > I didn't say the additions were useless, just that the added footprint
>   > wasn't negligible.
> 
> I had no intention of implying that. Just that the additional space is
> well explained by the added functionality

How does one know it is ``well explained''?  I asked in this thread,
near its now long forgotten beginning, how does one estimate the
amount of pure space taken by a set of Lisp files, but didn't hear any
answers.  (And now I think there is no way of making such
estimations.)

Re: PURESIZE increased (again)

[Richard Stallman]

    Given that the above additions to loadup.el have added some desirable
    functionality to the default emacs build, the increase in pure memory
    use should be acceptable.

Even more cogent, the reason for this change is that Emacs always
loaded those files just after it started up.  My change is such that
they are preloaded rather than loaded each time.

Re: PURESIZE increased (again)

[Luc Teirlinck]

>From my previous message:

   Comparing my present pure-bytes-used of 1200904 with the 1036280 from
   an old CVS version of 2005-02-07, suggests that pure-bytes-used is
   currently growing faster than 13 percent a year (between the the
   reference points I used it grew at an annualized rate of 13.7
   percent).

And that was with a feature freeze in effect during the entire period.

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Richard Stallman]

    > I think we simply
    > _must_ understand why on similar systems the numbers are so different

    Why?  What's the _downside_ of adding a fudge factor to puresize?  Is
    it worth the time to debug?  [On a modern system.]

If the numbers vary a little, that's not important in itself.
But if they are very different, that suggests something else is wrong.
It's worth tracking down the root cause, just to see if it has
other consequences more important than variation in puresize.

Re: PURESIZE increased (again)

[Luc Teirlinck]

Richard Stallman wrote:

   If the numbers vary a little, that's not important in itself.
   But if they are very different, that suggests something else is wrong.

They are not very different.  They only differ by a fraction of one
percent.  Running temacs under GDB and putting a breakpoint on Fload
(as Eli suggested) just does not seem to work for me.  After byte-run
is loaded, pure_bytes_used is 76028 for me, which is less than the
79800 posted by Eli at the same point in temacs.  After that `c' in
GDB just continues, apparently completely ignoring the breakpoint on
Fload.

After bootstrapping, my pure-bytes-used is 1200904, which is
apparently less than the 1216528 reported by Eli.  Both numbers are
larger than BASE_PURESIZE used to be before it got increased from
1200000 to 1205000.  After that increase I no longer get the overflow
warning.

I have a few tiny uninstalled changes, but if these make a difference,
the fudge factor would really be to small.

Sincerely,

Luc.

Re: PURESIZE increased (again)

[Eli Zaretskii]

> Date: Sat, 22 Apr 2006 20:05:17 -0500 (CDT)
> From: Luc Teirlinck <teirllm@dms.auburn.edu>
> CC: miles@gnu.org, eliz@gnu.org, romain@orebokech.com, Reiner.Steib@gmx.de,
>         emacs-devel@gnu.org
> 
> After that `c' in GDB just continues, apparently completely ignoring
> the breakpoint on Fload.

That is very strange indeed.

> After bootstrapping, my pure-bytes-used is 1200904, which is
> apparently less than the 1216528 reported by Eli.

This part I understand: I measured the values on MS-Windows, which
loads a bunch of files other systems don't (that's why it defines a
25KB extra for PURESIZE).

> Both numbers are larger than BASE_PURESIZE used to be before it got
> increased from 1200000 to 1205000.  After that increase I no longer
> get the overflow warning.

Fine, then I think we can close the issue.  Thanks.

Re: PURESIZE increased (again)

[Richard Stallman]

    They are not very different.  They only differ by a fraction of one
    percent.

In that case, perhaps it is not worth investigating.

Re: PURESIZE increased (again)

[Romain Francoise]

Andreas Schwab <schwab@suse.de> writes:

> Where do you see that?  I see only 1198536 bytes (1877040 on 64bit).

After rebuilding Emacs from a fresh checkout I get only 1199624 bytes,
so the (- 1200000 1199624) => 376 remaining bytes must be taken by local
changes.  I have local items in the menu bar and it looks like they're
the cause of the size increase; I didn't think they could.

I reverted my change and will change the pure size as a local change,
too; the trunk doesn't need it at the moment.  Thanks for making me
aware of this issue.

-- 
Romain Francoise <romain@orebokech.com> | The sea! the sea! the open
it's a miracle -- http://orebokech.com/ | sea! The blue, the fresh, the
                                        | ever free! --Bryan W. Procter

Re: PURESIZE increased (again)

[Eli Zaretskii]

> From: Romain Francoise <romain@orebokech.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 16 Apr 2006 12:56:42 +0200
> 
> I grant you that a 10,000 bytes increase for an overflow of 696 bytes
> might be too much, but since the size keeps going up it's just preparing
> for future changes...

I don't see any need to prepare for future changes.  When those
changes come, if they require to enlarge PURESIZE, we will do it at
that time.

> As you may or may not know, since August 2005 I've been packaging weekly
> snapshots of Emacs CVS for the Debian GNU/Linux operating system;
> this gives me the opportunity to build Emacs automatically with a
> variety of different configuration options and compiler flags every
> week, so I catch these kinds of problems pretty quickly.  I think the
> overflow is caused this week by Bill's latest batch of changes to
> Customize.

My build is from today, so the Customize changes are already in.

Re: PURESIZE increased (again)

[Bill Wohler]

Romain Francoise <romain@orebokech.com> writes:

> I think the overflow is caused this week by Bill's latest batch of
> changes to Customize.

And if the other packages implement the :package-version keyword, it
will get bigger as well. I did change the value from (symbol string)
to (symbol . string) which saved a tiny bit of space; if anyone sees
where we can get more savings, please let me know.

-- 
Bill Wohler <wohler@newt.com>  http://www.newt.com/wohler/  GnuPG ID:610BD9AD
Maintainer of comp.mail.mh FAQ and MH-E. Vote Libertarian!
If you're passed on the right, you're in the wrong lane.