64-bit lossage

64-bit lossage

[Dave Love]

It looks as though the current code is broken generally on 64-bit
systems (by induction from alpha-osf and mips-sgi, so not
endian-related).  I don't know since when, but it predates the
emacs-unicode branch, which it's hampering me working on.  The symptom
is a crash (segv, if I remember correctly) in GC during dumping.

I made a bug report some time ago, but it doesn't seem to have been
fixed.  Could someone look into it?  I assume the commercial outfits
(at least SuSE?) are interested in 64-bit.

(It isn't due to an obvious lack of prototypes, although I found and
fixed a few on the unicode branch.)

Re: 64-bit lossage

[Richard Stallman]

I don't know if we have anyone who supports 64-bit platforms.
We may have to wait until someone who cares about those platforms
wants to debug the problem.

Re: 64-bit lossage

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

|> I don't know if we have anyone who supports 64-bit platforms.
|> We may have to wait until someone who cares about those platforms
|> wants to debug the problem.

I'm running Emacs on ia64, no problems.  The current incarnation has been
built from CVS about a week ago.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: 64-bit lossage

[Dave Love]

Andreas Schwab <schwab@suse.de> writes:

> I'm running Emacs on ia64, no problems.

So much for extrapolation...

Can you test on GNU/Alpha, which might help to narrow it down?  I
don't have a dual-boot system to try.

I assume there weren't any ia64-specific changes which should be made
elsewhere.

Re: 64-bit lossage

[Andreas Schwab]

Dave Love <d.love@dl.ac.uk> writes:

|> Andreas Schwab <schwab@suse.de> writes:
|> 
|> > I'm running Emacs on ia64, no problems.
|> 
|> So much for extrapolation...
|> 
|> Can you test on GNU/Alpha, which might help to narrow it down?  I
|> don't have a dual-boot system to try.

Just built Emacs from CVS head as of today on alpha-suse-linux, and it
works.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: 64-bit lossage

[Dave Love]

Andreas Schwab <schwab@suse.de> writes:

> Just built Emacs from CVS head as of today on alpha-suse-linux, and it
> works.

Thanks.

I made a guess and found what causes the problem (segv in GC while
shrinking the gap).

You lose when USE_MMAP_FOR_BUFFERS is defined on 64-bit targets.  [For
some reason, this isn't defined on Solaris, 32- or 64-bit, which I
think means you can't reclaim the memory used by deleted buffer.]  I
assume there is some type error introduced since 21.2, which works.
Could someone have a look at that code?  I worked on the mmap stuff
originally, but I don't remember much about it, and it's all changed
since then.

I still think the special-purpose mmap-ism should die in favour of
using dlmalloc on non-GNU systems instead of the currently-provided
gmalloc.  As far as I remember, the only objection was that it might
not be as portable; I haven't seen any evidence for that, and it is
claimed to be `among the most portable' mallocs.

Re: 64-bit lossage

[Ken Raeburn]

Dave Love <d.love@dl.ac.uk> writes:
> You lose when USE_MMAP_FOR_BUFFERS is defined on 64-bit targets.  [For
> some reason, this isn't defined on Solaris, 32- or 64-bit, which I
> think means you can't reclaim the memory used by deleted buffer.]  I
> assume there is some type error introduced since 21.2, which works.
> Could someone have a look at that code?  I worked on the mmap stuff
> originally, but I don't remember much about it, and it's all changed
> since then.

Is the patch I checked in on Sunday enough to fix it?

> I still think the special-purpose mmap-ism should die in favour of
> using dlmalloc on non-GNU systems instead of the currently-provided
> gmalloc.  As far as I remember, the only objection was that it might
> not be as portable; I haven't seen any evidence for that, and it is
> claimed to be `among the most portable' mallocs.

Perhaps.

Re: 64-bit lossage

[Dave Love]

Richard Stallman <rms@gnu.org> writes:

> I don't know if we have anyone who supports 64-bit platforms.

In principle I do -- at least Tru64 -- but I know there's commercial
interest in them, and I'm not in a position to sort this out.  (Debian
supports at least Alpha, SPARC, MIPS and IA64 of the 64-bit
architectures I know of, though I don't know if MIPS has 64-bit
userland.)

> We may have to wait until someone who cares about those platforms
> wants to debug the problem.

This sort of breakage would be a lot easier to deal with if someone
tested the codebase regularly on such platforms, as I used to; I'd be
surprised if no-one else has them available.  In my experience it's
typically much easier if you can narrow down when it occurred and look
for suspicious changes.  Going months without being able to build on
Alphas is bad news.

Re: 64-bit lossage

[Stefan Monnier]

> This sort of breakage would be a lot easier to deal with if someone
> tested the codebase regularly on such platforms, as I used to; I'd be

As a matter of principle, I've always done my best to compile/test
Emacs on all the different systems I have access to.  10 years ago
that included 64bit machines, but nowadays all I have is x86.
VHS vs Beta...


	Stefan

Re: 64-bit lossage

[Richard Stallman]

    This sort of breakage would be a lot easier to deal with if someone
    tested the codebase regularly on such platforms, as I used to; I'd be
    surprised if no-one else has them available.  In my experience it's
    typically much easier if you can narrow down when it occurred and look
    for suspicious changes.

I will ask the pretesters if anyone has these machines available.

Re: 64-bit lossage

[Ken Raeburn]

I have two Alpha systems available.

One runs Debian, very slowly; it's an old 166MHz box without much
memory.  (If anyone knows where I can find a fast, cheap Alpha that
runs Linux, I'm interested in upgrading.)

The other is at work and runs Tru64 5.1; it's a better build machine
at 500MHz but I need to remember to keep resynchronizing my sources
manually until someone gets OpenAFS ported. *sigh*

I'll try to get some builds done on them more often, but I'll note one
thing right off -- the m/alpha.h file breaks the USE_LISP_UNION_TYPE
code because it unconditionally redefines some of the macros for
picking apart an integer used as a Lisp_Object.  At a glance, it
appears that m/ia64.h and others do this too.

I'd say the macros in lisp.h need revising to better support 64-bit
architectures; this shouldn't be needed on a machine-by-machine basis.
If no one else gets around to it, I may take a shot at it....

Ken

Re: 64-bit lossage

[Richard Stallman]

Would you like to fix some of the broken macros?

Re: 64-bit lossage

[Ken Raeburn]

> Would you like to fix some of the broken macros?

I will try.

Re: 64-bit lossage

[Dave Love]

Ken Raeburn <raeburn@raeburn.org> writes:

> One runs Debian, very slowly; it's an old 166MHz box without much
> memory.

It probably runs Debian faster than the 48MB P100 I've done most
development on...

> I'll try to get some builds done on them more often, but I'll note one
> thing right off -- the m/alpha.h file breaks the USE_LISP_UNION_TYPE
> code because it unconditionally redefines some of the macros for
> picking apart an integer used as a Lisp_Object.

Yes.  Why is that a problem in practice?  (Also, lisp.h doesn't always
undef such things before re-defining them, with possible warning
storms as a consequence.)

> I'd say the macros in lisp.h need revising to better support 64-bit
> architectures;

As far as I know, they currently work, but there are probably more
serious 64-bit problems elsewhere unless someone has re-done what
eggert and I both started on.

Adding long long support (per TODO) seems most useful in that area.

Re: 64-bit lossage

[Ken Raeburn]

Dave Love <d.love@dl.ac.uk> writes:
>> I'll try to get some builds done on them more often, but I'll note one
>> thing right off -- the m/alpha.h file breaks the USE_LISP_UNION_TYPE
>> code because it unconditionally redefines some of the macros for
>> picking apart an integer used as a Lisp_Object.
>
> Yes.  Why is that a problem in practice?  (Also, lisp.h doesn't always
> undef such things before re-defining them, with possible warning
> storms as a consequence.)

I'm using the Lisp union type to ensure we've got reasonable type
handling, that we're not confusing Lisp objects and actual integer
values we want to use.  In the current representation the bit patterns
are the same for non-negative integers, but if we ever want to be able
to change the representation -- not necessarily switching to Guile,
but perhaps moving the tag bits to the low bits that are always zero
in a properly aligned struct pointer -- we still need to fix all those
cases.

A necessary early step in the Guile work I want to do is to get the
ability to make such changes.  Even if we never switch to Guile, I
believe that ability is important to have.

From what I've seen so far, on all the platforms where the Lisp object
manipulation macros were redefined, there are probably better ways to
do it.  And sometimes the reason for the redefinition is just bogus;
for example, MARKBIT was redefined on the alpha because of some
problem with an #if test that as far as I can see, doesn't exist now.
Generally the macros duplicate values that lisp.h or config.h would've
selected anyways.

I could work on these changes without changing m/alpha.h; I just
couldn't work on them on the Alpha.

As far as Joe Random downloading and building Emacs, it's not directly
causing any problem at all, aside from those redefinition warnings.

>> I'd say the macros in lisp.h need revising to better support 64-bit
>> architectures;
>
> As far as I know, they currently work, but there are probably more
> serious 64-bit problems elsewhere unless someone has re-done what
> eggert and I both started on.
>
> Adding long long support (per TODO) seems most useful in that area.

On most (all?) of the 64-bit configurations I've worked on, "long" is
64 bits; "long long" is mostly interesting only if "long" isn't as big
as a pointer, or we really really want to store more information than
a pointer.  If we hit a platform where only "long long" is pointer
sized, yes, we'll need to use it.  But if the lisp-union code compiles
without complaint, I bet "long long" can be made to work quite easily.

Ken

P.S.  I'm running an Emacs binary right now that was built to use a
union for a Lisp_Object, compiled for x86.  Most of what I'm doing is
interactive, but I haven't noticed any performance issues.  I did one
timing test with "make autoloads", and out of about four minutes of
CPU time, the "union" version was slower by about five seconds.
That's probably doing lots of searching in the regexp code;
byte-compiling might be a better test case to look for worst-case
results, but I haven't tried it yet.

Re: 64-bit lossage

[Dave Love]

Ken Raeburn <raeburn@raeburn.org> writes:

> I'm using the Lisp union type to ensure we've got reasonable type
> handling, that we're not confusing Lisp objects and actual integer
> values we want to use.

I don't think that needs to be done specifically on Alpha, though.

> Generally the macros duplicate values that lisp.h or config.h would've
> selected anyways.

I know there's lots of that sort of thing in the build system overall,
but we were forbidden to clean it up in the past.

> On most (all?) of the 64-bit configurations I've worked on, "long" is
> 64 bits;

Of course.  The idea was to be able to use long long on 32-bit
systems.

Re: 64-bit lossage

[Ken Raeburn]

Dave Love <d.love@dl.ac.uk> writes:
> It looks as though the current code is broken generally on 64-bit
> systems (by induction from alpha-osf and mips-sgi, so not

Which OS are you using on the Alpha?

On the 5.1 machine I've got at work, with a source tree I've already
started banging on for some of the lisp implementation issues I was
complaining about recently, I'm having some problems with mmap.  Not
using mmap for buffer storage seems to fix the problem for the moment,
but I'll investigate that further.

This debugging is more tedious than expected -- no dbx license
installed, and gdb-3.1 doesn't work.  Let's hear it for printf! :-)

I also noticed a number of compiler warnings about pointer mismatches
where the pointer targets were of the wrong type (e.g., int vs long);
I'll try to clean some of that up later.  But they kind of get lost
in the long list of complaints about mixing "char *" and "unsigned
char *".

> endian-related).  I don't know since when, but it predates the
> emacs-unicode branch, which it's hampering me working on.  The symptom
> is a crash (segv, if I remember correctly) in GC during dumping.

My problem showed up in some buffer manipulation code, so maybe it's
not the same problem.  But like I said, I've already started making
other changes in my tree....  Could you try the patch below?

If it works, try re-enabling the use of mmap and see if that part is
working on your system.

It's not ready to be checked in, but your results will let me know if
I'm on the right track.

Ken


Index: mem-limits.h
===================================================================
RCS file: /cvsroot/emacs/emacs/src/mem-limits.h,v
retrieving revision 1.29
diff -p -u -r1.29 mem-limits.h
--- mem-limits.h	11 Dec 2001 22:18:16 -0000	1.29
+++ mem-limits.h	21 Jul 2002 11:18:17 -0000
@@ -81,8 +81,10 @@ extern POINTER start_of_data ();
 
 #ifdef BSD_SYSTEM
 #ifndef DATA_SEG_BITS
+#ifndef DATA_START
 extern char etext;
 #define start_of_data() &etext
+#endif
 #endif
 #endif
 
Index: puresize.h
===================================================================
RCS file: /cvsroot/emacs/emacs/src/puresize.h,v
retrieving revision 1.65
diff -p -u -r1.65 puresize.h
--- puresize.h	16 Apr 2002 17:27:42 -0000	1.65
+++ puresize.h	21 Jul 2002 11:18:17 -0000
@@ -42,7 +42,7 @@ Boston, MA 02111-1307, USA.  */
 #endif
 
 #ifndef BASE_PURESIZE
-#define BASE_PURESIZE (830000 + SYSTEM_PURESIZE_EXTRA + SITELOAD_PURESIZE_EXTRA)
+#define BASE_PURESIZE (930000 + SYSTEM_PURESIZE_EXTRA + SITELOAD_PURESIZE_EXTRA)
 #endif
 
 /* Increase BASE_PURESIZE by a ratio depending on the machine's word size.  */
Index: m/alpha.h
===================================================================
RCS file: /cvsroot/emacs/emacs/src/m/alpha.h,v
retrieving revision 1.45
diff -p -u -r1.45 alpha.h
--- m/alpha.h	15 Apr 2002 13:16:00 -0000	1.45
+++ m/alpha.h	21 Jul 2002 11:18:17 -0000
@@ -29,8 +29,6 @@ NOTE-END
 
 */
 
-#define BITS_PER_LONG 64
-#define BITS_PER_EMACS_INT 64
 #ifndef _LP64
 #define _LP64			/* This doesn't appear to be necessary
 				   on OSF 4/5  -- fx.  */
@@ -56,14 +54,8 @@ NOTE-END
 
 /* Use type EMACS_INT rather than a union, to represent Lisp_Object */
 /* This is desirable for most machines.  */
-
 #define NO_UNION_TYPE
 
-/* Define the type to use.  */
-#define EMACS_INT long
-#define EMACS_UINT unsigned long
-#define SPECIAL_EMACS_INT
-
 /* Define EXPLICIT_SIGN_EXTEND if XINT must explicitly sign-extend
    the 24-bit bit field into an int.  In other words, if bit fields
    are always unsigned.
@@ -113,6 +105,7 @@ NOTE-END
 #ifdef __ELF__
 #undef UNEXEC
 #define UNEXEC unexelf.o
+#define DATA_START    0x140000000
 #endif
 
 #ifndef __ELF__
@@ -122,11 +115,6 @@ NOTE-END
 #define TEXT_START    0x120000000
 #define DATA_START    0x140000000
 
-/* This is necessary for mem-limits.h, so that start_of_data gives
-   the correct value */
-
-#define DATA_SEG_BITS 0x140000000
-
 /* The program to be used for unexec. */
 
 #define UNEXEC unexalpha.o
@@ -164,28 +152,6 @@ NOTE-END
 #define LINUX_SBRK_BUG
 #endif
 
-
-#define PNTR_COMPARISON_TYPE unsigned long
-
-/* On the 64 bit architecture, we can use 60 bits for addresses */
-
-#define VALBITS         60
-
-
-/* This definition of MARKBIT is necessary because of the comparison of
-   ARRAY_MARK_FLAG and MARKBIT in an #if in lisp.h, which cpp doesn't like. */
-
-#define MARKBIT         0x8000000000000000L
-
-
-/* Define XINT and XUINT so that they can take arguments of type int */
-
-#define XINT(a)  (((long) (a) << (BITS_PER_LONG - VALBITS)) >> (BITS_PER_LONG - VALBITS))
-#define XUINT(a) ((long) (a) & VALMASK)
-
-/* Define XPNTR to avoid or'ing with DATA_SEG_BITS */
-
-#define XPNTR(a) XUINT (a)
 
 #ifndef NOT_C_CODE
 /* We need these because pointers are larger than the default ints.  */
Index: s/osf5-0.h
===================================================================
RCS file: /cvsroot/emacs/emacs/src/s/osf5-0.h,v
retrieving revision 1.6
diff -p -u -r1.6 osf5-0.h
--- s/osf5-0.h	10 Oct 2001 12:21:58 -0000	1.6
+++ s/osf5-0.h	21 Jul 2002 11:18:17 -0000
@@ -24,4 +24,4 @@
 /* We have missing/inconsistent prototypes on 5.0, at least.  */
 #define INHIBIT_X11R6_XIM
 
-#define USE_MMAP_FOR_BUFFERS	1
+/*#define USE_MMAP_FOR_BUFFERS	1*/

Re: 64-bit lossage

[Ken Raeburn]

I wrote:
> On the 5.1 machine I've got at work, with a source tree I've already
> started banging on for some of the lisp implementation issues I was
> complaining about recently, I'm having some problems with mmap.

I've checked in a fix for the mmap problem.

Please let me know if it, or the patch I sent, fixes your problem on
the Alpha.

Ken

Re: 64-bit lossage

[Dave Love]

Ken Raeburn <raeburn@raeburn.org> writes:

> I've checked in a fix for the mmap problem.

Could you please send me a patch.

> Please let me know if it, or the patch I sent, fixes your problem on
> the Alpha.
> 
> Ken

Re: 64-bit lossage

[Dave Love]

Ken Raeburn <raeburn@raeburn.org> writes:

> Which OS are you using on the Alpha?

Tru64 5.1a, but I doubt that matters.  Irix64 does just that same.

> On the 5.1 machine I've got at work, with a source tree I've already
> started banging on for some of the lisp implementation issues I was
> complaining about recently,

What issues?

> I'm having some problems with mmap.  Not using mmap for buffer
> storage seems to fix the problem for the moment, but I'll
> investigate that further.

Yes -- I read this too late -- but what has changed in that area since
21.2?

> and gdb-3.1 doesn't work.

Indeed; I've worked on it (insufficiently).  The GNUPro one from the
`The Linux and Tru64 UNIX Portability' (?) CD works, but I've not been
able to extract the sources for it from Compaq^WHP -- I must try
again.

> I also noticed a number of compiler warnings about pointer mismatches
> where the pointer targets were of the wrong type (e.g., int vs
> long);

Some of that is problems with system headers that I may have kludges
for already.

> I'll try to clean some of that up later.  But they kind of get lost
> in the long list of complaints about mixing "char *" and "unsigned
> char *".

Yes, see etc/TODO.  See cc(1) for how to disable specific warnings,
e.g. `-msg_disable ptrmismatch1,promotmatchw', which I have currently.
I mostly used gcc to find such problems, though, and the result is the
same as with cc currently.  [The Compaq compiler supports some gcc
extensions, so testing __GNUC__ may not be optimal.]

> My problem showed up in some buffer manipulation code, so maybe it's
> not the same problem.  But like I said, I've already started making
> other changes in my tree....  Could you try the patch below?

What problem is it trying to solve?  I don't understand.  It doesn't
fix Tru64 or affect Solaris, Irix.

Re: 64-bit lossage

[Ken Raeburn]

Dave Love <d.love@dl.ac.uk> writes:
>> Which OS are you using on the Alpha?
> Tru64 5.1a, but I doubt that matters.  Irix64 does just that same.

I don't think I've got access to an Irix64 system to test with; I'm
fairly sure all the ones I've got access to are 32-bit only.  But
you're probably using irix5-0.h (which gets included by irix6-5.h and
others), which also turns on the mmap code.

>> On the 5.1 machine I've got at work, with a source tree I've already
>> started banging on for some of the lisp implementation issues I was
>> complaining about recently,
> What issues?

Per-machine definitions of macros dependent on low-level details of
the lisp implementation.  It's cleaner if the lisp implementation is
based on information about the architecture, and not partially
rewritten for some architectures.

The macros as they stand in lisp.h do appear to be fairly clean for
64-bit support.  They assume that "long" will be 64 bits when pointers
are, and that "_LP64" is defined in that case, but switching to "long
long" if "long" isn't big enough should actually be quite easy.

> Yes -- I read this too late -- but what has changed in that area since
> 21.2?

I think the make_gap_smaller code is new on the trunk since the
current release branch was started; that's what caused mmap_realloc to
be called to unmap some pages.  I want to look a bit more closely,
though, and see if there are other cases that could cause similar
problems on the release branch.

> Yes, see etc/TODO.  See cc(1) for how to disable specific warnings,
> e.g. `-msg_disable ptrmismatch1,promotmatchw', which I have currently.

Thanks, I'll try those.

> I mostly used gcc to find such problems, though, and the result is the
> same as with cc currently.  [The Compaq compiler supports some gcc
> extensions, so testing __GNUC__ may not be optimal.]

>> My problem showed up in some buffer manipulation code, so maybe it's
>> not the same problem.  But like I said, I've already started making
>> other changes in my tree....  Could you try the patch below?
> What problem is it trying to solve?  I don't understand.  It doesn't
> fix Tru64 or affect Solaris, Irix.

It wasn't trying to fix one problem.  The patches I had in progress
made the problem disappear for me; I wanted to know if they were
sufficient for you or if you were seeing a different problem.

The other patch I sent, to fix the computed number of pages to unmap,
should've fixed this specific problem, and across multiple platforms.
The other Alpha-specific changes shouldn't have had any impact in this
area, but when I sent them to you I was still not quite certain of
that.

Ken

Re: 64-bit lossage

[Dave Love]

Ken Raeburn <raeburn@raeburn.org> writes:

> I don't think I've got access to an Irix64 system to test with;

For what it's worth, any remotely-recent system can run 64-bit (r4000
up, probably with Irix 5.2 up, but I can't remember for sure that for
back).

> Per-machine definitions of macros dependent on low-level details of
> the lisp implementation.  It's cleaner if the lisp implementation is
> based on information about the architecture, and not partially
> rewritten for some architectures.

The machine and system files are a horrible mess in general, and that
often leads to real problems.

> The macros as they stand in lisp.h do appear to be fairly clean for
> 64-bit support.  They assume that "long" will be 64 bits when pointers
> are, and that "_LP64" is defined in that case, but switching to "long
> long" if "long" isn't big enough should actually be quite easy.

Good but it wasn't trivial when we last tried.

> I think the make_gap_smaller code is new on the trunk since the
> current release branch was started; that's what caused mmap_realloc to
> be called to unmap some pages.  I want to look a bit more closely,
> though, and see if there are other cases that could cause similar
> problems on the release branch.

Good.  (I use the released version in anger on tru64 without
problems.)

> It wasn't trying to fix one problem.  The patches I had in progress
> made the problem disappear for me;

Not for me, for what it's worth.  Only disabling use of mmap did.

Re: 64-bit lossage

[Ken Raeburn]

Dave Love <d.love@dl.ac.uk> writes:
> Ken Raeburn <raeburn@raeburn.org> writes:
>> I don't think I've got access to an Irix64 system to test with;
> For what it's worth, any remotely-recent system can run 64-bit (r4000
> up, probably with Irix 5.2 up, but I can't remember for sure that for
> back).

At work, we've got R5000s running 32-bit kernels, and I don't control
them.

>> Per-machine definitions of macros dependent on low-level details of
>> the lisp implementation.  It's cleaner if the lisp implementation is
>> based on information about the architecture, and not partially
>> rewritten for some architectures.
>
> The machine and system files are a horrible mess in general, and that
> often leads to real problems.

Yes, some of the OS stuff does look pretty messy, but simple
architecture issues can be cleaned up.

>> The macros as they stand in lisp.h do appear to be fairly clean for
>> 64-bit support.  They assume that "long" will be 64 bits when pointers
>> are, and that "_LP64" is defined in that case, but switching to "long
>> long" if "long" isn't big enough should actually be quite easy.
>
> Good but it wasn't trivial when we last tried.

There shouldn't be any cases of mixing int and EMACS_INT values that
could inadvertently create broken lisp objects, and the pure numeric
manipulation should be fine.

Conversions between pointers and integers in Emacs all seem to assume
that EMACS_INT is the same size as a pointer, and that would need
fixing, mostly outside of lisp.h.  Probably an Emacs version of C99's
intptr_t would do the trick.

>> It wasn't trying to fix one problem.  The patches I had in progress
>> made the problem disappear for me;
>
> Not for me, for what it's worth.  Only disabling use of mmap did.

Interesting.  I'd like to look into that further.  Should I try out
the unicode branch?



>> I'm using the Lisp union type to ensure we've got reasonable type
>> handling, that we're not confusing Lisp objects and actual integer
>> values we want to use.
> I don't think that needs to be done specifically on Alpha, though.

Only if I want to test things on 64-bit platforms.

>> Generally the macros duplicate values that lisp.h or config.h would've
>> selected anyways.
> I know there's lots of that sort of thing in the build system overall,
> but we were forbidden to clean it up in the past.

Oh?  I don't want to go traipsing around in areas we're not supposed
to mess with...

>> On most (all?) of the 64-bit configurations I've worked on, "long" is
>> 64 bits;
> Of course.  The idea was to be able to use long long on 32-bit
> systems.

Ah.  To get the full range of pointer values, and larger integers?

Having just heard from someone who wants to edit a 300M file on a
32-bit platform, I may look a little more closely at this, but it's
not a priority.

If we used something more like the Guile object representation -- tag
bits at the bottom instead of the top, and more of the type info in
the pointed-to structure -- we might be able to get 31-bit integers
and 32-bit word pointers.  Since structure pointers are -- or can be
made to be -- aligned on 4- or 8-byte boundaries, we can produce the
pointers by masking off those bits, without interfering with the upper
bits of the pointer at all.  I think XEmacs does this too.

Ken

Re: 64-bit lossage

[Richard Stallman]

    Conversions between pointers and integers in Emacs all seem to assume
    that EMACS_INT is the same size as a pointer, and that would need
    fixing, mostly outside of lisp.h.  Probably an Emacs version of C99's
    intptr_t would do the trick.

What is the case you'd like it to handle
where the two sizes are different?

Re: 64-bit lossage

[Ken Raeburn]

Richard Stallman <rms@gnu.org> writes:
>     Conversions between pointers and integers in Emacs all seem to assume
>     that EMACS_INT is the same size as a pointer, and that would need
>     fixing, mostly outside of lisp.h.  Probably an Emacs version of C99's
>     intptr_t would do the trick.
>
> What is the case you'd like it to handle
> where the two sizes are different?

Wider integer support on a 32-bit platform, so larger files can be
supported, buffer offsets can still be described, etc.  They would
still be bounded by the address space, of course, so it could only
gain maybe a factor of 4 or 8 at most, unless we work up a way to edit
huge files we can't copy into memory all at once.  It would also allow
handling of true 32-bit values (or even somewhat larger) for file
sizes, uids, etc.

Many of the cases where conversions are done are probably safe
already, compiler warnings aside, but I suspect sign-/zero-extension
mistakes might be made in a few cases, if there aren't any
configurations of this sort that have been made to work before.

But after looking at it a little, I think doubling the size of a
Lisp_Object is probably not worth it.

There are other possible approaches to getting bigger integer values
supported in Emacs.  I'm not really inclined to do that much work
rewriting the existing lisp system, though, even if it did look like a
good idea, since switching to Guile objects would have a similar
effect.

Ken

Re: 64-bit lossage

[Paul Eggert]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 01 Aug 2002 13:19:44 -0400
> 
> But after looking at it a little, I think doubling the size of a
> Lisp_Object is probably not worth it.

I agree in general, but Emacs on x86 is quite popular, and the current
28-bit address limit is starting to bite.  A few years ago I started
to work on supporting 64-bit Lisp_Object on 32-bit hosts and I got
about half the way through.  It's not that hard, but it is a bit
tedious (among other things, every function really needs to be
prototyped).

Changing the subject slightly, can't we increase address space and
improve performance on both 32- and 64-bit hosts, without widening
Lisp_Object, by moving the 4 tag bits to the low-order end of the
Lisp_Object, and ensuring that all non-Lisp_Int objects are aligned at
a multiple of 16?  That way, we would get all of the 32-bit address
space.  Lisp_Int addition and subtraction would still be fast, since
the Lisp_Int code is zero.  (Multiplication and division would be
slower, but that's rare.)  And we would speed up access to
non-Lisp_Int objects, since the tag bits in many cases could be
removed at zero cost during normal address arithmetic.

> Wider integer support on a 32-bit platform, so larger files can be
> supported, buffer offsets can still be described, etc....

This would require 64-bit ints, yes.  But I'm more worried about the
current 28-bit limit on Lisp object addresses on 32-bit hosts.  That's
only 256 MB.

Re: 64-bit lossage

[Ken Raeburn]

Paul Eggert <eggert@twinsun.com> writes:
> I agree in general, but Emacs on x86 is quite popular, and the current
> 28-bit address limit is starting to bite.

Starting? :-)
Yeah, part of the reason I looked into this at all was someone at MIT
asking about using Emacs to edit a 300M file.

Unfortunately, x86 is register-starved, so I expect the performance
cost to be rather severe, and perhaps some gcc bugs to surface.  It
might not be so bad if we used 32 bits for the value, and 32 for type
and mark bits.  Then at least the math might be easier on the
compiler, and we wouldn't need any checks to make sure we didn't have
pointer values outside the 32-bit address space.  But we'd still be
pushing 64-bit values around.

>    A few years ago I started
> to work on supporting 64-bit Lisp_Object on 32-bit hosts and I got
> about half the way through.  It's not that hard, but it is a bit
> tedious (among other things, every function really needs to be
> prototyped).

That would certainly help.  But I think the work I've been doing with
using "union Lisp_Object" has shaken out a lot of the problems where
"Lisp_Object" might be confused with integral types, and a few other
people seem to be compiling that way too.

> Changing the subject slightly, can't we increase address space and
> improve performance on both 32- and 64-bit hosts, without widening
> Lisp_Object, by moving the 4 tag bits to the low-order end of the
> Lisp_Object, and ensuring that all non-Lisp_Int objects are aligned at
> a multiple of 16?  That way, we would get all of the 32-bit address
> space.  Lisp_Int addition and subtraction would still be fast, since
> the Lisp_Int code is zero.  (Multiplication and division would be
> slower, but that's rare.)  And we would speed up access to
> non-Lisp_Int objects, since the tag bits in many cases could be
> removed at zero cost during normal address arithmetic.

That would get us more of the address space to use for Lisp objects on
some platforms, but it doesn't get us bigger integers.  We still need
to be able to convert "point" and markers into lisp integers.

We could use just the low bit or two to indicate an integer, but that
also dictates what's in those low bits for all the other lisp types,
which means we probably need to steal another bit for non-integer type
tagging (five bits, or 32-byte alignment, which would greatly increase
the storage used for some smaller structures like Lisp_Cons and
Lisp_String), or move some of that type info into the pointed-to
objects, the way Lisp_Vector and Lisp_Misc each support several data
types.

Or some combination of the two, roughly like this:

   ... XXXX XX0M - integer
   ... XXXX X01M - mask off low two bits to get pointer to Lisp_Cons
   ... XXXX X11M - mask off low three bits to get pointer to struct
                   containing type tag
   M = mark bit

Then cons cells and other objects would have to be aligned on 8-byte
boundaries, and those "other" types would need an extra few bits at
the front as a type tag, like Lisp_Misc has already, and Lisp_Vector
sort of does in the high bits of the size field.

We might get another bit by moving the GC mark bit into the pointed-to
storage, but then cons cells might have to grow.  (You need to be able
to mark a cons cell containing two integers, somehow.)

If the mark bit is moved outside the object entirely -- for example,
use a block of mark bits that correspond to a collection of objects,
with the mark bits being located by doing arithmetic on the object
address -- then the storage still grows, but perhaps a bit more
efficiently.  Locating the mark bit is more expensive, but we get back
a bit in the Lisp_Object representation, and when we pull an object's
mark bit into the CPU cache, we get mark bits for nearby objects as
well; that may gain back a little performance during the mark phase,
and the sweep phase wouldn't have to touch the objects at all.
(Considering the high cost of cache misses relative to arithmetic in
terms of CPU cycles these days, it might even be a net benefit; I
haven't looked at the literature on this issue in a while.  Extending
this to multiple pages per "block" might reduce swapping in some cases
too, I'm not sure.)

There are other games that might be played by address manipulation.
For example: Allocate groups of lisp object structures on page
boundaries, with all the current primary object types each being
allocated on separate pages.  Mask off the low bits of the address,
and look at bookkeeping information at the start of the page, which
includes the type of the objects stored on that page.  Then the type
information doesn't need to be stored per object.


I'm not too inclined to tackle this myself right now.  I'm happy to
discuss it, and perhaps help out in some small ways if someone else
wants to work on it, but my focus is on the Guile work.  And if the
low level details of the Lisp system wind up being thrown out as part
of that work, any effort put in on them would be discarded.

Ken

Re: 64-bit lossage

[Paul Eggert]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 01 Aug 2002 19:37:09 -0400
> 
> I'm not too inclined to tackle this myself right now.

I understand.  I haven't had the time to work on it either, at least
not in the last couple of years.

> my focus is on the Guile work.  And if the low level details of the
> Lisp system wind up being thrown out as part of that work, any
> effort put in on them would be discarded.

Can you estimate the chances of that happening?  (Sorry, I don't know
the status of the Emacs-in-Guile project.)

Re: 64-bit lossage

[Ken Raeburn]

Paul Eggert <eggert@twinsun.com> writes:
> Can you estimate the chances of that happening?  (Sorry, I don't know
> the status of the Emacs-in-Guile project.)

No, I think it's too early to say.

I got an Emacs to the point of running using the Guile allocator and
GC once before, with most object types being Guile "smobs" rather than
normal Scheme object types.  Some aspects of the approach I used were
less than optimal, because I wanted to reduce the internal API changes
scattered through the Emacs code (e.g., pass around a lisp object
instead of a struct pointer), and the performance was rather poor.

This time around, I'm integrating the more reasonable and
Guile-independent changes into the main code base, and looking at
making some of those API changes to make the Guile work cleaner.  I'm
hoping that soon I'll be ready to start a branch and see if I can get
other interested people to help out.  In fact, I should probably put
some thought into how development should go once the branch is
started, and what's really needed in order to start.  Perhaps it could
be started now....

Once something is actually running again, and the interface changes
are done, we can evaluate the performance issues and the status of the
Guile lisp-loader code, and see where we want to go from there.

Ken

Re: 64-bit lossage

[Richard Stallman]

    Changing the subject slightly, can't we increase address space and
    improve performance on both 32- and 64-bit hosts, without widening
    Lisp_Object, by moving the 4 tag bits to the low-order end of the
    Lisp_Object, and ensuring that all non-Lisp_Int objects are aligned at
    a multiple of 16?

A cons cell is 8 bytes, so it would have to be a multiple of 8.

Anyway, is the limiting factor really address space for Lisp objects?
If the issue is the size of a single buffer, that is concerned
with the range of integers, not addresses of Lisp objects.

Re: 64-bit lossage

[Paul Eggert]

> Date: Fri, 2 Aug 2002 16:13:36 -0600 (MDT)
> From: Richard Stallman <rms@gnu.org>
> 
>     can't we increase address space and improve performance on both
>     32- and 64-bit hosts, without widening Lisp_Object, by moving
>     the 4 tag bits to the low-order end of the Lisp_Object, and
>     ensuring that all non-Lisp_Int objects are aligned at a multiple of 16?
> 
> A cons cell is 8 bytes, so it would have to be a multiple of 8.

That could be arranged by sort of a Huffman coding for type bits, as
Ken Raeburn suggested.  E.g., if M denotes a mark bit and XX...X
denotes 28 value bits, we could do this:

 XX...X000M Lisp_Int
 XX...XX01M Lisp_Cons (note that it has 29 value bits)
 XX...X010M Lisp_Symbol
 XX...X011M Lisp_Misc
 XX...X100M Lisp_String
 XX...X110M Lisp_VectorLike
 XX...X111M Lisp_Float

Lisp_Symbol etc. would need to be on 16-byte boundaries, but that
could be arranged too.  Of course we don't need such niceties on
64-bit hosts, but even there I think it would be a performance win to
move the tag bits to the low-order end.

> Anyway, is the limiting factor really address space for Lisp objects?
> If the issue is the size of a single buffer, that is concerned
> with the range of integers, not addresses of Lisp objects.

To represent Elisp integers that do not fit in 28 bits, we could use
floating point.  IEEE-floating-point hosts will give us integers up to
2**51, which is enough.  There is room for a marker bit in 'struct
Lisp_Float.type' to say "This number is really an integer, and
operations like integerp should treat it as one".

(Another possibility is to use bignums, but I assume you've already
considered this.)

At the C level, buffer-size-related integers are currently 'int' or
'EMACS_INT'.  We need to change this to a size_t-width int.  This will
require a lot of C-level coding cleanup but should be invisible to the
Elisp user.  This low-level work needs to be done anyway, since
currently Emacs doesn't work with buffers larger than 2GB even on
64-bit hosts, since such hosts typically have 32-bit 'int'.

Re: 64-bit lossage

[Richard Stallman]

    To represent Elisp integers that do not fit in 28 bits, we could use
    floating point.

That is a possible approach for many kinds of data, that are used in
only a few places in the C code, but it would not be feasible to do
this for buffer indices.  There are lots many places in the code that
convert buffer position from Lisp integers and C ints and back.

Re: 64-bit lossage

[Stefan Monnier]

> > But after looking at it a little, I think doubling the size of a
> > Lisp_Object is probably not worth it.
> 
> I agree in general, but Emacs on x86 is quite popular, and the current
> 28-bit address limit is starting to bite.  A few years ago I started
> to work on supporting 64-bit Lisp_Object on 32-bit hosts and I got
> about half the way through.  It's not that hard, but it is a bit
> tedious (among other things, every function really needs to be
> prototyped).

There are 4 limits:
- the number of objects we can point to
- the largest elisp integer
- the actual address space available to the Emacs process
- the largest buffer that Emacs can handle correctly

The first one (such as using lower bits for tagging) is irrelevant
to this discussion.  The second one is currently set at 128MB for
32bit systems.  The third one is usually around 2GB but can be close
to 4GB if you're lucky (on 32bit systems as well) and Emacs
really can't do much about it.
The fourth one can be 4GB even on 64bit systems because the
Emacs C code often uses `int' to represent buffer positions.

We can try to change the tagging system to move the 128MB limit
closer to the 2-4GB "hard" limit, but I don't see the point:
a 129MB file is most likely program-generated rather than
human-generated, so next week's file might just as well be 4.1GB
instead and we won't be able to do anything about that one.

So I think if we want to deal with "large files on 32bit systems"
we'd better try to see how to allow Emacs to view/edit files
piecemeal (where only part of the file is in memory at any point
in time).

This said, adding arbitrary-integer support to Emacs (see the
gerd-gmp branch for a start) might be a good thing as well.


	Stefan