Some OpenWrt port related problems

Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1.1: Type: text/plain, Size: 1916 bytes --]

Hi,

I'm currently working on an OpenWrt port [1] for Emacs23 to use on the
Ben NanoNote PDA [2].

OpenWrt packages use cross-compilation, so Emacs is used in a NO_DUMP
configuration, loading loadup.el every time it starts.  This causes at
least one problem with environment variables, that I already fixed [3].

Now I'm hitting another problem when using org-mode:

 File mode specification error: 
 (wrong-type-argument stringp (require . t-mouse))

After some debugging this looks like being caused by variable
load-history containing the element:

  ((require . t-mouse))

This looks a little broken, since all other elements have a
filename-string in front of that cons cell, e.g.:

 ("/usr/share/emacs/23.2/lisp/term/linux.el"
  (defun . terminal-init-linux))

Now org-mode calls eval-after-load which calls
load-history-filename-element which causes the wrong-type-argument error
when using the 'car' of ((require . t-mouse)) with string-match.

I can sanitize load-history in site-start.el, removing the broken entry,
but that's not a very clean fix (also it takes time and memory, what a
waste).

Anybody knows who's fault that error is anyways?  is ((require
. t-mouse)) a valid entry?  Is eval-after-load broken?  How does that
entry get inserted into load-history in the first place?

Full output of C-h v load-history attached.

thanks for any help,

cheers,

David

PS: if it helps I could hand out SSH accounts to the nanonote to
individual developers

[1]
http://projects.qi-hardware.com/index.php/p/openwrt-packages/source/tree/master/emacs
[2] http://en.qi-hardware.com/wiki/Ben_NanoNote
[3]
http://projects.qi-hardware.com/index.php/p/openwrt-packages/source/tree/master/emacs/patches/010-fix-init-env.patch
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40


[-- Attachment #1.2: load-history.gz --]
[-- Type: application/octet-stream, Size: 24031 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Stefan Monnier]

> OpenWrt packages use cross-compilation, so Emacs is used in a NO_DUMP
> configuration, loading loadup.el every time it starts.

That's unfortunate, as this is completely untested and unsupported.
I expect you'll bump into further problems.  Maybe a "simpler" solution
is to setup a simulation environment where you could perform the dump.

> This causes at least one problem with environment variables, that
> I already fixed [3].

Not sure if that's the right fix, but indeed there's a bug there that
shows up when using NO_DUMP.  Make sure you record it via M-x
report-emacs-bug.

> Now I'm hitting another problem when using org-mode:
>  File mode specification error: 
>  (wrong-type-argument stringp (require . t-mouse))
> After some debugging this looks like being caused by variable
> load-history containing the element:
>   ((require . t-mouse))
> This looks a little broken, since all other elements have a
> filename-string in front of that cons cell, e.g.:

That one doesn't remind me of anything.

> Anybody knows who's fault that error is anyways?

I don't.

> is ((require . t-mouse)) a valid entry?

No.

> Is eval-after-load broken?

Not that I know.

> How does that entry get inserted into load-history in the first place?

That's the question, yes.  But no, I have no idea how this can happen.


        Stefan

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 2905 bytes --]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> OpenWrt packages use cross-compilation, so Emacs is used in a NO_DUMP
>> configuration, loading loadup.el every time it starts.

> That's unfortunate, as this is completely untested and unsupported.  I
> expect you'll bump into further problems.  Maybe a "simpler" solution
> is to setup a simulation environment where you could perform the dump.

Well unfortunate is that Emacs' Makefiles make a number of assumptions
that are completely incompatible with cross-compilation :) Like
compiling and executing the resulting program in a single Makefile rule.
The dumping is even more hackier.  Didn't even know that such a
procedure is possible on current operating systems :)

The problem with a simulated environment, that you're suggesting, is,
that OpenWrt packages (or rather the recipies used to compile a package)
need to be self-contained inside OpenWrt.  If I needed a target system
emulator I'd have to write a recipie how to compile one on any host and
add a build-dependency to it.  Quite some work, and maybe not even
possible in a completely stable way.  Qemu user-space emulation for the
MIPS32 target seems currently broken (at least it doesn't work for me),
so I'd even need to write a rule for how to generate a root-filesystem
to get into a simulated environment.  And that would have to work for
every of the target CPUs that OpenWrt supports :/

>> This causes at least one problem with environment variables, that I
>> already fixed [3].

> Not sure if that's the right fix, but indeed there's a bug there that
> shows up when using NO_DUMP.  Make sure you record it via M-x
> report-emacs-bug.

Going to do that eventually.

>> Now I'm hitting another problem when using org-mode: File mode
>> specification error: (wrong-type-argument stringp (require
>> . t-mouse)) After some debugging this looks like being caused by
>> variable load-history containing the element: ((require . t-mouse))
>> This looks a little broken, since all other elements have a
>> filename-string in front of that cons cell, e.g.:

> That one doesn't remind me of anything.

Hmm, so maybe related to the CANNOT_DUMP config (note: it's not NO_DUMP,
my mistake).

>> Anybody knows who's fault that error is anyways?

> I don't.

>> is ((require . t-mouse)) a valid entry?

> No.

>> Is eval-after-load broken?

> Not that I know.

>> How does that entry get inserted into load-history in the first
>> place?

> That's the question, yes.  But no, I have no idea how this can happen.

Ok, thanks for the answers.  Then it might be time to do some more
in-depth debugging to see where in loadup.el that entry originates.

cheers,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Stefan Monnier]

> Well unfortunate is that Emacs' Makefiles make a number of assumptions
> that are completely incompatible with cross-compilation :) Like
> compiling and executing the resulting program in a single Makefile rule.
> The dumping is even more hackier.  Didn't even know that such a
> procedure is possible on current operating systems :)

Yes, it's clearly not friendly to cross-compilation.
There's a lot of room for improvement there, but given the reliance on
dumping, there is very little motivation to actually make it
cross-compilation friendly.

> The problem with a simulated environment, that you're suggesting, is,
> that OpenWrt packages (or rather the recipies used to compile a package)
> need to be self-contained inside OpenWrt.  If I needed a target system

I'm somewhat familiar with OpenWRT, so I have an idea of how much work
this would require in general.  But some targets may come with "readily
available" simulators, so I thought maybe that was the case for the
NanoNote as well.

>>> Now I'm hitting another problem when using org-mode: File mode
>>> specification error: (wrong-type-argument stringp (require
>>> . t-mouse)) After some debugging this looks like being caused by
>>> variable load-history containing the element: ((require . t-mouse))
>>> This looks a little broken, since all other elements have a
>>> filename-string in front of that cons cell, e.g.:
>> That one doesn't remind me of anything.
> Hmm, so maybe related to the CANNOT_DUMP config (note: it's not NO_DUMP,
> my mistake).

Yes, very likely.

>> That's the question, yes.  But no, I have no idea how this can happen.
> Ok, thanks for the answers.  Then it might be time to do some more
> in-depth debugging to see where in loadup.el that entry originates.

Indeed.  Grepping for Vload_file_name, Vload_history, and CANNOT_DUMP
might give you a good start.  Good luck,


        Stefan

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The problem with a simulated environment, that you're suggesting, is,
>> that OpenWrt packages (or rather the recipies used to compile a
>> package) need to be self-contained inside OpenWrt.  If I needed a
>> target system

> I'm somewhat familiar with OpenWRT, so I have an idea of how much work
> this would require in general.  But some targets may come with
> "readily available" simulators, so I thought maybe that was the case
> for the NanoNote as well.

Well, if I'm going through the hassle of porting a package to NanoNote
I'm trying to make it general enough to work on all OpenWrt platforms.
So finally we'll be able to run Emacs on off-the-shelf routers :)

>>> That's the question, yes.  But no, I have no idea how this can
>>> happen.
>> Ok, thanks for the answers.  Then it might be time to do some more
>> in-depth debugging to see where in loadup.el that entry originates.

> Indeed.  Grepping for Vload_file_name, Vload_history, and CANNOT_DUMP
> might give you a good start.  Good luck,

Maybe not before next year :)

cheers,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Richard Stallman]

Why do you want to avoid dumping?

-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

> Why do you want to avoid dumping?

Because OpenWrt is using cross-compilation for everything.  I.e.  the
system used for compilation is usually an x86 system, but the generated
executables are for MIPS32 (when compiling for NanoNote).

I could dump Emacs in the target system, from a wrapper script, when
launched for the first time.  But last time I tried that it failed with
insufficient memory (32Mb RAM, no swap), so I'm going without dumping
for now.

So far its working out pretty well.  These are just the compromises
necessary on OpenWrt (and Emacs is certainly not the only package in
need of heavy patching).

cheers,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Ken Raeburn]

On Dec 29, 2010, at 04:28, David Kuehling wrote:
>>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:
> 
>> Why do you want to avoid dumping?
> 
> Because OpenWrt is using cross-compilation for everything.  I.e.  the
> system used for compilation is usually an x86 system, but the generated
> executables are for MIPS32 (when compiling for NanoNote).
> 
> I could dump Emacs in the target system, from a wrapper script, when
> launched for the first time.  But last time I tried that it failed with
> insufficient memory (32Mb RAM, no swap), so I'm going without dumping
> for now.

It sounds like running Emacs on such a system is going to be pretty marginal in any case, but do you recall what part of it failed?  Finding the doc strings?  The actual dumping?

Ken

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

> On Dec 29, 2010, at 04:28, David Kuehling wrote:

>> I could dump Emacs in the target system, from a wrapper script, when
>> launched for the first time.  But last time I tried that it failed
>> with insufficient memory (32Mb RAM, no swap), so I'm going without
>> dumping for now.

> It sounds like running Emacs on such a system is going to be pretty
> marginal in any case, but do you recall what part of it failed?
> Finding the doc strings?  The actual dumping?

Ok, just recompiled emacs with dumping support, and running

  $ emacs -Q --batch --eval \
    '(dump-emacs "./demacs" "/usr/bin/emacs")'

on the NanoNote gave me:

  [..]
  Loading ediff-hook...
  Finding pointers to doc strings...
  Finding pointers to doc strings...done
  emacs: Can't allocate buffer for /usr/bin/emacs

So it wants to pull a full copy of the emacs binary into memory?
This problem I can workaround by changing the Linux overcommit setting,
but then dumping fails with another problem:

  $ echo "1" > /proc/sys/vm/overcommit_memory 
  $ emacs -Q --batch --eval \
     '(dump-emacs "./demacs" "/usr/bin/emacs")'

  [..]
  Finding pointers to doc strings...
  Finding pointers to doc strings...done
  emacs: Program segment above .bss in /usr/bin/emacs

So what's that supposed to mean?  

cheers,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Andreas Schwab]

David Kuehling <dvdkhlng@gmx.de> writes:

>   emacs: Program segment above .bss in /usr/bin/emacs
>
> So what's that supposed to mean?  

Probably that unexelf.c needs to grow support for some special MIPS
sections.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Some OpenWrt port related problems

[Eli Zaretskii]

> From: David Kuehling <dvdkhlng@gmx.de>
> Date: Sat, 01 Jan 2011 15:20:58 +0100
> Cc: rms@gnu.org, Emacs Dev <emacs-devel@gnu.org>
> 
>   Loading ediff-hook...
>   Finding pointers to doc strings...
>   Finding pointers to doc strings...done
>   emacs: Can't allocate buffer for /usr/bin/emacs
> 
> So it wants to pull a full copy of the emacs binary into memory?

It tries to mmap it, yes:

  old_file_size = stat_buf.st_size;
  old_base = mmap (NULL, old_file_size, PROT_READ | PROT_WRITE,
		   MAP_ANON | MAP_PRIVATE, mmap_fd, 0);
  if (old_base == MAP_FAILED)
    fatal ("Can't allocate buffer for %s\n", old_name, 0);

>   $ echo "1" > /proc/sys/vm/overcommit_memory
>   $ emacs -Q --batch --eval \
>      '(dump-emacs "./demacs" "/usr/bin/emacs")'
> 
>   [..]
>   Finding pointers to doc strings...
>   Finding pointers to doc strings...done
>   emacs: Program segment above .bss in /usr/bin/emacs
> 
> So what's that supposed to mean?

This message comes from this portion of unexelf.c:

   /* Fix up a new program header.  Extend the writable data segment so
      that the bss area is covered too. Find that segment by looking
      for a segment that ends just before the .bss area.  Make sure
      that no segments are above the new .data2.  Put a loop at the end
      to adjust the offset and address of any segment that is above
      data2, just in case we decide to allow this later.  */

   for (n = new_file_h->e_phnum - 1; n >= 0; n--)
     {
       /* Compute maximum of all requirements for alignment of section.  */
       ElfW(Word) alignment = (NEW_PROGRAM_H (n)).p_align;
       if ((OLD_SECTION_H (old_bss_index)).sh_addralign > alignment)
	 alignment = OLD_SECTION_H (old_bss_index).sh_addralign;

 #ifdef __sgi
	   /* According to r02kar@x4u2.desy.de (Karsten Kuenne)
	      and oliva@gnu.org (Alexandre Oliva), on IRIX 5.2, we
	      always get "Program segment above .bss" when dumping
	      when the executable doesn't have an sbss section.  */
       if (old_sbss_index != -1)
 #endif /* __sgi */
       if (NEW_PROGRAM_H (n).p_vaddr + NEW_PROGRAM_H (n).p_filesz
	   > (old_sbss_index == -1
	      ? old_bss_addr
	      : round_up (old_bss_addr, alignment)))
	   fatal ("Program segment above .bss in %s\n", old_name, 0);

I don't know enough about unexelf, nor about ELF in general, to tell
what this means or what to do about this, though.  Anyone?

Re: Some OpenWrt port related problems

[Ken Raeburn]

On Jan 1, 2011, at 10:34, Eli Zaretskii wrote:
>> From: David Kuehling <dvdkhlng@gmx.de>
>> Date: Sat, 01 Jan 2011 15:20:58 +0100
>> Cc: rms@gnu.org, Emacs Dev <emacs-devel@gnu.org>

>>  Finding pointers to doc strings...done
>>  emacs: Can't allocate buffer for /usr/bin/emacs
>> 
>> So it wants to pull a full copy of the emacs binary into memory?
> 
> It tries to mmap it, yes:

Contrary to the comment in unexelf.c saying "We do not use mmap because that fails with NFS." :-)

Perhaps we could reduce the memory requirement a bit by (in the ELF case) mapping the ELF header and read-only sections as shared, read-only data, so the process wouldn't need space from the OS for private modifications to those pages.  It would take a bit more work to do that, though.  (Strawman design: Map the whole file read-only and shared.  Parse the program headers and build a list of file-offset ranges that need write access.  Then map private copies of the necessary pages of the original file in over the read-only ones.)  Or perhaps some of the read-write regions could just be written out from the running process, ignoring the on-disk version; I haven't looked at how unexelf.c works very closely.

If enabling overcommit or swapping temporarily on the few systems where Emacs can't build gets us around the problem, I'm not sure if it's worth it, but if some OpenWrt hacker wants to experiment with it, I don't think it would be a bad change.


>>  Finding pointers to doc strings...done
>>  emacs: Program segment above .bss in /usr/bin/emacs
>> 
>> So what's that supposed to mean?
> 
> This message comes from this portion of unexelf.c:
> 
>   /* Fix up a new program header.  Extend the writable data segment so
>      that the bss area is covered too. Find that segment by looking
>      for a segment that ends just before the .bss area.  Make sure
>      that no segments are above the new .data2.  Put a loop at the end
>      to adjust the offset and address of any segment that is above
>      data2, just in case we decide to allow this later.  */
> 
>   for (n = new_file_h->e_phnum - 1; n >= 0; n--)
>     {
>       /* Compute maximum of all requirements for alignment of section.  */
>       ElfW(Word) alignment = (NEW_PROGRAM_H (n)).p_align;
>       if ((OLD_SECTION_H (old_bss_index)).sh_addralign > alignment)
> 	 alignment = OLD_SECTION_H (old_bss_index).sh_addralign;
> 
> #ifdef __sgi
> 	   /* According to r02kar@x4u2.desy.de (Karsten Kuenne)
> 	      and oliva@gnu.org (Alexandre Oliva), on IRIX 5.2, we
> 	      always get "Program segment above .bss" when dumping
> 	      when the executable doesn't have an sbss section.  */
>       if (old_sbss_index != -1)
> #endif /* __sgi */
>       if (NEW_PROGRAM_H (n).p_vaddr + NEW_PROGRAM_H (n).p_filesz
> 	   > (old_sbss_index == -1
> 	      ? old_bss_addr
> 	      : round_up (old_bss_addr, alignment)))
> 	   fatal ("Program segment above .bss in %s\n", old_name, 0);
> 
> I don't know enough about unexelf, nor about ELF in general, to tell
> what this means or what to do about this, though.  Anyone?

It appears to be looking for any higher-numbered segment in the old executable (despite the NEW_* macros, printf shows the numbers as being those from temacs) which ends at an address higher than where the BSS section started; it quits the loop when finding a normal (PT_LOAD) segment where the on-disk data ends on the page where the BSS section starts.  The latter is to find the segment where our former BSS storage gets added as initialized data.

It certainly makes sense to me that we don't want to find the BSS start within initialized data or other mapped-in segments; that would be inconsistent.  In fact, the whole region of former-BSS and dynamically allocated storage that we want to turn into initialized data shouldn't overlap with pre-existing initialized data or other mapped-in segments.  But why we don't check for the segment start addresses, or why the segment numbering matters, I'm not sure.  (Most of my experience with ELF format was in reading and writing object files, where the sections are all that matters and segments aren't defined, and I didn't deal with MIPS much.  And I'm dredging up decade-plus-old memories for this, so I'm a bit rusty.)  I assume there's some (official or de facto) standard to the segment ordering involved here.  Nor is it clear to me why simply shutting off the check, instead of adjusting it, is the right thing for executables with .sbss sections on SGI systems specifically.

I did a MIPS32 (Lemote laptop) GNU/Linux build of the latest alpha snapshot; no error was produced.  According to "readelf -e", temacs has no .sbss section.  The on-disk part of segment 4 does end at the start of BSS, and it's the segment with the highest memory address (not file offset) overall, so there's nothing mapped higher than the BSS.  So I'm afraid I can't help much, at least without more info (e.g., "readelf -e temacs", to see how it's different from mine).  And I'm not sure I know enough about the linker view on MIPS to help anyways.

Ken

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 2152 bytes --]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

> On Jan 1, 2011, at 10:34, Eli Zaretskii wrote:
>>> From: David Kuehling <dvdkhlng@gmx.de> Date: Sat, 01 Jan 2011
>>> 15:20:58 +0100 Cc: rms@gnu.org, Emacs Dev <emacs-devel@gnu.org>

>>> Finding pointers to doc strings...done emacs: Can't allocate buffer
>>> for /usr/bin/emacs
>>> 
>>> So it wants to pull a full copy of the emacs binary into memory?
>> 
>> It tries to mmap it, yes:

> Contrary to the comment in unexelf.c saying "We do not use mmap
> because that fails with NFS." :-)

> Perhaps we could reduce the memory requirement a bit by (in the ELF
> case) mapping the ELF header and read-only sections as shared,
> read-only data, so the process wouldn't need space from the OS for
> private modifications to those pages.  

Well, if those pages are not modified, no memory is needed from the OS
anyway (i.e. copy-on-write/lazy copy).  Just that linux VM manager seems
to usually check whether it has enough pages just-in-case.

Similar problems seem to crop up with fork();exec() inside emacs.  So
enabling overcommitting on the NanoNote may be a good thing in general.

[..]
> I did a MIPS32 (Lemote laptop) GNU/Linux build of the latest alpha
> snapshot; no error was produced.  According to "readelf -e", temacs
> has no .sbss section.  The on-disk part of segment 4 does end at the
> start of BSS, and it's the segment with the highest memory address
> (not file offset) overall, so there's nothing mapped higher than the
> BSS.  So I'm afraid I can't help much, at least without more info
> (e.g., "readelf -e temacs", to see how it's different from mine).  And
> I'm not sure I know enough about the linker view on MIPS to help
> anyways.

  $ readelf -t /usr/bin/emacs

  There are no sections in this file.

:) 

Could it be that 'sstrip' (that's no typo, it's not vanilla 'strip')
used for openwrt packages causes collateral damage here?  Emacs won't be
the only package effected.

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Stefan Monnier]

>> (e.g., "readelf -e temacs", to see how it's different from mine).  And
[...]
>   $ readelf -t /usr/bin/emacs
                         ^^
                          t

-- Stefan

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> (e.g., "readelf -e temacs", to see how it's different from mine).
>>> And
> [...]
>> $ readelf -t /usr/bin/emacs

In the openwrt port I just copy temacs over to emacs, as dumping is not
possible.  so /usr/bin/emacs _is_ temacs (although the openwrt
packageing scripts seem to mangle it with sstrip).

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Ken Raeburn]

On Jan 2, 2011, at 08:53, David Kuehling wrote:
> Well, if those pages are not modified, no memory is needed from the OS
> anyway (i.e. copy-on-write/lazy copy).  Just that linux VM manager seems
> to usually check whether it has enough pages just-in-case.
> 
> Similar problems seem to crop up with fork();exec() inside emacs.  So
> enabling overcommitting on the NanoNote may be a good thing in general.

Eh.. I've never been convinced that it's a good thing.  I like the fact that mmap/malloc can fail, and give you a chance to recover, rather than simply having a process blown out of the water when it turns out that a page isn't actually available after all.  But that's just me....

>  $ readelf -t /usr/bin/emacs
> 
>  There are no sections in this file.
> 
> :) 
> 
> Could it be that 'sstrip' (that's no typo, it's not vanilla 'strip')
> used for openwrt packages causes collateral damage here?  Emacs won't be
> the only package effected.

Okay, then you are doing something different...  I don't know how unexelf.c is going to handle a file with no section headers.  As best I recall, they're not critical for execution, but unexelf.c may be making additional assumptions based on how other systems tend to operate.  Ideally, I think it should be possible to just extent the loadable data sections, but that's not how unexelf.c operates.  If you can bypass 'sstrip' for a package, or just one executable in the package (emacsclient should be fine to strip, for example), that might fix the problem and allow you to have it dump during installation.

Ken

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 2352 bytes --]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

> On Jan 2, 2011, at 08:53, David Kuehling wrote:
>> Well, if those pages are not modified, no memory is needed from the
>> OS anyway (i.e. copy-on-write/lazy copy).  Just that linux VM manager
>> seems to usually check whether it has enough pages just-in-case.
>> 
>> Similar problems seem to crop up with fork();exec() inside emacs.  So
>> enabling overcommitting on the NanoNote may be a good thing in
>> general.

> Eh.. I've never been convinced that it's a good thing.  I like the
> fact that mmap/malloc can fail, and give you a chance to recover,
> rather than simply having a process blown out of the water when it
> turns out that a page isn't actually available after all.  But that's
> just me....

Yes don't like it too much either, just a workaround as nobody is going
to fix the mmap() in the near future :)  Plus I just witnessed GNU Octave
use 64MB of virtual memory on the 32MB RAM nanonote, so over-allocating
memomory seems to be a common disease nowadays.

>> $ readelf -t /usr/bin/emacs
>> 
>> There are no sections in this file.
>> 
>> :)
>> 
>> Could it be that 'sstrip' (that's no typo, it's not vanilla 'strip')
>> used for openwrt packages causes collateral damage here?  Emacs won't
>> be the only package effected.

> Okay, then you are doing something different...  I don't know how
> unexelf.c is going to handle a file with no section headers.  As best
> I recall, they're not critical for execution, but unexelf.c may be
> making additional assumptions based on how other systems tend to
> operate.  Ideally, I think it should be possible to just extent the
> loadable data sections, but that's not how unexelf.c operates.  If you
> can bypass 'sstrip' for a package, or just one executable in the
> package (emacsclient should be fine to strip, for example), that might
> fix the problem and allow you to have it dump during installation.

The best solution will be to use strip instead of sstrip, and I think
the NanoNote firmware is going to use that very soon (since more sstrip
problems have been cropping up recently).

Going to post how that turns out.

cheers,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

>>>>> "David" == David Kuehling <dvdkhlng@gmx.de> writes:

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

>> On Jan 2, 2011, at 08:53, David Kuehling wrote:
>>> Could it be that 'sstrip' (that's no typo, it's not vanilla 'strip')
>>> used for openwrt packages causes collateral damage here?  Emacs
>>> won't be the only package effected.

>> Okay, then you are doing something different...  I don't know how
>> unexelf.c is going to handle a file with no section headers.  As best
>> I recall, they're not critical for execution, but unexelf.c may be
>> making additional assumptions based on how other systems tend to
>> operate.  Ideally, I think it should be possible to just extent the
>> loadable data sections, but that's not how unexelf.c operates.  If
>> you can bypass 'sstrip' for a package, or just one executable in the
>> package (emacsclient should be fine to strip, for example), that
>> might fix the problem and allow you to have it dump during
>> installation.

> The best solution will be to use strip instead of sstrip, and I think
> the NanoNote firmware is going to use that very soon (since more
> sstrip problems have been cropping up recently).

> Going to post how that turns out.

Ok here we go.  Recompiled with strip instead of sstrip.  Now Emacs is
killed by the OOM killer when attempting to dump :(

  emacs -Q --batch --eval \
        '(dump-emacs "./demacs" "/usr/bin/emacs")'
  [..]
  Loading vc-hooks...
  Loading ediff-hook...
  Finding pointers to doc strings...
  Finding pointers to doc strings...done
  Killed

(the kernel log contains a message about "out of memory: kill process
652...")

Now I'm out of ideas.

Thanks for the help, cheers,

David
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Some OpenWrt port related problems

[Richard Stallman]

    I could dump Emacs in the target system, from a wrapper script, when
    launched for the first time.  But last time I tried that it failed with
    insufficient memory (32Mb RAM, no swap), so I'm going without dumping
    for now.

If it has enough memory to run Emacs, it has enough to dump Emacs.
The latter requires all the space needed for the former.


-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org