Re: Preview: portable dumper

[Daniel Colascione <dancol@dancol.org>]

On 02/19/2018 05:30 AM, Ken Brown wrote:
> On 2/17/2018 7:02 PM, Daniel Colascione wrote:
>> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>
>>     On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>>      > On 02/15/2018 05:56 PM, Ken Brown wrote:
>>      >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>      >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>     [...]
>>      >>>     I just tried to build on 64-bit Cygwin, and the build 
>> fails as
>>      >>> follows:
>>      >>>
>>      >>>     [...]
>>      >>>     Dumping under the name bootstrap-emacs.pdmp
>>      >>>     dumping fingerprint:
>>      >>>     
>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>      >>>     Dump complete
>>      >>>     Byte counts: header=80 hot=13187392 discardable=119424
>>     cold=9086640
>>      >>>     Reloc counts: hot=919268 discardable=5790
>>      >>>     make -C ../lisp compile-first
>>     EMACS="../src/bootstrap-emacs.exe"
>>      >>>     make[2]: Entering directory
>>      >>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>      >>>     emacs: could not load dump file
>>     "../src/bootstrap-emacs.pdmp": out
>>      >>>     of memory
>>      >>>
>>      >>>     There's probably some obvious explanation, but I don't see
>>     it at the
>>      >>>     moment.
>>      >>>
>>      >>>
>>      >>> I'm not entirely surprised to see Cygwin fall over here. We 
>> could
>>      >>> just use the Windows memory mapping functions directly, but I'd
>>      >>> prefer to stick with the POSIX API if we can make it work.
>>      >>
>>      >> I agree.
>>      >>
>>      >>> Any idea where in the mmap sequence we fail?
>>      >>
>>      >> I haven't looked at the code yet, so I don't understand the
>>     question.
>>      >> If you give me some guidance, I'll try to investigate.
>>      >
>>      > Thanks. I think the trouble must be in dump_mmap_contiguous. We
>>     report
>>      > all errors from this function as "out of memory".
>>     dump_mmap_contiguous
>>      > takes an array of mapping descriptors (think ELF segments or
>>     something)
>>      > and maps them all into a single contiguous region of virtual
>>     memory. On
>>      > POSIX systems, we reserve a chunk of address space with a big
>>     PROT_NONE
>>      > anonymous mapping, then carve it up into the separate mappings
>>     that we
>>      > really want. Windows doesn't support atomic mmap replacement, so
>>     Cygwin
>>      > has to emulate it, and I wouldn't be surprised if something's 
>> going
>>      > wrong in this emulation.
>>
>>     You're right that the problem is in dump_mmap_contiguous.  I stepped
>>     through it in gdb and found the following:
>>
>>     1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>>
>>     2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is
>>     called, which calls dump_map_file_posix with protection =
>>     DUMP_MEMORY_ACCESS_READWRITE.
>>
>>     3. This calls mmap at line 4219 with mem_prot = PROT_READ | 
>> PROT_WRITE
>>     and mem_flags = MAP_PRIVATE | MAP_FIXED.
>>
>>     4. This mmap call fails.
>>
>>     I don't know if this is a bug in Cygwin's mmap or simply a limitation
>>     that has to be worked around.  Should I take this to the Cygwin 
>> mailing
>>     list, or do you have other ideas as to how to proceed?
>>
>> I'd definitely let the Cygwin people know. If the problem is that 
>> Cygwin doesn't properly support atomic map replacement, I think we can 
>> work around the problem by using the same unmap-before-remap approach 
>> we use for the native Windows code, but keep using the POSIX memory 
>> functions. I can come up with a patch blind, but you'd be in a better 
>> position to iterate quickly.
> 
> OK, here's Corinna's answer:
> 
>    https://cygwin.com/ml/cygwin/2018-02/msg00202.html
> 
> It seems like she's suggesting the same unmap-before-remap approach that 
> you mentioned.

Thanks. I've pushed a fix, I think. Try it now.