From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ken Brown Newsgroups: gmane.emacs.bugs Subject: bug#50666: 28.0.50; Fix native compilation on Cygwin Date: Thu, 23 Sep 2021 13:13:05 -0400 Message-ID: References: <9f20194e-b1ba-9417-4f18-caa1d80b5568@cornell.edu> <01a89ba6-2786-df04-0181-069b50a70331@cornell.edu> <835yux5dn1.fsf@gnu.org> <87bl4pf3s1.fsf@Otto.invalid> <83tuih3uvr.fsf@gnu.org> <877dfcg5tu.fsf@Otto.invalid> <83pmt44vn1.fsf@gnu.org> <83mto84r9l.fsf@gnu.org> <83fsu04mai.fsf@gnu.org> <1a5e01a2-2247-2f68-82f6-2075577e02b6@cornell.edu> <837dfc4hi1.fsf@gnu.org> <4ae8067f-55b2-d243-66f3-f76493095a39@cornell.edu> <83o88jvity.fsf@gnu.org> <8e8e74ce-0deb-bcdc-d298-be2e9d4636d7@cornell.edu> <83bl4juu2c.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33683"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 Cc: Stromeko@nexgo.de, 50666@debbugs.gnu.org, akrl@sdf.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Sep 23 19:25:09 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mTST4-0008Vl-B2 for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 23 Sep 2021 19:25:06 +0200 Original-Received: from localhost ([::1]:47500 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mTST3-0004y3-9k for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 23 Sep 2021 13:25:05 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:42788) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTSIM-0004CI-TP for bug-gnu-emacs@gnu.org; Thu, 23 Sep 2021 13:14:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:44215) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mTSIM-0007vF-J2 for bug-gnu-emacs@gnu.org; Thu, 23 Sep 2021 13:14:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mTSIM-0007v7-Ex for bug-gnu-emacs@gnu.org; Thu, 23 Sep 2021 13:14:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Ken Brown Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 23 Sep 2021 17:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50666 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 50666-submit@debbugs.gnu.org id=B50666.163241719630375 (code B ref 50666); Thu, 23 Sep 2021 17:14:02 +0000 Original-Received: (at 50666) by debbugs.gnu.org; 23 Sep 2021 17:13:16 +0000 Original-Received: from localhost ([127.0.0.1]:55760 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTSHb-0007tq-TG for submit@debbugs.gnu.org; Thu, 23 Sep 2021 13:13:16 -0400 Original-Received: from mail-bn7nam10on2131.outbound.protection.outlook.com ([40.107.92.131]:15247 helo=NAM10-BN7-obe.outbound.protection.outlook.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTSHa-0007tc-5t for 50666@debbugs.gnu.org; Thu, 23 Sep 2021 13:13:14 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Vo6/y1XtwdgSHKo/t+4YxtWanHtR/wT7iaA2VPq7ihzAYVmo6AERu2dIbUPP7rEng6brB76/9rdhj/lxnddMEuZaFXpEShslsGwEIBBveMa0FLkPZ72pc/bb6jHez51XJQeQGWWq0D6LLIAMugD1V8HS3WzqifcOpfrSR485OXKX1RAtuztMGCN/BZL1Ny6BAlosuwjPffyP3Sd6+2C7b2ixlehTCgCUM4Fd6s9XvtyUNT4+GjTYN92tQHgPz/08N2hQErcLo1piYeIHfg6RRA2mQWL9RSYo3uN0mwoY6cWcvJDpk+TXqfLy0eclILZRvF9aqSwsafAuy0Lca4q+GA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=36YSoJ6jPb3pQo6E+NuZoMJD/R747GyaTsw2qUg00nA=; b=chuUmG7IDsNT0xWY1C3fZUK2nxEpkyx2aIUql8x3ih2BFuRgxLP8ATYyR/1Xgjb5gYUsvTT/1BTNu0OP5Foc6SB4sChd2osPax2J6xkaWOgE7ykHIdV92UiISYVVOzO3AiAvOzvaW0o066kvm5KrzJFcJ2vEathBITocycY4E6je5QZrRwIDWMjSACqKsPh7gUcvLQAWFJ4noIUyTJ0S3e+h4A0fld2XBXStCocE+D/F5SDZOkW3gMh2jCgkS7NntgVGXJ2TEJz63ugtuMxnk2iG8HnQLCqs45nD0AN+IsL+N3cl20hDo7zDDgUJuR7bRbn4oXCySTjUfAUJbLWJvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=36YSoJ6jPb3pQo6E+NuZoMJD/R747GyaTsw2qUg00nA=; b=RsxlR0HwEDjwbgY9W61YNt5cTKDreJ6ceajXbfp5wKj8RuNIBC5fr3g0ztkEdLJoBSpvlCO9QJxUo30UQJiUhAe1LesTuMhyp+hUpo7InWhnnL9gp3Gd4Tfdjt1UBHmqzVTCY695BPrkXuxedIuSVpPvjqaUA5m3YHrHOgluUxk= Authentication-Results: debbugs.gnu.org; dkim=none (message not signed) header.d=none;debbugs.gnu.org; dmarc=none action=none header.from=cornell.edu; Original-Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN6PR04MB0354.namprd04.prod.outlook.com (2603:10b6:404:99::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.13; Thu, 23 Sep 2021 17:13:07 +0000 Original-Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66%6]) with mapi id 15.20.4523.022; Thu, 23 Sep 2021 17:13:06 +0000 In-Reply-To: <83bl4juu2c.fsf@gnu.org> Content-Language: en-US X-ClientProxiedBy: CH2PR11CA0029.namprd11.prod.outlook.com (2603:10b6:610:54::39) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) Original-Received: from [IPv6:2603:7081:7e3f:3419:8038:2245:c6fc:6fdc] (2603:7081:7e3f:3419:8038:2245:c6fc:6fdc) by CH2PR11CA0029.namprd11.prod.outlook.com (2603:10b6:610:54::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.15 via Frontend Transport; Thu, 23 Sep 2021 17:13:06 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b952a529-1f4e-4de3-7594-08d97eb56903 X-MS-TrafficTypeDiagnostic: BN6PR04MB0354: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: S7l1VB99f+HHDX4Dm3krk2ATue6WNNjUCjjRTMxH+bgPBVSNm1mWENbMeE0WVEf0asiARdtzmQ7wrlTMdIe05pXN7zwGz3WGA5ejVX1mDp6QyWES6sWzlNA/1lyEbvTaWJDLwlWFtB7BjrhGVtwz69ioHzxoo4q/hFFKjuLtoU8JXtIIRRMM2iByLM9zI3vc6u8k8F8Fp4pORLvKbnLlVdOGPOptuguw3Nyca50XnDunj0znHEIBEL+4I+TY/NFJU3jVivtUvFYpapqpQ36Vay9iW/pjwbePWUw1vqpxIsZC2LgSOgFiiwYVX5VkPHm+Ny0lzFyrus8eHXgJgrLmprzMgfjEH+gYIw5agPiDm7Up02kr7eHNOiG3nhMnhm1lUR32nY5XXa45iUzYR4dWBAVD6vLvkdcWCi+MJfAJsLI7nGiBBQ7LZ94yit25kQwy6Hk4wNLIMVm/FEsnX7ICVjzn25e3/AMvFvztUBXT0ng2GMYx8LNhcBPMaB7///LUkPbESoZWEqlsrROJQRXGjmIdSuXFrq71PzyZiwsHJZVIy2WGGFf9aEgVBtJvPOyeCA9I6sDvxu2CzoAyz1tyTq+OJOPhaEEcAXAckiCpBb7UEeNu888sYgaZfK2UtvF5Uwsi/aPcFJGA016FyBKR0fifuUEnJQPtHsnFpMA/tgZ80xl2eF4ebpo6dZ1Pdx4+AwONkmvDYACYTLN2lflaBsYvHudPjtUhouGeQG9/o/3v8t85AjJWuvTpilQZB 6ChDExm8aL3P766AOcRN0Qy7uEKHH+ihV5TaIAcWsGteRZyOiwlgQ3M+vP4AnqPkQBX X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(5660300002)(2616005)(31696002)(38100700002)(36756003)(83380400001)(4326008)(86362001)(316002)(508600001)(66946007)(66476007)(66556008)(2906002)(186003)(6486002)(75432002)(966005)(53546011)(31686004)(6916009)(8936002)(8676002)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Zt5+1HJurtd8oV4Gtz9rTvLg/NUGhiMTvsjG7BiJF3XlwacLWAuQ6q7kINmMK4K4kFNhElLtXYhxHl7gedyY5v+wdX5aHMDa8AU8+n2EH+DJFqmCtgF9h/e2/EefaWO7kWQVLb9seNZGOwD4ca9ybQIkJQD/3krLd02VZV9K3M09MbyznRCBRyVuSUi2fJXb74raEOYM7eenX5LRfmRHhA625HTEiOhwnd+g71pDx/DzhNo3GjGajJ0j8vuizzgfhMvmLbvXmB2oBd+ebc/Ym7+QL7MUHy3ysBr9wQj1LFm5PRj8HyI4c7O/PYI5oHhNKtC3MeOa7K6c24MXSWsyjCDX68m7Z3bfsqySRxI5hUzcXqjvkoE/5d7gpoeVtytCuZncmz9vzQDPSzC8xtat/ICh7gYYwUpPtWCHA+srA4lKPORilp+kAegn2z9r5ilGfoSgpiwcc7/1L8TgR2NsrF1sAvhNOFLTg8+dtmWu6e9h0GflHV4SGTkiLs6hkULPeZGITeDEf+zAGczXPC+IIL0FlHl223Np9ZIDsjfwDV1uYg5wIjq7rO4cf+hAIrK2RzEO4uLuBLxyi4O6R6elCLk1VSESGR6vzI4o+UxA2cmn4rlKKVWfqnLtTuhHVufOLyEGe+1Hx3WTkUivvMpVI08uJrM589KKTNqbwM3hBoj6jPN7W/f1ofsOoRC9gTISxMz4DtgMXtv+f3nN3szlliyQ/Cpkh1btrO5uGf1X6eDmgbmEhjZQlIXfh4 4N/iHQUBVrvzYh7yBZSILxH7QS5qLlEBEXH1P/khXzFyu4GXnabUvUklOq3LOxWcNLhptPrWYn9LrzFL9Ipe/o4MIL1mMnUWnI X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: b952a529-1f4e-4de3-7594-08d97eb56903 X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2021 17:13:06.6939 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AbFCOg97gc4Mu44pItYRk9re1uQhOBHJd25XBc4Rkbclp228AQ14KRy8sRuYF+Ca97rN+wh/V6o//0/XCzv/mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR04MB0354 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:215207 Archived-At: On 9/23/2021 12:37 PM, Eli Zaretskii wrote: >> Cc: Stromeko@nexgo.de, 50666@debbugs.gnu.org >> From: Ken Brown >> Date: Thu, 23 Sep 2021 10:20:19 -0400 >> >>> Is it really necessary to rebase the *.eln files before each startup? >>> Isn't it enough to rebase each of the .eln files just once, when it is >>> produced? If indeed this is needed every time, can you explain why? >> >> We have to distinguish between system libraries and user libraries. Libraries >> installed by Cygwin packages are system libraries. The *.eln files in >> ~/.emacs.d/eln-cache are user libraries. Cygwin maintains a database of all >> system libraries and their base addresses. Whenever a Cygwin package is >> installed or updated, Cygwin rebases all system libraries and updates the database. >> >> Each time Emacs starts, it has no way of knowing whether the system libraries >> have been rebased since the last time Emacs was run. So the user's *.eln files >> could now have base address conflicts with system libraries. Rebasing the *.eln >> files fixes this problem. > > What do you mean by "system libraries" here? Does it, for example, > include the DLLs distributed in the Cygwin port of libpng or libjpeg? Yes. > Or does that include only the basic libraries: the Cygwin DLL, the C > runtime, etc.? > >>> The non-preloaded *.eln files are all loaded by native-elisp-load, so >>> I guess the rebase should be launched from there? The preloaded *.eln >>> files are loaded in pdumper.c:dump_do_dump_relocation, but do we need >>> to support non-rebased preloaded *.eln files? >> >> The preloaded *.eln files will be installed by Cygwin's package manager when >> emacs is installed. They are therefore system libraries and are automatically >> rebased as needed. > > If the only problem is with non-preloaded *.eln files, why not rebase > them on the fly, when they are loaded. That is, run the 'rebase' > command from the native-elisp-load function, before it actually loads > the file. User libraries are never loaded during startup, only when > some Lisp requires them. Good idea. >>> When an updated .eln file is produced for .eln that is loaded into >>> some running Emacs, Emacs on Windows renames the original .eln to >>> avoid a similar problem. >> >> I hadn't thought of this issue. We may have to use a similar technique... >> >>> Can't you use the same technique, to avoid >>> the need of rebasing on each start? >> >> ...but it wouldn't eliminate the need for rebasing at each start for the reasons >> explained above. > > Perhaps that need could be eliminated after all, see above. > >>> Please note that users could >>> place *.eln files in unusual locations (and customize >>> native-comp-eln-load-path to reflect that), so finding _all_ of the >>> relevant *.eln files from a shell script might not be easy. In fact, >>> even without customizing the load-path, I don't think I understand how >>> will that script you propose know where to find all the *.eln files. >> >> The current proposal that Achim and I are looking at would require each user to >> maintain a file ~/.config/rebase/dynpath.d/emacs containing a list of >> directories where the .eln files can be found. By default, this file would >> contain one line, which is the path to the standard eln-cache directory. Users >> who customize native-comp-eln-load-path would have to modify that file accordingly. > > That's tough on users, because Emacs by default automatically compiles > every .el file it loads into .eln, if there's no up-to-date .eln file > already, and the compilation runs in the background (by forking > additional Emacs sub-processes that run in batch mode). In addition, > the native-compilation process sometimes decides that it needs to > create a special "trampoline" .eln file (if you want to know why, I'm > sure Andrea can explain) that correspond to parts of the *.el files. > The upshot is that users may not even be aware that new *.eln files > have been created as part of their session, and may not know their > names. Unless the automatic rebase process, which runs from > native-elisp-load, will also update the file in dynpath.d, I don't see > how users could maintain such a database by hand in practice. > > There's one more aspect that you should be aware of. A single file > FOO.el could give birth to several different .eln files, for example > if they are compiled by different Emacs binaries and/or from different > source directories and/or from somewhat different versions of FOO.el. > For example, I now have 3 different versions of .eln files > corresponding to window.el, in the same directory: > > window-0d1b8b93-3370bedb.eln > window-0d1b8b93-7d08b7b4.eln > window-0d1b8b93-f8fc9683.eln > > This makes the job of maintaining the database by hand even harder and > more error-prone. > >> Finally, as a side note, I don't think it would be a tragedy if this just turns >> out to be too complicated and we have to disable native compilation on 32-bit >> Cygwin. The Cygwin home page at https://cygwin.com/ already contains the following: >> >> Address space is a very limiting factor for Cygwin. These days, a full >> 32 bit Cygwin distro is not feasible anymore, and will in all likelihood >> fail in random places due to an issue with the fork(2) system call. >> >> Therefore we recommend using 32 bit Cygwin only in limited scenarios, with >> only a minimum of necessary packages installed, and only if there's no way >> to run 64 bit Cygwin instead. > > My point is that maybe we should make that decision already, before > burning too much time and energy on it. You might be right. I wasn't aware of all the complications you mentioned above. We still need to do something for 64-bit Cygwin. Even though address collisions are unlikely they could still happen theoretically. But there might be a much easier solution that doesn't necessarily require rebasing. For example, Achim mentioned earlier the possibility of marking the eln as ASLR w/ high-entropy and large address aware. > Maybe you should ask on the > Cygwin list whether somebody will object to making 32-bit Cygwin Emacs > a second-class citizen. Well, 32-bit Cygwin is already a second-class citizen, so we might just have to do that whether someone objects or not. But I'll continue the discussion with Achim on the cygwin-apps list before making a final decision. Thanks for all your comments and suggestions. Ken