From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ken Brown Newsgroups: gmane.emacs.bugs Subject: bug#50666: 28.0.50; Fix native compilation on Cygwin Date: Thu, 23 Sep 2021 10:20:19 -0400 Message-ID: <8e8e74ce-0deb-bcdc-d298-be2e9d4636d7@cornell.edu> References: <9f20194e-b1ba-9417-4f18-caa1d80b5568@cornell.edu> <01a89ba6-2786-df04-0181-069b50a70331@cornell.edu> <835yux5dn1.fsf@gnu.org> <87bl4pf3s1.fsf@Otto.invalid> <83tuih3uvr.fsf@gnu.org> <877dfcg5tu.fsf@Otto.invalid> <83pmt44vn1.fsf@gnu.org> <83mto84r9l.fsf@gnu.org> <83fsu04mai.fsf@gnu.org> <1a5e01a2-2247-2f68-82f6-2075577e02b6@cornell.edu> <837dfc4hi1.fsf@gnu.org> <4ae8067f-55b2-d243-66f3-f76493095a39@cornell.edu> <83o88jvity.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="14552"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 Cc: Stromeko@nexgo.de, 50666@debbugs.gnu.org To: Eli Zaretskii , Andrea Corallo Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Sep 23 16:21:26 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mTPbJ-0003b9-Mq for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 23 Sep 2021 16:21:26 +0200 Original-Received: from localhost ([::1]:45824 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mTPbI-000334-9B for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 23 Sep 2021 10:21:24 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:49562) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTPax-0002kN-09 for bug-gnu-emacs@gnu.org; Thu, 23 Sep 2021 10:21:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:43767) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mTPaw-0007Gc-Np for bug-gnu-emacs@gnu.org; Thu, 23 Sep 2021 10:21:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mTPaw-0006kq-EB for bug-gnu-emacs@gnu.org; Thu, 23 Sep 2021 10:21:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Ken Brown Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 23 Sep 2021 14:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50666 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 50666-submit@debbugs.gnu.org id=B50666.163240683825906 (code B ref 50666); Thu, 23 Sep 2021 14:21:02 +0000 Original-Received: (at 50666) by debbugs.gnu.org; 23 Sep 2021 14:20:38 +0000 Original-Received: from localhost ([127.0.0.1]:55313 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTPaX-0006jl-Lk for submit@debbugs.gnu.org; Thu, 23 Sep 2021 10:20:38 -0400 Original-Received: from mail-sn1anam02on2100.outbound.protection.outlook.com ([40.107.96.100]:63559 helo=NAM02-SN1-obe.outbound.protection.outlook.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTPaS-0006jO-Mn for 50666@debbugs.gnu.org; Thu, 23 Sep 2021 10:20:36 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cYtDeqXz66iVKW4v+oWdoePFTR+KWG0hEMzo+A3jRZlidA3vLS3nyrw5Vj4DFFBpwvAHtMfv4h1JfD3n39BqcfB1aMIEM+wk2NmPuib+thVULpZb6L7Qz20srCxq5usapTGDfFQryw0GtYe7nuxBoUtCkiN9G9E3Ipe24UGd/Ea1a+b77wMp+Bf2u4uy+PJiSujM/VICYp5be0KhuGSSwnlrbJo27j9F7SkWog31kxsbelsqzf6Y6CKnT6kL1/XlAC5AD1gJkS8VbJX+pvHNkEKnJvoQY5+IUMndhjyMTalos30L5/YDKofJ5pXIzgq/rU2k/YspNNhm/NWWEyT0pQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=4epyYpGxvGq3Wc3A0y9mOWhctXT08WzeZiXSLQLO/qg=; b=R0cKUgtKrpAoIw4x9MjHMh+MX4160aU9lKxFC3bs/rEEYCC1VkptZWm+7f7UjJVboClrmOy5wmKa5yQInpSthIZ7TJxUShHmCq9DsXuGyvnQ4PXCkRJIUzzOA5B5scXyE43SUYwmMEgrDvo9El4JG7SofkySRakV5tUzZY8rZI26SRSqRAa8s2NO+v2MsYI/tIrh7tigCrVF9PEjpVhdTqO07h5AZzUEqmPd3BglSGah9QbEDNIn8zRz/jkaYa28HE+k/JlrXA5W8lQ/cvNTogyE+K89PsIpECNJ94ydSfgEkQbultGODYhVb6fqUBUSObgaCd2Qenl6xUdqFcFFKQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4epyYpGxvGq3Wc3A0y9mOWhctXT08WzeZiXSLQLO/qg=; b=IDn3Mud5AT1QdmcxmAgv3DnZkgHz5ILCmNC/6O36GLeBTKpe6kuG1cP+v+Xtieju7zw+G+Pb4xUOJDsH8s7QQJXjGIadFH2ZI38/XodOFZASYpA12mtddL+celwZN6X5RUunoBStasQIZnC3JjGB9FLPC2pJRhU5td83UK0tU2A= Authentication-Results: debbugs.gnu.org; dkim=none (message not signed) header.d=none;debbugs.gnu.org; dmarc=none action=none header.from=cornell.edu; Original-Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN8PR04MB5715.namprd04.prod.outlook.com (2603:10b6:408:74::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.13; Thu, 23 Sep 2021 14:20:22 +0000 Original-Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66%6]) with mapi id 15.20.4523.022; Thu, 23 Sep 2021 14:20:21 +0000 In-Reply-To: <83o88jvity.fsf@gnu.org> Content-Language: en-US X-ClientProxiedBy: CH0PR03CA0392.namprd03.prod.outlook.com (2603:10b6:610:11b::31) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) Original-Received: from [IPv6:2603:7081:7e3f:3419:8038:2245:c6fc:6fdc] (2603:7081:7e3f:3419:8038:2245:c6fc:6fdc) by CH0PR03CA0392.namprd03.prod.outlook.com (2603:10b6:610:11b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.15 via Frontend Transport; Thu, 23 Sep 2021 14:20:21 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fb0d26a9-0fa8-493b-e4e4-08d97e9d46ff X-MS-TrafficTypeDiagnostic: BN8PR04MB5715: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: k0KYJxykEgPlR21W/po38N56jis5d/yvrWgFPqwE2pAS1c/AGCogUol8ZgcILJwozayMEVdP6Nhn5sqhIrq2Sp81j0kvVEufcpdGBj44jIvsF70HeLrI9/cU5sNGdd3o8YcxsRvxxOgkwrw2FzKr78eq9b0bjO1kqsXAKQIFu9RQ4842XFtS7Vav3XcsQLULIwgFFUcbgcgG5VQrXZA2qQoIULWXRthhyV/mXlEdGloNO/UYR8ZPOCkf0ErMTyTYKHHQeYhE5TobT5je2QZvoWuQH+fAB+FWgseWePibQmRy+pkkxBKRm4v2WgiRxFlWoPAzDp0WaT/GK8Ty0vfWGhMtH1NHiflZZgN+VyhIehRpHFauimP4OgMBMF+omrgEgPn8IsjtFOznf+JQERJ30AZSiwppK9cvd91lu3nu3MD3NBhUtn3VvfEBgyoUaHHdsDCeSX/J63LwhuP1XU91/h9sv+PsNJ1YyVoP6roCeMaa1evzIYL0hyr2EEa1+4MKaL8Xt1S2WHAT4bDPOEvdrQbDm4PwGb8rGZCIkV5gmuQmBs0dbNei+3/WUYfCfEHkB/+ym43HwbgqVT3XpFgc+AZI/5sIcy4cwzd5BjuWiAmWdC+5hYSydnq+3hZySXM/R63ZBhcOBKhqmzx8GjKyzWhUSPpKDVyWiPrsiuUDjjMCbM2zBSlmR662R394I0MSXUS9fRmFpOz8chjiUTQJ5a84jh8oCa00sd7rBN4P3WhM2p3bpH8VeZLz9s7lv fs9+2WbZQPMRPvimDrMxOmenaZ6pOyhh/VyXhMyxKzs9mhPwQbq3r6XGMoG1htWWd8w X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(84040400005)(5660300002)(2616005)(31686004)(38100700002)(316002)(966005)(8676002)(66556008)(66476007)(508600001)(66946007)(53546011)(4326008)(31696002)(8936002)(110136005)(86362001)(2906002)(36756003)(75432002)(83380400001)(6486002)(186003)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: B6XNwdXiaeVPylQON97BQbKg0xbAd8Xgcp4pxXiCahtNiWwmiXs6aL3ptPsWAnE35heGIgZl5rGJqYlaeb2GECJ7OcKSCVXwd18O/6C80Llw7/ae1xc3x0wcugh/PtXWeCMl/SGyQbfqkichhdpE4f7L/76POR4fleVjZ7afpEJ76fTa3/wdBExkOfK33QKBQQurii7oygE+oFCLPQwLB4lZ0fZKQHJnzRx5nmRfTSIFyjcg4KORaAnlKZAxd1bnNCnmuX8/o7hV31cEe4Ba/dcnCp3dr9w/2N7HYRy7dItSxGANK+kUIvhUmSuA71N/StNtC/GWSFkhTR1rf4QyyClMKz5DgEwP7mfvNiAceas1PJNDCBEQ6hoY3Np2dt6jh2IiV8CitE/QLg7nOAhUEm7+M/pe5y0Ttj6NhB83MHI5jRm9C+icl6X3UA4rpofQuHzCHn7Qm15MZfLADrKd7aewOdrnXwLSQp57qmaWPme/Sk2n5cO/PPaRHeQ+5FTUqYCPyH5YZlgDm2xK2VxwFwgvMzAinmxw+IKYvB47kqi91LhseXIYAm5dtF7Kyr1UUegsHsEnN9NqqZfT2I/xQeEIczWjyFN7vo/TyZOR6XnmI1LjPvSjmXDh0zsbNBxkTUOt4DT+0xPfOy8fG9HvXZMzd1gyValHCXYXYMATGb0MIvV0/mYrysqCRy+112fdMMa0oL2g0y5bFTRGggT9TcmbR7Qseb+J8sbpkBL9aYMpNvi5Mx6Xgc7PV1 6yu8EbcIcxZNpCiboM4/UzoAG9aipPJ/rYuvPWw0UtMScpmFfWem0f3h/Npb9uR+ATNfF/+nq8Bv/5OzJh9yIMf8kSCsbyFa3M X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: fb0d26a9-0fa8-493b-e4e4-08d97e9d46ff X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2021 14:20:21.8265 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XCIQO6MYSWtEVDVM6KfxDXRQ+okpbke5uICCQvwAxJJT8KlIvn3de9ES9GLmpAvgBoq2e9nxlXJwETuDhoUI2A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR04MB5715 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:215187 Archived-At: On 9/23/2021 3:42 AM, Eli Zaretskii wrote: >> Cc: Stromeko@nexgo.de, 50666@debbugs.gnu.org >> From: Ken Brown >> Date: Wed, 22 Sep 2021 17:35:28 -0400 >> >> We've made a good start on the Cygwin side, but I have a question about how to >> integrate it into Emacs. > > I added Andrea to this discussion, as he knows more than anyone else > about the Emacs side of this stuff. > >> Let's say we have a script that I'll call "rebase" for the purpose of this >> discussion, which rebases all the eln files in ~/.emacs.d/eln-cache. The user >> would then start Emacs via a script that first calls rebase and then starts >> Emacs. > > Is it really necessary to rebase the *.eln files before each startup? > Isn't it enough to rebase each of the .eln files just once, when it is > produced? If indeed this is needed every time, can you explain why? We have to distinguish between system libraries and user libraries. Libraries installed by Cygwin packages are system libraries. The *.eln files in ~/.emacs.d/eln-cache are user libraries. Cygwin maintains a database of all system libraries and their base addresses. Whenever a Cygwin package is installed or updated, Cygwin rebases all system libraries and updates the database. Each time Emacs starts, it has no way of knowing whether the system libraries have been rebased since the last time Emacs was run. So the user's *.eln files could now have base address conflicts with system libraries. Rebasing the *.eln files fixes this problem. >> Within Emacs, I would then want to do something like >> >> (if (eq system-type 'cygwin) >> (call-process "rebase" nil >> '(:file "") >> nil "" ...)) >> >> after every compilation but before the compiled file is loaded. >> >> I'm not familiar enough with native compilation to know where this should go. > > The non-preloaded *.eln files are all loaded by native-elisp-load, so > I guess the rebase should be launched from there? The preloaded *.eln > files are loaded in pdumper.c:dump_do_dump_relocation, but do we need > to support non-rebased preloaded *.eln files? The preloaded *.eln files will be installed by Cygwin's package manager when emacs is installed. They are therefore system libraries and are automatically rebased as needed. >> Can you help? > > I hope the above helps. But I think we should understand better all > the aspects of this issue, to make sure it will work correctly in the > wild. The *.eln files bring quite a few new and unusual aspects and > dependencies to Emacs, they are not just another kind of DLLs loaded > dynamically. So I'd appreciate if you explained: > > . why is rebasing needed (I think I understand that part, but I'd > prefer an explanation from the experts) I'm not as expert as I'd like to be on the intricacies of Cygwin's fork implementation, but here's my best attempt. Achim may want to correct it or add to it. When Cygwin forks, it creates a new process and copies the memory image of the parent to the child. But Cygwin uses the Windows loader to load DLLs, so it has to ensure that Windows will load all DLLs to the same addresses in the child at which they were loaded in the parent. The problem occurs when there's an address collision, i.e., two DLLs have the same hard-wired base address. Window then resolves the collision by loading one of them at a different address. But there's no guarantee that it will resolve the collision in the child the same way it resolved it in the parent. That leads to a fork failure. Cygwin tries to head-off such problems before they occur by rebasing as I've described above at package-installation time. > . how will the 'rebase' utility determine to which address to remap > each .eln file It simply chooses an address that doesn't conflict with any of the system libraries and the already-rebased user libraries. >> P.S. The rebase script will fail to rebase eln files that are already loaded, >> but that's harmless in the scenario above. > > When an updated .eln file is produced for .eln that is loaded into > some running Emacs, Emacs on Windows renames the original .eln to > avoid a similar problem. I hadn't thought of this issue. We may have to use a similar technique... > Can't you use the same technique, to avoid > the need of rebasing on each start? ...but it wouldn't eliminate the need for rebasing at each start for the reasons explained above. > Please note that users could > place *.eln files in unusual locations (and customize > native-comp-eln-load-path to reflect that), so finding _all_ of the > relevant *.eln files from a shell script might not be easy. In fact, > even without customizing the load-path, I don't think I understand how > will that script you propose know where to find all the *.eln files. The current proposal that Achim and I are looking at would require each user to maintain a file ~/.config/rebase/dynpath.d/emacs containing a list of directories where the .eln files can be found. By default, this file would contain one line, which is the path to the standard eln-cache directory. Users who customize native-comp-eln-load-path would have to modify that file accordingly. For example, I currently have a second line pointing to the native-lisp directory of my emacs build directory, so that I can do testing of my built emacs without having to install it. Finally, as a side note, I don't think it would be a tragedy if this just turns out to be too complicated and we have to disable native compilation on 32-bit Cygwin. The Cygwin home page at https://cygwin.com/ already contains the following: Address space is a very limiting factor for Cygwin. These days, a full 32 bit Cygwin distro is not feasible anymore, and will in all likelihood fail in random places due to an issue with the fork(2) system call. Therefore we recommend using 32 bit Cygwin only in limited scenarios, with only a minimum of necessary packages installed, and only if there's no way to run 64 bit Cygwin instead. Ken