From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id UAvdAwRnNGACDQAA0tVLHw (envelope-from ) for ; Tue, 23 Feb 2021 02:23:00 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id s3KLOQNnNGBLcQAAbx9fmQ (envelope-from ) for ; Tue, 23 Feb 2021 02:22:59 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 41BD48883 for ; Tue, 23 Feb 2021 03:22:59 +0100 (CET) Received: from localhost ([::1]:54624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lENLm-0005s8-5Z for larch@yhetil.org; Mon, 22 Feb 2021 21:22:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:52388) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lENKB-0004qG-Fz for guix-devel@gnu.org; Mon, 22 Feb 2021 21:21:25 -0500 Received: from imta-37.everyone.net ([216.200.145.37]:42558 helo=imta-38.everyone.net) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lENK8-00083y-Of for guix-devel@gnu.org; Mon, 22 Feb 2021 21:21:19 -0500 Received: from pps.filterd (omta004.sj2.proofpoint.com [127.0.0.1]) by imta-38.everyone.net (8.16.0.43/8.16.0.43) with SMTP id 11N2K9Rq018205; Mon, 22 Feb 2021 18:21:07 -0800 X-Eon-Originating-Account: -E-Nfh-i2vVVUMS4qC9gfF-ANWFd83-V9hG1RMD0DJk X-Eon-Dm: m0116293.ppops.net Received: by m0116293.mta.everyone.net (EON-AUTHRELAY2 - 53b92078) id m0116293.6000aa54.371d03; Mon, 22 Feb 2021 18:21:06 -0800 X-Eon-Sig: AQMHrIJgNGaSKe6bMgIAAAAD,057fd26f491741f0f40009e32ffa3003 X-Eip: 6B9NMhTCT-vLGTopbcaM9OnVqevbEX7ndPOIfhqWbwo Date: Tue, 23 Feb 2021 03:20:51 +0100 From: Bengt Richter To: Evan Rowley Subject: Installing via iso.xz, really best idea? -- was Re: Gnome Boxes Message-ID: <20210223022051.GA15276@LionPure> References: <878s7lqop7.fsf@gnu.org> <87blchm9oj.fsf@nckx> <8C025335-C317-4853-ACE6-9E746E46539B@lepiller.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-02-22_08:2021-02-22, 2021-02-22 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 malwarescore=0 clxscore=1034 adultscore=0 lowpriorityscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 phishscore=0 impostorscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102230017 Received-SPF: pass client-ip=216.200.145.37; envelope-from=bokr@oz.net; helo=imta-38.everyone.net X-Spam_score_int: -23 X-Spam_score: -2.4 X-Spam_bar: -- X-Spam_report: (-2.4 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bengt Richter Cc: guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.87 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 41BD48883 X-Spam-Score: -1.87 X-Migadu-Scanner: scn0.migadu.com X-TUID: heTD9y9HAO0M Hi, On +2021-02-22 00:39:36 -0500, Evan Rowley wrote: > FreeBSD also provides .iso.xz. Some examples here: > http://ftp-archive.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/13.0/ > Looking there for an example: --8<---------------cut here---------------start------------->8--- FreeBSD-13.0-BETA2-amd64-bootonly.iso 363927552 2021-Feb-12 07:47 FreeBSD-13.0-BETA2-amd64-bootonly.iso.xz 82351604 2021-Feb-12 07:47 --8<---------------cut here---------------end--------------->8--- That seems like a really worthwhile compression! (ratio 4.4:1) However, I wonder if big .iso's are really the best way to take advantage of the xz (or any) compression for installation purposes. If the point of having the information in an .iso is to have it be bootable with everything (drivers, UI, etc) needed to install to any raw disk anywhere, ISTM that is wastefully general for my usual case: I have several laptops all able to dd to a USB-attached SSD or thumbdrive. I trust their impure apps to do the xfr and dd, if I can verify result hashes independently. I don't need to boot anything but my laptop for these apps: they're on my laptops. I don't intend to use my foreign laptop to synthesize content of the files to xfr by dd, so there is only dependency on the pure guix versions of fdisk, mkfs, etc used on the machine that executed guix system -make-custom-bootstrap-scripted-tarball my-neat-system.scm (in my dreams :) (maybe a modded version of guix pack could do it, but sizes?) ┌─────────────────────────────────────────────────────────────────────────────────┐ │ I don't want to modify the laptop I'm using (in fact, I want a guarantee │ │ that it WON'T be modified as I use it to dd-write to the attached USB storage). │ └─────────────────────────────────────────────────────────────────────────────────┘ (boxed for emphasis :) All I really need is a tarball with a bootstrap script that essentially does a series of dd file transfers to selected block sequences on the target disk(s). I am thinking this could be smaller and easier to use than using guix pack to concoct the same functionality. As the script starts, IWBN to be able to select target disk(s) interactively, probably using lsblk to probe and display, but I want to avoid partitioning target disk(s) with foreign tools (i.e. writing to the medium with other than dd). I think read-only access to get existing (if any) partition locations and sizes and types is ok, so the script can potentially choose between sparse files to dd into common sizes of disk and partition spaces where e.g. just padding with zeroes is not workable. IWBN to be able to target a toy 32-gb USB as well as the front end of a 500GB SSD with the same script/tarball. Those files for dd can contain ready-made images of GPT pieces -- superblock, VFAT fs, legacy-bootable grub blocks, /boot and / and maybe other specialized block-sequence images (with mkfs and transfers into the populated fs images already done on a pure guix system using loopback-mounted files for fs images), or whatever else may be needed. (caveat hunch: subject to some init-hook binary patching on first boot (hopefully no grub-launched or BIOS-validated thing like firmware update necessary if kernel init can do it ;/ ). For the case where I want to modify my own laptop, e.g. to boot from the USB-attached storage I just prepared, that depends on what its BIOS does for booting, but I guess that is commonly just a matter of updating grub's .cfg. For the case where I want to install to my own laptop's disk(s), I still don't like booting a monster iso. If my laptop's existing OS can't go into a safe maintenace mode, where the target disk is synced and unmounted, then I will have to reboot. If that is from a bootable installer USB, I want a rescue-size bootable, not the monster iso including everything (nor the job and wait of writing the iso image to a USB just so I can boot it once). ISTM a manifest for the installation job could be a file of symbolic by-hash links or equivalent sha256sum listing, and a list of mirrors, specifying directories where the by-hash links are findable, e.g. like https://example.com/pub/filedir/by-hash/ and links in that directory looking like e.g., e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -> ../zerolengthfile (which are easy to generate selectively or wholesale for a filedir/ directory just creating the ./by-hash/ subdirectory and making the links)) ISTM like a custom bootstrap.sh could safely wget each file, piping directly to dd writing these chunks to raw disk slots, afterwards checking the sha256 of every block sequence it wrote. Done simply -- and irrespective of whether the by-hash links and files were stored in a git repo or plain file directories. I think "guix system" could automate the production of customized my-neat-system-boostrap.sh scripts containing the right wget and dd commands and sha256 verifications, while providing safe interactive choice of target disk(s) at the start of running it. Such a script, built to run on a specific but non-guile/guix-dependent "foreign" platform like Linux x86_64 would be a nice way to share a "my-neat-system" ... or a Guix release. So it seems to me, anyway :) (I know you can do all kinds of things with laptops already running guix. A main point here is to be able to use a (borrowed even) foreign, non-guile/guix laptop to install something onto raw USB-attached storage without transferring anything from the state of the laptop doing the work (i.e., xfr only, no synthesis). I.e., the system or data being transferred could be for a totally different architecture or purpose. For the paranoid, a first-boot (assuming new thing is bootable) hook could re-check all the hashes on the new system. (Please excuse all the handwaving :) > On Thu, Feb 18, 2021 at 4:23 PM Julien Lepiller wrote: > > > Sorry, a compressed .iso is probably common, a .iso.xz is very uncommon > > :). We even have had some reports of people trying to copy that directly to > > an installation media. > > > > Le 18 février 2021 14:56:44 GMT-05:00, Tobias Geerinckx-Rice > > a écrit : > >> > >> Julien Lepiller 写道： > >> > >>> Usually compression is provided by the webserver, but maybe ours > >>> is not configured to do that. I think we're the only distro to > >>> provide compressed isos. > >>> > >> > >> Really? Most distribution ISOs use squashfs or similar with > >> XZ/LZMA compression. It doesn't make sense to compress that over > >> the wire. > >> > >> That said: XZ compression currently saves 27% (559M -> 405M). > >> Transparently serving pre-compressed ISOs with nginx (gzip level > >> 9) would save about 25% (559M -> 415M), which is surprisingly > >> similar. > >> > >> Kind regards, > >> > >> T G-R > >> > >> > > -- > - EJR Hm, I wonder how large a tarball "guix pack" as it works now would create to create something runnable with the my-neat-system-bootstrap.sh functionality described above, vs depending on the foreign system's bash, wget, dd, etc. -- Regards, Bengt Richter