From mboxrd@z Thu Jan 1 00:00:00 1970 From: elaexuotee@wilsonb.com Subject: Re: Guix System on a GCE VM -- Success Date: Thu, 09 Apr 2020 11:11:52 +0900 Message-ID: <3FHAUW8KV3W8J.3BSZFDDKCPXJN@wilsonb.com> References: <35A9N567H32M5.257INESNBD8T8@wilsonb.com> <878sj6gpw8.fsf@jlicht.xyz> <2IHLQK6FN24VX.3ND6783BBUH9Y@wilsonb.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="----_=_56b9c506729dad2c1adc6f0a_=_" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:44265) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jMMfv-0001d7-GI for guix-devel@gnu.org; Wed, 08 Apr 2020 22:12:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jMMft-0007zW-85 for guix-devel@gnu.org; Wed, 08 Apr 2020 22:12:15 -0400 Received: from m42-5.mailgun.net ([69.72.42.5]:58646) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jMMfp-0007yd-TM for guix-devel@gnu.org; Wed, 08 Apr 2020 22:12:13 -0400 In-Reply-To: <2IHLQK6FN24VX.3ND6783BBUH9Y@wilsonb.com> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: Jelle Licht , guix-devel@gnu.org This is a multipart message in MIME format. ------_=_56b9c506729dad2c1adc6f0a_=_ MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_7c2b74c53c12a6d522a7f004_=_" This is a multipart message in MIME format. ------_=_7c2b74c53c12a6d522a7f004_=_ Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable elaexuotee--- via "Development of GNU Guix and the GNU System distribution.= " wrote: > > This all looks pretty nifty! Just to be clear, under which license did > > you make this script available? >=20 > Ouch. That was a major oversight. Thanks for bringing this up! > I went ahead and put it under the BSD-3-CLAUSE. Attached is the same sour= ce > with a copyright notice and license header. It looks like I messed up the attachment. Trying again. ------_=_7c2b74c53c12a6d522a7f004_=_ Content-Disposition: attachment; filename=gce.sh Content-Type: text/plain Content-Transfer-Encoding: quoted-printable #!/usr/bin/env sh set -o errexit -o noclobber -o nounset # Copyright (c) 2020 elaexuotee@wilsonb.com. All rights reserved. # # Licensed under # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are me= t: # # 1. Redistributions of source code must retain the above copyright notice,= # this list of conditions and the following disclaimer. # # 2. Redistributions in binary form must reproduce the above copyright noti= ce, # this list of conditions and the following disclaimer in the documentat= ion # and/or other materials provided with the distribution. # # 3. Neither the name of the copyright holder nor the names of its contribu= tors # may be used to endorse or promote products derived from this software # without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS I= S" # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE= # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOS= E # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE= # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF TH= E # POSSIBILITY OF SUCH DAMAGE. ## GCE Custom Image # # This script intends to be "executable documention" for the process of set= ting # up a Google Compute Engine (GCE) virtual machine from a custom image. usage() { cat <<-USAGE Usage: $(basename "${0}") [options] [ [..]] This script aids in setting up Google Compute Engine with an instance running a custom image. Execute commands in order provided at the command line. The procedure for a fresh setup follows the order listed below in Status Commands. Setup Commands: mkconf Configure files on raw image partition mkraw Convert image to raw format mksize Grow image to integer multiple of GBs mktar Archive raw image mkgsb Create GS bucket togsa Upload archive to GS bucket togci Create GCE image from GS archive togvm Create GCE instance from GCE image Status Commands: ckconf Check contents of qcow2 image ckraw Check integrity of raw image cksize Check size of raw image cktar Check contents of tar archive ckgsb Check existence of GS bucket ckgsa Check existence of tar archive in GS bucket ckgci Check existence of GCE image ckgvm Check status of GCE instance ttysro View GCE instance's serial output ttysrw Connect to instance serial port Removal Commands: rmgvm Remove GCE instance rmgci Remove GCE image rmgsa Remove archive from GS bucket rmgsb Remove GS bucket Metainfo Commands: help Show this help message vars Print value of configuration variables Options: -f Family name for GCE image -v Version string of GCE image -n Name of GCE image -N Name of GCE instance -s Subnet name of GCE instance -m Machine type of GCE instance -B URI of GS bucket -A URI of GS tar file -p Partition of image on which / resides -S Serial port of GCE instance -b Common path base for files and names -g Path on image of grub.cfg -i Local path of source image file -r Local path of raw image file -a Local path of tar file -h Show this help message USAGE } fail() { msg=3D${1}; errno=3D${2-1} >&2 printf 'Error: %s\n' "${msg}" >&2 usage exit "${errno}" } show_vars() { printf 'gce_image_family=3D%s\n' "${gce_image_family}" printf 'gce_image_version=3D%s\n' "${gce_image_version}" printf 'gce_image_name=3D%s\n' "${gce_image_name}" printf '\n' printf 'gce_vm_name=3D%s\n' "${gce_vm_name}" printf 'gce_vm_subnet=3D%s\n' "${gce_vm_subnet}" printf 'gce_vm_type=3D%s\n' "${gce_vm_type}" printf '\n' printf 'ord_partition=3D%s\n' "${ord_partition}" printf 'ord_serial_port=3D%s\n' "${ord_serial_port}" printf '\n' printf 'path_base=3D%s\n' "${path_base}" printf 'path_grub_cfg=3D%s\n' "${path_grub_cfg}" printf 'path_qcow2=3D%s\n' "${path_qcow2}" printf 'path_raw=3D%s\n' "${path_raw}" printf 'path_tar_gz=3D%s\n' "${path_tar_gz}" printf '\n' printf 'gs_uri_bucket=3D%s\n' "${gs_uri_bucket}" printf 'gs_uri_tar=3D%s\n' "${gs_uri_tar}" } size_mod_gb() { img=3D${1} size=3D$(stat -c '%s' "${img}") mod_gb=3D$((size / (1024*1024*1024))) rem_gb=3D$((size % (1024*1024*1024))) printf '%s %s\n' "${mod_gb}" "${rem_gb}" } ## GCE requires image sizes to be an integer number of GBs grow_to_ciel_gbs() { img=3D${1} size_mod_gb "${img}" \ | if read -r sz_mod_gb sz_rem_gb _; then [ "${sz_rem_gb}" -eq 0 ] && return dd if=3D/dev/zero of=3D"${img}" \ bs=3D1G count=3D0 seek=3D$((sz_mod_gb + 1)) fi } ## GCE puts a few contraints on kernel cmdline arguments # # Remove unsupported: `splashimage`, `rhgb`, and `quiet` # Enable serial console: `console=3DttyS0,38400n8d` prepare_grub_cfg() { grub_cfg=3D${1} sudo sed -i -f - "${grub_cfg}" <<-'SED' /^\s\+linux/ { s/\s\+\(rhgb\|quiet\|splashimage\)//g } /^\s\+linux/ { s/\s\+console=3DttyS0,38400n8d//g s/$/ console=3DttyS0,38400n8d/ } SED } check_grub_cfg() { grub_cfg=3D${1} sudo sed -n -f - "${grub_cfg}" <<-'SED' /^\s\+linux.*\(rhgb\|quiet\|splashimage\)/ { p } /^\s\+linux/ { s/\s\+console=3DttyS0,38400n8d/&/g t; F; p } SED } free_nbd_dev() { sudo modprobe -v nbd for nbd in /sys/class/block/nbd*; do [ "$(cat "${nbd}"/size)" -ne 0 ] && continue dev_block=3D$(cat "${nbd}/dev") readlink -vf "/dev/block/${dev_block}" break done } mount_image_part() { img=3D${1}; part=3D${2}; mnt=3D${3} dev=3D$(free_nbd_dev) sudo qemu-nbd --format=3Dqcow2 --connect=3D"${dev}" "${img}" mkdir -vp "${mnt}" sudo mount -v "${dev}p${part}" "${mnt}" } umount_image() { mnt=3D${1} dev=3D$(mount | awk "\$3 ~ \"${mnt}\" {print \$1}") sudo umount -v "${mnt}" rmdir -v "${mnt}" sudo qemu-nbd --disconnect "${dev}" } configure_files() { img=3D${1}; part=3D${2}; grub_cfg=3D${3} mnt=3D$(mktemp -d) mount_image_part "${img}" "${part}" "${mnt}" prepare_grub_cfg "${mnt}/${grub_cfg}" umount_image "${mnt}" } check_files() { img=3D${1}; part=3D${2}; grub_cfg=3D${3} mnt=3D$(mktemp -d) mount_image_part "${img}" "${part}" "${mnt}" check_grub_cfg "${mnt}/${grub_cfg}" umount_image "${mnt}" } ## GCE image source expects oldgnu gzip tar with image named `disk.raw` archive_to_tar_gz() { raw=3D${1}; tar_gz=3D${2} dir=3D$(mktemp -d) ln -vs "${raw}" "${dir}/disk.raw" tar --verbose --sparse --dereference --format=3Doldgnu --create --gzip \ --directory "${dir}" --file "${tar_gz}" disk.raw rm -v "${dir}/disk.raw" rmdir -v "${dir}" } ## To debug boot, we must explicitly enable serial I/O gce_create_vm() { name=3D${1}; image=3D${2}; subnet=3D${3}; type=3D${4}; port=3D${5} gcloud compute instances create "${name}" \ --image "${image}" \ --subnet "${subnet}" \ --machine-type "${type}" gcloud compute instances add-metadata "${name}" \ --metadata "serial-port-enable=3D${port}" } while getopts ':a:A:b:B:f:g:i:m:n:N:p:r:s:S:v:h' opt "${@}"; do case "${opt}" in f) opt_gce_image_family=3D${OPTARG};; v) opt_gce_image_version=3D${OPTARG};; n) opt_gce_image_name=3D${OPTARG};; N) opt_gce_vm_name=3D${OPTARG};; s) opt_gce_vm_subnet=3D${OPTARG};; m) opt_gce_vm_type=3D${OPTARG};; B) opt_gs_uri_bucket=3D${OPTARG};; A) opt_gs_uri_tar=3D${OPTARG};; p) opt_ord_partition=3D${OPTARG};; S) opt_ord_serial_port=3D${OPTARG};; b) opt_path_base=3D${OPTARG};; g) opt_path_grub_cfg=3D${OPTARG};; i) opt_path_qcow2=3D${OPTARG};; r) opt_path_raw=3D${OPTARG};; a) opt_path_tar_gz=3D${OPTARG};; h) usage; exit;; :) fail "Expected argument to option -${OPTARG}";; *) fail "Unrecognized option: -${OPTARG}";; esac done shift $((OPTIND - 1)) gce_image_family=3D${opt_gce_image_family:-gce-vm} gce_image_version=3D${opt_gce_image_version:-} gce_image_name=3D${opt_gce_image_name:-${gce_image_family}${gce_image_versi= on:+-}${gce_image_version}} gce_vm_name=3D${opt_gce_vm_name:-${gce_image_name}} gce_vm_subnet=3D${opt_gce_vm_subnet:-default} gce_vm_type=3D${opt_gce_vm_type:-f1-micro} ord_partition=3D${opt_ord_partition:-1} ord_serial_port=3D${opt_ord_serial_port:-1} path_grub_cfg=3D${opt_path_grub_cfg:-/boot/grub/grub.cfg} path_base=3D${opt_path_base:-${PWD}/imgs/${gce_image_name}} path_qcow2=3D${opt_path_qcow2:-${path_base}.qcow2} path_raw=3D${opt_path_raw:-${path_base}.raw} path_tar_gz=3D${opt_path_tar_gz:-${path_base}.tar.gz} gs_uri_bucket=3D${opt_gs_uri_bucket:-gs://$(date +%s.%N)} gs_uri_tar=3D${opt_gs_uri_tar:-${gs_uri_bucket}/$(basename "${path_tar_gz}"= )} [ $# -gt 0 ] || fail "Expected command" for command in "${@}"; do case "${command}" in # Setup mkconf) configure_files "${path_qcow2}" \ "${ord_partition}" \ "${path_grub_cfg}";; mkraw) qemu-img convert -O raw "${path_qcow2}" \ "${path_raw}";; mksize) grow_to_ciel_gbs "${path_raw}";; mktar) archive_to_tar_gz "${path_raw}" \ "${path_tar_gz}";; mkgsb) gsutil mb "${gs_uri_bucket}";; togsa) gsutil cp "${path_tar_gz}" "${gs_uri_bucket}";; togci) gcloud compute images create "${gce_image_name}" \ --source-uri "${gs_uri_tar}" \ --family "${gce_image_family}";; togvm) gce_create_vm "${gce_vm_name}" \ "${gce_image_name}" \ "${gce_vm_subnet}" \ "${gce_vm_type}" \ "${ord_serial_port}";; ttysro) gcloud compute instances get-serial-port-output \ "${gce_vm_name}" \ --port "${ord_serial_port}";; ttysrw) gcloud compute connect-to-serial-port \ "${gce_vm_name}" \ --port "${ord_serial_port}";; # Status ckconf) check_files "${path_qcow2}" \ "${ord_partition}" \ "${path_grub_cfg}";; ckraw) qemu-img compare "${path_qcow2}" "${path_raw}";; cksize) size_mod_gb "${path_raw}" | awk '$2 !=3D 0 {exit 1}';; cktar) tar -tf "${path_tar_gz}" | sed '/^disk.raw$/!q1';; ckgsb) gsutil du "${gs_uri_bucket}";; ckgsa) gsutil stat "${gs_uri_tar}";; ckgci) gcloud compute images describe "${gce_image_name}";; ckgvm) gcloud compute instances describe "${gce_vm_name}";; # Removal rmgvm) gcloud compute instances delete "${gce_vm_name}";; rmgci) gcloud compute images delete "${gce_image_name}";; rmgsa) gsutil rm "${gs_uri_tar}";; rmgsb) gsutil rb "${gs_uri_tar}";; # Metainfo help) usage;; vars) show_vars;; *) fail "Unrecognized command: ${command}";; esac done ------_=_7c2b74c53c12a6d522a7f004_=_-- ------_=_56b9c506729dad2c1adc6f0a_=_ Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYIADUWIQQ7FdZn/PDWvxE6cmR2pStZ7i7CgQUCXo6EYxccZWxhZXh1b3Rl ZUB3aWxzb25iLmNvbQAKCRB2pStZ7i7CgW+pAQDCIf57Xhzb9Tm2L7Slq/k3uvXR w7CVbTDevuYA4FXokQD/QUjnsxH5TmHq6btSu4s6fcCd/J9pS0CqP9zQPp51WwA= =Qfz0 -----END PGP SIGNATURE----- ------_=_56b9c506729dad2c1adc6f0a_=_--