From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 4L8MNeabpGItTAAAbAwnHQ (envelope-from ) for ; Sat, 11 Jun 2022 15:43:02 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 4CX8NOabpGIIggEA9RJhRA (envelope-from ) for ; Sat, 11 Jun 2022 15:43:02 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EACA23D67D for ; Sat, 11 Jun 2022 15:43:01 +0200 (CEST) Received: from localhost ([::1]:35620 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o01OG-0004w0-Qk for larch@yhetil.org; Sat, 11 Jun 2022 09:43:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44882) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01MM-0003DD-S9 for bug-guix@gnu.org; Sat, 11 Jun 2022 09:41:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:58311) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o01MM-0001mY-K6 for bug-guix@gnu.org; Sat, 11 Jun 2022 09:41:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1o01MM-00027F-GM for bug-guix@gnu.org; Sat, 11 Jun 2022 09:41:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#55907: VFIO kernel module fails to capture PCI device Resent-From: "Nick Zalutskiy" Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 11 Jun 2022 13:41:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 55907 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 55907@debbugs.gnu.org X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16549548498099 (code B ref -1); Sat, 11 Jun 2022 13:41:02 +0000 Received: (at submit) by debbugs.gnu.org; 11 Jun 2022 13:40:49 +0000 Received: from localhost ([127.0.0.1]:52206 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o01M8-00026T-C9 for submit@debbugs.gnu.org; Sat, 11 Jun 2022 09:40:48 -0400 Received: from lists.gnu.org ([209.51.188.17]:40738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o01M5-00026K-Tv for submit@debbugs.gnu.org; Sat, 11 Jun 2022 09:40:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44846) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01M4-00032v-Ox for bug-guix@gnu.org; Sat, 11 Jun 2022 09:40:45 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:45857) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01M1-0001l0-LZ for bug-guix@gnu.org; Sat, 11 Jun 2022 09:40:43 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 75BD65C00EA for ; Sat, 11 Jun 2022 09:40:39 -0400 (EDT) Received: from imap43 ([10.202.2.93]) by compute2.internal (MEProxy); Sat, 11 Jun 2022 09:40:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=const.fun; h=cc :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm1; t= 1654954839; x=1655041239; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05 HA6cm9hY=; b=bk3wh5x0Tj3niWJPpN5j5x1nX46gb6JGdEGwYvR/Qm+xRHG+zYu OfF+4qY/hZ8rvisV0TfSiwlsFy6px+Sz1t5g64s1Ax7kHnSnTHpwCe6dNqCHbjZQ XTmmUIwvAfIihO53uoxxq3FG84379Rklmq48zTWR5tymK7qk2YARL2BwK5+FW+Oj GSFsi0MpmuoGwr6EoZsPhWh8T0eYgU9rX3cXL9IuAJdPlBYWcnQv/5FmpjcWnIvs NDlujdsC94ntkytr5NliFADKkUannQz5rI1LcjvD4r2SgP2fOV9qmUOlhZBWECKg LOK30L0ttZX7qJ6vlg8h6ZsN6ZSMmMY9mIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1654954839; x= 1655041239; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05HA6cm9hY=; b=X tt6Kf8boA33/VuJfKwZQDuPqNT3jqSpBUe9f8sMsG3eCCiXPrleWF8XyVr1iFQi9 z2wqDVJY+AUdkfVjxZnHsDYBUm1NT8Ov9ZCFtSn7VebESQhsO4zjGRTSd2iHgjDE ccQnvzSfKU141U0uSL4GvrhNaS5FNyWyy8IpR7o10X2lEVf6jOFedm0+mctEurBl 4AFwrtT2nfurHIN7lQf/FJ5TnUakKgLM7j9qLtcj0O10URebLBO845i5Hu3hB0eb bBmvXu4Tu/ynE0mOH/76fQjpjMYAetz1YDmEOyOMSA8MEWAvzlkEq9EoKYcoLy8R pL5+fvD0ykjCEXUhcFOZw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddufedgieehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsegrtderre erredtnecuhfhrohhmpedfpfhitghkucgkrghluhhtshhkihihfdcuoehnihgtkhestgho nhhsthdrfhhunheqnecuggftrfgrthhtvghrnhepffethfdthefgkedugeetieekueevfe fhgfeiffetgfdvgeehleeitefhheehfeeinecuffhomhgrihhnpehgihhthhhusgdrtgho mhdprghrtghhlhhinhhugidrohhrghdpkhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepnhhitghksegtohhnshhtrdhf uhhn X-ME-Proxy: Feedback-ID: i25a94707:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 276A62D40070; Sat, 11 Jun 2022 09:40:39 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-692-gb287c361f5-fm-20220603.003-gb287c361 Mime-Version: 1.0 Message-Id: Date: Sat, 11 Jun 2022 09:40:18 -0400 From: "Nick Zalutskiy" Content-Type: multipart/alternative; boundary=88555654cab94e5396cdcfa3e24335ec Received-SPF: pass client-ip=66.111.4.26; envelope-from=nick@const.fun; helo=out2-smtp.messagingengine.com X-Spam_score_int: 17 X-Spam_score: 1.7 X-Spam_bar: + X-Spam_report: (1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_SUSPICIOUS_NTLD=0.499, FROM_SUSPICIOUS_NTLD_FP=1.997, HTML_MESSAGE=0.001, PDS_OTHER_BAD_TLD=1.997, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1654954982; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05HA6cm9hY=; b=fuT3SNeW+U8T5qMc68/wOGi/m1Ve9RSqmpbhAJJ3Dpc70+82rl2byLrrDhZvvkbPhlbhWY oWmjsgQ251mA/9qk4kMTS+Z8GBAbQQqydKBQDeOvqwvNoJ7426NIFKyxMfaebZGJX8rHpT Hy4JHjxOzw3+j60O/OyIhiIOcIAs3IZNr8hazQKbsnu1EfbVdg4X2NOBEpTjp8Yu8sg6uy rcAA/saE4kjAgG2R6/FvBOkp0ipa+3hJJEBSCzAfHzfxmSj1LFKhIqvv8Rb9kZfeQbVJVH MfIRVKW5CfXeeAbn81mm/tp+ULebY6dAvJ0PP2auH7mLKFrlogj42o/zF+kLhw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1654954982; a=rsa-sha256; cv=none; b=a+icbepZkZNZoba7wnyXpwCCzyQFVRXWUR6oXN822bKYEKRSINXFtGAcPaven+wxQOzjIt /8s7dQOclmDeU8I1fZtMt3osVuOCGXEKsOiGY8LHafLnXMflIxZ38T7QskFM2KqnMoxiV5 NvRwP97InUzJEOGTjdx+8xWAb3smWX0iTOT0KlnGWPmqJFu6j8K4CYjn4lfAMgzpl7qXK/ Qtb2cLfhJUjG9mA/JXj4INkjaqxHgO6F6yBZjHLF1s6lwsg71uP+hwYnXhG7VMeL3r3Uy3 H0pV7NEbY+CTHqOQGei61yeL3g/sNJV1A0F5t16dDzjMqHxSLlXT8J9/r4+OnA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=const.fun header.s=fm1 header.b=bk3wh5x0; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="X tt6Kf8"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 0.33 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=const.fun header.s=fm1 header.b=bk3wh5x0; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="X tt6Kf8"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: EACA23D67D X-Spam-Score: 0.33 X-Migadu-Scanner: scn0.migadu.com X-TUID: qjD+OoxGMATB --88555654cab94e5396cdcfa3e24335ec Content-Type: text/plain Hello all, I am trying to capture my graphics card at initrd, using vfio, to later pass it through to a virtual machine. Judging by dmesg, the VFIO module does load early, however, the card is not captured at that point and the amdgpu driver is later loaded instead. This is what I have in my `operating-system` config: > (kernel-arguments '("iommu=pt" "vfio-pci.ids=1002:73bf")) > (initrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %base-initrd-modules)) There are two video cards in the system, both AMD, but different models. The video card of interest is in a separate IOMMU group and the : combination is correct for my machine. Best I can tell, vfio-pci.ids argument is not propagated to the module by initramfs. See the following: Searching online I came up against a GitHub issue for a different initramfs generator that exhibited the same symptoms: VFIO module was loaded, kernel arguments were correct, yet the card was not captured by the vfio driver. The maintainer there did a great job tracking down and fixing the issue and came up with this insight https://github.com/anatol/booster/issues/20#issuecomment-808956316 > After reading kmod code I found that kernel does not use cmdline params for loadable modules. It was surprising for me. Instead it is expected that userspace handles cmdline parsing and provides required module params explicitly. Another way to attach the correct driver to the gpu is to run a script at initrd, which I don't know how to accomplish with Guix. This approach has the advantage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs I tried following the kernel docs to rebind a different driver after boot, but I believe this doesn't work for video cards, and hasn't worked for me. Any help is greatly appreciated! Links: Kernel docs for vfio https://www.kernel.org/doc/html/latest/driver-api/vfio.html Arch guide for GPU passthrough https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF Thank you! -Nick --88555654cab94e5396cdcfa3e24335ec Content-Type: text/html Content-Transfer-Encoding: quoted-printable
Hello all,
<= /div>

I am trying to capture my graphics card at init= rd, using vfio, to later pass it through to a virtual machine. Judging b= y dmesg, the VFIO module does load early, however, the card is not captu= red at that point and the amdgpu driver is later loaded instead.

This is what I have in my `operating-system` confi= g:

(kernel-argume= nts '("iommu=3Dpt" "vfio-pci.ids=3D1002:73bf"))
  (in= itrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %= base-initrd-modules))

There ar= e two video cards in the system, both AMD, but different models. The vid= eo card of interest is in a separate IOMMU group and the <vendor id&g= t;:<device id> combination is correct for my machine.

Best I can tell, vfio-pci.ids argument is not propagate= d to the module by initramfs. See the following:

Searching online I came up against a GitHub issue for a different = initramfs generator that exhibited the same symptoms: VFIO module was lo= aded, kernel arguments were correct, yet the card was not captured by th= e vfio driver. The maintainer there did a great job tracking down and fi= xing the issue and came up with this insight https://github.com/a= natol/booster/issues/20#issuecomment-808956316

After reading kmod code I found that k= ernel does not use cmdline params=20 for loadable modules. It was surprising for me. Instead it is expected=20 that userspace handles cmdline parsing and provides required module=20 params explicitly.

Another way= to attach the correct driver to the gpu is to run a script at initrd, w= hich I don't know how to accomplish with Guix. This approach has the adv= antage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_pass= through_via_OVMF#Using_identical_guest_and_host_GPUs
<= br>
I tried following the kernel docs to rebind a different dr= iver after boot, but I believe this doesn't work for video cards, and ha= sn't worked for me.

Any help is greatly app= reciated!

Links:
Kernel docs = for vfio

Arch guide for GPU passthrou= gh

Thank you!

-Nick

--88555654cab94e5396cdcfa3e24335ec--