Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added additional solution to Blackscreen #96

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added 257606.rom
Binary file not shown.
100 changes: 51 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,17 @@

### If you go to reddit (specifically [r/vfio](https://www.reddit.com/r/vfio)) for help, try to mention [u/lellow_yedbetter](https://www.reddit.com/user/lellow_yedbetter/). You will probably get help faster from users in the sub, but it is easier for me to keep an eye on the github issues.

# Single GPU Passthrough on Linux
This guide is to help people through the process of using GPU Passthrough via libvirt/virt-manager on systems that only have one GPU.
This guide is to help people through the process of using GPU Passthrough via libvirt/virt-manager on Legion 5 Pro in Discrete Graphics Mode.

## Special Thanks to:
### [The Passthrough post](https://passthroughpo.st)
For hosting news and information about VFIO passthrough, and for the libvirt/qemu hook helper in this guide.

### andre-ritcher
For providing the vfio-pci-bind tool. A tool that is no longer used in this guide, but was previously used and he still deserves thanks.

### Matoking
For the Nvidia ROM patcher. Making passing the boot gpu to the VM without GPU bios problems.
Patching the rom is no longer required but I never would have written this guide without the original work so I'm keeping them here.

### Sporif
For diagnosing, developing, and testing methods to successfully rebind the EFI-Framebuffer when passing the video card back to the host OS.

### droidman
For instructions on manually editing the vBIOS hex for use with VFIO passthrough

### [Yuri Alek](https://gitlab.com/YuriAlek/vfio)
A guide that is no doubt better than mine. Learning a few things from his implementation that can help me out a bit. This guide depends on libvirt at the base where as his has implementations that do not.
### JoeKnock90 on Github.
For providing the start and stop scripts that I used to bind and unbind the gpu dynamically. This repo was forked from his original one and repurposed for mostly personal reference when implementing vfio on the Lenovo Legion 5 Pro.

#### So many other people and organizations I need to thank. If feel your name should be here, please contact me. Credit where credit is due is very important to me, and to making the Linux community a better place.
#### If feel your name should be here, please contact me. Credit where credit is due is very important to me, and to making the Linux community a better place.

## Contents

Expand All @@ -38,24 +25,14 @@ A guide that is no doubt better than mine. Learning a few things from his implem
# Disclaimer
You are completely responsible for your hardware and software. This guide makes no guarentees that the process will work for you, or will not void your waranty on various parts or break your computer in some way. Everything from here on out is at your own risk.

# Background
Historically, VFIO passthrough has been built on a very specific model. I.E.

* 2 GPUs, 1 for the host, and one for the VM
* 2 monitors *OR* a monitor with 2 inputs *OR* a KVM switch

I personally, as well as some of you out there, might not have those things available. Maybe You've got a Mini-ITX build with no iGPU. Or maybe you're poor like me, and can't shell out for new computer components without some financial planning before hand.

Whatever your reason is. VFIO is still possible. But with caveats. Here's some advantages and disadvantages of this model.

This setup model is a lot like dual booting, without actually rebooting.

# Advantages
* As already stated, this model only requires one GPU
* The ability to switch back and forth between different OSes with FULL use of a discrete graphics processor (Linux on Host with full GPU, Windows 10 Guest with Full GPU, MacOS guest with full GPU)
* Bragging rights
* Could be faster than dual booting (this depends on your system)
* Using virtual disk images (like qcow) gives you management of snapshots, making breaking your guest os easy to recover from.
* Its really cool!!!
* Near native performance. More performance than dual on laptop at least.

# Disadvantages
* Can only use one OS at a time.
Expand All @@ -66,21 +43,13 @@ This setup model is a lot like dual booting, without actually rebooting.
* If you DO have a second video card, solutions like looking-glass are WAYYY more convenient and need active testing and development.
* All VMs must be run as root. There are security considerations to be made there. This model requires a level of risk acceptance.

For my personal use case. This model is worth it to me and it might be for you too!
For my personal use case. This model is worth it to me and I just think its really really cool to do!

# Prerequisites and Assumptions

## Assumptions
This guide is going to assume a few things

1. You have a system capable of VFIO passthrough. I.E. a processors that supports IOMMU, sane IOMMU groups, and etc.
2. I am going to start in a place where you have a working libvirt config, or qemu script, that boots a guest OS without PCI devices passed through.

I am not going to cover the basic setup of VFIO passthrough here. There are a lot of guides out there that cover the process from beginning to end.

What I will say is that using the [Arch Wiki][arch_wiki] is your best bet.

Follow the instructions found [here][arch_wiki]
You have read and understood thouroughly/implemented vfio using the [Arch Wiki][arch_wiki].

[arch_wiki]: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Expand All @@ -98,6 +67,15 @@ With all this ready. Let's move on to how to actually do this.

# Procedure

# Automatic Install

Just run the `setup-single-gpu-passthrough.sh` script as root:
```
sudo ./setup-single-gpu-passthrough.sh
```
It will take care of setting everything up. It automates the processes detailed below.

# Manual Install
## Setting up Libvirt hooks

Using libvirt hooks will allow us to automatically run scripts before the VM is started and after the VM has stopped.
Expand All @@ -121,10 +99,10 @@ Anything in the directory ````/etc/libvirt/hooks/qemu.d/{VM Name}/prepare/begin`

Anything in the directory ````/etc/libvirt/hooks/qemu.d/{VM Name}/release/end```` will run when your VM is stopped

### Libvirt Hook Scripts]
#### Do not copy my scripts. Use them as a template, but write your own.
### Libvirt Hook Scripts
Ensure that you have edited the pci ids according to the output of `lspci -nnk` to match your gpu. Otherwise the vfio-pci driver will not recognize and bind to your gpu.

I've made my start script ```/etc/libvirt/hooks/qemu.d/{VMName}/prepare/begin/start.sh```
I've edited the start script ```/etc/libvirt/hooks/qemu.d/{VMName}/prepare/begin/start.sh```


### Start Script
Expand All @@ -135,6 +113,12 @@ set -x

# Stop display manager
systemctl stop display-manager.service
systemctl stop sddm.service
pulse_pid=$(pgrep -u igneel pulseaudio)
pipewire_pid=$(pgrep -u igneel pipewire-media)
kill $pulse_pid
kill $pipewire_pid

## Uncomment the following line if you use GDM
#killall gdm-x-session

Expand All @@ -148,12 +132,22 @@ echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
# Avoid a Race condition by waiting 2 seconds. This can be calibrated to be shorter or longer if required for your system
sleep 2

modprobe -r nvidia-drm
modprobe -r nvidia-uvm
modprobe -r snd_hda_intel
modprobe -r i2c_nvidia_gpu
modprobe -r nvidia

sleep 2

# Unbind the GPU from display driver
virsh nodedev-detach pci_0000_0c_00_0
virsh nodedev-detach pci_0000_0c_00_1
virsh nodedev-detach pci_0000_01_00_0 #Replace numbers with your specific pci id. Use lspci -nnk
virsh nodedev-detach pci_0000_01_00_1 # This one too

# Load VFIO Kernel Module
modprobe vfio-pci
modprobe vfio
modprobe vfio_iommu_type1
```
NOTE: Gnome/GDM users. You have to uncommment the line ````killall gdm-x-session```` in order for the script to work properly. Killing GDM does not destroy all users sessions like other display managers do.

Expand All @@ -163,12 +157,18 @@ My stop script is ```/etc/libvirt/hooks/qemu.d/{VMName}/release/end/revert.sh```
```
#!/bin/bash
set -x


modprobe -r vfio-pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

# Re-Bind GPU to Nvidia Driver
virsh nodedev-reattach pci_0000_0c_00_1
virsh nodedev-reattach pci_0000_0c_00_0
virsh nodedev-reattach pci_0000_01_00_1 #Replace id with your gpu id number. Use lspci -nnk.
virsh nodedev-reattach pci_0000_01_00_0 #This too

# Reload nvidia modules
modprobe snd_hda_intel
modprobe nvidia
modprobe nvidia_modeset
modprobe nvidia_uvm
Expand Down Expand Up @@ -207,6 +207,8 @@ Logs can be found under /var/log/libvirt/qemu/[VM name].log
3. If there is a problem here, typically the command will hang. That would signify a problem with the VM libvirt configuration.
4. If you are returned to the prompt, check if the vm is in a running state by using `sudo virsh list`
5. If it's running fine, and you've made sure that you are not having the issue in step 1 and 2, yell at me in the issue tracker or reddit
3. **My 3070 Laptop GPU does not work well with the vfio driver hence it does not bind to the device properly and the guest cannot make proper use of the gpu. When I ran in hybrid mode with the iGPU enabled I noticed lower performance in games or outright crashing. In single-gpu mode it seems the problem is a blackscreen as soon as the display is handed off to the gpu. The solution is to send your gpu vbios explicitly through the xml. I have uploaded a rom file I got from TechPowerUp. I will upload my own VBIOS later. It is recommended that you dump the vbios for your own gpu and use it for best compatibility and security. **
`<rom file='path/to/file'/>`

### Audio
Check out the ArchWIKI entry for tips on audio. I've used both Pulseaudio Passthrough but am currently using a Scream IVSHMEM device on the VM.
Expand All @@ -220,15 +222,15 @@ This will keep services running even when your account is not logged in. I do no
## Personal Touches
Here's a few things I do to make managing the host easier.

1. Start a VNC server on the host in the start script
2. Set pulseaudio volume to 100%
1. Isolate all but 1 core from host for more performance.
2. Enable Remote Desktop on your VM and note down IP Address from Virt-Manager.
3. Anything you want the host to do upon VM activation.


# Let me know what works and what doesnt!
Let me know your success and failure stories.


#### [Fuel my coffee addiction or help me test new hardware](https://www.paypal.com/donate?business=87AQBT5TGFRJS&item_name=Github+Testing&currency_code=USD)
#### [This guide was forked from joeknock90 and edited for Legion 5 Pro. Here's a link to send some support if you desire.](https://www.paypal.com/donate?business=87AQBT5TGFRJS&item_name=Github+Testing&currency_code=USD)
#### Always appreciated, never required.

34 changes: 34 additions & 0 deletions qemu
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
#
# Author: SharkWipf
#
# Copy this file to /etc/libvirt/hooks, make sure it's called "qemu".
# After this file is installed, restart libvirt.
# From now on, you can easily add per-guest qemu hooks.
# Add your hooks in /etc/libvirt/hooks/qemu.d/vm_name/hook_name/state_name.
# For a list of available hooks, please refer to https://www.libvirt.org/hooks.html
#

GUEST_NAME="$1"
HOOK_NAME="$2"
STATE_NAME="$3"
MISC="${@:4}"

BASEDIR="$(dirname $0)"

HOOKPATH="$BASEDIR/qemu.d/$GUEST_NAME/$HOOK_NAME/$STATE_NAME"

set -e # If a script exits with an error, we should as well.

# check if it's a non-empty executable file
if [ -f "$HOOKPATH" ] && [ -s "$HOOKPATH" ] && [ -x "$HOOKPATH" ]; then
eval \"$HOOKPATH\" "$@"
elif [ -d "$HOOKPATH" ]; then
while read file; do
# check for null string
if [ ! -z "$file" ]; then
eval \"$file\" "$@"
fi
done <<< "$(find -L "$HOOKPATH" -maxdepth 1 -type f -executable -print;)"
fi

10 changes: 8 additions & 2 deletions example-revert.sh → revert.sh
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
#!/bin/bash
set -x


modprobe -r vfio-pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

# Re-Bind GPU to Nvidia Driver
virsh nodedev-reattach pci_0000_0c_00_1
virsh nodedev-reattach pci_0000_0c_00_0
virsh nodedev-reattach pci_0000_01_00_1 #Replace id with your gpu id number. Use lspci -nnk.
virsh nodedev-reattach pci_0000_01_00_0 #This too

# Reload nvidia modules
modprobe snd_hda_intel
modprobe nvidia
modprobe nvidia_modeset
modprobe nvidia_uvm
Expand Down
10 changes: 10 additions & 0 deletions setup-single-gpu-passthrough.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/fish
sudo mkdir -p /etc/libvirt/hooks/qemu.d/win11/{prepare/begin,release/end}

sudo cp qemu /etc/libvirt/hooks/
sudo chmod +x /etc/libvirt/hooks/qemu

sudo cp start.sh /etc/libvirt/hooks/qemu.d/win11/prepare/begin/
sudo cp revert.sh /etc/libvirt/hooks/qemu.d/win11/release/end/

sudo systemctl restart libvirtd
23 changes: 21 additions & 2 deletions example-start.sh → start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ set -x

# Stop display manager
systemctl stop display-manager.service
systemctl stop sddm.service
pulse_pid=$(pgrep -u igneel pulseaudio)
pipewire_pid=$(pgrep -u igneel pipewire-media)
kill $pulse_pid
kill $pipewire_pid

## Uncomment the following line if you use GDM
#killall gdm-x-session

Expand All @@ -17,6 +23,19 @@ echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
# Avoid a Race condition by waiting 2 seconds. This can be calibrated to be shorter or longer if required for your system
sleep 2

modprobe -r nvidia-drm
modprobe -r nvidia-uvm
modprobe -r snd_hda_intel
modprobe -r i2c_nvidia_gpu
modprobe -r nvidia

sleep 2

# Unbind the GPU from display driver
virsh nodedev-detach pci_0000_0c_00_0
virsh nodedev-detach pci_0000_0c_00_1
virsh nodedev-detach pci_0000_01_00_0 #Replace numbers with your specific pci id. Use lspci -nnk
virsh nodedev-detach pci_0000_01_00_1 # This one too

# Load VFIO Kernel Module
modprobe vfio-pci
modprobe vfio
modprobe vfio_iommu_type1