Re: [RFC] Resizable BARs vs bridges with BARs
From: Christian König
Date: Sat Nov 19 2022 - 14:14:52 EST
Hi Alex,
Am 19.11.22 um 15:07 schrieb Alex Williamson:
Hi Christian,
On Sat, 19 Nov 2022 12:02:55 +0100
Christian König <christian.koenig@xxxxxxx> wrote:
Hi Alex,
Am 19.11.22 um 00:09 schrieb Alex Williamson:
Hi,
I'm trying to get resizable BARs working in a configuration where my
root bus resources provide plenty of aperture for the BAR:
pci_bus 0000:5d: root bus resource [io 0x8000-0x9fff window]
pci_bus 0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]
pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window] <<<
pci_bus 0000:5d: root bus resource [bus 5d-7f]
But resizing fails with -ENOSPC. The topology looks like this:
+-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0 Intel Corporation DG2 [Arc A380]
\-04.0-[61]----00.0 Intel Corporation Device 4f92
The BIOS is not fluent in resizable BARs and only programs the root
port with a small aperture:
5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) (prog-if 00 [Normal decode])
Bus: primary=5d, secondary=5e, subordinate=61, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: b9000000-ba0fffff [size=17M]
Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M]
Kernel driver in use: pcieport
The trouble comes on the upstream PCIe switch port:
5e:00.0 PCI bridge: Intel Corporation Device 4fa1 (rev 01) (prog-if 00 [Normal decode])
>>> Region 0: Memory at b010000000 (64-bit, prefetchable)
Bus: primary=5e, secondary=5f, subordinate=61, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled]
Memory behind bridge: b9000000-ba0fffff [size=17M]
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M]
Kernel driver in use: pcieport
Note region 0 of this bridge, which is 64-bit, prefetchable and
therefore conflicts with the same type for the resizable BAR on the GPU:
60:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05) (prog-if 00 [VGA controller])
Region 0: Memory at b9000000 (64-bit, non-prefetchable) [disabled] [size=16M]
Region 2: Memory at bfe0000000 (64-bit, prefetchable) [disabled] [size=256M]
Expansion ROM at <ignored> [disabled]
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
It's a shame that the hardware designers didn't mark the upstream port
BAR as non-prefetchable to avoid it living in the same resource
aperture as the resizable BAR on the downstream device.
This is expected. Bridges always have a 32bit non prefetchable and a
64bit prefetchable BAR. This is part of the PCI(e) spec.
To be clear, the issue is a bridge implementing a 64-bit, prefetchable
BAR at config offset 0x10 & 0x14, not the limit/base registers that
define the bridge windows for prefetchable and non-prefetchable
downstream resources.
WHAT? I've never heard of a bridge with this configuration. I don't
fully remember the spec, but I'm pretty sure that this isn't something
standard.
Can you give me the output of "sudo lspci -vvvv -s $busID" for this device.
In any case, it's my understanding that our bridge drivers don't generally make use
of bridge BARs. I think we can test whether a driver has done a
pci_request_region() or equivalent by looking for the IORESOURCE_BUSY
flag, but I also suspect this is potentially racy.
That sounds like we have a misunderstanding here how those bridges work.
The upstream bridges should include all the resources of the downstream
devices/bridges in their BARs.
Correct, and the issue is that the bridge at 5e:00.0 _consumes_ a
portion of the window we need to resize at the root port.
Root port:
Prefetchable memory behind bridge: 000000bfe0000000-000000bff07fffff [size=264M]
Upstream switch port:
Region 0: Memory at b010000000 (64-bit, prefetchable)
Prefetchable memory behind bridge: 000000bfe0000000-000000bfefffffff [size=256M]
It's that Region 0 resource that prevents resizing.
Could it be that some of the ACPI tables are broken and because of this
we add a fixed resource to this device?
Otherwise I have a hard time coming up with a way for a bridge to have a
BAR in the config space.
The patch below works for me, allowing the new resourceN_resize sysfs
attribute to resize the root port window within the provided bus
window. Is this the right answer? How can we make it feel less
sketchy? Thanks,
The correct approach is to remove all the drivers (EFI, vesafb etc...)
which are using the PCI(e) devices under the bridge in question. Then
release the resources and puzzle everything back together.
See amdgpu_device_resize_fb_bar() how to do this correctly.
Resource resizing in pci-sysfs is largely modeled after the amdgpu
code, but I don't see any special provisions for handling conflicting
resources consumed on intermediate devices. The driver attached to the
upstream switch port is pcieport and removing it doesn't resolve the
problem. The necessary resource on the root port still reports a
child.
Is amdgppu resizing known to work in cases where the GPU is downstream
of a PCIe switch that consumes resources of the same type and the root
port aperture needs to be resized? I suspect it does not. Thanks,
Well we have the possibility to add extra space to bridges on the kernel
command line for this.
This is used for things like hotplug behind bridges with limited address
space.
Quite a while ago there was also a patch set which dynamically
binds/unbinds drivers from resources to resize the BARs. But that never
got far because of locking problems.
Regards,
Christian.
Alex