Talk about flakey behaviour! Luckily I’m not the first but probably not the last to figure out that some USB sticks just doesn’t play nice with some Raspberrys.
My Aeotec Z-Stick Gen 5 wasn’t recognized as expected by my RPi4.
The solution is rather simple, put a dumb USB 2.0 hub in between, and the USB stick is recognized by the Raspberry!

When using a USB 2.0 hub, the stick magically appears, as expected!

And is now selectable to pass to the VM

And is usable within the guest VM (Ubuntu hosting HA in a Docker container)