[Xen-users] HVM domU on storage driver domain

[G.R.]

Hi all,
I'm trying out the storage driver domain feature (one of the major
motivation to drive me out of my stable but ancient XEN 4.3.2 version, the
other motivation being the DISCARD feature).

I understand that HVM is NOT officially supported, but I played some tricks
to work it around.
The trick is to use file based storage and mount it through NFS and make
the same file accessible through the same path in both dom0 && NAS domU.
I have to disable one sanity check in libxl.c to make it stop complaining
about non-local disk image.

It's worthy to note that I wanted to grant read-only access to dom0 to
avoid any consistency / corruption issue.
I think this should be plausible for BIOS booting purpose. But for some
reason qemu refused to run with read-only permission.
Any solution?


What I get now is a booting HVM domU on storage driver domain.
By booting, I mean:
1. I passed the BIOS bootstrap phase which is supposed to backed by qemu.
2. The windows 7 OS is stuck in the boot screen forever :-)

Really a black joke, huh? The fact is that I have no way to determine what
is going on here.
Is it a performance issue or something has simply gone wrong?
Any suggestion on how to diagnose further?

Some observations:
1. The whole system is pretty idle with the booting win7 + NAS domU.
2. When I get the win7 into recover mode the disk spins up happily.
3. When win7 is booting, lsof reports nothing to the disk image on both
dom0 && NAS, as if the disk image is not being accessed.
4. In recover mode, I can see qemu accessing the disk image on dom0 from
lsof.
5. The NAS is freeNas 9.10 with FreeBSD 10 kernel. But I couldn't find any
clue about it hosting a xen disk (either dmesg or ps tree)

My current suspect is that may be the win 7 are using improper driver
during boot time?
(But both emulated / PV device should be accessible, hmm)

Suggestions on diagnose are welcome!

Thanks,
G.R.

[Austin S. Hemmelgarn]

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature (one of the major
motivation to drive me out of my stable but ancient XEN 4.3.2 version,
the other motivation being the DISCARD feature).
I understand that HVM is NOT officially supported, but I played some
tricks to work it around.
The trick is to use file based storage and mount it through NFS and make
the same file accessible through the same path in both dom0 && NAS domU.
I have to disable one sanity check in libxl.c to make it stop
complaining about non-local disk image.
It's worthy to note that I wanted to grant read-only access to dom0 to
avoid any consistency / corruption issue.
I think this should be plausible for BIOS booting purpose. But for some
reason qemu refused to run with read-only permission.
Any solution?
What I get now is a booting HVM domU on storage driver domain.
1. I passed the BIOS bootstrap phase which is supposed to backed by qemu.
2. The windows 7 OS is stuck in the boot screen forever :-)
Really a black joke, huh? The fact is that I have no way to determine
what is going on here.
Is it a performance issue or something has simply gone wrong?
Any suggestion on how to diagnose further?
1. The whole system is pretty idle with the booting win7 + NAS domU.
2. When I get the win7 into recover mode the disk spins up happily.
3. When win7 is booting, lsof reports nothing to the disk image on both
dom0 && NAS, as if the disk image is not being accessed.
4. In recover mode, I can see qemu accessing the disk image on dom0 from
lsof.
5. The NAS is freeNas 9.10 with FreeBSD 10 kernel. But I couldn't find
any clue about it hosting a xen disk (either dmesg or ps tree)
My current suspect is that may be the win 7 are using improper driver
during boot time?
(But both emulated / PV device should be accessible, hmm)
Suggestions on diagnose are welcome!

I don't have any specific suggestions regarding figuring out what's
going on, but I do have a suggestion for a work-around. Based on what
you've said, what you want is to use a domU running FreeNAS to provide
storage for other domU's on the same system. The de-facto method of
doing this sanely is to use a block-storage protocol instead of a
network filesystem to export the storage devices, and import the
block-device into dom0 (or a PV domU running as a storage driver domain)
then pass them to the VM's (or, if you're not using them as boot
devices, just import them in the VM directly). iSCSI is what usually
gets used for this, although if FreeNAS supports it I'd suggest using
ATAoE (which require near zero client-side setup when the client is
Linux (literally just load the module)) or NBD (which requires
significantly less client-side setup than iSCSI) as they're both much
simpler to set-up on the client side and have much less protocol
overhead than iSCSI. While I've never seen anyone try with a NAS
running as a domU itself, I know all three options work fine with Xen
when set up correctly.

[G.R.]

I don't have any specific suggestions regarding figuring out what's going
on, but I do have a suggestion for a work-around. Based on what you've
said, what you want is to use a domU running FreeNAS to provide storage for
other domU's on the same system. The de-facto method of doing this sanely
is to use a block-storage protocol instead of a network filesystem to
export the storage devices, and import the block-device into dom0 (or a PV
domU running as a storage driver domain) then pass them to the VM's (or, if
you're not using them as boot devices, just import them in the VM
directly). iSCSI is what usually gets used for this, although if FreeNAS
supports it I'd suggest using ATAoE (which require near zero client-side
setup when the client is Linux (literally just load the module)) or NBD
(which requires significantly less client-side setup than iSCSI) as they're
both much simpler to set-up on the client side and have much less protocol
overhead than iSCSI. While I've never seen anyone try with a NAS running
as a domU itself, I know all three options work fine with Xen when set up
correctly.

Hi Austin,
First of all thanks you for your suggestion.
What I'm trying to do here is partially for fun. I really doubt if I could
see real perf difference in production.
But isn't that really cool if it works?

Currently I'm using both NFS && iSCSI since they are supported by freeNAS
out-of-box.
Haven't payed attention to AOE && NBD before but will take a look for
future reference.

BTW, the NAS domU works great, really.

_______________________________________________
Xen-users mailing list
https://lists.xen.org/xen-users

[Kuba]

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature

Hi

A while ago, with a great deal of help from Roger Pau Monné, I managed
to use FreeBSD domU as storage driver domain to provide storage for
other domUs.

The main difference is that it didn't require any network-based protocol
(iSCSI etc.) between the domains.

Typically your domU's frontend driver is connected to a block device
inside dom0 via dom0's backend driver. But Xen has the ability to
connect your domU's frontend driver directly to the backend driver of
another domU. In short, you can create a storage driver domain, create a
block device inside it (e.g. a zvol) and than create another domU using
this block device directly, just as if it was provided by dom0.

Here you can find the steps that should get you started. It was a while
ago and required to apply a patch to Xen; I don't know what's its status
right now, but since FreeNAS is based on FreeBSD, it might be worth to
take a look:

https://lists.xenproject.org/archives/html/xen-users/2014-08/msg00003.html

Some time later I managed to set up a "true" storage driver domain using
PCI passthrough to assign SATA controller directly to the driver domain
and used that domain to provide storage for Windows-based guests. It
worked flawlessly. I believe this idea might be interesting to you too.

Regards,
Kuba

[G.R.]

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature

Hi
A while ago, with a great deal of help from Roger Pau Monné, I managed to
use FreeBSD domU as storage driver domain to provide storage for other
domUs.
The main difference is that it didn't require any network-based protocol
(iSCSI etc.) between the domains.
Typically your domU's frontend driver is connected to a block device
inside dom0 via dom0's backend driver. But Xen has the ability to connect
your domU's frontend driver directly to the backend driver of another domU.
In short, you can create a storage driver domain, create a block device
inside it (e.g. a zvol) and than create another domU using this block
device directly, just as if it was provided by dom0.
Here you can find the steps that should get you started. It was a while
ago and required to apply a patch to Xen; I don't know what's its status
right now, but since FreeNAS is based on FreeBSD, it might be worth to take
https://lists.xenproject.org/archives/html/xen-users/2014-08/msg00003.html

Hi Kuba,
The information you provided sounds fairly interesting! Thank you soooo
much~~
Strangely enough, the same patch quoted in your link is still relevant and
required after 2.5 years and 4 major release!
Roger, do you meant to submit your patch but some how get it lost?

Without the patch:
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring

With the patch:
Using xvda for guest's hda
******************* BLKFRONT for /local/domain/9/device/vbd/51712 **********


backend at /local/domain/1/backend/vbd/9/51712
156250000 sectors of 512 bytes
**************************
blk_open(/local/domain/9/device/vbd/51712) -> 5

However, I do NOT have the luck as Kuba had for a working system. (My first
attempt yesterday at least give me a booting screen :-))
What I see is the following errors:
Parsing config from ruibox.cfg
libxl: error: libxl_dm.c:1963:stubdom_xswait_cb: Stubdom 9 for 8 startup:
startup timed out
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model
did not start: -9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/9/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/8/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 8
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 8
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest
with domid 8
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 8 failed

Attaching the config and qemu-dm log for triage purpose.

Some time later I managed to set up a "true" storage driver domain using
PCI passthrough to assign SATA controller directly to the driver domain and
used that domain to provide storage for Windows-based guests. It worked
flawlessly. I believe this idea might be interesting to you too.

Actually my domU NAS has being running with PCI passhtroughed SATA
controller for 4 years.
It's just that the storage driver domain was NOT available in version 4.1.x
~ 4.3.2 when I built my box.

Regards,
Kuba
_______________________________________________
Xen-users mailing list
https://lists.xen.org/xen-users

[G.R.]

Hi Roger,
In case you missed this thread -- is stubdom supposed to work with storage
driver domain? I would need some help here.

Thanks,
Timothy

Post by G.R.

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature

Hi
A while ago, with a great deal of help from Roger Pau Monné, I managed to
use FreeBSD domU as storage driver domain to provide storage for other
domUs.
The main difference is that it didn't require any network-based protocol
(iSCSI etc.) between the domains.
Typically your domU's frontend driver is connected to a block device
inside dom0 via dom0's backend driver. But Xen has the ability to connect
your domU's frontend driver directly to the backend driver of another domU.
In short, you can create a storage driver domain, create a block device
inside it (e.g. a zvol) and than create another domU using this block
device directly, just as if it was provided by dom0.
Here you can find the steps that should get you started. It was a while
ago and required to apply a patch to Xen; I don't know what's its status
right now, but since FreeNAS is based on FreeBSD, it might be worth to take
https://lists.xenproject.org/archives/html/xen-users/2014-08
/msg00003.html

Hi Kuba,
The information you provided sounds fairly interesting! Thank you soooo
much~~
Strangely enough, the same patch quoted in your link is still relevant and
required after 2.5 years and 4 major release!
Roger, do you meant to submit your patch but some how get it lost?
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected
backend `/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected
backend `/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
Using xvda for guest's hda
******************* BLKFRONT for /local/domain/9/device/vbd/51712 **********
backend at /local/domain/1/backend/vbd/9/51712
156250000 sectors of 512 bytes
**************************
blk_open(/local/domain/9/device/vbd/51712) -> 5
However, I do NOT have the luck as Kuba had for a working system. (My
first attempt yesterday at least give me a booting screen :-))
Parsing config from ruibox.cfg
startup timed out
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device
model did not start: -9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/9/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/8/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 8
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 8
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy
guest with domid 8
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 8 failed
Attaching the config and qemu-dm log for triage purpose.

Some time later I managed to set up a "true" storage driver domain using
PCI passthrough to assign SATA controller directly to the driver domain and
used that domain to provide storage for Windows-based guests. It worked
flawlessly. I believe this idea might be interesting to you too.

Actually my domU NAS has being running with PCI passhtroughed SATA
controller for 4 years.
It's just that the storage driver domain was NOT available in version
4.1.x ~ 4.3.2 when I built my box.

Regards,
Kuba
_______________________________________________
Xen-users mailing list
https://lists.xen.org/xen-users

[Roger Pau Monné]

Post by G.R.

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature

Hi
A while ago, with a great deal of help from Roger Pau Monné, I managed to
use FreeBSD domU as storage driver domain to provide storage for other
domUs.
The main difference is that it didn't require any network-based protocol
(iSCSI etc.) between the domains.
Typically your domU's frontend driver is connected to a block device
inside dom0 via dom0's backend driver. But Xen has the ability to connect
your domU's frontend driver directly to the backend driver of another domU.
In short, you can create a storage driver domain, create a block device
inside it (e.g. a zvol) and than create another domU using this block
device directly, just as if it was provided by dom0.
Here you can find the steps that should get you started. It was a while
ago and required to apply a patch to Xen; I don't know what's its status
right now, but since FreeNAS is based on FreeBSD, it might be worth to take
https://lists.xenproject.org/archives/html/xen-users/2014-08/msg00003.html

Hi Kuba,
The information you provided sounds fairly interesting! Thank you soooo
much~~
Strangely enough, the same patch quoted in your link is still relevant and
required after 2.5 years and 4 major release!
Roger, do you meant to submit your patch but some how get it lost?

I guess I completely forgot about it and never properly sent it to the list,
sorry. The problem is that now I don't have a system that would allow me to
test it, so if I need to resend it I would need some confirmation that it's
still working as expected. From code inspection the issue seem to be there
still.

Post by G.R.
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
Using xvda for guest's hda
******************* BLKFRONT for /local/domain/9/device/vbd/51712 **********
backend at /local/domain/1/backend/vbd/9/51712
156250000 sectors of 512 bytes
**************************
blk_open(/local/domain/9/device/vbd/51712) -> 5
However, I do NOT have the luck as Kuba had for a working system. (My first
attempt yesterday at least give me a booting screen :-))
Parsing config from ruibox.cfg
startup timed out
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model
did not start: -9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/9/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/8/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 8
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 8
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest
with domid 8
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 8 failed

I'm not really sure about what's going wrong here, did you create the driver
domain guest with "driver_domain=1" in the config file?

Roger.

[G.R.]

Post by G.R.

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature

Hi
A while ago, with a great deal of help from Roger Pau Monné, I managed to
use FreeBSD domU as storage driver domain to provide storage for other
domUs.
The main difference is that it didn't require any network-based protocol
(iSCSI etc.) between the domains.
Typically your domU's frontend driver is connected to a block device
inside dom0 via dom0's backend driver. But Xen has the ability to connect
your domU's frontend driver directly to the backend driver of another domU.
In short, you can create a storage driver domain, create a block device
inside it (e.g. a zvol) and than create another domU using this block
device directly, just as if it was provided by dom0.
Here you can find the steps that should get you started. It was a while
ago and required to apply a patch to Xen; I don't know what's its status
right now, but since FreeNAS is based on FreeBSD, it might be worth to take
https://lists.xenproject.org/archives/html/xen-users/2014-

08/msg00003.html

Post by G.R.
Hi Kuba,
The information you provided sounds fairly interesting! Thank you soooo
much~~
Strangely enough, the same patch quoted in your link is still relevant and
required after 2.5 years and 4 major release!
Roger, do you meant to submit your patch but some how get it lost?

I guess I completely forgot about it and never properly sent it to the list,
sorry. The problem is that now I don't have a system that would allow me to
test it, so if I need to resend it I would need some confirmation that it's
still working as expected. From code inspection the issue seem to be there
still.

Yes, that patch is definitely required and working.
While through eyeballing I think the patch should be able to apply
directly, the automatic patch failed and I had to do it manually. I suspect
this is due to some format change to the patch in the email archive.

Post by G.R.
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
Using xvda for guest's hda
******************* BLKFRONT for /local/domain/9/device/vbd/51712 **********
backend at /local/domain/1/backend/vbd/9/51712
156250000 sectors of 512 bytes
**************************
blk_open(/local/domain/9/device/vbd/51712) -> 5
However, I do NOT have the luck as Kuba had for a working system. (My first
attempt yesterday at least give me a booting screen :-))
Parsing config from ruibox.cfg
startup timed out
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model
did not start: -9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/9/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/8/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 8
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 8
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest
with domid 8
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 8 failed

I'm not really sure about what's going wrong here, did you create the driver
domain guest with "driver_domain=1" in the config file?

Yes I did, even though I do not understand what is going on behind that
config. The manual is not clear enough and the tutorial even does not
mention it at all.

Do you have any suggestion about what I should do next to help
understanding the situation here?

G.R.


Roger.

[Roger Pau Monné]

Hello,

Could you please fix your mail client so that it properly quotes messages?
Not adding ">" to quotes makes it hard to follow the conversation.

Post by Kuba

Post by G.R.

Post by Kuba

Post by G.R.
Hi all,
I'm trying out the storage driver domain feature

Hi
A while ago, with a great deal of help from Roger Pau Monné, I managed

to

Post by G.R.

Post by Kuba
use FreeBSD domU as storage driver domain to provide storage for other
domUs.
The main difference is that it didn't require any network-based protocol
(iSCSI etc.) between the domains.
Typically your domU's frontend driver is connected to a block device
inside dom0 via dom0's backend driver. But Xen has the ability to

connect

Post by G.R.

Post by Kuba
your domU's frontend driver directly to the backend driver of another

domU.

Post by G.R.

Post by Kuba
In short, you can create a storage driver domain, create a block device
inside it (e.g. a zvol) and than create another domU using this block
device directly, just as if it was provided by dom0.
Here you can find the steps that should get you started. It was a while
ago and required to apply a patch to Xen; I don't know what's its status
right now, but since FreeNAS is based on FreeBSD, it might be worth to

take

Post by G.R.

Post by Kuba
https://lists.xenproject.org/archives/html/xen-users/2014-

08/msg00003.html

Post by G.R.
Hi Kuba,
The information you provided sounds fairly interesting! Thank you soooo
much~~
Strangely enough, the same patch quoted in your link is still relevant and
required after 2.5 years and 4 major release!
Roger, do you meant to submit your patch but some how get it lost?

I guess I completely forgot about it and never properly sent it to the list,
sorry. The problem is that now I don't have a system that would allow me to
test it, so if I need to resend it I would need some confirmation that it's
still working as expected. From code inspection the issue seem to be there
still.
Yes, that patch is definitely required and working.
While through eyeballing I think the patch should be able to apply
directly, the automatic patch failed and I had to do it manually. I suspect
this is due to some format change to the patch in the email archive.

OK, will try to find time to submit it.

Post by Kuba

Post by G.R.
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
frontend `/local/domain/5/device/vbd/51712' devtype `vbd' expected backend
`/local/domain/0/backend/qdisk/5/51712' got
`/local/domain/1/backend/vbd/5/51712', ignoring
Using xvda for guest's hda
******************* BLKFRONT for /local/domain/9/device/vbd/51712

**********

Post by G.R.
backend at /local/domain/1/backend/vbd/9/51712
156250000 sectors of 512 bytes
**************************
blk_open(/local/domain/9/device/vbd/51712) -> 5
However, I do NOT have the luck as Kuba had for a working system. (My

first

Post by G.R.
attempt yesterday at least give me a booting screen :-))
Parsing config from ruibox.cfg
startup timed out
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model
did not start: -9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/9/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 9
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed out
while waiting for /local/domain/1/backend/vbd/8/51712 to be removed
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 8
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 8
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy

guest

Post by G.R.
with domid 8
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 8

failed
I'm not really sure about what's going wrong here, did you create the driver
domain guest with "driver_domain=1" in the config file?
Yes I did, even though I do not understand what is going on behind that
config. The manual is not clear enough and the tutorial even does not
mention it at all.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Roger.

[G.R.]

Post by Roger Pau Monné
Hello,
Could you please fix your mail client so that it properly quotes messages?
Not adding ">" to quotes makes it hard to follow the conversation.

For unknown reason, gmail is using html mode... Should be fixed now.
Sorry for the inconvenience.

Post by Roger Pau Monné

Post by G.R.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Please find the xl log in the attachment.
Thanks!

[G.R.]

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Please find the xl log in the attachment.
Thanks!

Hi Roger,
Did you find any useful information in the xl -vvv log (from the previous mail)?

Thanks!
Rui

[G.R.]

Post by G.R.

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Please find the xl log in the attachment.
Thanks!

Hi Roger,
Did you find any useful information in the xl -vvv log (from the previous mail)?

Hi Roger,
Besides, the log attached above, I also find one old message from you:
https://lists.freebsd.org/pipermail/freebsd-xen/2016-June/002731.html
While you emphasized that's for dom0 support, I'm just wondering if,
by any tiny chance, it would affect FreeBSD as storage driver domain?
I'm on XEN 4.8 && FreeNAS 9.10 (FreeBSD 10.3 based) now.

Thanks,
Rui

_______________________________________________
Xen-users mailing lis

[G.R.]

Post by G.R.

Post by G.R.

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Please find the xl log in the attachment.
Thanks!

Hi Roger,
Did you find any useful information in the xl -vvv log (from the previous mail)?

Hi Roger,
https://lists.freebsd.org/pipermail/freebsd-xen/2016-June/002731.html
While you emphasized that's for dom0 support, I'm just wondering if,
by any tiny chance, it would affect FreeBSD as storage driver domain?
I'm on XEN 4.8 && FreeNAS 9.10 (FreeBSD 10.3 based) now.

Hi Roger,
Could you please provide some suggestions here? Really need some help.

Thanks,
Rui

[G.R.]

Post by G.R.

Post by G.R.

Post by G.R.

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Please find the xl log in the attachment.
Thanks!

Hi Roger,
Did you find any useful information in the xl -vvv log (from the previous mail)?

Hi Roger,
https://lists.freebsd.org/pipermail/freebsd-xen/2016-June/002731.html
While you emphasized that's for dom0 support, I'm just wondering if,
by any tiny chance, it would affect FreeBSD as storage driver domain?
I'm on XEN 4.8 && FreeNAS 9.10 (FreeBSD 10.3 based) now.

Hi Roger,
Could you please provide some suggestions here? Really need some help.

Hi Roger,
In case this is missed -- any chance you could take a look and comment?
Thanks,
Rui

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné
Hello,
Could you please fix your mail client so that it properly quotes messages?
Not adding ">" to quotes makes it hard to follow the conversation.

For unknown reason, gmail is using html mode... Should be fixed now.
Sorry for the inconvenience.

Post by Roger Pau Monné

Post by G.R.
Do you have any suggestion about what I should do next to help
understanding the situation here?

Can you provide the output with `xl -vvv ...`? That will be more verbose.

Please find the xl log in the attachment.
Thanks!

Sorry, I'm quite busy ATM and I don't have time to setup an environment in
order to try to reproduce your failure.

[...]

Post by G.R.
libxl: debug: libxl_device.c:361:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy
libxl: debug: libxl_device.c:270:disk_try_backend: Disk vdev=xvda, is using a storage driver domain, skipping physical device check
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x56365a68a700 wpath=/local/domain/1/backend/vbd/92/51712/state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a68a700 wpath=/local/domain/1/backend/vbd/92/51712/state token=3/1: event epath=/local/domain/1/backend/vbd/92/51712/state
libxl: debug: libxl_event.c:878:devstate_callback: backend /local/domain/1/backend/vbd/92/51712/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a68a700 wpath=/local/domain/1/backend/vbd/92/51712/state token=3/1: event epath=/local/domain/1/backend/vbd/92/51712/state
libxl: debug: libxl_event.c:874:devstate_callback: backend /local/domain/1/backend/vbd/92/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0x56365a68a700 wpath=/local/domain/1/backend/vbd/92/51712/state token=3/1: deregister slotnum=3
libxl: debug: libxl_device.c:1059:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x56365a68a700: deregister unregistered
libxl: debug: libxl_device.c:1114:device_hotplug: Backend domid 1, domid 0, assuming driver domains
libxl: debug: libxl_device.c:1117:device_hotplug: Not a remove, not executing hotplug scripts
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x56365a68a800: deregister unregistered
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: /usr/local/lib/xen/bin/qemu-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -d
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 92
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -domain-name
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: ruibox-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 0.0.0.0:0
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vncunused
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: event epath=/local/domain/0/device-model/92/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 92 device model: spawn watch p=(null)
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: event epath=/local/domain/0/device-model/92/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 92 device model: spawn watch p=running
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: deregister slotnum=3
libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 92 device model (dying as expected) [733] died due to fatal signal Killed

This seems suspicious, you shouldn't need a device-model for the subdomain
itself, can you please paste your domain config file?

Roger.

[G.R.]

Post by Roger Pau Monné

Post by G.R.
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: /usr/local/lib/xen/bin/qemu-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -d
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 92
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -domain-name
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: ruibox-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 0.0.0.0:0
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vncunused
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: event epath=/local/domain/0/device-model/92/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 92 device model: spawn watch p=(null)
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: event epath=/local/domain/0/device-model/92/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 92 device model: spawn watch p=running
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: deregister slotnum=3
libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 92 device model (dying as expected) [733] died due to fatal signal Killed

This seems suspicious, you shouldn't need a device-model for the subdomain
itself, can you please paste your domain config file?

So glad to see your comment!.
I can debug with your instruction, so no need to reproduce from your
side unless absolutely necessary.

The domU config file could be found in previous mails in this thread.
But reattach anyway:

There is yet other question I would like to see your comment, just to

Post by Roger Pau Monné
https://lists.freebsd.org/pipermail/freebsd-xen/2016-June/002731.html
While you emphasized that's for dom0 support, I'm just wondering if,
by any tiny chance, it would affect FreeBSD as storage driver domain?
I'm on XEN 4.8 && FreeNAS 9.10 (FreeBSD 10.3 based) now.

The background is that, as I mentioned in the very beginning of this
thread, I'm also trying to use local qemu device model + driver domain
by NFS mounting the remote disk image.
The domU appears to go through the BIOS boot into windows while get
stuck in the boot screen.
According to Paul the win-pv-driver owner, the debug log looks just
fine and it's probably that the back-end that is not working properly.
Email thread could be found here:
https://lists.xen.org/archives/html/win-pv-devel/2017-02/msg00027.html

Post by Roger Pau Monné
Roger.

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: /usr/local/lib/xen/bin/qemu-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -d
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 92
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -domain-name
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: ruibox-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 0.0.0.0:0
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vncunused
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: event epath=/local/domain/0/device-model/92/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 92 device model: spawn watch p=(null)
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: event epath=/local/domain/0/device-model/92/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 92 device model: spawn watch p=running
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0x56365a684860 wpath=/local/domain/0/device-model/92/state token=3/2: deregister slotnum=3
libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 92 device model (dying as expected) [733] died due to fatal signal Killed

This seems suspicious, you shouldn't need a device-model for the subdomain
itself, can you please paste your domain config file?

So glad to see your comment!.
I can debug with your instruction, so no need to reproduce from your
side unless absolutely necessary.
The domU config file could be found in previous mails in this thread.

Hm, AFAICT the problem seem to be that there's a device model launched to serve
the stubdomain, and that's completely wrong. The stubdomain shouldn't require a
device model at all IMHO.

Post by G.R.
There is yet other question I would like to see your comment, just to

Post by Roger Pau Monné
https://lists.freebsd.org/pipermail/freebsd-xen/2016-June/002731.html
While you emphasized that's for dom0 support, I'm just wondering if,
by any tiny chance, it would affect FreeBSD as storage driver domain?
I'm on XEN 4.8 && FreeNAS 9.10 (FreeBSD 10.3 based) now.

That's only relevant to FreeBSD >= 11, 10.3 should be fine without needing
anything else.

Post by G.R.
The background is that, as I mentioned in the very beginning of this
thread, I'm also trying to use local qemu device model + driver domain
by NFS mounting the remote disk image.
The domU appears to go through the BIOS boot into windows while get
stuck in the boot screen.
According to Paul the win-pv-driver owner, the debug log looks just
fine and it's probably that the back-end that is not working properly.
https://lists.xen.org/archives/html/win-pv-devel/2017-02/msg00027.html

Is the NFS share on a guest on the same host? I remember issues when trying to
do NFS from a guest and mounting the share on the Dom0.

Can you please try the following patch below and post the log of xl -vvv again:

---8<---
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 281058d..14014f4 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -2370,7 +2370,10 @@ int libxl__need_xenpv_qemu(libxl__gc *gc, libxl_domain_config *d_config)
goto out;
}

+ LOGD(DEBUG, domid, "Testing if domain needs DM");
+
if (d_config->num_vfbs > 0) {
+ LOGD(DEBUG, domid, "DM needed for vfbs");
ret = 1;
goto out;
}
@@ -2387,6 +2390,7 @@ int libxl__need_xenpv_qemu(libxl__gc *gc, libxl_domain_config *d_config)
for (i = 0; i < num; i++) {
if (dt->dm_needed(libxl__device_type_get_elem(dt, d_config, i),
domid)) {
+ LOGD(DEBUG, domid, "DM needed for disk %d", i);
ret = 1;
goto out;
}
@@ -2398,6 +2402,7 @@ int libxl__need_xenpv_qemu(libxl__gc *gc, libxl_domain_config *d_config)
/* xenconsoled is limited to the first console only.
Until this restriction is removed we must use qemu for
secondary consoles which includes all channels. */
+ LOGD(DEBUG, domid, "DM needed for console %d", i);
ret = 1;
goto out;
}

[G.R.]

Post by Roger Pau Monné

Post by G.R.

Post by G.R.
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: /usr/local/lib/xen/bin/qemu-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -d
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 92
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -domain-name
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: ruibox-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 0.0.0.0:0
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vncunused
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIMIT=1048576

Hm, AFAICT the problem seem to be that there's a device model launched to serve
the stubdomain, and that's completely wrong. The stubdomain shouldn't require a
device model at all IMHO.

I don't think I agree with your judgment.
With stubdom config, I could see a new domain launched along with the
domU in question:
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 6 r----- 137.0
nas 1 7936 2 -b---- 444.0
ruibox 6 2047 1 --p--- 0.0
ruibox-dm 7 32 1 -b---- 0.0

The 'ruibox-dm' name is referenced in the qemu-dm parameter log.
I think this comes from spawn_stub_launch_dm() which calls
libxl__spawn_local_dm() and prints out the log in question.

Also the debug patch you created are not being triggered at all.
The libxl__need_xenpv_qemu() function you touched are called in two situations:
1. in domcreate_launch_dm() for PV path, which is not applicable here
since it's HVM.
2. Indirectly called through libxl__dm_check_start() (likely device
hotplug related).

For #2, I don't see libxl_create.c call it directly.

Post by Roger Pau Monné

Post by G.R.
The background is that, as I mentioned in the very beginning of this
thread, I'm also trying to use local qemu device model + driver domain
by NFS mounting the remote disk image.
The domU appears to go through the BIOS boot into windows while get
stuck in the boot screen.
According to Paul the win-pv-driver owner, the debug log looks just
fine and it's probably that the back-end that is not working properly.
https://lists.xen.org/archives/html/win-pv-devel/2017-02/msg00027.html

Is the NFS share on a guest on the same host? I remember issues when trying to
do NFS from a guest and mounting the share on the Dom0.

Yes, NFS share exposed from the freeNAS domU on the same host.
It has to be that case since the same domU is being used as storage
driver domain in my experiment here.
I also experienced issue when mounting the share on Dom0 before.
The issue was discussed in this list years ago and is root caused to
be kernel config related.
I haven't see any other issue after fixing the kernel config.
In case you are interested in the background:
https://lists.xen.org/archives/html/xen-devel/2013-04/msg01302.html

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: /usr/local/lib/xen/bin/qemu-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -d
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 92
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -domain-name
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: ruibox-dm
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: 0.0.0.0:0
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -vncunused
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIMIT=1048576

Hm, AFAICT the problem seem to be that there's a device model launched to serve
the stubdomain, and that's completely wrong. The stubdomain shouldn't require a
device model at all IMHO.

I don't think I agree with your judgment.
With stubdom config, I could see a new domain launched along with the
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 6 r----- 137.0
nas 1 7936 2 -b---- 444.0
ruibox 6 2047 1 --p--- 0.0
ruibox-dm 7 32 1 -b---- 0.0

Yes, that's expected, it's the stubdomain (the domain running the device
model).

Post by G.R.
The 'ruibox-dm' name is referenced in the qemu-dm parameter log.
I think this comes from spawn_stub_launch_dm() which calls
libxl__spawn_local_dm() and prints out the log in question.

If it calls local_dm, it means it's launching a local device model (apart from
the stubdomain) which AFAICT it's wrong. The stubdomain itself shouldn't
require a device model on Dom0 in order to run (or at least that's my
recollection).

Post by G.R.
Also the debug patch you created are not being triggered at all.
1. in domcreate_launch_dm() for PV path, which is not applicable here
since it's HVM.

The stubdomain (ruibox-dm) it's a PV domain, so it should go that route
(libxl__need_xenpv_qemu).

Can you please paste the log with the patch applied? (and xl -vvv ...)

Roger.

[G.R.]

Post by Roger Pau Monné
Can you please paste the log with the patch applied? (and xl -vvv ...)

Thank you for clarifying my misunderstandings.
Please find the log in the attachment.
To be honest, I wasn't able to find any fundamental difference.
I even suspect if I applied the patch correctly. But I think its
correctly patched:

/usr/local/lib# grep 'Testing if domain needs DM' *.so
Binary file libxenlight.so matches

Thanks,
Rui

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné
Can you please paste the log with the patch applied? (and xl -vvv ...)

Thank you for clarifying my misunderstandings.
Please find the log in the attachment.
To be honest, I wasn't able to find any fundamental difference.
I even suspect if I applied the patch correctly. But I think its

Yes, it's correctly patched, it's just not using the function that I've
patched. Stupid question, but does the same exact config file work if you move
the virtual disk to Dom0? (but still using a stubdomain, and not changing any
other options).

Thanks, Roger.

[G.R.]

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné
Can you please paste the log with the patch applied? (and xl -vvv ...)

Thank you for clarifying my misunderstandings.
Please find the log in the attachment.
To be honest, I wasn't able to find any fundamental difference.
I even suspect if I applied the patch correctly. But I think its

Yes, it's correctly patched, it's just not using the function that I've
patched. Stupid question, but does the same exact config file work if you move
the virtual disk to Dom0? (but still using a stubdomain, and not changing any
other options).

I'll not be able to perform the exact experiment as you suggest since
my dom0 only has access to a usb drive with 2GB partition...
What I managed to do is to use a minimal liveCD as disk image (I also
changed the domain name, which shouldn't be relevant).

The stubdomain config works this time. Comparing the -vvv log, I can
also see the 'Spawning device-model ...' log.
The only difference is that, for the failing run, it shows:
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x5573d7180a18 wpath=/local/domain/9/device-model/8/state token=3/4:
register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x5573d7180a18 wpath=/local/domain/9/device-model/8/state token=3/4:
event epath=/local/domain/9/device-model/8/state

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: Stubdom 9 for 8 startup: xswait timeout (path=/local/domain/9/device-model/8/state)

libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x5573d7180a18 wpath=/local/domain/9/device-model/8/state token=3/4:
deregister slotnum=3
libxl: error: libxl_dm.c:1963:stubdom_xswait_cb: Stubdom 9 for 8
startup: startup timed out
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x5573d7180a18: deregister unregistered
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device
model did not start: -9

While for the working run, it shows:
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x55f894156668 wpath=/local/domain/20/device-model/19/state
token=3/4: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55f894156668 wpath=/local/domain/20/device-model/19/state
token=3/4: event epath=/local/domain/20/device-model/19/state

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x55f894156668 wpath=/local/domain/20/device-model/19/state token=3/4: event epath=/local/domain/20/device-model/19/state

libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x55f894156668 wpath=/local/domain/20/device-model/19/state
token=3/4: deregister slotnum=3
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x55f894163c50 wpath=/local/domain/0/backend/vif/19/0/state
token=3/5: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55f894163c50 wpath=/local/domain/0/backend/vif/19/0/state
token=3/5: event epath=/local/domain/0/backend/vif/19/0/state
libxl: debug: libxl_event.c:878:devstate_callback: backend
/local/domain/0/backend/vif/19/0/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55f894163c50 wpath=/local/domain/0/backend/vif/19/0/state
token=3/5: event epath=/local/domain/0/backend/vif/19/0/state
libxl: debug: libxl_event.c:874:devstate_callback: backend
/local/domain/0/backend/vif/19/0/state wanted state 2 ok

It's not clear to me what the time-outed watch is waiting for.
While the disk back-end is highly suspicious, the context in the
passing log looks more vif related.

BTW, a side product of my typo shows how the system responses to a
non-existing disk image in the storage driver domain.
This time vbd is explicitly mentioned in the log while no 'Spawning
device-model...' log is found.
I think this suggests that the back-end in FreeNAS domU is at least
not entirely unresponsive.

libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55bbe3fc9270 wpath=/local/domain/1/backend/vbd/16/51712/state
token=3/0: event epath=/local/domain/1/backend/vbd/16/51712/state
libxl: debug: libxl_event.c:878:devstate_callback: backend
/local/domain/1/backend/vbd/16/51712/state wanted state 2 still
waiting state 1
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: backend
/local/domain/1/backend/vbd/16/51712/state (hoping for state change to
2): xswait timeout (path=/local/domain/1/backend/vbd/16/51712/state)
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x55bbe3fc9270 wpath=/local/domain/1/backend/vbd/16/51712/state
token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:862:devstate_callback: backend
/local/domain/1/backend/vbd/16/51712/state wanted state 2 timed out
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x55bbe3fc9270: deregister unregistered
libxl: debug: libxl_device.c:1059:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x55bbe3fc9270: deregister unregistered
libxl: error: libxl_device.c:1074:device_backend_callback: unable to
add device with path /local/domain/1/backend/vbd/16/51712
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x55bbe3fc9370: deregister unregistered
libxl: error: libxl_create.c:1255:domcreate_launch_dm: unable to add
disk devices

Thanks,
Rui

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné
Can you please paste the log with the patch applied? (and xl -vvv ...)

Thank you for clarifying my misunderstandings.
Please find the log in the attachment.
To be honest, I wasn't able to find any fundamental difference.
I even suspect if I applied the patch correctly. But I think its

Yes, it's correctly patched, it's just not using the function that I've
patched. Stupid question, but does the same exact config file work if you move
the virtual disk to Dom0? (but still using a stubdomain, and not changing any
other options).

I'll not be able to perform the exact experiment as you suggest since
my dom0 only has access to a usb drive with 2GB partition...
What I managed to do is to use a minimal liveCD as disk image (I also
changed the domain name, which shouldn't be relevant).
The stubdomain config works this time. Comparing the -vvv log, I can
also see the 'Spawning device-model ...' log.
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
event epath=/local/domain/9/device-model/8/state

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: Stubdom 9 for 8 startup: xswait timeout (path=/local/domain/9/device-model/8/state)

libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
deregister slotnum=3
libxl: error: libxl_dm.c:1963:stubdom_xswait_cb: Stubdom 9 for 8
startup: startup timed out
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x5573d7180a18: deregister unregistered
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device
model did not start: -9
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x55f894156668 wpath=/local/domain/20/device-model/19/state
token=3/4: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55f894156668 wpath=/local/domain/20/device-model/19/state
token=3/4: event epath=/local/domain/20/device-model/19/state

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x55f894156668 wpath=/local/domain/20/device-model/19/state token=3/4: event epath=/local/domain/20/device-model/19/state

libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x55f894156668 wpath=/local/domain/20/device-model/19/state
token=3/4: deregister slotnum=3
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x55f894163c50 wpath=/local/domain/0/backend/vif/19/0/state
token=3/5: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55f894163c50 wpath=/local/domain/0/backend/vif/19/0/state
token=3/5: event epath=/local/domain/0/backend/vif/19/0/state
libxl: debug: libxl_event.c:878:devstate_callback: backend
/local/domain/0/backend/vif/19/0/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x55f894163c50 wpath=/local/domain/0/backend/vif/19/0/state
token=3/5: event epath=/local/domain/0/backend/vif/19/0/state
libxl: debug: libxl_event.c:874:devstate_callback: backend
/local/domain/0/backend/vif/19/0/state wanted state 2 ok
It's not clear to me what the time-outed watch is waiting for.
While the disk back-end is highly suspicious, the context in the
passing log looks more vif related.

Neither to me, it seems like the stubdomain doesn't start correctly, but the
stubdom log doesn't give any clues about what might cause it (or at least I'm
not able to identify them). Can you please apply the following patch, try to
create the domain (this time using the stubdom), and after 30s do `xenstore-ls
-fp` in the Dom0 and paste the results here?

This way I will know if the frontends in the stubdom have attached correctly.

Roger.

---8<---
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7722665..4d4d18f 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -95,7 +95,7 @@
#define LIBXL_DEVICE_MODEL_START_TIMEOUT 60
#define LIBXL_DEVICE_MODEL_SAVE_FILE XEN_LIB_DIR "/qemu-save" /* .$domid */
#define LIBXL_DEVICE_MODEL_RESTORE_FILE XEN_LIB_DIR "/qemu-resume" /* .$domid */
-#define LIBXL_STUBDOM_START_TIMEOUT 30
+#define LIBXL_STUBDOM_START_TIMEOUT 120
#define LIBXL_QEMU_BODGE_TIMEOUT 2
#define LIBXL_XENCONSOLE_LIMIT 1048576
#define LIBXL_XENCONSOLE_PROTOCOL "vt100"

[G.R.]

Post by Roger Pau Monné

Post by G.R.
It's not clear to me what the time-outed watch is waiting for.
While the disk back-end is highly suspicious, the context in the
passing log looks more vif related.

Neither to me, it seems like the stubdomain doesn't start correctly, but the
stubdom log doesn't give any clues about what might cause it (or at least I'm
not able to identify them). Can you please apply the following patch, try to
create the domain (this time using the stubdom), and after 30s do `xenstore-ls
-fp` in the Dom0 and paste the results here?
This way I will know if the frontends in the stubdom have attached correctly.

Please find the requested log in the attachment.
I also attached the qemu log from the stubdomain. Not very interesting
from my POV, just for reference.

Quoting how VBDs are shown up on different domU.
It sounds like both are correctly connected, even though different
number of ring pages are used.
*xvda for the running FreeNAS domU:
/local/domain/1/device/vbd = "" (n0,r1)
/local/domain/1/device/vbd/51712 = "" (n1,r0)
/local/domain/1/device/vbd/51712/backend =
"/local/domain/0/backend/vbd/1/51712" (n1,r0)
/local/domain/1/device/vbd/51712/backend-id = "0" (n1,r0)
/local/domain/1/device/vbd/51712/state = "4" (n1,r0)
/local/domain/1/device/vbd/51712/virtual-device = "51712" (n1,r0)
/local/domain/1/device/vbd/51712/device-type = "disk" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref0 = "768" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref1 = "769" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref2 = "770" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref3 = "771" (n1,r0)
/local/domain/1/device/vbd/51712/num-ring-pages = "4" (n1,r0)
/local/domain/1/device/vbd/51712/ring-page-order = "2" (n1,r0)
/local/domain/1/device/vbd/51712/event-channel = "8" (n1,r0)
/local/domain/1/device/vbd/51712/protocol = "x86_64-abi" (n1,r0)

*xvda for the ruibox-dm stubdomain:
/local/domain/3/device/vbd = "" (n0,r3)
/local/domain/3/device/vbd/51712 = "" (n3,r1)
/local/domain/3/device/vbd/51712/backend =
"/local/domain/1/backend/vbd/3/51712" (n3,r1)
/local/domain/3/device/vbd/51712/backend-id = "1" (n3,r1)
/local/domain/3/device/vbd/51712/state = "4" (n3,r1)
/local/domain/3/device/vbd/51712/virtual-device = "51712" (n3,r1)
/local/domain/3/device/vbd/51712/device-type = "disk" (n3,r1)
/local/domain/3/device/vbd/51712/protocol = "x86_64-abi" (n3,r1)
/local/domain/3/device/vbd/51712/ring-ref = "1789" (n3,r1)
/local/domain/3/device/vbd/51712/event-channel = "5" (n3,r1)


One interesting info is that failed stubdomain creation appears to
leave some trash in the system.
After some failed attempts, the storage driver domU stopped to response and the
xl -vvv create cannot proceed to 'Spawn device-model' step.
When doing xenstore-ls in that situation, I could find a long list of
VBDs in the FreeNAS domU's backend session.
After a reboot, everything backs to normal and I was able to collect
the attached log.

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
It's not clear to me what the time-outed watch is waiting for.
While the disk back-end is highly suspicious, the context in the
passing log looks more vif related.

Neither to me, it seems like the stubdomain doesn't start correctly, but the
stubdom log doesn't give any clues about what might cause it (or at least I'm
not able to identify them). Can you please apply the following patch, try to
create the domain (this time using the stubdom), and after 30s do `xenstore-ls
-fp` in the Dom0 and paste the results here?
This way I will know if the frontends in the stubdom have attached correctly.

Please find the requested log in the attachment.
I also attached the qemu log from the stubdomain. Not very interesting
from my POV, just for reference.
Quoting how VBDs are shown up on different domU.
It sounds like both are correctly connected, even though different
number of ring pages are used.
/local/domain/1/device/vbd = "" (n0,r1)
/local/domain/1/device/vbd/51712 = "" (n1,r0)
/local/domain/1/device/vbd/51712/backend =
"/local/domain/0/backend/vbd/1/51712" (n1,r0)
/local/domain/1/device/vbd/51712/backend-id = "0" (n1,r0)
/local/domain/1/device/vbd/51712/state = "4" (n1,r0)
/local/domain/1/device/vbd/51712/virtual-device = "51712" (n1,r0)
/local/domain/1/device/vbd/51712/device-type = "disk" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref0 = "768" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref1 = "769" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref2 = "770" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref3 = "771" (n1,r0)
/local/domain/1/device/vbd/51712/num-ring-pages = "4" (n1,r0)
/local/domain/1/device/vbd/51712/ring-page-order = "2" (n1,r0)
/local/domain/1/device/vbd/51712/event-channel = "8" (n1,r0)
/local/domain/1/device/vbd/51712/protocol = "x86_64-abi" (n1,r0)
/local/domain/3/device/vbd = "" (n0,r3)
/local/domain/3/device/vbd/51712 = "" (n3,r1)
/local/domain/3/device/vbd/51712/backend =
"/local/domain/1/backend/vbd/3/51712" (n3,r1)
/local/domain/3/device/vbd/51712/backend-id = "1" (n3,r1)
/local/domain/3/device/vbd/51712/state = "4" (n3,r1)
/local/domain/3/device/vbd/51712/virtual-device = "51712" (n3,r1)
/local/domain/3/device/vbd/51712/device-type = "disk" (n3,r1)
/local/domain/3/device/vbd/51712/protocol = "x86_64-abi" (n3,r1)
/local/domain/3/device/vbd/51712/ring-ref = "1789" (n3,r1)
/local/domain/3/device/vbd/51712/event-channel = "5" (n3,r1)

Hm, can you please paste the full output of xenstore-ls -fp, I also want to see
if the stubdomain correctly sets the "initialized" stated
(/local/domain/XX/device-model/YY/state), and the state of the backend in the
driver domain (/local/domain/1/backend/).

Roger.

[G.R.]

Post by Roger Pau Monné

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
It's not clear to me what the time-outed watch is waiting for.
While the disk back-end is highly suspicious, the context in the
passing log looks more vif related.

Neither to me, it seems like the stubdomain doesn't start correctly, but the
stubdom log doesn't give any clues about what might cause it (or at least I'm
not able to identify them). Can you please apply the following patch, try to
create the domain (this time using the stubdom), and after 30s do `xenstore-ls
-fp` in the Dom0 and paste the results here?
This way I will know if the frontends in the stubdom have attached correctly.

Please find the requested log in the attachment.
I also attached the qemu log from the stubdomain. Not very interesting
from my POV, just for reference.
Quoting how VBDs are shown up on different domU.
It sounds like both are correctly connected, even though different
number of ring pages are used.
/local/domain/1/device/vbd = "" (n0,r1)
/local/domain/1/device/vbd/51712 = "" (n1,r0)
/local/domain/1/device/vbd/51712/backend =
"/local/domain/0/backend/vbd/1/51712" (n1,r0)
/local/domain/1/device/vbd/51712/backend-id = "0" (n1,r0)
/local/domain/1/device/vbd/51712/state = "4" (n1,r0)
/local/domain/1/device/vbd/51712/virtual-device = "51712" (n1,r0)
/local/domain/1/device/vbd/51712/device-type = "disk" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref0 = "768" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref1 = "769" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref2 = "770" (n1,r0)
/local/domain/1/device/vbd/51712/ring-ref3 = "771" (n1,r0)
/local/domain/1/device/vbd/51712/num-ring-pages = "4" (n1,r0)
/local/domain/1/device/vbd/51712/ring-page-order = "2" (n1,r0)
/local/domain/1/device/vbd/51712/event-channel = "8" (n1,r0)
/local/domain/1/device/vbd/51712/protocol = "x86_64-abi" (n1,r0)
/local/domain/3/device/vbd = "" (n0,r3)
/local/domain/3/device/vbd/51712 = "" (n3,r1)
/local/domain/3/device/vbd/51712/backend =
"/local/domain/1/backend/vbd/3/51712" (n3,r1)
/local/domain/3/device/vbd/51712/backend-id = "1" (n3,r1)
/local/domain/3/device/vbd/51712/state = "4" (n3,r1)
/local/domain/3/device/vbd/51712/virtual-device = "51712" (n3,r1)
/local/domain/3/device/vbd/51712/device-type = "disk" (n3,r1)
/local/domain/3/device/vbd/51712/protocol = "x86_64-abi" (n3,r1)
/local/domain/3/device/vbd/51712/ring-ref = "1789" (n3,r1)
/local/domain/3/device/vbd/51712/event-channel = "5" (n3,r1)

Hm, can you please paste the full output of xenstore-ls -fp, I also want to see
if the stubdomain correctly sets the "initialized" stated
(/local/domain/XX/device-model/YY/state), and the state of the backend in the
driver domain (/local/domain/1/backend/).

Hi Roger,
The full log is in the attachment file 'store9'.

Thanks,
Rui

[Roger Pau Monné]

Post by G.R.
Hi Roger,
The full log is in the attachment file 'store9'.

AFAICT there are no attachments on this email.

Roger.

[G.R.]

Post by Roger Pau Monné

Post by G.R.
Hi Roger,
The full log is in the attachment file 'store9'.

AFAICT there are no attachments on this email.
Roger.

I mean the previous one. Anyway, reattach.

[Roger Pau Monné]

Post by G.R.

Post by Roger Pau Monné

Post by G.R.
Hi Roger,
The full log is in the attachment file 'store9'.

AFAICT there are no attachments on this email.
Roger.

I mean the previous one. Anyway, reattach.

Oh, sorry, I didn't realize there was an attachment to that email. It looks
like the stubdom is able to connect to all the frontends and backends (both the
disk and the nic are in state "4"), however the QEMU inside the stubdomain is
not able to start. It should set:

/local/domain/3/device-model/2/state = "running"

And that's not happening. I suggest that you add some debug printf's to QEMU
and rebuild the stubdomain in order to know why QEMU is not starting up
properly.

Roger.

[G.R.]

Post by Roger Pau Monné
Oh, sorry, I didn't realize there was an attachment to that email. It looks
like the stubdom is able to connect to all the frontends and backends (both the
disk and the nic are in state "4"), however the QEMU inside the stubdomain is
/local/domain/3/device-model/2/state = "running"
And that's not happening. I suggest that you add some debug printf's to QEMU
and rebuild the stubdomain in order to know why QEMU is not starting up
properly.
Roger.

I'm trying as you suggested, but still get confused.

The log I'm seeing is:
stubdom dbg=13
stubdom dbg=14
stubdom dbg=15, vm_running=1, paused=0
xen be: console-1: xen be: console-1: initialise() failed
initialise() failed

Which corresponds to the following debug log I added:
626 printf("stubdom dbg=13\n");
627 xenstore_record_dm_state("running");
628
629 printf("stubdom dbg=14\n");
630 qemu_set_fd_handler(xenstore_fd(), xenstore_process_event, NULL, NULL);
631
632 printf("stubdom dbg=15, vm_running=%d, paused=%d\n",
vm_running, xen_pause_requested);
633 while (1) {
634 while (!(vm_running && xen_pause_requested))
635 #ifdef CONFIG_STUBDOM
636 /* Wait up to 10 msec. */
637 main_loop_wait(10);
638 #else
639 /* Wait up to 1h. */
640 main_loop_wait(1000*60*60);
641 #endif
642
643 printf("stubdom dbg=16\n");

So the 'running' stage should have been reached, at least once.
But still, libxl complain about device-model timeout:
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x5597e5a36a18 wpath=/local/domain/13/device-model/12/state
token=3/4: register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch
w=0x5597e5a36a18 wpath=/local/domain/13/device-model/12/state
token=3/4: event epath=/local/domain/13/device-model/12/state
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: Stubdom 13
for 12 startup: xswait timeout
(path=/local/domain/13/device-model/12/state)
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x5597e5a36a18 wpath=/local/domain/13/device-model/12/state
token=3/4: deregister slotnum=3
libxl: error: libxl_dm.c:1963:stubdom_xswait_cb: Stubdom 13 for 12
startup: startup timed out
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x5597e5a36a18: deregister unregistered
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device
model did not start: -9

***Is the timeout really waiting for 'running' state?***
***Is the 'running' state going to be consumed and need to be regenerated?***
***Is there any other event that libxl is watching for?***

BTW2, there are some time out error waiting for the disk from storage
driver domain to be removed upon destroy:
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 13
libxl: debug: libxl.c:1712:devices_destroy_cb: forked pid 12559 for
destroy of domain 13
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: removal of
backend path: xswait timeout
(path=/local/domain/1/backend/vbd/12/51712)
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x5597e5a404d0 wpath=/local/domain/1/backend/vbd/12/51712 token=1/9:
deregister slotnum=1
libxl: error: libxl_device.c:1264:device_destroy_be_watch_cb: timed
out while waiting for /local/domain/1/backend/vbd/12/51712 to be
removed
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x5597e5a404d0: deregister unregistered
libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy
failed for 12