Post by Roger Pau MonnéHello
This is my first ever email list help request, apologies if I have not 100%
got the protocol correct.
I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
guests successfully using DRBD9 as primary root disk, but when testing an
HVM PfSense (BSD) guest, I am having some issues.
DRBD9 is successfully set up and when creating the guest using the config as
follows
disk=['drbd:disk1-firewall2,hda,w']
The guest will not create with the following info
xl -v create /etc/xen/firewall2.cfg
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: Loader: 0000000000100000->000000000015b844
xc: detail: Modules: 0000000000000000->0000000000000000
xc: detail: TOTAL: 0000000000000000->000000001f800000
xc: detail: ENTRY: 0000000000100630
xc: detail: 4KB PAGES: 0x0000000000000200
xc: detail: 2MB PAGES: 0x00000000000000fb
xc: detail: 1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74 device
model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device model
did not start: -3
libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy guest
with domid 74
libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
failed
Contents of guest log
qemu-system-i386: -drive
Could not open 'disk1-firewall2': No such file or directory
This is wrong, file= should point to some block device created by the
drbd hotplug script.
Does this mean there is an issue with the handoff from Xen to qemu while
building whichever command is required to start the HVM guest. I have
used the hotplug script in the documented method which works for PV
guests. Are you saying that using the following format
disk=['drbd:disk1-firewall2,hda,w']
Is not the correct way for HVM guests? I have confirmed, that using the
actual dev mount works correctly
(disk=['phy:/dev/vg/disk1-firewall2_00,hda,w']), but this will not live
migrate, or even start on any node besides the last node that was primary.
Post by Roger Pau MonnéContents of hotplug log
disk1-firewall2: State change failed: (-12) Device is held open by someone
Command 'drbdsetup secondary disk1-firewall2' terminated with exit code 11
I'm afraid this is due to block hotplug scripts being run twice for
HVM guests, once for the PV disk and another time for the emulated
disk.
This could be a red herring, the row show here may be from previous
attempts to start the guest. Subsequent creates have not seemed to
create this line. I have deleted the hotplug.log and rerun, and the file
is still empty, I have restarted the xen service, in case a new file
handle was required, but it still is empty after a few failed create
attempts.
Post by Roger Pau MonnéCould you please paste the full output of `xl -vvv create ...`, that
should contain all the calls to the hotplug scripts.
Parsing config from firewall2.cfg
libxl: debug: libxl_create.c:1563:do_domain_create: ao 0x1462fb0:
create: how=(nil) callback=(nil) poller=0x1462960
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=unknown
libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses
script=... assuming phy backend
libxl: debug: libxl_device.c:366:libxl__device_disk_set_backend: Disk
vdev=hda, using backend phy
libxl: debug: libxl_create.c:947:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV
domain, skipping bootloader
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1460808: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail: Loader: 0000000000100000->000000000015b844
xc: detail: Modules: 0000000000000000->0000000000000000
xc: detail: TOTAL: 0000000000000000->000000001f800000
xc: detail: ENTRY: 0000000000100630
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail: 4KB PAGES: 0x0000000000000200
xc: detail: 2MB PAGES: 0x00000000000000fb
xc: detail: 1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f53733d5000 -> 0x7f5373426dd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=phy
libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses
script=... assuming phy backend
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0:
register slotnum=3
libxl: debug: libxl_create.c:1586:do_domain_create: ao 0x1462fb0:
inprogress: poller=0x1462960, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60
wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:884:devstate_callback: backend
/local/domain/0/backend/vbd/85/768/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60
wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:880:devstate_callback: backend
/local/domain/0/backend/vbd/85/768/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0:
deregister slotnum=3
libxl: debug: libxl_device.c:991:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1463b60: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and
environment ready
libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug
script: /etc/xen/scripts/block-drbd add
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block-drbd add
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1463c60: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec
1, not running hotplug scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1463c60: deregister unregistered
libxl: debug: libxl_dm.c:1796:libxl__spawn_local_dm: Spawning
device-model /usr/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
/usr/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -xen-domid
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 85
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -no-shutdown
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -nodefaults
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -name
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: firewall2
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: :0,to=99
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -display
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -serial
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: pty
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -device
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
cirrus-vga,vgamem_mb=8
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -boot
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: order=c
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -net
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -machine
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: xenfv
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -m
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 504
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: -drive
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback
libxl: debug: libxl_dm.c:1800:libxl__spawn_local_dm: Spawning
device-model /usr/bin/qemu-system-i386 with additional environment:
libxl: debug: libxl_dm.c:1802:libxl__spawn_local_dm:
XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1:
register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1460b00
wpath=/local/domain/0/device-model/85/state token=3/1: event
epath=/local/domain/0/device-model/85/state
libxl: debug: libxl_exec.c:399:spawn_watch_event: domain 85 device
model: spawn watch p=(null)
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1:
deregister slotnum=3
libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 85
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device
model did not start: -3
libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2:
register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1467550
wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: event
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:880:devstate_callback: backend
/local/domain/0/backend/vbd/85/768/state wanted state 6 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2:
deregister slotnum=3
libxl: debug: libxl_device.c:991:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467550: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and
environment ready
libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug
script: /etc/xen/scripts/block-drbd remove
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block-drbd remove
libxl: debug: libxl_event.c:545:watchfd_callback: watch
epath=/local/domain/0/backend/vbd/85/768/state token=3/2: empty slot
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467650: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec
1, not running hotplug scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467650: deregister unregistered
libxl: debug: libxl_linux.c:267:libxl__get_hotplug_script_info:
backend_kind 6, no need to execute scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467ca0: deregister unregistered
libxl: debug: libxl.c:1738:devices_destroy_cb: forked pid 26093 for
destroy of domain 85
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462fb0:
complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462fb0: destroy
libxl: debug: libxl.c:1477:libxl_domain_destroy: ao 0x1462250: create:
how=(nil) callback=(nil) poller=0x1462960
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 85
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy
guest with domid 85
libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 85
failed
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462250:
complete, rc=-21
libxl: debug: libxl.c:1486:libxl_domain_destroy: ao 0x1462250:
inprogress: poller=0x1462960, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462250: destroy
xc: debug: hypercall buffer: total allocations:648 total releases:648
xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
xc: debug: hypercall buffer: cache current size:3
xc: debug: hypercall buffer: cache hits:632 misses:3 toobig:13
Post by Roger Pau MonnéAnd the guest is not started
xl block-list 0
Vdev BE handle state evt-ch ring-ref BE-path
xl block-attach 0 'drbd:disk1-firewall2,hda,w' (no errors or output)
xl block-list 0
Vdev BE handle state evt-ch ring-ref BE-path
768 0 0 4 102 8 /local/domain/0/backend/vbd/0/768
xl block-detach 0 768 (no errors or output)
xl block-list 0
Vdev BE handle state evt-ch ring-ref BE-path
So it mounts and unmounts fine manually, but not via the hotplug script?
When you attach to Dom0 you are also using the hotplug script.
OK, which is possibly a useless test then, as the hotplug script then
works as expected for PV, and when run manually via block-attach, so I
think this confirms the script in fact works, just not in the HVM case.