[Xen-users] Future of 32-bit PV support

[Juergen Gross]

In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
anyone would see problems with:

- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13

- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer

And related to that:

- is there any Linux distribution still shipping 32-bit PV-capable
systems?

- what about BSD? Is 32-bit PV support important there?


Juergen

[Jan Beulich]

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

* would this proposal affect the ability to use a 32-bit dom0?

If the Dom0 is to be PV - yes, of course. For the time being there's
no complete PVH Dom0 support, so if 32-bit is needed here, PV is
for now indeed the only option.

* is this proposal to affect x86 only or also for ARM?

There's no PV on ARM.

Jan

[Juergen Gross]

Post by Jan Beulich

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

* would this proposal affect the ability to use a 32-bit dom0?

If the Dom0 is to be PV - yes, of course. For the time being there's
no complete PVH Dom0 support, so if 32-bit is needed here, PV is
for now indeed the only option.

And to be more precise: the first step would be to remove 32-bit PV
support from upstream Linux kernel. This would result in the loss of
the ability to use a _new_ Linux (e.g. >= 4.20 / 5.0) as a 32-bit dom0.
A 32-bit dom0 using a kernel <= 4.19 would still work until we remove
32-bit PV support from the hypervisor (which we wouldn't do before full
support of PVH dom0, I guess).

Is there a special reason you want to use a 32-bit dom0?


Juergen

[Christopher Clark]

Post by Juergen Gross

Post by Jan Beulich

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the

newest

Post by Jan Beulich

Post by Juergen Gross
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

* would this proposal affect the ability to use a 32-bit dom0?

If the Dom0 is to be PV - yes, of course. For the time being there's
no complete PVH Dom0 support, so if 32-bit is needed here, PV is
for now indeed the only option.

ack. I asked because it's not necessarily obvious to all that dom0 is
included in the term "guest", and it widens the consequences of this change.

Post by Juergen Gross
And to be more precise: the first step would be to remove 32-bit PV
support from upstream Linux kernel. This would result in the loss of
the ability to use a _new_ Linux (e.g. >= 4.20 / 5.0) as a 32-bit dom0.
A 32-bit dom0 using a kernel <= 4.19 would still work until we remove
32-bit PV support from the hypervisor (which we wouldn't do before full
support of PVH dom0, I guess).

That makes sense.

Post by Juergen Gross
Is there a special reason you want to use a 32-bit dom0?

In short, no.
OpenXT currently uses a 32-bit PV dom0, but work is already under way to
migrate to 64-bit. I think your proposal is justified and good.

Christopher

[Florian Heigl]

Hi Juergen,

this will sound like only a rant now, but the answer appears very simple and logical
"If you have to ask, it's far too late..."

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether

Noone would benefit rely on any software flavour where awareness is so low that the core team doesn't even know if someone uses it.
(Yes, I have used it. 32bit Xen stuff was already undermaintained almost a decade ago. Definitely it has not biased me in a positive way, and yes I also lack recent experience since I will NEVER try that again)

It is likely better if you drop the support.

Greetings,
Flo

tl;dr: The more time you got for other parts of Xen that are widely used, the better.

[Roger Pau Monné]

Post by Florian Heigl
Hi Juergen,
this will sound like only a rant now, but the answer appears very simple and logical
"If you have to ask, it's far too late..."

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether

Noone would benefit rely on any software flavour where awareness is so low that the core team doesn't even know if someone uses it.
(Yes, I have used it. 32bit Xen stuff was already undermaintained almost a decade ago. Definitely it has not biased me in a positive way, and yes I also lack recent experience since I will NEVER try that again)

This is being actively tested by osstest, the Xen test system:

http://logs.test-lab.xenproject.org/osstest/logs/125892/test-amd64-i386-xl/info.html

Just search for i386 on the osstest matrix:

http://logs.test-lab.xenproject.org/osstest/logs/125892/

osstest have a bunch of tests that run on a 32bit PV Dom0 and a bunch
of them that create 32bit PV guests, so it's certainly not
unmaintained.

Roger.

[Roger Pau Monné]

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13

This means 32bit PV support would be switched off by default in the
hypervisor build, but distros or individuals could still enable it
(like the build system will enable it for the shim).

Post by Juergen Gross
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

FTR, NetBSD is the only BSD to have PV support, both 32 and 64bits.

Roger.

[Andy Smith]

Hi Juergen,

As this was also addressed to -user I'm going to assume that you do
want user response as well.

Post by Juergen Gross
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13

Although amd64 has been the default for us for many years, at the
moment we still have 64% of our customers running 32-bit PV. If
there remains a way for us to boot them through PV-shim and then
pvgrub2 with no functional changes and no work inside the guest then
that's fine, we'll adapt.

Post by Juergen Gross
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer

I doubt there is any technical reason why they can't switch to
64-bit, it's just that in the majority of cases that involves a
complete reinstall and the users just haven't bothered to.

If they are forced to switch because an impending kernel update will
leave them with a kernel that doesn't boot, they are going to be
upset that they are forced to reinstall their guest, or switch to a
64-bit kernel with their existing 32-bit userland.

It will of course help if they have plenty of warning that they need
to make the switch. But unless we're talking 2+ years of warning I'm
sure there will be some who will be unhappy.

I was hoping to transition to PVH guests as soon as possible, but
last time I looked into it there was a problem booting the stable
Linux kernel under PVH, and also no support in grub2.

Will it remain possible to boot a 32-bit Linux guest in PVH mode?

If so, could the final removal of 32-bit PV in the Linux kernel be
held off until there is:

1) a kernel shipping in Debian stable, Ubuntu LTS and CentOS that
boots under PVH, and;

2) support in grub2 so I can build a grub image that boots under
PVH?

If grub PVH support is not going to happen, what is the roadmap for
user-specified guest kernels under PVH?

Post by Juergen Gross
- is there any Linux distribution still shipping 32-bit PV-capable
systems?

Debian stable 32-bit kernels still boot under PV, as do Ubuntu 18.04
LTS ones. Ubuntu LTS releases are supposed to be supported (by
Canonical) for 5 years, and while of course Xen does not fall under
the category of software that they support, there will be people
sticking with 18.04 LTS as long as they can.

I'm not saying that people running 32-bit PV Ubuntu 18.04 are right
to expect that to continue being supported until 2023. I'm just
saying that human nature dictates that those sorts of expectations
will exist.

It will help a lot if there is an easy way for us to switch them
from 32-bit PV to PVH, while still letting them install their own
kernels.

Cheers,
Andy

[Juergen Gross]

Post by Florian Heigl
Hi Juergen,
As this was also addressed to -user I'm going to assume that you do
want user response as well.

Right. Thanks for responding.

Post by Florian Heigl

Post by Juergen Gross
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13

Although amd64 has been the default for us for many years, at the
moment we still have 64% of our customers running 32-bit PV. If
there remains a way for us to boot them through PV-shim and then
pvgrub2 with no functional changes and no work inside the guest then
that's fine, we'll adapt.

Post by Juergen Gross
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer

I doubt there is any technical reason why they can't switch to
64-bit, it's just that in the majority of cases that involves a
complete reinstall and the users just haven't bothered to.

Is something like missing Meltdown mitigation for 32-bit PV guest a
technical reason?

Post by Florian Heigl
If they are forced to switch because an impending kernel update will
leave them with a kernel that doesn't boot, they are going to be
upset that they are forced to reinstall their guest, or switch to a
64-bit kernel with their existing 32-bit userland.
It will of course help if they have plenty of warning that they need
to make the switch. But unless we're talking 2+ years of warning I'm
sure there will be some who will be unhappy.
I was hoping to transition to PVH guests as soon as possible, but
last time I looked into it there was a problem booting the stable
Linux kernel under PVH, and also no support in grub2.

Okay, noted.

Post by Florian Heigl
Will it remain possible to boot a 32-bit Linux guest in PVH mode?

Yes.

Post by Florian Heigl
If so, could the final removal of 32-bit PV in the Linux kernel be
1) a kernel shipping in Debian stable, Ubuntu LTS and CentOS that
boots under PVH, and;
2) support in grub2 so I can build a grub image that boots under
PVH?

I think this is a reasonable request.

Post by Florian Heigl
If grub PVH support is not going to happen, what is the roadmap for
user-specified guest kernels under PVH?

I have a patch series lying around for grub2 PVH support. It requires
some rework and another kernel enhancement. I'll try to resume work on
the patches soon.

Post by Florian Heigl

Post by Juergen Gross
- is there any Linux distribution still shipping 32-bit PV-capable
systems?

Debian stable 32-bit kernels still boot under PV, as do Ubuntu 18.04
LTS ones. Ubuntu LTS releases are supposed to be supported (by
Canonical) for 5 years, and while of course Xen does not fall under
the category of software that they support, there will be people
sticking with 18.04 LTS as long as they can.

I guess they will stick to the stable kernel they are using now? Then
this will be no problem.

Post by Florian Heigl
I'm not saying that people running 32-bit PV Ubuntu 18.04 are right
to expect that to continue being supported until 2023. I'm just
saying that human nature dictates that those sorts of expectations
will exist.
It will help a lot if there is an easy way for us to switch them
from 32-bit PV to PVH, while still letting them install their own
kernels.

In the end it should be just a switch of domain type and boot loader
(PV -> PVH, grubxen -> grubxen-pvh). The kernel needs to be configured
to support PVH, of course.

Thanks for the very valuable input!


Juergen

[Hans van Kranenburg]

Post by Juergen Gross

Post by Florian Heigl
Hi Juergen,
[...]
If so, could the final removal of 32-bit PV in the Linux kernel be
1) a kernel shipping in Debian stable, Ubuntu LTS and CentOS that
boots under PVH, and;
2) support in grub2 so I can build a grub image that boots under
PVH?

I think this is a reasonable request.

Post by Florian Heigl
If grub PVH support is not going to happen, what is the roadmap for
user-specified guest kernels under PVH?

I have a patch series lying around for grub2 PVH support. It requires
some rework and another kernel enhancement. I'll try to resume work on
the patches soon.

+1, Yes please!

Do I understand correctly ('another kernel enhancement') that you found
more things which need to be done than the 32-bit memory map limitation
issue at the grub side?

Post by Juergen Gross

Post by Florian Heigl
[...]

Thanks,
Hans

[Juergen Gross]

Post by Hans van Kranenburg

Post by Juergen Gross

Post by Florian Heigl
Hi Juergen,
[...]
If so, could the final removal of 32-bit PV in the Linux kernel be
1) a kernel shipping in Debian stable, Ubuntu LTS and CentOS that
boots under PVH, and;
2) support in grub2 so I can build a grub image that boots under
PVH?

I think this is a reasonable request.

Post by Florian Heigl
If grub PVH support is not going to happen, what is the roadmap for
user-specified guest kernels under PVH?

I have a patch series lying around for grub2 PVH support. It requires
some rework and another kernel enhancement. I'll try to resume work on
the patches soon.

+1, Yes please!
Do I understand correctly ('another kernel enhancement') that you found
more things which need to be done than the 32-bit memory map limitation
issue at the grub side?

The kernel is still lacking support for handing over RSDP address from
grub to the kernel. The initial patches adding that haven't been taken
as several distributions are shipping a patched grub2 which is breaking
the grub/kernel boot protocol by clobbering kernel data.

I need to extend that protocol to detect such a clobbering in order to
be able to tell the difference between grub having set a new parameter
or just written garbage to it.


Juergen

[Stefano Stabellini]

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

Hi Juergen,

Although I can see that deprecating 32-bit PV guest support is
desirable, and it might not cause any problems to Linux and
BSDs, we need to be careful about unikernels.

There are probably unikernels out there that only support PV 32bit
still. And why not? If you are designing a unikernel today, it would
still make sense to use PV 32bit or PVH.

[Juergen Gross]

Post by Florian Heigl

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

Hi Juergen,
Although I can see that deprecating 32-bit PV guest support is
desirable, and it might not cause any problems to Linux and
BSDs, we need to be careful about unikernels.
There are probably unikernels out there that only support PV 32bit
still. And why not? If you are designing a unikernel today, it would
still make sense to use PV 32bit or PVH.

PVH will still work, of course. 32- and 64-bit.


Juergen

[Juergen Gross]

Post by Juergen Gross
In the Xen x86 community call we have been discussing whether anyone
really is depending on 32-bit PV guests. We'd like to evaluate whether
- deprecating 32-bit PV guest support in Xen, meaning that we'd
eventually switch to support 32-bit PV guests only via PV-shim from
Xen 4.12 or 4.13
- dropping 32-bit PV support from upstream Linux kernel, resulting in
current 32-bit PV guests no longer being able to upgrade to the newest
kernel version any longer
- is there any Linux distribution still shipping 32-bit PV-capable
systems?
- what about BSD? Is 32-bit PV support important there?

Hello,
Actually, NetBSD only supports PV (there is work toward PV drivers for HVM,
and PVH, but this is making slow progress). 32-bit PV is faster than 64-bit PV
so all my domUs are 32bits these days.

Thanks for the feedback.


Juergen