Home
Solaris and Xen

My experience
Last Update: 24.07.2007 18:00



Please note that as of 24.07.2007 I stopped updating this page.
My experience with Solaris Xen 07/07 and newer is now documented here.


Solaris and Xen

  1. Solaris and Xen
    1. Introduction
    2. Environment used
    3. Xen on Solaris Details
      1. DomU Images tested
    4. Missing Features
      1. Hiding devices from Dom0
      2. Full Virtualization Support
      3. Dynamic Hardware changes for DomU
      4. xm reboot / xm shutdown
      5. prtdiag does not work
      6. ACPI is not (fully) implemented
    5. Bugs
      1. USB driver scsa2usb in the Xen bfu image is outdated
      2. The bge driver in the Xen bfu image is outdated
      3. Invalid GDM login service
      4. Shutdown not working reliable after installing Xen
      5. keyboard not working reliable after installing Xen
      6. flarcreate issue: do not use -S
      7. DNS Name resolution seems not to work in a Solaris DomU
      8. format does not work inside a Solaris DomU
      9. How to get rid of a zombie domU
    6. Trouble Shooting
      1. General
      2. How to change the output of uname
      3. NameError: name 'FutureWarning' is not defined
      4. Error: [Errno 2] No such file or directory: '/proc/xen/balloon'
      5. error: (11, 'Resource temporarily unavailable')
    7. HowTos
      1. Solaris Dom0 Howtos
        1. How to install Xen on Solaris
        2. What SMF services are used in the Dom0 for Xen?
        3. What Solaris Installation cluster is necessary to install Xen?
        4. Can I start a 32 Bit DomU in a 64 Bit Dom0 or vice versa?
        5. Where is the config file for Xen?
        6. Where are the logfiles from Xen?
        7. The Xen Webserver (localhost:8000) only shows two links to himself
        8. How to view information about the installed Xen version
        9. How to view the Xen kernel messages
        10. How to convert a DomID to a DomName and vice versa
        11. How to pause/unpause a DomU
        12. How to save and restore a DomU
        13. Is there a limit for file based disk images like in Linux Dom0s?
        14. How to check if a we're running in a Xen Domain (either Dom0 or DomU)
      2. General DomU HowTos
        1. How to get a graphical login for a DomU
        2. How to increase the memory of a running DomU
        3. How to attach another network adapter to a running Solaris DomU
        4. How to attach a disk image or disk device in read/write mode
        5. How to list the settings of a running DomU
        6. How to start a DomU from a logical partition
        7. How to create a disk image file for a DomU
        8. How to copy a DomU image file created as sparse file
      3. Solaris DomU HowTos
        1. How to create a Solaris DomU
        2. How to format a disk image
        3. How to install VNC Server in a Solaris DomU
        4. How to mount a Solaris DomU disk image in the Dom0
        5. How to attach a disk image file to a running Solaris DomU
        6. How to attach an ISO image file to running Solaris DomU
        7. How to attach the physical CDROM drive to a running Solaris DomU
        8. How to resize a Solaris DomU disk image
        9. How to reboot a Solaris DomU
      4. Linux DomU HowTos
        1. How to create sample Linux DomUs without installing Linux
        2. Which Linux Kernel is necessary for Xen 3.0 Domains?
        3. How to get the Network working in a Linux DomU
        4. How to mount a Linux DomU disk image in the Solaris Dom0
        5. How to attach a disk image file to a running Linux DomU
        6. How to attach an ISO image file to a running Linux DomU
        7. How to attach the physical CDROM drive to a running Linux DomU
        8. How to resize a Linux DomU disk file image
    8. Files
    9. Links
      1. General
      2. Documentation
      3. Xen and OpenSolaris
      4. Xen DomU Images and utils
      5. German Links
    10. Credits
    11. History


Introduction


On this page I document my experience with Xen on Solaris 10 x86.

Due to the nature of this issue - Xen support for Solaris is just starting - this web site is far from being complete and always more or less work in progress. If you find a bug or know information not mentioned here please let me know - either via email or through the little mail interface on this website.

I'll continue to update this web site and record the changes in the history section below.

Frankfurt, 02.08.2006    Bernd Schemmer


Update 20.11.2006 (for German readers):

Wer sich ernsthaft mit Xen beschaeftigen will, sollte sich mal das Buch "Xen3" [20.11.2006] von Andrej Radonic und Frank Meyer anschauen (Franzis Verlag, ISBN 9 783772 378997)

Das Buch bietet eine ausfuehrliche Beschreibung von Xen inklusive Xen unter OpenSolaris und ist sehr zu empfehlen.

(Weitere Links mit Informationen ueber Xen in Deutsch sind weiter unten aufgelistet)


Update 19.07.2007

A new drop of Xen is out - this time Xen is distributed as full Solaris Installation Image see the download links below. But an BFU image for upgrading an existing Solaris installation is also available on the download page.

Xen 07/2007 is based on build 66.

A list of notable enhancements in this drop is here: http://blogs.sun.com/levon/entry/solaris_xen_update
The docs for the new drop are here: http://opensolaris.org/os/community/xen/docs/

My own experiences with this Xen Release are documented below.



Some notes about the notation used on this site


Every paragraph is marked with a stamp like this: [02.08.06 snv_43 Xen_07_2006]

This means the information was added on 02.08.2006 while I was using Xen Release 07/2006 on top of Solaris Nevada build 43.


Every link is marked with a stamp like this: [19.01.06]

The date is the last date that I checked the website referenced by the link.


Infos from other sources (like Blogs or Mailing lists) are marked with a stamp like this: (Source: Casper Dik, 14.01.06 [19.01.06])

This means the entry was made by Casper Dik on 14.01.06 and the last time I checked this link was 19.01.2006.


Text that I borrowed from other sources is printed with grey background like this

This text was taken from another source




back to top

Environment used

My environment is as follows:

I tested Xen and Solaris on an Acer Ferrari 4005. My general experience with Solaris on the Acer Ferrari 4005 is documented here.


[02.08.06 snv_43 Xen 07_2006]

I installed the "July 2006 release" of Xen (packages and bfu) on top of Solaris snv_43


[14.09.06 snv_46 Xen 08_2006]


All further tests are done with the "August 2006 release" of Xen (packages and bfu) on top of Solaris snv_46.


[21.07.07 Xen 07_2007]

All tests are now done with the 07/07 Release of Xen




Xen on Solaris Details


[24.07.06 snv_43 Xen_07_2006]

I started playing with Xen and Solaris today. First tests with the Xen Demo Disk (see Xen link list below) were succesfull - Xen does work in general on the Acer Ferrari 4005. Next step is to install Solaris from scratch and add the Xen software.


[25.07.06 snv_43 Xen_07_2006]

I installed Xen according to the Installation Instructions on the Open Solaris web page (see Xen link list below) - works: I now have a running Xen Dom0 with Solaris!

While installing Xen I created a little script to execute the necessary tasks: install_xen.sh.

I installed Xen on my Ferrari 4005 running a fresh installed Solaris Nevada snv_43 using the instructions from this page
http://www.opensolaris.org/os/community/xen/How-To-07-2006/install/ [25.07.06].

The installation worked without errors (a reconfigure reboot was necessary to get it working)

After that I installed the Linux Images from the Xen Demo CDROM using the instructions from this page
http://www.dme.org/log/2006/07/17/solaris-on-xen-a-linux-domu-without-installing-linux/ [25.07.06].

Results until now:

[26.07.06 snv_43 Xen 07_2006]

Just learned that Xen for Solaris does not yet support the assigning of PCI devices to other domains than dom0:

There have been two different approaches to this.  In Xen 2.x it was  possible to tell dom0 to ignore some PCI devices and make them  visible to a domU.  This was never supported in the 'Solaris on Xen  2.x' port (which was released much earlier this year).  In Xen 3.x it  seems that a different approach has been adopted, where it's  necessary to have PCI front and backend drivers to mediate the  availability of PCI devices in domU.  Solaris doesn't currently have  that support (in either dom0 or domU).

USB devices are managed in a similar way and are similarly not yet  available.

This is on our list of things to do, but the code is relatively new  in Xen (well, more accurately, in the Linux port to Xen) and we  haven't got there yet.
(Source: David Edmondson,  26.07.06 in an email)

 
Also Xen for Solaris 64 Bit is less stable then Xen for Solaris 32 Bit and running a 32 Bit OS in a 64 Bit domain does not work at all.

So I booted Xen for Solaris 32 bit and started a Linux Domain with the Linux images from the Xen Demo disk - and that worked out of the box!


[27.07.06 snv_43 Xen 07_2006]

Sample domain configuration for the Centos image from the Xen Demo Disk [download here]:

kernel = "/export/xen-images/centos1/vmlinuz-2.6.16.13-xen"
memory = 128
name = "xen-centos1"
disk = ['file:/export/xen-images/centos1/rootfs.img,sda1,w']
vif = [ 'mac=00:16:3E:00:00:15' ]
root = "/dev/sda1 ro"


Note: The MAC address is optional but I want the Domain to use always the same IP address via DHCP.

Sample script to start the Centos domain (console & vncviewer) [download here]:

#!/usr/bin/ksh

DOMAIN_NAME="centos1"
DOMAIN_IP="192.168.178.115"

echo "Starting the Xen Domain \"${DOMAIN_NAME}\" ..."
gnome-terminal --title "Xen - ${DOMAIN_NAME} console" --command "xm create -c  /export/xen-images/${DOMAIN_NAME}.phy"

echo "Waiting for the domain to come up ..."
while [ 0 = 0 ] ; do
  echo ".\c"
  ping ${DOMAIN_IP}
  [ $? -eq 0 ] && break
  sleep 1
done
sleep 5
echo ""
echo "Starting the VNC viewer for \"${DOMAIN_IP}\" ..." 
/opt/sfw/bin/vncviewer ${DOMAIN_IP}:5900 &



[28.07.06 snv_43 Xen 07_2006]

Got the Debian Image from the Xentos Demo CD working now - I copied the missing ethtool from the Centos Image.

Note that Linux distributions based on the 2.4 kernel do not work in a Xen 3.0 domU right now:

2.4 kernels only work in either Xen 2.x or fully virtualized, as there's no patches to the 2.4 kernel. It would be possible to create 2.4 kernel patches, but I think it's very low on the priority list...
> 
> I have a customized harddisk image of DSL that I would like
> to run in a Xen DomU.
>
> Can this be done without recompiling the kernel and/or libs?

In full virtualization (HVM = VT/SVM capable hardware) you can run without any recompilation or other changes to the system. Only Xen 3.x support full virtualization, and of course, you do need recent hardware to get the necessary processor features...

(Source: Mats Petersson on the Xen User Mailing list,  27.07.06 [27.07.06])


[01.08.06 snv_43 Xen 07_2006]

Created my first Xen DomU running Solaris (32 Bit) today using the instructions from this page
Creating Solaris domU virtual disk images at  Xen at OpenSolaris.org [09.09.06] and it works out of the box!!

That's cool again!

If you're going to do the same (and I encourage you to do so!) be aware that it takes a long time to create the DomU.

After I got the Solaris DomU running I installed Xvnc using the instructions from this page: Xvnc with Solaris on Xen [03.08.06] .

Update: 03.08.2006:

The bugs listed here in the past for the page Xvnc with Solaris on Xen [03.08.06] are now corrected.


[02.08.06 snv_43 Xen 07_2006]

The smallest dom0 I've built had the "Core" cluster and then the  following packages:

        SUNWtoo SUNWpoolr SUNWpool SUNWadmc SUNWadmfr SUNWadmfw
        SUNWlur SUNWluu SUNWluzone SUNWzoner SUNWzoneu SUNWinst
        SUNWPython

With the addition of the Xen related packages from this site, this is enough to allow you to create a file-backed domU using vbdcfg.
(Source: David Edmondson,  02.08.06 in the Open Solaris Xen Forum)

Using the configuration above I was almost able to create a minimal Solaris DomU.  Almost because I had to install two additional packages before I could install the bfu:

bash-3.00# pkginfo | grep bind
system      SUNWbind                     BIND DNS Name server and tools
system      SUNWbindr                    BIND Name server Manifest



Update 03.08.2006:

The typo on the page Creating Solaris domU virtual disk images at Xen at OpenSolaris.org  [03.08.06] mentioned here in the past is corrected.


[08.08.06 snv_43 Xen 07_2006]

Succesfully booted Solaris 64 Bit DomUs on Solaris 64 Bit Dom0 today!

Unfortunatley I did not find any other 64 Bit DomU image until now.


[10.08.06 snv_43 Xen 07_2006]

Succesfully booted a FreeBSD 7.0 DomU. Thanks to Mark Johnson for the link to the FreeBSD image, the config file for that DomU, and the other information.

Mark's list of successfully running DomUs is:

  32-centos.cfg
  32-freebsd.cfg
  32-slackware.cfg
  32-debian.cfg
  32-gentoo.cfg
  32-solaris.py
  32-fedora.cfg
  32-opensuse.cfg
  32-ttylinux.cfg
  64-solaris.py


[31.08.06 snv_43 Xen 07_2006]

A new release of the Xen for Solaris bfu is out, see the Release Notes [31.08.06]  (not tested yet).


[09.09.06 snv_46 Xen 08_2006]

Reinstalled Solaris (snv_46) and Xen (08/2006) successfully.

Succesfully booted my local Ubuntu 6.06.1LTS installation on hda5 in a Xen DomU running under Solaris (see below for the config file)!


[13.09.06 snv_46 Xen 08_2006]

Succesfully created and started a Xen DomU with Ubuntu 6.06.1LTS using the instructions from this page [13.09.06].


[26.10.06 snv_46 Xen 08_2006]

Looks like the Xen people updated the Xen Demo Disk with a Xen kernel not working in DomU on a Solaris Dom0 (see here)


[19.07.07 snv_66 Xen 07_2007]

A new drop of Xen is out - this time Xen is distributed as full Solaris Installation Image see the download links below. Not tested yet but will do that as soon as possible.


[21.07.07 Xen 07_2007]

I successfully reinstalled my Ferrari yesterday from the DVD with Solaris 10 and Xen 07/07.

The installation worked without problems. In the default configuration Solaris will boot without Xen so you should change the default entry in the file /boot/grub/menu.lst manually if required (see also this document [21.07.07])

And please note that even if you boot the Xen enabled kernel the Xen SMF services are still disabled and you must enable them manually via svcadm (but only once; see also this blog entry [21.07.07])

After all Xen services are enabled they should look like this:

dominion# svcs -a | grep xctl
online 10:51:04 svc:/system/xctl/store:default
online 10:51:11 svc:/system/xctl/xend:default
online 10:51:11 svc:/system/xctl/console:default
online 10:51:16 svc:/system/xctl/domains:default


On my Ferrari with a 64 Bit AMD CPU Solaris boots into 64 Bit mode in the default configuration (with Xen enabled and without) but due to the lack of Xen DomUs for 64 Bit I changed this to 32 Bit with this entry:

title Solaris dom0 32Bit
kernel /boot/xen.gz
module /platform/i86xpv/kernel/unix /platform/i86xpv/kernel/unix
module /platform/i86pc/boot_archive


After rebooting the Ferrari with Xen in 32 Bit mode I tried to start my old Linux and Solaris DomUs. Unfortunately these DomUs all use non-PAE kernels and Xen 07/07 seems not to like them. The error message from Xen was very clear here:

bash-3.00# xm create -c solaris-b44.phy
Using config file "./solaris-b44.phy".
Error: (2, 'Invalid kernel', 'Non PAE-kernel on PAE host.')


I'm not sure if this a non-correctable situation so I started a thread on OpenSolaris [21.07.07] regarding this issue:

>If I understand this correct, I can not run the older DomUs with Solaris 
> Xen 07/07 in Dom0?

As shipped, no. But what you could do (though I haven't tested this), is
downlaod our xen-gate sources and modify sunos.hg/bin/build-xen such that PAE=n
around line 199. If you install the resultant packages from running
build-workspace, I /think/ the dom0 kernel should boot under non-PAE just fine
and you can run the older domUs as well as the newer ones.
(Source: John Levon,  21.07.07 in the Open Solaris Xen Forum)



My next step was to create a new Solaris 10 DomU using the instructions from the Xen Webpage on opensolaris.org [21.07.07]. The installation of the Solaris DomU worked flawless after I used an ISO image instead of the real DVD as installation source.

The config file for the Solaris 10 DomU after the installation now looks like

name = "solaris_10_July_07"
memory = "368"
vcpus = "1"
disk = [ 'file:/export/xen-images/Solaris10_xen0707.raw,0,w' ]
root = "/dev/dsk/c0d0s0"
vif = [ 'mac=00:0e:0c:3e:18:cd' ]
on_shutdown = 'destroy'
on_reboot = 'restart'
on_crash = 'destroy'


and the DomU works perfect:

bash-3.00# xm console 16
SunOS Release 5.11 Version xen-nv66-2007-06-24 32-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
ip: joining multicasts failed (4) on xnf0 - will use link layer broadcasts for multicast
Hostname: dhcppc1
syslogd: line 45: WARNING: loghost could not be resolved

dhcppc1 console login: root
Password:
Last login: Sat Jul 21 14:56:52 on console
Jul 21 15:05:56 dhcppc1 login: ROOT LOGIN /dev/console
Sun Microsystems Inc.   SunOS 5.11      xen-nv66-2007-06-24     October 2007
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
xnf0: flags=201005843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,DHCP,IPv4,CoS> mtu 1500 index 2
        inet 192.168.1.21 netmask ffffff00 broadcast 192.168.1.255
        ether 0:e:c:3e:18:cd
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128
#


I don't know if I have to think about the error message regarding multicast so I ignore it for now

Update [22.07.07]

This is a known bug that can be ignored, see this bug [22.07.07].


[22.07.07 Xen 07_2007]



Today I successfully started a Solaris DomU in a Linux Dom0. The Linux Dom0 was running Xen 3.0.4 under Ubuntu Feisty Fawn 7.04:

xtrnaw7@t30:~$ uname -a
Linux t30 2.6.19-4-server #2 SMP Thu Apr 5 06:07:34 UTC 2007 i686 GNU/Linux

root@t30:~# xm info
host                   : t30
release                : 2.6.19-4-server
version                : #2 SMP Thu Apr 5 06:07:34 UTC 2007
machine                : i686
nr_cpus                : 1
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 1
threads_per_core       : 1
cpu_mhz                : 1794
hw_caps                : 3febf9ff:00000000:00000000:00000080
total_memory           : 1022
free_memory            : 63
xen_major              : 3
xen_minor              : 0
xen_extra              : .3-0
xen_caps               : xen-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
cc_compile_by          : buildd
cc_compile_domain      : buildd
cc_compile_date        : Sat Mar 24 16:01:08 UTC 2007
xend_config_format     : 2



First I started the Solaris DomU created on my Solaris Dom0. Then I installed a new Solaris DomU under Linux using the Solaris DVD image. Both methods worked and the result looks like this:

root@t30:/data/xen# xm create -c Solaris10_xen0707.phy
Using config file "Solaris10_xen0707.phy".
Started domain solaris_10_July_07
SunOS Release 5.11 Version xen-nv66-2007-06-24 32-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
WARNING: Found xen v3.0.3-0 but need xen v3.0.4-1-sun
WARNING: The kernel may not function correctly
ip: joining multicasts failed (4) on xnf0 - will use link layer broadcasts for multicast
Hostname: dhcppc1

dhcppc1 console login: syslogd: line 45: WARNING: loghost could not be resolved

dhcppc1 console login: root
Password:
Jul 22 06:38:17 dhcppc1 login: ROOT LOGIN /dev/console
Last login: Sat Jul 21 22:07:14 on console
Sun Microsystems Inc.   SunOS 5.11      xen-nv66-2007-06-24     October 2007
#




back to top

DomU Images tested


Infos about using Linux Images and Solaris Images in a DomU in detail [Last update: 09.09.06 /bs]:


[21.07.07 Xen 07_2007]

All Images below use non-PAE kernels. It looks like non-PAE kernels are not supported for DomUs  in the  07/07 Drop of Xen for Solaris.


Image
32/64 Bit?
Status
Date
Centos image from the Xen Demo Disk
32 Bit
- boots
- gets an IP address from the DHCP server
- ping from the Domain to the outside world works
- TCP and UDP connections work after issuing
ethtool -K eth0 tx off
- VNC connection to the Domain works

Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[02.08.06 snv_43 Xen 07_2006]
OpenSUSE image from the Xen Demo Disk
32 Bit
- boots
- does not get an IP address automatically but the IP address can be configured manually
- ping from the Domain to the outside world works
- TCP and UDP connections work after issuing
ethtool -K eth0 tx off
- VNC connection to the Domain works

Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[02.08.06 snv_43 Xen 07_2006]
Debian image from the Xen Demo Disk
32 Bit
- boots
- gets an IP address from the DHCP server
- ping from the Domain to the outside world works
- network connections work after copying the ethtool binary from the Centos image and issuing

ethtool -K eth0 tx off

- VNC connection to the Domain works

Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[02.08.06 snv_43 Xen 07_2006]
Solaris 10 snv43 DomU (SUNWXCall with Xen 07/2006 bfu)
32 Bit
- works
- VNC connections to the Domain work after installing Xvnc Server in the domain

Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[03.08.06 snv_43 Xen 07_2006]
Solaris 10 snv43 DomU (SUNWcore with add. packages, [see here] and 07/2006 Xen bfu)
32 Bit
- works
- gets an IP address from the DHCP server

Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[03.08.06 snv_43 Xen 07_2006]
Solaris 10 snv43 DomU (SUNWXCall with 07/2006 Xen bfu) 64 Bit
- works
- gets an IP address from the DHCP server

[08.08.06 snv_43 Xen 07_2006]
Solaris 10 snv43 DomU (SUNWcore with add. packages, [see here] and 07/2006 Xen bfu) 64 Bit
- works
- gets an IP address from the DHCP server
[08.08.06 snv_43 Xen 07_2006]
gentoo.2006 image from jailtime.org [07.08.2006]
32 Bit
- works
- gets an IP address from the DHCP server
- network connections work after copying the ethtool binary from the Centos image and issuing

ethtool -K eth0 tx off

Notes:

Use the kernel from the Xen Demo Disk to boot this image

AFAIK there's no X or VNC server installed in this image
Sample config file for this image:

kernel = "/export/xen-images/centos1/vmlinuz-2.6.16.13-xen"
memory = 128
name = "gentoo-2006"
disk = ['file:/export/xen-images/gentoo.2006/gentoo.2006-0.img,sda1,w',
'file:/export/xen-images/gentoo.2006/gentoo.swap,sda2,w']
vif = [ 'mac=00:16:3E:00:00:20' ]
root = "/dev/sda1 ro"
extra = "gentoo=nodevfs"


Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[07.08.06 snv_43 Xen 07_2006]
slackware 10.2 from jailtime.org [07.08.2006]
32 Bit
- works
- gets an IP address from the DHCP server
- network connections work after issuing

ethtool -K eth0 tx off

Notes:

Use the kernel from the Xen Demo Disk to boot this image

AFAIK there's no X or VNC server installed in this image
Sample config file for this image:

kernel = "/export/xen-images/centos1/vmlinuz-2.6.16.13-xen"
memory = 128
name = "slackware"
disk = ['file:/export/xen-images/slackware/slackware.10-2.img,sda1,w',
'file:/export/xen-images/slackware/slackware.swap,sda2,w']
vif = [ 'mac=00:16:3E:00:00:21' ]
root = "/dev/sda1 ro"

Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
07.08.06 snv_43 Xen 07_2006]
FreeBSD 7.0 from FreeBSD Xen Image  [10.08.06]
32 Bit
- works
- does not get an IP address from the DHCP server but you can manually configure an IP address (I think DHCP is not configured in the image; but I'm no FreeBSD guru)
- network connections work without disabling checksuming
- looks like there's no VNC server installed in the image

Sample config file for the FreeBSD 7.0 DomU [Source: Mark Johnson]:

kernel = "/export/xc/xvm/kernels/32-freebsd-7.0"
memory = 256
name = "freebsd"
vif = [ 'bridge=xenbr0' ]
dhcp = "dhcp"
disk = ['file:/export/xc/xvm/disks/32-freebsd-7.0,hda1,w']
extra = "boot_verbose"
extra += ",boot_single"
extra += ",vfs.root.mountfrom=ufs:/dev/xbd769a"
extra += ",kern.hz=100"


Works also with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[10.08.06 snv_43 Xen 07_2006]
Solaris 10 snv_44 DomU from opensolaris.org
32 Bit
Works, but issues a warning:

WARNING: Found xen v3.0.2-sun but need xen v3.0.2-3-sun
WARNING: The kernel may not function correctly

After rebooting the DomU the image file was corrupted; therefor I suggest you should not use this image with Xen 07/2006!

Works with the Xen Release 08/2006 [09.09.06 snv_46 Xen 08_2006]
[09.09.06 snv_43 Xen 07_2006]
Ubuntu 6.06.1 LTS 32 Bit
Succesfully booted my local Ubuntu 6.06.1LTS installation on hda5 in a Xen DomU running under Solaris
[09.09.06 snv_46 Xen 08_2006]
Ubuntu 6.06.1LTS DomU Image created with the instructions from this site
[13.09.2006]
32 Bit
- I had to use the Xen kernel from the Xen Demo Disk (vmlinuz-2.6.16.13-xen) to get this to work; the Xen kernel from Ubuntu did not work

- network connections work after issuing

ethtool -K eth0 tx off

[13.09.06 snv_46 Xen 08_2006]




back to top

Missing Features

This section conatins a (surely not complete) list of features not implemented in the current release of Xen for Solaris

see also the Release Notes at Xen at OpenSolaris.org [09.09.06]


Hiding devices from Dom0


[08.08.06 snv_43 Xen 07_2006]

This features is not yet implemented in the current release of Xen for Solaris. The necessary pci.backend and pci.frontend drivers  are supposed to be in one of the next releases of Xen for Solaris


Full Virtualization Support


[08.08.06 snv_43 Xen 07_2006]

Full Virtualization Support is currently not implemented in Xen for Solaris

[19.07.07 Xen 07_2007]

Xen for Solaris drop 07/2007 now has Full Virtualization Support for the approbiate CPUs. To check if your CPU has the necessary capabilities use the C program posted in this blog entry:

http://blogs.sun.com/cwb/entry/starting_out_with_solaris_on [21.07.07]


Dynamic Hardware changes for DomU


[08.08.06 snv_43 Xen 07_2006]

Dynamic Hardware changes for DomUs are not fully implemented yet


xm reboot / xm shutdown


[08.08.06 snv_43 Xen 07_2006]

"xm reboot" and "xm shutdown" do not work for Solaris DomUs yet, see here

[21.07.07 Xen 07_2007]
 
"xm reboot" and "xm shutdown" do work for Solaris DomU if running Xen 07/07 in the Dom0.


prtdiag does not work

[08.08.06 snv_43 Xen 07_2006]

prtdiag is not supported in the current version of Xen for Solaris - neither in the Dom0 nor in a DomU

[21.07.07 Xen 07_2007]

prtdiag does work in the Dom0 of Xen 07/07 (but not in a Solaris DomU)


ACPI is not (fully) implemented

[09.08.06 snv_43 Xen 07_2006]

Because the ACPI support is not yet implemented in Xen powernowd does not work

(bash):root@ferrari:/export/xen-images # cat /etc/svc/volatile/site-powernow:default.log
[ Aug  9 18:12:35 Enabled. ]
[ Aug  9 18:12:35 Executing start method ("/usr/sbin/powernowd") ]
/dev/powernow: No such file or directory
[ Aug  9 18:12:35 Method "start" exited with status 95 ]
(bash):root@ferrari:/export/xen-images # /usr/sbin/powernowd
/dev/powernow: No such file or directory

(bash):root@ferrari:/export/xen-images # ls -l /dev/powernow
lrwxrwxrwx   1 root     root          37 Aug  9 01:24 /dev/powernow ->
../devices/pseudo/powernow@0:powernow

(bash):root@ferrari:/export/xen-images # ls -l /devices/pseudo/power\@0
power@0/              power@0:power_button

(bash):root@ferrari:/export/xen-images # ls -l /devices/pseudo/powernow*
/devices/pseudo/powernow*: No such file or directory




Bernd Schemmer: >>looks like the powernowd does not run under Xen:
...
Bernd Schemmer: >> Is this correct?

Casper Dik: >>Possibly; powernow needs to have access to ACPI and all the machine
Casper Dik: >>specific registers controlling powernow.

Right.  There has been some discussion in the xen community on how to make this happen.  I haven't followed the details of these conversations, but I don't expect a quick resolution.

In addition, if you try frkit on our version of i86pc (i.e. a hardware boot, not under Xen), a couple of the modules will fail to install.  We have moved a lot of the kernel modules to new locations, and we have yet to update frkit to look in both places.  For now, you can work around these problems by running:

# ln -s /platform/i86pc/kernel/mach/pcplusmp /kernel/mach/
# ln -s /platform/i86pc/kernel/mach/amd64/pcplusmp /kernel/mach/amd64/
(Source: Ryan Scott in the Xen Discussion on opensolaris.org, 09.08.06)



back to top

Bugs


This section list some bugs and some things that might look like a bug (e.g. the outdated driver in the Xen bfu are surely not a bug at all)

see also Known Issues (bugs) with Xen on Solaris [26.07.06]


USB driver scsa2usb in the Xen bfu image is outdated


[01.08.06 snv_43 Xen 07_2006]

After installing the Xen bfu the USB 2.0 Memorystick again does not work (see this thread on the Laptop Mailing list [02.08.06])
To avoid this, backup the following files before installing the bfu and restore the files again after installing the bfu:

/kernel/drv/amd64/scsa2usb
/kernel/drv/scsa2usb.conf          
/kernel/drv/scsa2usb


[09.09.06 snv_46 Xen 08_2006]

Fixed in the Xen 08/2006 release


The bge driver in the Xen bfu image is outdated


[27.07.06 snv_43 Xen 07_2006]

The bge driver in the bfu archive is out-dated and does not work reliable on the Ferrari. Therefor you must save the existing bge driver before installing the bfu and restore the original files after installing the bfu. The files to backup are:

/kernel/drv/amd64/bge
/kernel/drv/bge     
/kernel/drv/bge.conf


[09.09.06 snv_46 Xen 08_2006]

Fixed in the Xen 08/2006 release



Invalid GDM login service

[09.08.06 snv_43 Xen 07_2006]

The bfu reinstalls the old GDM2 login SMF service gdm2-login; this service was replaced with the SMF service graphical-login in newer Solaris Nevada releases.

Use the following code to remove this service:

echo "Checking the GDM login service ..."

# gdm login frmi
#
INVALID_GDM2_FRMI="svc:/application/gdm2-login:default"
NEW_GDM2_FRMI="svc:/application/graphical-login/gdm:default"

if [ "$( svcs -o STATE -H ${NEW_GDM2_FRMI} )"x != ""x ] ; then
  echo "SMF service \"${NEW_GDM2_FRMI}\" found."
  if [ "$( svcs -o STATE -H ${INVALID_GDM2_FRMI} )"x != ""x ] ; then
    echo "Removing the invalid SMF service \"${INVALID_GDM2_FRMI}\" ..."
    svcadm disable -s "${INVALID_GDM2_FRMI}"
    svccfg delete "${INVALID_GDM2_FRMI}"
  else
    echo "Invalid SMF service \"${INVALID_GDM2_FRMI}\" not found"
  fi
else
  echo "SMF service \"${NEW_GDM2_FRMI}\" not found."
fi


Note: Looks like the bfu does not contain a manifest file for the SMF service gdm2-login.


[09.09.06 snv_46 Xen 08_2006]

Fixed in the Xen 08/2006 release



Shutdown not working reliable after installing Xen

[01.08.06 snv_43 Xen 07_2006]

A shutdown does not work reliable anymore after installing the Xen bfu on Solaris snv_43. Solaris hangs while shuting down most of the time and I must press the power button to turn the Ferrari off. I'm still investigating this bug


[09.09.06 snv_46 Xen 08_2006]

Seems to be fixed in the Xen 08/2006 release



keyboard not working reliable after installing Xen


[27.07.06 snv_43 Xen 07_2006]

If the keyboard does not work anymore after booting, telnet to the machine and apply the workaround from the bug http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6447269 [26.07.06]


[09.09.06 snv_46 Xen 08_2006]

Seems to be fixed in the Xen 08/2006 release


flarcreate issue: do not use -S

[02.08.06 snv_43 Xen 07_2006]

Do not use the  -S of flarcreate if you want to create a DomU Image from the flar file according to the instructions on this page:
Creating Solaris domU virtual disk images at Xen at OpenSolaris.org [02.08.06]


If the flar file is created with the -S switch you might get an error message like:

ERROR: Unable to process archive file /export/Sol10_snv43_core_xen.flar.
vbdcfg: Warning: Failed to extract archive /export/Sol10_snv43_core_xen.flar into /export/xc/proto/filep roto/vmnt


This is a known bug.

[21.07.07 Xen 07_2007]

In the Xen Release 07/07 flarcreate is not neccessary anymore to create a DomU


DNS Name resolution seems not to work in a Solaris DomU


[01.08.06 snv_43 Xen 07_2006]

I found a problem with the DNS resolution in a Solaris 10 DomU.

The Test Environment was:

I installed the Xen bfu on top of Solaris snv43 and booted the Dom0 with Solaris. Then I installed and booted a Solaris 10 DomU.

In the Solaris 10 Dom0 the DNS resolution works; in the Solaris 10 DomU the DNS resolution does not work as expected:

Inside the Solaris 10 DomU I can't use hostnames resolved by the DNS, e.g.

bash-3.00# ping www.heise.de
ping: unknown host www.heise.de


see this thread on opensolaris.org



[03.08.06 snv_43 Xen 07_2006]


Resolved:

bash-3.00# getent hosts www.heise.de  
193.99.144.85   www.heise.de    
     
bash-3.00# ping www.heise.de          
www.heise.de is alive                 

but I don't know why. Only rebooted the Dom0 ...


format does not work inside a Solaris DomU


[03.08.06 snv_43 Xen 07_2006]


Calling format in a Solaris 10 DomU created with vbdcfg crashes the domain:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c0d0 <drive type unknown>
/xendev/xvbd@0
Specify disk (enter its number): p
`p' is not an integer.
Specify disk (enter its number): 0

0

/dev/dsk/c0d0s0 is currently mounted on /. Please see umount(1M).


AVAILABLE DRIVE TYPES:
0. other
Specify disk type (enter its number): 0
Enter number of data cylinders: WARNING: init(1M) exited on fatal signal
10: restarting automatically
WARNING: exec(/sbin/init) failed with errno 5.
WARNING: failed to restart init(1M) (err=5): system reboot required


see this thread on opensolaris.org

The error with format is a known missing feature but the crash not


[04.08.06 snv_43 Xen 07_2006]

format does not work with virtual block devices yet; just treat it as a single partition/slice.

(Source: David Edmondson,  04.08.06 in the Open Solaris Xen Forum [04.08.06].



[14.09.06 snv_46 xen 08_2006]

format does now work in a running Solaris DomU (see How to format a disk image)





How to get rid of a zombie domU


[09.08.06 snv_43 Xen 07_2006]

The only way to get rid of  a zombie domU is to reboot the Dom0.

Use xm list to check for a zombie domU:

(bash):root@ferrari:/export/xen-images # xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      512     1 r-----   577.2
Zombie-xen-debian                 10      128     1 --p--d     0.0


see this thread on opensolaris.og [09.08.06]




back to top

Trouble Shooting


General

[08.08.06 snv_43 Xen 07_2006]

In case of an error always check the Xen logfiles.


How to change the output of uname


Some scripts and binaries do not know the output of uname if a Xen kernel is booted. In this case you can use sloggi [09.09.06] to fake the uname output:

(bash):root@ferrari:/root # uname -a
SunOS ferrari 5.11 matrix-aug i86xen i386 i86xen


(bash):root@ferrari:/root # slogctl -h
Usage: slogctl [-hV] [-s sysname] [-r release] [-v version] [-m machine]
Options:
   -s sysname   Set the name of the operating system.
                [Default: SunOS]
   -r release   Set the operating system release level.
                [Usually: 5.8, 5.9, ...]
   -v version   Set the the operating system version.
                [Default: Generic]
   -m machine   Set the machine hardware name (class).
                [Usually: sun4m, sun4u, i86pc, ...]
   -h           Print this help screen.
   -V           Print version information.
No options restore the original uname() setup

(bash):root@ferrari:/root # slogctl -r 5.10 -v Generic -m i386
SunOS ferrari 5.10 Generic i386

(bash):root@ferrari:/root #


A compiled binary for Solaris 10 is available on my homepage. in the download section.


NameError: name 'FutureWarning' is not defined

[09.08.06 snv_43 Xen 07_2006]

(bash):root@ferrari:/export/xen-images # xm create -c centost1.phy
Traceback (most recent call last):
  File "/usr/sbin/xm", line 8, in ?
    from xen.xm import main
  File "/usr/lib/python/xen/xm/main.py", line 31, in ?
    warnings.filterwarnings('ignore', category=FutureWarning)
NameError: name 'FutureWarning' is not defined



Solution: You installed an invalid Python version after installing the Xen bfu

Use pkginfo to check which Python version is installed and remove the invalid version:

(bash):root@ferrari:/export/xen-images # pkginfo | grep -i py
system      SFWpy                            Python - an interpreted, interactive, OO prog. language
GNOME2      SUNWPython                       The Python interpreter, libraries and utilities
GNOME2      SUNWPython-devel                 The Python interpreter, libraries and utilities - development files
GNOME2      SUNWPython-extra                 Supplemental Python libraries and utilities
GNOME2      SUNWPython64                     The 64-bit Python interpreter, libraries and utilities for Xen
GNOME2      SUNWPython64-devel               The 64-bit Python interpreter, libraries and utilities for Xen - development files
GNOME2      SUNWgnome-python-desktop         Python support desktop libraries for GNOME
GNOME2      SUNWgnome-python-libs            Python support libraries for GNOME
GNOME2      SUNWgnome-python-libs-devel      Python support libraries for GNOME - development files
system      SUNWlxml-python                  The XML library - Python bindings
system      SUNWlxsl-python                  The XSLT library - Python bindings



Error: [Errno 2] No such file or directory: '/proc/xen/balloon'

[08.08.06 snv_43 Xen 07_2006]

bash-3.00# xm create -c ./mydomu1.phy
Using config file "./mydomu1.phy".
Error: [Errno 2] No such file or directory: '/proc/xen/balloon'


Reason:  Not enough free memory - check the domain configuration

Use xentop or "xm list" to get the memory already used:

bash-3.00# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      512     1 r-----   413.3
mydomU1                            3      276     1 ------    13.8
xen-centos1                        1      128     1 ------    11.2



error: (11, 'Resource temporarily unavailable')


[08.08.06 snv_43 Xen 07_2006]

This is one of the methods Xen is telling you that you're trying to run a 32 Bit DomU in a 64 Bit Dom0.

[27.10.06 snv_46 Xen 08_2006]

Another possible reason for this error is: You're trying to start a DomU with a not supported kernel version

Check the log file /var/log/xend-debug.log

If there's a message like "ERROR: PAE-kernel on non-PAE host." you're using an invalid kernel for your DomU.
To fix this error; try to start the DomU with the non-PAE kernel (see the filelist below).


[19.07.07 Xen 07_2007]

The Xen drop 07/2007 does support PAE-kernels and non-PAE kernels



back to top

HowTos

This section contains some HowTos for Solaris and Linux DomUs running on Solaris Dom0. Note that some of the information below is also in the sections above.

[08.08.06 snv_43 Xen 07_2006]

Please note that the HowTos in this section are based on my experiences with Xen on Solaris. They may be completly wrong!

Notes:

All xm opperations rely upon the Xen control daemon, aka xend. For any xm commands to run xend must also be running. For this reason you should start xend as a service when your system first boots using xen.

Most xm commands require root privledges to run due to the communications channels used to talk to the hypervisor. Running as non root will return an error.

Most xm commands act asynchronously, so just because the xm command returned, doesn't mean the action is complete. This is important, as many operations on domains, like create and shutdown, can take considerable time (30 seconds or more) to bring the machine into a fully compliant state. If you want to know when one of these actions has finished you must poll through xm list periodically.
(Source: http://www.die.net/doc/linux/man/man1/xm.1.html[08.08.06])

see also the Xen FAQ  [08.08.06]

Solaris Dom0 Howtos



How to install Xen on Solaris


[07.08.06 snv_43 Xen 07_2006]

see http://www.opensolaris.org/os/community/xen/How-To-07-2006/install/ [25.07.06].


Some hints for installing


Install the packages and the bfu images right after a fresh installation - this makes life easier

According to the bfu documentation you should only bfu a system while on the console. Well, I did it in an Gnome Terminal running on the local display and it worked also  - but your mileage may vary.

Be aware that  the bfu will downgrade parts of your system if you add it to a newer Solaris release then the one the bfu was created for. In my case (installing the 07/2006 Xen bfu on snv_43) the bfu downgraded the bge driver and scsa2usb driver

Make sure that you do a reconfigure boot after installing Xen

Be aware that the Xen bfu changes the Solaris boot process. After installing this bfu the format of the entries in the /boot/grub/menu.lst is different; see this blog entry  GRUB menu.lst entries if using Xen [31.07.06]


[09.09.06 snv_46 Xen 08_2006]

see  the Installation instructions at Xen at OpenSolaris.org [09.09.06]

In the 08_2006 drop I did not find any outdated driver yet.


[19.07.07 Xen 07_2007]


Beginning with Xen Drop 07/2007 Xen is distributed as complete Solaris InstallationDVD. This make the installation of the Dom0 and DomUs much easier.


What SMF services are used in the Dom0 for Xen?

[19.07.07 Xen 07_2007]


# svcs -a | grep xctl
online         10:51:04 svc:/system/xctl/store:default
online 10:51:11 svc:/system/xctl/xend:default
online 10:51:11 svc:/system/xctl/console:default
online 10:51:16 svc:/system/xctl/domains:default
Note that the Xen services are disabled by default - you must manually enable them.


What Solaris Installation cluster is necessary to install Xen?

[07.08.06 snv_43 Xen 07_2006]

To install Xen you must install at least the SUNWCore Installcluster and the packages



(*) At least in my tests these packages were also neccessary


Can I start a 32 Bit DomU in a 64 Bit Dom0 or vice versa?

[07.08.06 snv_43 Xen 07_2006]

No, you can't. All DomUs must be the same architecture as the Dom0


Where is the config file for Xen?


/etc/xen/xend-config.sxp

[20.07.07 Xen 07_2007]

In Xen 07/2007, all of the properties from xend-config.sxp have been put into SMF xctl/xend (config/*).


Where are the logfiles from Xen?

[07.08.06 snv_43 Xen 07_2006]

The log files for Xen are:

-rw-r--r--   1 root     root        1224 Jul 26 18:54 /var/log/xen-debug.log   
-rwxr-xr-x   1 root     root        1296 Jul 25 23:10 /var/log/xend-debug.log  
-rw-r--r--   1 root     root      104259 Jul 26 19:47 /var/log/xend.log        
-rw-r--r--   1 root     root       10186 Jul 26 19:47 /var/log/xendev-event.log




The Xen Webserver (localhost:8000) only shows two links to himself


[07.08.06 snv_43 Xen 07_2006]

It looks like the Webserver is only for communication between the Xen daemons .


[15.09.06 snv_46 Xen 08_2006]

Looks like there is a more or less hidden http interface; use

http://localhost:8000/xend

to get the http interface of xen

Source: http://lists.xensource.com/archives/html/xen-users/2006-03/msg00020.html [15.09.06]

Note that you must activate the webserver in the Xen config file /etc/xen/xend-config.sxp and restart xend to get it to work


How to view information about the installed Xen version


[08.08.06 snv_43 Xen 07_2006]

bash-3.00# xm info
host                   : unknown
release                : 5.11
version                : matrix-build-2006-07-14
machine                : i86xen
nr_cpus                : 1
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 1
threads_per_core       : 1
cpu_mhz                : 1989
hw_caps                : 078bfbff:e3d3fbff:00000000:00000010:00000001:00000001:00000001
total_memory           : 1023
free_memory            : 86
xen_major              : 3
xen_minor              : 0
xen_extra              : .2-sun
xen_caps               : xen-3.0-x86_32
platform_params        : virt_start=0xfc000000
xen_changeset          : Tue Jul 11 15:33:52 2006 -0700 9584:65e1a0cd24aa
cc_compiler            : gcc version 3.4.3 (csl-sol210-3_4-20050802)
cc_compile_by          : xen-discuss
cc_compile_domain      : opensolaris.org
cc_compile_date        : Fri Jul 14 05:19:14 PDT 2006


[21.07.07 Xen 07_2007]

host                   : dhcppc0
release                : 5.11
version                : xen-nv66-2007-06-24
machine                : i86pc
nr_cpus                : 1
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 1
threads_per_core       : 1
cpu_mhz                : 1989
hw_caps                : 078bfbff:e3d3fbff:00000000:00000010:00000001:00000001:00000001
total_memory           : 1022
free_memory            : 33
xen_major              : 3
xen_minor              : 0
xen_extra              : .4-1-sun
xen_caps               : xen-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : Mon Jun 25 06:41:06 2007 -0700 13212:6d81d051701c
cc_compiler            : gcc version 3.4.3 (csl-sol210-3_4-20050802)
cc_compile_by          : xen-discuss
cc_compile_domain      : opensolaris.org
cc_compile_date        : Mon Jun 25 06:44:38 PDT 2007
xend_config_format     : 3



How to view the Xen kernel messages


[08.08.06 snv_43 Xen 07_2006]

bash-3.00# xm dmesg
 __  __            _____  ___   ____
 \ \/ /___ _ __   |___ / / _ \ |___ \    ___ _   _ _ __
  \  // _ \ '_ \    |_ \| | | |  __) |__/ __| | | | '_ \
  /  \  __/ | | |  ___) | |_| | / __/|__\__ \ |_| | | | |
 /_/\_\___|_| |_| |____(_)___(_)_____|  |___/\__,_|_| |_|

 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0.2-sun (xen-discuss@opensolaris.org) (gcc version 3.4.3 (csl-sol210-3_4-20050802)) Fri Jul 14 05:19:14 PDT 2006 Latest ChangeSet: Tue Jul 11 15:33:52 2006 -0700 9584:65e1a0cd24aa

(XEN) Physical RAM map:
(XEN)  0000000000000000 - 000000000009f800 (usable)
(XEN)  000000000009f800 - 00000000000a0000 (reserved)
(XEN)  00000000000d0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000003fef0000 (usable)
(XEN)  000000003fef0000 - 000000003fefb000 (ACPI data)
(XEN)  000000003fefb000 - 000000003ff00000 (ACPI NVS)
(XEN)  000000003ff00000 - 0000000040000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec10000 (reserved)
(XEN)  00000000fed20000 - 00000000fed90000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) System RAM: 1022MB (1047100kB)
(XEN) Xen heap: 10MB (10320kB)
(XEN) Using scheduler: Simple EDF Scheduler (sedf)
(XEN) PAE disabled.
(XEN) found SMP MP-table at 000f6ab0
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: RSDP (v000 PTLTD                                 ) @ 0x000f6a80
(XEN) ACPI: RSDT (v001 PTLTD    RSDT   0x06040000  LTP 0x00000000) @ 0x3fef491f
(XEN) ACPI: FADT (v001 ATI    Firanha  0x06040000 ATI  0x000f4240) @ 0x3fefadff
(XEN) ACPI: SSDT (v001 PTLTD  POWERNOW 0x06040000  LTP 0x00000001) @ 0x3fefae73
(XEN) ACPI: MADT (v001 PTLTD     APIC   0x06040000  LTP 0x00000000) @ 0x3fefaf6a
(XEN) ACPI: MCFG (v001 PTLTD    MCFG   0x06040000  LTP 0x00000000) @ 0x3fefafc4
(XEN) ACPI: DSDT (v001    ATI    SB400 0x06040000 MSFT 0x0100000e) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 15:4 APIC version 16
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 1, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 21 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Initializing CPU#0
(XEN) Detected 1989.863 MHz processor.
(XEN) CPU0: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 1024K (64 bytes/line)
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: AMD Turion(tm) 64 Mobile Technology ML-37 stepping 02
(XEN) Total of 1 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
(XEN) ...trying to set up timer (IRQ0) through the 8259A ...  failed.
(XEN) ...trying to set up timer as Virtual Wire IRQ... works.
(XEN) Platform timer is 1.193MHz PIT
(XEN) Brought up 1 CPUs
(XEN) Machine check exception polling timer started.
(XEN) Using IPI Shortcut mode
(XEN) *** LOADING DOMAIN 0 ***
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   08000000->10000000 (98304 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: 40000000->4080e96c
(XEN)  Init. ramdisk: 4080f000->46198000
(XEN)  Phys-Mach map: 46198000->46218000
(XEN)  Start info:    46218000->46219000
(XEN)  Page tables:   46219000->46233000
(XEN)  Boot stack:    46233000->46234000
(XEN)  TOTAL:         40000000->46400000
(XEN)  ENTRY ADDRESS: 40800000
(XEN) Dom0 has maximum 1 VCPUs
(XEN) Initrd len 0x5989000, start at 0x4080f000
(XEN) Scrubbing Free RAM: ...........done.
(XEN) Xen trace buffers: disabled
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen).
(XEN) (file=io_apic.c, line=1979)
(XEN) ioapic_guest_write: apic=0, pin=19, old_irq=19, new_irq=19
(XEN) ioapic_guest_write: old_entry=0000a9b1, new_entry=0001a9b1
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) (file=io_apic.c, line=1979)
(XEN) ioapic_guest_write: apic=0, pin=19, old_irq=19, new_irq=19
(XEN) ioapic_guest_write: old_entry=0000a9b1, new_entry=0001a9b1
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) (file=io_apic.c, line=1979)
(XEN) ioapic_guest_write: apic=0, pin=21, old_irq=21, new_irq=21
(XEN) ioapic_guest_write: old_entry=0000a9a9, new_entry=0001a9a9
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) APIC error on CPU0: 00(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)
(XEN) APIC error on CPU0: 40(40)


How to convert a DomID to a DomName and vice versa


[08.08.06 snv_43 Xen 07_2006]

Use "xm domid" and "xm domname" to convert a DomID to a DomName and vice versa, e.g.

bash-3.00# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      512     1 r-----   823.9
mydomU1                            6      276     1 ------    14.9
xen-centos1                        1      128     1 ------    17.2

bash-3.00# xm domid xen-centos1
1

bash-3.00# xm domname 1
xen-centos1

bash-3.00# xm domname 0
Domain-0



How to pause/unpause a DomU

[08.08.06 snv_43 Xen 07_2006]

use "xm pause" and "xm unpause" to pause or resume a DomU, e.g.:

bash-3.00# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      512     1 r-----   823.9
mydomU1                            6      276     1 ------    14.9
xen-centos1                        1      128     1 ------    17.2

bash-3.00# xm pause myDomU1
Error: <Fault 3: 'myDomU1'>

bash-3.00# xm pause 6

bash-3.00# xm unpause 6


How to save and restore a DomU


[08.08.06 snv_43 Xen 07_2006]

Use "xm save" and "xm restore" to save or restore a DomU:

restore state-file

Build a domain from an xm save state file. See save for more info.

save domain-id state-file

Saves a running domain to a state file so that it can be restored later. Once saved, the domain will no longer be running on the system, thus the memory allocated for the domain will be free for other domains to use. xm restore restores from this state file.

This is roughly equivalent to doing a hibernate on a running computer, with all the same limitations. Open network connections may be severed upon restore, as TCP timeouts may have expired

(Note: not tested yet)


[21.07.07 Xen 07_2007]

In Xen 07/07 use the XM parameter new, start, save, and restore to save and restore DomUs


Is there a limit for file based disk images like in Linux Dom0s?

[08.08.06 snv_43 Xen 07_2006]

There's no limit for file based disk images in Solaris Dom0

(Note: By default, Linux only allows up to 8 loop-back devices to be active. You can increase this number by setting max_loop=32 (etc) on the domain 0 kernel command line in Grub. You also need to create new loop[n] device nodes in /dev).

Solaris uses lofi Devices for file based disk images. Therefor you can use lofiadm to check which file based disk images are currently in use, e.g.:

bash-3.00# lofiadm
Block Device             File
/dev/lofi/1              /export/xen-images/centos1/rootfs.img
/dev/lofi/2              /export/xen-images/datadisk.img
/dev/lofi/3              /export/xen-images/mydomU1/root.file
/dev/lofi/4              /export/xen-images/soldatadisk.img



How to check if a we're running in a Xen Domain (either Dom0 or DomU)


[19.07.07  Xen 07_2007]

Beginning with Xen 07_2007 you can use the uname command to do that

# uname -i
i86xpv

i86xpv is the platform for Solaris Xen Domains.



General DomU HowTos


How to get a graphical login for a DomU

[07.08.06 snv_43 Xen 07_2006]

Install and start VNC server inside the DomU (if not already installed) and use the VNC viewer to connect to the DomU, e.g.

	vncview 192.168.178.18:5900

[21.07.07 Xen 07_2007]


If you don't have a vncviewer installed, you can use the one which is part of vino (remote desktop). Here's what I have on some of my systems.

$ cat `which vncviewer`
#!/bin/sh
exec java -jar /usr/share/gnome/vino/vino-client.jar  ${1+"$@"}
(Source:  http://blogs.sun.com/mrj/entry/the_latest_solaris_on_xen [21.07.07])



How to increase the memory of a running DomU


[07.08.06 snv_43 Xen 07_2006]

This feature is not supported in the current Solaris Xen implementation

To increase the memory of a DomU shutdown the DomU, change the config file for the DomU and restart the DomU.

[19.07.07 Xen 07_2007]

The Xen drop 07/2007 does support ballooning so that you can increase the memory of a running DomU (note: not tested yet)



How to attach another network adapter to a running Solaris DomU


[08.08.06 snv_43 Xen 07_2006]

Use "xm network-list <domname>" to list the currently attached network adapter of a DomU, e.g.

bash-3.00# xm network-list xen-centos1
Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
0   0  00:16:3E:00:00:15    0     4      8     522  /523     /local/domain/0/backend/vif/1/0


Use

network-attach domain-id [script=scriptname] [ip=ipaddr] [mac=macaddr] [bridge=bridge-name] [backend=bedomain-id]

to attach another network adapter to a domain; use

network-detach domain-id devid

to detach a network adapter from a domain

(not tested yet)


How to attach a disk image or disk device in read/write mode

[08.08.06 snv_43 Xen 07_2006]

Use the "w" for the mode parameter of "xm block-attach" to attach a disk image or disk device in read/write mode; use "r" to attach in read-only mode.  xend in Solaris only looks at the first chracter of the mode parameter.

see this thread in the Xen discussion on Opensolaris.org


How to list the settings of a running DomU


[08.08.06 snv_43 Xen 07_2006]

bash-3.00# xm list mydomU1 --long
(domain
    (domid 3)
    (uuid 4f4a0c95-66f6-29cd-a39e-5bb345bd6e1b)
    (ssidref 0)
    (vcpus 1)
    (cpu_weight 1.0)
   
   
    (memory 276)
    (maxmem 276)
    (name mydomU1)
    (on_poweroff destroy)
    (on_reboot restart)
    (on_crash destroy)
    (image
        (linux
            (kernel /export/xen-images/mydomU1/platform/i86xen/kernel/unix)
            (ramdisk /export/xen-images/mydomU1/platform/i86pc/boot_archive)
            (root /dev/dsk/c0d0s0)
            (args /platform/i86xen/kernel/unix)
        )
    )
    (device
        (vif
            (backend 0)
            (script vif-bridge)
            (bridge xenbr0)
            (mac 00:16:3e:00:00:18)
        )
    )
    (device
        (vbd
            (backend 0)
            (dev 0)
            (uname file:/export/xen-images/mydomU1/root.file)
            (mode w)
        )
    )
    (state ------)
    (shutdown_reason poweroff)
    (cpu_time 14.475556563)
    (online_vcpus 1)
    (up_time 176.174789906)
    (start_time 1155020971.72)
    (store_mfn 67201)
    (console_mfn 67200)
)



How to start a DomU from a logical partition


Solaris does not support Logical Partitions - therefor it's difficult to start a DomU from a logical partition. One method is to give the DomU access to the whole disk, e.g. the following Xen DomU config file works:


kernel = "/export/xen-images/centos1/vmlinuz-2.6.16.13-xen"
memory = 128
name = "ubuntu6"
disk = [ 'phys:/dev/dsk/c0d0p0,hda,r' ]
vif = [ 'mac=00:16:3E:00:00:21' ]
root = "/dev/hda5 ro"


But be aware that this is a very (!) dangerous method! You should only use this on test systems!


How to create a disk image file for a DomU

The easiest way to create a disk image for a DomU is to use qemu-img. qemu-img creates a sparse file . E.g.

bash-3.00# qemu-img create -f raw Solaris10_xen0707.raw 10G
Formating 'Solaris10_xen0707.raw', fmt=raw, size=10485760 kB

bash-3.00# ls -l Solaris10_xen0707.raw
-rw-r--r--   1 root     root     10737418240 Jul 22 13:03 Solaris10_xen0707.raw

bash-3.00# du -ks Solaris10_xen0707.raw
24      Solaris10_xen0707.raw


see also How to copy a DomU image file created as sparse file

How to copy a DomU image file created as sparse file


To copy a sparse file from one machine to another first tar the file with gnu tar (otherwise you loose the wholes in the sparse file).

Thanks to  John Levon for the hint  [21.07.07].

Example:


# on the source machine runnning Solaris
#
bash-3.00# ls -l Solaris10_xen0707.raw
-rw-r--r--   1 root     root     10737418240 Jul 21 23:50 Solaris10_xen0707.raw

bash-3.00# du -ks Solaris10_xen0707.raw
1351176 Solaris10_xen0707.raw

bash-3.00# ls -l sol10xenimg.tar
-rw-r--r--   1 root     root     590704640 Jul 22 08:44 sol10xenimg.tar



# on the target machine running Linux (the filesystem is ext3):
#
root@t30:/data/incoming# ls -l sol10xenimg.tar
-rw-r--r-- 1 xtrnaw7 xtrnaw7 590704640 2007-07-22 08:53 sol10xenimg.tar

root@t30:/test# ls -l export/xen-images/Solaris10_xen0707.raw ;
-rw-r--r-- 1 root root 10737418240 2007-07-21 23:50 export/xen-images/Solaris10_xen0707.raw

root@t30:/test# du -ks export/xen-images/Solaris10_xen0707.raw
603908  export/xen-images/Solaris10_xen0707.raw



Solaris DomU HowTos


How to create a Solaris DomU


[07.08.06 snv_43 Xen 07_2006]

see Creating Solaris domU virtual disk images at Xen at OpenSolaris.org  [03.08.06]

see also flarcreate issue:  Do not use -S

[21.07.07 Xen 07_2007]

Note that the method to create Solaris DomUs is much easier now with Xen 07/07!

How to format a disk image


[07.08.06 snv_43 Xen 07_2006]

In the current release of Xen for Solaris you can NOT format a disk image inside a Solaris DomU. Just treat the disk as one big slice and use
   
    newfs /dev/dsk/c0d0s0

to create a filesystem on the slice.

see also format does not work inside a Solaris DomU


[14.09.06 snv_46 Xen 08_2006]

formating a disk image is supported in the 08-2006 drop fo Xen  - see here [14.09.06]


Example:

# ### in the Dom0

(bash):root@ferrari:/export/xen-images # xm block-attach solaris-b44  "file://export/xen-images/datadisk.img" c3  w

(bash):root@ferrari:/export/xen-images # xm block-list solaris-b44
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/4/0
195    0    0     1      -1     -1    /local/domain/0/backend/vbd/4/195


# ### in the DomU

bash-3.00# devfsadm

bash-3.00# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 1988 alt 0 hd 128 sec 32>
          /xendev/xvbd@0
       1. c0d195 <DEFAULT cyl 500 alt 0 hd 64 sec 32>
          /xendev/xvbd@c3
Specify disk (enter its number): 1
selecting c0d195
[disk formatted, no defect list found]


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        show       - translate a disk address
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> fdisk
No fdisk table exists. The default partition for the disk is:

  a 100% "SOLARIS System" partition

Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
y
format> p


PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 499 + 0 (reserved cylinders)

Part      Tag    Flag     Cylinders       Size            Blocks
  0 unassigned    wm       0 - 498      499.00MB    (499/0/0) 1021952
  1 unassigned    wm       0              0         (0/0/0)         0
  2     backup    wu       0 - 498      499.00MB    (499/0/0) 1021952
  3 unassigned    wm       0              0         (0/0/0)         0
  4 unassigned    wm       0              0         (0/0/0)         0
  5 unassigned    wm       0              0         (0/0/0)         0
  6 unassigned    wm       0              0         (0/0/0)         0
  7 unassigned    wm       0              0         (0/0/0)         0
  8       boot    wu       0 -   0        1.00MB    (1/0/0)      2048
  9 unassigned    wm       0              0         (0/0/0)         0

partition> label
Ready to label disk, continue? y

Warning: no backup labels
partition> ^D

bash-3.00# newfs /dev/rdsk/c0d195s0
newfs: construct a new file system /dev/rdsk/c0d195s0: (y/n)? y
/dev/rdsk/c0d195s0:     1021952 sectors in 499 cylinders of 64 tracks, 32 sectors
        499.0MB in 32 cyl groups (16 c/g, 16.00MB/g, 7680 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 32832, 65632, 98432, 131232, 164032, 196832, 229632, 262432, 295232,
 721632, 754432, 787232, 820032, 852832, 885632, 918432, 951232, 984032,
 1016832



Note: Do NOT use format/fdisk for the root disk of the running DomU!



How to install VNC Server in a Solaris DomU

[07.08.06 snv_43 Xen 07_2006]

see Xvnc with Solaris on Xen [03.08.06]



How to mount a Solaris DomU disk image in the Dom0


[07.08.06 snv_43 Xen 07_2006]

Use lofiadm to mount the disk image of a Solaris DomU in the Solaris Dom0, e.g.


bash-3.00# lofiadm -a /export/xen-images/mydomU1/root.file
/dev/lofi/1

bash-3.00# mount -o ro /dev/lofi/1 /mnt
bash-3.00# ls /mnt
archives                      etc                           mnt                           sbin
bfu.child                     export                        net                           solcor
bfu.conflicts                 home                          opt                           system
bfu.parent                    install_xen.log               osox-bfu-2006-07-14.acr.done  tmp
bin                           kernel                        osox-bfu-2006-07-14.bfu.done  usr
boot                          lib                           platform                      var
dev                           lost+found                    proc
devices                       mainroot                      root

bash-3.00# df -k /mnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/lofi/1          1028250  839753  126802    87%    /mnt



[21.07.07 Xen 07_2007]

In Xen 07/07 the disk images are partitioned disk images. Therefor you can not simply mount the image files with lofiadm.


How to attach a disk image file to a running Solaris DomU

[08.08.06 snv_43 Xen 07_2006]

# ### in the Dom0

bash-3.00# xm block-list mydomU1
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/10/0

bash-3.00# xm block-attach mydomU1 file://export/xen-images/soldatadisk.img c3 w

bash-3.00# xm block-list mydomU1
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/10/0
195    0    0     1      -1     -1    /local/domain/0/backend/vbd/10/195


# ### in the Solaris DomU
#
bash-3.00# ls /dev/dsk
c0d0s0  c0d0s1  c0d0s2  c0d0s3  c0d0s4  c0d0s5  c0d0s6  c0d0s7

bash-3.00# devfsadm

bash-3.00# ls /dev/dsk
c0d0s0    c0d0s2    c0d0s4    c0d0s6    c0d195s0  c0d195s2  c0d195s4  c0d195s6
c0d0s1    c0d0s3    c0d0s5    c0d0s7    c0d195s1  c0d195s3  c0d195s5  c0d195s7


bash-3.00# mount -o rw /dev/dsk/c0d195s0 /mnt

bash-3.00# ls /mnt
bash        lost+found

bash-3.00# cp /usr/bin/basename /mnt

bash-3.00# ls /mnt
basename    bash        lost+found

bash-3.00# df -k /mnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d195s0      47897    1681   41427     4%    /mnt


bash-3.00# umount /mnt


# ### in the Dom0
#

bash-3.00# xm block-detach mydomU1 c3

bash-3.00# xm block-list mydomU1
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/10/0



# ### in DomU
#
bash-3.00# ls /dev/dsk
c0d0s0    c0d0s2    c0d0s4    c0d0s6    c0d195s0  c0d195s2  c0d195s4  c0d195s6
c0d0s1    c0d0s3    c0d0s5    c0d0s7    c0d195s1  c0d195s3  c0d195s5  c0d195s7

bash-3.00# devfsadm -C

bash-3.00# ls /dev/dsk
c0d0s0  c0d0s1  c0d0s2  c0d0s3  c0d0s4  c0d0s5  c0d0s6  c0d0s7



How to attach an ISO image file to running Solaris DomU

[08.08.06 snv_43 Xen 07_2006]

# ### in the Dom0
#
 xm block-attach mydomU1 file://export/iso/hwb.iso c3 r
 
# ### in the Solaris DomU

# before the block-attach
#
 bash-3.00# ls /dev/dsk
c0d0s0    c0d0s2    c0d0s4    c0d0s6    c0d193s0  c0d193s2  c0d193s4  c0d193s6
c0d0s1    c0d0s3    c0d0s5    c0d0s7    c0d193s1  c0d193s3  c0d193s5  c0d193s7

# after the block-attach
#
bash-3.00# devfsadm -v
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s0 -> ../../devices/xendev/xvbd@c3:a
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s0 -> ../../devices/xendev/xvbd@c3:a,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s1 -> ../../devices/xendev/xvbd@c3:b
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s1 -> ../../devices/xendev/xvbd@c3:b,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s2 -> ../../devices/xendev/xvbd@c3:c
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s2 -> ../../devices/xendev/xvbd@c3:c,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s3 -> ../../devices/xendev/xvbd@c3:d
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s3 -> ../../devices/xendev/xvbd@c3:d,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s4 -> ../../devices/xendev/xvbd@c3:e
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s4 -> ../../devices/xendev/xvbd@c3:e,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s5 -> ../../devices/xendev/xvbd@c3:f
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s5 -> ../../devices/xendev/xvbd@c3:f,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s6 -> ../../devices/xendev/xvbd@c3:g
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s6 -> ../../devices/xendev/xvbd@c3:g,raw
devfsadm[100376]: verbose: symlink /dev/dsk/c0d195s7 -> ../../devices/xendev/xvbd@c3:h
devfsadm[100376]: verbose: symlink /dev/rdsk/c0d195s7 -> ../../devices/xendev/xvbd@c3:h,raw
devfsadm[100376]: verbose: SUNW_port_link: port monitor ttymon0 added
devfsadm[100376]: verbose: SUNW_port_link: /dev/term/0 added to sacadm

bash-3.00# ls /dev/dsk
c0d0s0    c0d0s3    c0d0s6    c0d193s1  c0d193s4  c0d193s7  c0d195s2  c0d195s5
c0d0s1    c0d0s4    c0d0s7    c0d193s2  c0d193s5  c0d195s0  c0d195s3  c0d195s6
c0d0s2    c0d0s5    c0d193s0  c0d193s3  c0d193s6  c0d195s1  c0d195s4  c0d195s7

bash-3.00# mount -o ro -F hsfs /dev/dsk/c0d195s0 /mnt

bash-3.00# df -k /mnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d195s0       4084    4084       0   100%    /mnt

bash-3.00# ls /mnt
about       circuit     images      mirror      table
adapter     comment     index.html  news        wanted
cable       connector   links       robots.txt

# ### in the Dom0

# detach the ISO image file

bash-3.00# xm block-detach mydomU1 c3

bash-3.00# xm block-list mydomU1
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/3/0



How to attach the physical CDROM drive to a running Solaris DomU

[08.08.06 snv_43 Xen 07_2006]

# ### in the Dom0
#

# stop the Volume Management daemon
#
/etc/init.d/volmgt stop


bash-3.00# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      512     1 r-----  1404.1
xen-centos1                        1      128     1 -b----    37.9

bash-3.00# xm block-attach mydomU1  phy://dev/rdsk/c1t0d0s0 c3  r

bash-3.00# xm block-list mydomU1
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/10/0
195    0    0     1      -1     -1    /local/domain/0/backend/vbd/10/195

# ### in the Solaris DomU
#
bash-3.00# ls /dev/dsk
c0d0s0  c0d0s1  c0d0s2  c0d0s3  c0d0s4  c0d0s5  c0d0s6  c0d0s7

bash-3.00# devfsadm

bash-3.00# ls /dev/dsk
c0d0s0    c0d0s2    c0d0s4    c0d0s6    c0d195s0  c0d195s2  c0d195s4  c0d195s6
c0d0s1    c0d0s3    c0d0s5    c0d0s7    c0d195s1  c0d195s3  c0d195s5  c0d195s7

bash-3.00# mount -o ro -F hsfs /dev/dsk/c0d195s0 /mnt

bash-3.00# ls /mnt
README.diskdefines  dists               pool                start.ini
autorun.inf         install             preseed             ubuntu
bin                 isolinux            programs            ubuntu.ico
casper              md5sum.txt          start.bmp
disctree            pics                start.exe

# to deattach

# ### in the Solaris DomU

bash-3.00# umount /mnt

# ### in the Dom0
#
bash-3.00# xm block-detach mydomU1 c3

bash-3.00# xm block-list mydomU1
Vdev  BE handle state evt-ch ring-ref BE-path
0      0    0     4      5      8     /local/domain/0/backend/vbd/10/0


# ### in the Solaris DomU
#
bash-3.00# devfsadm -C

bash-3.00# ls /dev/dsk
c0d0s0  c0d0s1  c0d0s2  c0d0s3  c0d0s4  c0d0s5  c0d0s6  c0d0s7


Note:

If the vold is running, replace the block-attach command with

xm block-attach mydomU1 pyh://vol/dev/rdsk/c1t0d0/ubuntu6.06i386  c3 r


How to resize a Solaris DomU disk image

[08.08.06 snv_43 Xen 07_2006]

# ### in the Dom0
# (after the shutdown of the Solaris DomU)

#

bash-3.00# df -k /
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d0s0      1028250  840072  126483    87%    /


bash-3.00# ls -l root.file
-rw------T   1 root     root     1088523264 Aug  8 09:09 root.file

# add 50 MB to the image file
#
bash-3.00# dd if=/dev/zero bs=1024000 count=50 >>./root.file
50+0 records in
50+0 records out


bash-3.00# ls -l root.file
-rw------T   1 root     root     1139723264 Aug  8 15:11 root.file


# ### in the Solaris DomU
#

bash-3.00# df -k /
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d0s0      1028250  840101  126454    87%    /

# grow the filesystem
#
bash-3.00# growfs -M / /dev/rdsk/c0d0s0
/dev/rdsk/c0d0s0:       2226000 sectors in 3710 cylinders of 1 tracks, 600 sectors
        1086.9MB in 116 cyl groups (32 c/g, 9.38MB/g, 2368 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 19232, 38432, 57632, 76832, 96032, 115232, 134432, 153632, 172832,
 2035232, 2054432, 2073632, 2092832, 2112032, 2131232, 2150432, 2169632,
 2188832, 2208032

bash-3.00# df -k /
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d0s0      1076790  840149  174946    83%    /


[14.09.06 snv_46 Xen 08_2006]

This approach may not work with a formated disk image in Xen 08_2006 (not tested yet!)



How to reboot a Solaris DomU

[08.08.06 snv_43 Xen 07_2006]

Always use the approbiate Solaris command inside a Solaris DomU to reboot or halt the Domain. "xm reboot" seems not to work for a Solaris DomU at this time.


[09.09.06 snv_46 xen 08_2006]

Looks like rebooting a Solaris DomU is not supported in this release: After issuing a "reboot" the DomU shuts down but does not automatically restart. Reooting works for Linux DomUs.

[21.07.07 Xen 07_2007]

Rebooting a Solaris DomU does work in Xen 07/07


How to turn off checksum offloading


Add

set xnf:xnf_cksum_offload = 0

to /etc/system



Linux DomU HowTos


[07.08.06 snv_43 Xen 07_2006]

see also the Documentation links below


How to create sample Linux DomUs without installing Linux


[07.08.06 snv_43 Xen 07_2006]

see http://www.dme.org/log/2006/07/17/solaris-on-xen-a-linux-domu-without-installing-linux/ [25.07.06].

or download a Linux image with Xen support from http://jailtime.org (or one of the other sources mentioned in the Links section below)

see also DomU Images tested


Which Linux Kernel is necessary for Xen 3.0 Domains?

[07.08.06 snv_43 Xen 07_2006]

At the time of writing Xen 3.0 is not (yet?) ported to the Linux Kernel 2.4.x. Therefor only Linux distributions with the Kernel 2.6.x will work in a Xen 3.0 domain.

Distributions based on Linux 2.4.x  should boot in Xen 3.0 Dom0s with Full Virtualization support - but that is not yet implemented in the Solaris Dom0.


How to get the Network working in a Linux DomU

[07.08.06 snv_43 Xen 07_2006]

At this time you must disable TX checksumming feature to get a reliable network connection. In most Linux Distribution this can be done with done with

    ethttool -K eth0 tx off

You should add this command to one of the startup scripts of the Linux DomU.

If the distribution you're using does not include the binary ethtool, copy the binary from the Centos image from the Xen Demo Disk.


How to mount a Linux DomU disk image in the Solaris Dom0


[07.08.06 snv_43 Xen 07_2006]

Due to the missing ext2/ext3 support in Solaris you can not mount a disk image from a Linux DomU in a Solaris Dom0.

To change the contents of the disk image of a Linux Dom0 mount the image file as second disk in another  Linux Dom0, e.g


kernel = "/export/xen-images/centos1/vmlinuz-2.6.16.13-xen"
memory = 128
name = "xen-centos1"
disk = ['file:/export/xen-images/centos1/rootfs.img,sda1,w','file:/export/xen-images/other_rootfs.img,sda2,w']
vif = [ 'mac=00:16:3E:00:00:15' ]
root = "/dev/sda1 ro"



[09.09.06 snv_46 Xen 08_2006]

Download the new packages with Miscellaneous Filesystem Support (NTFS, Ext2Fs) for OpenSolaris  from the Belenix Homepage [09.09.06].  These packages contain the binaries for read-only support of ext2, ext3, and ntfs filesystems in Solaris.

How to attach a disk image file to a running Linux DomU


[07.08.06 snv_43 Xen 07_2006]

# ### in the Dom0
#
# Liste the block devices of the Linux DomU (info only!)
#
bash-3.00# xm block-list xen-debian
Vdev  BE handle state evt-ch ring-ref BE-path
2049    0    0     4      6      8     /local/domain/0/backend/vbd/10/2049

# attach another image file to the DomU
#
bash-3.00# xm block-attach xen-debian file:/export/xen-images/datadisk.img sda2 w

# list the block devices of the Linux DomU
#
bash-3.00# xm block-list xen-debian
Vdev  BE handle state evt-ch ring-ref BE-path
2049    0    0     4      6      8     /local/domain/0/backend/vbd/10/2049
2050    0    0     4      8      546   /local/domain/0/backend/vbd/10/2050

# ### in the Linux DomU mount the new block device (after issuing mkfs, of course)
#
localhost:~# mount /dev/sda2 /mnt
localhost:~# ls /mnt
ethtool  lost+found

# umount the device again
#
umount /dev/sda2


# ### in the Dom0
#
# detach the image file from the Linux DomU
#
bash-3.00# xm block-detach xen-debian sda2

# list the attached block devices again
#
bash-3.00# xm block-list xen-debian
Vdev  BE handle state evt-ch ring-ref BE-path
2049    0    0     4      6      8     /local/domain/0/backend/vbd/10/2049



How to attach an ISO image file to a running Linux DomU


[08.08.06 snv_43 Xen 07_2006]


# ### in the Dom0
#
bash-3.00# xm block-attach xen-centos1 file://export/iso/hwb.iso /dev/hdc r

bash-3.00# xm block-list xen-centos1
#Vdev  BE handle state evt-ch ring-ref BE-path
2049    0    0     4      6      8     /local/domain/0/backend/vbd/1/2049
2050    0    0     4      7      9     /local/domain/0/backend/vbd/1/2050
5632    0    0     4      9      547   /local/domain/0/backend/vbd/1/5632

# ### in the Linux DomU

mount -o ro /dev/hdc /mnt

# ### in the Dom0 use the following commands to detach the ISO image
#
bash-3.00# xm block-detach xen-centos1 5632

bash-3.00# xm block-list xen-centos1
Vdev  BE handle state evt-ch ring-ref BE-path
2049    0    0     4      6      8     /local/domain/0/backend/vbd/1/2049
2050    0    0     4      7      9     /local/domain/0/backend/vbd/1/2050



How to attach the physical CDROM drive to a running Linux DomU

[08.08.06 snv_43 Xen 07_2006]

Stop the Volume Management in the Dom0:

/etc/init.d/volmgt stop

and attach the drive to the Linux DomU:

xm block-attach xen-centos1 phy://dev/rdsk/c1t0d0s0 /dev/sda3 r

In the Linux DomU use

mount -o ro /dev/sda3 /mnt

to mount the drive and

umount /mnt

to umount the drive.


To detach isssue in the Dom0:

xm block-detach xen-centos1 /dev/sda3


If vold is running, replace the attach command with

xm block-attach xen-centos1 pyh://vol/dev/rdsk/c1t0d0/ubuntu6.06i386 /dev/sda3 r


How to resize a Linux DomU disk file image

[08.08.06 snv_43 Xen 07_2006]

see here




back to top

Files


This is a list of files and scripts for Xen on Solaris I put on my website that maybe useful.

install_xen_0706.sh Script to install Xen (pkgs, bfu, and necessary config file changes) according to the instructions on this Web Site: Installation Instructions at OpenSolaris.org [24.07.06]
and my experiences while installing and testing Xen

Source: me, [Last update of the script: 09.08.06/bs]

## History
##   25.07.2006 v1.0.0.0 /bs
##     initial release for Xen bfu 2006-07-14
##   26.07.2006 v1.0.0.1 /bs
##     added workaround for the PS/2 keyboard bug
##   27.07.2006 v1.0.0.2 /bs
##     added code to save & restore the original bge driver
##   01.08.2006 v1.0.0.3 /bs
##     added code to save & restore the original USB driver
##   09.08.2006 v1.0.0.4 /bs
##     added code to remove the invalid gdm2 SMF service
##

This script is only tested with the 07/2006 Xen release installed on Solaris snv_43!
[09.08.06 snv_43 Xen 07_2006]
centos1.phy
Xen configuration file for the Centos image from the Xen Demo Disk
[27.07.06 snv_43 Xen 07_2006]
start_xen_centos1.sh
Sample script to start a Xen DomU with the Centos image from the Xen Demo Disk
[27.07.06 snv_43 Xen 07_2006]
start_xen.sh
General kornshell script to start a Xen DomU and/or attach a console and/or run the vncviewer for the DomU. There's no documentation for the script right now - so please check the source code. The important part is the part with the configuration variables at the start of the script
[14.09.06 snv_46 Xen 08_2006]
vmlinuz-2.6.16.13-xen
Working Kernel for the Linux DomUs from the Xen Demo Disk

Note:

This is a non-PAE kernel - it does not work for DomUs running under a Solaris Dom0 with Xen 07/07
[26.10.06 snv_Xen 08_2006]




back to top

Links



General


Xen Homepage [24.07.06]


Xen Demo Download [24.07.06]

[26.10.06] Update:

Looks like the current Xen Demo Disk contains the Linux Kernel vmlinuz-2.6.16.29-xen which is not working in a DomU on a Solaris Dom0. Please use the kernel vmlinuz-2.6.16.13-xen instead.

[19.07.07] Update:

Xen Drop 07/2007 supports PAE kernel therefor the Xen Demo Disk should work in a Solaris Dom0 with Xen Drop 07/2007 [not tested yet]


Documentation

Communitiy Documentation[07.08.06]

Xen HowTos [27.07.06]

Xen FAQ [27.07.06]

Small Xen Howto [07.08.06]

Xen Networking [08.08.06]

Xen Security [10.08.06]


Xen and OpenSolaris


Xen (July 2007, based on Xen 3.0.4 and build 66 of Nevada) announcement [19.07.07]

Xen (July 2007) ISO Image download [19.07.07]

Xen (July 2007) BFU download [21.07.07]


Xen bfu image August 2006 (based on snv_44) [31.08.06]

Xen for dummies [31.07.06]

Open Solaris on Xen - some internals[31.07.06]

Live migration of Solaris instances [31.07.06]

Xvnc with Solaris on Xen [03.08.06]

Installation Instructions at OpenSolaris.org (07/2006)  [24.07.06]

Installation Instructions at OpenSolaris.org(08-15-06) [31.08.06]

Known Issues (bugs) with Xen on Solaris [26.07.06]

GRUB menu.lst entries if using Xen [24.07.06]

Hints for Xen drop 07/2007  [19.07.07]


Xen DomU Images and utils


Xen - a Linux Dom without Linux [24.07.06]
[27.10.06] Update: see my notes here

Creating Solaris domU virtual disk images at OpenSolaris.org [03.08.06]

Ready to use Solaris 10 domU image [31.08.06]
(Download link) [20.11.06]


Xenoppix 5.0.1 Xen3.0.3 (http, ftp, and bittorent) [20.11.06]



jailtime.org - OS Images with Xen Support [07.08.06]

As of 07.08.2006 here are these images:


www.xen-get.org - OS Images with Xen Support [10.08.06]

Looks like there are only description of images here  - but not the images ;-((

[27.10.06] Update

The images described on that site can be downloaded from this site:

ftp://mirror.lost-oasis.fr/xen-get/ [27.10.06]

As of 27.10.2006 here are these images:


http://www.suse.de/~garloff/linux/xen/images/ttylinux/ [20.11.06]

Various Linux DomU Images; as of 20.11.06 here are these images:


http://cdprojekte.mattiasschlenker.de/Public/Xen-Images/ [05.12.06]

Various Linux DomU Images; as of 05.12.06 here are these images:

Note:

The description of the images (in German) is here  http://news.mattiasschlenker.de/ [05.12.06]


FreeBSD Xen Image (thanks to Mark Johnson for the link) [10.08.06]


NetBSD/Xen HowTo (thanks to Mark Johnson for the link) [10.08.06]

Note: As of 08/10/06 the NetBSD domU is not working but that's a NetBSD problem - not a Solaris problem

Instructions for installing Xen in Ubuntu 6.06.1LTS and creating a Ubuntu DomU [13.09.06]


qemu OS images (without Xen!) [27.10.06]




Belenix Homepage (contains Solaris packages with ext2, ext3, and ntfs support) [09.09.06]


Sloggi Homepage [09.09.06]

Note: A compiled binary for Solaris 10 is available on my homepage. in the download section.





German Links


The following links all point to German Webpages

Webpage with links to OS Images with Xen Support [07.08.06]

German Xen Forum [07.08.06]

Xen Installation Documentation for Linux [07.08.06]

German Xen Docs [10.08.06]

Xen 3 Handbook in German  [10.08.06]

Xen 3 (very good book about Xen in german) [20.11.06]



back to top

Credits

Mark Johnson - infos about his experiences with Xen on Solaris and the link to the FreeBSD Xen Image

And everyone else mentioned above, of course.



back to top

History


Date
Comment
23.07.2007
New: How to turn off checksum offloading
22.07.2007
New: How to copy a DomU image file created as sparse file
New: How to create a disk image file for a DomU
21.07.2007
New: Where is the config file for Xen
Update: DomU Images tested
Update:  xm reboot and xm_shutdown
Update: prtdiag does not work
Update: How to save and restore a DomU
Update: How to get a graphical login for a DomU
Update: How to create a Solaris DomU
Update: How to mount a Solaris DomU disk image in the Dom0
Update: How to reboot a Solaris DomU
New: My first experience with Xen 07/07
19.07.2007
New: Xen Drop July 2007  avaiable for download
New: How to check if a we are running in a Dom0
Update: Full Virtualization Support
Update: error: 11 Resource temporarily
Update: How to increase the memory of a running DomU
Update: How to install Xen on Solaris

05.12.2006
Update: Download sources for Xen Images
20.11.2006
Update: Links
03.11.2006
Update: Link to German Xen Forum
27.10.2006
Update: Links
Update: error: 11 Resource temporarily
26.10.2006
New: Added kernel image to the download section
Update: Infos about the Xen Demo Disk
15.09.2006
Update: The Xen Webserver
New: Where is the Xen config file?
14.09.2006
Update: How to format a disk image
New: start_xen.sh - a general kornshell script to start Xen DomUs in a Solaris Dom0
13.09.2006
Created and tested a DomU with Ubuntu 6.06.1LTS
09.09.2006
Tested the Solaris 10 DomU from the opensolaris website
Reinstalled Solaris (snv_46) and Xen (08/2006)
Update: How to mount a Linux DomU disk image in Solaris
Update: The bge driver in the Xen bfu image is outdated
Update: Invalid GDM login service
Update: Shutdown not working reliable after instaling Xen
Update: keyboard not working reliable after installing Xen
New: How to change the output of uname?
New: How to start a DomU from a logical partition?
31.08.2006
Update: Links

10.08.2006
Update: Links
Update: DomU Images tested

09.08.2006
New: NameError: name FutureWarning is not
Update: Missing Features
New: Invalid GDM login service
Update: install_xen.sh
New: How to get rid of a zombie domU
08.08.2006
New: Trouble Shooting
Update: HowTos
Corrected:  How to attach a disk image file to a running Solaris DomU
07.08.2006
Update: Links
New: German Links
New: HowTos
Moved the "Hints for installing and using Xen" to the new HowTo section
Update: DomU Images tested

04.08.2006
Update: Format does not work in Solaris DomU
Update: Some hints for installing and using Xen

03.08.2006
removed now not necessary information about bugs on some webpages
move some paragraphs to the bugs section
New: format does not work inside a Solaris
02.08.2006
Initial release of this Webpage