A study on Xen & XenServer internals

Update

xeninternals — Wed, 30 Nov 2011 08:19:58 +0000

Maintaining a blog is more of a challenge than I initially anticipated. In my defence, I also seem to have a lengthy and varied list of things I want to learn and be involved in all the same time. As a result, I often find my focus and attentiveness drifting to greener pastures. This might be because of some fundamental gaps in my knowledge that need to be filled in order to understand the whole picture of Xen and which are in constant competition with other interests. Finding the motivation, time and the skills to fill out these gaps is a matter of disposition and having a well defined and clear goal in mind, which I don’t seem to have as we speak, but hopefully I will be able to tackle this problem. One of the things I wanted to make out of this blog was to make it a place-holder for my own personal notes on Xen, QEMU, blktap and Linux. This hasn’t been the case much, but it’s not forgotten (yet.)

Excuses aside, I would like to congratulate the Xen development team for the great work in up-streaming the main components of Xen dom0 drivers in the release 3.0 of Linux — a version I will likely stick around with for some time.

For the follow up posts, I intend to go over configuration management, git, etc.

Stay tuned for the next update!

Where were we?

xeninternals — Sat, 16 Apr 2011 01:52:11 +0000

After a long silence, I am back and eager to pick up from where I dropped off (which is nearly the same as trying to start from the ground of

Considering the lack of comments, I assume this is still exclusively my playground. This is somehow positive, as the pressure decreases somewhat, but not at all ideally as the pressure makes us work a tiny better. Oh well!

For reference, this is the HVM config for a Windows7 guest I am using in current testing of qemu’s blktap-alike implementation.

import os, re
arch = os.uname()[4]

if re.search(’64′, arch):
arch_libdir = ‘lib64′
else:
arch_libdir = ‘lib’

kernel = “/usr/lib/xen/boot/hvmloader”
builder=’hvm’
memory = 1024

# Should be at least 2KB per MB of domain memory, plus a few MB per vcpu.
shadow_memory = 8
name = “win7″
vif = [ 'type=ioemu, bridge=xenbr0' ]
acpi = 1
apic = 1
disk = [ 'file:/root/vm/win2008/disk1.img,hda,w', 'file:/root/images/win7.iso,hdc:cdrom,r' ]

device_model = ‘/usr/’ + arch_libdir + ‘/xen/bin/qemu-dm’

#—————————————————————————–
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot=”dc”
sdl=0
vnc=1
vncconsole=1
vncpasswd=”

serial=’pty’
usbdevice=’tablet’

Error: Device 768 (vbd) could not be connected. Path closed or removed during

xeninternals — Thu, 17 Jun 2010 12:59:59 +0000

This is one of those errors that merits a better description, or an additional “Have you checked this or that”. Fortunately, the problem is very simple to solve:

Add the following line to your dom0 kernel .config:

CONFIG_XEN_BLKDEV_BACKEND=y

n.b: replace “=y” with “=m”, if you prefer modules.

Getting started

xeninternals — Thu, 17 Jun 2010 02:32:10 +0000

First, I’ll start by confessing that this post is mainly notes to myself to describe how I setup my development environment to start learning about and hacking Xen which I hope will be useful to others as well. I may not have all the steps clearly described here mainly because I spent 1 week and a few days trying different things until I finally decided to settle for Debian Lenny, as such, some of the things I’ve been thru may not be quite clear, but I’ll update if something pops to mind. Also, the choice of Lenny was not difficult, I’ve been running it at home and it is my understanding a number of Xen developers use Debian too, so I hope it wasn’t a bad choice afterall!

My environment is composed of:

1x HP DL 365 g1 / 5Gb RAM
Debian Lenny 5.0 x86_64 with only “base system” plus the following packages which were installed on an as-needed basis, but I decided to pack them all together.

apt-get install build-essential \ libc-dev \ openssh-server \ screen \ git-core \ curl \ ncurses-dev \ mercurial \ ccache \ ctags \ libz-dev \ gettext \ libx11-dev \ uuid-dev \ openssl-dev \ openssl \ libssl-dev \ iasl \ bin86 \ bcc \ python-dev \ iproute \ bridge-utils

Make ccache the default ‘gcc’ by means of a symbolic link. This is very useful when you plan to do a lot of compile work on the same barely-changed sources as it speeds up the compilation considerably.

ln -s /usr/bin/ccache /usr/local/sbin/gcc

Download Jeremy’s tree as detailed here:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen $ cd linux-2.6-xen $ git reset --hard $ git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x Branch xen/stable-2.6.32.x set up to track remote branch refs/remotes/origin/xen/stable-2.6.32.x. Switched to a new branch "xen/stable-2.6.32.x" $ git pull $ cp /boot/config-2.6.26-2-amd64 .config $ make oldconfig $ make -j 8 > /dev/null $ make -j 8 modules_install $ update-initramfs -c -k 2.6.32.15 # At the time of this writing, Jeremy's PVOPS is at 2.6.32.15

The next step is to download Xen 4.0 testing (4.0.1 rc2 is the current) and the debuggers patchset from Mukesh Rathor:

$ hg clone http://xenbits.xensource.com/xen-4.0-testing.hg

As it turns out, the debuggers tree is way too old and doesn’t appear to work with the current PVOPS kernel, as such I had to abbandon it temporarily until I manage to rebase the patches from Mukesh and patch the testing tree, however I just noticed 4.0.1rc2 already contain gdbsx so only kdb-xen would be required to be ported. This will come later at some point, possibly when 4.0.1 hits the door.

The steps I’ve used to compile are pretty generic and simple, too:

$ make xen $ make tools $ make install-xen $ make install-tools

My xen grub entry looks as the following:

title 2.6.32-xen (debuggers) root (hd0,0) kernel /xen.gz com1=115200,8n1 console=com1 module /vmlinuz-2.6.32-xen root=/dev/cciss/c0d0p2 ro console=hvc0 earlyprintk=xen module /initrd.img-2.6.32-xen

And finally to be able to login, I had to follow the configuration suggested here, edit the /etc/inittab and add the following lines:
1:2345:respawn:/sbin/getty 38400 hvc0 2:23:respawn:/sbin/getty 38400 tty1

And reboot. Hopefully I didn’t forget any crucial step, if one is following these notes religiously, no warranty on the accuracy, although I’ll edit it again if I missed something..

Hello Xen World!

xeninternals — Wed, 10 Feb 2010 00:09:44 +0000

Hello and welcome!

This blog is coming to life out of a personal whim and interest to learn Xen and its internals. I am mainly using this blog as sort of a public notes repository that anyone can benefit from. I will hopefully update it with as much information as possible and strive to be consistently accurate.

One of the biggest challenges I am facing, as I write this entry, is that I still haven’t got around to choose where to start. I have a few good areas I would like to tackle:

Storage
Network
Debugging

So for now, I’ll probably pick one component and go random with it. I will hopefully pick one that isn’t too tough, as the time to market for a blog is crucial…