Here's implementation experience about virtualization, Cloud Computing using open source Libvirt.

1. sean666666@gmail.com P1
Control Hypervisor via Libvirt
Sean Chang
2011/4

2. sean666666@gmail.com P2
大綱
• 為什麼要用 libvirt
• Lifecycle of VM
• 順帶一提 virsh
• 順帶一提 xm
• 進入重點 libvirt
• Libvirt 使用範例
• Libvirt 使用經驗
• Libvirt 議題
• DEMO

3. sean666666@gmail.com P3
管理 hypervisor 的方法
• 下指令
o xm
o virsh
• 自己寫程式
o Libvirt
o A virtualization API
o Open source
o 可程式化

4. sean666666@gmail.com P4
飲水思源
了解VM(Guest domain)的生命週期

5. sean666666@gmail.com P5
Guest domain status
State Description
Undefine
d
This is the baseline state. An undefined guest domain has not been
defined or created in any way.
Defined A defined guest domain has been defined but is not running. This
state could also be described as Stopped.
Running A running guest domain is defined and being executed on a hypervisor.
Paused A paused guest domain is in a suspended state from the Running
state. Its memory image has been temporarily stored, and it can be
resumed to the Running state without the guest domain operating
system being aware it was ever suspended.
Saved A saved domain has had its memory image, as captured in the
Paused state, saved to persistent storage. It can be restored to the
Running state without the guest domain operating system being
aware it was ever suspended.

6. sean666666@gmail.com P6
Guest domain lifecycle

7. sean666666@gmail.com P7
VM(Guest domain) status

8. sean666666@gmail.com P8
VM Lifecycle
What does
Libvirt actully
do？

9. sean666666@gmail.com P9
工欲善其事
必先利其器

10. sean666666@gmail.com P10
xm
• Xen management user
interface
• The xm program is the main
interface for managing Xen
guest domains. The program
can be used to create, pause,
and shutdown domains. It can
also be used to list current
domains, enable or pin
VCPUs, and attach or detach
virtual block devices.
• Example
o xm create Fedora12
o xm start Fedora12
o xm help
o xm list
o xm mem-set {domain-id,
mem}
o xm migrate {domain-id,
host-ip}
o etc…沒事多玩指令，多
玩指令沒事

11. sean666666@gmail.com P11
virsh(1/2)
• virsh 是​個​用​來​管​理​客​座​端​和​
hypervisor（管​理​程​序​）的​指​令​
列​介​面​工​具​。
• 不過說穿了，這個指令列也是
用 libvirt 作為底層函式庫寫出
來的。
指​令​ Description
help 印​出​基​本​協​助​資​訊​。​
list 列​出​所​有​客​座​端​。​
dumpxml 輸​出​客​座​端​的​ XML 配​置​檔​案​。​
create 由​一​個​ XML 配​置​檔​案​來​建​立​客​座​端​並​啟​用​新​
的​客​座​端​。​
start 啟​用​一​個​未​啟​用​中​的​客​座​端​。​
destroy 強​制​客​座​端​停​下​。​
define 輸​出​客​座​端​的​ XML 配​置​檔​案​。​
domid 顯​示​客​座​端​的​ ID。​
domuuid 顯​示​客​座​端​的​ UUID。​
dominfo 顯​示​客​座​端​的​相​關​資​訊​。​
domname 顯​示​客​座​端​的​名​稱​。​
domstate 顯​示​客​座​端​的​狀​態​。​
quit 退​出​互​動​式​終​端​機​。​
reboot 重​新​啟​動​客​座​端​。​
restore 恢​復​之​前​已​儲​存​於​一​個​檔​案​中​的​客​座​端​。​
resume 恢​復​一​個​暫​停​中​的​客​座​端​。​
save 將​客​座​端​目​前​的​狀​態​儲​存​至​一​個​檔​案​中​。​
shutdown 正​常​地​停​用​客​座​端​。​
suspend 將​客​座​端​暫​停​。​
undefine 刪​除​與​某​個​客​座​端​有​關​聯​的​所​有​檔​案​。​
migrate 將​客​座​端​遷​移​至​另​一​部​主​機​上​。​

12. sean666666@gmail.com P12
virsh(2/2)
• Example
o # virsh connect {hostname
OR URL}
o # virsh create
configuration_file.xml
o # virsh suspend {domain-
id, domain-name or
domain-uuid}
o etc…沒事多玩指令，多
玩指令沒事
指​令​ Description
setmem 為​客​座​端​設​置​分​配​記​憶​體​。​
setmaxmem 設​定​ hypervisor 所​能​使​用​的​記​憶​體​最​大​限​制​。​
setvcpus 更​改​分​配​給​客​座​端​的​虛​擬​ CPU 數​量​。​
vcpuinfo 顯​示​有​關​於​客​座​端​的​虛​擬​ CPU 資​訊​。​
vcpupin 控​制​客​座​端​的​虛​擬​ CPU 相​似​性​。​
domblkstat 顯​示​正​在​執​行​中​的​客​座​端​的​區​塊​裝​置​數​據​。​
domifstat 顯​示​執​行​中​的​客​座​端​的​網​路​介​面​卡​數​據​。​
attach-device 透​過​使​用​某​個​ XML 檔​案​中​的​裝​置​定​義​來​將​裝​置​附​加​至​客​座​端​上​。​
attach-disk 附​加​新​磁​碟​裝​置​至​客​座​端​。​
attach-interface 附​加​新​網​路​介​面​卡​至​客​座​端​。​
detach-device 將​裝​置​從​客​座​端​上​分​離​，接​受​與​ attach-device 指​令​相​同​類​型​的​
XML 描​述​。​
detach-disk 將​磁​碟​裝​置​由​客​座​端​上​移​除​。​
detach-interface 將​網​路​介​面​卡​由​客​座​端​上​移​除​。​

13. sean666666@gmail.com P13
不是不寫程式
不過時後也已經到了

14. sean666666@gmail.com P14
Libvirt(http://libvirt.org/index.html)
• A toolkit to interact with the virtualization
capabilities of recent versions of Linux (and
other OSes), see our project goals for details.
• Free software available under the GNU Lesser
General Public License.(LGPL)
• A long term stable C API
• A set of bindings for common languages.(Java、
Python…etc)
• Latest release: 0.8.8: Feb 17 2011

15. sean666666@gmail.com P15
Terminology and goals
Xen
…
一般稱呼 Libvirt的術語 解釋
VM Domain An instance of an operating system (or
subsystem in the case of container
virtualization) running on a virtualized machine
provided by the hypervisor
Xen Hypervisor A layer of software allowing to virtualize a
node in a set of virtual machines with possibly
different configurations than the node itself
Physical
machine
Node A single physical machine

16. sean666666@gmail.com P16
Libvirt drivers
• Hypervisor drivers
o LXC - Linux Containers
o OpenVZ
o QEMU
o Test - Used for testing
o UML - User Mode Linux
o VirtualBox
o VMware ESX
o VMware
Workstation/Player
o Xen
o 支援Multi-Vendor(表面上…)
• Storage drivers
o Directory backend
o Local filesystem backend
o Network filesystem backend
o Logical Volume Manager
(LVM) backend
o Disk backend
o iSCSI backend
o SCSI backend
o Multipath backend

17. sean666666@gmail.com P17
API concepts(1/2)
Objects exposed

18. sean666666@gmail.com P18
API concepts(2/2)
Libvirt drivers
就是OS
或是應用程式
API呼叫
deamon
remotelocal

19. sean666666@gmail.com P19
Xen config V.S. XML
• VM的設定檔
o For Xen
o For libvirt

20. sean666666@gmail.com P20
細述 XML
unique
全虛擬
最大的實體資源分配
網路採bridge方式對外連通
這張虛擬網卡
掛在node的xenbr0
VNC port

21. sean666666@gmail.com P21
Python example – 基本VM操作

22. sean666666@gmail.com P22
Python example – 掛載/卸載磁碟
已經準備好
掛載的image空間

23. sean666666@gmail.com P23
我不入地獄 誰…
往下挖

24. sean666666@gmail.com P24
認證
• 呼叫遠端的libvirtd需要先經過認證，認證方
式如下
• ssh
o ssh username root: xen+ssh://root@node.example.com/
o 然後輸入帳號密碼，缺點，如果程式要全自動化會被卡在這裡。
• authorized key
o 製作Public / private key
o #ssh-keygen -t rsa
o 放在相對應的目錄: /root/.ssh/

25. sean666666@gmail.com P25
Debug / loggin
• Log message
o Generated at runtime by the libvirt code
o Timestamp, a category, a priority level, function name and line number indicating
where the message originated from, and a formatted message.
• Priority levels
o 1 (or debug) - log all messages
o 2 (or info) - log all non-debugging information
o 3 (or warn) - log only warnings and errors - this is the default
o 4 (or error) - log only errors
• Filter
o x:name
o 1:qemu
o 1:qemu 4:remote
• Log output
o 3:syslog:libvirtd 1:file:/tmp/libvirt.log

26. sean666666@gmail.com P26
虛擬裝置也支援熱插拔嗎?
• 半虛擬技術支援在不開機的情況下，CPU、MEM、DISK
都可以做調整。
• 全虛擬CPU、MEM要調整一定得重開機。
• 全虛擬DISK熱插拔則視VM的kernel版本而定。
• The Xen device model is more or less unchanged in the pv-ops
kernel. Converting a driver from the xen-unstable or 2.6.18-xen tree
should mostly be a matter of getting it to compile. There have been
changes in the Linux device model between 2.6.18 and 2.6.26, so
converting a driver will mostly be a matter of forward-porting to the
new kernel, rather than any Xen specific issues.
• Kernel version > 2.6.26 可能比較穩定有機會實現全虛擬的硬碟熱插
拔。

27. sean666666@gmail.com P27
Xen disk kinds and supporting
• Disk type
o IDE
o Virtual disk
o iSCSI
o USB Disk
• 實測熱插拔
xen 4.0 + 2.6.32.11
/ Dell
Storage
IDE Virtual SCSI USB Disk
centos5.4final
Kernel 2.6.18
× ○ × ×
RH6beta
Kernel 2.6.18
× × × ×

28. sean666666@gmail.com P28
API Issues
• create 跟 define 有什麼不同?
o 都是用來創建VM的API
o 差別在生命週期不完全相同
o create: 創建VM之外，順便把VM開起來，shutdown後該VM自動
從hypervisor卸除。
o define: 僅創建VM，須靠start API把VM開起來，但永遠定義在
hypervisor裡。甚至連migrate後也在。
• shutdown 跟 destroy 有什麼不同?
o 都是要把VM的狀態從running改成stop
o Shutdown：就是一般作業系統的關機。
o Destroy：以實體主機來看，就是直接把插頭拔掉。

29. sean666666@gmail.com P29
Development trick(1/3) - Problem
• Libvirt 在偵測 xen 上的VM實際狀態不穩定
• 連續下五十次偵測VM status的結果
[2, 665600L, 444416L, 1, 178818312669L][2, 665600L, 444416L, 1, 178818786889L][2, 665600L, 444416L, 1, 178819270686L][2
665600L, 444416L, 1, 178819782827L][2, 665600L, 444416L, 1, 178820286716L][0, 665600L, 444416L, 1, 178825799216L][2,
665600L, 444416L, 1, 178826484955L][2, 665600L, 444416L, 1, 178827054507L][2, 665600L, 444416L, 1, 178827646602L][2,
665600L, 444416L, 1, 178828162295L][2, 665600L, 444416L, 1, 178828718993L][2, 665600L, 444416L, 1, 178830313690L][2,
665600L, 444416L, 1, 178831831052L][2, 665600L, 444416L, 1, 178832305724L][2, 665600L, 444416L, 1, 178832859023L][2,
665600L, 444416L, 1, 178833339605L][2, 665600L, 444416L, 1, 178833848167L][2, 665600L, 444416L, 1, 178834372305L][2,
665600L, 444416L, 1, 178834895557L][2, 665600L, 444416L, 1, 178835386865L][2, 665600L, 444416L, 1, 178835922103L][2,
665600L, 444416L, 1, 178836494779L][2, 665600L, 444416L, 1, 178836973848L][2, 665600L, 444416L, 1, 178837452908L][2,
665600L, 444416L, 1, 178838008290L][2, 665600L, 444416L, 1, 178838553784L][2, 665600L, 444416L, 1, 178839024984L][2,
665600L, 444416L, 1, 178839462699L][2, 665600L, 444416L, 1, 178839987321L][2, 665600L, 444416L, 1, 178840510368L][2,
665600L, 444416L, 1, 178841185524L][2, 665600L, 444416L, 1, 178841873368L][2, 665600L, 444416L, 1, 178842393659L][2,
665600L, 444416L, 1, 178842910878L][2, 665600L, 444416L, 1, 178843454749L][2, 665600L, 444416L, 1, 178843947320L][2,
665600L, 444416L, 1, 178844427452L][2, 665600L, 444416L, 1, 178844942837L][2, 665600L, 444416L, 1, 178845472009L][2,
665600L, 444416L, 1, 178846036556L][2, 665600L, 444416L, 1, 178846502921L][2, 665600L, 444416L, 1, 178847082404L][2,
665600L, 444416L, 1, 178847612475L][2, 665600L, 444416L, 1, 178848158639L][2, 665600L, 444416L, 1, 178848527989L][2,
665600L, 444416L, 1, 178849177317L][2, 665600L, 444416L, 1, 178849704749L][2, 665600L, 444416L, 1, 178850274864L][2,
665600L, 444416L, 1, 178850717630L][2, 665600L, 444416L, 1, 178851307315L][2, 665600L, 444416L, 1, 178852079044L][2,
665600L, 444416L, 1, 178852861964L][2, 665600L, 444416L, 1, 178854056260L][2, 665600L, 444416L, 1, 178854907434L][2,
665600L, 444416L, 1, 178856457345L][2, 665600L, 444416L, 1, 178856918293L][2, 665600L, 444416L, 1, 178857394787L][1,
665600L, 444416L, 1, 178857716163L][2, 665600L, 444416L, 1, 178858177594L][2, 665600L, 444416L, 1, 178858773954L][2,
665600L, 444416L, 1, 178859309076L][2, 665600L, 444416L, 1, 178859897892L][2, 665600L, 444416L, 1, 178860400666L][2,
665600L, 444416L, 1, 178860937146L][2, 665600L, 444416L, 1, 178861383848L][1, 665600L, 444416L, 1, 178861889219L][2,
665600L, 444416L, 1, 178862333614L][2, 665600L, 444416L, 1, 178862848250L][2, 665600L, 444416L, 1, 178863325489L][2,
665600L, 444416L, 1, 178863674106L][2, 665600L, 444416L, 1, 178864153743L][2, 665600L, 444416L, 1, 178864647934L][2,
665600L, 444416L, 1, 178864970657L][2, 665600L, 444416L, 1, 178865558487L][2, 665600L, 444416L, 1, 178865890459L][2,
665600L, 444416L, 1, 178866309383L][2, 665600L, 444416L, 1, 178866793058L][2, 665600L, 444416L, 1, 178867306348L][2,

30. sean666666@gmail.com P30
Development trick(2/3) - Solution
• 如果狀態不穩，我們無法取得VM確實的狀態。
• 參考 VirtManager 的解法，利用以下API的特性
o 找出可以幫忙的 libvirt API。
o listDomainsID(): 列出所有xen上在跑的domain。
o 因為只有running的VM狀態不穩，所以利用listDomianID()這個
API把這些可能不穩的VM先抓出來，在把0,1,2等狀態都歸納為
runnung。

31. sean666666@gmail.com P31
Development trick(3/3) - Code
31
1
2
3
4
5

32. sean666666@gmail.com P32
libvirt 處理 Xen 與 KVM 的行為
• 不同的 hypervisor，libvirt的行為也會有所差異
• 以下以Xen跟KVM為例：

33. sean666666@gmail.com P33
開發心得
• 多用python的特有的API查詢指令help()、dir()來
幫助自己掌握libvirt的版況
• 不要太期待libvirt可以幫你做太多很強大的事情，
官網很多只定義好介面，尚未實作。舉例:
• 要熟悉XML config
• 要熟悉hypervisor的行為

34. sean666666@gmail.com P34
參考資料
• http://docs.fedoraproject.org/zh-
TW/Fedora/12/html/Virtualization_Guide/chap-Virtualization_Guide-
Managing_guests_with_virsh.html
• http://linux.die.net/man/1/xm
• http://libvirt.org/index.html
• http://libvirt.org/sources/virshcmdref/html/
• http://wiki.xensource.com/xenwiki/XenParavirtOps

35. sean666666@gmail.com P35
DEMO
• 準備 VM
o define
o create
• 操作 VM
o start
o suspend
o resume
o reboot
o shutdown
• 添加/卸載設備
o attach
o detach
• 取得 VM
o lookupByXXX
• 跨主機(下回分曉…)
o virtManager 介紹
o migration
o live migration

36. sean666666@gmail.com P36
Thank you
• 末記
o 這份資料的技術實作於2010~2011，如非最新資料，敬
請見諒
• End
o The technology information was implemented in 2010 ~
2011, sorry for no update on time.