The document summarizes troubleshooting techniques for the XenServer virtualization platform. It describes the XenServer architecture including the Xen domain 0, guest OS internals, XAPI management interface, and storage manager. It then covers various troubleshooting tools and techniques such as useful Linux commands, XenServer log files, debugging XenCenter logs, and examining the object model and CLI.
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Citrix XenServer 5.5 Troubleshooting
1.
2.
3.
4. XenServer Architecture 04/19/10 (c) 2009 Thomas Krampe SM Virtualized Hardware Hardware VT/AMD-V XAPI Drivers Storage CLI NFS iSCSI FC
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23. Summary 04/19/10 (c) 2009 Thomas Krampe xen-bugtool generates XE Server diagnostic file for submission to Technical Support Representative. xe host-bugreport-upload runs xen-bugtool and uploads the result to XenSource support. xe host-bugtool can be used to save Domain 0 state, to assist during emergency recovery Use Linux commands (network, storage, modules) to troubleshoot server install.
34. Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe Each Entry looks like [timestamp][hostname][level] [ref-1] [(num-1) unix_rpc] text-1 text2 text3…. [timestamp] Timestamp of log entry [hostname] Host name (in a pool) [level] Debug level [ref-1] ID of the task on whose behalf this is created num-1 Thread ID, local to the current process unix_rpc Unix Domain Socket (as opposed to inet_rpc=TCP) text-1 Subsystem text2/text3 more information on action performed
35. Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe xensource.log [20100209 16:58:28.688|debug|localhost|3078703|SR scanner D:dfd9ef0b708d|helpers] login done [20100209 16:58:28.921|debug|localhost|3078706 unix-RPC|SR.scan R:1ac783270499|taskhelper] task destroyed [root@localhost log]# Xenstored-access.log [20100209 16:49:05.381] D35 write data/meminfo_free 551040 [20100209 16:49:05.381] D35 write data/meminfo_total 1035760 [20100209 16:49:05.381] D35 write data/updated 1 messages Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078916|SR scanner D:dfd9ef0b708d|dispatcher] spawning a new thread to handle the current task Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078917 unix-RPC|session.slave_login D:58b671575b1e|xapi] Session.create trackid=ea252f3e59cc6f0d2d3de1345967d54b pool=true uname= is_local_superuser=true auth_user_sid= Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078919 unix-RPC|dispatch:SR.scan D:64e76e160ebc|taskhelper] task SR.scan R:98637b78c34f (uuid:6e8469dc-724d-0260-a584-44d36bca2953) created (trackid=ea252f3e59cc6f0d2d3de1345967d54b) by task D:dfd9ef0b708d
The XS installation CD runs a minimal Linux environment, so most standard Linux commands can be used to diagnose install problems. There are three “virtual terminals” available during installation, which display the install program, an interactive console and a detail event log. Use ALT + Function Keys to switch between the virtual terminals. Login to the interactive console with userid ‘root’. Technical Support may request that the command “support.sh” be run to create a diagnostic tarball to be submitted when filing a new case about an installation problem.
After logging into the interactive virtual terminal (Alt-F2), here are some example of standard Linux commands that provide information about devices. Fdisk will list all disks that can be seen as a result of the loaded storage device drivers. If a particular device driver did not load (e.g. for a RAID card), then the disks attached to that card will not be visible to “fdisk”. Ifconfig will show the network configuration of physical NICs, e.g. IP address and gateway. By default, the XE Install CD assumes the presence of a DHCP server on the local network. If no DHCP server is present, a static IP address can be configured with the command “ifconfig eth0 ip.ad.dr.es” (replacing ip.ad.dr.es with a valid static IP address on your local network). Ping can be used to verify network connectivity from the XE Server to a remote IP address.
Xen-bugtool: This is the most important command when submitting an XE Server problem report to Technical Support. It captures static and runtime state on a XenEnterprise Server, for use in diagnosing a reported problem. The compressed file can be unpacked on a Linux system and manually inspected. The support file may contain sensitive information about the customer’s operational environment and should not be posted to public mailing lists. We have also included a command line option that will run xen-bugtool remotely package up the results and upload them to an incoming support server at XenSource. This removes the need for a user to transfer the xen-bugtool output to another computer and then e-mail in. This command can be run on the XenSource server or run remotely from an administrator system.
While xen-bugtool is first and foremost a technical support diagnostic tool, the state that it records can be useful in manual emergency recovery of a critical system failure. Such a manual procedure is beyond the scope of this document or field technical staff, but it is recommended that copies of the xen-bugtool tarball be backed up to a safe location off-host from the XE Server.
XE proprietary agent uses syslog for all log messages. Can edit /etc/syslog.conf to send messages to central log server. For diagnosis of specific failures that are not easy to reproduce, Technical Support Representative may request an increase in log level, which is performed using the XE CLI (local or remote). xend.log and xend-debug.log contain messages from the Xen OSS tool stack, which does not use syslog (local logs only). All server log files are captured by the xen-bugtool inventory command (see later slide), for filing a case with Technical Support.
Administrator Console log files are used to diagnose: connection problem to XE Servers crash of the admin console Java application functional failure of admin console Java application File locations (logs, log property file, client property file) are different in Windows and Linux. For diagnosis of specific failures that are not easy to reproduce, Technical Support Representative may request an increase in log level, which is performed by editing a text file. Windows XE 3.1 clients have a per-user log file. Windows XE 3.0 clients have a central log file. Linux XE 3.0 and 3.1 clients have a per-user log file. When there is a per-user log file and that user needs to run multiple XE console instances (e.g. 3.0 and 3.1 in parallel), the startup script (Linux) or shortcut (Windows) must be manually edited to specify a per-instance log file, instead of the default per-user log file.