VMworld 2016: Troubleshooting 101 for Horizon

1. Troubleshooting 101 for Horizon
Linus Bourque, VMware, Inc
Joseph Cooper, VMware, Inc
EUC8243R
#EUC8243R

2. • This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
Disclaimer
CONFIDENTIAL 2

3. You Are Here
3
1 Introduction to Troubleshooting
2 Troubleshooting Tools
3 The Top Ten
4 Where To From Here
CONFIDENTIAL

4. Learner Objectives
4
Use a structured approach
to solve configuration and
operational problems.
Apply troubleshooting
methodology to logically
diagnose faults and improve
troubleshooting efficiency.
Document all steps taken
to resolve the problem.
1 2 3
By the end of this session, you should be able to meet the following objectives:
CONFIDENTIAL

5. 5
Definition of a System Problem
System problems arise from various sources:
Configuration
Issues
Resource
Contention
Network
Attacks
Software
Bugs or
Interoperability
Hardware
Failures or
Compatibility
A system problem is
a fault in a system, or one
of its components, that
negatively affects the
services needed for
normal production.
CONFIDENTIAL

6. VMware Severity Definitions and Response Targets
6
Severity Definition
Mission-Critical
Response
Critical 1
Production server or other mission-critical systems are
down and no workaround is immediately available.
30 minutes or less, 24x7
Major 2 Major functionality is severely impaired. 2 business hours, 12x7
Minor 3 Partial, noncritical loss of functionality of the software. 4 business hours, 12x5
Cosmetic 4 General usage questions. 8 business hours, 12x5
For a full list of VMware’s response times and a description of problem severity, go to
https://www.vmware.com/support/policies/severity.
CONFIDENTIAL

7. VMware Compatibility Guide
7CONFIDENTIAL

8. VMware Product Interoperability
8CONFIDENTIAL

9. Solution and Database Interoperability
9CONFIDENTIAL

10. Effects of a System Problem
10
These problems can affect
certain aspects of a system:
• Perceived effects or symptoms are
generally exposed and reported.
Symptoms of a system problem often
appear to be the problem itself.
• Your must look at all of the symptoms of
a system problem to determine the root
cause.
Usability Accuracy
Reliability Performance
CONFIDENTIAL

11. Adopting a Troubleshooting Process
11
Troubleshooting is a systematic approach to identifying the root cause of a problem and
defining steps to resolve the problem.
The troubleshooting process consists of several tasks:
Defining
the problem.
Identifying the
likely cause of
the problem.
Resolving
the problem.
• Identifying symptoms.
• Gathering
supplemental
information.
• Identifying possible
causes.
• Determine the root
cause.
• Identifying possible
solutions.
• Implement the
best solution.
Document the
problem, the
steps taken,
and the solution.
CONFIDENTIAL

12. Collecting Symptoms of a Problems
12
• Collecting symptoms and isolating the problem is the first step in troubleshooting.
• A single root cause often presents itself as several symptoms that users report.
• Differentiating between symptoms and the root cause of a problem is imperative.
Poor virtual desktop performance
Unable to provision or recompose
VMware View desktops
Horizon Client has a black screen
Horizon Client has a black screen
Non-optimized desktop image
Failed to set the computer’s
domain account password
PCoIP ports blocked
vRAM shortage on the View
virtual machine
Root CauseSymptom
CONFIDENTIAL

13. Gathering Additional Information
13
Can the problem be reproduced?
Provide a repeatable means to recreate
the problem and a way to validate that the
problem was resolved.
What is the scope?
Does the problem affect only one object or
multiple objects?
1
2
3
4
Was the system working before?
If so, what changed in the enviroment or
configuration?
Is the problem a known problem?
Consult references such as release notes
and VMware knowledge base articles to
determine whether the problem is
documented.
Ask questions to gather additional information to define the problem:
CONFIDENTIAL

14. Benchmarks
14
Real program
Microbenchmark
Component benchmark
Synthetic benchmarking
I/O, database benchmarks
VMware vRealize® Operations™
Various ways to quantify the problem:
CONFIDENTIAL

15. Viewing and Interpreting Diagnostic Information
15
View diagnostic messages in
the GUI or in log files.
Example: No Desktop Available
Interpret the diagnostic
messages to find the root
cause.
CONFIDENTIAL

16. Identifying Possible Causes and Taking
an Appropriate Approach
16
• A structured approach to
troubleshooting enables you to
determine the root cause quickly and
effectively.
• Based on the problem’s
characteristics, take one of the
following troubleshooting approaches:
Bottom-up
(Most General)
Approach
the cause
by halves
Hardware
(CPU, memory,
network, storage)
Virtual Desktop
ESXi Host
Horizon
Top-down
(Most Specific)
Investigate cause top down
Investigate cause bottom up
Approach the cause by halves
Horizon Client
CONFIDENTIAL

17. Determining the Root Cause
17
To determine the root cause, test your environment
and eliminate possible causes.
Example: No Desktop Available
Pool status. Lack of network connection to
VMware ESXi™ host.
Customization error. Incorrect domain credentials.
Not enough host resources (CPU, memory,
storage) are available.
Possible Causes
Physical resources are inaccessible.
Hardware
(CPU, memory,
network, storage)
Virtual Desktop
ESXi Host
Horizon
Horizon Client
CONFIDENTIAL

18. Resolving the Problem
18
After identifying the root cause,
assess the impact of the problem
on operations:
• High impact: Resolve immediately
• Medium impact: Resolve when
possible
• Low Impact: Resolve during next
maintenance window.
3
21 Identify possible solutions
and their impact on the VMware
Horizon enviroment:
• Short-term solution: Workaround
• Long-term solution: Reconfiguration
• Impact analysis: Assess the impact
of the solution on operations
Resolve the problem
by implementing the most
effective solution.
4 Document the problem
and solution.
CONFIDENTIAL

19. Example: Determining the Root Cause: Black Screen
19
Success?
Only when
using PCoIP?
Using the correct
video driver?
Success?
Yes
No
Start here:
Enough video memory
for the resolutions
Success?
Verify that the agents were
installed in the correct order
Success?
No
No
No
Yes
Yes
Yes
Yes
If you can, test possible causes by using a repeatable flow to determine the root cause.
Document all the steps that you take.
Test next
possible cause
Further
investigation
necessary
Root cause
identified
Root cause
identified
Root cause
identified
CONFIDENTIAL

20. Example: Resolving the Problem
20
In this example, assume that the root cause was not enough video memory for the
resolutions for the user’s dual-monitor setup.
Assess the impact of the problem on operations:
• Probably high impact: The problem affects any desktop that is
using dual monitors.
Identify possible solutions to resolve the problem:
• Short-term solution: Change the vRAM memory for the user’s VM.
• Long-term solution: Change the vRAM memory on the master image, and change
vRAM memory for all guests.
• Test the solution for unintended consequences
Implementing the solution should require minimal downtime.
3
2
1
CONFIDENTIAL

21. Common Troubleshooting Tools
(Manual and automated)
21

22. Methods to Run Commands
VMware vSphere®
Management Assistant :
vSphere ESXi Shell,
which includes:
• esxcli commands
• A set of esxcfg-* commands:
The esxcfg-* commands are
deprecated in favor of the newer
esxcli command-line utility.
• A set of other troubleshooting
commands.
• With the installed VMware
vSphere® Command-Line
Interface package, an
administrator can carry out
configuration and troubleshooting
tasks.
You can obtain command-line access on an ESXi host in several ways:
CONFIDENTIAL 22

23. Accessing vSphere ESXi Shell
You can access vSphere ESXi Shell in the
following ways:
23
• Enable the vSphere ESXi Shell service, either in the DCUI or in
vSphere Web Client.
• Access vSphere ESXi Shell from the DCUI by pressing Alt-F1.
• Disable the vSphere ESXi Shell service when not using it.
Local access through the
Direct Console User Interface (DCUI):
• Enable the SSH service, either in the DCUI or VMware vSphere®
Web Client.
• Use an SSH client, such as PuTTY, to access vSphere ESXi Shell.
• Disable the SSH service when not using it.
Remote access through SSH:
CONFIDENTIAL

24. vSphere Management Assistant Components
26
• Enables you to run common
system administration
commands against ESXi
hosts, such as:
• esxcli
• vmware-cmd
• vicfg-* commands
• Requires credential
connection options to a server
• Automates authentication to
the VMware vCenter Server®
system or ESXi host targets
• Relieves the user from having
to continually add login
credentials to every command
that is executed
• Facilitates unattended scripted
operations
• Entering this command
changes the command prompt,
which indicates the target
server designation.
vSphere CLI command set:
vi-fastpass authentication
component:
The vifptarget command
sets the target server for
vSphere CLI commands:
vSphere Management Assistant is a virtual appliance that includes components for
running vSphere commands:
CONFIDENTIAL

25. ESXCLI Commands
• esxcli vm namespace
• esxcli system namespace
• esxcli device namespace
• esxcli graphics namespace
• esxcli sched namespace
• esxcli vsan namespace
• esxcli esxcli namespace
• esxcli esxcli command list for a full listing
27
• esxcli namespace
• esxcli fcoe namespace
• esxcli hardware namespace
• esxcli iscsi namespace
• esxcli license namespace
• esxcli network namespace
• esxcli software namespace
• esxcli storage namespace
The esxcli command offers options in the following namespaces:
CONFIDENTIAL

26. vicfg-* Commands
vicfg-*
• You can use vicfg-* commands to manage your storage, network, and
host configuration.
• For example, you can run the vicfg-vmknic -l command to display
the IP information of your VMkernel interfaces.
32CONFIDENTIAL

27. vmware-cmd Command on vSphere Management Assistant
The vmware-cmd command is used for configuring virtual machines and
gathering information about them.
The vmware-cmd command can be run on vSphere Management Assistant
(not available on the local or remote vSphere ESXi Shell).
33CONFIDENTIAL

28. Common Windows Commands
• ipconfig[-options] target_name
• ping [-options] target_name
• tracert[-options] target_name
• pathping [-options] target_name
• nslookup[-options] host
• gpupdate[-options] target_name
• gpresult[-options]
• rsop
• Set
36
You can use the
following commands,
among others:
CONFIDENTIAL

29. 37
VDI Benchmarking
The following tools are used to
measure the scalability and
performance of the virtual
desktop infrastructure:
• VMware View Planner
• Login VSI
CONFIDENTIAL

30. Logging…
38CONFIDENTIAL

31. Location of vCenter Server Logs
40
• The vCenter Server log files are on the vCenter Server system:
– Location for vCenter Server on Windows 2008/2012:
• %ALLUSERSPROFILE%VMWarevCenterServerlogs
– Location for VMware vCenter® Server Appliance™:
• /var/log/vmware/
• Subdirectories exist for vCenter Server components, such as vCenter
Service, VMware vCenter® Single Sign-On, vCenter Inventory Service,
and vSphere Web Client.
CONFIDENTIAL

32. vCenter Server Logs Useful for Troubleshooting
The vpxd.log file is the main log file for vCenter Server.
Most vCenter Server actions are captured in vpxd.log.
41
Log File Purpose
vpxd.log The main vCenter Server log, consisting of all VMware vSphere®
Client™ and VMware vSphere® Web Services SDK connections,
internal tasks and events, and communication with the vCenter
Server agent (vpxa) on managed ESXi hosts
vpxd-profiler.log
profiler.log
scoreboard.log
Profiled metrics for operations performed in vCenter Server
cim-diag.log
vws.log
Common Information Model (CIM) monitoring information, including
communication between vCenter Server and the managed host’s
CIM interface
CONFIDENTIAL

33. Location of ESXi Host Logs
The ESXi host log files are on the ESXi host at /var/log.
42CONFIDENTIAL

34. ESXi Host Logs Useful for Troubleshooting
ESXi hosts write to multiple log files, depending on which action is being performed.
43
Log File Purpose
hostd.log Host management service logs
syslog.log
Management service initialization, watchdogs, scheduled tasks,
and DCUI use
vmkernel.log
Core VMkernel logs, including device discovery, storage and
networking device and driver events, and virtual machine
startups
vmkwarning.log
A summary of warning and alert log messages excerpted from
the VMkernel logs
vmksummary.log
A summary of ESXi host startup and shutdown, and an hourly
heartbeat with uptime, number of virtual machines running, and
service resource consumption
CONFIDENTIAL

35. Viewing Log Files in vSphere Web Client
• You can use vSphere Web Client to view and search log files on vCenter Server
systems and ESXi hosts.
• The log browser is on the Monitor tab for vCenter Server systems and ESXi hosts.
44CONFIDENTIAL

36. Using the vm-support Command to Export Data
1. Run vm-support at the command line.
• By default, the command does not require arguments.
• The command creates a ZIP file that includes the host
name and the time stamp in the filename:
– Example: esx-esxi01.vclass.local-2013-01-23--
14.09.tgz
2. Use the tar command to view the contents of the file.
• Example: tar –tzf /var/tmp/filename
49
You can use the vm-support command utility
at the local or remote vSphere ESXi Shell to
collect and compress the following types of data:
• Log files
• System status
• Configuration files
To collect and
compress useful
troubleshooting data:
CONFIDENTIAL

37. Log Locations
CONFIDENTIAL 50
Horizon Security Servers
Use support tool installed
on Connection Server or
launch support.bat from
C:Program FilesVMware
ViewServerDCT
View Composer
C:WindowsTempvmware-
viewcomposer-ga-
new.log on the linked-
clone desktop
Horizon Client
C:Users%username%AppDataLocal
VMwareVDMlogs
View Connection Server Instances
Use support tool installed
on Connection Server or
launch support.bat from
C:Program FilesVMware
ViewServerDCT
View Composer
Server
Horizon Agent
C:ProgramDataVMwareVDM
logs
Active Directory
Use the Event Viewer to view
AD-related logs.

38. Enabling Verbose Logging for View Composer
51
On the desktop through Regedit
You should only enable
View Composer verbose
logging for a short period
of time due to size
considerations. On the View Composer server through a file
2
1
You can enable verbose logging in the following ways:
CONFIDENTIAL

39. Collecting Windows Horizon Client Log Files
Log collection on Windows devices is set to basic logging by default.
You can enable advanced log collecting in Horizon Client for Windows by
running the support.bat script.
52CONFIDENTIAL

40. Collecting Horizon Client for Mac OS X Log Files
Horizon Client for Mac OS X does not have a
script to collect logs. You must manually collect
these files for diagnostic and troubleshooting
purposes or use the VMware Labs Fling.
53
1
2
3
In View 5 and later, the Horizon Client logs
on a Mac can be found in
~/Library/Logs/VMware Horizon
View Client.
You can download Horizon Collector for Mac VMware Fling at
https://labs.vmware.com/flings/horizon-collector-for-mac.
CONFIDENTIAL

41. Collecting iOS Horizon Client Log Files
You can solve most Horizon Client for iOS problems by resetting
the desktop or reinstalling the app.
54
For more information about enabling and collecting iOS logs
and files, see Using VMware Horizon Client for iOS at
https://www.vmware.com/pdf/horizon-view/horizon-client-ios-document.pdf.
iOS clients do not have a script to collect logs. You must manually
enable and collect these files for diagnostic and troubleshooting
purposes.
CONFIDENTIAL
1
2
3

42. Collecting Android Horizon Client Log Files
Log collection on Android devices is disabled by default because it will affect
the performance of the device. Disable log collection after you finish debugging
because log collection affects the performance of the device.
55
For more information about enabling and collecting iOS logs
and files, see Using VMware Horizon Client for Android at
https://www.vmware.com/pdf/horizon-view/horizon-client-android-document.pdf.
Use the options in Horizon Client for Android to enable log collection.
CONFIDENTIAL
1
2
3

43. vRealize Log Insight
Machine-generated log
data is typically massive in
scale, difficult to manage,
and overwhelming.
56
vRealize Log Insight provides a single location
to collect, store, and analyze logs at scale.
vRealize Log Insight is a log management solution
for physical, virtual, and cloud environments.
vRealize Log Insight helps monitor events and metrics.
It analyzes root causes and performs intelligent grouping,
which helps you troubleshoot efficiently.
vRealize Log Insight provides faster analytical queries
and aggregation than traditional tools, especially on larger
data sets. It adds structure to all types of unstructured log
data, so administrators can troubleshoot quickly.
CONFIDENTIAL

44. Searching and Filtering Log Events
You can search and filter log events by specifying the keywords,
time range, field operations, and so on.
57
• Event types grouping uses machine
learning to group similar events,
making root-cause analysis and
troubleshooting faster and easier.
• You can view events in context
and analyze event trends to uncover
anomalies.
• You can also save, export, rename,
share, or delete a query.
CONFIDENTIAL

45. Analyzing Logs with the Interactive Analytics Charts
• You can select different chart types to
graphically analyze log events.
• You can modify the aggregation and
grouping of query results to correlate
events and make the chart meaningful
for troubleshooting.
58
The charts provide visual representation of data and
enable you to perform visual analysis on your query results.
CONFIDENTIAL

46. Dynamic Field Extraction
When facing a large environment with many log events, you can locate
the data fields that are important to you and extract and save them.
59
Log field extraction
turns unstructured
data into structured
data that is meaningful
for troubleshooters.
CONFIDENTIAL

47. Troubleshooting Using Customized Dashboards
• vRealize Log Insight
dashboards are collections
of chart, field, table, and
query list widgets.
• You can customize
dashboards by adding,
modifying, and deleting
them.
• For example, you can save
a filtered query to your
custom dashboard by
creating a query list widget.
60CONFIDENTIAL

48. Monitoring Log Events and Sending Alerts
61
You can run specific alert queries at scheduled intervals.
When the query exceeds the preconfigured threshold, alerts can be sent to your email
or to VMware vRealize® Operations Manager™ for Horizon.
CONFIDENTIAL

49. The Top 10!
Per VMware Global Support Services, July 2016

50. Issue #1: Blackscreen
• Most well known issue and usually isolated to PCoIP protocol.
• Remember that PCoIP is made up of two parts to the connection (this is true for most
remote protocols):
– A TCP connection for authentication, machine selection, and other management activities.
The FQDN of the connection server that is entered in Horizon Client is used to establish
the connection.
– A connection to the machine
Resources (Variety of KBs written on the subject):
• Misconfiguration of connection server settings. (KB 1028332)
• Video RAM shortage on the View virtual machine. (KB 1027899)
• Incorrect video driver version installed on the View virtual machine. (KB 1029706)
• Disclaimers or login banners are being used. (KB 1016961)
• Agents were not installed in the correct order. (KB 1018158)
• Network ports are not open. (KB 1026498)
64
1
CONFIDENTIAL

51. • Order, order, order. The installation order of the Horizon Agents and other VMware
products can result in misconfigurations as can forgetting to select the right options
(e.g., Linked Clones vs Instant Clones, USB redirection, etc.)
Resources:
• Troubleshooting Agent Unreachable status in VMware Horizon View (2083535)
• Installation order of End User Computing Agents for User Environment Manager
(UEM) and App Volumes (2118048)
Issue #2: Misconfigured Agent
65
2
CONFIDENTIAL

52. Issue #3: SSL Certs
• Process is simple:
– Get Certification from CA and export
– Import and change friendly name
• Remember:
– Friendly name is vdm (not View, not Horizon, not Billy-Bob…)
– Key length: 2048 or greater (not 2047, not 1024, not thrice...)
PFX/PKCS#12 when importing into Windows cert store
Resources
• Using Microsoft Certreq to generate singed SSL certificates in VMware Horizon View
(2032400)
• Scenarios for Setting up SSL Certificates for View (Horizon View 6.0)
66
3
CONFIDENTIAL

53. • Remember this caveat: “Just because you can doesn’t mean you should!!”
• PCoIP Settings that could cause problems:
– Floor minimum
– Bandwidth maximum
– Frame rate
– Enable/disable build-to-lossless (version dependent)
– And many more!
• Test! Test! Test!
Issue #4: Network Configurations/Misconfigured GPOs
67
4
CONFIDENTIAL

54. Issue #5: Upgrading/upgrade Order
68
5
• Upgrading can be tricky. Always remember: backup, backup, backup.
• Plan your upgrade.
• Horizon then vSphere.
Resources:
1. Installation order of EUC agents for UEM/AppVolumes:
http://kb.vmware.com/kb/2118048
2. Update sequence for vSphere 6.0 and compatible VMware products:
http://kb.vmware.com/kb/2109760
CONFIDENTIAL

55. Issue #6: Forgetting to Install nVIDIA Pieces
(VIB and/or Guest Driver)
• Just installing the card isn’t enough. You need to install the ESXi VIB and Guest OS
Driver to take full advantage of those cards.
Resources to help:
1. NVIDIA GRID™ VGPU™ DEPLOYMENT GUIDE FOR VMWARE HORIZON 6.1:
https://www.vmware.com/files/pdf/products/horizon/grid-vgpu-deployment-guide.pdf
2. Installing and configuring the NVIDIA VIB on ESXi: http://kb.vmware.com/kb/2033434
69
6
CONFIDENTIAL

56. • End user experience impacts you.
• A happy user means a happy desktop admin. (Moar Warcraft; less reboots!)
Resources:
1. Fling: https://labs.vmware.com/flings/vmware-os-optimization-tool
2. Optimization Guide:
http://www.vmware.com/techpapers/2010/optimization-guide-for-windows-
7-and-windows-8-vir-10157.html
Issue #7: Unoptimized Guest OS
70
7
CONFIDENTIAL

57. • Tasks between Connection Servers and Composer get hung and nothing
provisions. Requires a View Environment (all connection servers down and then
powered on one at a time) reboot to get going.
Resources:
1. Restart order of the View environment to clear ADLDS (ADAM) synchronization
in View 4.5-6.1: https://kb.vmware.com/kb/2068381
Issue #8: Hung VCcache Issues
71
8
CONFIDENTIAL

58. • Sometimes VMs go missing because they are being managed outside of Horizon.
When it’s removed as part of the activities of the vSphere Administrator, Horizon is
often unaware that this and thus labels it as missing.
Resources:
• Manually deleting linked clones or stale virtual desktop entries from the View Composer
database in VMware View Manager and VMware Horizon View:
https://kb.vmware.com/kb/2015112
• Removing invalid linked clone entries automatically using the ViewDBChk tool in VMware
Horizon View 5.3 and later versions: https://kb.vmware.om/kb/2118050
Issue #9: Missing/Orphaned or Stuck in Deleing VM
72
9
CONFIDENTIAL

59. • Customization failures can be frustrating. It’s important to make sure you account
for each of the below, otherwise no provisioning will occur.
Resources:
• VMXNET 3 adapter on Parent Image for linked clones misconfigured (MS
KB2550978 & VMware KB: https://kb.vmware.com/kb/2007319)
• not enough ports on vSwitch/dvPortGroup
• typos on passwords.
Issue #10: Customization Failures
73
10
CONFIDENTIAL

60. Where to from Here?
74

61. Learn More
Visit us at VMworld
• Education & Certification Lounge: Mandalay Bay, VMvillage
• Certification Exam Center: Mandalay Bay, North Convention Center, Mariner’s A&B
Visit our website
• VMware training and certification: www.vmware.com/education
CONFIDENTIAL 75

62. Q&A
For questions
Linus at lbourque@vmware.com OR @linusbourque (Twitter)
Joe at cooperj@vmware.com OR @joeicooper (Twitter)
76
?
CONFIDENTIAL

65. Troubleshooting 101 for Horizon
Linus Bourque, VMware, Inc
Joseph Cooper, VMware, Inc
EUC8243R
#EUC8243R