Citrix Troubleshooting 101: How to Resolve and Prevent Business-Impacting Citrix Problems

© eG Innovations, Inc. | www.eginnovations.com
Citrix Troubleshooting 101
How to Resolve and Prevent Business-Impacting Citrix Problems
─ An eG Innovations Webinar ─

Know Your Speaker
Vinod Mohan
Sr. Product Marketing Manager
eG Innovations
Moderator
George Spiers
Citrix Technology Professional (CTP)
and EUC Architect
Speakers
Srinivas Ramanathan
Founder and CEO
eG Innovations
Over 15 years of experience working with Citrix
technologies.
srinivas@eginnovations.com | www.eginnovations.com
Citrix Certified Expert and Microsoft Solutions
Expert. One of the 50 CTPs in the world.
george@jgspiers.com | www.jgspiers.com

Agenda
• Understanding the Impact of Poor Citrix Performance
• 3 Key Steps to Troubleshoot Citrix Problems
─ Determining the Scope, Magnitude, and Source of the Problem
• Troubleshooting Tips from the Trenches
─ Citrix tiers, supporting infrastructure, client-side problems
• Tips to Optimize your Citrix Deployment
• How Automated Monitoring Aids in Troubleshooting

Citrix is Performance Sensitive
48%
45%
7%
The most performance sensitive service
Similar in performance demands as other services
Not that performance sensitive
93% believe that Citrix is one of the
performance sensitive services in
enterprise networks.
Source: 2016 Citrix Performance Survey
https://www.eginnovations.com/solutions/view-resources?whitepaper=36
• Many components are involved
in Citrix delivery
• Too many interdependencies
between tiers
• Any fault or failure cascades into
a bigger problem that affects
performance
• Performance relies also on
supporting infrastructure

What Happens When Users Complain that “Citrix is Slow”?
Users start to complain that Citrix has suddenly become very slow
Helpdesk starts getting user complaints
Helpdesk alerts the Citrix administration team
Citrix admins begin investigating the cause for slowness
• Loss of productivity to users • Pressure on IT to fix the problem soon• Customer satisfaction is affected

Slowness is Expensive!
Loss of Productivity
Application slowness for 3 hours; User
productivity loss: $200/hr X 3 hours X
1000 users = $600,000 per incident
Operations Cost
Expert IT staff involved in troubleshooting
costing $200/hr X 10 staff X 3hrs = $6,000
per incident

Slowness is Expensive!
Unforeseen Capital Expense
Projection
50 users/
server
Actual
25 users/
server
Double the number of servers
needed; $50,000 per server,
5 servers = $250,000 additional cost
Failure of Transformation Project
Performance
slower than
physical
desktops
Users unhappy; want physical
desktops back; Cost increase: $600
per user * 1000 users = $600,000

Performance Must Be Considered at Every Stage of
the Citrix Deployment Lifecycle
Design
Test &
Deploy
Operate
&
Maintain
Trouble-
shoot
Optimize
Rectify Monitor
Trouble-
shoot

Troubleshooting Tips
from the Trenches
Solve Top Citrix Problems

Common Citrix Problems
Logon is
slow
Application
is slow to
launch
Keystrokes
don’t show
up on time
Multimedia
playback is
choppy
User
ComplaintsPrinting
doesn’t
work
Session gets
disconnected
Where to begin troubleshooting?

3 Key Steps to Troubleshooting
#1 Determine SCOPE of the problem
#2 Determine MAGNITUDE of the problem
#3 Determine SOURCE of the problem

Determining The SCOPE of The Problem
Troubleshooting must be problem specific
• For e.g.: Citrix logon slowness will have different reasons and
troubleshooting fixes from slow application access
Is it just one task that the user is facing an issue with,
or are all tasks slow?
Logon
Session
Setup
App
Launch
App
Usage
Remote
Access
Printing

Determining The MAGNITUDE of the Problem
One user is impacted Several users are impacted
Less impact Moderate impact
All users are impacted
Critical impact
• Which user? Where? When?
• Is there a problem with the
user's network?
• What version of Citrix
Workspace app does the
user have?
• Problem with a specific
application or all
applications?
• Is there any commonality between
these users?
─ Same server?
─ Same apps?
─ Same domain?
─ Same location?
─ Same Receiver version?
• Is there DC network latency?
• Is the Citrix Provisioning server or
license server having an issue?
• Are StoreFront and Citrix ADC
working fine?
• Is the virtual environment
experiencing a bottleneck?
• Is there a problem in the SAN
environment?

Determining The SOURCE of the Problem
StoreFront Delivery Controller
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Active DirectorySQL Server
Citrix Virtual Apps
Server
Hypervisor
Internet/
WAN
Storage
Client Side
Citrix Delivery
Infrastructure
Supporting
Infrastructure
Citrix End Users
Firewall
Corporate
Network
Client
Network
Corporate
Network

Best Practices for Troubleshooting Common Citrix Problems
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Citrix Virtual Apps
Server
Hypervisor
Internet/
WAN
Storage
Citrix Delivery
Infrastructure
Citrix End Users
Firewall
Corporate
Network
Client
Network
 Slow Logon
 Slow App Launch
 Session Disconnects
 Frozen Sessions
 Session Latency

Slow Logon is the #1 Problem for Citrix Admins
Source: 2018 Citrix Migration Survey
www.eginnovations.com/Citrix-Migration-Trends
• In an environment that averages 500 user
logons per business day, an average 10-
second increase in logon times over a 5-day
week period results in over 6 hours of lost
productivity!
• Slow logons are ranked the #1 problem for
Citrix admins
#1 Slow Logon

Investigating Citrix Logon Slowness
This will help isolate which part of the infrastructure to go look for issues
Identify which step of the logon process is taking more time to execute
#1 Slow Logon

Monitoring Citrix Logons: No Longer a Science
Logon Simulation
#1 Slow Logon
Simulated logon via
Citrix Gateway
Applications
enumerated on the
StoreFront console
Launching application
on StoreFront
Citrix Director
Real User Monitoring

Tips to Avoid Logon Slowness
• If using Citrix Profile Management, make sure your storage location is configured with high speed
access to your desktops and applications
• Make sure that you have enough resource available to cope with demand. Always size for peak load.
• Optimise Group Policies. E.g., use Citrix Workspace Environment Management (WEM) to map printers,
drives and other actions after logon.
• Make use of Citrix Profile Management policy settings such as:
─ Virtual Apps Optimization
─ Profile streaming
─ Large file handling
─ Logon exclusion check (keeps profile size minimised)
• Optimise your images – Disable unneeded services, scheduled tasks, and tweak the OS for better
performance
#1 Slow Logon
─ https://www.jgspiers.com/windows-server-2012-r2-optimisation-script/ - for Windows Server 2012 R2
─ https://www.jgspiers.com/windows-server-2016-optimisation-script/ - for Windows Server 2016
─ https://www.jgspiers.com/windows-10-1709-optimisation-script/ - for Windows 10

Citrix Workspace Environment Management (WEM)
With WEM
Without WEM
All the processing is
done after logon
Logon speed is
improved
All the processing steps happen before logon
#1 Slow Logon

Large File Handling
• Large files can be created as symbolic links
• Avoids synchronizing the file down to the VDA
• Supported on Virtual Apps and Desktops 7.16+
#1 Slow Logon

Investigating Slow Application Launch
• Is the launch for only one application slow, or for all applications?
• Is slowness affecting application launch on one server or all servers?
• Check versions of applications in use: are they uniform?
• Compare workloads of servers in use
• Published apps may involve user session establishment. So brokering, GPOs etc.
can cause slowness.
• Look for add-ins/plugins (e.g., Outlook, Browser, etc.) that may be causing app
launch slowness
#2 Slow App/Desktop Launch

Monitoring Slow Application Launch
eG Enterprise
• Application availability and launch with
simulation
─ App Probing using Director
─ Logon simulation tools
• Monitor actual application launch
times (using Windows APIs)

Tips to Avoid Slow Application/Desktop Launch
• Optimise images
• Make use of session prelaunch for published applications
• Make sure VDAs are spec’d appropriately and not overloaded, especially during
peak times
• Make sure you have enough VDAs powered on and available to handle requests
• Make sure you have enough DDCs to handle requests
• Make sure you have enough StoreFront servers to deal with requests
• Ensure your Citrix profile/3rd party profile solution is optimized & not a
bottleneck
• Keep your Virtual Apps and Desktops deployment up-to-date
• Make use of Zones to broker users to desktops/applications closest to them if
you have multiple-datacentres

Implement Brokering Improvements (7.11+)
Improvements to Latency and SQL Blocking Query
Before XA/XD 7.11
(90ms latency)
XA/XD 7.11+
(90ms latency)
Brokering requests per sec 3.7 12.6
Time to launch 10k users 44m 55s 13m 10s
Quicker brokering = quicker launch times!

Use Session Prelaunch (7.6+)
• Session is pre-prepared when user logs on to StoreFront
• When user clicks to launch application, the majority―if not entire―logon process
would have already completed in background
• Not always will a user launch a Virtual Apps
application straight away, they may check
emails. In the meantime, let Virtual Apps
prepare the session in the background.
• Keep in mind that licenses are used, so you
can set a time limit to end pre-launched
sessions if they are not used after a period of
time.

Investigating User Session Disconnects
• Follow a process of elimination to determine if:
─ This happens to particular users, or all users
─ Happens when using specific desktops or published applications
─ Happens to users from particular offices and so on
#3 User Session Disconnects
• Check Event Logs on the VDAs around the time when disconnects happened
• Check Citrix Receiver/Workspace app logs
• Run network connectivity tests for affected users
• Run tests on VDAs to make sure they are not losing network connection briefly
• Check that firewalls and antivirus products are not interfering with ICA traffic

Tips to Avoid User Session Disconnects
• Make sure your environment is on the latest updates/versions:
─ Citrix Receiver/Workspace app should be up to date
─ VDAs are always patched with the latest releases from Microsoft
─ Update Hypervisor tools such as VMware Tools
─ Ensure that Citrix Virtual Apps and Desktops is kept up to date
• Ensure network monitoring is in place and the networking fabric is kept up to
date to ensure optimal performance and stability
• Keep a record of what has changed in the environment so you can roll-back
easily if disconnects happen suddenly after a change

Enable Receiver Logging
https://www.jgspiers.com/citrix-receiver-windows-logging/
General Receiver logging:
Set ReceiverVerboseTracingEnabled to 1 under
HKLMSOFTWARECitrix for 32-bit machines or
HKLMSOFTWAREWOW6432NodeCitrix for 64-bit machines

Investigating Frozen Sessions
• Are all users on the same VDA affected or just one?
• Does exiting Receiver, re-launching and connecting back to the
session work?
• Can you force the user off via Director, or by killing processes via
Task Manager on the VDA or Director?
• Is a process thrashing CPU, memory, or disk on the VDA(s)?
• Does the VDA need to be drained and rebooted to clear the frozen
session?
#4 Frozen Sessions

Tips to Avoid Frozen Sessions
• Make sure your environment is on the latest updates/versions:
─ Citrix Receiver/Workspace app should be up to date
─ VDAs are always patched with the latest releases from Microsoft
─ Update Hypervisor tools such as VMware Tools
─ Hypervisors should have latest patches
─ Ensure that Citrix Virtual Apps and Desktops is kept up to date
• Make sure that VDAs are spec’d appropriately if the session froze due to CPU
thrashing or lack of RAM. WEM can help here
• If using Citrix Provisioning (PVS), make sure your Write Cache set to “Cache on
device RAM with overflow to HDD”
─ Ensure the Write Cache for PVS and MCS is large enough to cope with writes
• Follow Citrix recommendations for Antivirus exclusions:
─ https://support.citrix.com/article/CTX127030
#4 Frozen Sessions

Hide Citrix Sessions
https://www.jgspiers.com/user-stuck-citrix-desktop-force-log-off/
Session frozen on VDA, but you
cannot reboot immediately
(Virtual Apps)?
#4 Frozen Sessions

Investigating Internal or External Session Latency
• If only external users are facing issues, but not internal users:
─ Are the Citrix ADCs (NetScalers) under load?
─ Is latency caused by routing or issues in the DMZ?
─ Could user's WAN connection be a problem?
#5 Session Latency
• If both internal and external users are affected, check StoreFront,
other Citrix components, and the datacentre network
StoreFront
Citrix ADC
Internet/
WAN
Firewall
WAN Latency DC Latency
Internal
User
External User

Monitoring Session Latency
• Use Citrix ADM (formerly NetScaler MAS) to get DC latency and WAN latency
metrics obtained by analysing the traffic flowing through Citrix ADC
#5 Session Latency
Customers not on Citrix
Gateway Premium licensing
can rely on AppFlow
monitoring tools from vendors
(eG Innovations, others)

Tips to Avoid Session Latency
• Monitor resource usage on your VDAs
─ Detect users/applications taking up too much resources proactively
• Monitor browser activity
─ Many monitoring tools can show URLs accessed by users
─ Limit browser access to web sites required for business activities
• For remote access, make sure Citrix ADC/Gateway (formerly NetScaler) is
configured with appropriate licenses that support throughput requirements
• Make sure the internal network is performing as should
─ Unforeseen problems can occur – e.g., with network teaming
#5 Session Latency

Enable Citrix HDX Adaptive Transport (EDT Protocol)
to Improve Latency and Throughput
A copy of a 45MB file over a 200ms line is still an incredible
36 seconds faster than the 100ms TCP copy.
#5 Session Latency

What Tools Citrix Offers for Troubleshooting
Citrix Smart Tools
• You can schedule a range of health
checks against your environment for:
– StoreFront health checks
– Citrix Site health checks
– Citrix Provisioning health checks
– License server health checks
Limitation: Does not provide in-depth visibility to diagnose issues. Does not help with
problems outside the Citrix stack.

What Tools Citrix Offers for Troubleshooting
Citrix Director
• Web-based, helpdesk-focused tool that
allows you to (based on licensing)
• Manage active sessions (remote control,
disconnect, logoff, stop processes)
• View logon times to VDAs (average logon
times across Delivery Groups, logon
times per user)
• View session establishment failures (why
a session failed to connect and a reason)
• Create alerts for long logon time averages, high session failure count and more
• View historical data and run reports on sessions, connections, logon times and more
Limitation: Functionality is restricted by Citrix Virtual Apps and Desktops licensing.
Does not help with problems outside the Citrix stack.

Best Practices for Troubleshooting Problems Originating
from the Client Side
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Citrix Virtual Apps
Server
Hypervisor
Internet/
WAN
Storage
Client Side
Citrix End Users
Firewall
Corporate
Network
Client
Network

Investigating Issues on The Client Side
• Is the client terminal running the latest version of Citrix Workspace app (Citrix
Receiver)?
• Is there anti-virus running on the client workstation that may be interfering with
ICA traffic?
• Is the browser that the user is using to access Citrix session slow?
─ Are there browser add-ons such as proxies or content filtering software that is blocking
or preventing ICA from optimally performing?
• Is there any user account lockout issue in the Active Directory?
• Is the client workstation having any problems?
─ A background application/process taking too much resources
Client-side Issues

Investigating Issues on The Client Side
• How is the network connectivity on the client side?
─ Is there any firewall setting on their workstation that’s affecting connectivity?
─ What is the network latency? Bandwidth?
─ Citrix APIs/counters provide latency and bandwidth information for virtual channels
Client-side Issues

Tips to Avoid Issues on the Client Side
• Make sure to have the most recent version of Citrix Workspace app (Receiver)
installed on the client terminal
• Test network connection strength from the user terminal to Citrix Site
─ Citrix network connection quality indicator tool can be used
(https://support.citrix.com/article/CTX220774)
• Check the user’s profile and see if there’s any issue, such as a large profile size
• Check Active Directory to see if the user account is configured correctly and does
not have any non-essential Group Policy settings applied or logon scripts
• Check for memory, disk and storage issues on the client workstation
• Make sure the browser used to access StoreFront or Citrix Gateway is of the latest
version and is not having any issues
Client-side Issues

Best Practices for Troubleshooting Problems in the
Infrastructure Supporting Citrix Deployments
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Citrix Virtual
Apps Server
Hypervisor
Internet/
WAN
Storage
Supporting
Infrastructure
Citrix End Users
Firewall
Corporate
Network
Client
Network

Investigating Issues with the Supporting Infra
• Network
─ Don’t assume, monitor! May not just be your routers, switches. Network card issues can cause problems.
─ Speed negotiation problems, driver issues can also affect performance
• Active Directory
─ Sites and Services may not be properly configured if you have multiple sites across your organisation
─ Logon scripts may be configured on user accounts through ADUC which can be bad for logon times
─ DNS if incorrectly configured could cause problems with applications and services
─ Kerberos authentication breaks if there is a five minute or more difference between the server and AD
• Hypervisor
─ Monitor resource usage – look for bottlenecks; Follow vendor-specified best practices
─ VM ready time, memory ballooning and swapping, disk queueing and aborts
─ Monitor resources used by other VMs
• Storage
─ Detect storage failures (failures remain hidden in RAID environments)
─ Monitor LUN-level I/O activity
• Cloud
─ Ensure you are not reaching capacity limits
─ Look for bandwidth / connection problems to the cloud
Infrastructure Issues

Tips to Avoid Issues in the Supporting Infra
Infrastructure Issues
• Make sure Group Policy is configured to apply only relevant policies. You can move a lot
of Group Policy settings off to WEM if you are entitled
• Size your hypervisors to cope with peak load. That is, the maximum number of sessions
you need to support concurrently with a 10-20% buffer.
─ Also factor in host failure or datacentre outage and how that will affect remaining healthy
infrastructure
• If using file servers for Citrix Profile Management or Folder Redirection, make sure enough
disk capacity, RAM and CPU is assigned to these servers, and that high availability is in place
• Keep the number of Organizational Units and Security Groups at a minimum. This
will make management easier and speed up troubleshooting
• Monitor and always regularly patch storage, networking and hypervisor infrastructure
• Maintain tight change-control and always keep track of changes to the environment

Optimizing the Citrix
Environment
For Maximum Performance

Lifecycle of a Citrix Deployment
Design
Test &
Deploy
Operate
&
Maintain
Trouble-
shoot
Optimize
Rectify Monitor

Achieve Persistency Even Across Non-Persistent Desktops
• Citrix Profile Management and Citrix WEM allow profiles, mapped drives, printers,
and other actions to roam with users regardless of which desktop they connect to,
providing a persistent feel
• Citrix App Layering (Elastic Layers) can solve niche cases where users need access to
specific applications, providing them with a unique desktop experience even on non-
persistent desktops
─ App Layering (User Layers) is now generally available, and will provide a storage location
for a users profile, applications and settings, which will roam from desktop to desktop

Achieve a Roaming OST for a Great Email Experience
Office 365 User Layers captures the
OST file in a VHDX container, which
can roam between desktops.
FSLogix Office 365 Containers
provide the same solution, plus
many additional advanced features.
Cache Exchange Mode with Office 365 is desired, but how
can that work effectively in a non-persistent environment?

Achieve Successful Patching of Citrix Images
• Review Citrix validated patching reports
─ https://support.citrix.com/pages/interop
• Perform your own tests on patched images before rolling out to production
• Carry out full change control procedures
• Roll out patches to 10% of estate initially, before rolling out to entire Site
• Use disk versioning to ensure successful rollback (Citrix Provisioning/PVS)
• Keep previous MCS images or image versions to ensure rollback of Machine Catalogs
• Use Citrix App Layering to reduce patching efforts

Image Optimization with Citrix App Layering
https://www.jgspiers.com/citrix-app-layering/

How to Keep on Top of Your Images & VDA Housekeeping Tips
• Consult with HR/Active Directory teams to be notified when staff have left
the company
─ Remove desktops or reassign to other staff
• Review desktop count regularly and reduce pool if required, or increase it
• Retire VDAs hosting applications that have since been migrated to newer
VDAs, or moved to the cloud/SaaS
• Use monitoring software that can alert you when VDAs have not been
logged on to for a period of time
• Take backups of images and then delete the live copies as part of
decommission process Script to find unused VDAs:
https://www.jgspiers.com/find-unused-vdas-script/

Successfully Deploy New Desktops or Applications to Citrix
• Consult with application vendors to understand h/w requirements to run their software,
especially in a multi-user environment
• Consult with application vendors to understand s/w requirements to run or work alongside the
applications, such as Outlook/Excel/Word/Java/.NET/Internet Explorer
• Deploy OS and s/w that both are in support by Microsoft, Citrix, and the application vendor(s)
• Profile staff, learning which peripherals they will use with the application, what drive maps they
need access to, what printers they need access to and so on
• Design desktops and application servers according to requirements
• Design Citrix and Group Policies appropriately according to requirements. Lifting restrictions on
clipboard access, client drive mapping and so on may be required.
• Run initial pilots to determine user acceptability, and gather resource consumption metrics to
determine impact through scale. Move pilot into staged production

Effectively Plan for Disaster Recovery
• Document steps required to recover a site in the event of a disaster, i.e. datacentre
outage. This allows staff to understand what they need to do
• Ensure that the load of a datacentre can be hosted elsewhere, i.e. cloud or a
recovery datacentre
• Use N+1 method to deploy StoreFront and Delivery Controllers
• SQL, Domain Controllers, hypervisors, and so on need to be redundant and cope
under partial failure
• Citrix ADC can help route traffic away from StoreFront etc. servers that are down
• Test Local Host Cache and configure your DDCs to cope with extra registration
requests, etc.
• Regularly review disaster plan and annually or bi-annually perform disaster testing
https://www.jgspiers.com/citrix-local-host-cache/

How Automated Monitoring
Aids in Citrix
Troubleshooting

Synthetic and Real User
Experience
Citrix and Supporting
Infrastructure
End-to-End
Performance Correlation
Reports, Analytics, Machine
Learning
Monitor Every Layer,
Every Tier
Pinpoint Root Cause of
Problems for Troubelshooting
Right-size & Optimize for
Maximum Performance
Measure All Aspects of Citrix
User Experience
1
2
3
4
End-to-End Monitoring with eG Enterprise

eG Enterprise Demo
• Client-side issue
– A non-corporate app taking up high CPU and
slowing the Citrix session
• Citrix issue
– GPO processing affecting logon time
• Infrastructure issue
– VMware bottleneck in
the hypervisor affecting
Citrix sessions

Dramatically simplify and
accelerate application
performance management.
Deliver peak performance,
user satisfaction and ROI.
User Satisfaction
IT Productivity
Higher service
uptime
Simplicity
Cost control & ROI
Total Citrix
Performance
Management
“
”
Benefits of Using eG Enterprise

Thank You!
For more information on Citrix
monitoring, contact eG Innovations:
www.eginnovations.com
info@eginnovations.com
For more information on Citrix
troubleshooting, contact George Spiers:
www.jgspiers.com
george@jgspiers.com

Citrix Troubleshooting 101: How to Resolve and Prevent Business-Impacting Citrix Problems

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Citrix Troubleshooting 101: How to Resolve and Prevent Business-Impacting Citrix Problems

Similaire à Citrix Troubleshooting 101: How to Resolve and Prevent Business-Impacting Citrix Problems (20)

Plus de eG Innovations

Plus de eG Innovations (20)

Dernier

Dernier (20)

Citrix Troubleshooting 101: How to Resolve and Prevent Business-Impacting Citrix Problems

Notes de l'éditeur