This session will cover advanced techniques in troubleshooting the Citrix NetScaler Appliance using tools such as Citrix TaaS, IPMI, nsconmsg, wireshark and log analysis. We will review usages of these tools along with case studies showing how to best troubleshoot common issues seen in operating Citrix NetScaler Appliances. What you will learn - Various tools available to troubleshoot issues and how to use them to isolate NetScaler Issues - Common deployment problems and how to isolate the causes

1. Advanced Tools and Techniques for
Troubleshooting NetScaler Appliances
Andrew Redman | Lead Escalation Engineer
Citrix Support Webinar
August 26 & 27, 2014

2. 2 © 2014 Citrix. Confidential.
NetScaler System Overview
Troubleshooting Tools & Techniques
Case Studies
Resources
Conclusion
Q&A
Agenda

3. NetScaler System Overview

4. Key NetScaler Processes
Process Description
ns_master/NSPPE
4 © 2014 Citrix. Confidential.
nsvpnd
nsaaad
nsconf
nsauthd
nslog.sh
nssync
nsreadfile
nslcd
nsfsyncd
nsnetsvc
nsconmsg
nscollect
Runs Citrix NetScaler OS
SSL VPN File Transfer
RBA and SSL VPN external authorization
Writes the ns.conf file
CLI authentication
Controls logging for the newnslog
HA synchronization
Used to read SSL certificate files
Runs the front panel LCD
Synchronizes bookmarks and SSL certificates
Used by the GUI for configuration changes
Controls writing of the newnslog
Statistics gathering for historical purposes

5. NetScaler File System
/var (hard drive) Logs - /var/log & /var/nslog
5 © 2014 Citrix. Confidential.
Install - /var/nsinstall
Trace - /var/nstrace
Core Dumps - /var/crash & /var/core
/flash (flash drive) Config - /flash/nsconfig
SSL Certificates - /flash/nsconfig/ssl

6. NetScaler File System (cont.)
/flash (cont.) User Monitors - /flash/nsconfig/monitors
6 © 2014 Citrix. Confidential.
Custom Options - /flash/nsconfig
/ (ram drive) OS - (operating system)

7. Troubleshooting Tools & Techniques

8. NetScaler Tech Support Bundle
> show techsupport
Critical
System Data
In-Depth
Performance
Monitoring
Stats
Detailed Log
Files
USER
Command
Logging
/var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz

9. The NetScaler Tech Support Bundle
9 © 2014 Citrix. Confidential.

10. Citrix Insight Services
10 © 2014 Citrix. Confidential.

11. 11 © 2014 Citrix. Confidential.
Data Collection Analyze Data
‘Single Mission … Data Collection’
FAQ: http://support.citrix.com/article/CTX131233
Tailored
Recommendations

12. https://taas.citrix.com
12 © 2014 Citrix. Confidential.

13. Technical Issues Flagged
13 © 2014 Citrix. Confidential.

14. Investigate Issues Further
The ‘BELL’
14 © 2014 Citrix. Confidential.

15. Crash File(s) Location
15 © 2014 Citrix. Confidential.
nscollect-542.gz

16. Intuitive Navigation
16 © 2014 Citrix. Confidential.
Select a different
newnslog file to view

17. Detailed Graphs
17 © 2014 Citrix. Confidential.
Mouse over graph
to see more detail
Informative
reference legend
Download the data
as an excel sheet

18. IPMI – Intelligent Platform Management Interface
Change NetScaler
18 © 2014 Citrix. Confidential.
Default LOM IP Address: http://192.168.1.3
IP Address
Obtain Health
Monitoring Detail
Harvest Serial
Number
Determine MAC
Address

19. Common CLI Show Commands
Common show commands for system information:
show node, show info, show license
Common show commands for vserver and service:
show lb vserver, show cs vserver, show service, show persistencesession
show connectiontable
Other common show commands:
show route, show ip
19 © 2014 Citrix. Confidential.

20. Common CLI Stat Commands
Common stat commands for system information:
stat ns, stat cpu, stat interface
Common stat commands for vserver and service:
stat lb vserver, stat cs vserver, stat service
Other common stat commands:
stat dns, stat ssl, stat http
20 © 2014 Citrix. Confidential.

21. Leveraging ‘nsconmsg’
Nsconmsg common use cases:
View events
View console messages
View statistics
Debug system counters
Debug load balancing issues
Debug CPU/Memory utilization
21 © 2014 Citrix. Confidential.
Make absolutely
sure that you
use a capital -K
and NOT a
lower-case -k

22. Example ‘nsconmsg’ Usage
# cd /var/nslog
# nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg
22 © 2014 Citrix. Confidential.
newnslog
current log file
fqdn-ssl-vip
name of vserver
ConLb=1
LB stats

23. Displaying debug performance information
NetScaler current V20 time Performance is Sun Mar Data
23 18:33:43 2014
NetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43
current time is Sun Mar 23 18:33:43 2014
-------------------------------------------------------
NATSession : Free(6553)A(6553)InUse(0)
NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0])
NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])
Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0
SSF: Conn (Srvr 0 Clnt 0) U:0
Mon: Probes: 434562009, Failed: 15
CM: VIP(Conn 10.54.169.75:(Srvr 0 Clnt 0) Sessions 443:UP:PCB LEASTCONNS): 0 NATPCB 0
Hits(7317, 0/sec) Mbps(0.00) Pers(OFF)
Z(SIP[S(10.54.148.201:0], C[0], SSL[0] Server[80:0] UP) SIPDIP[Hits(0] 7317, DIP[0] SO[0/sec, 0])
P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0
Mon: Probes: 434562009, Failed: 15
VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0
S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms)
Load(0) LConn_Idx: (C:0; V:0,I:1)
-------------------------------------------------------
CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014
23 © 2014 Citrix. Confidential.
kbits) RspTime(0.00 ms)
CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014

24. # nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more
# nsconmsg -K newnslog -s ConMon=1 -d oldconmsg
# nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg
# nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg
ConDebug - Debugging
ConLb - Load Balancing
ConMon - Monitoring Probes
ConMEM - Memory Management
ConCSW - Content Switching
ConSSL - SSL Offload
ConCMP - Compression
ConIC - Integrated Caching
24 © 2014 Citrix. Confidential.

25. Log File Analysis
# cd /var/log
# zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE)
ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED
2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success"
ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED
2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success"
25 © 2014 Citrix. Confidential.
interface down vServer down panic signaled

26. NetScaler + Wireshark = ‘thumbs up’
http://www.wireshark.org
26 © 2014 Citrix. Confidential.

27. RED HOT Wireshark Tip
27 © 2014 Citrix. Confidential.
Custom
Columns
Custom Menu
Options
Pre-build
Custom Filters
Much Faster
Analysis
Get the red hot details on how to empower your default Wireshark
configuration in the Reference Section at the end of this presentation.

28. Troubleshooting Techniques & Case Studies

29. Top Tips
29 © 2014 Citrix. Confidential.
Use Citrix Insight Services
Pay attention to the issues!
Note the highlighted counter(s)
Use nsconmsg to see even more detail
Correlate time-frames in other log files
Achieve root cause analysis faster!

30. Case #1 - High Availability Synchronization

31. Same type of
appliance
31 © 2014 Citrix. Confidential.
HA Pre-requisites
Same firmware
version
Recommend
same nsroot
password
Same RPC
Node password
Open requisite
TCP ports
Primary NetScaler Secondary NetScaler

32. Insight Services Flags The Issues
32 © 2014 Citrix. Confidential.

33. The HA Pair Struggled To Synchronize
# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted)
57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1)
57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2)
9861 0 65 nic_err_bdg_muted interface(0/1)
9862 0 65 nic_err_bdg_muted interface(0/2)
33 © 2014 Citrix. Confidential.
71,837,018 MAC Moves
65 Interface Mutes

34. The ‘newnslog’ Time-Frame
# nsconmsg -K newnslog -d setime
Displaying start and end time information
NetScaler V20 Performance Data
NetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41
start time Fri Feb 28 21:49:58 2014
total duration 00.00:03:30
end time Fri Feb 28 21:53:28 2014
total duration 00.00:03:30
data size 1,718,949 bytes
34 © 2014 Citrix. Confidential.

35. Case #2 - XA/XD Slow Performance

36. Smartphones XenDesktop
Tablets XenApp
36 © 2014 Citrix. Confidential.
Insight
Services
Critical Insight
Gleaned
Preventative
Approach
Don’t
Underestimate
XA/XD Slow Performance

37. Insight Services Again Flags The Issues
37 © 2014 Citrix. Confidential.
http://support.citrix.com/article/CTX136926

38. Performance Was Extremely Latent
# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved
4263 0 23 nic_tot_bdg_mac_moved interface(0/1)
4264 0 51 nic_tot_bdg_mac_moved interface(1/1)
4265 0 28 nic_tot_bdg_mac_moved interface(1/2)
38 © 2014 Citrix. Confidential.
23, 51 & 28 MAC Moves

39. Networking Issues Again?
# nsconmsg -K newnslog -d statswt0 | grep nic_err
4274 0 1995 nic_err_rl_pkt_drops interface(1/1)
4275 0 40736 nic_err_rl_pkt_drops interface(1/2)
4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1)
4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2)
4678 0 42731 allnic_err_rl_rate_pkt_drops
39 © 2014 Citrix. Confidential.
System Limits Exceeded
Rate-limited Packets!

40. 40 © 2014 Citrix. Confidential.
The Moral of the Story
Leverage Citrix Insight Services
Leverage
Insight Services
Pay Attention
Gain Quick
Insight
Dig Into
‘nsconmsg’
On Target for
Success!

41. Resources

42. Helpful Resources
Comprehensive NetScaler Counters
Wireshark Developer Editions
Customizing Wireshark Tutorial
Citrix Insight Services Forum
NSTRACE Options
How To Manage VLAN’s, Interfaces and Subnets
42 © 2014 Citrix. Confidential.

43. Conclusion

44. What We’ve Actually Covered
An Overview of the NetScaler System to give you a high-level understanding of the core system.
I shared with you some excellent Troubleshooting Tools that are available at your disposal.
I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues.
I then highlighted two different Case Studies leveraging the tools & techniques that I shared with
you in the presentation.
In addition I provided you with a few Resources for your future reference and edification.
44 © 2014 Citrix. Confidential.

45. Fuel your talent with continuous learning.
93% of Citrix Education students became more effective in their role after attending a course.
TVID: CFB-61B-A26
Citrix Education offers the following technical training for Networking professionals:
CNS-205: Citrix Netscaler 10 Essentials and Networking
CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam
CNS-301: Citrix NetScaler 10 Advanced Implementation
45 © 2014 Citrix. Confidential.
Visit (bit.ly/05Webinar) to save 10% off through September 30*
*Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply.

46. Simplify your journey, let us guide you.
Accelerate your implementation and minimize risk by taking advantage of Citrix
Consulting. You’ll get the expertise of certified Citrix Consulting Architects to
successfully deploy Citrix solutions in any phase of your project.
93% of Citrix Education students
became more effective in their role
after attending a course.
TVID: CFB-61B-A26
Visit bit.ly/CTXConsulting to learn more about our proven methodology.
46 © 2014 Citrix. Confidential.

47. 47 © 2014 Citrix. Confidential.
WORK BETTER. LIVE BETTER.