-Good morning and welcome to “Citrix Group Policy Troubleshooting for XenApp and XenDesktop”, my name is Rick Berry and I’ll be your guide to help clear up the mysterious world of Citrix Group Policies.
-A little bit about myself:
I’ve been on our Global Escalation team for 8 years and I work on both XenApp and XenDesktop related issues.
I specialize in the Citrix Group Policy components and work closely with our Group Policy architects in addition to providing troubleshooting assistance to our global support teams.
-When you’re having Group Policy issues, they can create a significant business impact in your environment, plus to complicate matters, you may not even know you’re having Citrix policy issues. So my key objectives for this session are to: Guide you through the Citrix Group Policy components, discuss troubleshooting tools and methods to help resolve your Group Policy issues or even better, prevent them altogether. Also, since the Citrix GP components are shared amongst our XenApp and XenDesktop platforms, the concepts I’ll be discussing today apply to both platforms.
-Lastly, we should have time for one or two questions. I’ll be covering a lot of material today, so if you have a question please hold it until this time. I’ll be close by after this session and later in the day over at the Ask the Experts bar in the Citrix booth if you think of something later. Plus this slide deck including my narrative will be available for download after Synergy in addition to the video recording being available on Citrix TV.
-So with that in mind, let’s get started
You can tweet about this session using #SYN406
There are multiple components involved with Citrix Group Policy and understanding how they work together is key to better troubleshooting Group Policy issues.
First, I want to establish a baseline for some terminology we’ll be using today on the different locations Citrix policies can be applied. This is key since troubleshooting methods can differ a bit depending on where you apply your Citrix policy settings.
First, you can set Citrix policies via the local group policies
-Each server and workstation has a single group policy object and Citrix policies can be set here
Next, we have Citrix Farm policies
-These are also known as IMA policies (usually in XenApp environments)
-These are policies that are set via AppCenter or the DSC (for XenApp 6.x) or Studio (for XenDesktop\XenApp 7.5)
-These policy settings are stored in the Farm datastore\database
Lastly, we can set Citrix policies via Active Directory Policies
-Set via Site, Domain or OU GPO’s
-Stored in Active Directory
-This allows for a single GPO object with both Citrix and Microsoft policy settings
Here are the methods for applying Citrix Policy settings
However, for todays presentation, I’ll be focusing on the Farm\IMA and Active Directory method since these are the most common
Next, it’s important to understand the processing and precedence order for policies. Especially if you are applying policies using different locations. So lets take a look at this. order for policies. This is the order that the GPO’s are reviewed for applicable policy settings.
Here are the available GPO types where we can place Citrix policy Settings starting at the local policies
This is the processing order, in other words, the order in which analyze each GPO type for applicable Citrix policy settings
Now the precedence order is opposite of this, meaning that policy settings in the higher GPO will win when there’s a conflict in settings
For example: say you have Client Drive Mapping enabled via a Citrix Policy in AD whereas in the local GPO you have it Prohibited, while processing policy settings, we’ll see both settings, but due to the precedence order, the resultant set of policies will contain the winning setting from the AD GPO
The first major component I’ll discuss is the Citrix Group Policy Management Console.
I’ll refer to this component as the Citrix GPMC from here on out, and provides Citrix policy visibility within the Microsoft GPMC
It allows you to manage Citrix policies via AppCenter\Studio (for Farm policies) or the Microsoft GPMC (for AD GPO’s)
This component also allows you to perform modeling or comparison of policy settings
It can be installed on a device for managing policies, requires that GPMC is installed
In case you ever need to review version details for these binaries during troubleshooting, the core binaries are located here
The second major component I’ll go over is the Citrix Group Policy Client Side Extension
I’ll refer to this from here on out as the Citrix CSE, this is separate from the MS CSE, and loaded via this DLL
This component is initialized via the Winlogon process (for both Computer and User)
It generates the policy requests
It will also retrieve certain aspects of the Computer or User connection, this is needed to see if any policy filters will be leveraged
This component also forwards policy request to the caching service
The core binaries for this component are located here
Next lets look at our Caching service
Handled via the CitrixCseEngine process which runs as a service, and it’s a part of the Citrix CSE
It handles the policy calculation in addition to writing the resultant settings to the registry
It also caches the Farm\AD GPO objects to optimize the policy application process
For AD\Farm GPO’s, the cache folder is located here
In addition, it caches per-computer and per-user policy files which we review in detail later in the sessions
Now that we’ve gone over the core components and services involved, let’s review what data files Citrix Group Policy uses.
First, we have the resultant set of policies file, known as RSOP.gpf, and exists per-computer and per-user
This binary file location is based on whether it’s a per-computer or per-user file
The RSOP.gpf file is parsed into registry settings and are applied to the proper location,
NOTE: the <SESSIONID> you see here is going to be used quite a bit for user policies
Let’s talk about removing or rolling back Citrix policy settings
We need a way to remove policy settings that aren’t explicitly configured, you may ask why ?? If a user logs in and gets say a printer virtual channel bandwidth setting via Citrix policy, if the next user to log in to that same session they may not have that setting configured, so we need to be able to “remove” or “undo” this settings
The data file for this mechanism is the Rollback.gpf file. It exists for both per-computer and per-user settings
The rollback.gpf file contains the instructions needed to remove pre-existing RSOP settings in the registry
The rollback file is cached to the same location as the RSOP.gpf file
Next, I want to review the Citrix Policy Filters
The Citrix policy filters allow granular control of policy settings
Allows filtering based on certain criteria, for instance workstations coming from a certain IP range, or connections coming thru your external Netscaler
The available filters vary based on whether you are dealing with a Computer or User policy
The filters can be used with any policy set with the exception of the “Unfiltered” default policy
Lets take a look at the Computer filters for XenApp 6.5 from AppCenter.
You can see the filter types listed here
We’ll zoom in a bit on the filter types, and we see that for a Computer policy we have these available filters
When selecting a filter type, you’ll get additional details on the filter in this pane
Zooming in again, we can see the products and versions the filter applies to plus details of the filter
Now lets take a look at that same details for User Policies.
We see the available filter types here
We’ll zoom in a bit to show you the additional filters that are present for User policies.
Let’s review our default policy information along with the built-in templates
There’s a default unfiltered policy (per-computer and per-user)
The policy settings you add to the Unfiltered policy apply to all related objects (Computer or User)
The unfiltered policy can be disabled if needed (if you disable the Unfiltered policy move it to the lowest priority)
By default there are pre-configured policy templates in place
Templates are group based on user connectivity methods (WAN, etc.)
In addition, once you create your own polices, they can be saved as a template, this is a good backup method, but this shouldn’t be your only backup method
Now, as you move from XenApp 6.x and XenDesktop 5.x to newer versions, you going to see some difference when managing Citrix policies.
This is the policy node from a XenApp 6.5 environment
In this case, you’ll see separate Computer and User policy nodes
Now for newer versions of XenDesktop 7.x and XenApp 7.5, this view is different. Here we see a Studio screen shot of the policy node from XenApp 7.5
We see that there's only a single node for the policy objects. There’s no differentiation between computer and user policies within Studio for these newer versions.
We take care of that on the backend when determining the resultant set of policies.
The exception to this is when you manage Citrix policies via AD GPO’s, in that case, because you’re using the GPMC to manage policy settings, there will still be separate Computer and User sections.
Here are some details surrounding policy refresh intervals, there are some differences to be aware of
For policies setup in Studio\AppCenter
The Computer and logged in User policies will refresh every 90 minutes
For AD GPO’s
The refresh interval is typically every 90 minutes plus or minus a random offset
This update interval can also be set via AD GPO at the domain level
For either method
Computer policies will update at machine startup
User policies will refresh during login or reconnection to an active or disconnected session
You can also manually update the policies by issuing gpupdate /force (this is helpful when you modify policies and want them to refresh quicker)
Now that we’ve reviewed the components and files involved in Citrix Group Policy, I want to tie this altogether for you by showing you how Citrix Group Policies are applied when a user logs in. NOTE, this process is similar for when a computer starts up.
Here we have an end user ready to log in <PAUSE><PAUSE> What, this isn’t your end user, lucky
Ok, here’s your happy end user logging in
Once the user logs in, the Microsoft Winlogon process starts up
Then it loads the available Client Side Extensions
This includes not only the Microsoft policy extension, the Citrix one as well
From this point I’ll focus on the Citrix CSE
Citrix CSE starts to process the policies and the local GPO’s are processed first
Then CSE process the Farm policies
Then lastly CSE processes the Active Directory policies
Now the precedence order is just the opposite of our processing order (so Active Directory policies l take precedence over Farm policies and so on), using this precedence order and any policy filters involved, a resultant set of policies file (RSOP.GPF) is created
This file is then used to make the actual policy settings in the registry (the registry location is based on whether we dealing with Computer or User policies)
Let’s take a closer look at this final stage of Citrix Group Policy processing, since there’s a lot more going on here and understanding this critical stage can help you significantly when troubleshooting
The Rollback.GPF file is loaded from the cache (keep in mind this is the file containing instructions to remove or undo the existing RSOP settings)
We then apply the new RSOP policy settings, this results in the registry being updated
We then remove the previous cached GPF files
Then we copy the new RSOP.gpf file to the cache and create a new Rollback.gpf file for the existing session which is cached as well
The newer Rollback.gpf file will then be used the next time the User logs in (or Computer starts up)
Then the LastUpdate value is updated in the registry for the Computer or Session
At this point, we’re done
Now, I want to review recommended practices based on issues encountered during Citrix Support cases.
First, we’ll discuss policy architecture
While it’s fully supported to use both Farm and AD policies together, it can lead to confusion during troubleshooting or when something is not working properly
If possible try to use one method or the other based on your requirements
We’ve seen some issues where the use of WMI filters on AD GPO’s (for example, you want to apply the AD GPO to a certain OS), can cause issues during session reconnects. This is due to some issues with the MS API’s used by Terminal Services during a reconnect and AD timing out for WMI calls.
Try to use WMI filters sparingly and if you have to, it maybe possible to prevent issues by using a setting we have, I’ll talk a bit about this more later
<????> I have a question for you, no need to raise your hand, think about if you’ve got your Citrix policies documented or even better backed up in a manner you could easily restore them
<PAUSE> Ok, that’s a good amount of you. Documenting policies is important especially when things go wrong. No worries for those of you that don’t have them documented yet as I’m going to go over steps you can take to easy accomplish this.
We have several methods to document these, based on how you apply your policies
For Farm policies applied via AppCenter or Studio
It’s best to get them written out say in Excel, etc. Or you can also leverage our Citrix Scout tool to get the settings and filters exported.
For policies applied via Active Directory
Use the GPMC and select the Save Report option on your GPOs, this will create a HTML report (NOTE: this needs to be done a device with both the MS GPMC and Citrix GPMC installed)
For either method, there are a few other tools that will help with this as well and I’ll go into more details on these in a bit
The first is CtxCseUtil, which will generate resultant set of policy reports
We also have a PowerShell module that contains cmdlets to export and import Citrix Group Policies (the export cmdlet is great since this gives you a snapshot of your policies that can be imported later)
There are certain actions, that if taken, can cause group policy consistency issues. These are some of the more difficult issues to troubleshoot and you may not even know you’re having an issue until it’s too late (you’ll get the dreaded call from security or your bosses, bosses boss). So here are some things you shouldn’t do.
Unless instructed by someone from Citrix Technical Support, don’t manually manipulate any of the Citrix Group Policy files\folders\registry areas
This includes these areas which relate to the policy cache and resultant registry areas. These areas are tightly linked together, so altering any one area can have negative effects.
There is a certain use case where we might ask you to clear these areas out, for instance a specific fix as referenced here for LA5051, in this case, we clearly document the steps needed.
Ok, now that I’ve gone over the architecture and components of Citrix group policy, let’s talk about some recommended practices based on issues we’ve seen while working on support cases.
When troubleshooting Citrix group policy issues, we have a recommended troubleshooting path. These steps will help you isolate the issue or in case you need to contact Citrix Technical Support for assistance, you’re going to have the information ready to go and help expedite the resolution process.
I’ll be covering these steps in greater detail in the upcoming slides
The best starting point when troubleshooting Citrix group policy issues is to know what the desired policy settings are for the given scenario, so knowing you’re baseline configuration is key.
Then to isolate the issue further, make sure you can answer what I call the four W’s
Who is seeing the issue, are there specific users in a separate office, or all users?
What issue is being seen, are they getting client drives mapped when they shouldn’t be or vice versa
When are they seeing the issue. For new sessions, reconnecting, smooth roaming or maybe all of those?
Lastly, where are they seeing the issue? Are they working from home and coming in via your Netscaler?
Or are they working in the corporate office?
The next step in the process is to be sure you know what version of components you are dealing with.
Now looking at this screen shot, we only see the core XenApp 7.5 install the related VDA, but no group Policy components
So what version am I really at?? In this case, the GP components are at the baseline versions that come with the core product. The best method to confirm the versions is to check the individual components.
Note that our Citrix Scout tool does collect versions of the Citrix Group Policy components as I’ve highlighted in this Scout data file for a XenApp 6.5 server
So let’s take a look at getting the CSE version.
You can look in the component directory
As I mentioned earlier, here’s the path to the CSE, you can right click on the CitrixCseEngine executable, which is the primary CSE binary
And going to the Details tab will show you this is version 2.1
Now to determine the GPMC version, we can do the same thing
We go to the components directory as I’ve referenced here and we can right click on the main binary
And we get the version here, in this v2.2
Now that we’ve reviewed how to get the component version here’s a reference chart to show the versions for our core XenApp products. This will help you when you have the baseline versions of the components installed.
I’ve provided the XenApp version, GPMC and CSE versions
For 6.0
For 6.5, When there’s a higher version in parenthesis, this means this component has a higher available version. You can review these on our support site and you’d likely be asked to upgrade if you’re having issues
And lastly for 7.5
Now that we’ve reviewed how to get the component version here’s a reference chart to show the versions for our core XenDesktop products.
Again showing the version of XenDesktop, GPMC and CSE
For 5.5, 5.6
For 7.1
And lastly for 7.5
Next we’d want to check our the locally cached policy objects, first we’ll look at cached objects from AD GPO’s
On the server\VDA in question, here’s a look at the GPO cache
As I noted earlier, it’s under PROGRAMDATA folder
Let’s look closer at the GPF files
In this case, we see a GUID in the file name
When we see GUID in the file name, this means it’s a cached AD GPO
You can there are two files with the GUID, the one with a 0 is the cached User policy
The 1 in the name is the cached Computer policy
Now that we have the GUID, let’s take a closer look in AD
We look at the local cache again and note the GUID
We can then search in the GPMC (or use PowerShell cmdlets) to find a match
Here we’ve found a GUID match, so this is our AD GPO in the GPMC
We then also know what the GPO display name is
So now we want to compare the time details for the AD GPO, here we see the creation date
Here is the modified date (meaning the last time a change was made to it), this is the date we’re really interested in
In this case, the dates are a match between the AD GPO and the locally cached version, so our cached version is up to date. If we didn’t match, usually issuing a gpupdate /force will resolve that, if not, there’s a likely an issue with the GPO in AD (look in event viewer for errors)
NOTE: There could be a difference in the modified timestamps of the cache files since there is a separate one for User and one for Computer. The modified timestamp in the GPMC is the last update and should reflect in one of the files.
Now for farm policies created with AppCenter or Studio, the cache files look a little different.
We’re looking in the same cache folder
Looking closer, we see these files lack the GUID
Which means they are Farm policies
But the User and Computer notations are similar
Now we should check the Computer and User cache files
Here we see the location of the cached RSOP and Rollback GPF files
The per-computer cache files are located in the root of this folder
Now we can look into the sub-folders, the folder is named for the SessionID, if you know of a specific user seeing the issue, you can issue a QUSER to get their session and check from there
These sub-folders are for the Per-User files (for the given SessionID)
These are the files for Session 2, note for either computer or users files, you should always have both rollback and rsop gpf files, if not, this needs to be resolved
Now, let’s review what’s in the registry for Citrix Group Policies
Here we see the core location for the per-computer settings
The various sets of Computer policies usually are contained in sub keys, here we see the Graphics and ICA policy settings
Looking at the values, we can see the final resultant policy settings, NOTE: we have references on our e-docs site to match up a registry value to the policy in question
Now we should review the per-user RSOP settings in the registry, especially if there’s some issue with a policy applying (or not applying)
For the per-user registry area, we need to so into the sub-key noted with the SessionID which we get from QUSER, in this case we go into Session 2
Here I’ve highlighted two sub-keys of interest, the Events key contains a last Update time of the policy settings whereas the Evidence key contains details of the connection itself (things like Client IP, Client Name, etc.)
Under the User sub-key we get the various groups of GPO’s and their settings
Now let’s look at connection details and how that relates to Citrix Group Policy.
<<QUESTION>>Can anyone tell me why we’d be interested in client connection details in relation to Citrix group policy? <PAUSE>
Correct, we’re interested in connections details because we can filter User policies based on certain connection filters
Things like Client IP address or Client name, so if connection info isn’t right, there’s no way we’ll be able to apply the right policies
Lets look at the connection details closer.
On the XenApp\XenDesktop device the end user is connecting to, we track the connections details under HKLM\Software\Citrix\ICA\Session, each session has a sub-key noted with the Session ID which we can get from running Quser on the device, then we can look in the Connection sub-key for details
Here we see the connections Client Name and Client Address, which is good to know in case we’re filtering on this. Issues with policies not applying might be related to sub-net changes or a rebuild of a client device
On a side note, you can also look up the full client version for connection here as well
CTXCseUtil is a Citrix tool that can translate the RSOP.gpf for a user or computer into a html report. This can be beneficial when troubleshooting policy issues.
Once we have CTXCseUtil on the Server\VDA, we can get to a cmd prompt on that device and run it (or we can run it against a remote device)
Usually, when you run CtxCseUtil the first time on a device, you’ll likely get a general WinRM error. CtxCseUtil needs WinRm configured so it can run.
The solution to this is to run WinRM quickConfig on the device
Here’s another common issue when running CtxCseUtil.
This “A specified Logon session” message error is another common issue
This error usually means that you are trying to get RSOP report when using AD GPO’s and you’ve logged in with a local Administrators account that doesn’t have access to AD, The solution is to log in or run the tool with an account that has AD privileges
NOTE: There are more details on using the tool in the Help Message document located in the same folder as the tool
Once you run CtxCseUtil, the resultant report will be created in the same folder. The file name is CitrixRsopResult.html.
Here’s a screen clip of the report detailing the User policy section
Here’s a shot of the Computer policy section, showing what filters were applied or not
Then lastly the report will show the Setting name, What Value was set and the GPO name the setting comes from
The next tool we’ll look at is our PowerShell Module for Citrix group policies.
This powerful tool contains cmdlets for managing all types of Citrix policies
The module is not built-in and needs to be imported, I have a link in the resources section to the module
It contains cmdlets to: set\get policy settings, export\import policy objects
The module will import\export policy settings, configuration details and filters used, so a this is the preferred method to backup your polices for easy restoral
So let’s walk thru using the PowerShell module to export our Farm policies
Here we have our PowerShell cmd prompt up and we’re in the directory where we’ve copied the PowerShell module to
Then we issue the Import-Module command
Once Loaded we can use the Get-Command on the module to see the available functions, we can also use the Get-Help command on any of these functions for more details, not only can you import or export policy settings but you can modify or create them as well. Be sure to have a backup of your existing policies before modifying your policies
Then we run the command to export the farm policies to a specific folder
Once that completes, you should have these files in the folder specified
When exporting Citrix policies from Active Directory GPO’s, there’s a few more steps involved.
We use the same Citrix PowerShell module and cmdlets
We create a PowerShell drive to the Active Directory GPO
We have all of the steps in this Citrix Support article
In the article we detail how to export settingsAnd how to import settings from AD GPO’s
Next, at some point you might need or be asked to capture a CDF trace of your policy issue and our latest CDFControl tool is a great way to expedite this process. <CLICK> Here we see that CDFControl already has a built in trace category for capturing Citrix Group Policy issues
<CLICK> Zooming in, we can capture all modules, just the CSE or GPMC, if there’s a need to contact Citrix Technical Support, having a clear snapshot of the issue and details involved will expedite the resolution process.
Now, I’m going to move on to two specific issues we’ve seen from time to time related to Group Policy issues. The first one relates a farm\IMA policy issue.
Has anyone ever seen this error when accessing your Computer or User policies within AppCenter or Studio?
With farm policies (not AD), the policy object is stored within the datastore\database as a single entity or blob
This error can occur when the policy object gets corrupted, one possible use case is when multiple people are access Citrix policies and making changes at the same time, this use case has been resolved in newer versions of AppCenter\Studio
This error will be seen when you attempt to access either the Computer or User policies node within AppCenter or Studio
Try not to just jump in and restore the datastore\database for this issue, since you may not know when the corruption occurred.
If you do see this error however, please contact Citrix Technical Support for assistance, we have a set of focused steps to help you resolve this issue
When using Farm type policies, it’s important to maintain a policy export, this can be used for easy restoral in cases like this
The second issue I want to bring up relates to the use of WMI filters on your AD GPO’s that contain Citrix policy settings.
If you are using WMI filters on your AD GPO’s, it’s possible you may see some issues such as
-Citrix policies not correct for reconnected sessions
-Or reconnects\logins taking a extremely long time to complete (especially, if the issue seems to resolve itself after a while)
In order to debug this further, it’s best to enable Microsoft Group Policy logging
- This can be accomplished using this registry value (I have resource link to a TechNet article on this)
This will create a log file here, look for this entry related to a FilterCheck API call, if it’s returning this error you are seeing AD time out on the WMI call
You should also scan the event logs for WMI related errors, if any of these are seen, you should contact Microsoft for further assistance, there are several MS hotfixes out there to help with this issue, but the particular events surrounding this need to be reviewed for your specific environment.
Here are the key items we reviewed during todays presentation.
This section contains references, articles and links to tools that I’ve discussed during the presentation.
Here are links for Group Policy Documentation
Here are links to Citrix Technical Support articles on Group Policy issues and a planning guide
And lastly here are links to tools related to Citrix Group Policy and Microsoft Group Policy.
I’ve shined the light on Citrix Group Policy architecture to help you isolate and troubleshoot Citrix Group Policy issues in your environment. Plus we reviewed the key points to help preventing Group Policy issues in your environment as well. Thank you for attending my session and enjoy the rest of Synergy, I’ll be out in the hallway for further questions and come over a see us at the Ask The Experts bar at the main Citrix Booth in Hall C.