Exchange 2010 Active Sync Troubleshooting

2. Activesync troubleshooting
architecture connectivity
troubleshooting performance

3. Activesync - architecture
Security
• SSL for encryption and server ID validation
• AD credentials or client certificates for
authentication
• Activesync Mailbox policies
• Remote Wipe architecture connectivity
troubleshooting performance

4. Activesync - architecture
Security
• SSL for encryption and server ID validation
• AD credentials or client certificates for
authentication
• Activesync Mailbox policies
• Remote Wipe architecture connectivity
• Allow/Block/Quarantine
• Throttling
troubleshooting performance

5. Activesync – architecture -ABQ a c
t p
Logic Flow
• Is the mobile device authenticated? If not, challenge the mobile device for the correct credentials. Otherwise, go on to the next step.
• Is Exchange ActiveSync enabled for the current user? If not, return an "access restricted" error to the device. Otherwise, go on to the next step.
• Are the mobile policy enforcement criteria met by the current mobile device? If not, block access. Otherwise, go on to the next step.
• Is this mobile device blocked by a personal exemption for the user? If so, block access. Otherwise, go on to the next step.
• Is this mobile device allowed by a personal exemption for the user? If so, grant full access. Otherwise, go on to the next step.
• Is this mobile device blocked by a device access rule? If so, block access. Otherwise, go on to the next step.
• Is this mobile device quarantined by a device access rule? If so, quarantine the device. Otherwise, go on to the next step.
• Is this mobile device allowed by a device access rule? If so, grant full access. Otherwise, go on to the next step.
• Apply the default access state per the Exchange ActiveSync organizational settings. This grants access, blocks access, or quarantines the current device,
depending on the organizational settings.

6. Activesync – architecture -ABQ a c
t p
ABQ - Block

7. Activesync – architecture -ABQ a c
t p
ABQ - Block

8. Activesync – architecture -ABQ a c
t p
ABQ - Block

9. Activesync – architecture -ABQ a c
t p
ABQ - Block

10. Activesync – architecture -ABQ a c
t p
ABQ - Block

11. Activesync – architecture -ABQ a c
t p
ABQ - Block

12. Activesync – architecture -ABQ a c
t p
ABQ - Block

13. Activesync – architecture -ABQ a c
t p
ABQ - Block

14. Activesync – architecture -ABQ a c
t p
ABQ - Block

15. Activesync – architecture -ABQ a c
t p
ABQ – Block
IIS logs - Provisioning
V0
200

16. Activesync – architecture -ABQ a c
t p
ABQ – Block
IIS logs - Attempted Foldersync
Error:DeviceIsBlockedForThisUser_As:BlockedG
200

17. Activesync – architecture -ABQ a c
t p
ABQ – Block

18. Activesync – architecture -ABQ a c
t p
ABQ – Block - Cons
• Telling the Admins
• No auto email
• Can only allow the device by using Powershell
Get-ActiveSyncDevice -mailbox ceo | where{$_.devicemodel -eq "iPhone"} | Set-CASMailbox -id CEO -ActiveSyncAllowedDeviceIDs ($_.DeviceId)

19. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

20. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine
• Account seems to sync fine
• At first nothing is synchronized
• GAL search fails
• No calendar or contact information synced to device from mailbox
• After the discovery process complete, the quarantine message is delivered to the device

21. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine
IIS logs - Discovery
As:DeviceDiscoveryG
200

22. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

23. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

24. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

25. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

26. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

27. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

28. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

29. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

30. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

31. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

32. Activesync – architecture -ABQ a c
t p
ABQ – Quarantine

33. Activesync – architecture -ABQ a c
t p
ABQ – Limitations
• User Agent
• Zero day exploits
• Firmware level agnostic
• ISA / TMG / other firewall solutions
• manual powershell after the fact

34. Activesync - architecture a c
t p
Airsync Protocol
Activesync features available in Exchange 2007 sp3
http://msdn.microsoft.com/en-us/library/aa996303(v=EXCHG.80).aspx
Activesync feature available in Exchange 2010 sp2
http://technet.microsoft.com/en-us/library/bb123484
List of Activesync build / features and what mobile devices implement
http://en.wikipedia.org/wiki/Comparison_of_Exchange_ActiveSync_Clients

35. Activesync - architecture a c
t p
ISAPI

36. Activesync - architecture
Internet facing CAS
- [internal site CAS]
- XSO RPC MBX
architecture connectivity
troubleshooting performance

37. Activesync - architecture
Internet facing CAS
- [internal site CAS]
- XSO RPC MBX
architecture connectivity
troubleshooting performance

38. Activesync - architecture
Partnership
architecture connectivity
troubleshooting performance

39. Activesync - architecture
Partnership
architecture connectivity
troubleshooting performance

40. Activesync - connectivity
Autodiscover
architecture connectivity
troubleshooting performance

41. Activesync - connectivity
Direct Push
architecture connectivity
troubleshooting performance

42. Activesync - connectivity a c
t p

43. Activesync - connectivity a c
t p

44. Activesync - connectivity a c
t p

45. Activesync - connectivity a c
t p

46. Activesync - connectivity a c
t p

47. Activesync - connectivity a c
t p

48. Activesync - connectivity a c
t p

49. Activesync - connectivity a c
t p

50. Activesync - connectivity a c
t p
Affinity

51. Exchange ActiveSync Common Status Codes
Ping Command Status
Value Meaning
1 The heartbeat interval expired before any changes occurred
in the folders being monitored. The client should reissue the
Ping command request.
2 Changes occurred in at least one of the folders that were
being monitored. The response includes the folders in which
these changes have occurred.
3 The client Ping command request did not specify all of the
necessary parameters. The client is expected to issue a
Ping request that includes both the heartbeat interval and
the folder list.
4 There has been a general error in the Ping request issued
by the client, which can be caused by poorly formatted
WBXML.
5 The heartbeat interval specified by the client is outside the
range set by the server administrator. If the specified
interval was too great, the returned interval will be the
maximum allowable value. If the specified interval was too
low, the returned interval will be the minimum allowable
value.
6 The Ping command request specified more folders to
monitor for changes than is allowed by the limit configured
by the server administrator. The response specifies the limit
in the MaxFolders element.
7 The client specified a folder that has been moved or deleted
or the server that the client has been accessing has been
upgraded from Exchange Server 2003 SP1 to SP2. The
client should issue a FolderSync request.

52. Exchange ActiveSync Common Status Codes
Sync Command Status
Value Meaning
1 Success.
2 Protocol version mismatch.
3 Invalid sync key.
4 Protocol error.
5 Server error.
6 Error in client/server conversion.
7 Conflict matching the client and server object.
8 Object not found.
9 User account may be out of disk space.
10 An error occurred while setting the notification
GUID.
11 Device has not been provisioned for
notifications yet.

53. Exchange ActiveSync Common Status Codes
Search Command Status
Value Meaning
1 Success.
2 Protocol Error.
3 An error on the Exchange server occurred.
4 Bad Link.
5 Access Denied.
6 Not Found.
7 Connection Failed.
8 Too Complex.
9 Index not loaded.
10 TimeOut.
11 NeedToFolderSync.
12 EndOfRetrieveableRangeWarning.

54. Exchange ActiveSync Common Status Codes
FolderSync Command Status
Value Meaning
1 Success.
2 A folder with that name already exists.
3 Folder is a special folder.
4 Folder not found.
5 The specified parent folder was not found.
6 An error on the Exchange server occurred.
7 Access denied.
8 The request timed out.
9 Sync key mismatch or invalid sync key.
10 Misformatted request.
11 An unknown error occurred.

55. Example of PING Server Response

56. Activesync - troubleshooting
Scoping questions:
• Is the device reaching the Internet facing CAS?
• Are all mobile devices affected?
• Which CAS do we need to troubleshoot?
• Is this an issue that’s well known?
architecture connectivity
troubleshooting performance

57. Activesync - troubleshooting
Troubleshooting service
• the browser test
https://CAS.contoso.com/microsoft-server-
activesync/default.eas
https://mail.contoso.com/microsoft-server- architecture connectivity
activesync/default.eas
[501 method not implemented is the expected
response]
troubleshooting performance

58. Activesync - troubleshooting
https://www.testexchangeconnectivity.com
Test-ActiveSyncConnectivity
Event logs (Source: MSExchange ActiveSync)
architecture connectivity
IIS logs (requests to /microsoft-server-activesync)
EAS Mailbox device logging
Windows Mobile emulator
troubleshooting performance
Failed request tracing
Perfmon

59. https://www.testexchangeconnectivity.com

60. Test-ActiveSyncConnectivity cmdlet

61. Event Log Example

62. W3SVC Log Example
_Fid:10_Ty:Em_Filt3_S
t:S_Sk:2063964464_SsCmt1_Srv:6a0c0d0s0e0r0A0sd_BR1_BPR0_
_LdapC23_RpcC116_RpcL203_Pk1087184048_S1_As:AllowedG_Mbx:E2K10M.x.ExchLab.loc
al_Throttle0_Budget:(

63. W3SVC Log Breakdown - Elements
Letter Element
identifier name Definition Possible values
V Protocol The protocol version the device is Value Meaning
version using to synchronize with the 120 Version 12
Exchange server. 25 Version 2.5
21 Version 2.1
20 Version 2.0
10 Version 1.0
Ty Type The type of folder that's being Value Meaning
synchronized. Em E-mail
Co Contacts
Ca Calendar
Ta Tasks
Fid Folder ID The ID of the folder that's being Positive Integer
synchronized.
Fc Folder count The number of folders that are Positive Integer
being synchronized.
Filt Filter type The data that the user requested. Value Meaning E-mail? Calendar?
Tasks?
0 No filter Yes Yes Yes
1 1 day back Yes No No
2 3 days back Yes No No
3 1 week back Yes No No
4 2 weeks back Yes Yes No
5 1 month back Yes Yes No
6 3 months back No Yes No
7 6 months back No Yes No
8 Incomplete No No Yes

64. W3SVC Log Breakdown - Elements
St Sync type The type of synchronization that's being performed. Value Meaning
F First sync
S Subsequent
R Recovery sync
I Invalid sync
Sk Sync key The actual sync key that's used between the mobile phone and Positive integer
the Exchange server.
Cli: Client Stores the count of each type of activity from the Client. Output Identifier value
statistics is in the form Cli: 0A0C3D1F0E. Meaning
A Adds
C Changes
D Deletes
F Fetches
E Errors
Svr: Server Stores the count of each type of activity from the server. Output Identifier Meaning
statistics is in the form Svr:2A0C2D1F1E. A Adds
C Changes
D Deletes
F Fetches
E Errors
E Number of The number of errors encountered in a request. Positive integer
errors
Io Items opened The number of items that were opened. This feature hasn't yet Positive integer
been implemented.
Hb Heartbeat The Heartbeat interval that's used for the PING command. Positive integer
interval

65. W3SVC Log Breakdown - Elements
Ssp SharePoint The number of files that were accessed from Windows Positive integer
documents SharePoint Services.
Sspb SharePoint bytes The number of bytes that were accessed from Windows Positive integer
SharePoint Services.
Unc UNC files The number of files that were accessed through Windows Positive integer
file shares.
Uncb UNC bytes The number of bytes that were accessed through Windows Positive integer
file shares.
Att Attachments The number of attachments that were retrieved. Positive integer
Attb Attachment bytes The number of bytes that were retrieved for attachments. Positive integer
Pk Policy key The element that's used by the client and server to Not applicable
received correlate acknowledgements to a particular policy setting.
Pa Policy The element that indicates success if all the policy settings Value Meaning
acknowledge were applied correctly. 1Policy was
status successfully applied
2Policy was partially
applied
3Policy was not
applied

66. W3SVC Log Breakdown - Elements
Oof OOf action The action that is performed on the Out of Value Meaning
Office status stored on the Exchange GetRetrieves the OOF status and
server. message
SetSets the OOF status and
message
UserInfo User The parameter that specifies retrieval of Get
information the user information data.
action
DevModel Device model The device information that is supplied by Possible values include
the device manufacturer. manufacturer name, model
name, and model number.
DevIMEI IMEI The International Mobile Equipment String
Identity (IMEI). It is a 15-digit code that's
assigned to each device.
DevName Device friendly This element stores the user's description String
name of their device.
DevOS Device OS The operating system that is running on String
the device.
DevLang Device OS The localized language of the device String
language operating system.
Error Error The error section of the request. String
S Status This element returns the status of the String
device.
R Not Relevant This element returns a count of items that Positive integer
have changed but aren't relevant to the
mobile phone or device.

67. W3SVC Log Breakdown - Elements
Pfs PerFolderStatus
BR BodyRequested
BPR BodyPartRequested
LdapC LdapCount
LdapL LdapLatency
RpcC RpcCount
RpcL RpcLatency
E NumErrors
Io NumItemsOpened

68. W3SVC Log Breakdown - Elements
DevAgent DeviceInfoUserAgent
Rto RequestTimedOut
Erq EmptyRequest
Ers EmptyResponse
Cpo CompletionOffset
Fet FinalElapsedTime
DevEnaSMS DeviceInfoEnableOutboundSMS
DevMoOp DeviceInfoMobileOperator

69. W3SVC Log Breakdown - Elements
RR NumberOfRecipientsToResolve
Fb "Fb"=AvailabilityRequested
Ct CertificatesRequested
Pic PictureRequested
As AccessStateAndReason
Ssu Ssu
Mbx MailboxServer
Dc DomainController
Throttle ThrottledTime

70. W3SVC Log Example
_Fid:10_Ty:Em_Filt3_S
t:S_Sk:2063964464_SsCmt1_Srv:6a0c0d0s0e0r0A0sd_BR1_BPR0
_LdapC23_RpcC116_RpcL203_Pk1087184048_S1_As:AllowedG_Mbx:E2K10M.x.ExchLab.loc
al_Throttle0_Budget:(

71. W3SVC Log Example Breakdown Server Stats
Adds 6
Protocol Version 14.1
Changes 0
Type E-mail
Deletes 0
Folder ID 10
Soft-Deletes 0
Folder Count 5
Errors 0
Filter Type 3 days back
LDAPCount 23
Sync Type Subsequent sync
RPCCount 116
Sync Key 2063964464 RPCLatency 203
Status Success PolicyKey 1087184048
BodyRequested 1 Status 1
AccessStateandReason Allowed
BodyPartRequested 0
Mailbox E2k10
Throttle 0

72. W3SVC Log Sample – Break it Down!
Example Ping command:
&Log=V120_Hb780_S1

73. W3SVC Log – Too Easy!
Protocol Version 12
Heartbeat Interval 780 sec (13min)
Status 1 (Success)

74. Log Parser Query and Results

75. Export-ActiveSyncLog Example

76. Export-ActiveSyncLog Example

77. Get-ActiveSyncDevice cmdlet

78. EAS Mailbox Logging
http://msexchangeteam.com/archive/2007/05/30/439568.aspx

79. EAS Mailbox Logging

80. EAS Mailbox Logging

81. EAS Mailbox Logging

82. EAS Mailbox Logging – WP7
Log Entry: 70
-----------------
RequestTime :
10/20/2011 11:00:19
Identifier :
70F0FE13

83. EAS Mailbox Logging – WP7
MS-ASProtocolVersion: 14.1

84. EAS Mailbox Logging – WP7
<Sync xmlns="AirSync:">
<HeartbeatInterval>1380</HeartbeatInterval>
</Sync>
WasPending :
[Response was pending]

85. EAS Mailbox Logging – WP7
ResponseHeader :
HTTP/1.1 200 OK
MS-Server-ActiveSync: 14.1
<SyncKey>268775212</SyncKey>
<CollectionId>5</CollectionId>
<Status>1</Status>
<Commands>
<Add>
<ServerId>5:11</ServerId>
<ApplicationData>
ResponseTime :
10/20/2011 11:01:46

86. EAS Mailbox Logging – WP7
<Sync xmlns="AirSync:">
<HeartbeatInterval>1380</HeartbeatInterval>
<Partial/>
</Sync>

87. EAS Mailbox Logging - iPhone
Log Entry: 61
-----------------
RequestTime :
10/20/2011 12:29:45
Identifier :
6E3B9610
WasPending :
[Response was pending]

88. EAS Mailbox Logging - iPhone
ResponseHeader :
HTTP/1.1 200 OK
MS-Server-ActiveSync: 14.1
<Status>2</Status>
<Folder>5</Folder>
ResponseTime :
10/20/2011 12:30:30

89. EAS Mailbox Logging - iPhone
Log Entry: 62
-----------------
RequestTime :
10/20/2011 12:31:01
<CollectionId>5</CollectionId>
<GetChanges/>

90. EAS Mailbox Logging - iPhone
ResponseHeader :
HTTP/1.1 200 OK
MS-Server-ActiveSync: 14.1
<SyncKey>2657206</SyncKey>
<Add>
<ServerId>5:10</ServerId>

91. EAS Mailbox Logging - iPhone
Log Entry: 63
-----------------
RequestTime :
10/20/2011 12:31:01
Identifier :
3BB1439B
Cmd=Sync
<SyncKey>2657206</SyncKey>
<CollectionId>5</CollectionId>
<Fetch>
<ServerId>5:10</ServerId>
</Fetch>

92. EAS Mailbox Logging - iPhone
ResponseHeader :
HTTP/1.1 200 OK
MS-Server-ActiveSync: 14.1
ResponseBody :
<?xml version="1.0" encoding="utf-8" ?>
<Sync xmlns="AirSync:">
<Collections>
<Collection>
<SyncKey>530022051</SyncKey>
<CollectionId>5</CollectionId>
<Status>1</Status>
<Responses>
<Fetch>
<ServerId>5:10</ServerId>
<Status>1</Status>

93. EAS Mailbox Logging – iPhone ???
<SyncKey>644101135</SyncKey>

94. EAS Mailbox Logging – iPhone ???
</Sync>
SyncCommand_GenerateResponsesXmlNode_AddChange_ConvertServerToClientObject_Exception :
Microsoft.Exchange.AirSync.ChangeTrackingItemRejectedException
at Microsoft.Exchange.AirSync.ChangeTrackingFilter.Filter(XmlNode xmlItemRoot, Nullable`1[] oldChangeTrackingInformation)
at Microsoft.Exchange.AirSync.SyncCollection.ConvertServerToClientObject(ISyncItem syncItem, XmlNode airSyncParentNode,
SyncOperation changeObject, GlobalInfo globalInfo)
at Microsoft.Exchange.AirSync.SyncCollection.<>c__DisplayClassd.<GenerateCommandsXmlNode>b__4(SyncOperation
changeObject)
<SyncKey>644101135</SyncKey>

95. Log Entry: 69
-----------------
RequestTime :
10/20/2011 12:49:23
ServerName :
E2K10CH
AssemblyVersion :
14.01.0325.000
Identifier :
7FF1CC78
& Cmd=Ping
X-Ms-Policykey: 2891930116
<Ping xmlns="Ping:">
<HeartbeatInterval>700</HeartbeatInterval>
</Ping>

96. EAS Mailbox Logging - iPhone
Log Entry: 70
-----------------
RequestTime :
10/20/2011 13:01:53
Identifier :
24B088EB
&Cmd=Ping
X-Ms-Policykey: 2891930116
<Ping xmlns="Ping:">
<HeartbeatInterval>801</HeartbeatInterval>
</Ping>

97. EAS Mailbox Logging - iPhone
Log Entry: 71
-----------------
RequestTime :
10/20/2011 13:15:21
Identifier :
47C28128
& Cmd=Ping
X-Ms-Policykey: 2891930116
<Ping xmlns="Ping:">
<HeartbeatInterval>700</HeartbeatInterval>
</Ping>

98. EXTRA

105. http://blogs.technet.com/b/exchange/archive/2007/09/17/3403937.aspx

112. appPoolId="MSExchangeSyncAppPool"
statusCode="401.3"
triggerStatusCode="401.3"
timeTaken="0"

113. ">C:Program FilesMicrosoftExchange ServerV14ClientAccesssyncdefault.eas<
FILE_CACHE_ACCESS_START

114. <Data Name="ContextId">{00000000-0000-0000-CB00-0080000000F5}</Data>
<Data Name="Successful">false</Data>
<Data Name="FileFromCache">false</Data>
<Data Name="FileAddedToCache">false</Data>
<Data Name="FileDirmoned">true</Data>
<Data Name="LastModCheckErrorIgnored">true</Data>
<Data Name="ErrorCode">2147942405</Data>
<Data Name="LastModifiedTime"></Data>
<Opcode>FILE_CACHE_ACCESS_END</Opcode>
<freb:Description Data="ErrorCode">Access is denied.
(0x80070005)</freb:Description>

115. Activesync - performance
Throttling
• EASMaxConcurrency : 10
• EASPercentTimeInAD :
• EASPercentTimeInCAS :
architecture connectivity
• EASPercentTimeInMailboxRPC :
• EASMaxDevices : 10
• EASMaxDeviceDeletesPerMonth :
troubleshooting performance

116. Activesync - performance a c
t p
Trending analysis
• using AD tools since partnership is kept in leaf object
Csvde –d “cn=users,DC=Contoso,DC=com” –r (objectclass=msexchactivesyncdevice) -l
dn,msExchDeviceUserAgent,whenChanged,whenCreated –f
c:allExchange2010mobiledevicepartnerships.csv
"CN=iPhone§Appl87831W4QY7H,CN=ExchangeActiveSyncDevices,CN=e14MobileTester,CN=Users,DC=Contoso,DC=com",20101111173928.0Z,20101111173948.0Z,Apple-iPhone1C2/802.117
"CN=PocketPC§BAD73E6E02156460E800185977C03182,CN=ExchangeActiveSyncDevices,CN=e14manager,CN=Users,DC=Contoso,DC=com",20101231183218.0Z,20101231183326.0Z,MSFT-
PPC/5.2.5001
"CN=WP§C01D49121ABAFAFD3C72924235668667,CN=ExchangeActiveSyncDevices,CN=wp7user,CN=Users,DC=Contoso,DC=com",20110421115008.0Z,20110421115100.0Z,MSFT-WP/7.0.7390
"CN=iPhone§Appl87831W4QY7H,CN=ExchangeActiveSyncDevices,CN=iuser01,CN=Users,DC=Contoso,DC=com",20110426120447.0Z,20110426120505.0Z,Apple-iPhone1C2/803.148
…
• Compare this to the shell approach. From Management Shell
Get-Mailbox alias | Get-ActivesyncDeviceStatistics | ft identity,DeviceType,DeviceModel

117. Activesync - performance
Log Parser Studio
#demo
architecture connectivity
troubleshooting performance

118. Performance Monitor a
t
c
p
http://technet.microsoft.com/en-us/library/ff367877.aspx
http://technet.microsoft.com/en-us/library/ff367871.aspx