This is the presentation deck file that I gave at CRMUG Summit 2015 on Oct 16, 2015. In this session, I talked about some best practices that you can use to speed up your Dynamics CRM data migration and integration.

1. #CRMUGSummit | #INreno15
#CRMUGSummit
Daniel Cai, KingswaySoft Inc.
SESSION CODE: ADC26
GETTING DATA INTO MICROSOFT DYNAMICS
CRM FASTER

2. #CRMUGSummit | #INreno15
INTRODUCTION - YOUR PRESENTER
 Daniel Cai
 Principal Developer @KingswaySoft
 7 Years of development experiences with Microsoft Dynamics CRM
– Mostly working as a developer
 5 times of Microsoft Dynamics CRM MVP since 2011
 About KingswaySoft
– A leading provider of data integration solutions for Microsoft Dynamics software and
beyond
• SSIS Integration Toolkit for Microsoft Dynamics CRM
• SSIS Integration Toolkit for Microsoft Dynamics AX
• SSIS Integration Toolkit for Microsoft Dynamics GP
• SSIS Integration Toolkit for Microsoft Dynamics Marketing
• SSIS Integration Toolkit for Microsoft Dynamics NAV
• SSIS Integration Toolkit for Microsoft Dynamics SL
• SSIS Integration Toolkit for Parature
• and many more…
– Thousands of enterprise clients worldwide depend on our integration solutions to
drive their data efficiency
2

3. #CRMUGSummit | #INreno15
 Discuss the techniques that you can use to get
data into Microsoft Dynamics CRM faster
 Discuss some best practices that you utilize in
your CRM data integration and migration
projects
3
SESSION OBJECTIVES

4. #CRMUGSummit | #INreno15
 SSIS Integration Toolkit is used in some occasions in this presentation to
discuss the techniques
– It is a data integration solution developed by KingswaySoft that I work for
– More information available at http://www.kingswaysoft.com/products/ssis-integration-
toolkit-for-microsoft-dynamics-crm
 The techniques should be transferrable to any other third-party or home-
grown solutions in a similar fashion, provided an equivalent option is
available
 I didn’t invent any tips or tricks, neither did I discover all of them by myself,
they all come from the community that I have been learning. The credits go
to the generous CRM community and some special individuals.
4
DISCLAIMER

5. #CRMUGSummit | #INreno15
 WHY?
 WHERE TO START FROM?
 HOW?
 Final Q&A
 Closing & Surveys
5
AGENDA

6. WHY?

7. #CRMUGSummit | #INreno15
 Web Service Interfaces are generally slow
– Dynamics CRM is no exception
 You are often constrained by the limited time window
– A couple to a few hours time window for your daily integration
– Typically a ~50 hours of time window for the initial data migration
• Friday 9pm to Monday 6am
 Your data volume could be significantly large
 You always want to be faster!
7
WHAT ARE THE PROBLEMS

8. WHERE TO START FROM?

9. #CRMUGSummit | #INreno15
 Identify Resources
– Technical Resources
 Get to know your infrastructure
– Networking
– Server Capacities
 Define a baseline benchmark
– How many records in a second are required in order to meet the time window allowed
 Define the infrastructure that can help you achieve the desired benchmark
 Decide between Buy vs Write
9
DEFINE YOUR STRATEGIES

10. #CRMUGSummit | #INreno15
 Optimized Infrastructure
– Networking
– Firewall
– Security Policies
– Network Load Balancer
 Server Capacity (CRM Server, Database Server, Integration
Server)
– CPU
– Memory
– I/O
10
HAVING THE RIGHT INFRASTRUCTURE

11. #CRMUGSummit | #INreno15
 CRM Data Import Wizard
 Message-driven Integration Solutions
– BizTalk
– Service Bus
– Message Queue
 Batch-based ETL Solutions
– SSIS
– Scribe
– Informatica
– IBM CastIron
11
CHOOSE THE RIGHT TOOLS

12. #CRMUGSummit | #INreno15
 Each tool works somewhat differently
 Establish a good understanding of your tools
– Know the capabilities
– Know the different options to achieve a particular
purpose
– Know the techniques that can be used to help achieve
better performance
12
TRY TO KNOW YOUR TOOLS

13. HOW?

14. #CRMUGSummit | #INreno15
MULTI-THREADING
Make sure to update your .config file in order to
overcome the connection limit of concurrent
service calls
<configuration>
...
<system.net>
<connectionManagement>
<add address="*" maxconnection="100"/>
</connectionManagement>
</system.net>
</configuration>
TheSSIS way

15. #CRMUGSummit | #INreno15
LEVERAGE MULTI-NODE CLUSTER
How to achieve this?
• CRM Server URL option in SSIS Integration Toolkit

16. #CRMUGSummit | #INreno15
BULK API
 Bulk API is more beneficial when
network latency is high
 There is an API throttling with CRM
Online
– At most 2 concurrent ExecuteMultiple
requests per organization
How to achieve this?
• Simply enter a Batch Size in SSIS destination
component.

17. #CRMUGSummit | #INreno15
 The less fields, the better performance
 Some fields may require additional services calls using special SOAP messages when
writing to CRM
– statecode/statuscode
– ownerid
– some more
• parentsytemuserid and businessunitid (systemuser entity)
• businessunitid (team and equipment entities)
• Parentbusinessunitid (businessunit entity)
– Those special service calls are not necessary if it is an Update and you are using v7.1 or later
• For Create or Upsert calls, you still need the additional service calls (as of v7.1).
 Any fields registered for CRM plugins/workflows or auditing will have certain performance
impact when submitted
17
MINIMIZE THE NUMBER OF FIELDS YOU WORK WITH

18. #CRMUGSummit | #INreno15
 Why?
– Smaller SOAP message, which in turn results in less networking time
– Can avoid firing unnecessary CRM plugins/workflows or auditing
that are registered for the fields
 How to achieve this?
– Simply turn on “Ignore Unchanged Fields” option in CRM destination
component
18
AVOID SUBMITTING UNCHANGED FIELDS

19. #CRMUGSummit | #INreno15
 CRM Diagnostics Tool
– http://CrmServer/tools/diagnostics/diag.aspx
 Ideally the latency should be as low as
possible
– Make sure the integration server (or
workstation) is close to the CRM Server
– A poor infrastructure can result in high
network latency, even the servers are close
to each other
 Bandwidth should be as high as possible
19
MIND YOUR NETWORK LATENCY

20. #CRMUGSummit | #INreno15
 Which servers
– CRM Server
– Database Server
– Integration Server (or workstation)
 What to watch
– CPU
– Memory
– I/O
– Other CRM performance counters
20
WATCH OUT RESOURCE USAGES ON SERVERS

21. #CRMUGSummit | #INreno15
 Plan CRM database growth
– Have the right server specification to accommodate the growth
– Define data archiving strategies as required
– For initial load, consider sizing the database to the target size after the completion of the load to avoid database growth during
the load
 Cleanup PrincipalObjectAccessBase table as required (to remove orphan records)
– http://support.microsoft.com/kb/2664150
• Consider implementing this as a SQL Server agent job to cleanup weekly or monthly to keep your system healthy
 Cleanup AsyncOperationBase table as required (to remove completed workflow logs)
– http://support.microsoft.com/kb/968520 for one-time cleanup
• Make sure you implement the index first, otherwise it could a long time to complete depending on how many records you have
• Consider implementing this as a SQL Server agent job to cleanup weekly or monthly to keep your system healthy
– For CRM workflows, consider choosing “Automatically delete completed workflow jobs (to save disk space)” option for automatic
cleanup
– For asynchronous plugin, consider choosing “Delete AsyncOperation if StatusCode = Successful” option for automatic cleanup
21
MANAGE DATABASE GROWTH

22. #CRMUGSummit | #INreno15
 If your data integration or migration process involves any querying, you should review your query performance
 Depending on the situation, if any query involves non-indexed CRM database field(s), you might want to add custom
index(es)
– CRM On-Prem, add custom indexes to CRM database directly
– CRM Online, request adding custom indexes by raising a support ticket. Alternatively, use the following techniques to add custom
indexes automatically.
• Add the field to CRM quick find view
• Create CRM alternate key – will have to be unique
 Note that having an excessive number of indexes in one database table (entity) can adversely affect your performance
 Where to start from?
– Look for long running queries in CRM Server Windows Log
• CRM logs any database queries that take longer than 10s
– SQL Server “Missing Index” tables
 Query Tuning
– Show Actual Execution Plan – SQL Server Management Studio (SSMS)
– SQL Server Database Engine Tuning Wizard
22
MANAGE DATABASE INDEXES

23. #CRMUGSummit | #INreno15
 Reschedule CRM Maintenance Jobs
– By default, the time when CRM maintenance jobs kick off
depends on when the server was installed (or the
organization being provisioned) – which could be in busy
business hours
– Ideally your integration job kicks off shortly after the
maintenance job finishes
 Disable the Reindex All job
– What it does?
• Reindex
• Shrink your CRM database
– Create your own CRM database maintenance jobs
23
MANAGE CRM MAINTENANCE JOBS

24. #CRMUGSummit | #INreno15
 Manual primary keys could cause performance issue
down the road
– CRM Server generates sequential GUID values for best
performance
– Use manual primary keys only for migration purpose
 Further read:
– The Dangers of Guid.NewGuid();
• http://blogs.msdn.com/b/crminthefield/archive/2015/01/19/the-dangers-of-guid-
newguid.aspx
24
AVOID GENERATING PRIMARY KEY (GUID) YOURSELF

25. #CRMUGSummit | #INreno15
 Turn on SQL Server RCSI (Read Committed Snapshot Isolation)
– https://msdn.microsoft.com/en-us/library/tcbchxcb(v=vs.110).aspx
ALTER DATABASE CrmOrgName_MSCRM
SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE CrmOrgName_MSCRM
SET READ_COMMITTED_SNAPSHOT ON
 Add NOLOCK hint to your FetchXML Query
<fetch mapping="logical" no-lock="true">
<entity name="account">
<attribute name="name" />
<filter>
<condition attribute="creditlimit" operator="gt" value="1000000" />
</filter>
</entity>
</fetch>
25
DEAL WITH LOCKS

26. #CRMUGSummit | #INreno15
 Help reduce database
deadlocks
 Recommended for CRM
database, but not for data
warehouse
26
SQL SERVER – MAX DEGREE OF PARALLELISM

27. #CRMUGSummit | #INreno15
 Set CRM database to Simple Recovery mode
– No SQL Server transaction logs
 Disable IIS Logging
 Disable all CRM workflows / plugins and auditing
 Running data integration on CRM server itself
– Best possible network latency
– You need to alternate the target service URL to point to local server as
shown previously
27
EVEN MORE AGGRESSIVE MEASURES – SOME MAY
ONLY BE PRACTICAL FOR INITIAL LOAD

28. #CRMUGSummit | #INreno15
 White Papers
– Microsoft Dynamics CRM 2015 White Papers & Technical Documentation
• http://blogs.msdn.com/b/crminthefield/archive/2015/01/05/microsoft-dynamics-crm-2015-white-papers-amp-technical-
documentation.aspx
– Optimizing and maintaining the performance of a Microsoft Dynamics CRM 2011 server infrastructure
• http://www.microsoft.com/en-ca/download/details.aspx?id=27139
 Wikipedia
– Dynamics CRM Quick Optimization Guide
• http://social.technet.microsoft.com/wiki/contents/articles/13661.dynamics-crm-2011-quick-optimization-guide.aspx
 Blog Posts
– http://blogs.msdn.com/b/crminthefield/archive/2012/04/26/avoid-performance-issues-by-re-scheduling-crm-2011-
maintenance-jobs.aspx
 Best Practices
– Best practices for developing with Microsoft Dynamics CRM
• https://msdn.microsoft.com/en-us/library/gg509027.aspx
28
RESOURCES

29. #CRMUGSummit | #INreno15
 Daniel Cai, KingswaySoft
– daniel.cai@kingswaysoft.com
– Twitter: danielcai
– LinkedIn: https://ca.linkedin.com/in/danielcai
– http://www.kingswaysoft.com
29
CONTACT INFO

30. #CRMUGSummit | #INreno15
THANK YOU!
 Please complete & turn in your survey now
 Session code: ADC26
 Visit the online CRMUG Summit 2015 community to
download these slides, ask questions, and connect
with participants
 Visit the CRMUG Medics in the HUB to get expert
advice on your CRM-related problems!
30