• Is your organization using any Enterprise Content Management systems besides SharePoint?
• Has your current ECM system been deprecated or require an expensive annual maintenance contract?
• Does your firm already use Microsoft SharePoint as an intranet/collaboration portal?
• Would you like to leverage the cutting edge ECM and taxonomy features in SharePoint 2010 or 2013?
If so, it may be time to migrate your scanned and other transactional documents from your legacy ECM system to SharePoint, and take advantage of innovative ECM and taxonomy features available in this powerful platform.
Top industry experts and influencers Joel Oleson and Tom Castiglia from Hershey Technologies will explain best practices to plan and implement a successful ECM content migration to Microsoft SharePoint. Join us on June 4th to learn about:
• Using ECM and taxonomy related features in SharePoint 2010 and 2013
• Reasons to migrate content to SharePoint (as well as reasons not to migrate!)
• Best practices for ECM architecture in SharePoint including security, taxonomy and governance
• 14 specific tips for planning your migration from any ECM system to SharePoint
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
14 Tips for Planning ECM Content Migration to SharePoint
1. 14Tips for Planning an ECM
Content Migration to SharePoint
Joel Oleson
HersheyTechnologies
@joeloleson
2. About HersheyTechnologies
• Founded in 1991
• Certified Microsoft Partner
• IT Solution Provider with deep expertise in…
• End to End SharePoint Consulting Services
• Document and Image Capture:
• Scanning and Imaging
• OCR solutions for automated document and
data capture
• ECM and Document Management
• BPM /Workflow
• Contact us…
• Follow us onTwitter: @HersheyTech
• www.hersheytech.com
• info@hersheytech.com
3. Agenda
•SharePoint ECM – SharePoint Online & SharePoint 2016
•Why migrate content from legacy ECM systems into
SharePoint?
•14 Planning tips for ECM content migration to SharePoint
4. About Joel Oleson
• Director of Marketing & Product Strategy @ Hershey
Tech
• AdventureTraveler Explorer
• First SharePoint Admin
• Rode a bike down the World’s Most Dangerous Road
• Architecting Customer ECM Solutions since 2000
• Launched SP Communities in Jamaica, Kenya, Jordan,
Philippines, Uruguay, and many more…
• Sharing best practices blogging, speaking, tweeting…
@joeloleson
Collabshow.com
Joleson@hersheytech.com
8. SharePoint Online – Office 365
• Your company may already have licenses… 80% of SharePoint
CALs cover SharePoint Online
• TB of storage per user in OneDrive for Biz
• 10 GB file upload
• Always up to date… Microsoft does the upgrades without
additional fees
• You get it first! O365 early adopter customers have had access to
Delve for over a year!
• Mobile just works
9. Coming Soon… SharePoint 2016 !
1.Durable Links
2.Compliance – DLP is the key term. Monitoring, and
protection of data through deep analysis of content.
3.Increased Boundaries – List views increased, single file
sizes up to 10GB,TB databases, and list thresholds
4.Minimal patching with Zero Downtime
5.Hybrid Deployment Automation
11. Tip # 1 – Information Architecture is Key -
Consider Refactoring such as splitting one legacy
repository into multiple SharePoint libraries
AVOID
item level
permissions
Reduce number
of documents
per library
(improves
performance)
SharePoint
allows unified
search across
multiple
libraries
File Share SP List
12. Vendor PO # Invoice # Division
Alpha, LLC 3456617 74584 ACSC
Bravo, Inc 3456633 88363 ACMO
Charlie Co. 3456641 56546454 ACSC
Alpha, LLC 3456648 74584 ACSC
Delta Signs 3456652 675676 ACTX
Echo Ink 3456661 INV-324454 ACTX
Bravo, Inc. 3456670 456546464 ACMO
Vendor PO # Invoice #
Alpha, LLC 3456617 74584
Charlie Co. 3456641 56546454
Alpha, LLC 3456648 74589
ACSC
Invoices
Vendor PO # Invoice #
Delta Signs 3456652 675676
Echo Ink 3456661 INV-324454
Vendor PO # Invoice #
Bravo, Inc 3456633 88363
Bravo, Inc. 3456670 456546464
ACMO
Invoices
ACTX
Invoices
Legacy Repository
SharePoint Libraries
Tip # 1 - Use folders, libraries or content types
for security
13. Tip #2 Permissions/Security Considerations
•Consider Adding AD Groups to SP Groups (instead of AD
Users)
• Adding AD users to SP Groups causes incremental/continuous
crawl to update to crawl ACLs. Crawl may go from 3 minutes
to 2 hours.
CC: Helloturkeytoe
14. Tip #3 If you want Scale you need to
PLAN!
• Even if you are migrating content with rich metadata, plan to import the
content into SharePoint using folders.
• SharePoint allows around 25M documents per library
• With 5,000 root level folders and 5,000 documents per folder this gives you 25M
documents/library
• Without use folders, Incremental/ContinuousCrawls may perform a table scans
• Easiest approach is to use Content Organizer Rules
17. Tip # 5 – Optimize…
Consider converting “choice” fields into Managed
Metadata, Lookup or External data columns
Managed
Metadata
Lookup External Data
(BCS)
Hierarchical YES NO NO
Reference
other columns
NO YES YES
Scope Farm Site Collection Enterprise
18. Tip # 6
Normalize with site columns or content types
for consistency
Policy#
Policy No
PolicyNumber
Legacy ECM System
Policy Number
SharePoint Site
ColumnRepository Field Name
Auto Policies PolicyNumber
Medical Policies PolicyNo
Home Policies Policy#
19. Tip # 7 – Don’t migrate Junk…
Age out, Archive, and Optimize…
“Garbage in = Garbage Out!”
cc: Swamp school
20. Tip # 8 – Understand the Environment:
Use Cases
Workflow automation and validation
EnsuringTerms and Metadata consistency
Ensuring there is NO Data or Metadata loss during migration
What does inbound data & capture look like? Scan/Fax/Email
Are there any specialized indexing needs
21. Tip # 9 - Understand the Environment:
Key Metrics
Dbs, folders, files and items
Structure and size of repository
Rate of change
Growth over time
Usage rates and concurrency
22. Tip #10: Migration Gotchas!
•Invalid Characters in the filenames
•Error handling
•Broken Links
•File Size
•Compliance and Sensitivity of documents
23. Migration Gotchas
File Conversions or third party viewer may be needed
Scanned
Documents
PDF
• Searchable
• Image-Only
TIF
• Multi-Page (1 file
per document)
• Single-Page
(multiple files per
document)
Engineering
Auto-Cad
• Dwg
•
Email and
Attachments
msg
eml
Multimedia
(Consider non MS
formats)
Audio
• mp3
• wmv
• wav
Video
• avi
• mpeg
• mp4
24. TIP #11 Infrastructure Considerations for
large scale
•Lessen impact existing SharePoint site collections when
starting large ECM migration
• Allocate dedicatedWEB front end servers to support higher
throughput
• Allocate dedicated SQL Server for legacy content
to host content DBs to avoid disk contention
timothymorgan
25. Tip # 12 –
Consider extending Search… Buy or Build
Consider Search
Refinement and
Rich filtering
SharePoint formats
Search results like a
“search engine”,
not a DMS.
SharePoint
metadata filtering
does not scale for
large libraries
26. If you put something in how will you find it?
How many docs
are returned in
typical query?
Save queries for
re-use?
Search metadata
vs. full-text
How many times
per day is
repository
queried?
What type of users
are searching?
28. Tip #13 –
XenDocs ECM SearchWithVizit™
Intuitively build
precise document
queries
Sortable search
results
Image previews
for scanned
images
29. Tip # 14 –
Understand migration tool options
• Existing 3rd party tools vs Custom tools
• Field mapping (different field names between source system and SharePoint)
• Filters to allow migrations to be performed in chunks
• E.g. Only migrate documents where DocType=‘Expense Report’ and
DocDate>’02/13/2009’
• Control: Ability to manage and audit status of every record
• File Conversions
• Single PageTIF to Multi-PageTIF
• TIF to PDF
• Image only PDF to Searchable PDF
• PDF to PDF/A
30. Advantages of 3rd party MigrationTools
• Allows you to skip versions. 2007 to 2013 or 2010 to 2016
• Incremental copy
• Fix broken and relative links
• Fix security issues
• Optimize for Search
• Remove Junk
• Refactor – Promote/demote, move/copy, split
• Remove legacy dependencies on code
33. In Closing…
•Hope these tips were helpful, if you need more help,
lets discuss…
•Demo of various migration tools
•Office 365 Hybrid strategy
•Metadata Search optimization
•Adoption strategy
At Hershey Technologies, we’ve been providing IT solutions for more than 20 years. We have deep, end-to-end SharePoint consulting and development expertise, and we specialize in document capture, document management and workflow solutions!
We’re also resellers for a number of major SharePoint ISVs such as Nintex, AvePoint, and Vizit. We’re also a Diamond partner with Kofax.
I’d like to note that today’s webinar is not going to be a “product pitch”. The content today will focus on how to plan a migration of content from another ECM system to SharePoint. Although I will reference a few software tools that can automate your migration, in my experience the success of most migrations to SharePoint has more to do with how SharePoint is architected and how the migration process was planned than it has to do with the tools used to perform the migration.
I’d like to make today’s presentation more interactive than a typical webinar. So please chime in with any questions using the “chat” window. The content today assume that you have some familiarity with SharePoint. However, if you would like me to explain further about specific SharePoint features and how they might be relevant to your migration plans, please let us know through the chat window.
Before we get started, I want to ensure our terms are clear.
In the SharePoint community, the term “migration” can have different meanings.
Often people use it to refer to an upgrade of SharePoint version X to version Y. However, for this presentation, I am referring to migration of content (documents and metadata) from some other ECM (non-SharePoint) platform into SharePoint
There has been a great deal of consolidation in the ECM market. Many of the larger vendors have purchased smaller competitors. In many cases, the new owner’s primary goal is not to continue innovating on the new product, but to push customers to move to main product line. A classic example of this is EMC, who owns Documentum and ApplicationXtender. Customers on the AX platform tell us how there are no longer new features added, and EMC wants them to move to Documentum.
But if you are an AX shop and you already have a mature SharePoint installation, its far more cost effective to migrate to SharePoint then Documentum.
Over the last few years we have seen increased interest from our clients in migrating content from what I’ll call “legacy” ECM products to SharePoint.
So if you are in this webinar, I presume that you already are using SharePoint or that your company is already planning to use SharePoint. The purpose of this presentation is focus on “how” to plan your migration, not “why” you should migrate to SharePoint.
That being said, I’ll review some of the key reasons why so many companies are deciding to migrate ECM content to SharePoint…
Content Types in SharePoint let us group related metadata columns into a re-usable template. This allows us to de-couple the metadata schema from document libraries, and gives us flexibility that many other ECM products don’t have.
A content type can be re-used in multiple document libraries and conversely, a single document library can store documents of different content types.
Sometimes a legacy ECM system may have a small number of very large repositories that ideally should be re-factored into multiple SharePoint libraries. Some document management systems might not allow a unified search across multiple repositories, and so documents get grouped into a single large repository to simplify the end user search experience.
This approach often forces systems to use document level security models which may reduce the scalability of the system. There may be business requirements to separate documents into various document libraries, by department, business unit or some other metadata attribute.
Although SharePoint also supports item-level permissions, using item-level permissions can limit scalability. So if your current system is using item level permissions, consider refactoring the system into multiple repositories to avoid use of item level permissions.
When your ECM solution in SharePoint is used for archive (read-only) purposes only, SharePoint can be support millions of documents in a single document library. But if your documents are also used in collaboration (read-write) scenarios or you have workflow processes on your documents, you need to architect your SharePoint taxonomy to keep the number of documents per library to much smaller numbers, which may require creating more libraries that you have in your legacy ECM system.
Many ECM document repositories have a metadata choice field named “Document Type”, which is typically a drop down list (choice) field of possible document types. This design can be migrated directly to SharePoint. However, sometimes documents should be treated differently based on the “document type”. For example, say we have repository for “contracts”, with a field named “contract type” containing options for: NDA, MSA, Maintenance Agreement, Partnership agreement, etc. Perhaps, the MSA contracts should have a different retention period than other contracts, and maybe Partnership Agreements require a different approval process than the others.
In this case, we can define a new “Contract” content type in SharePoint which contains any other contract metadata fields (except for document type). Then we can create additional content types for NDA, MSA, Maintenance Agreement, Partnership agreement – each of these will inherit from the Contract content type, so they all share the same metadata. But we can define unique retention policies, workflow processes and templates for each one.
Many ECM document repositories have a metadata choice field named “Document Type”, which is typically a drop down list (choice) field of possible document types. This design can be migrated directly to SharePoint. However, sometimes documents should be treated differently based on the “document type”. For example, say we have repository for “contracts”, with a field named “contract type” containing options for: NDA, MSA, Maintenance Agreement, Partnership agreement, etc. Perhaps, the NDA contracts should have a different retention period than other contracts, and maybe the MSAs require a custom approval process than the others.
The best approach for this is by creating custom Content Types, and leveraging content type inheritance. As a general recommendation, we suggest that companies always create a general content type called “[YourCompany] Document”. This content type will inherit from the generic “Document” content type, and all other custom content types that you create will inherit from “[Your Company] Document”.
In this case, we’ll define a new “Contract” content type in SharePoint which contains any other contract metadata fields (except for document type). Then we can create additional content types for NDA, MSA, Maintenance Agreement, Partnership agreement – each of these will inherit from the Contract content type, so they all share the same metadata. But we can define unique retention policies, workflow processes and templates for each one.
We can now these content types in the same document library or multiple document libraries, and the documents will adhere to the retention rules and workflow processes defined by their content type.
All legacy document management systems support “choice fields”, where metadata values are constrained to a list of values, typically presented to users as a drop down list. Of course, SharePoint supports Choice fields as well. But SharePoint has several other options, which could provide long term benefits for certain use cases:
Managed Metadata: If the options in a choice field need to be re-used in many different use cases and across many SharePoint sites and site collections, or if the options in a choice field are hierarchical in nature, then consider defining columns using Managed Metadata.
Lookup columns: behave similar to a choice field, except that the list of choices are sourced from a SharePoint list that is in the same site as the document library. Lookup columns are great when you want to enable end users to dynamically add, remove or change choices available for that field, but the choices don’t need to be re-used across multiple sites.
BCS (External Data): behave similar to Lookup columns, except that the values are sourced from an external database, rather than a list in SharePoint. Columns based on external data can be re-used in multiple sites and site collections.
Note that all of these options complicate the migration process. So these options should not be considered lightly. Remember that your migration is a one time project, but your taxonomy lasts forever. So weigh whether the long time benefit of these features outweighs the short term costs to use these features.
Sometime users have created multiple repositories that have the same logical fields. Some ECM systems don’t support column re-use, and therefore certain columns may have been “manually copied” or re-created in multiple repositories. If users were not very careful, columns that should have shared the identical name were given slightly different names in each repository.
By creating “Site Columns” instead of “list columns”, we can make sure that columns are properly re-used to create a consistent environment. Mistakes happen! But there’s no reason not to correct those mistakes before starting your migration to SharePoint.
Don’t assume that the current capture system will (or should) work exactly the same way with SharePoint as it does with the legacy ECM system. In many cases, you will be able to keep the identical capture process, and simply swap out an “export connector” for the legacy system with a new connector for SharePoint… but this isn’t always the case. Depending on your SharePoint taxonomy, your capture system may or may not have the features needed to support all of the relevant SharePoint features (e.g. columns types, such as Managed Metadata, BCS and Lookup columns).
If you are using Choice fields, Managed Metadata, Lookup or BCS columns in SharePoint, think about how you will sync these terms with your document capture system, For example, Kofax Capture allows users to reference Managed Metadata termsets from SharePoint while they are validating documents. So the Kofax admins don’t need to worry about syncing the SharePoint terms with Kofax.
Make sure you inventory all of the channels that can deliver content to your legacy ECM system, and validate that each capture sub-system is compatible with SharePoint.
Be on the lookout for any specialized or custom document indexing processes, such as “One to many indexing” where a single document may contain many related metadata records.
This information is needed to help architect your SharePoint farm:
Number of site collections and content databases
Create one per repository
Put all repositories in one site collection
Group certain repositories in one site collection and others in a different site collection
Warrant an additional Physical SQL server (or multiple SQL servers)
Storage (SAN, NAS, IOPS)
This information is important to know for a few reasons –
It may impact your choice of migration tools and the approach used for the migration
Depending on the file types, you’ll need to make sure that you have integrated viewers and editors for various file formats with SharePoint
There are two major features that most document management systems have:
-intuitive metadata filtering/query builder
-search results displayed in a tabular grid
SharePoint’s native metadata filtering displays resulting a grid, but it doesn’t support column types like single line of text, and it doesn’t scale well in document libraries with more than 5000 documents.
SharePoint Search provides the advanced search web part, but this can be challenging to configure and formats results like a search engine, rather than in a tabular display.
Make sure you understand and document exactly how users interact with your current ECM system. Site down and watch how users really do their job on a daily basis. This is especially important for any users that use your ECM system on a daily basis to get their primary job done.
XenDocs Search provides a user experience that is similar to virtually all other document management systems, including intuitive metadata filtering, tabular search results, document sorting, document previews, etc.
XenDocs allows SharePoint users to have users the “best of both worlds” – all of the web content management and document collaboration features AND a great solution for managing transactional document.
There are many migration tools for SharePoint available. Depending on your source ECM system you may find an off-the-shelf migration product that meets all of your needs for a low cost. In many cases, some level of customization may be needed. Features to look for include:
Field mapping, in case columns in SharePoint are different from the legacy system.
Ability to migrate sub-sets of documents in filtered chunks
Ability to Start/Pause/Stop/Resume a migration
Extremely robust auditing of every document that is successfully migrated or that encounters an error. This is critical to troubleshooting any migration problems. This data should be stored in a database, not exported to a text file.
File Conversions (especially to PDF, very common!)
Support for advanced SharePoint column types (e.g. MMS, Lookup, BCS). However, there are work-arounds in SharePoint for transforming data from text columns to other column types, so if the migration tool doesn’t support this, its not a huge problem. Either way, you need to know what column types are supported.
Our XenDocs ECM Migration toolset is comprised of two major components:
Export modules, which exports batches of content from your legacy ECM systems, such as ApplicationXtender, FileNet, File Magic, Liberty, Sire, LaserFiche, etc.
File Importer for SharePoint which imports these batches into the appropriate sites, libraries, content types and columns. The File Importer can also perform image processing on your documents during the migration, such as OCR, convert to searchable PDF, auto-rotate upside down pages, etc. This tool can also be used after your migration project to integrate multi-function scanners, fax servers or other line of business applications with SharePoint on an ongoing basis.
If you have a migration project in mind we’d be happy to provide you with a personalized demonstration.