A IBM Connect 2013 session by David O'Neal (Consultant, Infrastructure Engineering, Nationwide), Wouter Aukema (CTO, Trust Factory) and Florian Vogler (CEO, panagenda)

1. SPN101 Leveraging the Power of
IBM Lotus Notes and Domino
at Nationwide
David O’Neal | Consultant, Infrastructure Engineering, Nationwide
Wouter Aukema | CTO, Trust Factory
Florian Vogler | CEO, panagenda
© 2013 IBM Corporation

2. What we‘ll cover today
 Introduction
 IBM Notes and Domino @ Nationwide
 What we did
– Collect Data (what sources & some stats)
 What we found
– Confirmations & Opportunities
– Configuration, Usage, Performance, Security
 What it means
– Short Term Quick Wins
– Long Term Strategic Takeaways
 Conclusions / What we learned
 Q&A
2

3. Introduction – who‘s who?
 About Nationwide
– Nationwide Mutual Insurance Company, based in Columbus, Ohio, is one of the largest
and strongest diversified insurance and financial services organizations in the U.S. and
is rated A+ by both A.M. Best and Standard & Poor’s. The company provides customers
a full range of insurance and financial services, including auto insurance, motorcycle,
boat, homeowners, pet, life insurance, farm, commercial insurance, annuities,
mortgages, mutual funds, pensions, long-term savings plans and specialty health
services.
 About Trust Factory
– Trust Factory‘s DNA provides true insight into server performance and scaling
opportunities. DNA is also used by IBM worldwide as Domino DoubleCheck.
 About panagenda
– With more than 5.5 million licenses of its products, panagenda helps customers in over
70 countries analyze and optimize their IBM environments.
3

4. IBM Notes and Domino @ Nationwide
 The Nationwide Notes/Domino Environment
– Production use began in 1997 with version 3.3
– Migrated/Consolidated to Notes from cc:Mail and a variety of different mainframe email
systems
– Current environment
• 6 Domino Domains
• 200+ Domino servers on Microsoft Windows® (8.x mixture – mostly 8.5.3 for mail)
(Mail, Management, Application, Blackberry and Good servers)
• Active / Passive clustering accross two data centers
• 56,000+ Notes clients (mostly 8.5.2)
• 15,000,000+ messages routed weekly
• ~20 Sametime 8.5.2 IFR1 servers using Domino and WebSphere
• ~1200-1400 Domino applications with 700 being active
4

5. IBM Notes and Domino @ Nationwide
 What is Nationwide trying to accomplish by performing this in-depth analysis?
– Server:
• Discover inconsistant configurations, and find gaps where Domino does not readily
report items that could potentially turn into problems.
– Client:
• Discover and inventory client side settings, configurations and local databases to get
a better understanding of client health and functionality.
– Environmental:
• Combine server and client findings to get a holostic view of our Notes/Domino
environment.
5

6. What we did
Collect data from Inventory Notes clients
Domino servers – notes.ini
– statrep.nsf – desktop, bookmarks,
– log.nsf names
– catalog.nsf – local databases
– directories (names.nsf, – various OS and Notes
DA) properties
Talk 
6

7. What we did
 Collect
– log, statrep and catalog from 151 servers
– 33,000 users used 35,000+ clients
– 690,000+ documents with 315,000+ attachments collected = 3,5 GB of raw data
– 1.6 million desktop icons, 1.5 million local databases, 5.4 million notes.ini entries,
8.5 million client and OS details
 Analyze
– DNA: Compared this engagement against 2+ million other users
– 100+ views created consuming 30+ GB of disk space
 Interpret and Correlate
 … and now for the meat …
7

8. Domino Environment Overview
1 Domino Directory
39,725 Users Registered
34,057 Users Active 47,178 Databases Touched
153 Servers Registered 494,006 Views Indexed
151 Servers Analyzed 133,540 Databases Deployed
4 Domino Releases 1,477,390 Views Defined
4 Operating Systems 381 View Storage (GB)
1,361,855 ACL Entries 82,131 Db Storage (GB)
39,369 Groups Registered 1,370,468 Group Members
6 February 8
2013

9. DNA Benchmark
Active versus Registered Users
100 %
34,057 active users
80 %
60 %
40 %
20 %
0%
Nationwide Lowest Customer DNA Average Highest Customer
Unused Licenses, Web Users, Regular Absense
6 February 9
2013

10. DNA Bechmark
Time Online
25 60
50
20
40
(mins per session)
Session Duration
15
(hours per user)
Online Time
On average
30
with DNA
10
20
5
10
- -
Nationwide Lowest Customer DNA Average Highest Customer
Session Duration 3 1 4 22
Online Time 24 2 23 77
6 February 10
2013

11. User Demand Profiling
(Nationwide, 34,057 active accounts)
Remote Workers Office Workers System Accounts
25%
20%
15%
10%
5%
0%
2 4 6 8 10 12 14 16 18 20 22 24
Distinct Hours Online per Day
6 February 11
2013

12. End User Demand Characteristics Only mail servers
Nationwide in Scope for DNA
100%
75%
50%
25%
0%
Notes Sessions Document Reads Document Writes Db Transactions Network Traffic Session Duration
check new mail 6% 0% 0% 1% 0% 0%
system dbs 6% 0% 0% 1% 0% 0%
mail files 80% 85% 79% 85% 86% 99%
directories 3% 1% 5% 4% 0% 0%
applications 5% 14% 16% 9% 14% 1%
6 February 12
2013

13. End User Demand Characteristics
Other IBM Customer
100%
75%
Extreme high
docreads on
Directory databases
50%
25%
0%
Notes Sessions Document Reads Document Writes Db Transactions Network Traffic Session Duration
check new mail 19% 0% 0% 3% 0% 1%
system dbs 13% 4% 0% 5% 1% 1%
mail files 33% 24% 76% 54% 55% 72%
directories 16% 41% 3% 11% 11% 4%
applications 19% 31% 22% 27% 33% 23%
6 February 13
2013

14. User Demand on 16739, Databases
Nationwide
100.000.000
Showing only databases touched by >1 users. Majority of apps
(47,175 databases touched by all users) are MC
10.000.000
1.000.000
K
B
n
e
o
y
v
S
s
t
r
.
l
i
100.000
10.000
1.000
100
10
1
0
1 10 100 1.000 10.000 100.000 1.000.000 10.000.000 100.000.000
KiloBytes Read from Server
369, Application Dbs 560, Domino Directory Dbs 15209, Mailfiles 55, Mailin databases 143, Server Mail Boxes 403, System databases
6 February 14
2013

15. End User Demand at Nationwide
Classified by Demand Level
Document
Writes
Document
Reads
Database
Transactions
Network Traffic 1 user does 15% of total
network demand
(client to server)
Network Traffic
(server to client)
User Sessions
0% 25% 50% 75% 100%
Extreme (1) Intensive (16) Moderate (804) Light (33,236)
6 February 15
2013

16. Domino Servers at Nationwide
Classified by Maximum Session Concurrency
95
90 Redistributing the load can reduce
nr. of servers w ith up to 87
85
80
75
70
65
60
55
50
45
40
35
30
25
20
15
10
5
0
Very Low Low Average Normal High
Level < 50 50 - 249 250 - 749 750 - 1499 >= 1500
Servers 87 23 23 17 1
6 February 16
2013

17. Concurrent User Sessions
6 February 17
5,000
10,000
15,000
20,000
25,000
30,000
0
2013
2012-12-04 00
04
08
12
16
20
2012-12-05 00
04
08
12
16
20
2012-12-06 00
04
08
12
16
20
2012-12-07 00
04
08
12
16
End User Demand
20
Session Concurrency
2012-12-08 00
04
08
12
16
20
2012-12-09 00
04
08
12
16
20
2012-12-10 00
04
08
12
16
Max Observed Maximum: 27,179
20

18. Network Compression
How Much is Notes Network Compression Used?
Very few customers
have this properly
Enabled
implemented
Includes Traffic
from Users and 75%
Servers
# Users making use of
Notes Network Compression
100%
Enabled
Disabled
75%
Disabled
25%
% Active Users
50%
25%
0%
Persons Servers
6 February 18
2013

19. Deployment Integrity
Entries appearing in multiple documents
Integrity check # Databases Document Type Item Type Nr of Documents
Duplicate Replica On Same Server 380 group docs listname 3
Duplicate Template On Same Server 341 mail-in docs fullname 22
Replicas Acting As Different Template 610 person docs fullname 2
Same Replica But Different Inheritance -
Grand Total 27
Grand Total 1,331
PubNames,
DirCat & DA at
risk (!)
11 Group Cycles Detected
6 February 19
2013

20. Basic Security Checks
Internet Password Strength Databases with Anonymous Access
Variations found Accounts Access Level Databases Templates
'password' 0 Author 84 102
'secret' 0 Editor 11 -
firstname 0 Manager 3 302
lastname 0 Reader 2,507 222
shortname 0
companyname 0
Grand Total 2,605 626
Grand Total 0
1st Customer with
NO issues :)
6 February 20
2013

21. And now for the client-side analysis …
21

22. Diving right into client-side analysis
 The following slides dive into
various client-side details
 In many cases, the Nationwide
environment is surprisingly clean
– Your environment will
most probably look very different
22

23. Notes 8.0.2 & 8.5.2
 Although there are 1,817 clients with 8.0.2,
only 26 have Create_R8_Databases enabled =
they do not leverage the benefits of ODS 48
23

24. Operating system details
 Various different client-side operating systems in use:
24

25. Local replicas of public addressbook
 Local replicas of the public addressbook
beyond cutoff
 Risk of replicating deleted documents
back into server-side replica
  Enable PIRC:
25

26. Local addressbooks:
Version mismatch
 Checked rows show configurations
where names design matches
client version
– (might still have wrong ODS, though)
 In general, design mismatch of
system databases
– slows down client startup and beyond
– causes unexpected behaviour or
non-functioning of Policies
  can be fixed by
– making sure clients have correct templates
– removing TemplateSetup= from notes.ini
26

27. Local bookmark.nsfs:
Version mismatch
 Checked rows show configurations
where bookmark design matches
client version
– (might still have wrong ODS, though)
27

28. Local cache.ndk:
Version mismatch
 Checked rows show configurations
where cache design does NOT match
client version
– (might still have wrong ODS, though)
 Cache.ndk must be deleted and
re-created together with
CREATE_R85_DATABASES=1
in notes.ini - for it to have proper
design and ODS
(make sure client has correct cache.ntf)
28

29. Local desktop.ndk:
ODS issues
 Checked rows show configurations
where desktop ODS is NOT ideal
29

30. Local log.nsfs:
Version mismatch
 Checked rows show configurations
where log design does NOT match
client version
– (might still have wrong ODS, though)
30

31. Notes.ini:Log=
 A couple of users have multiple log= lines in notes.ini
 Since only the first entry is actually read in such a case,
logging does not work as expected for those users
31

32. More on ODS levels
 Various databases and templates do not have
an ideal ODS …:
 Adding
CREATE_R85_DATABASES=1
and
NSF_UpdateODS=1
to notes.ini can help!
32

33. More notes.ini entries …
 Less than 1% of all users have port compression
disabled, but 25% of all traffic is uncompressed
 must be enabled on BOTH servers and clients
 identify servers that are used by users but have
port compression disabled
33

34. EXTMGR_ADDINS= …
 Various users have
EXTMGR_ADDINS
entries in notes.ini which are
seperated by a blank
  surprisingly DOES work
(side-effects unknown)
34

35. Mail
 Who encrypts email when saving emails?
 Who encrypts sent email?
 Who signs sent emails?
35

36. Calendar
 Users with wrong(?) calendar settings?
 1=Sunday
2=Monday
3=Tuesday
5=Thursday
36

37. Cache.ndk
 Users where Cache= in notes.ini points to
– A dedicated file/path
– Partly filepaths in which users
might not have write permissions
(e.g. Notes program files directory)
37

38. Clustering / Loadbalancing
 FailoverSilent (defaults to 0) is disabled for almost all users:
38

39. Client Configuration Health
 Basic vs. Standard clients
 Clients with wrong InstallMode= setting in notes.ini
39

40. … same for ini:InstallType=
 Identifying Client/Admin/Designer configurations:
40

41. Hardware/OS details: disk space
 Users with too little free disk space
– might soon call helpdesk
– may experience stability issues
– have high disk fragmentation = slooooow
41

42. Hardware/OS details: memory
 Users with too little memory
– again, sloooow(er) client experience
42

43. Locations: do not use IP addresses as mailserver names …
 A couple of users have an IP address configured as their mailserver
breaks Policies
 DNS names as mailservers could become a problem if the DNS domain were ever
to be renamed …
43

44. Mailfile replicas?
 98 users work off a local replica
+330 managed replica users
 BUT: 3,407 users have a local replica
and: 149 users have more than one mail replica …
– 39 of these local replicas are beyond cutoff:
44

45. ECL
 Open doors 
45

46. Analyzing desktop icons (special vs. local vs. server)
46

47. Analyzing desktop icons (details)
 196,930 local databases with an icon (e.g. bookmark.nsf)
 380,243 local databases without an icon (e.g. help files, cache.ndk, …)
 1,266 templates on desktops
 37,108 templates not on desktops (think shared data directory)
 36,865 replicas without any icon
 267 replicas without a local icon
 2,686 replicas without a server icon
 862,395 template replicas without any icon
 14 template replicas without a local icon
 765 template replicas without a server icon
47

48. Summing it up …
48

49. Conclusions
 Mission accomplished
– Provided a holistic view across servers and clients
 Mission not accomplished (yet)
– This is work in process
 Nationwide is the 1st customer out of many that leverages most of the
features/potentials of the N/D 8.5 platform
– Optimization potential almost exclusively in areas “without features”
– Implemented Domino password security the way it should be
49

50. What we learned
 Detailed data helps to leverage IBM Notes and Domino to its fullest potential
 … and helps shifting from reactive to proactive
  Assumptions vs. Evidence
– Eliminate best guess/hope based working
 Find out and focus on what really matters
50

51. Q&A
 Thank you for listening!
51

52. Legal disclaimer
© IBM Corporation 2013. All Rights Reserved.
The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided
AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be
responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating
any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this
presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way.
Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon
many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance
can be given that an individual user will achieve results similar to those stated here.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics
may vary by customer.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
52 © 2013 IBM Corporation