BP116

BP116 Backup 101:
The What, How, and When
Jim Casale | Lotus Consultant | GROUP Business
Software

© 2011 IBM Corporation

Welcome

Jim Casale
Lotus Consultant - GROUP Business Software

Website: www.jimcasale.net
Email: jimc@5jcs.com
Twitter: @jimcasale

© 2011 IBM Corporation 2

Evaluations Count
● Please fill out your evaluation before you leave or online at
http://www.lsonline.info
─ “Click here to fill out an individual session survey”
● Evaluations help speakers improve the sessions they deliver to you
● If you don't fill out your evaluation you will be locked in a room with me for 24
hours while I tell jokes


Agenda
● What are backups?

● IBM® Lotus® Product Architecture Overview

● What needs to be backed up

● When it needs to be backed up

● How it needs to be backed up

● What to restore when that time comes

● IBM® Lotus® Domino® and DAOS – It's not all that hard


What this session is not
● It will not make you a DBA, IBM® WebSphere® Administrator, or Backup
Administrator
● It will not show you every conceivable backup configuration for your
environment or anyone else
● It will not allow you to tell people how to do their jobs better (although we all
know in some cases that may be true)


What this session is
● This session will give you a basic understanding of the what, how, and when of
backing up several popular Lotus technologies
● It will give you the information you need to understand what your backup
administrator is babbling about
● about 55 minutes in duration from now


What are backups?
● Backups are copies of data stored onsite or offsite
─ Operating Systems
─ User Data
─ Configuration Data
● Types of backups
─ Full
– All encompassing backup of all data associated with a particular system or
server
─ Incremental
– Backup of data that has changed or been modified since the last backup
─ Differential
– Backup of data that has changed or been modified since the last full backup


Backup Software
● Not all Backup Software is created equal
─ Traditional Backup Software
– Used to backup files. Utilizes Backup Agents in some cases

● Image Backup Software
─ Utilizes special routines and APIs to create image backup of a volume
● Continuous Data Protection
─ Protects data by continuously backing up data as it is modified

● Backup Methodology
─ Daily
─ Weekly
─ Monthly
─ Policy


Things to consider

● What level of recovery are you looking for?
● What are the SLA agreements, if any?
● What backup system are you using?
● Who will be performing the backup and restore?
● What type of Domino environment do you have?
● What type of network environment do you have?


Lotus Product Architecture
● Not all Lotus Products are the same
─ Some products are a combination of different IBM technologies
─ e.g. IBM® Sametime®, IBM® Lotus® Connections®, etc
● Size of environment dictates who is responsible for which part
─ Larger environments may have a dedicated WAS administrator, IBM® DB2®
Administrator, etc
● If you are not responsible for backing up these systems you should still know
how it works – trust but verify


Lotus Product Architecture cont'd
● Core Products
─ IBM® Lotus Notes® and Domino®
– Client/server application environment based on Notes Storage Facility
─ IBM® Lotus® Sametime®
– Unified communications and collaboration server
● As of Sametime 8.5 there are Domino and WebSphere based
versions
─ IBM® Lotus® Connections®
– Social software application server
─ IBM® Lotus® Quickr®
– Team collaboration software server
● Like Sametime, there are Domino and WebSphere versions


Lotus Product Architecture cont'd
● Core Technologies
─ IBM® DB2®
─ IBM® WebSphere® Application Server
– Used by many Lotus Products
– Limited use license
– Customized installation depending on the application


Let The Games Begin
● IBM DB2
● IBM WebSphere
─ IBM Lotus Sametime
─ IBM Lotus Connections
─ IBM Lotus Quickr
● IBM Lotus Domino


IBM DB2
● What needs to be backed up?
─ Databases, Log Files
● When does it need to be backed up?
─ Before major changes or upgrades
─ Daily incrementals
─ Weekly Full Backups
● How to back it up?
─ Offline
– “backup db2 database ....” command
– Use file level backup to backup the DB2 backup files
─ Online
– “backup db2 database ....” command
– Must use Archive Logging


IBM DB2 cont'd
● Document your installation
─ DB2 version
─ DB2 Fixpack level
─ Use the db2level command to obtain information including 32/64 bit information
─ Where DB2 and associated database is located


IBM DB2 How to
● Offline DB2 Backup
─ From a IBM DB2 Command Window
– db2 backup database STMS

● Online DB2 Backup
─ Online backups are only available for databases configured with logretain or userexit
enabled
─ To enable online DB2 backups
– db2 update database configuration for STMS using logretain recovery


IBM DB2 How to cont'd
─ db2 get database configuration for STMS (backup pending = Yes)

─ db2 backup database STMS to D:DB2BACKUP


IBM DB2 How to cont'd
─ After the DB2 configuration change has been made you need to stop and start DB2
– db2stop
– db2start
─ DB2 is now enabled for online backups. You must now backup the DB2 log files as
they are now using “archival logging”
─ Use file level backup or “include logs” db2 backup command parameter
● DB2 Restore
─ db2 restore database STMS from D:DB2BACKUP
─ You need access to the DB2 server and the DB2 backup files
─ If you need to perform a point-in-time restore you need to know the date and time of
the backup you want to restore


IBM WebSphere Server
● IBM WebSphere Application Server is the heart of many Lotus products
─ IBM Lotus Sametime
─ IBM Lotus Connections
─ IBM Lotus Quickr

● Each installation of WebSphere is tailored to the individual product but there are
some common procedures, files, and commands that are used across all
WebSphere based technologies


IBM WebSphere Server cont'd
─ Databases, Configuration Files, Customized Files
─ Daily incremental
─ Backup DB2 database
─ Backup WebSphere configuration files
– All properties files from <WASHOME>/properties directory
– All files from <WASHOME>/etc directory
– Or just backup the entire Websphere directory – it's easier
─ Use the backupconfig command for each profile
– Simple to use – run from the bin directory in each profile directory
– Backs up the configuration information for that profile in an appropriately named
zip file – e.g. WebSphereConfig_2010-12-18.zip


● backupconfig example
─ In it's simplest form of backup - just type the command. There are switches to tell the
system where to save the file, name the file, etc.


─ It's easier Alert - the advantages of using a file system backup and restore over
the backupConfig method are that fixes and fixpacks are restored instantly. This
method also includes any customized/modified files
─ Another It's easier Alert – If you backup the registry in Windows you will save
a lot of work if you ever have to restore to new hardware. All WebSphere services
will be setup if you restore them from registry backup
● Document your installation
─ WebSphere version
─ WebSphere Fixpack level
─ Use the versionInfo.bat command to obtain system information – you will need this
information when you open a PMR
─ Where WebSphere is located
– And I hope it's not under C:Program FilesIBMWebSphere...


● IBM WebSphere Application Server Restore
─ All WebSphere and DB2 services need to be shutdown
─ Have all your system documentation ready
─ Depending on the situation you may want to make a backup of what you have before
you restore.
─ Restore the backup to the file system making sure you overwrite the existing files
─ You may want to put services to manual until you have checked out the restored files
─ When restore is complete restart server


IBM Lotus Sametime
─ Backup DB2 if WebSphere based Sametime
─ Backup WebSphere if it's WebSphere based Sametime
─ Backup Domino
─ Backup Sametime configuration files
– sametime.ini
– userinfoconfig.xml
– stsecurity.ini
– stldap.ini
– vpuserinfo.nsf, stconfig.nsf, stcenter.nsf. stpolicy.nsf, etc


IBM Lotus Sametime cont'd
● If you are using the new Sametime Meeting server please be aware that
meeting materials are stored in DB2 - not the file system
● Sametime Restore
─ What is restored depends on the reason you are restoring
– If you have a corrupt nsf you can restore just that database (while Sametime is
down)
– If you are doing a full system restore because of hardware issues you will have
to plan out your restore.
● What gets restored first and in what order
● Configure services for manual start until all aspects check out OK
● Possible scenario includes
– DB2
– Websphere
– Domino


IBM Lotus Connections
─ Backup DB2
─ Backup WebSphere
─ Backup Connections configuration files
– Lotus Connections installation directory – LotusConnections
– profileRegistry.xml
– LotusConnections-config.xml
– Any customized configuration files
● header and footer HTML files, CSS and JSP files, themes


IBM Lotus Connections
─ HTTPServer directory


IBM Lotus Quickr for WebSphere Portal
─ Backup DB2
─ Backup WebSphere
─ Backup Quickr Places


IBM Lotus Quickr for WebSphere Portal cont'd
● Quickr Places
─ Enable backup of Places
– Done for each Place to backup
– Open your place and click Customize -> Properties -> Edit Properties -> Edit
– Enable Application Backup for Archiving and Restore
– You can manage backup, restore, and archive of places from Manage Places
● This has changed in Quickr 8.5 – now called Quickr Administration
Console or Places Administration
● HTTPServer directory


Don't forget to fill out your evaluations


IBM Lotus Quickr for Domino
─ YAY – no DB2 or WebSphere to backup
─ Backup Domino
─ Backup Quickr for Domino configuration files
─ Backup Quickr Places


IBM Lotus Quickr for Domino cont'd
● Backup Quickr for Domino configuration files
─ qpconfig.xml
─ Domino databases
● Backup Quickr Places
─ load qptool archive -p MyPlace -dir D:QUICKRBACKUP
─ load qptool archive -a


IBM Lotus Domino
● How to back it up? ← This is my favorite part of the session
─ Document your installation
– All third party addins and extensions
– notes.ini
– Custom templates


IBM Lotus Domino cont'd
● What files to backup
─ From a Domino perspective
– Application files (nsf)
– Customized template files (ntf)
– Transaction Logs

─ From an OS level perspective
– server.id
– SSL keyfiles (.kyr and .sth)
– notes.ini
– third party addins, extensions, utilities
● Excluded from OS backup
– nsf, ntf, box, txn, and lfh files


● What files not to backup
─ Standard template files (ntf)
– Depending on your environment
● log.nsf
● mail.box(s)
● Help files


● You know what to backup and what not to backup – now what?
─ Plan your backup
– Never run Domino maintenance and a backup at the same time
– Whenever possible use a Domino specific backup agent
● IBM® Tivoli® Storage Manager for Mail or equivalent
– Don't run maintenance on Domino that will work against you
● e.g. These commands will fragment your nsf's and, in the case of
transaction logging backups, will change the DBIID
● Compact -B

● Compact -C

● Fixup -F -J (Use on a corrupt database but do full backup of

database as soon as possible)


● Use transaction logging and the Domino backup API whenever possible
● In other words use the Domino specific backup agent
─ It reads nsf files at the Domino API level, not the OS file system level
─ You will not have issues with open files*
─ You can cut your backup time dramatically

● In the Domino world most every nsf is considered “changed” from the last time
it was backed up. Essentially OS level backup software will want to perform a
full backup
● API level backup provides the ability to backup only those transactions since
the last backup. Instead of a backup taking 7-8 hours to finish your backups will
take only minutes
● The drawback is your restore times can be longer


● Transaction Log Backup
─ Sometime referred to as “incremental” backup
─ Uses the least amount of resources, and take the least amount of time
─ Can be used to restore to “Point-in-Time”
─ Can be run multiple times throughout the day with no performance impact on
Domino
─ Can take longer to restore data
● Best Practices for Transaction Log Backup
– Full backup on weekend – which day depends on your Domino maintenance
schedule.
– Archive logs at least once a day – more often on heavily used servers
– Inactivate logs at least once a week
– Do not scan transaction logs or the transaction log control file with OS antivirus. If it
quarantines a transaction log your Domino server DIES and you will have to
recreate your logs on the next Domino startup


● Point-in-Time Restores
– You can recover documents to a point in time as long as you know when they
existed in the application
– Don't advertise it as guaranteed in case you can't get a document back
– You can get documents back even though they haven't been backed up yet (if you
really like your users)
– PIT restores depend on the right transaction log files and control file. If you have to
recreate your logs and control file PIT will not work for prior backups
– If you suffer from corrupt a transaction log and recreate them, you will get errors
when trying to restore PIT from before the corruption. Domino PIT restores depend
on the control file (lth file). If you run into this error you will get complaints from the
backup software saying Recovery Manager is not happy with what you are trying to
do
– You cannot restore from Trash. Well you can but it's a pain and not recommended


● The most frequently searched for terms on my blog are..

DAOS and backup


DAOS

● New 8.5 feature to save space in DAOS enabled applications
● Can save 40-60 % in application space used (your mileage may vary)
● Low level service for the stripping of attachments from DAOS enabled
applications
● Does its work in the background. Domino doesn't know it exists
● Moves attachments outside the NSF realm (e.g. to less expensive disk
subsystems)
● Brings new issues to the Domino backup world


DAOS cont'd
● Backing up a DAOS enable application will NOT backup the associated
attachments
● You should continue to run your normal Domino backups*
● Schedule incremental backups of the DAOS datastore at the OS level
● You do not have to do full backups of DAOS – one full backup and then
incrementals.
● If you are paranoid you can do monthly full backups of DAOS
● If you want to do daily full backups of DAOS then go ahead, it won't hurt
anything


DAOS cont'd
● How DAOS works (simple explanation)?
─ DAOS strips attachments out of documents at a low level and moves them to the
directory specified in the server document. This process happens in the background
and is transparent to the Domino server and the end user
● DAOS System Files
─ The heart of DAOS are
– daoscat.nsf
– daos.cfg
– These DAOS system files hold the information needed for DAOS to operate


DAOS cont'd
● How DAOS works (simple explanation)? cont'd
─ DAOS Commands
– DAOS has tell commands like other Domino services e.g compact, updall, etc
– Can be used to inquire about the health of the DAOS system or to run
maintenance commands
– Output can be redirected to text file for later examination or processing
─ NLO directory
– Where all good attachments go for retirement (in a friendly sort of way)
– The NLO files are stored with encrypted file names e.g.
0A65CAD58DE5836169CFE27556C88ADF6DF6602500008690.nlo


Disclaimer
● I am not an IBM employee and I do not sell software
● The following slides refer to IBM Tivoli Storage Manager and IBM Tivoli Storage
Manager for Mail because they work well
● Your mileage may vary


DAOS cont'd
● DAOS order of backup
─ NSF Files
– Best practice is to use a backup system that using the Lotus Domino API such
as Tivoli Storage Manager for Mail
─ Transaction Logs
– DAOS requires transaction logging, but not “archive” type transaction logging
specifically. But utilizing archive style transaction logs will allow for ultra fast
weekday backups as well as “Point In Time” restores
─ NLO Files
– The NLO backup needs to cover the entire DAOS directory including
subdirectories
– DAOS can handle 40 million NLO files. Your backup system needs to handle
that
● Automation
─ Backups are best done using automation. Scripted/Scheduled backups work best
● Don't forget!
─ Don't forget to test your backups and check backup log files for errors


How simple can DAOS backup be?
cdProgra~1TivoliTSMdomino

domdsmc selective * /loggedonly /subdir=yes /adsmoptfile=dsm.opt /logfile=domsel.log >> domsel.log

domdsmc archivelog /adsmoptfile=dsm.opt /logfile=domasch.log >> domarc.log

cdProgra~1TivoliTSMbaclient

dsmc incremental g:DAOS -subdir=yes >> nlobackup.log


Yeah, but I have to restore data too
● Once again, it's not as hard as it sounds
─ But some backup systems are better/easier than others
– NSF File – this is the very easy part. Best practice would be to restore to an
alternate server (very common in the Tivoli Storage Manager for Mail world)
– Transaction Logs – if doing a “Point In Time” restore you would use the backup
system to restore and play back the transactions
– NLO's – Run your DAOS command to list the NLO files you need to restore

▬ tell daosmgr listnlo -o ssquarepants.txt ssquarepants.nsf
– When complete this command will leave you with a text file in the Domino data
directory listing all the missing NLO's
– Use the output file to restore the NLO files from backup


DAOS Restore – Start Here
● On your restore server use your Tivoli Storage Manager for Mail client to
restore the last backup copy of the mail file
● Once the mail file has been restore we need to find out what the missing NLO
files are. The restore server only uses the Domino executables and Server ID. It
is not configured to be a running Domino server
─ If there is a daos.cfg and daoscat.nsf delete them before proceeding. They will be
recreated when you run the first command
─ Change to the Lotus Domino executable directory from a command prompt and
issue the following command. Remember to prefix the command with “n” since we
are not issuing these commands from a Domino console
─ Nfixup -f -j mailssquarepants.nsf – where mailssquarepants.nsf is the name of the
mail file you are restoring


DAOS Restore cont'd
● When fixup runs on the mail file the daos.cfg and daoscat.nsf files will be
created
● Before we find out what NLO files are missing we need to resync the DAOS
Catalog by issuing the following command

ndaosmgr resync

● Resyncing the catalog will prepare the catalog to find out what NLO files are
missing

● We are now ready to get a listing of missing NLO's


DAOS Restore cont'd
ndaosmgr listnlo -o ssquarepants.txt ssquarepants.nsf

● This will give you a listing of missing NLO files
D:DOMINO>ndaosmgr listnlo -o ssquarepants.txt mailssquarepants.nsf
01/10/2010 03:15:17 PM DAOSMGR: ListNLO started for
NSF:E:Datamailssquarepants.nsf
01/10/2010 03:15:17 PM DAOSMGR: ListNLO output file:
E:Datassquarepants.txt.txt
01/10/2010 03:15:17 PM DAOSMGR: ListNLO completed - 608 entries
processed

● Note:The output of this file will show the DAOS path on the server the
command is being run from. You may need to tweak the file when using the
output file for NLO restores


Sample output file for missing NLO's

G:DAOS0001C22A7354C87D0EC23FD2433A7D0530F4205912F10001D842.nlo
G:DAOS000180EF611583B9CE2187F59FF78AF8CBB747B720660001DB9F.nlo
G:DAOS0001D0E07F6B19E4CBD6051AF73B265D5810248AF107000003E0.nlo
G:DAOS00012DB55D70ACC0970139346B28846CC3C56460D92C0001EF86.nlo
G:DAOS00019A5BA6D9299BCA37BC83931F981345690BB294D200014CA8.nlo
G:DAOS00018E773F92D87D340BFA3E72796F74470A64FB3681000004F7.nlo
G:DAOS000140DCA47464080F2958954AD9A636799B9890945F00000E97.nlo
G:DAOS00016FC25B33B33E8E5DDE1F5CA97836C7C6ECB022C100000430.nlo
G:DAOS0001C495BD38A088BFAE81AD47BFA060A68D6B711648000004E3.nlo
G:DAOS0001FB56605047DA65B1D1B5E65394EDC29A76FB38FC000004A1.nlo


So what do we do now?
● Now that we have a listing of missing NLO files we need to restore them from
backup
● This is why TSM is your friend
● To restore the missing NLO files without losing your wits
dsmc restore -filelist=ssquarepants.txt -latest or
dsmc restore -filelist=C:ssquarepants.txt -latest
● You may need to edit the file list depending on your environment. TSM may
require a UNC path for the restore as well as an explicit path to the directory
you want to restore the NLO files into
G:DAOS00018E773F92D87D340BFA3E72796F74470A64FB3681000004F7.nlo
to
<source
server>g$DAOS00018E773F92D87D340BFA3E72796F74470A64FB36810000
04F7.nlo


Restoring the NLO files
● You will see the following output when restoring the NLO files
Restoring 49,428 <source
server>g$DAOS0001FB73DB4873C09FA3932250AB554BF8F78C56CC9E0000C01C.nlo --> <destination
server>g$FB73DB4873C09FA3932250AB554BF8F78C56CC9E0000C01C.nlo [Done]
server>g$DAOS0001FD2C1A37EB81B4E66DBD495384C61B2D38DABB39000031B3.nlo --> <destination
server>g$FD2C1A37EB81B4E66DBD495384C61B2D38DABB39000031B3.nlo [Done]
server>g$DAOS0001FE5708DEFB914CB2081C5B81190629D086D34F950002D000.nlo --> <destination
server>g$FE5708DEFB914CB2081C5B81190629D086D34F950002D000.nlo [Done]

Total number of objects restored: 608
Total number of objects failed: 0
Total number of bytes transferred: 70.14 MB
Data transfer time: 2.33 sec
Network data transfer rate: 30,709.24 KB/sec
Aggregate data transfer rate: 1,810.56 KB/sec
Elapsed processing time: 00:00:39


A Few Tips
● Do not AV scan the NLO files or directory
● More frequent full backups of the NLO files will make restore easier but will
make backup times longer and use more backup storage
● Alternate server restores are the preferred way to restore mail if using Tivoli
Storage Manager for Mail
● If tight security is a major concern then an alternate server restore is NOT for
you – it requires a copy of the server ID on the alternate restore server


The Demo


Lotupshere Sessions to see
● ID105 DAOS Deployment and Best Practices
─ Speakers: Patrick Mancuso, Gary Rheaume
─ DL S. Hemisphere III - Tuesday 3:30pm – 4:30pm
● IDOL101 Lotusphere IDOL
─ DL S. Hemisphere IV-V - Tuesday 1:30pm – 2:30pm
● BP102 Get Started Installing IBM Lotus Sametime 8.5.1 – You Too Can Be a
WAS Admin!
─ Speaker: Gabriella Davis
─ SW 5-6 - Tuesday 3:30pm - 4:30pm


Don't forget your evaluations
● Remember what happens if you don't fill in your evaluations....24 hours of
─ Why did the chicken cross the road??
─ Knock knock
─ What do you get when you cross a....


Q&A
● You can always contact me after the session if you have any questions


Legal Disclaimer
© IBM Corporation 2011. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without
warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of
the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors,
or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change
at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor
shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or
both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.


BP116

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to BP116

Similar to BP116 (20)

Recently uploaded

Recently uploaded (20)

BP116