1. Quality of Experience
Performance Benchmark on Messaging solutions
Microsoft Office 365, IBM LotusLive and Google Apps
Webinar October 6 th 2011
Alexis Blandin, Cloud & Social Business Leader | Synergie Informatique
Arnaud Bécart, International Presales Engineer| ip-label
David Wais, Business Development Leader| ip-label

2. Synergie Informatique
 Cloud Services Provider, France
 Social Business Specialist
 Expert in messaging and
collaboration solutions
integration
 IBM, Microsoft and Google
partner

3. ip-label.newtest
Around the world, ip-label.newtest solutions measure the quality of applications
from the user’s point of view: business applications as well as web, voice and
even TV over IP platforms.
Over 1500 direct and indirect customers rely on ip-label.newtest to validate the
end-to-end reliability of their websites and critical applications worldwide

4. Cloud goes Mainstream
“ Cloud Computing Market Will Top $241 Billion in 2020 ” – Forrester Research
“10% of Enterprise Email Will Be Based on a Cloud/SaaS Model in 2014” – Gartner
 Innovation = Public versus Enterprise
 Social Network
 Mobility, highly connected, convergence
 Increasing Data
 SaaS, a serious consideration

5. Messaging in the line of sight
Cloud Usage Scenarios
CIO Expected Advantages
1. Prototyping  Commodities delegation
2. Test & Development  Industrialisation
 Fast deployment
3. Messaging  Savings, ROI
4. Collaboration  Self-service, flexibility / PAYG
 Innovation (« beta »)
5. Web Applications  Standardization, integration
 New challenges (mobility, …)
6. Backup, Archiving

6. The CIO Evolution
 Guarantees, disponibility (SLA)
 Warranty and « Quality » improvement
 Security, externalisation, reversibility
 Sales relationships (B2B)
 Contracting and legal management, (Patriot Act, CNIL)
 Going from CAPEX to OPEX (subscription)
 Operational and organisational restrains

7. Adapted Needs
 Governance, « Cloud » Best Practices
 Adapted procedures flowchart
 More straight to the point (support, specifics, infrastructures)
 Dependant to the service/ network
Metrology Tools
 User experience

8. Editor’s Focus / Collaborative Suite
1911 1975 1998
Identity,
Services / Softwares Windows / Office Search
Strategy
Social Business « All In » Pure Player
Product LotusLive Office 365 Google Apps For Business
Acquisition By
Yes Progressive (options) No
modules
€/U/Y
34 to 113 € 43 to 306 € 40 €
(public prices)
• Social and Collaborative
• Pack Office Integrated • Web Experience
Advantages Network
• Sharepoint Framework • Google ecosystem
• External Opening
• Some missing features • SharePoint Out-of-the-Box • No web conference
Weaknesses (online office suite, (ergonomy, features) • Very basic integrated social and
blogs/wikis, call audio/video) • Document-Centric collaborative tools

9. E-Mail Systems Focus
“ Inbox = Tetris getting zero lines doesn’t win
it just makes the next move easy “
Product • LotusLive iNotes/Notes • Exchange Online • GMail
• Codes different than a classic
• Classic codes • Classic codes messaging suite
Features
• Native Lotus Notes included • Native Outlook integrated • GMail offline (Google
Chrome)
• Two offers: basic(web mail) +
Technical advanced (Domino) • Hybride Mode • Full Web
• Hybrid Mode
• Leader On-Premise
• Lack of visibility, can look strong • In position of conquest
• Late Coming-Out Cloud
Analysis, enough just for Domino clients • Big efforts made in some
• Difficult move from BPOS to
trend • Domino/LotusLive future vision points (sécurity /
Office 365 : which
available administration)
consequeces ?
Space 25 Go 25 Go 25 Go
SLA,
Credit according 99,9 % 99,9 % 99,9 %
to the available 2 à 10% monthly cost 25 à 100% monthly cost 3 to 15 days
%

10. Methodology
Data Solution
 Pilot for all web applications , rich web or client (
Office 365, Outlook…) run like a user: mouse
click, filling application forms, text control, click
on image etc.
 Fast launch on one of the 70 measuring points in
the world (5 operators are tested in France,
Orange, Neuf-SFR, Iliad-Free, Colt, Cogent)
 Mode « Saas » on demand, from set up tests to
operational robot and servers maintenance
 Access to all measures via an extranet, reports
sent via mail and warnings on incidents
 Access to back up for all set up , expertise and
training requests
 Scenari recording tools for clients wanted to be
autonomous

11. Solution Architecture

12. Tests Description
 3 platforms tested
 3 types of tests
– Access to the portal from 5 spots in France
• https://portal.microsoftonline.com
• https://apps.lotuslive.com
• https://www.google.com/a/synergie-informatique.fr/
• Tests origins: France Telecom, Neuf-SFR, Iliad-Free, Colt, Cogent
• Tests frequency: 15 minutes
• Tests duration: 21/07/11 au 30/09/11 (71 jours)
– Scenario SENDING from 1 spot in France
• Sending a mail from the Cloud platform to a external messging address
• Delivery mail authentication in less than 60 seconds
• Tests frequency: 30 minutes
• Tests duration: 1/09/11 au 30/09/11 (30 jours)
– Scenario RECEIVING from 1 spot inFrance
• Sending a mail from an external messaging address rto the Cloud platform
• Receiving mail authentication in less than 60 seconds
• Tests frequency: 30 minutes
• Tests duration: 1/09/11 au 30/09/11 (30 jours)

13. Tests Description
 Scenario SENDING
– Connexion to the portal
– Authentication
– Access to the e-mail
– New online message to mailXX@ip-label.net
– Send the message and wait for POP from ip-label (mail.ip-label.com) – waiting time max 60 seconds
 Scénario RECEIVING
– Connexion to the portal
– Authentication
– Access to the e-mail
– Send a mail via SMTP from a server from ip-label (mail.ip-label.com)
– Wait for mail receipt on the tested platform – waiting time max 60 seconds
Each sent mail is automatically deleted after each test
Each portal account uses a distinctive external messaging account

14. Tests Description
 When the portal we are trying to access is considered unavailable?
– Non loading of the portal in less than 60 seconds
– Network incidents detected ( DNS faulty resolution , failed connexion) and
server(loading slowness)
 When a scenario is considered unavailable ?
– Non complete loading of a page in less than 30 seconds linked to network or
server incidents
– Keywords not found
• Username to validate the loading of the portal
• Username/Password, connexion button, then button ‘new mail to validate the
authentication step
• SEND : button new mail and send
• RECEIVING : refresh button to verify the mail had been received
– Non transmission of the e-mail ( external SMTP  online messaging or online
messaging SMTP) in less than 60 seconds

15. Portal Access
Availibility and average access time to each portal
Results from 21/07/11 to 30/09/11 – Around 34000 tests per portal
1,541 seconds 1,097 seconds 3,552 seconds
99.980 % 99.977 % 99.968 %
google apps
Résolution DNS
lotus live Temps de connexion
Temps de réponse du serveur
Office 365 Chargement du contenu
0 500 1000 1500 2000 2500 3000 3500 4000
– Google has the best portal loading time
– Lotus Live is quite fast (1,5 seconds) but the page is longer to start (600ms), with longer DNS resolution,
connexion and server time
– Office 365 is the longest to load it is also the heaviest (1 Mo, versus 17 ko for google and 32 ko for Lotus
Live)
– Lotus Live shows the best availibility with 7 failures, Google (8 échecs) and Office 365 (11 failures)

16. Portal Access Access to portal availibility
Access to portal performance Evolution over a 3 months period
4000 100
100
99,99
99,98
99,98
99,98
99,98
99,98
3500 3750 99,95
99,96
3474 3432
3000
99,9
2500
1982 Office 365
2000 99,85
1357 lotus live
1500 1286
99,83
google apps
99,8
1000
1119 1113 1061
500 99,75
0
Juillet Aout Septembre 99,7
Juillet Aout Septembre
Conclusion
- For the LotusLive portal , the answering time network and
server has increased since 31/08/11 ( for an identical content)
- Main errors encountered
- Google : 5 DNS resolution errors and 3 time outs
Lotus Live : 4 interrupted connections and 2 unreachable servers
Office 365 : 4 DNS resolution errors, 5 unreachable servers and 2
timeouts
- A maintenance intervention on the Office 365 portal on 9/9/11
from 5am to 8;15am was not taken into account

17. Authentication
Authentication availibility and online messaging access
Results from 01/09/11 to 30/09/11 –About 2800 tests per portal
Performance Authentication and online
99,76 messaging access
Disponibilité
99,86
24h/24
99,17
11438
Disponibilité 99,89 Office 365
Heures Ouvrées 99,89 google apps
8h-23h 99,56 15238
lotus Live
Disponibilité 99,54
19054
HNO 99,81
23h-8h 98,52
0 5000 10000 15000 20000
97,50 98,00 98,50 99,00 99,50 100,00
– Google is the strongest plaftform in terms of authentication 99.86 % in 24/24
– Office 365 99.76% in 24/24 on authentication (mainly linked to some timeouts and DNS errors); those
incidents usually occur at night and from 8h to 23h the availibility is the same as Google
– LotusLive has the most errors at the authentication (99.17%); even if those errors occur mainly at night time,
(99.56% availibility between 8h and 23h), one incident last September 18th between 8h and 10h penalise their
position on this scale
– Google is the fastest platform to authentify yourself and access to messaging : 11,5 seconds, LotusLive 15
seconds ( in one page load in opposition to the others)
– Office 365 is the slowest to reach online Outlook : 19 seconds

18. Authentication
Authentication Incident for Lotus Live on 18/09
Timeout : incomplete page after
login (88k out of 212ko expected)
Page error after login
l 18

19. Authentication
Authentification errors examples
Authentification timeout on Office 365
Authentification timeout on Google Apps

20. Authentication
Examples : Authentication errors on Google Apps portal
Erreur du serveur (HTTP 500) sur l’objet Dashboard qui devrait rediriger sur le portail
Erreur
Cas normal
Impact utilisateur : le champ login absent de la page et l’utilisateur ne peut pas s’authentifier

21. Sending/Receiving mail
Performance
Average length of time to send and receive e-mails
12271
Temps de réception 23213
18895
google apps
lotus live
7558
Office 365
Temps d'envoi 10049
15095
0 5000 10000 15000 20000 25000
– On average, to send a mail from a Cloud platform is faster than receiving a mail from an
external platform
– Google is the platform which transmits mails the fastest, as well as sending to an external
account (12,2 seconds on average) and receiving from an external account(7.5 seconds)
– Office 365 is faster than Lotus Live for receiving mail, but slower than LotusLive to send mails

22. Sending/Receiving mail
Stability
Average availibility for sending and receiving mails
86,17
Disponibilité de réception (<60s) 99,18
71,66
google apps
lotus live
85,17
Office 365
Disponibilité d'envoi (<60s) 97,62
74,81
0 10 20 30 40 50 60 70 80 90 100
– Lotus Live is the most available platform in terms of sending mails, if we consider as
unavailable a length of time superior to 60 seconds ,to send a mail CloudExterne or
ExterneCloud
– Office 365 (~75%) and Google Apps (~85%) : This means even with a fast sending and
receiving mail process (<15 secondes), engorgements regularly slow down the e-mail
sending or receiving process
– Sending and receiving mail process is more unstable for Google and Office 365

23. 2 1 3
 Google Apps Gmail can be considered as a reference in terms of online Scores
Google Office
messaging access speed and you are guaranteed fast sending and Apps Lotus Live 365
receiving mail Portal performance 3 2 1
Portal availibility 2 3 1
 IBM Lotus Live is a stable platform in terms of sending and receiving Authentication availability and
online messaging access 3 2 1
mails but can be a disappointment for users because of access Authentifcation availability
unavailibility and average access time (15 sec) (working hours) 3 1 3
Receiving Time 3 1 2
 Microsoft Office 365 has a good messaging availibility (99.89% during Sending Time 3 2 1
Receiving Stability 2 3 1
working hours) but some progress need to be made on page loading
Sending Stability 2 3 1
time ( particularly on authentication and access to Outlook) and on
Total 21 17 11
sending stability

24. Conclusions
Results Details
 Gmail Google
– Best Portal Access Performance
– Best authentication performance and availibity
– Best sending time to an external e-mail address and receiving from an external e-mail address
– Relative unstability in transfert time frames
• 15 mails out of 100 are sent in more than 60 seconds
• 14 mails out of 100 are received in more than 60 seconds
 IBM Lotus Live
– Look out on the highest portal loading time since the beginning of September
– Best portal access availibility
– Best transfert stability
1 mails out of 100 are sent in more than 60 seconds
3 mails out of 100 are received in more than 60 seconds
– Authentication performance and availibility is not good
 Microsoft Office 365
– Very good authentication disponibility during working hours.
– Portal access performance and access to messaging to be improved (19 seconds)
– Stabilité des durées de transmission des mails à améliorer
• 25 mails out of 100 are sent in more than 60 seconds
• 28 mails out of 100 are sent in more than 60 seconds

25. Conclusions
Each editor commits himself to 99.9% availibility
– « Microsoft provides a financially-backed 99.9% uptime guarantee »
• http://www.microsoft.com/fr-fr/office365/what-is-office365.aspx
– « LotusLive Notes delivers the capabilities your business users need to increase efficiency while
reducing IT complexity and associated bottom line costs - all backed by a dedicated IBM team with
24x7 support and a 99.9% up-time service level objective »
• https://www.lotuslive.com/styles/tours/transcripts/LLNotes_datasheet.pdf
– « Google Apps includes a 99.9% uptime SLA and synchronous replication »
• http://www.google.com/apps/intl/en/business/messaging.html
As a reminder data for September 2011…
According to you, what is the quality really
perceived by the online messaging users ?

26. Q/A
 Were mails send with attachments ?
– No , only text content; the add on of attachments of different sizes could be a criteria
to do more tests
 Which internet navigator was used for the tests?
– Internet Explorer 8
 Can those tests be run on a client (Lotus Notes, outlook) ?
Yesthe robots from ip-label can simulate any user running windows
 Can this benchmark be testing with files /document sharing?
– Sharing files tools can also be tested by the robots ip-label
 What about other navigators , especially if Microsoft Exchange is losing out on
fonctionnality ?
– The robots can test Explorer or Firefox; the multi navigators cans be given through
other solutions like Cloud Observer for instance
 Can we audit audio / video flow?
– Ip-label can measure those flows using different measuring solutions on a user
standpoint (streaming, voice quality analysis, video tests /audio conference, etc.)

27. Thank You
Synergie-informatique.fr
@SynergieInfo