2. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
2
Coverity Scan Service
The Coverity Scan™ service began as the largest public-private sector research project in the world focused on open source
software quality and security. Initiated in 2006 with the U.S. Department of Homeland Security, Coverity now manages the
project, providing our development testing technology as a free service to the open source community to help them build
quality and security into their software development process. With almost 700 projects participating, the Coverity Scan service
enables open source developers to scan–or test–their Java, C and C++ code as it is written, flag critical quality and security
defects that are difficult (if not impossible) to identify with other methods and manual reviews, and provide developers with
actionable information to help them to quickly and efficiently fix the identified defects. More than 20,000 defects identified by
the Coverity Scan service were fixed by open source developers in 2012 alone.
We’ve expanded beyond our annual Coverity Scan Report to create a series of open source project spotlights.This spotlight
features LibreOffice, an open source office suite developed by The Document Foundation, and a member of the Scan service
since October of 2012.
Introduction to LibreOffice
LibreOffice got its start in 2010 as an offshoot of the OpenOffice open source collaboration suite. OpenOffice had its roots
in StarOffice, which Sun Microsystems acquired in 1999. In starting LibreOffice,The Document Foundation set out to build
a better open source software suite by eschewing strict rules about design protocol and strict restrictions, and adopting a more
“people-centered” philosophy.
LibreOffice embodies the collaborative philosophy which is often associated with open source projects. Its philosophy was
described by one of their committers, Norbert Thiebaud, as “believe in the people.” People are willing to do the right thing and
you must trust them.The community is focused on mentoring and supporting new members. Long-time members lead by
example to communicate the values of the project, with new members emulating that behavior. At the core of these principles
is the promise of better quality and higher reliability software that gives greater flexibility at zero cost.
Today, the project has the backing of all the major Linux distributions, including Red Hat, Novell and Ubuntu, as well as the
support of the Free Software Foundation (FSF), Google, Intel and AMD. It is the default office suite of the most popular
Linux distributions, and is available in more than 112 languages and for a variety of computing platforms, including Microsoft
Windows, Mac OS X and Linux.
3. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
3
LibreOffice joined Coverity Scan in October of 2012. When the LibreOffice team first started using the Scan service, the
project had a defect density of 1.11.The team has been working diligently to reduce the number of defects in the code, fixing
more than 2,000 defects so far, and improving their defect density by almost 25%.
LIBREOFFICE ANALYSIS: 2013
Year Version
Lines of Code
Analyzed
Outstanding
Defects
Defects Fixed
2013 to date 4.1 9,016,902 4,717 2,150
DEFECT DENSITY BY PROJECT SIZE
Lines of Code LibreOffice Open Source Proprietary
More than 1 million .84 .75 .66
4. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
4
The following are the specific types of defects outstanding in LibreOffice:
OUTSTANDING DEFECTS AS OF OCTOBER 4, 2013
API usage errors 23
Class hierarchy inconsistencies 3
Code maintainability issues 79
Concurrent data access violations 2
Control flow issues 447
Error handling issues 1628
Incorrect expression 49
Insecure data handling 11
Integer handling issues 252
Memory – corruptions 56
Memory - illegal accesses 42
Null pointer dereferences 886
Parse warnings 1
Performance inefficiencies 3
Program hangs 4
Resource leaks 74
Security best practices violations 26
Uninitialized members 1090
Uninitialized variables 40
Various 1
Grand Total 4717
5. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
5
As of October 4, 2013, they had fixed the following high- and medium-impact defects:
DEFECTS FIXED YEAR AS OF OCT. 4, 2013
API usage errors 14
Class hierarchy inconsistencies 1
Code maintainability issues 7
Concurrent data access violations 2
Control flow issues 133
Error handling issues 1386
Incorrect expression 26
Insecure data handling 3
Integer handling issues 39
Memory – corruptions 17
Memory - illegal accesses 34
Null pointer dereferences 122
Performance inefficiencies 4
Program hangs 6
Resource leaks 133
Security best practices violations 9
Uninitialized members 188
Uninitialized variables 26
Grand Total 2150
6. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
6
Coverity Scan uncovered a buffer overrun in the LibreOffice code, which leads to illegal memory access. Specifically, they
defined a static array of 12 8-byte structures starting on line 225 in the image below:
While iterating over that array, they specify the upper bound for the index as the size of the array (96), rather than the number
of elements (12). Each element is 8 bytes, so element 96 would start at offset 760. As a result, the code on line 245 will access
memory all the way up to 768 bytes past the beginning of the array. Since the array only occupies 96 bytes, the loop will read
and operate on 672 bytes of illegally accessed data!
7. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
7
To correct this issue, they now divide the size of the array by the size of each element, giving them the number of elements in
the array to loop over:
8. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
8
Q&A Norbert Thiebaud
A Committer to LibreOffice
Q: Can you tell us a little bit about the philosophy of the LibreOffice project?
A: We have a very “people centered” philosophy that can be summarized as, “believe in the people.” People are willing
to do the right thing.The core group of engineers who started the project created a community of being accepting
and helpful. When people want to help with the project, be nice.There is a willful effort to be accepting.The
experienced developers recognize and value people who are trying to contribute and make a special effort to help
them be successful. Long-time members of the project lead by example to communicate the values of the project,
and new members try to emulate those values.
Q: Why do you think you are able to achieve high levels of quality?
A: LibreOffice is a very large project, with a very, very long history, and has more than seven million net lines of code,
some of which are more than 20 years old. Automation and tools are a must to be able to cope with the scale, to
catch potential problems as early as possible. For the ones that get away, we have a great, dedicated group of QA
volunteers that help triage bugs reported by our fantastic and diverse user base, to help channel issues to developers.
We’ve also developed tools like bi-bisect, which allows testers to bisect between binary versions of the product to
identify when a particular bug or regression appeared. We have an ever-growing stack of unit tests and integration
tests. We regularly use dynamic analysis tools like Valgrind to identify performance and/or memory issues. And
of course we benefit from static analysis tools like Coverity that can do what no single developer can: review the
entirety of the code base, every time.
Q: What is it about the developers on your project that you think enables them to create high-quality code?
A: LibreOffice has many, very senior developers that have been with the codebase for a decade or so... and they are
quite accessible to the less seasoned developers.They mentor new volunteers and expose them to the tools and
methods we use to avoid pitfalls. We are also growing a gerrit-based infrastructure to foster code review and it is
integrated with buidlbots to be able to build and test patches as early as possible in the process, at the patch level.
Q: What happens if you have a developer working on the project who submits code that doesn’t meet your quality expectations?
A: LibreOffice has essentially two levels of access to the code.The first level of access is Committers that can push
code, and the second is Contributors that can propose patches via gerrit. For the latter, the patch has to be reviewed
and approved by a Committer. At that stage, if there are issues with the patch, the work is commented on and a
new version of the patch can be uploaded in gerrit based on these comments.
For Committers, they can still use gerrit to seek the opinion of their peers, but they can push their work out
directly.That work will generate activity on a commit-list ML which can trigger some post-facto review comments.
In general, there is little problem with Committers not meeting quality expectations, and certainly nothing that is
not solved quickly by peer pressure and cooperation with other developers to address any concerns.
9. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
9
Over time, Contributors who prove they are doing a good job can become Committers. It’s very organic when this
happens; there is no set limit of patches or set amount of time.
Q: What sort of process do you follow to ensure high-quality code?
A: I would say an 'all-of-the-above' process.That is, there is not a rigid, formal process, but a wide variety of tools
combined with motivated developers, QA and users that are all aiming for a quality product. In other words, use
all the technology help you can get and trust your extended teams to each do their part to maintain the highest
level of quality. But without a rigid process, that could, at times, be used as an excuse: "But I followed the process,
so it must be good." Process, or more exactly guidelines, are there to guide and help achieve the goals, but are
no substitute for personal involvement and responsibility. In LibreOffice, we believe that having people that are
involved and take ownership of their work is much more important to achieve quality than any process.
Q: Do you have people in formal roles to ensure the quality of the code?
A: No, we do have tools and people that manage these tools, who can and do raise attention to them if and when
things slip, but quality is everybody's concern and priority. In general, the project tries to keep the structure as
flat as possible. From the development perspective, there are essentially only two formal levels: Contributor and
Committer. In LibreOffice, like in any open source project, some people by virtue of their skills and experience
become informal, de-facto leads but these are not formal roles and arise organically.
Q: Can you describe how development testing and the Coverity Scan service fit into your release process?
A: We have set-up a buildbot that regularly uses Coverity Scan. We are still ramping up that infrastructure, but due to
the size of the codebase (and therefore the size of the resulting scan), we are doing these builds every two to three
weeks right now. Eventually, I think we will get to a weekly build.
Some developers are reviewing the output of Coverity Scan, triaging and fixing the bugs that need fixing.The long
history, the complexity of the LibreOffice code base and the fact that the Coverity scans are relatively recent tools
in our arsenal mean that we are still quite a bit behind with regard to triaging all the defects reported by Coverity,
but we are catching up!
Q: What tools do you use, beside Coverity, and how do they impact your ability to deliver high-quality code?
A: We use anything and everything we can get our hands on and find the man-power to run.The latter means that
the frequency of use of some of these tools can be variable, but that includes things like other static code analysis
tools, code coverage, unit tests, automated integration tests, and of course good old elbow-grease by a small army of
LibreOffice volunteers.
Q: What challenges do you face with regard to maintaining high-quality code that are unique to open source and how do you
overcome these challenges?
A: The best tool in the world can only get you so far.The magic ingredient to a successful project is people. So the
greatest challenge is to make sure we retain and continue to attract great people, and help them with all the tooling
and support we can to leverage their enthusiasm and dedication.
10. COVERITY SCAN PROJECT SPOTLIGHT: LIBREOFFICE
10
Q: Do you have any last comments you would like to make?
A: LibreOffice started its new adventure, under the “umbrella” of The Document Foundation, about three years
ago, building on the very large existing codebase inherited from OpenOffice.org. We have been making great
progress at re-structuring, cleaning up and improving the quality of LibreOffice and Coverity makes an important
contribution there.Thank you!
Conclusion and Next Steps for Coverity Scan
LibreOffice has done an excellent job of addressing key defects in their code in the short time they have been part of the
Coverity Scan service. We would like to thank the LibreOffice team for their participation in the service.
Register your open source project with Coverity Scan or sign up for a free trial to get visibility into the quality and security of
your software code.
11. Software
Integrity
Report
Project Name: LibreOffice
Version: 4.1
Project Description: Scan project LibreOffice
Project Details:
Lines of Code Inspected: 9,016,902
Target Level 1 ACHIEVED
Project Defect Density: 0.51
High-Impact and Medium-Impact Defects: 4608
Company Name: LibreOffice
Point of Contact: Coverity Scan
Client email: scan-admin@coverity.com
Report Date: Oct 3, 2013 12:21:30 PM
Report ID: 7346e342-5dfe-4271-bde8-010a74ed206f
Coverity Product: Static Analysis
Product Version: 6.6.1
Coverity Point of Contact: Integrity Report
Coverity email: integrityrating@coverity.com
The Coverity Integrity Rating Program provides a standard way to objectively measure the integrity of your own
software as well as software you integrate from suppliers and the open source community. Coverity Integrity
Ratings are established based on the number of defects found by Coverity® Static Analysis when properly
configured, as well as the potential impact of defects found. Coverity Integrity Ratings are indicators of software
integrity, but do not guarantee that certain kinds of defects do not exist in rated software releases or that a release is
free of defects. Coverity Integrity Ratings do not evaluate any aspect of the software development process used to
create the software.
A Coverity customer interested in certifying their ratings can submit this report and the associated XML file to
integrityrating@coverity.com. All report data will be assessed and if the Coverity Integrity Rating Program
Requirements are met, Coverity will certify the integrity level achieved for that code base, project, or product.
12. High-Risk Defects
High-impact defects that cause crashes, program
instability, and performance problems.
Medium-Risk Defects
Medium-impact defects that cause incorrect results,
concurrency problems, and system freezes.
Defect Risk by Component
Component Owner
Defect
Density
Other 0.63
writer 1.52
calc 1.19
draw 0.95
filters 0.26
- - 0.00
- - 0.00
- - 0.00
- - 0.00
- - 0.00
- - 0.00
13. Defects by Assigned Severity
High-severity defects have been tagged by developers as a
clear threat to the program's stability and/or security.
Defect Severities by Component
Component Owner
Defect
Density
Other 0.63
writer 1.52
calc 1.19
draw 0.95
filters 0.26
- - 0.00
- - 0.00
- - 0.00
- - 0.00
- - 0.00
- - 0.00
Defects by Triage State
14. Coverity Software Integrity Report
The Coverity Software Integrity Rating is an objective standard used by developers, management, and business executives to
assess the software integrity level of the code they are shipping in their products and systems.
Coverity rating requirements are based on an assessment of several factors:
• Defect density: For a given component or code base, the number of high-risk and medium-risk defects found by static
analysis divided by the lines of code analyzed. Defect density excludes fixed defects and defects dismissed as false
positives or intentional. For example, if there are 100 high-risk and medium-risk defects found by static analysis in a
code base of 100,000 lines of code, the defect density would be 100/100,000 = 1 defect per thousand lines of code.
• Major severity defects: Developers can assess the severity of defects by marking them as Major, Moderate, or Minor
(customizations might affect these labels). We consider all defects assigned a severity rating of Major to be worth
reporting in the Integrity Report regardless of their risk level because the severity rating is manually assigned by a
developer who has reviewed the defect.
• False positive rate: Developers can mark defect reports as false positives if they are not real defects. We consider a
false positive rate of less than 20% to be normal for Coverity Static Analysis. A false positive rate above 20% indicates
possible misconfiguration, incorrect inspection, use of unusual idioms in the code, or a flaw in our analysis.
Coverity Integrity Level 1 requires the software has less than or equal to 1 defect per thousand lines
of code, which is approximately the average defect density for the software industry.
Coverity Integrity Level 2 requires the software has less than or equal to 0.1
defect per thousand lines of code, which is approximately at the 90th percentile for
the software industry. This is a much higher bar to satisfy than Level 1. A million-line
code base would have to have 100 or fewer defects to qualify for Level 2.
Coverity Integrity Level 3 This is the highest bar in the rating system today. All
three of the following criteria need to be met:
• Defect density less than or equal to 0.01 defect per thousand lines of code, which is approximately in the 99th
percentile for the software industry. This means that a million-line code base must have ten or fewer defects
remaining. The requirement does not specify zero defects because this might force the delay of a release for a few stray
static analysis defects that are not in a critical component (or else giving up on achieving a target Level 3 for the
release).
• False positive rate less than 20%. If the rate is higher the results need to be audited by Coverity to qualify for this
integrity rating level. A higher false positive rate indicates misconfiguration, usage of unusual idioms, or incorrect
diagnosis of a large number of defects. The Coverity Static Analysis has less than 20% false positives for most code
bases, so we reserve the right to audit false positives when they exceed this threshold.
• Zero defects marked as Major severity by the user. The severity of each defect can be set to Major, Moderate, or Minor.
This requirement ensures that all defects marked as Major by the user are fixed because we believe that once human
judgment has been applied, all Major defects must be fixed to achieve Level 3.
Level Not Achieved indicates that the target level criteria are not met. This means that the software has too many
unresolved static analysis defects in it to qualify for the desired target integrity level. To achieve the target integrity level
rating, more defects should be reviewed and fixed.
How to Use Your Software Integrity Rating
Set software integrity standards for your projects, products, and teams.
It is often difficult for developers and development management to objectively compare the integrity of code bases, projects,
and products. The Coverity Software Integrity Rating is a way to create "apples-to-apples" comparisons and promote the
success of development teams that consistently deliver highly-rated software code and products. Development teams can also
use these ratings as objective evidence to satisfy requirements for quality and safety standards.
Audit your software supply chain.
It is challenging for companies to assess the integrity of software code from suppliers and partners that they integrate with
their offerings. The Coverity Software Integrity Rating is a way to help companies create a common measurement of software
integrity across their entire software supply chain.
Promote your commitment to software integrity.
The integrity of your software has a direct impact on the integrity of your brand. Showcasing your commitment to software
integrity is a valuable way to boost your brand value. It indicates that they are committed to delivering software that is safe,
secure, and performs as expected.
