1. Developing B2 for
Performance and Scalability
Noriaki Tatsumi
Kevin Weisser

2. Why Performance Matters?
• User experience
– Your users don’t like slow and unresponsive apps
– Slow applications lose users in frustrations
– Users tend to wander off beyond captive response
time of 7 – 10 seconds
• User adoption
– Google saw 25% decline in searches within 6
weeks after introducing 0.5s delay (~$2.5B loss)
– Wikia observed up to 20% user loss as the page
load time reached 5 seconds

3. Wikia – Exit rate (abandon)
http://assets.en.oreilly.com/1/event/27/Varnish%20-%20A%20State%20of%20the%20Art%20High-Performance%20Reverse%20Proxy%20Presentation.pdf

4. “With great power comes great
responsibility” – Uncle Ben, Spiderman
• Reputations at stake
• System resources are shared and limited
• B2s with scalability problem impacts system
performance
– System-wide performance degradation
– Server downtime

5. What is an anti-pattern?
• Some repeated pattern of action, process or
structure that initially appears to be beneficial,
but ultimately produces more bad consequences
than beneficial results
• A refactored solution exists that is clearly
documented, proven in actual practice and
repeatable.
- Definition from Wikipedia

6. How can anti-patterns help?
• Prevents performance as an afterthought
• Helps identify performance issues that typically
don’t arise until deployed in production
• Prevents pre-mature optimizations

7. How do I use anti-patterns?
• Identify and correct the occurrences of anti-
patterns during:
– Design review
– Implementation/code review
– Execution of experiments and profiling
– Static code analysis

8. Top 10 violated high-level performance
anti-patterns fixed in Blackboard Learn™
Inefficient Filtering
Unnecessary Processing
The Ramp
Data Overreach
Frequency (%)
Unlimited Resource Usage
Functional Bug
Excessive Granularity
More is Less
Usage of Outdated Tools
and APIs
Mishandled Concurrency
0 5 10 15 20 25 30

9. Top 10 violated low-level performance
anti-patterns fixed in Blackboard Learn™
Missing or non-optimal index [db]
Inefficient or erroneous criteria [db]
Unnecessary data retrieval [app}
Too many objects in memory [app}
Presentation of unlimited results [ui]
Frequency (%)
Inefficient access path [ui]
No data caching [app}
Not releasing resource (immediately) after use [app}
Incompatiblity with HA deployment [app}
Excessive DB round trips [app}
0 5 10 15 20 25

10. Most violated high-level anti-pattern
#1: Inefficient Filtering
• Description
– Querying data using inefficient logic, algorithm, or
technique
• Problem
– Consumes resources that can be used by other
processes
– Increases execution/response time
• Solution
– Store data in both centralized and organized manner
– Search against smaller data set first, fail fast, and don’t
look at unnecessary data
– Leverage indexing and other efficient
techniques/algorithms

11. Most violated high-level anti-pattern
#1: Inefficient Filtering
Missing or non-optimal index (DB)
• Most violated low-level anti-pattern
• Problem
– Missing appropriate index on search columns leads to
inefficient data retrieval operation by SQL query optimizer
– Having index can inversely add unnecessary overhead if the
column is never used for searched criteria
• Solution
– Analyze SQL execution plan to verify usage of proper index
– Create index on foreign keys
– Create unique index on unique columns
– Use an index to eliminate unnecessary full-table or index
scans to access small percentage of data on a large table
– Use covering index for key lookup

12. Creating Database Objects with B2
• Release 9.1 SP1 and above
– B2s authored by independent developers can now
define and install schema elements
• Not enabled by default
– Administrators must enable this functionality from
the B2 management section of the admin panel
• Documentation
– http://www.edugarage.com/display/BBDN/Creating+
Database+Objects+with+Building+Blocks

13. Most violated high-level anti-pattern
#1: Inefficient Filtering
Inefficient or erroneous criteria (DB)
• The second most violated low-level anti-pattern
• Problem
– Duplicate, unnecessary, incorrect, or missing predicate
checks and joins exist in SQL
– Increases response time, system resource consumption, and
locking
• Solution
– Be specific on what data you need
– Keep it simple and avoid unnecessary checks
– Use JOIN criteria that aligns with index
– Analyze execution plan to derive efficient and balanced query

14. Most violated high-level anti-pattern
#1: Inefficient Filtering
Wrapping filtering column in a function (DB)
• Problem
– The query optimizer does not see the column
– Available index likely does not get used
• Solution
– Rewrite query to avoid using function if possible
– Format bind variables in application tier

15. Example of bad T-SQL function usage
- Without -

16. Example of bad T-SQL function usage
- With LOWER-

17. Example of bad T-SQL function usage
- Comparison -
Without T-SQL function With T-SQL function
Logical I/O reads 3 3238
Estimated CPU cost 0.396307 15.454153

18. Most violated high-level anti-pattern
#1: Inefficient Filtering
Other low-level anti-pattern examples
• Filtering larger query first
• Implicit data type conversion
• Inefficient CSS selectors
• Inefficient DOM element lookup
• Inefficient regex
• Unnecessary query on VIEW instead of table
• Usage of "NOT IN" comparison rather than JOIN
• Usage of wrong index
• Using UNION when returned records always unique

19. Most violated high-level anti-pattern
#2: Unnecessary Processing
• Description
– Processing when not needed or not needed at the time
– Excessive or duplicate executions
• Problem
– Consumes resources that can be used by important
work
– Increases execution/response time
• Solution
– Remove, postpone (asynchronous), prioritize, or reorder
the processing steps
– Leverage caching to reuse loaded or calculated data

20. Most violated high-level anti-pattern
#2: Unnecessary Processing
Presentation of Unlimited Results (UI Design)
• 5th most violated low-level anti-pattern
• Problem
– Users typically do not consume all data when presented
with large amount so the used resources are wasted
– Severe system performance issue can arise depending
on the volume of prepared results
• Solution
– Paginate results
– Guide users to narrow down search criteria
– Limit amount of data that can be returned

21. Example usage of pagination and limiting
the number of returned results

22. Most violated high-level anti-pattern
#2: Unnecessary Processing
Inefficient access path (UI Design)
• 6th most violated low-level anti-pattern
• Problem
– Room for optimization exists in actions/clicks users
must take to complete his/her task
– Results to creating additional work for the system
(also impacts user’s time to task)
• Solution
– Refactor UI or click path to let users accomplish
their tasks more efficiently

23. Example of enhancing access path

24. Most violated high-level anti-pattern
#2: Unnecessary Processing
Other low-level anti-pattern examples
• Excessive executions of maintenance task
• HTTP 404 error on cacheable static content
• Missing short-circuit logic
• No data caching
• No http caching
• Unnecessary cache reload
• Unnecessary conditional check
• Unnecessary object creation
• Updating unchanged data

25. Most violated high-level anti-pattern
#3: The Ramp
• Description
– Processing time increases or the system crashes as the
application gets used over a period of time
• Problem
– The application cannot handle the increased volume of
data/traffic
– Resource consumption increases or exhausts
• Solution
– Select algorithms or data structures based on maximum
size expected
– Use algorithms/designs that adapt to the increasing size
or doesn’t get effected by it (ex. flyweight
pattern, streaming technique)

26. Most violated high-level anti-pattern
#3: The Ramp
Too many objects in memory
• #4 most violated low-level anti-pattern
• Problem
– Loading large amount of data or objects at once for
processing can result to out-of-memory error and other
system-level performance impact
– Examples include unmarshalling entire DB resultset in
heap and generation of large XML documents using
DOM object
• Solution
– Flyweight and lazy loading patterns
– Streaming and chunking techniques

27. Example of streaming database result set
- Overview -
• DataList/DataListIterator
– Interface for a list/iterator data structure materialized
from a data source, such as a database or a file
– Extends the List/Iterator interface to leverage
polymorphism
• ResultSetDataList
– Implementation of DataList that’s materialized from a
SQL resultset
– Unmarshalls the resultset rows one by one as the getter
method is called
• ResultSetDataListIterator
– Iterator that’s materialized from ResultSetDataList

28. Example of streaming database result set
- Usage considerations -
Avoid out-of-memory and SQL timeout
• Use ResultSet.TYPE_FORWARD_ONLY
• Set Statement fetch size
– Default for Oracle JDBC = 10
– Default for i-net MERLIA = 0
• ‘0’ uses client cursor that reads all rows into memory
• Specify fetch size to use cursor on server side
• Use flyweight pattern in the unmarshaller if possible

29. Example of streaming database result set
- Code sample -
• See Appendix A

30. Most violated high-level anti-pattern
#3: The Ramp
Not releasing resource (immediately) after use
• #8 most violated low-level anti-pattern
• Problem
– Cause of connection leaks, deadlocks, performance
degradation and other unexpected behavior
– Threads, sockets, database connection, file
handler, and other resources can be a victim
• Solution
– Use finally block to ensure release
– Release resource as quickly as possible

31. Example of resource management
- CSContext -
CSContext csCtxt=null;
try {
csCtxt = CSContext.getContext();
// your code
} catch (Exception e) {
csCtxt.rollback();
csCtxt=null;
} finally {
if (csCtxt!=null) {
csCtxt.commit();
}

32. Most violated high-level anti-pattern
#3: The Ramp
Other low-level anti-pattern examples
• Exceeding max number of expressions in a list
• No data caching
• No priority logic
• Not releasing resource (immediately) after use
• Presentation of unlimited results
• Transaction isolation unnecessarily wide
• Unlimited cache storage

33. Low-level performance anti-patterns
fixed in Blackboard Learn™ by tier
Application
Database
Frequency (%)
UI Design
Browser
Network
0 10 20 30 40 50 60

34. Tools to identify anti-patterns:
First line of defense
• Peer code review
– Atlassian Crucible
– Review Board
• Static code analyzer
– FindBugs
– PMD
– Sonar
– Custom rules
• Performance unit testing
– JUnitPerf

35. Tools to identify anti-patterns:
Deeper analysis
• Java
– VisualVM
– Memory Analyzer (MAT)
– YourKit
• Browser
– Firebug
– Google Page Speed
– dynaTrace AJAX Edition
– Fiddler
– WebPagetest

36. Tools to identify anti-patterns:
Deeper analysis
• Database
– MS SQL Server
• SQL Server Management Studio
• SQL Server Profiler
• Performance Dashboard
– Oracle
• SQL*Plus
• Automatic Workload Repository (AWR)
• Active Session History (ASH)

37. Takeaways
• The importance of performance and scalability
for your B2
• Top violated performance anti-patterns in
Blackboard Learn™
• Forensics Tools

38. Please provide feedback for this session by emailing
DevConFeedback@blackboard.com.
The title of this session is:
Developing B2 for Performance and Scalability

39. Appendix A:
Example of streaming database result set

40. Example of streaming database result set
- DataList.java -
public interface DataList<E> extends List<E>
{
/**
Closes the list and releases all of the resources that back
the list
*/
void close();
}

41. Example of streaming database result set
- ResultSetDataList.java -
public <X> ResultSetDataList( DataListSelectQuery<X> query )
{
// make sure the query object and resultset object in the query object are not null
…
try {
if ( query.getRst().isClosed() ) {
throw new IllegalArgumentException( "Result set must not be closed" );
}
// Use this information to restrict the usage of methods that require scrolling
_resultSetType = query.getRst().getType();
} catch ( SQLException e ) {
throw new IllegalArgumentException( "Unable to validate input result set", e );
}
_query = query;
_rst = query.getRst();
_unmarshaller = query.getUnmarshaller();
…

42. Example of streaming database result set
- ResultSetDataList.java -
public E get( int index ) {
if ( _resultSetType == ResultSet.TYPE_FORWARD_ONLY ) {
String msg = "You must use the Iterator because your ResultSet is set to
TYPE_FORWARD_ONLY.";
throw new IllegalArgumentException( msg );
}
if ( index < 0 ) {
throw new IndexOutOfBoundsException();
}
try {
if ( _rst.absolute( index + 1 ) ) {
if ( _rst.getRow() != index + 1 ) {
throw new IndexOutOfBoundsException();
}
return (E) _unmarshaller.unmarshall();
} else {
throw new IndexOutOfBoundsException();
}…

43. Example of streaming database result set
- ResultSetDataList.java -
/**
The close() method closes the prepared statement and result set wrapped in this
object. This is called by the finalize method that garbage collector executes when
appropriate. However, it should still be called as soon as this object is no longer
necessary by the programmer. Please do not rely on the garbage collector.
*/
@Override
protected void finalize() throws Throwable {
try {
close();
} finally {
super.finalize();
}
}

44. Example of streaming database result set
- ResultSetDataListIterator.java -
@Override
public E next()
{
if ( !hasNext() )
{
throw new NoSuchElementException();
}
try
{
E element = (E) _list._unmarshaller.unmarshall();
_elementRetrieved = true;
return element;
}
//handle exceptions
…