Slides from Lightning talk on Faban at FOSDEM 2010

1. Developing benchmarks and workloads using
Faban 1.0
Amanda Waite – Sun Microsystems, UK

2. What is Faban anyway?
● Framework and toolkit to aid test/benchmark
development and automate test run execution
● Developed by Sun - Open Source
● http://faban.sunsource.net
● Harness – benchmark process automation,
collect statistics, display results
● Driver Framework – benchmark development
framework and component model
● Analysis Tool – graph, tabulate and compare
results

4. Detailed result graphs

5. System Stats

6. Faban Architecture
Control
Master
Agent
Agent
Agent
Web Server Driver Driver Driver
Run Queue
Load
Log Server Agent
Agent
SUT SUT
System System
Benchmark
Control

7. Stuff You Give to Faban
● Systems
● System(s) Under Test (host:port)
● DB Server(s)
● File Server(s)
● Reverse Proxies (host:port)
● Memcache instance(s) (host:port)
● Benchmark Driver
● Configuration

8. Stuff Faban Gives You
● A harness that:
● runs your driver code
● manages all of your systems
– Including running stat tools
● gathers data (run analysis and system stats/config)
● manages your runs
● A driver API and all of the plumbing
● Configuration screens (web based)
● Auto-generated from configuration file

9. Configuration

10. More Configuration

11. Pluggable Services/Tools
Infrastructure
My Service
My Service
Tools
ToolSets
ApacheHttpd
ApacheHttpd
WebLogic
SPECjEnt
WebLogic
SPECjEnt
Olio
Service
Olio
Faban
Faban

12. Service/Tool Control
Start
Configure
Start
Steady State ● Life-cycle control
for services/tools
Benchmark Run
● Collects/post-
Service
processes logs
Tool
and stats
● Archives
service/tool data
Stop with run
Ramp Down
Post-Process/Get ● Implementation
Post-Process
knows to invoke
Stop/Get external service or
external tool

13. Faban Driver
● Java Class (or classes) that defines a workload
● Used by Faban to Drive load
● Uses annotations to define Operations
● Operations are complete interactions, i.e.:
● GET html page
● Load static resources and images called out in page
● Optionally redirect based on HTTP return code
● Validate success of operation
● Operation Mix defines how and when
operations are executed

14. Driver Framework
@BenchmarkDefinition(name = "OlioRails",
version = "0.2",
scaleName = "Concurrent Users")
@BenchmarkDriver(name = "UIDriver",
threadPerScale = 1)
// 90/10 Read/Write ratio
@MatrixMix(
operations = {"HomePage", "Login", "TagSearch", "EventDetail", "PersonDetail",
"AddPerson", "AddEvent"},
mix = {@Row({0, 11, 52, 36, 0, 1, 0}), // Home Page
@Row({0, 0, 60, 20, 0, 0, 20}), // Login
@Row({21, 6, 41, 31, 0, 1, 0}), // Tag Search
@Row({72, 21, 0, 0, 6, 1, 0}), // Event Detail
@Row({52, 6, 0, 31, 11, 0, 0}), // Person Detail
@Row({0, 0, 0, 0, 100, 0, 0}), // Add Person
@Row({0, 0, 0, 100, 0, 0, 0}) // Add Event
})
@NegativeExponential(cycleType = CycleType.CYCLETIME,
cycleMean = 5000,
cycleDeviation = 2)
public class UIDriver {

15. Driver Framework
@BenchmarkOperation(name = "HomePage",
max90th = 1,
timing = Timing.AUTO)
public void doHomePage() throws IOException {
http.fetchURL(homepageURL);
@BenchmarkOperation(name = "AddEvent",
max90th = 4,
timing = Timing.MANUAL)
public void doAddEvent() throws Exception {
if (!isLoggedOn) {
throw new IOException("User not logged when trying to add an event");
}
ctx.recordTime();
loadStatics(addEventStatics);
GetMethod eventForm = new GetMethod(addEventURL);

16. Get it, try it, contribute
http://faban.sunsource.net