The document discusses how Blackboard sizes its Academic Suite software based on benchmarking. It provides details on the benchmarking methodology, including modeling user behavior, data growth, and performance objectives. The results showed how the software performed under different workload levels on various hardware configurations. The last part discusses using the benchmark results and sizing guide to determine an institution's adoption profile and appropriate hardware configuration based on factors like sessions per hour and page loads.

1. How We Size the Academic Suite: Benchmarking at Blackboard TM Speaker: Steve Feldman Director, Software Performance Engineering and Architecture [email_address]

6. Part 1: Introduction and Methodology

7. The Performance Lifecycle Complete End to End Performance Engineering Refactoring and Optimizing End to End Performance Testing Modeling, Profiling and Simulation Data Collection & Usage Analysis Strategy, Methodology and Best Practices SPE Methodology

13. Part 2: Academic Suite Benchmark Review

17. Performance Scenarios Identical workload to the under-loaded Learning System with Community System model, but with the definition of 50 complex domain relationships. Response times calibrated to under-loaded system comparison (~5s.) Calibrated Academic Suite with Complex Domains Combination of Learning System and Community System use case interactions with 40% of the workload in a controlled Assessment Concurrency Problem. Response times calibrated to under-loaded system comparison (~5s.) Calibrated Learning System and Community System with Concurrency Model for Assessments Combination of Learning System, Community System and Content System use case interactions to reflect the budding adoption of the full Academic Suite. Response times calibrated to under-loaded system comparison (~5s.) Calibrated Academic Suite Regression test case from 6.3 performing a mix of student viewing/activity, instructor authoring and minimal administrator management. Meant to be an over-loaded system. Response times < 15s. Over-Loaded Learning System and Community System Regression test case from 6.3 performing a mix of student viewing/activity, instructor authoring and minimal administrator management. Meant to be an calibrated loaded system. Response times < 10s. Calibrated Learning System and Community System Regression test case from 6.3 performing a mix of student viewing/activity, instructor authoring and minimal administrator management. Meant to be an under-loaded system. Response times < 5s. Under-Loaded Learning System and Community System Summary/Description Workload

19. Performance Objective #1 58% 2650 minutes (4 threads) 6360 minutes Large Institution (Sun Microsystems) 58% 130 minutes (4 threads) 309 minutes Moderate Institution (Sun Microsystems) 36% 16 minutes (4 threads) 25 minutes Small Institution (Sun Microsystems) Improvement Benchmark #2 (Min. Threads) Benchmark #1 Model Name NA Not Valid 989 minutes Large Institution (Windows) 40% 2120 minutes 5389 minutes Large Institution (Linux) NA Not Valid 196 minutes Small Institution (Windows) 37% 107 minutes (4 threads) 288 minutes Moderate Institution (Linux) NA Not Valid 9 minutes Small Institution (Windows) 12 minutes (4 threads) 21 minutes Small Institution (Linux) Improvement Benchmark #2 (Min. Threads) Benchmark #1 Model Name

20. Performance Objective #2

21. Performance Objective #3

22. Performance Objectives #7 and #9

23. Performance Objective #9

24. Performance Objective #7 20343 Sessions/HR 51 UPL/Second 1,145,440 Bytes/Second 145,287 Transactions 14913 Sessions/HR 33 UPL/Second 695,319 Bytes/Second 94,181 Transactions 8212 Sessions/HR 22 UPL/Second 488,168 Bytes/Second 54,049 Transactions R7.1 High-Level R3 (Workload of 360 Possible Concurrent Simulations Learning System/Community System) R2 (Workload of 240 Possible Concurrent Simulations Learning System/Community System) R1 (Workload of 120 Possible Concurrent Simulations Learning System/Community System) Workload 24034 Sessions/HR 65 UPL/Second 1,329,037 Bytes/Second 157,629 Transactions (6-Nodes) 18455 Sessions/HR 50 UPL/Second 1,102,667 Bytes/Second 130,811 Transactions 17288 Sessions/HR 42 UPL/Second 901,103 Bytes/Second 118,754 Transactions 16063 Sessions/HR 45 UPL/Second 968,128 Bytes/Second 106,659 Transactions (4-Nodes) 13341 Sessions/HR 34 UPL/Second 729,616 Bytes/Second 90,353 Transactions 12459 Sessions/HR 31 UPL/Second 640,958 Bytes/Second 87,433 Transactions 10455 Sessions/HR 25 UPL/Second 544,673 Bytes/Second 59,239 Transactions (2 Nodes) 8080 Sessions/HR 22 UPL/Second 480,824 Bytes/Second 53,780 Transactions 7238 Sessions/Hr 19 UPL/Second 311,656 Bytes/Second 51,888 Transactions R7.1 HL Clustered R7.1 Mid-Level R7.1 Entry-Level

25. Performance Objective #7 (Cont.) 20207 Sessions/Hr 47 UPL/Second 1,014,189 Bytes/Sec 130,907 Transactions (3 Nodes) 14668 Sessions/Hr 32 UPL/Second 676,802 Bytes/Second 96,742 Transactions (2 Nodes) 12974 Sessions/Hr 35 UPL/Second 735,846 Bytes/Second 84,970 Transactions R7.1 High-Level R9 (Workload of 600 Possible Concurrent Simulations Full Academic Suite) R8 (Workload of 400 Possible Concurrent Simulations Full Academic Suite) R7 (Workload of 200 Possible Concurrent Simulations Full Academic Suite) Workload 27997 Sessions/Hr 71 UPL/Second 1,527,433 Bytes/Sec 181,121 Transactions (6 Nodes) 23056 Sessions/Hr 63 UPL/Second 1,196,553 Bytes/Sec 149,709 Transactions 12652 Sessions/Hr 25 UPL/Second 451,975 Bytes/Second 64,289 Transactions 24034 Sessions/Hr 65 UPL/Second 1,392,037 Bytes/Sec 157,629 Transactions (4 Nodes) 18857 Sessions/Hr 53 UPL/Second 1,157,486 Bytes/Second 118,353 Transactions 11908 Sessions/Hr 34 UPL/Second 668,189 Bytes/Second 77,553 Transactions 13804 Sessions/HR 36 UPL/Second 763,955 Bytes/Second 90,941 Transactions (4 Nodes) 12548 Sessions/Hr 33 UPL/Second 728,082 Bytes/Second 84,004 Transactions 5721 Sessions/Hr 13 UPL/Second 275,672 Bytes/Second 37,313 Transactions R7.1 HL Clustered R7.1 Mid-Level R7.1 Entry-Level

26. Performance Objectives #5, 6, 7, 8 and 10

27. Part 3: Working with the Sizing Guide

42. Storage 3:1 Greater then 1 TB 600,000 500,000 4:1 800 GB 300,000 50,000 5:1 200 GB 50,000 5,000 10:1 20 GB 7,000 500 Ratio of File System to Database Storage File System Size Number of Existing Users Number of Existing Courses

44. Part 4: References and Resources

47. Questions?