2. EXECUTIVE SUMMARY
With online learning taking a more central role in education, continuous system
availability has become a critical requirement. Many institutions are implementing
their Blackboard Learn™ environments with cluster configurations to reduce the
risk of service outages.
This paper discusses how Blackboard Learn service levels can be increased using
Windows 2008 R2 cluster services for Microsoft SQL Server 2008 R2 running on
Dell PowerEdge servers, Dell EqualLogic and Dell PowerVault storage.
Utilizing the capabilities of the Dell Public Solutions Center, Blackboard, Dell, and
Microsoft partnered to create a solution architecture blueprint that delivers high
availability for Blackboard Learn. This paper describes the solution architecture
and provides configuration guidelines as well as recommended best practices.
Benefits of the Solution Architecture
The solution architecture is designed to help customers achieve:
Reduced risk of downtime through a reliable technology infrastructure,
jointly developed and tested by Dell, Blackboard and Microsoft. The reliable
infrastructure translates into continuous learning as everyone on campus
has access to the data they need.
Increased uptime through high availability features in Dell server and
storage systems, replicated components in the application tier, a storage
configuration with RAID 5 or 10 protection, and a failover cluster service for
the database server.
Investment protection through the scalable Dell server and storage com-
ponents as well as the ability to use Microsoft Windows and Microsoft SQL
server technologies and to leverage existing staff expertise.
Lower TCO due to outstanding price/performance of the virtualized envi-
ronment and simplified management of virtualized resources, including VM
cloning techniques that enable rapid provisioning.
2
4. Introduction
The Blackboard Learn architecture has built-in availability for the application tier when implemented using horizontally
scaled application instances. Adding clustering to the database server is the recommended method for improving service
level availability even further. Clustering enables failover between redundant physical servers or between redundant
virtual machines (VMs) on the same physical server. Many customers today are deploying their application tier instances
in a virtualized manner and a similar approach can also be taken with the database tier when deploying a database cluster.
The two primary benefits for deploying the database in a clustered configuration are:
Service level availability
When a hardware or software failure occurs on one database server node, the SQL Server database can failover
to the other node with minimal impact to end users. Often end users don’t even notice the slight delay or tempo-
rary slow down in performance during the failover process, which takes a matter of seconds.
Simplified management
A single point of administration for the cluster and its resources makes the cluster easier to manage than manag-
ing two servers or VMs individually.
The solution architecture blueprint is based on Microsoft Windows Server 2008 R2 and Microsoft SQL Server 2008 R2,
Dell PowerEdge servers, Dell EqualLogic storage, and Blackboard Learn. The hardware and software components in
the architecture have been tested together and are validated to achieve optimal performance for supporting the two
workloads, Standard and Advanced Sizing configurations for Blackboard Learn.
DELL AND MICROSOFT HIGHLY AVAILABLE ARCHITECTURE FOR
BLACKBOARD LEARN™
The virtualized highly available architecture from Dell, architecture utilizes server virtualization with Microsoft
Microsoft, and Blackboard offers increased availability Hyper-V to deliver maximum throughput on multicore
at an affordable price. Delivering the solution on a pure Dell PowerEdge servers.
Microsoft platform enables institutions to deploy the
Figure 1 shows a logical representation of the virtualized
Blackboard Learn platform in a consistent and scalable
architecture. Solution components are deployed across an
manner with other enterprise solutions that may already
application tier, a database tier, and a storage tier. It should
be running on a Microsoft platform.
be noted that while this paper provides specific Dell server
The solution is built on a robust server and storage archi- and storage recommendations for two Blackboard Learn
tecture based on virtualized Dell PowerEdge servers with configurations, Dell offers numerous server and storage
Intel Xeon Processor E7 family CPUs and Dell EqualLogic options. A Dell representative can design a customized
iSCSI and Dell PowerVault CIFS storage systems. The configuration to meet any unique customer requirements.
4
5. VIRTUALIZATION WITH
VIRTUALIZED DELL AND MICROSOFT MICROSOFT WINDOWS SERVER
ARCHITECTURE FOR BLACKBOARD LEARN 2008 R2 HYPER-V
The virtualization infrastructure is implemented
Application Tier using Microsoft Windows Server 2008 R2 Hyper-
VM # 1
Microsoft
Hyper-V V, which provides a dynamic, reliable, and scal-
Blackboard Learn Virtualization
able virtualization platform. It also offers a single
VM # n
Blackboard Learn
set of integrated management tools to manage
Storage Tier
Dell PowerEdge both physical and virtual resources, enabling the
R810 Servers
creation of agile and dynamic data centers.
Dell PowerVault
NX3100
Some of the key features of Hyper-V R2 that
Dell EqualLogic Database Tier
PS6000 Family
Storage Systems
Dell PowerVault
NX1950
make it a good fit for the Blackboard Learn solu-
VM # 1
Microsoft SQL
Server 2008 R
Dell PowerVault
NX1950
Dell PowerVault
NX1950
tion include:
VM # n
Microsoft SQL
Server 2008 R
Virtual Machine Live Migration so that VMs
can be migrated from one server to another
R2
Dell PowerEdge
R810 Servers without disruption.
Support for Cluster Share Volumes (CSV)
that enable clustered VMs to easily share a
Figure 1: Logical diagram of the solution architecture.
single disk volume.
Dynamic reallocation of memory
between different VMs in response to
changing workloads.
SERVER VIRTUALIZATION FLEXIBILITY Integration with System Center Virtual
Machine Manager 2008 R2 (SCVMM 2008
Virtual Machine # A-1: App Server Virtual Machine # B-n: Database Server R2) for simplified management of both
8-32 GB memory 64-256 GB memory
One virtual CPU thread
virtual and physical server resources.
2 -4 virtual CPU threads
for every two in App tier
Blackboard Learn
Windows Server 2008 R2
Web/App server
SQL Server 2008 R2
Building Availability into the
Virtual Machine # A-2: App Server
Virtualized Environment
Virtual Machine # B-2: App Server
8-32 GB memory 8-32 GB memory
2-4 virtual CPU threads 2-4 Virtual CPU threads
Blackboard Learn Blackboard Learn
For maximum availability, it is recommended to
Web/App server Web/App server
have at least two physical servers in the archi-
tecture so that the loss of one physical server
Virtual Machine # A-n: Database Server Virtual Machine # B-n: App Server
64-256 GB memory will not bring down all instances of the database
8-32 GB memory
One virtual CPU thread
for every two in App tier 2-4 Virtual CPU threads nor all application tier instances. Figure 2 shows
Windows Server 2008 R2 Blackboard Learn
SQL Server 2008 R2 Web/App server an example of how database and application tier
instances can be spread across two Dell Power-
Edge R810 servers to avoid having a single point
Dell PowerEdge R810 Server A Dell PowerEdge R810 Server B of failure.
Figure 2: VMs can be deployed across multiple physical servers for
maximum availability.
blackboard.com
6. HOW MICROSOFT SQL SERVER 2008 Failover can be initiated automatically in the event of a failure
FAILOVER CLUSTERING WORKS (e.g. hardware, operating system, application, or service
SQL Server failover clustering provides high-availability failure), or manually such as for a planned upgrade. During
support for an entire SQL Server instance and is built on a failover, the ownership of the cluster resource group is
top of an established Windows Server failover cluster. transferred to another node in the cluster. In the case of the
A Windows Server failover cluster aims to provide high Blackboard Learn database server, only two cluster nodes
availability for services or applications that run within the are required, so ownership is simply transferred to the other
failover cluster. It contains a group of independent cluster identically configured database server node.
nodes (physical servers or virtual machines) that work The Windows Server failover cluster service monitors and
together to increase the availability of applications and ser- manages the SQL Server 2008 instances and assigns their
vices. Failover clustering can protect against hardware and network names and IP addresses. A virtual name and IP
software failures by failing over resources from one cluster address is assigned to the entire cluster resource group,
node to another as required. providing a unique identifier that clients (application
Failover is the process of taking a clustered service or servers) can use to connect to the SQL Server instance
application offline on one node and bringing it back online regardless of which node is currently the active instance. As
on another node. This process is typically transparent to illustrated in Figure 3, when the resource group fails over,
the users, who should experience a minimal disruption of the virtual name and IP address are redirected to the new
service when a failover occurs. active node. This process is transparent to clients so that
there is no need to change the name or IP address that they
From a logical standpoint, a SQL Server failover cluster
are using to connect to the SQL Server database.
appears on the network as a single SQL Server instance on
a single computer. The cluster nodes even use a common One important new feature of Windows Server 2008 to
LUN for accessing shared storage as described in the improve availability is the removal of the previous depen-
storage architecture section below. At any point in time, dency on a single shared disk to establish a quorum. In
only one of the cluster nodes is active and owns the cluster cluster configurations, the term quorum is used to define
resource group including the shared storage resources. All the list of components that must be online for that cluster
client requests are served by the active SQL Server instance to continue running. In Windows Server 2003, a node
and the other SQL Server instance is in passive mode until could continue participating in the cluster only as long as
a failover is initiated. it remained in communication with a shared disk known
Active Database Connections Are
Blackboard Blackboard
Server Node Fails Users Users Re-established to
New Active Node
Failover
Blackboard Blackboard
Application Instances Results Application Instances
in Role
Reversal for
DB Server
Nodes
Database Database Database Database
Server Instance # 1 Server Instance # 2 Server Instance # 1 Server Instance # 2
Active Passive Passive Active
Cluster Cluster
Shared Shared
Storage Storage
Figure 3: Database server cluster failover process.
6
7. as the quorum disk. This quorum disk was the same for all The Dell PowerEdge servers can be configured to support
nodes, thus making it a single point of failure. This single either a Standard Sizing configuration or an Advanced
point of failure was removed in Windows Server 2008 by Sizing configuration. The Standard Sizing configuration is
adding support for four quorum modes that give increased recommended for institutions that have minimal deploy-
flexibility for how quorums are established. ment of distance learning or connected classroom initia-
tives (less than 5%). It is designed to balance availability
The Blackboard Learn database server employs a cluster
and performance using cost-effective platforms.
of two or more virtual or physical nodes. Each node runs
an instance of the SQL Server database as shown in Figure The Advanced Sizing configuration is designed for institu-
2 in the above section. In the event of a failover, users may tions of varying sizes that are deploying distance learning
notice a brief delay of a few seconds, but can continue their programs or connected classroom initiatives to a portion of
Blackboard Learn session from wherever they were before their communities. The configuration is designed to deliver
the failover. There is no need for users to login again or to both high-availability and high-performance.
restart their Blackboard Learn session. If a user is in the
The performance capacity of both the Standard and
middle of a task during a failover, it will appear completely
Advanced Sizing configurations can be scaled by adding
seamless with the unlikely loss of connectivity to the appli-
more VMs with Blackboard Learn application instances to
cation or loss of data.
the application tier and simultaneously increasing the CPU
RECOMMENDED DELL SERVER and memory that is allocated to the database instances.
CONFIGURATIONS The same standardized VM is deployed for the application
The entire solution architecture runs on Dell PowerEdge
tier in both configurations. The standardized application
R810 servers, which offer outstanding density for virtualized
tier VM runs a single Blackboard Learn application instance
workloads because of their large memory footprint and
and a Web/application server instance, and is configured
support for up to four, ten-core CPUs from the Intel® Xeon®
with 2-4 CPUs and 8-32 GB of memory. These standardized
processor E7 family. The Intel Xeon processor E7 family sup-
VMs are then used as building blocks for adding scalable
ports hyper-threading technology, which means that each of
performance and availability to the application tier.
the ten CPU cores on a single processor provide two threads.
Thus a single CPU with 10 cores provides 20 CPU threads.
Description Standard Sizing Configuration Advanced Sizing Configuration
Requirements for a Single • 2-4 CPU threads or 1-2 processor cores • 2-4 CPU threads or 1-2 processor cores
Application Tier VM • 8-32 GB memory • 8-32 GB memory
instance
Number of VMs in the
2-6 6-12
Application Tier
Requirements for each • 1 CPU thread for every 2 CPU threads in • 1 CPU thread for every 2 CPU threads in
identical Database Tier VM the application tier the application tier
instance • 32 GB memory plus 3 GB for every VM • 32 GB memory plus 3 GB for every VM
in the application tier in the application tier
Number of VMs in the
2 2
Database Tier
Total hardware requirements Two Dell PowerEdge R810 servers Three Dell PowerEdge R810 servers
• Two 10-core Intel Xeon E7 Processors • Two 10-core Intel Xeon E7 Processors
for a total of 40 CPU threads per server for a total of 40 CPU threads per server
• 64 to 256 GB memory per server • 64 to 256 GB memory per server
Supported number of
20,000 40,000
concurrent user sessions
Table 1: Standard and Advanced Sizing configuration examples.
blackboard.com
8. Table 1 provides the sizing guidelines for the Standard and STORAGE ARCHITECTURE
Advanced Sizing configurations. The ranges shown for CPU Storage architecture is a critical design consideration for
threads, memory requirements, and number of VMs in the Hyper-V cloud solutions. One of the key principles in the
application tier indicate the minimum and desired amounts. storage architecture of this solution is the use of Cluster
The stated user loads supported by each configuration are Share Volumes. Volumes that are configured as Cluster
based on deployments with four CPU threads and 32 GB of Shared Volumes can be accessed by all nodes of a failover
memory per application tier VM. A total of four such VMs cluster. This is what enables all Blackboard Learn database
in the Standard Sizing configuration will support 20,000 server nodes or VMs to open and manage files on the same
concurrent user sessions for a peak hour. A total of eight storage volume.
such VMs in the Advanced Sizing configuration will support Another key design feature of this storage architecture is
40,000 concurrent user sessions for a peak hour. A concur- the use of a network attached storage (NAS) gateway. In
rent session is a user who has logged into the application this example configuration, the Dell PowerVault NX3100
within the last hour. acts as a front-end to the Dell EqualLogic storage arrays,
For sizing the database tier, the rule of thumb is that each enabling them to be accessed via an NFS mount from VMs in
database server instance needs half as many CPU threads the application tier. The NFS mount enables the Blackboard
as there are total CPU threads in the application tier and it Learncontent file system as well as system infrastructure
needs an additional 3 GB of memory (beyond the base 32 files such as operating system and VM configuration files to
GB) for every VM deployed in the application tier. Thus if be accessed via the CIFS protocol.
there are four application tier VMs and each application tier The Dell EqualLogic storage arrays are also set up as direct
VM is configured with four CPU threads (16 threads total), attached storage for the Blackboard Learn database server
the database tier would need 8 CPU threads and at least 44 VMs. This enables the database server VMs to gain access
GB of memory. to the shared cluster storage via an iSCSI connection.
LOGICAL STORAGE ARCHITECTURE
Pool of VM servers Dell EqualLogic PS Series storage pool
Common LUNs for accessing Cluster Share Volumes
Cluster Share Volumes with iSCSI connections
DB tier VM containing via Hyper-V Disk Volume #1
Blackboard DB Server Instance System Infrastructure (Hyper-V VM Images)
DB tier VM containing Disk Volume #2
Blackboard DB Server Instance Blackboard Database
Hyper-V R2
App tier VM containing Disk Volume #3
Blackboard Application Instance System Infrastructure (Windows OS Images)
App tier VM containing Dell PowerVault Disk Volume #4
Blackboard Application Instance NX3100 iSCSI connections Blackboard Content File System
NFS exports to Dell EqualLogic
Storage
App tier VM containing Disk Volume #n
Blackboard Application Instance Blackboard Content File System
Figure 4 Illustrates the logical design of the storage architecture. The following section explains the Blackboard Learn file systems that
are highlighted in the figure.
8
9. Blackboard Learn File Systems Recommended Storage
The Blackboard Learn storage environment consists of three Configurations
primary file systems as follows: There are two primary aspects for sizing the
storage arrays. The arrays must support a high
Blackboard Learn content file system—This file system con-
tains course data that is accessed by teachers and students enough throughput of IOPS (I/O operations per
when using the Blackboard Learn application. The stored second) so that the database server or Black-
artifacts can be large files such as videos or graphics so board Learn application instances will not be idle
cost effective storage capacity is important. File access is waiting for storage. Secondly, the storage arrays
occasional, making I/O throughput less critical than capac- must have enough capacity to store the expected
ity. It is recommended to use high capacity SATA drives for
data files.
this file system and the storage area must be configured for
network access via the CIFS protocol. Table 2 shows the recommended Dell storage
Blackboard Learn database—The Blackboard Learn data- solutions to match the performance and capacity
base generally requires only about 20% of the capacity of needs for Blackboard Learn Standard Sizing and
the Blackboard Learn content file system. However, it can Advanced Sizing configurations. The Blackboard
have a high volume of I/O activity and thus requires a high Learn database and the system infrastructure files
performance storage solution, especially when there are share the same physical storage array and thus
more users as in the Advanced Sizing configuration. The
are listed together in the first row of the table.
storage system used to maintain the database files must
However, these file systems will each have their
have a direct connection to the physical server that houses
the Blackboard Learn database. own volume on the shared storage array.
System infrastructure files—The Blackboard Learn virtual- As mentioned above, a Dell NAS device is used as
ized server architecture requires a shared storage area a gateway to the Dell EqualLogic storage arrays.
where the Microsoft Windows operating system files and The final row of Table 2 shows the recommended
Hyper-V virtual machine images can be maintained. These Dell PowerVault NX3100 network attached
files do not take up much space, but must have reasonable
storage system. It communicates with the Dell
I/O performance for access by virtual machines in the
EqualLogic storage arrays via the iSCSI protocol
application tier using the CIFS protocol. It is recommended
to place this file system on the same physical storage and then presents the appropriate disk volumes to
environment with the Blackboard Learn database. the application tier via an NFS mount.
Description Standard Sizing Configuration Advanced Sizing Configuration
Blackboard Learn Dell EqualLogic PS6000XV iSCSI SAN Dell EqualLogic PS6000XVS virtualized
database and system Storage iSCSI storage array
infrastructure files • Approximately 3,400 IOPS • Approximately 7,000 IOPS
• Approximately 3.7 TB usable space • Approximately 2.5 TB usable space
with RAID 10 with RAID 6
Blackboard Learn Dell EqualLogic PS6000E iSCSI Array Dell EqualLogic PS6500E iSCSI Array
content file system • Approximately 1,000 IOPS • Approximately 2,600 IOPS
• Approximately 12.6 TB usable space • Approximately 18.0 TB usable space
with RAID 10 with RAID 5
Network Attached Dell PowerVault NX3100 Network Dell PowerVault NX3100 Network
Storage gateway Attached Storage Attached Storage
Table 2: Storage configuration examples.
blackboard.com
10. ADDITIONAL BEST PRACTICES AND CONFIGURATION GUIDELINES
The guidelines in this section represent best practices not only for high availability, but also for configuring the environment to
achieve the stated performance for the Standard and Advanced Sizing workloads. The guidelines are listed below by category.
Application Tier Configuration Guidelines RAID 10 configuration is recommended for the disk
Allocate four to 32 GB of memory for the Java Heap drives that store database files.
in each VM in the application tier. Most configurations RAID 5 or RAID 10 can be used for the disk drives
run with eight to 16 GB of heap. that store the Blackboard Learn content file system. In
The Blackboard Learn content system files are recom- cases where the selected storage solution offers more
mended to be stored on a filer on the network rather than adequate I/O throughput, use RAID 5. Otherwise,
than in local storage on a server to make it easy to move use RAID 10.
Blackboard Learn application tier VM instances between For best database performance, use separate drives (on
servers without having to remount the file systems. the same storage device) to store database data files vs.
log files and tempdb files. This allows these files to be
Storage Tier Configuration Guidelines read/written in parallel (reducing I/O wait time)
The consolidated storage environment should have
dual connections between all servers and storage Network Configuration Guidelines
devices for high availability Servers that will be housing application tier VMs
While storage capacity requirements for the should have at least two NIC ports dedicated to iSCSI
Blackboard Learn content file system are significant, traffic. This provides redundancy for iSCSI com-
access speed is less important. Therefore 7200 or munication between the Hyper-V R2 hosts and the
10K RPM SATA drives are generally sufficient for the EqualLogic PS storage array.
Blackboard Learn content file system. Servers that will be housing database server instances
The database files require high performance access. should have four NIC ports dedicated to iSCSI traffic.
This can either be achieved by using 15,000 RPM SAS The unified network storage system should have four
drives for the file system containing the database files NIC ports dedicated to iSCSI traffic and two NIC ports
or by combining solid-state drives (SSDs) with SAS dedicated to NFS file server traffic.
drives such as in the award winning Dell EqualLogic
Jumbo frames should be enabled on the network switch.
PS6000XVS virtualized iSCSI storage array.
SUMMARY
This solution architecture blueprint is a result of the combined efforts of Blackboard, Dell, and Microsoft to design a cost
effective virtualized infrastructure that offers high service levels and predictable scalability. Customers can leverage this
solution to:
Reduce risk by deploying a reliable technology infrastructure in which the components have been tested together
Increase uptime through high availability features and a highly available architecture
Protect investments through scalable Dell systems and the ability to leverage a Microsoft infrastructure
Reduce TCO with an affordable approach to high availability and outstanding price/performance
10
11. For More Information
To find out more about Blackboard, Dell, and Microsoft offerings for highly available online learning solutions, contact a
Blackboard representative or visit the Web sites listed in Table 3 below.
URL Description
http://www.blackboard.com Blackboard Inc. home page
http://library.blackboard. Blackboard Learn Hardware Sizing Guide for Dell
com/d/?bfb0e63e-4ef7-44de-b86b-86e8c74f6f08 Deployments (Release 9.1, June 2010)
http://www.dell.com/hied Dell solutions for higher education
http://www.dell.com/poweredge Dell PowerEdge servers
http://www.dell.com/equallogic Dell EqualLogic PS6000 Series iSCSI SAN array
http://www.microsoft.com/windowsserver2008/ Microsoft Windows Server 2008 R2
http://www.microsoft.com/windowsserver2008/en/us/
Microsoft Windows Server 2008 R2 Hyper-V
hyperv-overview.aspx
http://www.microsoft.com/sqlserver/2008/en/us/R2.aspx Microsoft SQL Server 2008 R2
Table 3: Web Links for Additional Information.
APPENDIX A—HARDWARE AND SOFTWARE COMPONENT DESCRIPTIONS
Tables 4 and 5 describe the recommended hardware and software components within the solution architecture.
Hardware Component Description
Dell PowerEdge R810 The large memory footprint of the Dell PowerEdge R810 server along with advanced reliability and
server availability features make it a great fit for virtualized environments. These servers, which support up to
four, ten-core CPUs from the Intel® Xeon® processor E7 family, are used in both the application tier and
the database tier.
Dell EqualLogic The Dell EqualLogic PS6000E is a virtualized iSCSI SAN that combines intelligence and automation
PS6000E iSCSI Array with fault tolerance to provide simplified administration, rapid deployment, enterprise performance
and reliability, and seamless scalability. With SATA disk drives the PS6000E delivers good cost-per-
Gigabyte with up to 32TB s of storage, making it a good fit for the Blackboard Learn content file
system.
Dell EqualLogic The Dell EqualLogic PS6500E is a high-density, virtualized iSCSI SAN that combines intelligence and
PS6500E iSCSI Array automation with fault tolerance to provide simplified administration, rapid deployment, enterprise
performance and reliability, and seamless scalability. With the lowest cost-per-GB in the EqualLogic
line, the EqualLogic PS6500E provides up to 48TB of SATA storage in a dense 4U chassis, making it a
good fit for storing the Blackboard Learn content file system for Advanced Sizing configurations.
Dell EqualLogic The Dell EqualLogic PS5000XV array is a virtualized iSCSI SAN solution that combines intelligence
PS6000XV iSCSI SAN and automation with fault tolerance to provide simplified administration, rapid deployment, enterprise
Storage performance and reliability, and seamless scalability. With 15,000 RPM SAS disk drives and automated
load balancing, the PS6000XV array is an ideal SAN platform for the database server in the standard
configuration.
Dell EqualLogic The PS6000XVS virtualized iSCSI storage array is ideal for optimizing responsiveness of I/O intensive
PS6000XVS iSCSI applications such as the Blackboard Learn database server. It provides intelligent data tiering within
Storage Array the array by combining low-latency solid-state drives (SSDs) with performance-sensitive 15,000 rpm
SAS hard disks to deliver more IOPS.
Dell PowerVault The Dell PowerVault NX3100 is a unified network storage solution that simultaneously stores both
NX3100 Network file and application data. It can be used as an integrated storage solution or a NAS gateway and is
Attached Storage deployed in this solution architecture as a NAS gateway connecting to the Dell EqualLogic storage
systems to enable access by application tier VMs.
Table 4: Recommended Server and Storage Components.
blackboard.com