Exalogic is an impressive piece of hardware offering immense performance. However the smallest configuration is 96 cores, 768 GB memory and a 40TB SAN... way bigger than many smaller customers could imagine using (even including test environments). This session takes a look at how you could use modern server technology, such as blades, to build a smaller version of Exalogic, and yet still benefit from some of the cost savings from sophisticated automation. This will include a case study of a mid-sized installation where these techniques have been used. Delivered on 5 December 2011 at UKOUG 2011 by Simon Haslam.

1. Build Your Own
Middleware Machine
Simon Haslam
Veriton Limited
1 (1.0h)

2. Simon Haslam / Veriton
Specialised consultant & Oracle Partner,
established for 15 years
Demanding web & call-centre applications
Architecture & development strategy;
health-checks; disaster recovery; tuning
Oracle Fusion Middleware
(JavaEE, SSO, OAM, OID, clustering)
ADF Applications (esp. strategy & admin)
Database & related technologies
(Solaris/Linux, load balancers, firewalls, …)
2 (1.0h)

3. Agenda
1. My Ideal Middleware Machine
2. Hardware
3. Software & Configuration
4. Non-functional Factors
5. Case Study
6. Summary

4. Background to this presentation
• Julian Dyke & I came up with ‘RAC-in-a-
BOX’ in 2006
– 2 * HP DL365 (8 cores each)
– 1 * MSA1000 SAN (HA controllers)
– 2 * Cisco 3750 switches
• Standard OFA style installation of RAC,
config of ASM etc
• Ready to plug into network & power
4 (1.0h)

5. From the ‘Old’ Days to Today
• Was physical hosts with DAS
• Storage arrays for consolidating storage
• Now: most mid-size sites
– Production: physical hosts for production
databases
– Dev & Test: virtualised environments very
common
• (leaving discussions about public cloud /
hybrid models)
5 (1.0h)

6. From now to future
Public
Cloud
Remote
On Premise
Dedicated
Hardware
for Application
Compute platform
Storage platform
Private
Appliance Cloud
Appliance
Or heavy
consolidation
Physical Virtualised
6 (1.0h)

7. Trends
• Green IT
– Carbon costing
– Power costs
– SCR
• Convergence
– FCoE
– iSCSI maturing
• Technology
– Speed of commodity hardware
– Virtualisation
– Networks VLANs: now traffic management, LACP etc.
– 10GbE more common
– Not the bandwidth/latency of InfiniBand but may be good enough
for most, certainly mid-size, sites
7 (1.0h)

8. Exalogic is BIG!
• Even ¼ rack Exalogic is:
– 8 compute nodes @ 12 = 96 cores
– WebLogic Suite list price = £1.4M
• Even if ½ processor & ½ NUP = £870k
– Add: WLS Management Pack, Cloud Management Pack..?
– List price of hardware $475k (~ £320k)
• Many of my customers only have two or three big web/call centre apps:
– modern processors => production can run on 2 compute nodes (for HA)
– with virtualisation 1 compute nodes (12 cores) can run multiple test envs
– many still have Forms & Reports – expansion of Java EE apps likely over
next 5 years but not from Day 1
– Web app to database traffic is modest
• Nothing wrong with Exalogic… it is just very BIG
Oracle OpenWorld 2011: announcement 1/8 rack Exalogic
(4 x 12 core 96GB compute nodes, same NAS and IB)
8 (1.0h)

9. My ideal ♥ Middleware Machine!
• Hardware
– Compute: fast processors & ‘plenty’ of memory
– Stateless compute nodes
– Networks for public, private, management
– Power management
– Totally managed remotely
– (note: I’ve not mentioned storage)
• Software
– WebLogic
– Web Tier
– Load Balancing
– Management
– Layered products (old and new)
9 (1.0h)

10. Confession: I am a blade convert!
10 (1.0h)

11. Agenda
1. My Ideal Middleware Machine
2. Hardware
3. Software & Configuration
4. Non-functional Factors
5. Case Study
6. Summary

12. Ingredients: 1. Compute
12 (1.0h)

13. Ingredients: 2. Power
• Single or 3 phase
• Note: health and safety implications
13 (1.0h)

14. Ingredients: 3. Storage
14 (1.0h)

15. Ingredients: 4. Network
• Core switches
• Load balancers
15 (1.0h)

16. Bake & Leave to Cool
16 (1.0h)

17. Traditional Rackmount vs Blade?
• Power consumption
http://www.veriton.co.uk/roller/fmw/entry/...
hp_blade_vs_rack_servers
• Costs
• Note some chassis-wide costs:
– Converged networking
– Administration modules
• Vendor lock-in
17 (1.0h)

18. Changes
from adding fans/
power supplies
Size of error
bars unknown!
18 (1.0h)

19. Exalogic
1/4 rack 8n
(inc storage)
5.3-7.2kW*
Exalogic
1/8th rack 4n
(inc storage)
2.4-4.3kW*
Exalogic
1/2 rack 16n: 8-11kW
1 rack 32n: 13-18kW
Note: Exalogic
has 96GB RAM
per node cf 24GB,
SSD boot disks
and slightly faster procs
(3.0GHz cf 2.8GHz)
19 (1.0h) * Exalogic “typical” to “max” (from Oracle datasheet 14-NOV-11) – rounded to 2 sig. fig.

20. Compute Node Storage
• What would you put on a server if you had
shared storage?
– Operating system files?
– Swap space
– Crash dumps
– Application log files
– Transaction logs?!
• Do you need SSD for this?
• As soon as you add storage to a server you
make it stateful
20 (1.0h)

21. Hardware Summary
• Very fast processors
• No local storage; boot from SAN
• Shared 10GbE networking (max 20GbE per
blade)
• Centralised hardware management
• Centralised power management
21 (1.0h)

22. Agenda
1. My Ideal Middleware Machine
2. Hardware
3. Software & Configuration
4. Case Study
5. Non-functional Factors
6. Summary

23. Provisioning
• Stateless blades (hardware)
– No disk (or SDcard!)
– No MAC addresses
– No WWNs
– No Lights-Out Management addresses
= ~virtualisation of the hardware
23 (1.0h)

24. Hardware/OS Configuration
• Pre-assign IP addresses
– Management
– Public Prod
– Test
– Private
– Cluster
24 (1.0h)

25. Software Installation - today
• Centralised Deployment
– Scripted (e.g. WLST & response files)
– Enterprise Manager – Deployment Pack (still
some work to be done)
25 (1.0h)

26. Virtualisation - notes
• VMs – build from kickstart/jumpstart etc
• There’s a lot to learn from AWS
• Oracle Virtual Assembly Builder is a very
interesting product for the future
– VM template libraries
• OVAB probably the future if you’re on OVM
• OVAB apparently underpinning Java Service of
Oracle Public Cloud
26 (1.0h)

27. Middleware Installation
• Shared middleware homes
– Pros and Cons
– Great for support ☺
• E.g. Exalogic EDG
27 (1.0h)

28. Agenda
1. My Ideal Middleware Machine
2. Hardware
3. Software & Configuration
4. Case Study
5. Non-functional Factors
6. Summary

29. Case Study
• Mid-sized UK customer
• ~800 agents in 4 call centres + web site
• Previously 10g iAS on 6/8 2U rack servers
(production, primary site)
• Migration to WebLogic Suite 11g (plus OID
11g & Reports 11g)
• Moving from Red Hat to Oracle Linux 5.6*
• Virtualised
• Centralised Oracle homes and management
29 (1.0h)

30. How did it go?
• Power was a challenge
– you need to size for full capacity
• Cooling
– fortunately able to free up space, but could have been
tricky due to heat density
• Boot from SAN
– Not really a problem – much more common these days
– I suspect (I) will get blamed for anything bad
• Networking
– Forced through a rethink of some of the VLANs and
subnets
– We did have some problems with LACP/trunking
30 (1.0h)

31. Now?
• A project appeared from blue, sneaked in
and went live on the blade system before
Oracle project out of testing!
• Chassis will probably be full within a year
31 (1.0h)

32. Agenda
1. My Ideal Middleware Machine
2. Hardware
3. Software & Configuration
4. Case Study
5. Non-functional Factors
6. Summary

33. Remember…
Licences
and support
Hardware
33 (1.0h)

34. Politics
• Multiple teams are often responsible for
support:
– Application Development
– Oracle Administration
– System (e.g. Unix) Administration
– Storage Management
– Data Centre Management
34 (1.0h)

35. Real-World Example in 2011
• Oracle team had bought lots of SOA Suite licences (sizing method a bit
unknown – partly historical, partly reseller)
• Production hardware still to be purchased
• Licences only efficiently “fitted” on certain combinations of hardware
at primary and DR site
• System Administration team different to Oracle/project team
• Cost of Oracle licences > 15x cost of hardware
• Cost centre for hardware/OS/VMware seemed to be different from cost
centre for Oracle => smallest amount of hardware bought*
Oracle perpetual licences plus Support/Updates unused
• This is not unusual behaviour!
* I do have some sympathy here – the licences should have been
right-sized in the first place
35 (1.0h)

36. Real-World Example:
What if they had chosen Exalogic?
• 2 Exalogic ¼ racks (production and DR,
would have probably had to run dev/test on
DR)
• Presumably Oracle licences would be
bought at same time so match
One decision to make
1 Purchase Order
Choice cannot be ‘unbalanced’
36 (1.0h)

37. Exalogic vs Build-Your-Own
©2010 Markus Eisele
37 (1.0h)

38. Exalogic
• Comes assembled & cabled up from Oracle
• All decisions have been taken for you – you can only
choose the size
• Oracle love you & you’ll probably get a better deal on your
licences
• Oracle remote diagnostics & single point of support
• You’ll get all the best Middleware features & super
performance, especially if you have Exadata too
• OVAB etc means it may be more “plug & go”
• Oracle “Red Stack Administrator” is King/Queen!
• But:
– May have to buy more hardware than you need up-front
– You’ll need at least 2 & possibly 3 – all are same (top) spec
– Your admins may not be used to Sun hardware, LOM etc
– (Vendor lock-in)
38 (1.0h)

39. Build Your Own (Blade)
• Flexibility to run other stuff on same infrastructure
• Very neat cabling
• Can buy the fastest processors and migrate slower ones to
test envs
• Great automation options
• Hardware will be consistent with other non-Oracle stuff
• Fits in with virtualised compute layer, storage layer,
network layer => very clean architecture
• But:
– You’ll probably need to hire a consultant to set up to get this
degree of automation (costly)
– You may have internal bickering over storage, network, boot from
SAN, spec of servers etc
– Oracle hold back best features for Exalogic
– With blades you will still have vendor lock-in
39 (1.0h)

40. Agenda
1. My Ideal Middleware Machine
2. Hardware
3. Software & Configuration
4. Case Study
5. Non-functional Factors
6. Summary

41. • Exalogic is bigger than many organisations need
• Reasons to buy Exalogic may be as much political
as technical
• Modern alternative ways to build a “middleware
machine” include stateless compute nodes booting
directly from centralised storage
• If building your own - careful configuration
required with scripted management – don’t
reinvent the wheel. Standardise!
41 (1.0h)

42. Thanks for listening!
Twitter: @simon_haslam
Oracle Fusion Middleware Admin blog:
http://simonhaslam.co.uk
42 (1.0h)