Sun sparc enterprise t5140 and t5240 servers technical presentation

Sun SPARC Enterprise
T5140 and T5240
Rack Servers
Technical Training Course
<Name>
<Job Title>

Sun Microsystems, Inc.

1

UltraSPARC T2: • Up to 8 cores @1.2GHz or 1.4GHz
TrueSystem On a Chip • Up to 64 threads per CPU
• Up to 16 FB-DIMMs, 4 memory
controllers
FB DIMM FB DIMM FB DIMM FB DIMM
> Up to 64GB memory (4GB DIMMs)
> 2.5x memory BW of UltraSPARC T1
FB DIMM FB DIMM FB DIMM FB DIMM • 8 x fully pipelined Floating Point units /
core, 1 per core
MCU MCU MCU MCU
• Dual 10Gbit Ethernet and PCI-E
integrated onto chip
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$
• 4MB L2 (8 banks) 16 way associative
Full Cross Bar • Enhanced MAU/Security co-processor
C0 C1 C2 C3 C4 C5 C6 C7 per core
FPU FPU FPU FPU FPU FPU FPU FPU
MAU MAU MAU MAU MAU MAU MAU MAU
> DES, 3DES, AES, RC4, SHA1,
DMA Sub-System SHA256, MD5, RSA to 2048 key,
ECC,CRC32
NIU Sys I/F
• Advanced Power saving features
(E-net+) Buffer Switch Core PCI-E
• 65nm process technology
Power ~95W
<1.5W/thread x8 @2.5GHz
2x 10GE Ethernet 4GBytes/s bi-directional
Sun Internal and Approved Partners Only 2

UltraSPARC T2 Processor
SPC SPARCv9 CMT Core
L2 Data L2 Data

Bank 0 Bank 4 CCX Core-Cache Crossbar
L2B0 SPARC SPARC SPARC SPARC L2B4 L2$ Level 2 Cache Bank
L2 Data
Core 0 Core 1 Core 5 Core 4
L2 Data CLC Coherence & Link Controller
Bank 1 Bank 5 MCU Memory Control Unit
L2B1 L2B5 SIU System Interface Unit
L2 L2 L2 L2
MCU0 MCU2 NCU Non-cacheable Unit
NCU

TAG0 TAG1 TAG5 TAG4 FSR
MAC Media Access Controller

EFU
MCU1 MCU3

DMU Data Management Unit

SIO
SII

L2B3 CCX L2B7
FSR CCU PEU PCI Express Unit
L2 Data L2 Data
Bank 3 Bank 7
L2 L2 L2 L2
L2B2 L2B6
TAG2 TAG3 TAG7 TAG6
L2 Data L2 Data

Bank 2 Bank 6

SPARC SPARC SPARC SPARC
DMU
Core 2 Core 3 Core 7 Core 6 RDP TDS

PEU

RTX
PSR ESR FSR MAC

For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4

UltraSPARC T2 Plus: • Per Socket
Multi-Socket US T2 > Up to 8 cores @1.2GHz or
Each Coherency Link 6.4GBytes/s per direction, 12.8GB/s total
1.4GHz
4 x Coherency links give Snoop bandwidth of 51.2GB/s > Up to 64 Threads per socket
4x
Coherency > 4MB L2$ 8 banks x 16 Way SA
Links
> Up to 8 FPU's, 1 per core
FB-DIMM FB-DIMM FB-DIMM FB-DIMM
> Up to 8 Crypto cores, 1 per core
FB-DIMM FB-DIMM FB-DIMM FB-DIMM > DES, 3DES, AES, RC4, SHA1, SHA256,
MD5, RSA to 2048 key, ECC,CRC32

MCU MCU
> 2 memory Controllers, 16 DIMMS
CU CU CU CU > 25Read+13WriteGB/s Memory BW
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ > One x8 PCIe interface
Full Cross Bar • Per system
CO C1 C2 C3 C4 C5 C6 C7 > 2 sockets glue-less
FPU FPU FPU FPU FPU FPU FPU FPU > 4 sockets using UST2 Plus XBR
MAU MAU MAU MAU MAU MAU MAU MAU
> 16 or 32 PCIe lanes on 2 or 4
System I/F PCI-Express Socket systems
• 65nm process technology
SSI x8 @2.5GHz
Bus Power ~105W 4GBytes/s
<1.5W/thread bi-directional


UltraSPARC T2 Plus Processor
SPC SPARCv9 CMT Core
CCX Core-Cache Crossbar
L2$ Level 2 Cache Bank
CLC Coherence & Link Controller
MCU Memory Control Unit
SIU System Interface Unit
NCU Non-cacheable Unit
NCX Non-cacheable Crossbar
DMU Data Management Unit
PEU PCI Express Unit


CMT Processor Comparison
UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus
Technology 90nm 65nm 65nm
# of Cores 8 8 8
# of Threads 32 64 64
# of Sockets 1 1 1–4
Core Frequency (Ghz) 1 – 1.4 1.2 – 1.4 1.2 – 1.4
Execution pipelines per core 1 2 2
Instructions per cycle per pipe 1 1 1
FPUs per chip 1 8 8
I$ size (per core) 16KB 16KB 16KB
D$ size (per core) 8KB 8KB 8KB
L2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-Way
Memory Type DDR2 FBDIMM-1 FBDIMM-1
Memory Bandwidth (GB/s) 21.3 42READ + 21WRITE 25READ + 13WRITE
Coherence Bandwdth (GB/s) - - 4 x (6.4READ +6.4WRITE)
IO Bus JBUS x8 PCIe x8 PCIe
IO Bandwidth (GB/s) 2.5 2+2 2+2
On-chip Networking - 2 x 10GE -
Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5
Power (Watts, Watts/thread) 72, 2.25 95, 1.5 105, 1.6 *


UltraSPARC T2 Plus 2-Socket System

Coherency Links
Dual Channel Dual Channel Dual Channel Dual Channel
FBDIMM FBDIMM FBDIMM FBDIMM

Memory Controller Memory Controller Memory Controller Memory Controller

Coherence Coherence Coherence Coherence Coherence Coherence Coherence Coherence
Unit Unit Unit Unit Unit Unit Unit Unit

UltraSPARC T2 Plus Cores, Crossbar, L2$ UltraSPARC T2 Plus Cores, Crossbar, L2$
(8 cores, 64 threads, 4MB L2$) (8 cores, 64 threads, 4MB L2$)

PCI-Express NCU, DMU NCX PCI-Express NCU, DMU NCX

System IO (Network, Disk, etc.)


UltraSPARC T2/Plus CPU Core Block Diagram
• IFU – Instruction Fetch Unit
> 16 KB I$, 32B lines, 8-way SA
TLU IFU > 64-entry fully-associative ITLB
• EXU0/1 – Integer Execution Units
> 4 threads share each unit
> Executes one integer
instruction/cycle
EXU0 EXU1
• LSU – Load/Store Unit
> 8KB D$, 16B lines, 4-way SA
MMU/ > 128-entry fully-associative DTLB
HWTW • FGU – Floating/Graphics Unit
• SPU – Stream Processing Unit
> Cryptographic acceleration
SPU FGU LSU
• TLU – Trap Logic Unit
> Updates machine state, handles
exceptions and interrupts
• MMU – Memory Management Unit
> Hardware tablewalk (HWTW)
Gasket – TLB miss handled by hardware
> 8KB, 64KB, 4MB, 256MB pages
xbar/L2

Floating Point Unit

• Each Core has its own Floating Point Unit. This removes
completely the 2% FP restriction of T1.
• Only a 6 cycle penalty to the FPU
> T1 required upwards of 28 cycles
• Fully-pipelined (except divide/sqrt)
Data
> Divide/sqrt in parallel with add or multiply operations of other
threads
• Full VIS 2.0 implementation in hardware, no emulation
• FPU performs integer multiply, divide, population count
• Multiplier incorporates enhancements for offloading modular
arithmetic operations to the Crypto Unit

●

● Sun Internal and Approved Partners Only 9

FB-DIMM Memory Interface

Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus
Require Pins 1000 440 208
Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socket
Up to 64x DIMM slots Up to 32x DIMM slots/socket
Memory Capacity up to 16x DIMM slots
T5x20 uses 16 DIMM slots T5x40 uses 8/16 DIMM slots
Data
Memory Controllers 4 4 2
Memory Latency 96ns 126ns 126ns
DIMM speeds 400MHz 667MHz 667MHz
DIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM

●


10/100/1000/10GbE Network Interfaces
• Uses the Neptune Chip on the Motherboard which provides
> Two RJ-45 ports dedicated for 10/100/1000
> Two RJ-45 ports capable of 10/100/1000
– Fitting a XAUI adapter disables the one of these ports per XAUI
• Same 10 GbE XAUI cards as T5x20 server
• 16 Rx and 24 Tx DMA channels for virtualization
– Integrated NIU in T2 has 16 Tx channels
Data
– Raw UDP Tx performance on Neptune > T2 NIU
– T2 processor network typically outperforms Neptune on most network
benchmarks
• Packet classification
> Rx data can be HW classified at layer 2 ,3 and 4
– Put into the separate Rx DMA buffers by classification rules
• Same 'nxge' device driver as T2 NIU.
• Performance estimates, testing/tuning ongoing
– 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle
●
– 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports)

T5140/T5240 Extreme Rackmount Density
Dual UltraSPARC T2 Plus • 1 / 2RU chassis, 28” depth
• Two sockets
> Up to 8 cores @ 1.2GHz at RR
– 1.4GHz to follow
> Up to 128 threads
• 161U/2U/ 322U FB-DIMM memory slots
> 64/128GB of memory, 4GB DIMMs
High Reliability
• Up to 128 threads / 128GB memory / 128 • Up to 4 (1U) or 8/16 (2U) Hot-Plug
LDoms in a 1U / 2U redundant chassis SATA/SAS 2.5” HDD (Raid 0,1)
> RAID 5 BBWC card post-RR
• Over 5,100 threads per rack !
• Redundant, hot-swappable PSUs and
• High RAS and I/O bandwidth Fans. DC Option for 2U
• Certain Configs are 'Hi-Line Power' only
• Scale-Up / Scale-Out of application
infrastructure Expandable
• 3/6 PCI-E expansion slots
> including 2 x 10 GBE I/0 XAUI slots
• 4 x 1GbE Ethernet as standard
• 4 USB ports

T5x40 Standard Configurations (PTO)
Part Number Description 1RU
2 x 4 Core, 64 Threads, 1.2Ghz, 8GB (8 X 1GB),
SETPAAE1Z
4 disk capable, 2x 146GB disks, 2x 720W PSU, DVD
SETPBGE1Z
SETPCJE1Z
SETPCPE1Z
Part Number Description 2RU
SEUPBAE1Z
SEUPCJE1Z
SEUPCPE1Z
SEUPCPE2Z 16 disk capable, 2x 146GB disks, 2x 1100W PSU, DVD
HIGHLINE POWER ONLY (220VAC)
SEUPDPE1Z 16 disk capable, 2x 146GB disks, 2x 1100W PSU, DVD
HIGHLINE POWER ONLY (220VAC)


Sun Fire T1000/T5120/T5140 Differences

Feature Sun Fire T1000 SPARC Enterprise T5120 SPARC Enterprise T5140
Form Factor 1U, 18¾” deep 1U, 28” deep 1U, 28” deep
UltraSPARC T1 UltraSPARC T2 2x UltraSPARC T2 Plus
CPU
1.0 GHz, 16-32 threads 1.2/1.4 GHz 32-64 threads 1.2/1.4 GHz 64-128 threads
DDR2 32GB Max FB-DIMM 64GB MAX FB-DIMM 64GB MAX
Memory
8x slots 16x slots 16x slots
4 x 1GbE or
Network 4 x GbE 4 x GbE + 2 x 10GbE 3 x 1Gbe + 1 x 10GbE or
2 x 1GbE + 2 x 10GbE
1x 3.5” SATA, non-hot-swap
Internal Storage Up to 8 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap
OR 2x 2.5” SAS
Removable Media None 1x DVD-RW 1x DVD-RW
Serial 1x RS-232, No USB 1x RS-232, 4x USB 1x RS-232, 4x USB
PCI Express slots 1x (low profile) 3x(low profile) 3x(low profile)
1x 300W AC 2x 650W AC & DC option 2x 720W AC
Power Supply
Non-Redundant Redundant/Hot-Swap Redundant/Hot-Swap
Fans Non-Hot-Swap 4 x Redundant/Hot-Swap 6 x Redundant/Hot-Swap


Sun Fire T2000/T5220/T5240 Differences

Feature Sun Fire T2000 SPARC Enterprise T5220 SPARC Enterprise T5240
Form Factor 2U, 24” deep 2U, 28” deep 2U, 28” deep
UltraSPARC T1 UltraSPARC T2 2x UltraSPARC T2 Plus
CPU
1.0 - 1.4GHz 32 threads 1.2/1.4 GHz 64 threads 1.2/1.4 GHz 64-128 threads
DDR2, 64GB MAX FB-DIMM, 64GB MAX FB-DIMM 128GB MAX
Memory
16x Slots 16x Slots 32x slots
4 x GbE or
Network 4x GbE 3 x GbE + 1 x 10GbE or
4x GbE + 2x 10GbE 2 x GbE + 2 x 10GbE
Internal Storage 4x 2.5” SAS, hot-swap 8 x 2.5” SAS, hot-swap Up to 16 x 2.5” SAS, hot-swap
Removable Media 1x DVD-ROM 1x DVD-RW 1x DVD-RW
Serial 1x RS-232, 4x USB 1x RS-232, 4x USB 1x RS-232, 4x USB
PCI Express slots 3x (low profile) 6x (low profile) 6x (low profile)
PCI-X slots 2x (low profile) None None
2x 450W AC 2x 750W AC 2x 1100W AC/2 x1200Watt DC
Power Supply
Redundant/Hot-Swap Redundant/Hot-Swap Redundant/Hot-Swap
Fans 3 x Redundant Hot-Swap 3 x Redundant Hot-Swap 5x Redundant/Hot-Swap


Front Panel
Dual USB 2.0 ports

Locator LED
/Button
DVD-RW Disk Drive Map
Fault LED Top Fault LED
Status LED
Power Button DISK 0 DISK 1 DISK 2 DISK 3 PSU Fault LED
Temp Fault LED

Serial Number label

Locator LED
/Button
Fault LED
Status LED Top Fault LED
Power Button PSU Fault LED
Temp Fault LED
DISK 1 DISK 3 DISK 5 DVD-ROM
Disk Drive Map
Serial Number DISK 0 DISK 2 DISK 4 DISK 6 DISK 7

Dual USB 2.0 ports
DISK 3 DISK 7 DISK 12 DISK 15
DISK 2 DISK 6 DISK 11 DISK 14
DISK 1 DISK 5 DISK 9 DVD-RW
DISK 0 DISK 4 DISK 8 DISK 10 DISK 13

Rear Panel

PCI/XAUI 0 PCI/XAUI 1 PCI 2
PSU PSU 0 PSU 1
Status
LED's

Chassis
Status LED's Quad Gigabit Dual POSIX DB-9
and Locator Button Ethernet Ports USB 2.0 Serial Port
SP SP Ports
Serial Network
Port Port

PSU 1
PCI 3 PCI 4 PCI 5
PCI/XAUI 0 PCI/XAUI 1 PCI 2
PSU 0

Chassis
PSU Status LED's Quad Gigabit POSIX DB-9
Status Dual
and Locator Button Ethernet Ports USB 2.0 Serial Port
LED's
SP SP Ports
Serial Network
Port Port


T5140 Block Diagram
CL
1RU=16 DIMMS

x8 SSI x8 SSI

x4
x4 PCIe x8 x8
PCIe
LSI x1
Switch Switch
Disk Chassis 1068E PLX 8548 PLX 8548 FPGA
1RU=4 Disks
x8 x8
PCIe x8
to x4 NEPTUNE
PCI
Ethernet
33MHz XAUI XAUI 2 x 10/100/1000
2 x 10/100/1000/10000
32bit

USB 2.0
to IDE PCI
to MPC885
DVD USB ILOM
USB 2.0 2.0 QUAD GbE Service
Hub
PHY
BCM5466R
Processor
USB
2.0

Front Panel 0 1 2 3
USB PCIe PCIe PCIe Quad GbE Serial Network POSIX
Rear Panel x16 x8 x8 Connectors Mgt Mgt Serial DB-9
Slot2 Slot1 Internal and Approved Partners Only
Sun Slot0 18

T5240 Block Diagram
2RU Memory Mezzanine Tray 2RU=16 DIMMS without Mezannine, 32 DIMMS with mezannine
Option
CL

x8 SSI x8 SSI

x4
x4 PCIe x8 x8
PCIe
LSI x1
Switch Switch
Disk Chassis x4 1068E PLX 8548 PLX 8548 FPGA
2RU = 8 or 16 Disks
x8 x8 x8
PCIe x8 x8 x8
to x4 NEPTUNE
PCI
Ethernet
33MHz XAUI XAUI 2 x 10/100/1000
2 x 10/100/1000/10000
32bit

USB 2.0
to IDE PCI
to MPC885
DVD USB ILOM
2RU 2RU 2RU
USB 2.0 2.0 QUAD GbE Service
Hub
PHY
BCM5466R
Processor
USB
2.0

Front Panel 0 1 2 3
USB PCIe PCIe PCIe PCIe PCIe PCI- Quad GbE Serial Network POSIX
Rear Panel x16 x8 x8 x8 x8 ex8 Connectors Mgt Mgt Serial DB-9
Slot2 Slot5 Slot1 Internal and Approved Partners Only
Sun Slot4 Slot0 Slot3 19

Internal View (T5140) Fault Remind System Configuration Service Processor
Button Card (SCC)EEPROM

FB0/FM0

FB1/FM0
FB0/FM0

UltraSPARC UltraSPARC
PCIe Slot 2

T2 Plus 0
T2 Plus 0
x16(m)/x8(e)
FB-DIMMs

FB0/FM1

FB1/FM1
FB0/FM1
PCIe Slot 1
x8(m)/x8(e)
or
XAUI 1

T2 Plus 1
T2 Plus 1
Disk
Chassis FB-DIMMs
FB1/FM2
FB0/FM2

PCIe Slot 0
x8(m)/x8(e)
or
XAUI 0

Neptune
PSU1 Network ASIC
Power Power
Board Supplies
PSU0

SAS x4 cable to Power BUS Bars Memory Mezannine Connectors
Disk Chassis 6 Fan Assemblies are required X4 SAS Connector 0 (cable attached)
RED = +12Volts (not used on T5140) X4 SAS Connector 1 (not used in 1RU)
(chassis can take up to 8) BLUE = Ground
Chassis Intrusion Reed Switch
(magnet on underside of lid)

Internal View (T5240 without Mezzanine) Fault Remind System Configuration Service Processor
Button Card (SCC)EEPROM

FB0/FM0

FB1/FM0
FB0/FM0
PCIe Slot 2

x8(m)/x8(e)

T2 Plus 0
T2 Plus 0
x16(m)/x8(e)
FB-DIMMs

PCIe Slot 1
x8(m)/x8(e)
x8(m)/x8(e)
FB0/FM1

FB1/FM1
FB0/FM1

FB1/FM1
or

T2 Plus 1
T2 Plus 1
XAUI 1

Disk
Chassis FB-DIMMs
PCIe Slot 0
x8(m)/x8(e)
x8(m)/x8(e)
or
XAUI 0
FB0/FM2
FB0/FM2

Power Power PSU's
Board Supplies Vertically
Stacked

Power BUS Bars Memory Mezannine Connectors X4 SAS Connector 0 (cable attached)
5 Fan Assemblies are required RED = +12Volts X4 SAS Connector 1 (not used in 1RU)

Internal View (T5240 with Mezzanine) Mezzanine Air Fault Remind System Configuration Service Processor
Guide Button Card (SCC)EEPROM

FB0/FM0

FB1/FM0
FB1/FM0
PCIe Slot 2
x8(m)/x8(e)
x16(m)/x8(e)
FB-DIMMs

PCIe Slot 1
x8(m)/x8(e)
x8(m)/x8(e)
FB0/FM1

FB1/FM1
FB0/FM1

FB1/FM1
or
XAUI 1

Disk
Chassis FB-DIMMs
PCIe Slot 0
x8(m)/x8(e)
x8(m)/x8(e)
or
XAUI 0
FB0/FM2
FB0/FM2

Power Power PSU's
Board Supplies Vertically
Stacked

Power BUS Bars Memory Mezannine Handles
5 Fan Assemblies are required RED = +12Volts

Motherboard Memory Physical Layout
Chassis Chassis
Front Rear
Channel0/DIMM 1/Branch 1
CMP0/Branch1/Channel1/DIMM0 1
CMP0/Branch1/Channel1/DIMM1
3 2

CMP0/Branch1/Channel0/DIMM0 1 SECOND
Install ALL DIMMs in

UltraSPARC
CMP0/Branch0/Channel0/DIMM0 1 Group 2

T2 Plus 0
CMP0/Branch0/Channel0/DIMM1 2 12 DIMM Config
FIRST
Group 1
8 DIMM Minimum
UltraSPARC

CMP1/Branch1/Channel1/DIMM0 1 Config
T2 Plus 1


Channel0/DIMM 1/Branch 1 1 THIRD
CMP1/Branch1/Channel0/DIMM0 Install ALL DIMMs in
CMP1/Branch0/Channel0/DIMM0 1 Group 3
CMP1/Branch0/Channel0/DIMM1 3 16 DIMM Config


At RR, Supported configs will be ... All DIMMs MUST be the same Size i.e 1GB, 2GB, 4GB, no mixing
Fully Populate Group 1 and 2 (8 DIMMS) All DIMMs in a Branch must be the same Sun Part Number
Fully Populate Groups 1, 2 and 3 (16 DIMMS) Unused DIMM locations must be occupied with DIMM Fillers

Mezzanine Memory Physical Layout
Chassis Chassis
Front Rear
CMP0/MR0/BR1/CH1/DIMM3 4
CMP0/MR0/BR1/CH1/DIMM2
3 4

CMP0/MR0/BR1/CH0/DIMM3 4 FOURTH
CMP0/MR0/BR0/CH0/DIMM3 4 Group 4
CMP0/MR0/BR0/CH0/DIMM2 4 24 DIMM Config



5 FIFTH
CMP1/MR1/BR1/CH0/DIMM3 Install ALL DIMMs in
CMP1/MR1/BR0/CH0/DIMM3 5 Group 5
CMP1/MR1/BR0/CH0/DIMM2 5 32 DIMM Config


At RR, Supported configs will be ... All DIMMs MUST be the same Size i.e 1GB, 2GB, 4GB, no mixing
Fully Populate Groups 1, 2, 3 , 4 and 5 (32 DIMMS) All DIMMs in a Branch must be the same Sun Part Number
Unused DIMM locations must be occupied with DIMM Fillers

Memory Configuration Rules
• Populate at least 4 DIMMS at a time
• Populate by complete group only (set 1, 2, 3, 4, 5)
> Groups may be 4 or 8 DIMMs, check service labels on lid
• Ensure proper installation of Airflow Guides
> All Configs require DIMM fillers for correct air-flow
> 1RU has no Airflow Guide
> 2RU has two configurations
– No Mezzanine
– Install Lower Air Guide
– With Mezzanine
– Install Lower PDB Guide and Mezzanine Guide

• All DIMMS must be the same size(i.e all 1GB, 2GB or 4GB, no mixing)
• In a Group all DIMMS must have the same SUN Part No.
• No clocking down of memory in any config.
• 8GB DIMMs will be qualified/evaluated after RR

Network Ports and Options

XAUI 0 XAUI 1

Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1
Driver nxge0 nxge1 nxge2 nxge3 nxge0* nxge1*
PCI Device PCI@300/0/8/0,0 PCI@300/0/8/0,1 PCI@300/0/8/0,2 PCI@300/0/8/0,3 PCI@300/0/8/0,0 PCI@300/0/8/0,1

NOTES
All Ethernet ports provided by a single Neptune Chip on the motherboard
NET 0 and NET 1 are disabled by XAUI cards if installed to take advantage of 10GbE networking
* If a XAUI card is installed, the matching RJ-45 port is disabled
XAUI cards take one of three current Fibre 10GbE transceivers
Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM)
Copper versions of the XAUI card are expected Post-RR
XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive
with PCI cards in the same location


10GbE XAUI Option Cards
• Interface to the Neptune chip to upgrade two of its ports to 10GbE
• Cards each consume a PCIe slot if installed
• Attach to dedicated XAUI sockets on special risers
• 10Gbe XAUI Card
– SESY7XA1Z
> Requires Fibre Tranceivers
– 10GbE Short Reach(SR)
– SESY7XT1Z
– 10GbE Long Reach (LR)
– SESY7XT2Z
– 10GbE Long Reach multimode (LRM)
– part No. TBD

• Copper connector XAUI versions to follow post-RR


PCI Riser Cards
• Various Riser Cards are used
• Risers should not be moved between slots
• All slots accommodate low Profile PCI-Express cards
• T5140 – 25 Watts per card, 75 Watts Max (3 Slots)
• T5240 – 25 Watts per card, 102 Watts Max (6 Slots)

> Riser Card Mechanical and Electrical widths for T5140 and T5240

T5140 Slot 0 Slot 1 Slot 2
Only Slot x8/XAUI(m) x8(e) x8/XAUI(m) x8(e) x16(m) x8(e)

T5240 Slot 1 Slot 1 Slot 2
Top Slot x8(m) x8(e) x8(m) x8(e) x8(m) x8(e)
Bottom Slot x8/XAUI(m) x8(e) x8/XAUI(m) x8(e) x16(m) x8(e)


Disk Cage
• Disk chassis is a FRU
– 4 screws and disconnect cables
• Contains DVD-RW/Dual USB module as a
sub-FRU/CRU
– Standard in all default configs
– remove disk 3/7 to access release latch, pre-RR
• Non-DVD module available as X-Option
– Still provides Dual USB ports Remove Disk and locate latch
beneath DVD
Press up and pull DVD out
• T5140 will be available with only a 4
disk chassis (8 disk may be post RR)
> Only uses one of the x4 SAS links
• T5240 will be available with 8 or 16 disk chassis
> 16 Disk systems requires High-Line Power (220VAC)


Disk Drives
• Disk drives are all 2.5” form factor
• Chassis support both SAS and SATA (post-RR)
• SAS Drives
– 73GB @ 10K RPM
– 146GB @ 10K RPM
– 73GB @ 15K RPM Improved
Airflow above
and below
– 146GB @ 15K RPM (post-RR) disks

• SATA Drives
– 300GB @ 5400 RPM (H2CY08)
– 500GB @ 7200 RPM (end CY08)
Disk LED's
• New mounting bracket “Marlin” Ready to
Remove
> Allows better cooling in denser chassis Fault
Status
• Same disks as T5x20 systems


Disk Controller Options
• T5x40 has embedded LSI1068E on the motherboard
– JBOD up to 8 disks
– Controls internal disks only
– RAID 0 (striping) and RAID 1(mirroring), using 'radictl'
– Hot-plug supported
– Hot-swap in non-RAID configs is done with 'cfgadm'
• Optional Extended RAID functionality cards
> SPARC Drivers are available, SAS cables are being produced now
> SGXPCIESAS-R-INT-Z
– 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available)
– Adaptec Card and Firmware with Intel SunRiseLake chipset
– Battery Backed Write Cache (BBWC) of 256MB
– BBWC battery at least 72 hours (much longer when new/fully charged)
– Only for internal disks, replaces cables to route to standard disk chassis
> SGXPCIESAS-R-EXT-Z
– External port version, for external disk arrays only (no internal ports)
> Not available for T5440 due to chassis design and only 4 disk capacity

Fans and Cooling
• T5140 Unused Fan slots can be populated, but not recommended, fans will
run and wear, so not a great way of keeping spares....
> Requires 6 Fan modules
– Main System Cooling

FBD1

FBD0
PSU's

FM3

FM3
FM3
– FANBD0/FM0

Disk Chassis
Disk Chassis
FBD1
– FANBD0/FM1

FBD0
FM2

FM2
FM2
– FANBD0/FM2
– FANBD1/FM0

FBD1

FBD0
Motherboard

FM1

FM1
FM1
– FANBD1/FM1
– FANBD1/FM2

FBD1

FBD0
FM0

FM0
FM0
– No Modules in FM3
• T5240 Back Front
> Requires 5 Fan Modules
– Main system Cooling PSU's

FBD1

FBD0
FM2

FM2
FM2

FM2
– FANBD0/FM0

Disk Chassis
Disk Chassis
– FANBD0/FM1

FBD1

FBD0
FM1

FM1
FM1

FM1
– FANBD0/FM2
Motherboard
– FANBD1/FM0
– FANBD1/FM1

FBD1

FBD0
FM0

FM0
FM0
– No Modules in FANBD1/FM2

Fans modules and boards
• Fans are speed controlled by the Service Processor
• Each fan has an Amber FAULT LED
– if lit the fan module is faulty
• 1RU modules
– 2 Fans at 15K RPM Max.
Fan Fault
• 2RU modules Amber LED

– 2 Fans at 12K RPM Max Lit if fan faulty

• Monitored through ILOM/IPMI/Web

Vibration
Absorbing
Silicon
Dampers


Power Supplies
• PSU's are high efficiency
• Output 12V (and 3.3v standby) PSU LED's
• Most DC-DC conversion done in MBD Ready to
Remove
Fault
• Shared with Doradi/o, Tucani/a AC
Present

• Integrated Fan
• T5140
> Side-by-Side
> AC 720Watts
• T5240
> Vertically Stacked
> 1100Watts
> DC 1200 Watts option coming post-RR
> DC or AC, no sharing
> BEWARE...
– Same form factor as SunFire X4200 PSU's ... DO NOT MIX
– 1.4Ghz and 16 Disk configs require 220VAC (called “High Line Power”)

Rack Mounting Options
• Two Rack mounting options are available..
• Standard Rack Mount (4 post)SESY9RK1Z
> Standard configs contain this kit
– ATO : SESY9RK1Z, X-ATO : SESY9RK1Z
• Tool-less Rack Kit (4 post)
> Optional kit, purchased seperately
> Snaps into most 3rd party Racks
> Requires no tools to fit or remove
– ATO : SESY9RK2Z, X-ATO : SESX9RK2Z


Solaris for Sun SPARC Enterprise T5x40
• T5x40 servers will require Solaris10 8/07 (update 4) with
patches
> A Patch CD is supplied in the box
– Mandatory patches = 127111-08,126434-05,125416-05
– Available patches = 124235-02, 125369-13,125476-02,127111-07
– Likely to be out of date, so customers should check SunSolve...
• This incorporates several specific UltraSPARC T2/T2 Plus
optimizations and for CMT servers in general
• Also includes some specific T5x40 optimisations
> UltraSPARC T2 Plus MPO optimisations
> nxge driver tweaks for 10GbE devices
> MMU DMA relaxed scheduling
> LDOMs pre-install upgrade to version 1.0.2 (US T2 Plus support)

Solaris FMA
• Now aware of CPU Core offlining
• PCIe Fabric diagnostics improved
• Crypto Unit offlining
• Neptune awareness (though XAUI's not diagnosed)
• POST memory CE tolerance improved
• FRU identity info improved in fault reports (part/ser #)
• Diagnostic Engines (DE)
– Existing DE have been extended for T5x40
– New Eversholt DE used for HostBridge, PCIe diagnostics
• Limitations...
– Can't isolate Uncorrectable Errors to a single DIMM
– PCIe riser cards not in topology
– XAUI card and XFP not in the topology
– SouthBridge chip is not diagnosed at this time (DVD, USB,...)
– IO Expansion box is not diagnosed via FMA

ILOM Features at a Glance
• Hardware Control, Inventory, Monitoring
• Host Configuration and Administration
• SP Configuration and Administration
• Logging, Auditing and Alerting
• Authentication :
– LDAP, RADIUS, Active Directory, Local Users
• Services :
– SSH, HTTP(S), NTP, Syslog, SMTP
• Out-of-Band
– Via Serial port, or dedicated Ethernet port
• In-Band
– Via Host OS, firmware upload, Explorer, Syslog


ILOM – Industry Standard Manageability
• ILOM enables Sun systems to be easily integrated into third
party management infrastructures
• Key functionality
> IPMI 2.0
> SNMP V1, V2c, V3
> SSH 2.0
> LDAP, RADIUS & MS Active Directory authentication
> Browser UI, SMASH DMTF CLP + ALOM style CLI
• NOTE : No RKVMS functionality like Galaxy ILOM
> No Java plugin required
• Browsers tested include, these base versions an later
releases
– Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox
1.0 and 1.5 on W2K server

ILOM – Industry Standard Manageability

• Identical to ILOM 2.0 for T5x20
• Default user is 'root' with password 'changeme'
> Use for ILOM WEB GUI access and SSH access and IPMI
• The ALOM 'admin' user is not defined by default, create this
user with an ALOM CLI
• Lost 'root' password and ILOM config is reset using jumpers
> T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2)
• Note... 'sunservice' user shares the password of the 'root'
user ONLY FOR SERVICE ACCESS*
• Passwords must be 8 to 16 chars (used to be max 8)
• Max of 10 users allowed (used to be 16)


ILOM Common Tasks
• Setting up an ALOM 'admin' user from the ILOM CLI
> SUNSP00144F4641BD login: root
Password: changeme
Waiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Warning: password is set to factory default.
->create /SP/users/admin role=Administrator cli_mode=alom
Creating user...
Enter new password: ********
Enter new password again: ********
Created /SP/users/admin2
-> exit
SUNSP00144F4641BD login: admin
Password: ********
Daemons ready
Version 2.0.4.0
sc>


ILOM Common Tasks
• Setting up ILOM IP configuration from the ILOM CLI
> SUNSP00144F4641BD login: root
Password: changeme
Daemons ready
Version 2.0.4.0
Warning: password is set to factory default.
-> set /SP/network state=enabled
Set ’state’ to ’enabled’
-> set /SP/network pendingipaddress=10.8.31.163
Set ’pendingipaddress’ to ’10.8.31.163’
-> set /SP/network pendingipdiscovery=static
Set ’pendingipdiscovery’ to ’static’
-> set /SP/network pendingipnetmask=255.255.252.0
Set ’pendingipnetmask’ to ’255.255.252.0’
-> set /SP/network pendingipgateway=10.8.31.248 Static IP
Set ’pendingipgateway’ to ’10.8.31.248’ configuration
-> set /SP/network commitpending=true
Set ’commitpending’ to ’true’
->

-> set/SP/pendingipdiscovery=dhcp
Set ’pendingipdiscovery’ to ’dhcp’
-> set /SP/network commitpending=true
Set ’commitpending’ to ’true’
->
Dynamic IP
(DHCP)
configuration


ILOM IPMI functionality
• IPMI is a widely used management protocol for x86 servers
• ... But its new for SPARC boxes...
• Download the required tools from
> The Solaris Supplemental CD (an older binary package)
> Latest is available at http://ipmitool.sourceforge.net
– Binary packages (usually older versions) are available for Solaris x86 and SPARC,
as well as Linux, Windows, Mac...etc
> There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC
now too)
– http://www.sun.com/blueprints/0107/820-1011.pdf

-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr
/MB/V_+1V0_VDD | 1.01 Volts | ok
/MB/V_+3V3_STBY | 3.36 Volts | ok Get the Sensor
/MB/V_+3V3_MAIN | 3.36 Volts | ok Data
/MB/V_+5V0_VCC | 4.97 Volts | ok Repository
/MB/V_+12V0_MAIN | 12.16 Volts | ok
..
..
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set
/SYS/LOCATE on
/SYS/LOCATE | ON Set the System
-bash-ventus2$ Locator LED
flashing


ILOM Common Tasks
• Logging into the ILOM Web GUI
• Point Web Browser at the IP address of
the ILOM
> Will automatically redirect to a
secure HTTPS port
> Best to disable any Web Proxies for
best performance
> Make sure port 80, 443 are open on
any firewall
• Log in as any defined user, default
Administration user is 'root'


ILOM/ALOM Common Tasks
• A Tour of the ILOM Web GUI • GUI provides ways to set up IP address
configurations for ILOM
• Configure additional ILOM/ALOM users
• Access Sensor Readings
• Control System configurations
• Carry out firmware updates.
• etc....


ALOM 2.0 vs. 1.4 CLI Differences
• Default CLI appearance is ILOM
– User is 'root'
– Must create an 'admin' Administrator user with the ALOM CLI
• Flashupdate in new ALOM expects a TFTP server not an FTP server
• ALOM IP address changes now active only after running ...
– 'setsc netsc_commit true '
– Previously changes were active only after SC reset
– DHCP is enabled by default
• View firmware versions with
– 'showhost version ' sc> showhost version
Sun System Firmware 7.0.0_build_13 2007/08/10 23:44

• Only Admin and Operator roles Host flash versions:
Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54
OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25

are defined Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD***
2007/08/10 13:51


ALOM backward Compatibilty CLI
• Uses ALOM-CMT 1.2 for a base feature set
• Includes ALOM-CMT 1.4 features...
> break -d
> Host Watchdog Timer
> Bootmode LDOMs awareness
> Forward SP events to Hot/Solaris syslog via inband channel


Logical Domains 1.0.2 Features
Release in Q1CY08 aligned with UltraSPARC T2 Plus launch
• UltraSPARC T2 Plus Platform Support
> 128 domains supported at RR
– 256 domains work on T5440, may be qualified later, requires Solaris fixes
• Feature Set very similar to 1.0.1
• I/O domain can reboot while the other domains keep running
> Dependant domains would wait for IO transactions to complete
• LDoms SNMP MIB
> Modeled after industry standard DMTF CIM (Common Information Model)
> Running on control domain to enable remote SNMP monitoring
• Domain minimization support
> Minimal/reduced Solaris installation
> Base minimal packages plus optional ones for guest, control, I/O or service
domains

Logical Domains 1.0.2 Requirements
• Solaris 10 08/07 (update 4) and required patches
• Systems firmware version
– T5x40 Firmware 7.1.0 = SP FW 2.0.4.20+
– T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+
– T2000/T1000/T6300/Pelton Firmware 6.5.x
• SUNWldm.v LDOMs Manager Package
– Version 1.0.2


CRU/FRU List - T5140
• Lots shared with other Jake and Elwood systems
Item Description FRU/CRU
Disk Backplane 4 drive Disk Backplane to support 4 Drives only FRU
Disk Backplane 8 drive Disk Backplane to support 8 Drives (post -RR) FRU
Fan Module 1U Dual Fan module, 4 populated in system CRU
Fan Board 1U Fan connector board to accommodate Fan Modules, 2 per system FRU
DVD Assembly DVD Assembly including front USB board FRU
Paddle board 1U Conection between PDB, fan boards and Disk Backplane FRU
PDB 1U Power distribution board, contains master chassis serial number FRU
PSU 1U Power supply (720 Watts) CRU
X8 &XAUI, 1U PCIe Riser PCI Riser assembly for left and centre motherboard slots CRU
x16 1U PCIe Riser PCI Riser for right motherboard slot CRU
Rack mount kit (requires tools) Rack mount kit CRU
Rack Mount Kit (tooless) Rack mount kit, no tools required for installation CRU
CMA Cable management arm CRU
Bus Bars Power Bus Bars between PDB and Motherboard FRU
PDB to MB Ribbon Cable Ribbon cable for control connections between PDB abd Motherboard FRU
Light pipe kit Light pipe assemblies for left and right of disk cage FRU
Motherboard assembly Motherboard assembly, part no varies depending on Core count FRU
DIMMs FBDIMMs (three different part numbers for 1GB, 2GB, 4GB) CRU
XAUI board optical/copper XAUI Adapter card (two different parts for copper/optical) CRU
XFP module XFP GBIC module for use in XAUI cards (three different parts SR, HR, HRMM) CRU
Battery Battery backup for NVRAM of Service Processor CRU
HDD 73GB/146GB SAS Hard Disk drive, different part numbers for capacities CRU
System Configurations PROM System Configuration PROM FRU
Maramba Cable Kit SAS Cable kit FRU


CRU/FRU List - T5240
• Lots shared with other Jake and Elwood systems
Item Description FRU/CRU
Fan Module 2U Dual Fan module, 3 populated in system CRU
Fan Board 2U Fan connector board to accommodate Fan Modules, 2 per system FRU
DVD Assembly DVD Assembly including front USB board FRU
Paddle board 2U Connection between PDB, fan boards and Disk Backplane FRU
PDB 2U Power distribution board, contains master chassis serial number FRU
PSU 2U Power supply (1100 Watts) CRU
X8,X8, XAUI, 2U PCIe Riser PCI Riser assembly for left and centre motherboard slots CRU
X8, x16 2U PCIe Riser PCI Riser for right motherboard slot CRU
Rack mount kit (requires tools) Rack mount kit CRU
Rack Mount Kit (tooless) Rack mount kit, no tools required for installation CRU
CMA Cable management arm CRU
Bus Bars Power Bus Bars between PDB and Motherboard FRU
PDB to MB Ribbon Cable Ribbon cable for control connections between PDB abd Motherboard FRU
Light pipe kit Light pipe assemblies for left and right of disk cage FRU
Motherboard assembly Motherboard assembly, part no varies depending on Core count FRU
DIMMs FBDIMMs (three different part numbers for 1GB, 2GB, 4GB) CRU
XAUI board optical/copper XAUI Adapter card (two different parts for copper/optical) CRU
XFP module XFP GBIC module for use in XAUI cards (three different parts SR, HR, HRMM) CRU
Battery Battery backup for NVRAM of Service Processor CRU
HDD 73GB/146GB SAS Hard Disk drive, different part numbers for capacities CRU
System Configurations PROM System Configuration PROM FRU
Maramba Cable Kit SAS Cable kit FRU


Option Parts List for T5140/T5240
Part Number Option Description Available
SECY9PS41Z AC PSU 1RU (720 Watts) RR
SEDY9PS51Z AC PSU 2RU (1100 Watts) RR
- DC PSU 2RU (1200Watts) post-RR
SESY2A1Z 2GB Memory (2x1GB) RR
SESY2B1Z 4GB memory (2x2GB) RR
SESY2C1Z 8GB Memory (2x4GB) RR
SESY9FF1Z FB-DIMM Filler panel RR
SEUY2MM1Z Memory Mezzanine Assembly 2U only RR
SEU9AD1Z Memory Mezzanine Air Duct RR
SETY9BA1Z 4 Disk Backplane for 1U RR
SETY9BB1Z 8 Disk Backplane for 1U post-RR
SEUY9BB1Z 8 Disk Backplane for 2U RR
SEUY9BC1Z 16 Disk Backplane for 2U (High Line only (220VAC) RR
SESY3A11Z 73GB, 10KRpm 2.5” SAS Drive RR
SESY3C11Z 146GB, 10KRpm 2.5” SAS Drive RR
SESX3A21Z 73GB, 15KRpm, 2.5” SAS Drive RR
- SATA Drives (300,500GB) post-RR
SESY9MF1Z HDD Filler Panel RR
SESY9DV1Z DVD, 8X RW, Slot Load RR
SESY9RK1Z Slide Rail Kit, requires screwdriver (1RU and 2RU) RR
SESY9RK2Z Slide Rail Kit, Tool-less (1RU and 2RU) RR


Option Parts List for T5140/T5240 (cont..)
Part Number Option Description Avialable Max no.(1RU/2RU)
SG-PCIE1FC-QF4 Qlogic, 4Gb, PCI-E Single Port RR 3/6
SG-PCIE2FC-QF4 Qlogic, 4Gb, PCI-E Dual Port RR 3/6
SG-PCIE1FC-EM4 Emulex, 4Gb, PCI-E Single Port RR 3/6
SG-PCIE2FC-EM4 Emulex, 4Gb, PCI-E Dual Port RR 3/6
SG-PCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 3/6
SG-PCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 3/6
SGCPCIESAS-R-INT-Z SAS, 8 Port, RAID, PCI-E, External Drive Controller RR 1/1
SGCPCIESAS-R-EXT-Z SAS, 8 Port, RAID, PCI-E, External Drive Controller RR 1/1
SECX9SA1Z Cable X-option for T5140 RR 1/1
SECX9SA2Z Cable X-option for T5240 RR 1/1
7281A-2 Northstar Dual port GigE MMF Low profile PCI-E RR 3/6
7280A-2 Northstar Dual port GigE UTP Low profile PCI-E RR 3/6
1236A-Z Infiniband HBA, Dual 4x ports, PCI-E RR 3/6
4447A-X Atlas Quad Port, GigE Copper, PCI-E (up to 3/6) RR 3/3
X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 3/3
X5558A Tranceiver 10GE SR XFP (XAUI/Atlas) RR -
X5560A-Z Tranceiver 10GE LR XFP (XAUI/Atlas) RR -
X3000A XVR-300, PCI-E x8 Graphics cards TBD 1/1
X4240A XVR-300, PCI-E x16 Graphics cards, requires 605-4668-04 cable TBD 1/1
SESY7XA1Z 10GbE XAUI card RR 2/2
TBD 10GbE XAUI Copper 10g Base-T TBD 2/2
X6000A Sun Crypto Accelerator 6000 RR 1/ 2


Sun sparc enterprise t5140 and t5240 servers technical presentation

Sun sparc enterprise t5140 and t5240 servers technical presentation

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (6)

Similaire à Sun sparc enterprise t5140 and t5240 servers technical presentation

Similaire à Sun sparc enterprise t5140 and t5240 servers technical presentation (20)

Plus de xKinAnx

Plus de xKinAnx (20)

Dernier

Dernier (20)

Sun sparc enterprise t5140 and t5240 servers technical presentation