SlideShare une entreprise Scribd logo
1  sur  36
High Availability != High-cost
   Building low-cost Master / Slave Clusters on a
Linux based Operating Systems to provide mission
                  critical services
                   Norman Maurer

                 ApacheCon EU 2009
                   March 27, 2009
Norman Maurer
     Member of the Apache Software Foundation
 

     Apache Software Foundation Infrastructure Team
 

     PMC Apache JAMES
 

     Senior Unix System Engineer
 




                                   norman@apache.org
                                   norman.maurer@heagmedianet.de
                                   http://www.heagmedianet.com
                                   http://myblog.kicks-ass.org


                                                                   2
The question should be:
“Why should you not use Linux-HA?“.. ENOCLUE!

          It's flexible
      



          It's opensource
      



          It has a very active community
      



          It's known to work for high-volume
      

          servers

          It's just freakin' cool!
      




                                                3
Components for building the
        Cluster



                              4
The Linux-HA Cluster uses two base components
that make up the Cluster..

      Heartbeat*
  



      DRBD**
  




                          * - http://www.linux-ha.org
                          **- http://www.drbd.org
                                                        5
Putting stuff together for an
  Apache HTTPD Cluster
         The magic glue




                                6
Brief overview of how the whole Cluster works

               Webserver01                    Webserver02
  Heartbeat        NIC            HA              NIC       Heartbeat

   DRBD                           Data                       DRBD
                   NIC                            NIC

  Services

  Virtual-IP



                         Master           Slave



                                                        Failover!!!
                                                                        7
                                  Users
DRBD a.k.a network raid1
  Mirror your data across the network




                                        8
Benefits when using DRBD

      Synchronous and asynchronous replication
  



      Works at the block level
  



      Many Distributions ship with pre-build DRBD
  

      modules / utils

      Works with every filesystem on-top while in
  

      Master / Slave-Cluster

      Master / Slave Cluster and Master / Master-
  

      Cluster possible
                                                    9
Limitations when using DRBD

      I/O throughput heavily depends on network
  

      throughput

      Only Master / Slave Cluster supported when
  

      using traditional filesystems (ext3, xfs, reiserfs)

      Master / Master Cluster only works with shared
  

      cluster file system (GFS, OCFS2)

      Data access is only possible from the Master
  

      Node in a Master / Slave Cluster

                                                            10
Brief overview on how DRBD works

             Webserver01              Webserver02
Filesystem                                            Filesystem


DRBD-Layer                                           DRBD-Layer
               NIC                        NIC


 Device /                                              Device /
  Disk                                                  Disk



                           Ethernet

                                            Sync data across network


                                                                       11
Installation

       Ubuntu
   

       # apt-get install drbd8-utils drbd8-module-source build-
       essential module-assistant && module-assistant auto-
       install drbd8

       Debian
   

       # apt-get install drbd8-utils drbd8-modules

       SLES
   

       # yast -i drbd

       Others
   

       Check if there are prebuild packages / build it yourself
       from source!
                                                                  12
Configuration of DRBD-Nodes
      Set correct config parameters on both Nodes
  

      /etc/drbd.conf:
                                                  Synchron write-operations
      global {
               usage-count yes;
      }
      common {
               protocol C;
                                                  Specify resource
      }
      resource r0 {
                                                  You can have more then one!
               on node01 {
                        device /dev/drbd0;
                        disk   /dev/sda7;
                        address 10.0.0.2:7789;
                                                   Configuration of DRBD-
                        meta-disk internal;
               }
                                                   Nodes
               on node02 {
                        device /dev/drbd0;
                        disk   /dev/sda7;
                         address 10.0.0.3:7789;
                        meta-disk internal;
               }
      }
                                                                                13
Create the device and up it

       Create device metadata
   

       # drbdadm create-md r0

       Attach to backing device,set synchronization
   

       parameters and connect to peer
       # drbdadm up r0

       Check state
   

       # cat /proc/drbd
       version: 8.0.11 (api:86/proto:86)
       GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by
            phil@mescal, 2008-02-12 11:56:43
       1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C
       r---
            …
            …
                                                                               14
Initial device synchronization

       Set Primary
   

       # drbdadm -- --overwrite-data-of-peer primary r0

       Check state again
   

       # cat /proc/drbd
       version: 8.0.11 (api:86/proto:86)
       GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by
       phil@mescal, 2008-02-12 11:56:43
       0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
       ….
       ….




                                                                          15
File system layout and data migration

       Put filesystem on-top
   

       # mkfs.ext3 /dev/drbd0

       Temporarily mount the filesystem to generate
   

       folder structure
       # mount /dev/drbd0 /drbd

       Create required folders and copy content
   

       # mkdir -p /drbd/etc /drbd/usr/lib /drbd/var
       # rsync -az /etc/apache2 /drbd/etc/
       # rsync -az /var/www /drbd/var/
       # rsync -az /usr/lib/cgi-bin /drbd/usr/lib/


                                                      16
Heartbeat
Failover on error / panic




                            17
Main features of Heartbeat

       Heartbeat packets via ICMP / UDP / Serial
   



       Works with “every“ unix daemon
   



       Be able to execute scripts on failover
   



       Automatic failback
   



       Support of “STONITH*“ devices
   



                                 *shoot the other node in the head


                                                                     18
Brief overview on how Heartbeat works

                                 Heartbeats
               Webserver01                       Webserver02

 Heartbeat                                                       Heartbeat
                 Checks                               Checks

                                  TCP/IP
                       NIC                            NIC
  Scripts                                                          Scripts
                       COM         Serial         COM

Start   Stop                                                    Start   Stop




                        Master                Slave
Called when becoming                             Called when becoming
Master-Node                                      Slave-Node


                                                                               19
Installation of Heartbeat

       Ubuntu / Debian
   

       # apt-get install heartbeat


       SLES
   

       # yast -i heartbeat

       Other Distributions
   

       Check for prebuild packages / build it yourself from
       source !




                                                              20
Installation of extra-scripts

       Ubuntu / Debian
   

       # apt-get install drbdlinks


       Other Distributions
   

       Check for prebuild packages / build it your from source!




                                                                  21
Configuration of extra-scripts

       Set correct config parameters for drbdlinks on
   

       both nodes
       /etc/drbdlinks.conf:

       link('/etc/apache2', '/drbd/etc/apache2')
       link('/var/www', '/drbd/var/www')
       link('/usr/lib/cgi-bin', '/drbd/usr/lib/cgi-bin')




                                                           Specify all directories which
                                                           needs to be “linked“ from the
                                                           mounted DRBD device to the
                                                           correct location.

                                                                                           22
Configuration of Heartbeat – Part I

       Set correct config parameters on both nodes
   


       /etc/heartbeat/ha.cf:

       debugfile         /var/log/ha-debug
       logfile           /var/log/ha-log
                                             Don't fallback when the
       Logfacility       local0
       keepalive         2
                                             primary node comes up
       deadtime          60
                                             again
       warntime          30
       initdead          120
       udpport           697
       auto_failback     off
       node              node01
       node              node02
       debug             0
                                             Nodes need to be
                                             resolvable


                                                                       23
Configuration of Heartbeat – Part II

       Configure script execution on failover
   

        /etc/heartbeat/haresources:

        node01 xxx.xxx.xxx.xxx drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 drbdlinks apache2




       Primary                        DRBD             Mount FS                Link Start
                       Virtual
                                      →
       Node                                                                    dirs Apache2
                       IP
                                      Primary

       Scripts needs to be located under:
   

         /etc/init.d/

         /etc/heartbeat/resources.d/
                                                                                                    24
Configuration of Heartbeat – Part III

       Setup encryption and passphrase
   


           /etc/heartbeat/authkeys:

                                               File needs to have
           auth 1
           1 sha1 your_super_secure_password
                                               perms 600!



           Supported encryption
       

             md5

             sha1
                                               Not really an
             crc
                                               encryption. Just
                                               packet corruption
                                               prevention
                                                                    25
Final Steps..

       Disable startup scripts for daemons started by
   

       Heartbeat!

       Reboot both servers to see if all services will
   

       be started on the next reboot

       Test failover to see if all works as aspected
   




                                                         26
Clustering in production
     What is known to work




                             27
What services are known to work ?
       Mail
   

       Ubuntu,Qmail,Vpopmail

       Webserver
   

       Ubuntu, Apache HTTPD 1.3.x / 2.0.x/ 2.2.x

       Database
   

       Ubuntu, MySQL 4.1 / 5.0 / 5.1

       Radius
   

       Ubuntu, Freeradius
       ......
                 ..... and thousands of happy
                 customers                         28
Best Practices – Part 1
       Set the heartbeat and DRBD utils/modules to
   

       “hold“ in package-management

       Disable Heartbeat on the Slave-Node while
   

       performing upgrades

       Use dedicated NIC
   



       Use decent NIC (like intel)
   



       Use gigabit ethernet
   




                                                     29
Best Practices – Part 2
       Use different UDP Ports (for heartbeats) per
   

       Linux-HA Cluster

       Test failover before putting the Cluster in
   

       production

       Keep the Heartbeat and DRBD config files in-
   

       sync

       Use „just“ an crossover-cable to connect the
   

       nodes if possible



                                                      30
Known problems
What problems often show up and how to
               fix them




                                         31
DRBD – Part I
      Split brain scenario
  



      Crappy throughput
  



      Version mismatch
  



      Kernel panic on both nodes
  



      UID/GID not match on both Nodes
  




                                        32
Heartbeat
      Split brain scenario
  



      Heartbeats get dropped
  



      Nodes not „see“ each other
  




                                   33
Useful resources
       Linux-HA
   
       http://www.linux-ha.org

       DRBD
   
       http://www.drbd.org


       DRBDLinks
   

       http://www.tummy.com/Community/software/drbdlinks/


       DRBD and MySQL
   

       http://dev.mysql.com/doc/refman/5.1/en/faqs-mysql-drbd-heartbeat.html




                                                                               34
Questions ?




              35
Thank you for your attention!




                                36

Contenu connexe

Tendances

Disaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBDDisaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBDViswesuwara Nathan
 
Disruptive IP Networking with Intel DPDK on Linux
Disruptive IP Networking with Intel DPDK on LinuxDisruptive IP Networking with Intel DPDK on Linux
Disruptive IP Networking with Intel DPDK on LinuxNaoto MATSUMOTO
 
イマドキなNetwork/IO
イマドキなNetwork/IOイマドキなNetwork/IO
イマドキなNetwork/IOTakuya ASADA
 
CASPUR Staging System II
CASPUR Staging System IICASPUR Staging System II
CASPUR Staging System IIAndrea PETRUCCI
 
Rh202 q&a-demo-cert magic
Rh202 q&a-demo-cert magicRh202 q&a-demo-cert magic
Rh202 q&a-demo-cert magicEllina Beckman
 
DLM knowledge-sharing
DLM knowledge-sharingDLM knowledge-sharing
DLM knowledge-sharingEric Ren
 
Building a Two Node SLES 11 SP2 Linux Cluster with VMware
Building a Two Node SLES 11 SP2 Linux Cluster with VMwareBuilding a Two Node SLES 11 SP2 Linux Cluster with VMware
Building a Two Node SLES 11 SP2 Linux Cluster with VMwaregeekswing
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunheut2008
 
Building a Virtualized Continuum with Intel(r) Clear Containers
Building a Virtualized Continuum with Intel(r) Clear ContainersBuilding a Virtualized Continuum with Intel(r) Clear Containers
Building a Virtualized Continuum with Intel(r) Clear ContainersMichelle Holley
 
DPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabDPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabMichelle Holley
 
Easy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to AdvancedEasy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to AdvancedChenkai Sun
 
Gluster Cloud Night in Tokyo 2013 -- Tips for getting started
Gluster Cloud Night in Tokyo 2013 -- Tips for getting startedGluster Cloud Night in Tokyo 2013 -- Tips for getting started
Gluster Cloud Night in Tokyo 2013 -- Tips for getting startedKeisuke Takahashi
 
Cgroup resource mgmt_v1
Cgroup resource mgmt_v1Cgroup resource mgmt_v1
Cgroup resource mgmt_v1sprdd
 
GlusterFS CTDB Integration
GlusterFS CTDB IntegrationGlusterFS CTDB Integration
GlusterFS CTDB IntegrationEtsuji Nakai
 
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)Ron Munitz
 
Loadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitLoadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitFrederic Descamps
 
いろいろ引き出し作って見ました
いろいろ引き出し作って見ましたいろいろ引き出し作って見ました
いろいろ引き出し作って見ましたMutsumi IWAISHI
 

Tendances (20)

Disaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBDDisaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBD
 
Disruptive IP Networking with Intel DPDK on Linux
Disruptive IP Networking with Intel DPDK on LinuxDisruptive IP Networking with Intel DPDK on Linux
Disruptive IP Networking with Intel DPDK on Linux
 
イマドキなNetwork/IO
イマドキなNetwork/IOイマドキなNetwork/IO
イマドキなNetwork/IO
 
CASPUR Staging System II
CASPUR Staging System IICASPUR Staging System II
CASPUR Staging System II
 
Rh202 q&a-demo-cert magic
Rh202 q&a-demo-cert magicRh202 q&a-demo-cert magic
Rh202 q&a-demo-cert magic
 
DLM knowledge-sharing
DLM knowledge-sharingDLM knowledge-sharing
DLM knowledge-sharing
 
Building a Two Node SLES 11 SP2 Linux Cluster with VMware
Building a Two Node SLES 11 SP2 Linux Cluster with VMwareBuilding a Two Node SLES 11 SP2 Linux Cluster with VMware
Building a Two Node SLES 11 SP2 Linux Cluster with VMware
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zun
 
100 M pps on PC.
100 M pps on PC.100 M pps on PC.
100 M pps on PC.
 
Introduction to UNIX
Introduction to UNIXIntroduction to UNIX
Introduction to UNIX
 
Building a Virtualized Continuum with Intel(r) Clear Containers
Building a Virtualized Continuum with Intel(r) Clear ContainersBuilding a Virtualized Continuum with Intel(r) Clear Containers
Building a Virtualized Continuum with Intel(r) Clear Containers
 
DPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabDPDK in Containers Hands-on Lab
DPDK in Containers Hands-on Lab
 
Easy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to AdvancedEasy backup & restore with Clonezilla - Tips form Basic to Advanced
Easy backup & restore with Clonezilla - Tips form Basic to Advanced
 
Gluster Cloud Night in Tokyo 2013 -- Tips for getting started
Gluster Cloud Night in Tokyo 2013 -- Tips for getting startedGluster Cloud Night in Tokyo 2013 -- Tips for getting started
Gluster Cloud Night in Tokyo 2013 -- Tips for getting started
 
Cgroup resource mgmt_v1
Cgroup resource mgmt_v1Cgroup resource mgmt_v1
Cgroup resource mgmt_v1
 
GlusterFS CTDB Integration
GlusterFS CTDB IntegrationGlusterFS CTDB Integration
GlusterFS CTDB Integration
 
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
 
Loadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitLoadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkit
 
いろいろ引き出し作って見ました
いろいろ引き出し作って見ましたいろいろ引き出し作って見ました
いろいろ引き出し作って見ました
 
(Free and Net) BSD Xen Roadmap
(Free and Net) BSD Xen Roadmap(Free and Net) BSD Xen Roadmap
(Free and Net) BSD Xen Roadmap
 

Similaire à High Availability != High-cost

drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015Alexandre Huynh
 
Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10Sean Hull
 
brief introduction of drbd in SLE12SP2
brief introduction of drbd in SLE12SP2brief introduction of drbd in SLE12SP2
brief introduction of drbd in SLE12SP2Nick Wang
 
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBITOpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBITOpenNebula Project
 
State of Containers and the Convergence of HPC and BigData
State of Containers and the Convergence of HPC and BigDataState of Containers and the Convergence of HPC and BigData
State of Containers and the Convergence of HPC and BigDatainside-BigData.com
 
DRBD é um amigo!
DRBD é um amigo!DRBD é um amigo!
DRBD é um amigo!eiichi2009
 
Practice and challenges from building IaaS
Practice and challenges from building IaaSPractice and challenges from building IaaS
Practice and challenges from building IaaSShawn Zhu
 
Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...
Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...
Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...Nagios
 
Linux fundamental - Chap 12 Hardware Management
Linux fundamental - Chap 12 Hardware ManagementLinux fundamental - Chap 12 Hardware Management
Linux fundamental - Chap 12 Hardware ManagementKenny (netman)
 
Coredns nodecache - A highly-available Node-cache DNS server
Coredns nodecache - A highly-available Node-cache DNS serverCoredns nodecache - A highly-available Node-cache DNS server
Coredns nodecache - A highly-available Node-cache DNS serverYann Hamon
 
Hands on Virtualization with Ganeti
Hands on Virtualization with GanetiHands on Virtualization with Ganeti
Hands on Virtualization with GanetiOSCON Byrum
 
Romanticos com drbd 2
Romanticos com drbd 2Romanticos com drbd 2
Romanticos com drbd 2eiichi2009
 
MySQL High Availability Sprint: Launch the Pacemaker
MySQL High Availability Sprint: Launch the PacemakerMySQL High Availability Sprint: Launch the Pacemaker
MySQL High Availability Sprint: Launch the Pacemakerhastexo
 
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE
 
The Forefront of the Development for NVDIMM on Linux Kernel
The Forefront of the Development for NVDIMM on Linux KernelThe Forefront of the Development for NVDIMM on Linux Kernel
The Forefront of the Development for NVDIMM on Linux KernelYasunori Goto
 
Big Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosBig Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosHeiko Loewe
 
Docker and coreos20141020b
Docker and coreos20141020bDocker and coreos20141020b
Docker and coreos20141020bRichard Kuo
 

Similaire à High Availability != High-cost (20)

drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015
 
Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10
 
brief introduction of drbd in SLE12SP2
brief introduction of drbd in SLE12SP2brief introduction of drbd in SLE12SP2
brief introduction of drbd in SLE12SP2
 
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBITOpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
 
State of Containers and the Convergence of HPC and BigData
State of Containers and the Convergence of HPC and BigDataState of Containers and the Convergence of HPC and BigData
State of Containers and the Convergence of HPC and BigData
 
DRBD é um amigo!
DRBD é um amigo!DRBD é um amigo!
DRBD é um amigo!
 
Practice and challenges from building IaaS
Practice and challenges from building IaaSPractice and challenges from building IaaS
Practice and challenges from building IaaS
 
Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...
Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...
Nagios Conference 2012 - Andrew Widdersheim - Nagios is down boss wants to se...
 
Linux fundamental - Chap 12 Hardware Management
Linux fundamental - Chap 12 Hardware ManagementLinux fundamental - Chap 12 Hardware Management
Linux fundamental - Chap 12 Hardware Management
 
MySQL Cluster Basics
MySQL Cluster BasicsMySQL Cluster Basics
MySQL Cluster Basics
 
Coredns nodecache - A highly-available Node-cache DNS server
Coredns nodecache - A highly-available Node-cache DNS serverCoredns nodecache - A highly-available Node-cache DNS server
Coredns nodecache - A highly-available Node-cache DNS server
 
7.pptx
7.pptx7.pptx
7.pptx
 
Hands on Virtualization with Ganeti
Hands on Virtualization with GanetiHands on Virtualization with Ganeti
Hands on Virtualization with Ganeti
 
Romanticos com drbd 2
Romanticos com drbd 2Romanticos com drbd 2
Romanticos com drbd 2
 
MySQL High Availability Sprint: Launch the Pacemaker
MySQL High Availability Sprint: Launch the PacemakerMySQL High Availability Sprint: Launch the Pacemaker
MySQL High Availability Sprint: Launch the Pacemaker
 
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
 
The Forefront of the Development for NVDIMM on Linux Kernel
The Forefront of the Development for NVDIMM on Linux KernelThe Forefront of the Development for NVDIMM on Linux Kernel
The Forefront of the Development for NVDIMM on Linux Kernel
 
Big Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosBig Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and Mesos
 
Docker and coreos20141020b
Docker and coreos20141020bDocker and coreos20141020b
Docker and coreos20141020b
 
Web Server Free Bsd
Web Server Free BsdWeb Server Free Bsd
Web Server Free Bsd
 

Dernier

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 

Dernier (20)

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
 
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 

High Availability != High-cost

  • 1. High Availability != High-cost Building low-cost Master / Slave Clusters on a Linux based Operating Systems to provide mission critical services Norman Maurer ApacheCon EU 2009 March 27, 2009
  • 2. Norman Maurer Member of the Apache Software Foundation  Apache Software Foundation Infrastructure Team  PMC Apache JAMES  Senior Unix System Engineer  norman@apache.org norman.maurer@heagmedianet.de http://www.heagmedianet.com http://myblog.kicks-ass.org 2
  • 3. The question should be: “Why should you not use Linux-HA?“.. ENOCLUE! It's flexible  It's opensource  It has a very active community  It's known to work for high-volume  servers It's just freakin' cool!  3
  • 4. Components for building the Cluster 4
  • 5. The Linux-HA Cluster uses two base components that make up the Cluster.. Heartbeat*  DRBD**  * - http://www.linux-ha.org **- http://www.drbd.org 5
  • 6. Putting stuff together for an Apache HTTPD Cluster The magic glue 6
  • 7. Brief overview of how the whole Cluster works Webserver01 Webserver02 Heartbeat NIC HA NIC Heartbeat DRBD Data DRBD NIC NIC Services Virtual-IP Master Slave Failover!!! 7 Users
  • 8. DRBD a.k.a network raid1 Mirror your data across the network 8
  • 9. Benefits when using DRBD Synchronous and asynchronous replication  Works at the block level  Many Distributions ship with pre-build DRBD  modules / utils Works with every filesystem on-top while in  Master / Slave-Cluster Master / Slave Cluster and Master / Master-  Cluster possible 9
  • 10. Limitations when using DRBD I/O throughput heavily depends on network  throughput Only Master / Slave Cluster supported when  using traditional filesystems (ext3, xfs, reiserfs) Master / Master Cluster only works with shared  cluster file system (GFS, OCFS2) Data access is only possible from the Master  Node in a Master / Slave Cluster 10
  • 11. Brief overview on how DRBD works Webserver01 Webserver02 Filesystem Filesystem DRBD-Layer DRBD-Layer NIC NIC Device / Device / Disk Disk Ethernet Sync data across network 11
  • 12. Installation Ubuntu  # apt-get install drbd8-utils drbd8-module-source build- essential module-assistant && module-assistant auto- install drbd8 Debian  # apt-get install drbd8-utils drbd8-modules SLES  # yast -i drbd Others  Check if there are prebuild packages / build it yourself from source! 12
  • 13. Configuration of DRBD-Nodes Set correct config parameters on both Nodes  /etc/drbd.conf: Synchron write-operations global { usage-count yes; } common { protocol C; Specify resource } resource r0 { You can have more then one! on node01 { device /dev/drbd0; disk /dev/sda7; address 10.0.0.2:7789; Configuration of DRBD- meta-disk internal; } Nodes on node02 { device /dev/drbd0; disk /dev/sda7; address 10.0.0.3:7789; meta-disk internal; } } 13
  • 14. Create the device and up it Create device metadata  # drbdadm create-md r0 Attach to backing device,set synchronization  parameters and connect to peer # drbdadm up r0 Check state  # cat /proc/drbd version: 8.0.11 (api:86/proto:86) GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r--- … … 14
  • 15. Initial device synchronization Set Primary  # drbdadm -- --overwrite-data-of-peer primary r0 Check state again  # cat /proc/drbd version: 8.0.11 (api:86/proto:86) GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r--- …. …. 15
  • 16. File system layout and data migration Put filesystem on-top  # mkfs.ext3 /dev/drbd0 Temporarily mount the filesystem to generate  folder structure # mount /dev/drbd0 /drbd Create required folders and copy content  # mkdir -p /drbd/etc /drbd/usr/lib /drbd/var # rsync -az /etc/apache2 /drbd/etc/ # rsync -az /var/www /drbd/var/ # rsync -az /usr/lib/cgi-bin /drbd/usr/lib/ 16
  • 18. Main features of Heartbeat Heartbeat packets via ICMP / UDP / Serial  Works with “every“ unix daemon  Be able to execute scripts on failover  Automatic failback  Support of “STONITH*“ devices  *shoot the other node in the head 18
  • 19. Brief overview on how Heartbeat works Heartbeats Webserver01 Webserver02 Heartbeat Heartbeat Checks Checks TCP/IP NIC NIC Scripts Scripts COM Serial COM Start Stop Start Stop Master Slave Called when becoming Called when becoming Master-Node Slave-Node 19
  • 20. Installation of Heartbeat Ubuntu / Debian  # apt-get install heartbeat SLES  # yast -i heartbeat Other Distributions  Check for prebuild packages / build it yourself from source ! 20
  • 21. Installation of extra-scripts Ubuntu / Debian  # apt-get install drbdlinks Other Distributions  Check for prebuild packages / build it your from source! 21
  • 22. Configuration of extra-scripts Set correct config parameters for drbdlinks on  both nodes /etc/drbdlinks.conf: link('/etc/apache2', '/drbd/etc/apache2') link('/var/www', '/drbd/var/www') link('/usr/lib/cgi-bin', '/drbd/usr/lib/cgi-bin') Specify all directories which needs to be “linked“ from the mounted DRBD device to the correct location. 22
  • 23. Configuration of Heartbeat – Part I Set correct config parameters on both nodes  /etc/heartbeat/ha.cf: debugfile /var/log/ha-debug logfile /var/log/ha-log Don't fallback when the Logfacility local0 keepalive 2 primary node comes up deadtime 60 again warntime 30 initdead 120 udpport 697 auto_failback off node node01 node node02 debug 0 Nodes need to be resolvable 23
  • 24. Configuration of Heartbeat – Part II Configure script execution on failover  /etc/heartbeat/haresources: node01 xxx.xxx.xxx.xxx drbddisk::r0 Filesystem::/dev/drbd0::/drbd::ext3 drbdlinks apache2 Primary DRBD Mount FS Link Start Virtual → Node dirs Apache2 IP Primary Scripts needs to be located under:   /etc/init.d/  /etc/heartbeat/resources.d/ 24
  • 25. Configuration of Heartbeat – Part III Setup encryption and passphrase  /etc/heartbeat/authkeys: File needs to have auth 1 1 sha1 your_super_secure_password perms 600! Supported encryption   md5  sha1 Not really an  crc encryption. Just packet corruption prevention 25
  • 26. Final Steps.. Disable startup scripts for daemons started by  Heartbeat! Reboot both servers to see if all services will  be started on the next reboot Test failover to see if all works as aspected  26
  • 27. Clustering in production What is known to work 27
  • 28. What services are known to work ? Mail  Ubuntu,Qmail,Vpopmail Webserver  Ubuntu, Apache HTTPD 1.3.x / 2.0.x/ 2.2.x Database  Ubuntu, MySQL 4.1 / 5.0 / 5.1 Radius  Ubuntu, Freeradius ...... ..... and thousands of happy customers 28
  • 29. Best Practices – Part 1 Set the heartbeat and DRBD utils/modules to  “hold“ in package-management Disable Heartbeat on the Slave-Node while  performing upgrades Use dedicated NIC  Use decent NIC (like intel)  Use gigabit ethernet  29
  • 30. Best Practices – Part 2 Use different UDP Ports (for heartbeats) per  Linux-HA Cluster Test failover before putting the Cluster in  production Keep the Heartbeat and DRBD config files in-  sync Use „just“ an crossover-cable to connect the  nodes if possible 30
  • 31. Known problems What problems often show up and how to fix them 31
  • 32. DRBD – Part I Split brain scenario  Crappy throughput  Version mismatch  Kernel panic on both nodes  UID/GID not match on both Nodes  32
  • 33. Heartbeat Split brain scenario  Heartbeats get dropped  Nodes not „see“ each other  33
  • 34. Useful resources Linux-HA  http://www.linux-ha.org DRBD  http://www.drbd.org DRBDLinks  http://www.tummy.com/Community/software/drbdlinks/ DRBD and MySQL  http://dev.mysql.com/doc/refman/5.1/en/faqs-mysql-drbd-heartbeat.html 34
  • 36. Thank you for your attention! 36