Architecture of the High Availability Solution for Ganesha and Samba with Kaleb Keithley
•
0 j'aime•1,364 vues
Architecture of the High Availability Solution for Ganesha and Samba with Kaleb Keithley at Gluster Developer Summit 2016
Recommandé
Contenu connexe
En vedette
Plus de Gluster.org
Plus de Gluster.org (20)
Dernier
Dernier (20)
Architecture of the High Availability Solution for Ganesha and Samba with Kaleb Keithley
- 1. Indiana Jones and the Temple of NFS Ganesha HA Kaleb Keithley Gluster Summit, Berlin 6 October, 2016 These aren't the droids you're looking for
- 2. pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd Setup Pacemaker, Gluster, and create volume But I was going into Tosche Station to pick up some power converters!
- 3. pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd My HA Cluster Create HA Cluster Don't call me a mindless philosopher, you overweight glob of grease.
- 5. Set Constraints pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha-active ? grace-active ? pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha-active ? grace-active ? pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha-active ? grace-active ? pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha-active ? grace-active ? Fast ship? You've never heard of the Millennium Falcon?
- 6. Start ganesha.nfsd pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha If there's a bright center to the universe, you're on the planet that it's farthest from.
- 7. ganesha_mon notices ganesha.nfsd is running pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active ganesha-grace Help me, Obi-Wan Kenobi; you're my only hope.
- 8. Ganesha.nfsd dies pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active ganesha-grace I used to bullseye womp rats in my T-16 back home, they're not much bigger than two meters.
- 10. surviving ganesha.nfsds placed into NFS-GRACE pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active Who's the more foolish? The fool, or the fool who follows him?
- 12. VIP fails over pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsdblock unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace VIP2 You can't win, Darth. If you strike me down I will become more powerful than you could possibly imagine.
- 13. Next Steps ● Switch to storhaug (anyone fancy a trip to Norway?) ● Fencing? I've got a bad feeling abou this.
- 14. ● https://github.com/linux-ha-storage/storhaug ● http://clusterlabs.org/ I've got to rest before I fall apart
- 15. Indiana Jones and the Temple of NFS Ganesha HA Kaleb Keithley Gluster Summit, Berlin 6 October, 2016 These aren't the droids you're looking for
- 16. pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd Setup Pacemaker, Gluster, and create volume But I was going into Tosche Station to pick up some power converters! Four node cluster Pacemaker + Corosync running Glusterd running + four brick volume is started
- 17. pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd pacemaker glusterd glusterfsd My HA Cluster Create HA Cluster Don't call me a mindless philosopher, you overweight glob of grease. Glusterd invokes ganesha-ha.sh –setup Setup create the pacemaker cluster, plus some other pacemaker voodoo magic, e.g. + authenticates w/ all members + creates cluster w/ name and members + starts cluster + waits for cluster to elect a domain controller (DC) + sets quorum policy if < 3 members + disables stonith ...
- 18. pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock Create Resource Agents pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock I find your lack of faith disturbing. Creates resources (resource agents) + Floating (or virtual) IPs (IPaddr) and portblock/unblock + ganesha_mon, ganesha_grace, ganesha_nfsd What they do: + ganesha_mon checks to see if ganesha.nfsd is running or not + ganesha_grace sends dbus signals to put ganesha.nfsd into grace + ganesha_nfsd fixes up /var/lib/nfs/. (but storehaug) Then lots of pacemaker constraint location voodoo to make VIPs fail over in a deterministic manner
- 19. Set Constraints pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha-active ? grace-active ? pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha-active ? grace-active ? pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha-active ? grace-active ? pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha-active ? grace-active ? Fast ship? You've never heard of the Millennium Falcon? VIPs want to be where ganesha-active attribute exists ganesha_grace run when ganesha-grace attribute exists. ganesha_grace uses a pacemaker feature called “notify” available to clone resources. If ganesha_grace stops on node2, the other ganesha_grace RAs are notified.
- 20. Start ganesha.nfsd pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha If there's a bright center to the universe, you're on the planet that it's farthest from. After everything is set up, gluster/glusterd starts the ganesha.nfsds on all the nodes
- 21. ganesha_mon notices ganesha.nfsd is running pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active ganesha-grace Help me, Obi-Wan Kenobi; you're my only hope. ganesha_mon sees that ganesha.nfsd is running and creates ganesha-active and ganesha-grace attributes. VIP RAs get happy, clients mount using the Floating IP addrs. Time passes…...
- 22. Ganesha.nfsd dies pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active ganesha-grace I used to bullseye womp rats in my T-16 back home, they're not much bigger than two meters. Oops, one of the ganesha.nfsds died.
- 23. ganesha_mon deletes grace-active pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active In my experience, there's no such thing as luck. ganesha_mon notices and deletes the ganesha-grace attr.
- 24. surviving ganesha.nfsds placed into NFS-GRACE pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha-active Who's the more foolish? The fool, or the fool who follows him? Pacemaker notifies the other ganesha_grace RAs, which send their ganesha.nfsd a dbus message putting them into NFS-GRACE The ganesha.nfsds do whatever they do in grace, e.g. migrate locks, etc.
- 25. pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP2 block unblock ganesha ganesha_mon deletes ganesha-active You came in that thing? You're braver than I thought. Then ganesha-mon deletes the ganesha-active attr….
- 26. VIP fails over pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsdblock unblock ganesha pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP1 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP3 block unblock ganesha ganesha-active ganesha-grace pacemaker glusterd glusterfsd mon nfsd grace VIP block unblock pacemaker glusterd glusterfsd mon nfsd grace VIP4 block unblock ganesha ganesha-active ganesha-grace VIP2 You can't win, Darth. If you strike me down I will become more powerful than you could possibly imagine. Which causes the floating IP addr to fail-over to the next node based on the location constraints. When the operator restarts the ganesha.nfsd, everything “unwinds”. The ganesha.nfsds are put into grace, and the floating IP fails back to the node. In storehaug (Jose’s version) gluster/glusterd don’t run ganesha-ha.sh –setup, that’s done by the admin (or some other thing TBD. gluster/glusterd doesn’t start the ganesha.nfsds. Instead they’re started by the ganesha_nfsd RA along with the Samba server after /var/lib/nfs is setup.
- 27. Next Steps ● Switch to storhaug (anyone fancy a trip to Norway?) ● Fencing? I've got a bad feeling abou this.
- 28. ● https://github.com/linux-ha-storage/storhaug ● http://clusterlabs.org/ I've got to rest before I fall apart