Recommandé
Recommandé
Contenu connexe
Similaire à Disasters!
Similaire à Disasters! (20)
Dernier
Dernier (20)
Disasters!
- 1. Hi
- 2. me
- 3. me mickey
- 4. me mickey
- 5. me mickey
- 6. me mickey
- 7. me mickey
- 8. me mickey
- 9. me mickey
- 10. me mickey
- 11. me mickey
- 12. me mickey
- 13. me mickey
- 14. me mickey
- 15. me mickey (^ full stop)
- 16. Disasters
- 17. Disasters (oh my!)
- 18. More disasters:
- 19. More disasters:
- 20. More disasters:
- 21. More disasters:
- 22. More disasters:
- 24. Emergencies
- 25. Emergencies (can be)
- 26. Emergencies (can be) Disastrous
- 27. Emergencies (can be) Disastrous
- 28. Plan! • Prevent • Communicate • Recover • Energize!
- 29. Communicate
- 30. Communicate :(
- 31. Communicate :( (share)
- 32. Communicate :( (share)
- 33. Test
- 36. Test (like voting) don’t be a Dick
- 37. pound of cure • Colocate / managed services • Good server/network design • Network backups (mozy, symantec, iron mountain)
- 39. In the closet • File server (if it’s a file, it should be on the server) • Local backups • Servers (v. workstations)
- 40. Out of the closet • Everything else. • Network backups
- 41. Security
- 42. Security DOS
- 43. Security DOS “hacking”
- 44. Security DOS “hacking” insider attacks, social engineering, etc.
- 45. Security DOS “hacking” insider attacks, social engineering, etc.
- 46. Communications • Diversify • who calls whom when and how?
- 47. the starting gun • When does an emergency start? • Milestones
- 48. Continuity planning • How long can you afford to be down? • How much does it cost to stay up?
- 49. 0-24hr downtime Hot standby Geographical redundancy “spare” servers Network connectivity?
- 50. Work from home
Notes de l'éditeur
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- Name is Mickey [cue x2], I used to work at Verio [cue x2] for 10 years where I managed the east coast network.. Now I co-own infamia [cuex2] with my business partner Ernesto [cue x2], which is also the name of a tropical storm [cuex2] I spent at anchor by Vera’s White Sands just down the street. Some people say I talk too fast [cue x2] so if you don’t understand me, please feel free to raise a hand or raise your voice and stop me [cue]. I’m here to talk a bit about Disasters.
- A few “disasters” I’ve had to work through: on a normal night in 2003 the northeeast looks like this, and I had at least one data center in every major city. When this [slide] happened, it wasn’t very much fun. Hurricane isabelle (pictured is Hatteras, before and after Isabelle), when I was carrying jerry cans of diesel to our generator because I couldn’t get in touch with the fuel truck. And 9/11, of which I have no pictures because nobody wants to see that again, but props to my man Louis who gave us updates when he could. Lastly: not all disasters make the news, so we’ll address small disasters here as well. Quite frankly the nature of the disaster does not matter much. What does matter is how it affects you, your network, your data, your clients. So instead of disasters I like to talk about emergencies.
- A few “disasters” I’ve had to work through: on a normal night in 2003 the northeeast looks like this, and I had at least one data center in every major city. When this [slide] happened, it wasn’t very much fun. Hurricane isabelle (pictured is Hatteras, before and after Isabelle), when I was carrying jerry cans of diesel to our generator because I couldn’t get in touch with the fuel truck. And 9/11, of which I have no pictures because nobody wants to see that again, but props to my man Louis who gave us updates when he could. Lastly: not all disasters make the news, so we’ll address small disasters here as well. Quite frankly the nature of the disaster does not matter much. What does matter is how it affects you, your network, your data, your clients. So instead of disasters I like to talk about emergencies.
- A few “disasters” I’ve had to work through: on a normal night in 2003 the northeeast looks like this, and I had at least one data center in every major city. When this [slide] happened, it wasn’t very much fun. Hurricane isabelle (pictured is Hatteras, before and after Isabelle), when I was carrying jerry cans of diesel to our generator because I couldn’t get in touch with the fuel truck. And 9/11, of which I have no pictures because nobody wants to see that again, but props to my man Louis who gave us updates when he could. Lastly: not all disasters make the news, so we’ll address small disasters here as well. Quite frankly the nature of the disaster does not matter much. What does matter is how it affects you, your network, your data, your clients. So instead of disasters I like to talk about emergencies.
- A few “disasters” I’ve had to work through: on a normal night in 2003 the northeeast looks like this, and I had at least one data center in every major city. When this [slide] happened, it wasn’t very much fun. Hurricane isabelle (pictured is Hatteras, before and after Isabelle), when I was carrying jerry cans of diesel to our generator because I couldn’t get in touch with the fuel truck. And 9/11, of which I have no pictures because nobody wants to see that again, but props to my man Louis who gave us updates when he could. Lastly: not all disasters make the news, so we’ll address small disasters here as well. Quite frankly the nature of the disaster does not matter much. What does matter is how it affects you, your network, your data, your clients. So instead of disasters I like to talk about emergencies.
- Emergencies have a tricky tendency of not always being associated with a disaster. But even when they’re not, they can be disastrous: a hard drive crash, power failure, a/c failure, connectivity failure all seem to happen at the worst time. We used to have a point of present in columbia called the “barn” (because it was in an actual barn). An entire disk array decided to go bad one night. Christmas Eve, I believe. How do we cope with hurricanes, floods, dust storms, and barn animals? Three things.
- Emergencies have a tricky tendency of not always being associated with a disaster. But even when they’re not, they can be disastrous: a hard drive crash, power failure, a/c failure, connectivity failure all seem to happen at the worst time. We used to have a point of present in columbia called the “barn” (because it was in an actual barn). An entire disk array decided to go bad one night. Christmas Eve, I believe. How do we cope with hurricanes, floods, dust storms, and barn animals? Three things.
- Emergencies have a tricky tendency of not always being associated with a disaster. But even when they’re not, they can be disastrous: a hard drive crash, power failure, a/c failure, connectivity failure all seem to happen at the worst time. We used to have a point of present in columbia called the “barn” (because it was in an actual barn). An entire disk array decided to go bad one night. Christmas Eve, I believe. How do we cope with hurricanes, floods, dust storms, and barn animals? Three things.
- Emergencies have a tricky tendency of not always being associated with a disaster. But even when they’re not, they can be disastrous: a hard drive crash, power failure, a/c failure, connectivity failure all seem to happen at the worst time. We used to have a point of present in columbia called the “barn” (because it was in an actual barn). An entire disk array decided to go bad one night. Christmas Eve, I believe. How do we cope with hurricanes, floods, dust storms, and barn animals? Three things.
- Have an emergency management/preparedness plan. Plan should include: preventative measures (backups, network redundancy, off-site backups, managed v. self-hosted, etc) communications strategies: don’t assume you’ll be able to reach the person you need in an emergency. Landlines and cellphones may be useless. Plan on diverse communications strategies Recovery methods. Will you need new hardware? different connectivity? What repair/replace times does your support/maintenance plan with your vendors promise you? Your backups vendor? lastly: When do you trigger your emergency plan? This is not always so simple in the middle of an ongoing disaster to decide that this really is a disaster and it won’t get better in about 35 seconds. Hope springs eternal… More on these, but first…
- So you got yourself a plan, you got it approved, printed at the printers with twenty seven eight-by-ten colour glossy photographs with circles and arrows and a paragraph on the back of each one just like Officer Obie…and then shit hits the fan and Mickey can’t find you and he calls up Fred, and guess what, Fred’s never even seen the plan. So sad. (like the officer Obie when he saw the blind judge) [cue] So, share. I know it’s your baby, but it does you no good if nobody knows it. Play well with others: share. [cue] You won’t know who’ll be around when disaster hits, so the question is not “who needs this”, but rather, why would anyone NOT have this? Hire a skywriter if you have to. Get the word out.
- So you got yourself a plan, you got it approved, printed at the printers with twenty seven eight-by-ten colour glossy photographs with circles and arrows and a paragraph on the back of each one just like Officer Obie…and then shit hits the fan and Mickey can’t find you and he calls up Fred, and guess what, Fred’s never even seen the plan. So sad. (like the officer Obie when he saw the blind judge) [cue] So, share. I know it’s your baby, but it does you no good if nobody knows it. Play well with others: share. [cue] You won’t know who’ll be around when disaster hits, so the question is not “who needs this”, but rather, why would anyone NOT have this? Hire a skywriter if you have to. Get the word out.
- So you got yourself a plan, you got it approved, printed at the printers with twenty seven eight-by-ten colour glossy photographs with circles and arrows and a paragraph on the back of each one just like Officer Obie…and then shit hits the fan and Mickey can’t find you and he calls up Fred, and guess what, Fred’s never even seen the plan. So sad. (like the officer Obie when he saw the blind judge) [cue] So, share. I know it’s your baby, but it does you no good if nobody knows it. Play well with others: share. [cue] You won’t know who’ll be around when disaster hits, so the question is not “who needs this”, but rather, why would anyone NOT have this? Hire a skywriter if you have to. Get the word out.
- And now that everyone has the plan, you need to test it. Early and often. [cue] And end-to-end. If you’re testing a backup, do erase the original file (make sure it’s backed up first). If you’re testing a generator, do turn off municipal power. If we had a guitar, I’d sing you the story of old Dick, our Facilities guy in Alexandria (VA). Dick used to test the generator religiously every month. Every month, on the firsrt Monday, Dick would fire it up, let it warm up, test voltages, fill in forms, take measurements, and on and on. Then we had a power outage. The generator fired up. The generator warmed up, the measurements were measured. Then the transfer switch saw voltage…for the first time in about 5 years, and about 4 copper bus bars promptly burnt to a crisp. The data center lost power. Surprisingly, Dick didn’t get fired, but the moral of the story is still the same [cue]: test the entire thing, end to end. Simulate a problem. Do it in controlled circumstances, and no one gets hurt.
- And now that everyone has the plan, you need to test it. Early and often. [cue] And end-to-end. If you’re testing a backup, do erase the original file (make sure it’s backed up first). If you’re testing a generator, do turn off municipal power. If we had a guitar, I’d sing you the story of old Dick, our Facilities guy in Alexandria (VA). Dick used to test the generator religiously every month. Every month, on the firsrt Monday, Dick would fire it up, let it warm up, test voltages, fill in forms, take measurements, and on and on. Then we had a power outage. The generator fired up. The generator warmed up, the measurements were measured. Then the transfer switch saw voltage…for the first time in about 5 years, and about 4 copper bus bars promptly burnt to a crisp. The data center lost power. Surprisingly, Dick didn’t get fired, but the moral of the story is still the same [cue]: test the entire thing, end to end. Simulate a problem. Do it in controlled circumstances, and no one gets hurt.
- And now that everyone has the plan, you need to test it. Early and often. [cue] And end-to-end. If you’re testing a backup, do erase the original file (make sure it’s backed up first). If you’re testing a generator, do turn off municipal power. If we had a guitar, I’d sing you the story of old Dick, our Facilities guy in Alexandria (VA). Dick used to test the generator religiously every month. Every month, on the firsrt Monday, Dick would fire it up, let it warm up, test voltages, fill in forms, take measurements, and on and on. Then we had a power outage. The generator fired up. The generator warmed up, the measurements were measured. Then the transfer switch saw voltage…for the first time in about 5 years, and about 4 copper bus bars promptly burnt to a crisp. The data center lost power. Surprisingly, Dick didn’t get fired, but the moral of the story is still the same [cue]: test the entire thing, end to end. Simulate a problem. Do it in controlled circumstances, and no one gets hurt.
- Back to prevention. Some more thoughts on that. This is a boring slide, so take notes. -Colocate or managed: that wiring closet full of your “server” that used to be janitorial supplies? Get rid of it. Data centers are built to house computers. Closets are filled to house janitorial supplies. Your janitor will love you, and your boss too. - strong network and server architecture from the start will help prevent a lot of problems, so you won’t need the backups. Ever wondered why some computers cost $300 and some $3000? Go for the $3000. Add RAID, redundant database/fileservers, etc. -Network backups: bla bla bla Or you can rely on Dick (remember Dick?) to take some tape home on Tuesdays.
- Don’t trust powerstrips. They’re good for maybe a year or so but at least used to be notoriously for failing. In addition, when the surge does trip the power strip, the machines go down. UPS (Uninterruptible Power Supply) systems are the way to go for your in-house server. (Do not plug printers into UPS ). The best-quality UPS put out cleaner AC power than your utility power.
- I did say you should not keep your servers in-house, but if you have an office, you’ll likely need to keep some server in-house. Your file server, print server, etc. Number one no-questions asked policy: Everyone keeps all the files in the file server. In addition, a local backup can help get you back up and running very quickly in the event of a single server failure.
- Everything that doesn’t need to be a stone’s throw away, should be outsourced, colocated, or hosted. (define these) Network backups: of your file server at least. Test backups end-to-end. (My quickbooks file was not compatible with a new qbooks install from DVD, and updates were no longer available from quckbooks)
- Denial of service attacks are annoying and cause significant productivity loss. These are a significant concern for hosted services rather than your in-house servers More lossy, hacking attack where an intruder gains access to your server may have more severe consequences, from loss of data to compromise of sensitive information Most attacks where the intruder gains access to a server/network are linked to someone on the “inside”: a disgruntled employee or ex-employee, someone’s personal computer/laptop was poorly secured and compromised, etc. A network is as secure as its weakest link…are your staff’s computers up-to-date with patches?
- Denial of service attacks are annoying and cause significant productivity loss. These are a significant concern for hosted services rather than your in-house servers More lossy, hacking attack where an intruder gains access to your server may have more severe consequences, from loss of data to compromise of sensitive information Most attacks where the intruder gains access to a server/network are linked to someone on the “inside”: a disgruntled employee or ex-employee, someone’s personal computer/laptop was poorly secured and compromised, etc. A network is as secure as its weakest link…are your staff’s computers up-to-date with patches?
- Denial of service attacks are annoying and cause significant productivity loss. These are a significant concern for hosted services rather than your in-house servers More lossy, hacking attack where an intruder gains access to your server may have more severe consequences, from loss of data to compromise of sensitive information Most attacks where the intruder gains access to a server/network are linked to someone on the “inside”: a disgruntled employee or ex-employee, someone’s personal computer/laptop was poorly secured and compromised, etc. A network is as secure as its weakest link…are your staff’s computers up-to-date with patches?
- Denial of service attacks are annoying and cause significant productivity loss. These are a significant concern for hosted services rather than your in-house servers More lossy, hacking attack where an intruder gains access to your server may have more severe consequences, from loss of data to compromise of sensitive information Most attacks where the intruder gains access to a server/network are linked to someone on the “inside”: a disgruntled employee or ex-employee, someone’s personal computer/laptop was poorly secured and compromised, etc. A network is as secure as its weakest link…are your staff’s computers up-to-date with patches?
- The only thing I can say about this is diversify. VoIP as well as landline as well as cell as well as satelite phones. Most importantly, communications should be a part of your plan. Your girl on the ground will be the one who knows best how to get word out. Will she know whom to call? With what information? Will the guy on the other end know what to tell her? how to send what she needs? On 9/11, Louis could only call out twice in 2 days. And he still had to call his family. Make sure that one call counts. Have bail money handy.
- Disasters cost money. Maybe a lot of money. When is it time to dispatch that executive jet to send replacement parts? (this is actually a service we used to have at Verio). The guy on the ground with his computers on fire should not be placed in a position to make that decision as well. Especially since the guy on the ground may not know the full cost/rewards scenario for your company. Your plan should have reactionary milestones: after 3 hours of downtime, for example. Or 75% of cusotmers without service.