Dibi Conference 2012
•Télécharger en tant que KEY, PDF•
5 j'aime•2,121 vues
My talk from the DIBI conference in Newcastle in April 2012
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (18)
Similaire à Dibi Conference 2012
Similaire à Dibi Conference 2012 (20)
Dernier
Dernier (20)
Dibi Conference 2012
- 1. Just Build It! Scott Rutherford, DIBI 2012
- 4. CERN
- 5. • Axomic (OpenAsset) • Caron Software • Caron Records / Publishing • Yabb / Cominded • ThatsUseful (BuddyMap, Boovu, et al.) • RecommendBox • Njoyed • SlimTimer
- 6. Market leading customer feedback & support solution 21 strong in San Francisco & Raleigh 90,000+ organizations have used us $2M+ raised over 2 funding rounds Tens of millions of registered users 182 people/sec see our product
- 7. Failing Up
- 8. Happen’s To The Best • Facebook Beacon (Feature) • Google Wave (Product) • Twitter - Scaling (Technology) • Coke - New Coke! (Marketing)
- 9. “we're not that dumb, and we're not that smart!” - Don Keough
- 10. Shit Happens!
- 11. You Can’t Plan For Everything! Things are going to go wrong. It’s the way we react at these moments that defines to a great degree how successful we will be. Don’t spend time planning for every eventuality. Just be prepared to move quickly and decisively when situations arise. Don’t build failure in though!!!
- 13. Just Build It!
- 14. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code
- 15. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code
- 16. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code
- 17. ActiveScaffold (AKA AjaxScaffold Plugin) One line of Code >100K Installs > 750K Visitors
- 21. Version 1.0 • Rails 1.2.3 / HopToad / NewRelic / CloudFront • Engine Yard • Shared MySQL database • 2 slices (and staging) • Admin section based on ActiveScaffold • No billing ability until the end of 2008 (then using Freemium OSS)
- 23. Building Fast • Created our own problems • Insufficient test coverage and no CI
- 24. Building Fast • Created our own problems • Insufficient test coverage and no CI • deploy production:rollback • Watch those AJAX calls
- 25. Building Fast • Created our own problems • Insufficient test coverage and no CI • deploy production:rollback • Watch those AJAX calls • Be careful when testing.....
- 26. Building Fast • Created our own problems • Insufficient test coverage and no CI • deploy production:rollback • Watch those AJAX calls • Be careful when testing.....
- 27. Stunning Timing
- 28. Stunning Timing April 2009 Raised $800k
- 29. Version 2.0
- 30. Features, Features, Features • Reporting / Administration Tools • Multiple Languages (now crowd sourced) • Search that worked (Sphinx) • Single Sign On • API • Billing (constant PITA)
- 31. Growing.... • 15 - 20% month • 6 web boxes • 2 databases (master / slave) • Staging • Admin • Office!
- 33. Problems • Shared database started to creak • Tables growing in size - queries not scaling • Overused MySQL - queue, stats • Not enough caching • Dark corners - data exports, email notifications • Still insufficient test coverage and no CI • Us...... (production deploy:rollback)
- 34. “Zee Germans”
- 35. Into The Cloud.... • Moved servers to SoftLayer • Passenger and Ruby Enterprise • Signed up 4 million new users in 2 hours • 6 slices expanded to 24 • Stability resumed.......
- 36. Still Shaky • Redis / Resque for the queue • Redis for stats • More memcached • Make as many things background jobs as possible • Things are under control • Or so we thought.....
- 37. Meltdown • 10 days of almost stability • Flew to SXSW • MySQL melted (still on 5.0) • Percona rescued us from the dreaded MYSQL AUTO INCREMENT BUG (Moved us to 5.1) • Finally have a scalable redundant system, right?
- 38. DNS can fail? • Using ZoneEdit (free) • All nameservers went down, all hell broke loose • Contacted by DynECT over Twitter! • Gave us a free account to get going again. Moved to them • Stopped thinking about DNS!
- 39. Retention / Metrics 91% :( 10 months
- 40. What now? • Ongoing monitoring - daily slice stored in Mongo • Surveys at exit • Addition of HelpDesk & Knowledge Base • Integrations • API, Facebook, iOS
- 41. “Zee Germans”
- 42. And back to Earth with a bump • Softlayer had a few major outages on their cloud infrastructure • Slices are still shared resources • You can’t get someone to just reboot the box • Lesson: cloud is great to scale with, not so good for stability
- 43. So where are we now? • 20 systems in production (2 cloud instances) • Web - 6x dual cpu/quad core 12G RAM nginx/passenger, memcache • Resque - 3x single cpu/quad core 8gb RAM • Redis - Dual CPU/quad core 12g RAM • Mongo - 2x dual cpu/hex core 48G RAM 12x 15k RPM SAS drives RAID10 • MySQL - 2x dual cpu/hex core 72G RAM 24x 15k RPM SAS drives RAID10 • 97% month / month retention
- 44. What have we learned?
- 45. What have we learned? if anything.....
- 46. Performance Realization > 200ms stuffed
- 48. Gems
- 49. More Gems
- 52. Mouth / UGC Styles
- 53. The right tool!
- 54. Databases • Redis - Queue, Stats / Admin • Mongo - Stats / Mouth, PageViews, Financials, Retention • MySQL - Main data store • Sphinx - Search • Memcached - Caching
- 55. Libraries • jQuery • Backbone • D3 • SVG • MooTools Server • SASS • Compass
- 56. Software • DSpam • Chef • Ganglia • Nagios • Capistrano • Passenger • Nginx / HAProxy
- 57. External Services • Mailgun • Akamai • Embedly • Akismet • Pusher App • Dynect • Airbrake • SoftLayer • S3 • GitHub
- 58. Wish We’d Had
- 59. K.I.S.S 200 moving parts to change a lightbulb.....
- 60. Localize
- 61. Unsinkable?
- 62. Stealth mode is for..
- 63. Listen, no really... "If I had asked people what they wanted, they would have said faster horses." Attributed to Henry Ford
- 64. Our Creations • Resque Sliders - https://github.com/ kmullin/resque-sliders • Serve - http://get-serve.com/ • Mouth - https://github.com/cypriss/ mouth • UGC Styles - https://github.com/ joshuarudd/ugc-styles • ActiveScaffold - https://github.com/ activescaffold/active_scaffold
- 65. Just Build It! The best time to start is NOW!!
- 66. Scott Rutherford @scottrutherford scott@uservoice.com
Notes de l'éditeur
- Great talk by Ted Roden, really glad I didn’t use the focus slide. I looked at exactly the same picture as he used, the dangers of going last! \n\nJetlag has seriously kicked in so if I just fall asleep standing here can someone throw one of those mics at me\n
- How many people are involved in startups / open source projects / code projects or have something in mind or gestating in their heads.\n
- music degree\nunfinished\ndistracted by dj’ing and dance music\nwoke up one day (a bit of a theme apparently) and decided to do a physics degree which led to the amazing experience of doing a PhD\n
- CERN, situated half in Geneva / France. 27km underground tunnel containing the world largest machine the LHC. The purpose of which is to investigate the GUT and the elementary particles of the universe. It also happens to be where Tim Berners Lee created the web.\n\nBecame a little obsessed with programming and computers\n
- That led to starting a software company called Axomic, which surprisingly was pretty successful. But lesson 1, try to pick a big market... Architectural image management is not a big pond\n\nFollowed by trying to start or at least build several other things. \n
- Those lead up to my current company and role which is as COO and co-founder of UserVoice.\n
- The observant amongst you, will of noticed that this was the original title of this talk, not a startup conference so I tried to make it more of a mix \n
- Perhaps over embraced in the US, not embraced enough elsewhere. \n
- \n
- \n
- \n
- November 1940, Tacoma Narrows Bridge, state of Washington, route 16. indexes, massive loops\n
- Try to be a bit more positive. Two elements to it, one being the just do it concept. If you have an idea, can get involved in open source, do so you never know where it will take you. Secondly, sometimes it seems like there is a mountain of stuff you need to know to do something might be true, but to be honest you can learn everything you need to know on the way, so. Along those lines, this is some of the story of UserVoice.\n
- It starts in 2006, been playing around with Rails. \n\nFirst experience of the power of creating a piece of OSS. Rails had scaffolding but it wasn’t production ready, I had grown tired of producing the same code over and over when building Perl administration systems for various ISP’s and other businesses. DHH was highly critical of high level components so I knew these wouldn’t appear in Rails!!! Found rrwhite generator. But it produced a lot of code, I wanted something simpler so I wrote a blog post proposing \n\nTo produce\n
- I was playing with the concept when skype went and it was Richard White, we decided to work on it together.\n\nCode got released, started to become quite popular. Even started to earn money off it.\n
- More people became involved and the project became ActiveScaffold, I started to take a less active role in the project as I was working on other startups. Specifically at the start of 2008 on a company called Njoyed. \n
- The project is still going though, to date\n
- 2008, out trying to figure out what to do with Njoyed, was raising viable? did I even care? who did I need to know? What the hell am I doing? \n
- This may sound daft but I found myself asking what I would do if I won the lottery in the UK, I realized would move to SF so why wait. So that’s that then, not leaving. But how to stay. Lunch with Richard. He had a basic beta version of UserVoice up and running, which he had built with Lance (the 3rd guy in ActiveScaffold), however Lance had decided that startup life wasn’t for him and left to start a family with his wife in Walla Walla just outside Washington.\n\nSo that was that, Richard asked if I wanted to join him at UserVoice!With pretty much no thought I said yes, clearly serendipity!\n
- rent paid by slimtimer, unpaid, using savings to fly back and forth to avoid visa issues, not a lot of sleep. \n
- Very much MVP, No real backup, admin in the main app, marketing site in the main app, memcached and basic page caching\n\nRevenue is part of product, we wanted to have billing it just wasn’t built and no saas services available. Previous experience with PayPal had put me off for life. Freemium built by Lance.\n
- \n
- speak softly and with palatable words, you never know when you might have to eat them\n
- speak softly and with palatable words, you never know when you might have to eat them\n
- speak softly and with palatable words, you never know when you might have to eat them\n
- Money becoming an issue, need to raise. We were making money but not enough to matter.\n
- A lot of ‘no’, which actually usually consists of this is very interesting, stay in touch and keep us informed. took a few meetings to realize that!!\n\n2 months on the road, 3G usb sticks in the back of cars, deploying running down the platform, in meetings hoping to have it fixed before the demo. \n
- Eventually met Steve Anderson from Baseline who just said yes, 15 mins. Then Dave McClure did the same and we could hire some other people and get some sleep (well, that was the plan)\n
- May 2009 moved to SF\n\nOffice in the front room, now sharing a flat rather than a room pretty liberating!\n
- \n
- \n
- 24th September 2009, 18 months after launch..... Feeling fairly in control, some might say even a little smug. Things were going well, growing quickly, systems seemed stable, even had time to add Nagios and Munin to monitor things, created an sms system for alerts to provide some pre-emptive warnings, not just Pingdom for you’re down. Unfortunately turns out that wasn’t enough protection. Protection from what, you might ask, well.....\n
- \n
- 4am Monday morning in February 2010. Put our tab on every page with no warning........ Agreed to a time the next week to try again. StudioVZ Germany’s largest social network. 400 million pages view / day. \n
- 4am Monday morning, 1st March 2010, Softlayer, only cloud ISP at the time with low latency between hosted boxes and the cloud slices, no databases in the cloud, freaks people out with privacy issues.\n
- \n
- In short words the problem is in case of insert into table with auto-increment column the special AUTO_INC table level lock is obtained, instead of usual row-level locks. With many concurrent inserted threads this causes serious scalability problems, and in our consulting practice we had a lot of customers who was affected by InnoDB auto-inc. For several of them we even advised to replace auto-inc column by that or another solution.\n
- Continuation of moving anything we could to external services\n
- \n
- \n
- Deleted account, dark corner, forgot to put that on the queue, no one really deleted!!!!\n
- Returned to dedicated boxes. At the mercy of too many influences and still quite pricey for the power. But if you don’t have a sys admin as we are now in the somewhat luxurious position of having, it is outsourcing something that is not your core business\n
- The major events that caused us to rethink where we were at, there were of course others. In fact SXSW 2011, the site fell over as soon as we got on the plane. Not sure what it is about SXSW but its a drama magnet!!\n* We run memcache on the same machines that run rails, this cut down on the number of machines we needed.\n* We have 2 staging environments, enabling our developers to quickly work/test on 2 different branches simultaneously\n* Busiest peak time is ~250/requests per sec.\n* We use chef for 100% of config management across all machines, and our chef-server is internally hosted.\n* We use logstash/graylog2/kibana for log analysis and log searching. while this can be vastly improved, we still use it\n* We use ganglia for system metrics.\n* We use nagios for system monitoring.\nHave CI and a lot of tests\n
- \n
- \n
- \n
- Don’t try to reinvent the wheel. True for companies and software projects, become an expert in your domain and trust in others to do the same. If it proves that further down the line this decision was wrong change things then. Lack of focus is the downfall on many things!!!\n
- \n
- \n
- \n
- \n
- \n
- You have to focus on what you do / the purpose of the solution. Its hard enough to get one thing right!!!\n
- \n
- \n
- \n
- \n
- \n
- may sound silly given the number of systems we have. but each is actually a very simple solution to its own problem. complexity was trying to make things do a job they shouldn’t be. unmaintainable, un-expandable and fragile\n
- \n
- Don’t fix problems you don’t have, wait for things to actually need fixing. You’ll never be able to figure out all the things that will go wrong anyway...... make sure you can rollback!!!!\n
- Quote from Darmesh Shah of HubSpot and I couldn’t agree more, I’ve been involved in stealth projects and they all failed to deliver. The faster you get something out the faster you will start to learn\n
- 37 signals, Rework, saying no is very important, but understanding the issues is vital, shameless plug for UserVoice.\n\nHe did build a faster horse (and didn’t say the quote). The key to good product is there, he had a vision that solved peoples problems in a way they didn’t perceive. Its important to listen to people’s feedback but implement it inside your vision, they may see 4 or 5 separate issues that you can solve with one change. \n\n
- \n
- Try to be a bit more positive. \n
- \n