6. Market leading customer feedback &
support solution
21 strong in San Francisco & Raleigh
90,000+ organizations have used
us
$2M+ raised over 2 funding
rounds
Tens of millions of
registered users
182 people/sec see our
product
11. You Can’t Plan For
Everything!
Things are going to go wrong. It’s the way we
react at these moments that defines to a
great degree how successful we will be.
Don’t spend time planning for every
eventuality. Just be prepared to move quickly
and decisively when situations arise. Don’t
build failure in though!!!
14. ActiveScaffold (AKA AjaxScaffold Plugin)
One line of Code
15. ActiveScaffold (AKA AjaxScaffold Plugin)
One line of Code
16. ActiveScaffold (AKA AjaxScaffold Plugin)
One line of Code
17. ActiveScaffold (AKA AjaxScaffold Plugin)
One line of Code
>100K Installs
> 750K Visitors
18.
19.
20.
21. Version 1.0
• Rails 1.2.3 / HopToad / NewRelic /
CloudFront
• Engine Yard
• Shared MySQL database
• 2 slices (and staging)
• Admin section based on ActiveScaffold
• No billing ability until the end of 2008 (then
using Freemium OSS)
24. Building Fast
• Created our own problems
• Insufficient test coverage and no CI
• deploy production:rollback
• Watch those AJAX calls
25. Building Fast
• Created our own problems
• Insufficient test coverage and no CI
• deploy production:rollback
• Watch those AJAX calls
• Be careful when testing.....
26. Building Fast
• Created our own problems
• Insufficient test coverage and no CI
• deploy production:rollback
• Watch those AJAX calls
• Be careful when testing.....
30. Features, Features, Features
• Reporting / Administration Tools
• Multiple Languages (now crowd sourced)
• Search that worked (Sphinx)
• Single Sign On
• API
• Billing (constant PITA)
33. Problems
• Shared database started to creak
• Tables growing in size - queries not scaling
• Overused MySQL - queue, stats
• Not enough caching
• Dark corners - data exports, email
notifications
• Still insufficient test coverage and no CI
• Us...... (production deploy:rollback)
35. Into The Cloud....
• Moved servers to SoftLayer
• Passenger and Ruby Enterprise
• Signed up 4 million new users in 2 hours
• 6 slices expanded to 24
• Stability resumed.......
36. Still Shaky
• Redis / Resque for the queue
• Redis for stats
• More memcached
• Make as many things background jobs as
possible
• Things are under control
• Or so we thought.....
37. Meltdown
• 10 days of almost stability
• Flew to SXSW
• MySQL melted (still on 5.0)
• Percona rescued us from the dreaded
MYSQL AUTO INCREMENT BUG (Moved
us to 5.1)
• Finally have a scalable redundant system,
right?
38. DNS can fail?
• Using ZoneEdit (free)
• All nameservers went down, all hell broke
loose
• Contacted by DynECT over Twitter!
• Gave us a free account to get going again.
Moved to them
• Stopped thinking about DNS!
42. And back to Earth
with a bump
• Softlayer had a few major outages on their
cloud infrastructure
• Slices are still shared resources
• You can’t get someone to just reboot the
box
• Lesson: cloud is great to scale with, not so
good for stability
43. So where are we now?
• 20 systems in production (2 cloud instances)
• Web - 6x dual cpu/quad core 12G RAM nginx/passenger,
memcache
• Resque - 3x single cpu/quad core 8gb RAM
• Redis - Dual CPU/quad core 12g RAM
• Mongo - 2x dual cpu/hex core 48G RAM 12x 15k RPM
SAS drives RAID10
• MySQL - 2x dual cpu/hex core 72G RAM 24x 15k RPM
SAS drives RAID10
• 97% month / month retention
Great talk by Ted Roden, really glad I didn’t use the focus slide. I looked at exactly the same picture as he used, the dangers of going last! \n\nJetlag has seriously kicked in so if I just fall asleep standing here can someone throw one of those mics at me\n
How many people are involved in startups / open source projects / code projects or have something in mind or gestating in their heads.\n
music degree\nunfinished\ndistracted by dj’ing and dance music\nwoke up one day (a bit of a theme apparently) and decided to do a physics degree which led to the amazing experience of doing a PhD\n
CERN, situated half in Geneva / France. 27km underground tunnel containing the world largest machine the LHC. The purpose of which is to investigate the GUT and the elementary particles of the universe. It also happens to be where Tim Berners Lee created the web.\n\nBecame a little obsessed with programming and computers\n
That led to starting a software company called Axomic, which surprisingly was pretty successful. But lesson 1, try to pick a big market... Architectural image management is not a big pond\n\nFollowed by trying to start or at least build several other things. \n
Those lead up to my current company and role which is as COO and co-founder of UserVoice.\n
The observant amongst you, will of noticed that this was the original title of this talk, not a startup conference so I tried to make it more of a mix \n
Perhaps over embraced in the US, not embraced enough elsewhere. \n
\n
\n
\n
November 1940, Tacoma Narrows Bridge, state of Washington, route 16. indexes, massive loops\n
Try to be a bit more positive. Two elements to it, one being the just do it concept. If you have an idea, can get involved in open source, do so you never know where it will take you. Secondly, sometimes it seems like there is a mountain of stuff you need to know to do something might be true, but to be honest you can learn everything you need to know on the way, so. Along those lines, this is some of the story of UserVoice.\n
It starts in 2006, been playing around with Rails. \n\nFirst experience of the power of creating a piece of OSS. Rails had scaffolding but it wasn’t production ready, I had grown tired of producing the same code over and over when building Perl administration systems for various ISP’s and other businesses. DHH was highly critical of high level components so I knew these wouldn’t appear in Rails!!! Found rrwhite generator. But it produced a lot of code, I wanted something simpler so I wrote a blog post proposing \n\nTo produce\n
I was playing with the concept when skype went and it was Richard White, we decided to work on it together.\n\nCode got released, started to become quite popular. Even started to earn money off it.\n
More people became involved and the project became ActiveScaffold, I started to take a less active role in the project as I was working on other startups. Specifically at the start of 2008 on a company called Njoyed. \n
The project is still going though, to date\n
2008, out trying to figure out what to do with Njoyed, was raising viable? did I even care? who did I need to know? What the hell am I doing? \n
This may sound daft but I found myself asking what I would do if I won the lottery in the UK, I realized would move to SF so why wait. So that’s that then, not leaving. But how to stay. Lunch with Richard. He had a basic beta version of UserVoice up and running, which he had built with Lance (the 3rd guy in ActiveScaffold), however Lance had decided that startup life wasn’t for him and left to start a family with his wife in Walla Walla just outside Washington.\n\nSo that was that, Richard asked if I wanted to join him at UserVoice!With pretty much no thought I said yes, clearly serendipity!\n
rent paid by slimtimer, unpaid, using savings to fly back and forth to avoid visa issues, not a lot of sleep. \n
Very much MVP, No real backup, admin in the main app, marketing site in the main app, memcached and basic page caching\n\nRevenue is part of product, we wanted to have billing it just wasn’t built and no saas services available. Previous experience with PayPal had put me off for life. Freemium built by Lance.\n
\n
speak softly and with palatable words, you never know when you might have to eat them\n
speak softly and with palatable words, you never know when you might have to eat them\n
speak softly and with palatable words, you never know when you might have to eat them\n
Money becoming an issue, need to raise. We were making money but not enough to matter.\n
A lot of ‘no’, which actually usually consists of this is very interesting, stay in touch and keep us informed. took a few meetings to realize that!!\n\n2 months on the road, 3G usb sticks in the back of cars, deploying running down the platform, in meetings hoping to have it fixed before the demo. \n
Eventually met Steve Anderson from Baseline who just said yes, 15 mins. Then Dave McClure did the same and we could hire some other people and get some sleep (well, that was the plan)\n
May 2009 moved to SF\n\nOffice in the front room, now sharing a flat rather than a room pretty liberating!\n
\n
\n
24th September 2009, 18 months after launch..... Feeling fairly in control, some might say even a little smug. Things were going well, growing quickly, systems seemed stable, even had time to add Nagios and Munin to monitor things, created an sms system for alerts to provide some pre-emptive warnings, not just Pingdom for you’re down. Unfortunately turns out that wasn’t enough protection. Protection from what, you might ask, well.....\n
\n
4am Monday morning in February 2010. Put our tab on every page with no warning........ Agreed to a time the next week to try again. StudioVZ Germany’s largest social network. 400 million pages view / day. \n
4am Monday morning, 1st March 2010, Softlayer, only cloud ISP at the time with low latency between hosted boxes and the cloud slices, no databases in the cloud, freaks people out with privacy issues.\n
\n
In short words the problem is in case of insert into table with auto-increment column the special AUTO_INC table level lock is obtained, instead of usual row-level locks. With many concurrent inserted threads this causes serious scalability problems, and in our consulting practice we had a lot of customers who was affected by InnoDB auto-inc. For several of them we even advised to replace auto-inc column by that or another solution.\n
Continuation of moving anything we could to external services\n
\n
\n
Deleted account, dark corner, forgot to put that on the queue, no one really deleted!!!!\n
Returned to dedicated boxes. At the mercy of too many influences and still quite pricey for the power. But if you don’t have a sys admin as we are now in the somewhat luxurious position of having, it is outsourcing something that is not your core business\n
The major events that caused us to rethink where we were at, there were of course others. In fact SXSW 2011, the site fell over as soon as we got on the plane. Not sure what it is about SXSW but its a drama magnet!!\n* We run memcache on the same machines that run rails, this cut down on the number of machines we needed.\n* We have 2 staging environments, enabling our developers to quickly work/test on 2 different branches simultaneously\n* Busiest peak time is ~250/requests per sec.\n* We use chef for 100% of config management across all machines, and our chef-server is internally hosted.\n* We use logstash/graylog2/kibana for log analysis and log searching. while this can be vastly improved, we still use it\n* We use ganglia for system metrics.\n* We use nagios for system monitoring.\nHave CI and a lot of tests\n
\n
\n
\n
Don’t try to reinvent the wheel. True for companies and software projects, become an expert in your domain and trust in others to do the same. If it proves that further down the line this decision was wrong change things then. Lack of focus is the downfall on many things!!!\n
\n
\n
\n
\n
\n
You have to focus on what you do / the purpose of the solution. Its hard enough to get one thing right!!!\n
\n
\n
\n
\n
\n
may sound silly given the number of systems we have. but each is actually a very simple solution to its own problem. complexity was trying to make things do a job they shouldn’t be. unmaintainable, un-expandable and fragile\n
\n
Don’t fix problems you don’t have, wait for things to actually need fixing. You’ll never be able to figure out all the things that will go wrong anyway...... make sure you can rollback!!!!\n
Quote from Darmesh Shah of HubSpot and I couldn’t agree more, I’ve been involved in stealth projects and they all failed to deliver. The faster you get something out the faster you will start to learn\n
37 signals, Rework, saying no is very important, but understanding the issues is vital, shameless plug for UserVoice.\n\nHe did build a faster horse (and didn’t say the quote). The key to good product is there, he had a vision that solved peoples problems in a way they didn’t perceive. Its important to listen to people’s feedback but implement it inside your vision, they may see 4 or 5 separate issues that you can solve with one change. \n\n