Cinchcast is an innovative new platform that enables large-scale conference calls and webcasts for your business. Our webcasting solution must be able to securely push out slides to tens of thousands of people while synchronizing with the audio stream. In this talk, we will discuss how we used NodeJS and Socket.IO to achieve this goal. Socket.IO has tremendous capabilities to overcome limitations of the web sockets in the browser ecosystem today. However, leveraging and scaling this framework can be both challenging and interesting across multiple servers: How do you keep state? Why do connections begin getting dropped? Is it really secure? We will answer all of these questions in this talk. We will focus on understanding Socket.IO scalability and high availability, and discuss some security pitfalls that you need to watch out for. Aleksandr Yampolskiy is a CTO of Cinchcast, cloud-based conferencing solution for enterprises, and BlogTalkRadio, the world's largest online radio network averaging 36 million unique visitors a month. Prior to joining Cinchcast, Alex was a Head of Security and Compliance at Gilt Groupe companies, building their team from scratch to a team serving over 1300 employees. Before that, he has worked at Goldman Sachs, Oracle, and Microsoft, where he was a lead technologist building large-scale enterprise software focused on IDM, SSO, authentication and authorization. Aleksandr has been cited in New York Times, ComputerWorld, Observer, and other media. He speaks regularly about software development processes and security. He has a B.A. in Mathematics and Computer Science from New York University, and a Ph.D. in Computer Science from Yale University. He is also an organizer of NYC Technology Startup (a group of over 1400 entrepreneurs and developers) and NYC REDIS NOSQL meetups. In his spare time, he enjoys wandering New York museums, playing chess, martial arts, and public speaking. Danny Gershman is a Principal Engineer at Cinchcast. He's spent over 16 years experience developing software and now focusing on R&D. Worked with various technologies, but currently I'm working with .NET, SQL Server, Redis, NodeJS, Socket.IO, jwPlayer. Also teller of bad jokes.

1. Sharing Slides Securely with 10,000 People in Real-
Time : Socket.IO and Node.JS in Production
Aleksandr Yampolskiy and Danny Gershman

2. Who Are We?
• Aleksandr Yampolskiy, CTO @ayampolskiy
(alexyampolskiy@cinchcast.com)
• Previously head of security and compliance for Gilt Groupe companies, in charge of
securing IT infrastructure, secure architecture, PCI/SOX compliance, etc.
• Various leadership roles in Goldman Sachs, Oracle, Microsoft building scalable,
enterprise software for IDM, SSO, AuthN/AuthZ.
• Ph.D. in Distributed Computing
• Hobbies: chess, Edward Hopper, Ray Bradbury, martial arts, lately foosball and
coffee.
• Danny Gershman @dannygnj, Principal Engineer
(dannygershman@cinchcast.com)
• He's spent over 16 years experience developing software and now focusing on R&D.
• Worked with various technologies such .NET, SQL Server, Redis, NodeJS, Socket.IO,
jwPlayer, Liquid Office, Teleform, Ascent Capture, Classic ASP, and GWBASIC and
DOS
• Paintballer, DJ, Cat-Lover, and from New Jersey (hold the applause please)

3. Cinchcast, Inc.
Cinchcast, an enterprise
technology company,
provides a cloud-based
solution for conference
calls and webcasts.
BlogTalkRadio, a
consumer media
company, is the largest
online radio network in
the world.
Patented, Cloud-Based Platform

4. 4
Marketing Events
Earnings/Analyst Calls Executive Communications
Employee Townhalls
Team Meetings Training
All-Hands Meetings
Cinchcast Connect
Enhancing internal and external corporate communications
while significantly reducing associated costs

5. DEMO TIME

6. Challenge
• Security for sensitive conversations
• Real-time update of slides and analytics
• 10,000s or more participants on various devices, including
older browsers
• No browser plugins + Minimal bandwidth (12 MB/hr)

7. How Does It All Work?
NodeJS
server
cluster
Cinchcast SaaS

8. What Do We Use Node.JS+Socket.IO for?
• Keeping track of real-time listeners on the permalink
page
• Pushing the slide notifications to thousands of viewers in
real-time!

9. Node.JS
• Node.JS = Javascript on your server.
• Asynchronous event loop.

10. NodeJS Security Issues
• Perennial input validation issues
– Rulle #1 – validate thy input
– Never assume the input is well-formed. Think like a hacker!
• JSON eval
– JSON.parse(str) vs eval(str)
– var queryData = url.parse(req.url, true).query;
– Eval(“console.log(‘”+queryData.log+”’)”);
– what if I call http://127.0.0.1/?log=1’);var sys=require(‘sys’); var
exec=require(‘child_process’).exec;function puts=….
• An unhandled exception can crash your server

11. Example - XSS

12. Socket.IO
• Socket.IO = Persistent client-server connection, cross-
browser compatible.
handshake
Handshake accepted ,
transports, connection id,
config

13. Socket.IO Security Issues
• Communication in ws:// protocol is unencrypted.
• Don’t trust the client! All origins are allowed by default.
• Have to build your own authentication/authorization
(https://github.com/LearnBoost/socket.io/wiki/Authorizing
)

14. Example – Origin
• Malicious client by Krysztof Kotowicz
(https://github.com/koto/socket_io_client)
• It can handshake with socket.io server, ignore origin
restrictions, handle heartbeats, fuzz messages

15. What’s Different About Node.JS+Socket.IO
Security?
• More code and complexity in Node.JS/Socket.IO apps.
• We now need to review client-side and server-side
code.
• Dynamic, agile development approach results in code
that’s not thoroughly tested
• Complicated UI frameworks may contain their own
subtle security bugs
• New security attacks

16. What’s Different About Web 2.0 Security?
• Web 2.0 has completely new app security threats
– Malicious AJAX code execution
– WSDL scanning and enumeration
– RSS injection
– XML poisoning
– CSRF attacks

17. Relax, it’s not that bad!

18. Web 2.0 Security Reality
• Fundamentals are still the same, for Web 1.0 and Web 2.0, and for
node.js+socket.io apps.
• Multilayered “onion security”.
• None of the “new” attacks appear on OWASP top 10 list of security
bugs.
• In fact, Verizon 2009 data breach report lists top data breach causes
as
- Weak or default passwords
- SQL injection attacks
- Improper access rights
- XSS attacks

19. Our Approach
• Security decisions are based on risk, not just threats and
vulnerabilities (risk = threat*vulnerability*cost).
• Don’t chase hot vulnerabilities of the day. Instead, mitigate top
risks.
• AAA and least privilege principle.
• Heavily based on policy and user education.
• “Onion security” – multiple protections at each layer.
• Achieve “essential”, then worry about “excellent”.
• Be a “how team” instead of a “no team”.
• Build security into the software development lifecycle.

20. What Do We Do To Protect?
HMAC-SHA1 digest authentication based off rooms and
user type. ACLs are applied one authenticated.

21. What Do We Do To Protect?
• Secure Web-sockets

22. Multi-core

23. Multi-core

24. Multi-server
• Wait…how will we share data?

25. Session Data in Socket.IO

26. Of Course
The Greatest Session Store of All Time Is…

27. Sharing Session

28. Redis Store for Socket.IO

29. Storing / Retrieving Data in Session with
Redis

30. This can start getting out of hand

31. Matryoshka Code

32. Async (https://github.com/caolan/async)

33. Eventing Across Nodes (Pub-Sub)

34. Multi-server
Each process gets its own port, then individually exposed via a load balancer
with a virtual IP. Uses Layer 4 level proxying and SSL certificate is on the load
balancer.
192.168.1.100
3001 3002
3003 3004
192.168.1.101
3001 3002
3003 3004
Load Balancer
129.186.73.100

35. Time Sync

36. Failback with Upstart and Monit

37. Gotchas

38. Conclusion
• Security problems may be new but old principles
apply
• Validate thy input
• HMAC-SHA1 digest authentication
• Know the gotchas for multitasking (time sync,
ulimits, data sharing, etc.)
• We will tweet the slides link.
• Talk to us @ayampolskiy or @dannygnj

39. PITCH: Use Us For Large-Scale Conference
Calls
Contact us at http://cinchcast.com/contact/