1. Building a platform from open source at Yahoo!
Dustin Whittle – Yahoo! Developer Network
Building a platform from open source at Yahoo! | Dus$n Whi*le
3. Overview
• Why symfony?
• symfony vs ysymfony
• What does scaling really mean?
• Social Search: Delicious and Answers
• Yahoo! Open Strategy
– What is the Yahoo! Open Stack?
– Application Platform + Developer Tool
• Developer Tools
– YUI, Design Patterns, Tutorials
• Data (YQL) & Social APIs
– YQL, Geo, Profiles, Connections, Updates, …
– YOS SDK for PHP
• Building an open application with symfony and YOS
– OAuth / YQL / OpenSocial
Building a platform from open source at Yahoo! | Dus$n Whi*le
4. Who am I?
• Working with symfony since open source
• symfony Core Team Member
• Responsible for the development and support of
symfony at Yahoo!
– Worked with Y! Answers, Delicious, Y! Widgets, Y!
Bookmarks, Yahoo! Application Platform
• Consultant
– Commercial symfony support + training (USA)
Building a platform from open source at Yahoo! | Dus$n Whi*le
5. EXAMPLES | TUTORIALS | CODE SAMPLES
DEVELOPER.YAHOO.COM
Building a platform from open source at Yahoo! | Dus$n Whi*le
6. FREEBSD | LINUX | APACHE | PHP | MYSQL | BUGZILLA | HADOOP | SYMFONY
YAHOO! IS POWERED BY OPEN SOURCE
TECHNOLOGIES
Building a platform from open source at Yahoo! | Dus$n Whi*le
7. W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL
YAHOO! EMBRACES OPEN STANDARDS
Building a platform from open source at Yahoo! | Dus$n Whi*le
8. RASMUS LERDORF | DOUG CROCKFORD | DOUG CUTTING | CHRISTIAN HEILMANN
YAHOO! HIRES OPEN SOURCE DEVELOPERS
Building a platform from open source at Yahoo! | Dus$n Whi*le
9. YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES
YAHOO! GIVES BACK TO OPEN SOURCE
Building a platform from open source at Yahoo! | Dus$n Whi*le
10. YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING |
HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER | MUSIC |
ANSWERS | SHOPPING | FINANCE | TRAVEL
YAHOO! SHARES ITS DATA THROUGH
OPEN APIS AND WEB SERVICES
Building a platform from open source at Yahoo! | Dus$n Whi*le
11. Conferences | Hack Days | HackU | Tech Talks | YDN Theater
YAHOO! ENGAGES COMMUNITIES WITH OPEN
HACK EVENTS AROUND THE WORLD
Building a platform from open source at Yahoo! | Dus$n Whi*le
13. Users
Load Balancers
Frontend
Apache PHP ysymfony / YUI
Custom Modules APC, PEAR, PECL, Custom Extensions
Backend
MySQL/Oracle Web Services Ad API User API
FreeBSD 4.x/6.x, Linux 2.6.x
Building a platform from open source at Yahoo! | Dus$n Whi*le
14. Y! needs from a frontend platfrom
• Fit existing environment (RHEL/PHP5/Apache)
• Development Cycle – How easy to develop, test, and
deploy?
• Clean separation between data, logic, and display (MVC)
• Independent model layer to fit service oriented
architecture
• Extensible and pluggable
• Internationalization and localization support
• Detailed documentation and active community of support
• Open source and ability to contribute back
Building a platform from open source at Yahoo! | Dus$n Whi*le
15. Why a frontend presentation platform?
• Rasmus says “frameworks are not well suited for Y!”
– Build applications to requirements
• Do exactly what you need: no more, no less
• Understand that frameworks add a lot of overhead
• Choosing functional components is a better fit
• Despite choosing open source or building your own
– Everyone uses a framework
– If you use open source, use only the pieces you need
Building a platform from open source at Yahoo! | Dus$n Whi*le
16. Why a framework at all?
• Another software layer (ysymfony, yphp, yapache)
• Factors out common patterns
– Code Layout
– Configuration
– URL Routing
– Authentication / Security
– Form Validation / Repopulation
– Internationalization / Localization
• Encourages good design
• Abstraction > Consistency > Maintainability
Building a platform from open source at Yahoo! | Dus$n Whi*le
17. The choice to adopt symfony?
• Philosophy
– Full-stack framework for building complex web applications
– Adopt best ideas from anywhere, using existing code if
available (Mojavi, Prado, Rails, Django)
• Design
– Clean separation between Model, View, and Controller
– Controller using modules and actions
– Views using templates in straight PHP with helpers
– Easy to reuse view modules to compose a page
• Layouts, Components, Partials, Slots
Building a platform from open source at Yahoo! | Dus$n Whi*le
18. The choice to use symfony
• Configurability / Flexibility
• Features we do not want are easily disabled
• Use of factories for easy customization
• Documentation / Support Community
• The Definitive Guide to symfony (free online)
• Excellent tutorials and example applications
– Askeet & Jobeet
• Active community with wiki, mailing lists, forums, irc
channel
Building a platform from open source at Yahoo! | Dus$n Whi*le
19. Why ysymfony for Yahoo! teams?
• Eliminate common patterns by adding a layer on PHP
– Code layout/structure (MVC)
– Configuration
– Internationalization
• ysymfony is just a toolkit
– Learn one set of tools
• Shift between multiple projects
• Consistency
– Long term maintainability through platform
Building a platform from open source at Yahoo! | Dus$n Whi*le
25. A look at Yahoo! Answers
• http://answers.yahoo.com
• Yahoo! Answers is the largest collection of human
knowledge on the Web with more than 135 million
users and 515 million answers worldwide (Yahoo!
Internal Data, March 2008).
• Yahoo! Answers is the 2nd ranked education &
reference site on the web (comScore)
• Available in 26 markets and 12 languages
Building a platform from open source at Yahoo! | Dus$n Whi*le
26. Yahoo! Answers at the beginning
• Started as a small development team on PHP4 from a
fork of Yahoo! Taiwan Knowledge+
• Launched December 2005 by December 2006 there
were 60 million users and 65 million answers
• The code base eventually became difficult to maintain
and iterate new features
• Large distributed development teams (US / UK)
Building a platform from open source at Yahoo! | Dus$n Whi*le
34. The big picture
• A complete platform for building web applications
from frameworks
– PHP Framework
– JavaScript Framework
– CSS Framework
– UI Design Patterns + Best Practices
– Development Tools (logger, profiler, debugger, docs)
– Unit + Functional Testing Frameworks (LIME / YUI Test)
– Deployment Tools (rsync deployment system)
Building a platform from open source at Yahoo! | Dus$n Whi*le
35. What does Yahoo! change?
• Minor changes to fit our environment (bsd/php/apache)
– Most of our changes are easily implemented via factories
• Added dimensions to configurations
(ysfDimensionsPlugin)
• Integrate R3 translation/template management
(ysfR3Plugin)
– R3 - http://developer.yahoo.com/r3/
• Dropped the ORM and pushed down the stack (SOA)
– Added a parallel API Dispatcher (ysfAPIClientPlugin)
• Created a build and deployment solution (ysfBuildPlugin)
• Integrate support for Y! User Interface libraries
(ysfYUIPlugin)
Building a platform from open source at Yahoo! | Dus$n Whi*le
36. Propel or Doctrine or ???
• No ORM for large projects
• Propel or Doctrine for large projects
– Doctrine for internal projects (best supported)
• Service Oriented Architecture
– Platforms as services (reusable to all)
– No heavy lifting, push down the stack
– Thin Controller/Fat Model (where model == services)
• Java/C++/Erlang + JSON/XML
Building a platform from open source at Yahoo! | Dus$n Whi*le
37. Localizing the experience with dimensions
• Cascading Configuration based on YAML
– Framework -> Project -> Application -> Module
• Extending the cascade to be based on dimensions
– Dimensions can be anything (and can be chained together)
• Data Center + Environment for customizing configurations
• Culture for localizing user interface + data
• Theme for customizing look and feel
• User info (is user on corporate intranet?)
• Caching
Building a platform from open source at Yahoo! | Dus$n Whi*le
38. symfony I18n + L10n
• Translation + Template Management
– Integrated with symfony i18n + view layers
• <r3:trans>hello</r3:trans> vs __(‘hello’)
• Dictionary Management (YALA)
– SQLite/XLIFF Backend
• Added tasks
– Importing/Exporting XLIFF dictionary
– Generating translations for projects
– Find missing strings
Building a platform from open source at Yahoo! | Dus$n Whi*le
39. A build and deployment system
• Aggregate and minify stylesheets and javascripts
• Rewrite templates, css, js for CDN (Akamai, S3, …)
• Generate translations for configurations + templates
• Generate configuration cache
• Aggregate core classes + remove debug statements
• Run lint, unit, functional tests
• Package applications as .tgz
• Deployment via rsync
Building a platform from open source at Yahoo! | Dus$n Whi*le
40. What does it mean to scale?
• A system whose performance improves after adding
hardware, proportionally to the capacity added, is said to
be a scalable system.
• High Availability + Scalability + Performance
• Bigger dataset, more traffic, maintainable
• Not about performance
– PHP is slow, but it is not your bottleneck
• Languages do not scale, architectures do.
• Planning to grow and planning to fail
– Capacity Planning
– Business Continuity Planning
Building a platform from open source at Yahoo! | Dus$n Whi*le
41. Scaling – Planning
• Planning hardware purchases and hosting options to
have as much as you need without breaking your
wallet
• Partitioning and distributing databases to support
large datasets and simultaneous transactions
• Monitoring your applications to find and clear
bottlenecks
• Providing services APIs and using services from other
providers to increase your site's reach and capabilities
• Think Minimal, Plan to grow, Plan to fail.
Building a platform from open source at Yahoo! | Dus$n Whi*le
42. Scaling – The basics in PHP
• PHP is rarely the bottleneck
• “Most performance comes not from the language, but from
application design” - Rasmus
• Share Nothing Architecture
– Independent, self-sufficient, no single point of contention
– No local storage = No PHP Sessions
• Use a database (works for distributed)
• Use a small signed cookie (ideal)
– Important data in database
– Individual expiration on session objects
– Small data items
– Use a distributed cache
• Memcache
• Forget about small efficiencies -- Premature optimization is the
root of all evil.
Building a platform from open source at Yahoo! | Dus$n Whi*le
43. Scaling Databases – The basics
• Master/Slave Replication
– First steps
– Helps with reads, writes are still bottleneck
• Partitioning
– Segmenting data
• Sharding (horizontal partitioning)
– Segmenting data onto different physical machines
– Make problems smaller, easier to grow
Building a platform from open source at Yahoo! | Dus$n Whi*le
44. Improving latency with Caching
• Always use PHP opcode cache (APC, Xcache, etc)
– Use for routing and i18n cache
• Memcache (distributed cache)
– Use for view cache
• Distributed invalidation can be a pain
• sfViewCacheManager makes this easy!
• Be intelligent about cache_keys (uri, user, state)
• There is a fine line to caching
– At what point do you spend more time managing the cache,
than reading from it?
Building a platform from open source at Yahoo! | Dus$n Whi*le
45. Tweaking Performance
• Don’t use features you do not need
– settings.yml / factories.yml
• Use core_compile (aggregate classes)
• Remove debug statements (sfOptimizerPlugin)
• Do not use .htaccess (move to real apache config)
• Set a minimal include path
• Increase realpath_cache_size + realpath_cache_ttl
• Use apc.stat=0
• Use @routeName
• Do not use components in loop
Building a platform from open source at Yahoo! | Dus$n Whi*le
46. Do it yourself for cheap
• Open source software = Free
– Apache
– PHP
– MySQL
– Memcache / Perlbal / MogileFS / Squid / Gearman
– symfony / Doctrine / Propel / Swift
– Nagios
• Amazon Shared Infrastructure = Cheap
– EC2 Cloud Computing
– S3 Distributed Storage
– SimpleDB
Building a platform from open source at Yahoo! | Dus$n Whi*le
47. Yahoo! Open Strategy
Building a platform from open source at Yahoo! | Dus$n Whi*le
48. OPENID | XRDS | OAUTH | PORTABLE CONTACTS | OPEN SOCIAL
OPEN PLATFORMS + COLLABORATION
Building a platform from open source at Yahoo! | Dus$n Whi*le
49. The Open Web
Building a platform from open source at Yahoo! | Dus$n Whi*le
50. Y! OS – The Open Stack
Building a platform from open source at Yahoo! | Dus$n Whi*le
51. What does Yahoo! offer developers?
• Yahoo! Developer Network
• Developer Tools (YUI, etc)
• Social APIs
• Profiles
• ConnecWons
• Updates
• Data APIs
• OAuth
• Yahoo! Query Language
• Yahoo! ApplicaWon PlaYorm
• OpenSocial
Building a platform from open source at Yahoo! | Dus$n Whi*le
53. developer.yahoo.com
Building a platform from open source at Yahoo! | Dus$n Whi*le
54. What is Yahoo! Developer Network?
The Yahoo! Developer Network offers open source tools and
open data APIs to make it easy for developers to build
applications and mashups.
• 50+ APIs / Web Services
– Developer Dashboard
– Tutorials + Code Samples
– API Documentation
• Yahoo! User Interface libraries + ASTRA
• Design Patterns Library
• PHP / Python / Ruby Dev Centers
• Evangelism: Conferences / Theater / Blogs / Events
Building a platform from open source at Yahoo! | Dus$n Whi*le
56. Y! Developer Network – YUI Javascript
• JavaScript Framework
– Utilities - YAHOO, Dom, Event, Animation, Browser History
Manager, Connection Manager, Cookie, DataSource, Drag and
Drop, Element, Get, ImageLoader, JSON, Resize, Selector,
Loader
– Controls / Widgets - AutoComplete, Button, Calendar, Charts,
Color Picker, DataTable, ImageCropper, Rich Text Editor,
Slider, Uploader
– Container (Module, Overlay, Panel, Tooltip, Dialog), Layout
Manager, Menu, TabView, TreeView
– Debug – Logger, Profiler, Test
Building a platform from open source at Yahoo! | Dus$n Whi*le
57. Y! Developer Network – YUI CSS
• CSS Foundation
– Reset - Neutralizes browser CSS styles
– Base - Applies consistent style foundation
– Fonts - Foundation for typography and font-sizing
– Grids - Thousands of wireframe layouts
• User Interface Design Patterns Library
– Proven solutions to common interfaces
– http://developer.yahoo.com/ypatterns/
– Grade Browser Support / Progressive Enhancement
Building a platform from open source at Yahoo! | Dus$n Whi*le
58. Y! Developer Network – Documentation
• More than 275 functional examples
– http://developer.yahoo.com/yui/examples/
• YSlow + Performance Rules
– http://developer.yahoo.com/performance
• YUI Blog
– http://yuiblog.com/
• Mailing List @ Yahoo! Groups
– http://tech.groups.yahoo.com/group/ydn-javascript/
Building a platform from open source at Yahoo! | Dus$n Whi*le
59. A Query Language for the Web
SELECT * FROM INTERNET
Building a platform from open source at Yahoo! | Dus$n Whi*le
60. Before YQL
• Thousands of Web Services that provide valuable data
• Require developers to read documentation and form
URLs/queries.
• Data is isolated
• Needs combining, tweaking, shaping even after it gets
to the developer.
Building a platform from open source at Yahoo! | Dus$n Whi*le
61. Y! Open Stack – YQL
• SQL-Like Language
– Synonymous with Data access
– Familiar to developers
– Expressive enough to get the right data
• Self Describing - show, desc table
• Allows you to query, filter and join data across Web
Services.
Building a platform from open source at Yahoo! | Dus$n Whi*le
63. YQL – Open Tables
• Twitter • Delicious
• Weather • Dopplr
• Wesabe • Friendfeed
• Whitepages • Github
• Zillow • New York Times
• …. • Shopping
Building a platform from open source at Yahoo! | Dus$n Whi*le
Available on github ‐ h[p://github.com/spullara/yql‐tables/
64. YQL – Open Tables
Building a platform from open source at Yahoo! | Dus$n Whi*le
65. YQL – Open Tables
Building a platform from open source at Yahoo! | Dus$n Whi*le
66. YQL – Open Tables
Building a platform from open source at Yahoo! | Dus$n Whi*le
67. YQL - Examples
• select * from social.connections
• select * from delicious.feeds.popular
• select * from flickr.photos.interestingness
• select * from friendfeed.status
• select * from github.checkins
Building a platform from open source at Yahoo! | Dus$n Whi*le
68. YQL – Javascript Execute
• Allows executing of javascript on the server side to
mashup (join, filter, etc)
• Makes combining many web services very simple
• Support for Oauth
Building a platform from open source at Yahoo! | Dus$n Whi*le
69. YQL – API End Points
OAuth Endpoint
• http://query.yahooapis.com/v1/yql?q=...
Public Endpoint
• http://query.yahooapis.com/v1/public/yql?q=
YQL Console
• http://developer.yahoo.com/yql/console
Building a platform from open source at Yahoo! | Dus$n Whi*le
70. Y! Open Stack – Application Platform
• Allows developers to deploy their own web based
applications on Yahoo!
• Multiple Views: Small and Canvas
• Social Context: the new Yahoo! Social Directory
• OpenSocial 0.8 Javascript APIs
Building a platform from open source at Yahoo! | Dus$n Whi*le
71. Y! Open Stack – Open Applications
Building a platform from open source at Yahoo! | Dus$n Whi*le
75. Y! Open Stack – SDKs
PHP SDK Available
Open and OAuth Applica$ons
Ac$onScript 3 SDK Available
Open Applica$ons
Objec$veC SDK Available
Open and OAuth Applica$ons
Building a platform from open source at Yahoo! | Dus$n Whi*le
77. YOSSDK – 2-Legged OAuth
Used For:
‐ Public user data and open APIs
Building a platform from open source at Yahoo! | Dus$n Whi*le
78. YOSSDK – 3-Legged OAuth
Used For:
‐ Private data access
Building a platform from open source at Yahoo! | Dus$n Whi*le
79. Building an Open App : CommonGround
Experience: Basic Web Dev Knowledge – PHP/HTML/CSS/
Javascript
What we are building: Common ground – Find out what
you have in common with your social graph: music,
movies, books, hobbies.
What we will use: YOSSDK, YQL, YAP
Building a platform from open source at Yahoo! | Dus$n Whi*le
87. CommonGround available on GitHub
http://github.com/dwhittle/commonground
Building a platform from open source at Yahoo! | Dus$n Whi*le
88. QUESTIONS?
(I will be around for the whole event, so please do say hello and ask questions)
Building a platform from open source at Yahoo! | Dus$n Whi*le
89. WANT TO JOIN YAHOO?
WE ARE HIRING AND HAVE INTERNSHIPS!
Building a platform from open source at Yahoo! | Dus$n Whi*le
90. ENJOY THE REST OF SYMFONYLIVE 2009
EXAMPLES | TUTORIALS | CODE SAMPLES
DEVELOPER.YAHOO.COM
Building a platform from open source at Yahoo! | Dus$n Whi*le
91. Books worth reading
Building a platform from open source at Yahoo! | Dus$n Whi*le