Introduction and tips and tricks to use YQL.

2. HACKING ON
STEROIDS WITH YQL
Saurabh Sahni
YDN Product Guy & Hacker
Twitter: @saurabhsahni

4. Hacking together
systems in 24 hours is
lot of fun.

5. Data manipulation
Hacks = +
Data visualization

6. The web has lot of data around

7. ProgrammableWeb.Com – 6831 APIs

8. Yahoo! has opened
up its data

9. http://developer.yahoo.com/everything.html

10. THE TROUBLE WITH DATA
• You need to find data API
• Get Access – Signup for key
• Find data endpoint
• Read docs to learn what parameters you
have
• Get data in obscure format
• Use data after converting and filtering
• More APIs you use, more is your
annoyance

11. To make data
access easy on the
web, Yahoo!
created YQL

12. YQL turns web
services and data on
the web into
databases.

13. select {what} from {where}
where {conditions}

14. You can select, filter,
sort and limit

15. You can even insert,
update and delete
from it.

16. FINDING VIDEOS ABOUT BANGALORE
SELECT * FROM youtube.search where
query='bangalore'

17. SELECTING PHOTOS OF HACKDAY
SELECT * FROM flickr.photos.search where
text="hackday” and api_key=“b5a60b2a…”

18. INSERTING DATA
INSERT INTO bitly.shorten (login, apiKey, longUrl)
VALUES ('ME', 'API_KEY', 'http://yahoo.com')

19. UPDATING DATA
UPDATE social.profile.status
SET status="Using YQL UPDATE”
WHERE guid="NJFIDHVPVVISDX7UKED2WHU"

20. RETRIEVING MY CONTACTS
SELECT * FROM social.contacts WHERE
guid=me

21. ACCESSING PRIVATE DATA
http://query.yahooapis.com/v1/yql
Uses OAuth 1.0 for authorization
OAuth is complicated – use one of our SDKs at
https://github.com/yahoo

22. You can also mix and
match several web
services using the in()
command.

23. select * from search.termextract
where context in (select
description from rss where
url='http://rss.news.yahoo.com/
rss/topstories')

24. Almost all the top
APIs on web are
accessible from YQL

26. Some
of
them
amazon foursquare peerindex
apple geo salesforce
bbc github slideshare
bible google themovideb
boss hackernews tumblr
campfire ign twitter
contentanalysis intuit vimeo
craigslist kiva weather
delicious klout yahoo
dopplr lastfm youtube
etsy netflix zillow
facebook paypal

27. You want even
more?

28. Alright, how about this?
atom json
csv microformats
feed rss
html xml

29. The easiest way to
start with YQL is to
use the console
http://developer.yahoo.com/yql/
console

30. YQL: http://developer.yahoo.com/yql/console

34. How to get this data
in your app?

35. YQL is a REST API
in itself and has two
endpoints

36. The public endpoint does not need
any authentication.
http://query.yahooapis.com/v1/public/
yql?q={query} &format={format}

37. The private endpoint needs oauth
authentication.
http://query.yahooapis.com/v1/yql?q=
{query}&format={format}

38. Output formats are XML
or JSON

39. LET’S SEE IT

40. QUERY EXAMPLES
select
*
from
yahoo.finance.quotes
where
symbol
in
("^IXIC","^DJI","YHOO
","AAPL")

41. QUERY EXAMPLES
select
*
from
weather.bylocaHon
where
locaHon
in
("bangalore,
in",
”new
york,
us")

42. QUERY EXAMPLES
Find hackday tweets:
SELECT * FROM twitter.search where q='hackday’
Search Yahoo! Answers for resolved questions about cars:
select * from answers.search where query="cars" and type="resolved”
Find distance between Bangalore and Mumbai:
select * from geo.distance where place1="bangalore" and
place2="mumbai”
Extract important terms from top stories on Yahoo! news:
select * from search.termextract where context in (select description
from rss where url='http://rss.news.yahoo.com/rss/topstories')

43. QUERY EXAMPLES
Get Olympic medal list
select * from html where url='http://sports.yahoo.com/olympics/
medals.html' and xpath='//*[@id='mediasportsoverallmedalcount']/div
[2]/table/tbody/tr/td/a'
Shorten a URL:
insert into yahoo.y.ahoo.it (url, keysize) values ('http://
www.javarants.com', 5)
Search apartments in criagslist:
select * from craigslist.search where location="bangalore" and
type="apa" and query="indiranagar”

44. QUERY EXAMPLES
Scrape news from Yahoo! Finance:
select * from html where url="http://finance.yahoo.com/q?
s=yhoo" and xpath='//div[@id="yfi_headlines"]/div[2]/ul/li/a’
Select, filter data from google spreadsheets:
select * from csv where url="https://
spreadsheets.google.com/pub?key=0ArYndzim-
lbrdF8wc3A5QWl1ZGRpdkxRZk80SU9zUXc&output=csv"
and col5 like 'Bangalore%’ ;

45. Let’s find hackday
photos on flickr

46. How about limiting
to those clicked in
Bangalore

48. MAKING REQUESTS: FLICKR URLS
<photo farm="3"
id="5708163920"
isfamily="0"
isfriend="0"
ispublic="1"
owner="31832337@N04"
secret="0075137487"
server="2496"
title="San Francisco"/>

49. MAKING REQUESTS: FLICKR URLS
Photo URL
http://farm{$farm}.static.flickr.com/{$server}/
{$id}_{$secret}.jpg
Photo Page URL
http://www.flickr.com/photos/{$owner}/{$id}
Photo Owner Profile URL
http://www.flickr.com/photos/{$owner}

51. https://github.com/saurabhsahni/Hacks/

52. Finding Recent Photos from flickr

53. Some YQL Hacks

56. ChromYQLip is a
chrome extension for
page scraping via YQL
Open
Hack
Bangalore
2010
Winner

57. http://bit.ly/chromeYQL

58. VIDEO CLIP

59. http://www.webmeme.in

60. WEBMEME.IN
Fetch multiple feeds in different formats like atom, RSS and
transform them into consistent RSS format
Select * from rss where url in (‘http://feeds.feedburner.com/pluggd’,
‘http://quatrainman.blogspot.com/atom.xml’, ‘…’)
Filter news containing “india” from multiple feeds:
select * from rss where url in ('http://feeds.feedburner.com/
TechCrunch', 'http://www.readwriteweb.com/rss.xml','http://
gigaom.com/feed/') and description like '%india%’

61. YQL is open – you
can get your data
tables in our system

62. All you need to do is
write an XML
schema and put it
on Github.

63. http://github.com/yql/yql-tables

64. Here is the craigslist
search table
https://github.com/yql/yql-tables/tree/master/
craigslist/craigslist.search.xml

66. USE INSTANTLY BY UPLOADING ON YOUR
SITE
USE 'http://www.mysite.com/my_table.xml'
AS mytable;
SELECT * FROM mytable
WHERE user='saurabh'

67. You can even write
server side
javascript to build a
webservice or
augment one.
hLp://developer.yahoo.com/yql/guide/yql-­‐execute-­‐chapter.html

68. There are lot of
things you can do
with YQL.

69. Play yourself
http://
developer.yahoo.com/yql/

70. One more thing

72. RESOURCES
All Yahoo! APIs and Services
http://developer.yahoo.com/everything.html
YQL Documentation
http://developer.yahoo.com/yql
YQL Console
http://developer.yahoo.com/yql/console
YQL Github Account (Contribute Tables)
http://github.com/yql/yql-tables

73. THANKS!
http://www.slideshare.net/
saurabhsahni
Saurabh Sahni
Twitter: @saurabhsahni
Github: http://github.com/saurabhsahni
Web: http://www.saurabhsahni.com