This presentation gives an introduction to Yahoo Pipes and Yahoo Query Language and shows how to create Data Mashups and Data aggregations using pipes.

1. Data Aggregation
using pipes
by Pratibha Gannavarapu

2. Who am I?
Have been a Technical yahoo for 4 years
Now a Programme Manager
Have done MIT from Sydney University

3. Creating data mashups
What am I doing here?

4. There is abundant data on the web;
In different formats like RSS, CSV, XML, Atom, Json;
displayed as web pages
Web

5. Web

6. Get the data from multiple locations
What is a mash?

7. Process for a purpose
Purpose can be an application
What is a mash?

8. Display the result
What is a mash?

9. Tools to create a Mash
Pipes
Yahoo Query Language

10. Pipes – What is it?
Pipes is a free online service that can be used for remixing
feeds and output in RSS 2.0, RSS 1.0, JSON, Atom.

11. Pipes – How to?
Pipes has a visual editor that makes creating a new
pipe as easy as.. Umm.. falling off a log :p

12. Pipes - editor
canvas
modules
debugger

13. More explanation?
Lets get our hands dirty
Pipes - Example

14. My purpose is my application.
“Get all finance news headlines for Yahoo! Nasdaq
ticker YHOO”
Pipes - Purpose

15. Pipes – Example

16. Pipes - Example

17. Pipes - Example
To achieve my purpose
I need RSS feeds from both the sites and merge them
together
http://www.google.com/finance/company_news?q=N
ASDAQ:YHOO&output=rss
http://finance.yahoo.com/rss/headline?s=YHOO

18. Pipes - Example

19. Pipes - Example

20. Pipes - Example

21. Pipes - Example

22. Pipes - Example

23. Pipes - Example

24. Pipes - Example

25. Pipes - Example

26. Pipes - Example

27. Pipes - Example

28. Pipes - anatomy
Each pipe consists of 2 or more modules and each
module has one or more terminals.
We wire the modules together.

29. Pipes - dictionary
Canvas
Modules
Wiring
Input and Output Terminals

30. Pipes – other stuff
Private pipes vs public pipes
Private pipes can be shared using their URLs
Pipes once published are available for search
engines to crawl
Hosted pipes make life much more easier

31. Pipes – what can you do?
Combine 2 or more feeds or take user input and
filter, sort or concatenate them!
You can also execute regular expressions, date
formatting and much much more!

32. Pipes – Engine
Currently pipes is using V1 engine. It will be
upgraded to V2 (Beta) which is also used for YQL.
So if you have a V1 pipe, migrate to V2

33. Pipes – Cool stuff
It is easy to use with its “visual editor”.
Pipes is a life savor in enabling to jumble data
without tumbling into code.
Hosting and SEO are taken care of!! Yay!

34. The other way
YQL

35. YQL - Intro
YQL is more Programmatic, more robust.
SQL-like language that is easy to learn and
understand to run queries.
It returns either XML or JSON

36. YQL – Example Query
select * from flickr.photos.search where
text=”Yahoo" limit 3

37. YQL - Console
rest query
query
results
tables

38. YQL
YQL is a more powerful engine
providing an ability to create open
data tables

39. Resources
http://developer.yahoo.com/pipes
http://developer.yahoo.com/yql

40. Blog – http://stills360.com/blog/
Twitter – @pratiba
Facebook - @pratiba

41. Workshop
Take user input and search for books & following tech
magazines
Google book search base url :
http://books.google.com/books/feeds/volumes
User input should be passed as param q. So the final query
looks as:
http://books.google.com/books/feeds/volumes?q=footbal
l+-soccer&start-index=11&max-results=10
http://feeds.informationweek.com/infoweek/all_stories_b
logs

42. Workshop
Take user input
Use it as param for book search
Get feeds from Magazine
Search for items with user input in title or description
Merge the feeds from both sources
Sort them to show latest ones first