Collecting web information with open source tools

•

1 j'aime•2,365 vues

Sammy Fung

my lightening talk slide at coscup 2011, taipei

Technologie

Collecting
useful information
from web with
open source tools

First chairman of
Hong Kong Linux User Group

How does
programmers
solve problems
in daily life ?
程式員解決
現實問題的方法 ?

a lot of popular
web sites
running on II$
in Hong Kong.
香港很多大型網站
都是用 II$

Very slow when you're using!
當你在用的時候，就會很慢！

Visiting websites manually,
repeatly for any latest update.
為了追蹤最新消息，
人手重覆重瀏覽同一網站

Will you still addicted to
plurk/twitter without
auto new response/reply
alert ?
如果沒有自動新回應提
示 , 你還會沉迷噗浪
和推特 ?

About Scrapy
written in python

x = HtmlXPathSelector(response)
torrent = TorrentItem()
torrent['url'] = response.url
torrent['name'] = x.select("//h1/text()").extract()

<h1>Hello World</h1>

all of above are
available in
open source!
以上所有的也有
開源軟件

Problem #1
a lot of popular web sites
running on II$
in Hong Kong.

develop a list of
football matches live
on cable tv
做了「電視足球直播時間表」

Problem #2
some web sites doesn't
provide data API.

Alerts of Tropical Cyclones
in Northwest Pacific Ocean
@tctrack @tropicalhk

Path and Forecast of
active tropical cyclone

Let's solve
your own problems with
open source tools.
所以多多利用開源軟件
來解決
你生活上遇到的問題吧

Thank you! 謝謝 !

solving problems with open
source.

Thank you.

Recommandé

Scrapy.for.dummiesChandler Huang

Web scraping 1 2-3 with python + scrapy (Summer BarCampHK 2012 version)Sammy Fung

How to Scrap Any Website's content using ScrapyTutorial of How to scrape (cra...Anton

Crawling the web for fun and profitFederico Feroldi

Taller de Scrapy - Barcelona ActivaDaniel Bertinat

When big data meet python @ COSCUP 2012Jimmy Lai

摘星zenyuhao

Scrapy-101Snehil Verma

Recommandé

Scrapy.for.dummiesChandler Huang

Web scraping 1 2-3 with python + scrapy (Summer BarCampHK 2012 version)Sammy Fung

How to Scrap Any Website's content using ScrapyTutorial of How to scrape (cra...Anton

Crawling the web for fun and profitFederico Feroldi

Taller de Scrapy - Barcelona ActivaDaniel Bertinat

When big data meet python @ COSCUP 2012Jimmy Lai

摘星zenyuhao

Scrapy-101Snehil Verma

Nautral Langauge Processing - Basics / Non Technical Dhruv Gohil

really really really awesome php application with bdd behat and iterfacesGiulio De Donato

OpenSource SmartGrid: Teeters tail-of-possibilities 8dec11Stan Curtis

HTML5 for mobile - DiGi Challenge for ChangeJan Jongboom

Cybercrime and the developer 2021 styleSteve Poole

Use open source software to develop ideas at workSammy Fung

OpenSourceAdeyemi Fowe

Buildinga billionuserloadbalancer may2015-sre-con15europe-shuffPatrick Shuff

Dylan Butler & Oliver Hager - Building a cross platform cryptocurrency appDevCamp Campinas

Creating Open Data with Open Source (beta2)Sammy Fung

Beyond the Hype: 4 Years of Go in ProductionC4Media

Devoxx France 2022: Game Over or Game Changing? Why Software Development May ...Steve Poole

The Poly Pinoy, ReduxMiguel Paraz

NASA Space AppsJoe Kutner

Innovating the other web - #wrocsharp keynote Christian Heilmann

Full stack development with Python and DjangoInstaily Academy

UCLA HACKU'11Gopal Venkatesan

Py4 inf 01-introIshaq Ali

Droidcon thessaloniki 2015Raymond Chenon

Walter apiNicholas Schiller

Python 爬網⾴工具 - Scrapy 介紹Sammy Fung

DevRel - Transform article writing from printing to onlineSammy Fung

Contenu connexe

Similaire à Collecting web information with open source tools

Nautral Langauge Processing - Basics / Non Technical Dhruv Gohil

really really really awesome php application with bdd behat and iterfacesGiulio De Donato

OpenSource SmartGrid: Teeters tail-of-possibilities 8dec11Stan Curtis

HTML5 for mobile - DiGi Challenge for ChangeJan Jongboom

Cybercrime and the developer 2021 styleSteve Poole

Use open source software to develop ideas at workSammy Fung

OpenSourceAdeyemi Fowe

Buildinga billionuserloadbalancer may2015-sre-con15europe-shuffPatrick Shuff

Dylan Butler & Oliver Hager - Building a cross platform cryptocurrency appDevCamp Campinas

Creating Open Data with Open Source (beta2)Sammy Fung

Beyond the Hype: 4 Years of Go in ProductionC4Media

Devoxx France 2022: Game Over or Game Changing? Why Software Development May ...Steve Poole

The Poly Pinoy, ReduxMiguel Paraz

NASA Space AppsJoe Kutner

Innovating the other web - #wrocsharp keynote Christian Heilmann

Full stack development with Python and DjangoInstaily Academy

UCLA HACKU'11Gopal Venkatesan

Py4 inf 01-introIshaq Ali

Droidcon thessaloniki 2015Raymond Chenon

Walter apiNicholas Schiller

Similaire à Collecting web information with open source tools (20)

Nautral Langauge Processing - Basics / Non Technical

really really really awesome php application with bdd behat and iterfaces

OpenSource SmartGrid: Teeters tail-of-possibilities 8dec11

HTML5 for mobile - DiGi Challenge for Change

Cybercrime and the developer 2021 style

Use open source software to develop ideas at work

OpenSource

Buildinga billionuserloadbalancer may2015-sre-con15europe-shuff

Dylan Butler & Oliver Hager - Building a cross platform cryptocurrency app

Creating Open Data with Open Source (beta2)

Beyond the Hype: 4 Years of Go in Production

Devoxx France 2022: Game Over or Game Changing? Why Software Development May ...

The Poly Pinoy, Redux

NASA Space Apps

Innovating the other web - #wrocsharp keynote

Full stack development with Python and Django

UCLA HACKU'11

Py4 inf 01-intro

Droidcon thessaloniki 2015

Walter api

Plus de Sammy Fung

Python 爬網⾴工具 - Scrapy 介紹Sammy Fung

DevRel - Transform article writing from printing to onlineSammy Fung

Introduction to Open Source by opensource.hk (2019 Edition)Sammy Fung

My Open Source Journey - Developer and CommunitySammy Fung

Introduction to development with Django web frameworkSammy Fung

香港中文開源軟件翻譯Sammy Fung

Open Data and Web APISammy Fung

Global Open Source Development 2011-2014 Review and 2015 ForecastSammy Fung

Mozilla - Openness of the WebSammy Fung

Open Source Technology and CommunitySammy Fung

Access Open Data with Open Source Software ToolsSammy Fung

Installation of LAMP Server with Ubuntu 14.10 Server EditionSammy Fung

Software Freedom and Open Source CommunitySammy Fung

Building your own job site with DrupalSammy Fung

Software Freedom and CommunitySammy Fung

Open Source Job BoardSammy Fung

Introduction of Mozilla Hong Kong (COSCUP 2014)Sammy Fung

Introduction of Open Source Job Board with Drupal CMSSammy Fung

Python, web scraping and content management: Scrapy and DjangoSammy Fung

Local Weather Information and GNOME Shell ExtensionSammy Fung

Plus de Sammy Fung (20)

Python 爬網⾴工具 - Scrapy 介紹

DevRel - Transform article writing from printing to online

Introduction to Open Source by opensource.hk (2019 Edition)

My Open Source Journey - Developer and Community

Introduction to development with Django web framework

香港中文開源軟件翻譯

Open Data and Web API

Global Open Source Development 2011-2014 Review and 2015 Forecast

Mozilla - Openness of the Web

Open Source Technology and Community

Access Open Data with Open Source Software Tools

Installation of LAMP Server with Ubuntu 14.10 Server Edition

Software Freedom and Open Source Community

Building your own job site with Drupal

Software Freedom and Community

Open Source Job Board

Introduction of Mozilla Hong Kong (COSCUP 2014)

Introduction of Open Source Job Board with Drupal CMS

Python, web scraping and content management: Scrapy and Django

Local Weather Information and GNOME Shell Extension

Dernier

Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays

ICT role in 21st century education and its challengesrafiqahmad00786416

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Angeliki Cooney

AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin

[BuildWithAI] Introduction to Gemini.pdfSandro Moreira

TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Orbitshub

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra

Introduction to Multilingual Retrieval Augmented Generation (RAG)Zilliz

CNIC Information System with Pakdata Cf In Pakistandanishmna97

WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelDeepika Singh

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

MS Copilot expands with MS Graph connectorsNanddeep Nachan

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays

MINDCTI Revenue Release Quarter One 2024MIND CTI

FWD Group - Insurer Innovation Award 2024The Digital Insurer

DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity

Dernier (20)

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

ICT role in 21st century education and its challenges

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

AWS Community Day CPH - Three problems of Terraform

[BuildWithAI] Introduction to Gemini.pdf

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Introduction to Multilingual Retrieval Augmented Generation (RAG)

CNIC Information System with Pakdata Cf In Pakistan

WSO2's API Vision: Unifying Control, Empowering Developers

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

presentation ICT roal in 21st century education

MS Copilot expands with MS Graph connectors

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

MINDCTI Revenue Release Quarter One 2024

FWD Group - Insurer Innovation Award 2024

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Collecting web information with open source tools

1. Collecting useful information from web with open source tools

2. @sammyfung

3. Hong Kong

4. First chairman of Hong Kong Linux User Group

5. opensource.hk webmaster

6. How does programmers solve problems in daily life ? 程式員解決現實問題的方法 ?

7. Coding! 就是寫程式 !

8. a lot of popular web sites running on II$ in Hong Kong. 香港很多大型網站都是用 II$

9. Very slow when you're using! 當你在用的時候，就會很慢！

10. Visiting websites manually, repeatly for any latest update. 為了追蹤最新消息，人手重覆重瀏覽同一網站

11. Will you still addicted to plurk/twitter without auto new response/reply alert ? 如果沒有自動新回應提示 , 你還會沉迷噗浪和推特 ?

12. What do you need ? 你需要甚麼 ?

13. Regular Expression

14. HTML Parser

15. Web Crawling Framework

16. scrapy.org

17. About Scrapy written in python x = HtmlXPathSelector(response) torrent = TorrentItem() torrent['url'] = response.url torrent['name'] = x.select("//h1/text()").extract() <h1>Hello World</h1>

18. all of above are available in open source! 以上所有的也有開源軟件

19. Problem #1 a lot of popular web sites running on II$ in Hong Kong.

20. develop a list of football matches live on cable tv 做了「電視足球直播時間表」

21. Problem #2 some web sites doesn't provide data API.

22. Hong Kong Weather Info 香港天氣

23. @weatherhk

24. Alerts of Tropical Cyclones in Northwest Pacific Ocean @tctrack @tropicalhk

25. Path and Forecast of active tropical cyclone

26. Let's solve your own problems with open source tools. 所以多多利用開源軟件來解決你生活上遇到的問題吧 Thank you! 謝謝 !

27. solving problems with open source. Thank you.