Sammy Fung presented on his open source weather projects hk0weather and TCTrack, as well as his views on open data and open government in Hong Kong. He described hk0weather, an open source project to extract weather data from the Hong Kong Observatory website and convert it to JSON format. He discussed challenges with obtaining weather data from the HKO and limitations of weather data provided through the Hong Kong government's open data portal. He also proposed ideas for improving the transparency of legislative council vote result data.

1. 從 hk0weather 到 Open Data, Open Government
From hk0weather to Open Data, Open
Government
Sammy Fung
BarCampHK 2013
http://slidesha.re/11ZPKvx

2. Sammy Fung
● Blogger: sammy.hk
● Coder: perl -> php -> python
● Linux User: server + desktop
● “Cloud”: OpenStack.
● Smartphone: iPhone 3G -> Android
● Organizer: Linux/Open Source Communities

3. Agenda
● My Products: WeatherHK, TCTrack
● My new opensource projects: hk0weather
● My comments to Open Data and Open
Government
● Q&A, Open Discussion to Floor.

4. My Products
● WeatherHK
● TCTrack

5. WeatherHK
● http://twitter.com/weatherhk
● Interview by MetroPop in 2009.

6. WeatherHK
● http://twitter.com/weatherhk
● Backend: Python + Scrapy + Database +
Twitter + NNTP......
● Frontend: Twitter + Newsgroup
● hourly current weather report
● weather forecast report
● tropical signal warning

7. My Products
● WeatherHK
● TCTrack

8. TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php
● http://twitter.com/tctrack
● Plot TC current and forecast tracks over
Google Map.
● Tweet JTWC updates for Northwest Pacific.
● Source: JTWC, HKO.

9. TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php
● http://twitter.com/tctrack
● Probably first tctrack map in HK using
GoogleMap
● Use of GMap: TCTrack -> Weather
Underground Hong Kong -> HKO

10. g0v.tw
● g0v.tw 台灣零時政府
● clkao 高嘉良 suggested to implement open
source weather tools for Hong Kong.

11. hk0weather
● https://github.com/sammyfung/hk0weather
● Open Source Hong Kong Weather Project.
● convert to JSON data from HKO webpages.
● python + scrapy
● 1st version: from current weather report,
extracting temperture and humidity from 20+
weather stations, export in json format.

12. hk0weather
● https://github.com/sammyfung/hk0weather
● $ virtualenv hk0weatherenv
● $ source hk0weatherenv/bin/activate
● $ pip install scrapy
● $ git clone
https://github.com/sammyfung/hk0weather.git
● $ cd hk0weather
● $ scrapy crawl currwx -t json -o testresult

13. hk0weather
[{"humidity": 80, "station": "hko", "temperture": 17, "time": 1360785720},
{"station": "kingspark", "temperture": 16, "time": 1360785720},
{"station": "wongchukhang", "temperture": 17, "time": 1360785720},
{"station": "takwuling", "temperture": 16, "time": 1360785720},
{"station": "laufaushan", "temperture": 15, "time": 1360785720},
{"station": "taipo", "temperture": 16, "time": 1360785720},
{"station": "shatin", "temperture": 17, "time": 1360785720},
{"station": "tuenmun", "temperture": 17, "time": 1360785720},
{"station": "tseungkwano", "temperture": 16, "time": 1360785720},
{"station": "saikung", "temperture": 16, "time": 1360785720},
{"station": "cheungchau", "temperture": 17, "time": 1360785720},
{"station": "cheungchau", "temperture": 17, "time": 1360785720},
{"station": "tsingyi", "temperture": 17, "time": 1360785720},
{"station": "shekkong", "temperture": 15, "time": 1360785720},
{"station": "tsuenwanhokoon", "temperture": 15, "time": 1360785720},
{"station": "tsuenwanshingmunvalley", "temperture": 17, "time": 1360785720},
{"station": "hongkongpark", "temperture": 17, "time": 1360785720},
{"station": "shaukeiwan", "temperture": 16, "time": 1360785720},
{"station": "kowlooncity", "temperture": 16, "time": 1360785720},
{"station": "happyvalley", "temperture": 18, "time": 1360785720},
{"station": "wongtaisin", "temperture": 17, "time": 1360785720},
{"station": "stanley", "temperture": 16, "time": 1360785720},
{"station": "kwuntong", "temperture": 15, "time": 1360785720},
{"station": "shamshuipo", "temperture": 17, "time": 1360785720}]

14. hk0weather
● Future Planning:
● Add more weather reports.
● Getting ideas and/or cooperate with 'pro'
Weather hobbists.
● Remarks:
● Development of hk0weather is started from
ZERO, its code is different than my twitter
@weatherhk.

15. Why python + scrapy ?
● python: my current favourite programming
language for few years.
● scrapy: web crawling framework written in
Python.

16. Scrapy
● scrapy: web crawling framework written in
Python.
● HtmlXPathSelector
● Output: built-in JSON, CSV, XML.
● Python: import re

17. Items.py
class Hk0WeatherItem(Item):
time = Field()
station = Field()
temperture = Field()
humidity = Field()
pass

18. Items.py
class Hk0WeatherItem(Item):
time = Field()
station = Field()
temperture = Field()
humidity = Field()
pass

19. Currwx.py
start_urls = (
'http://www.weather.gov.hk/wxinfo/currwx/curr
entc.htm',
)

20. Currwx.py
def parse(self, response):
laststation = ''
temperture = int()
stations = []
hxs = HtmlXPathSelector(response)
report = hxs.select('//div[@id="ming"]')

21. libhk0
class hk0:
stations = [
(u' 天 文 台 ', 'hko'),
(u' 京 士 柏 ', 'kingspark'),
(u' 黃 竹 坑 ', 'wongchukhang'),
(u' 打 鼓 嶺 ', 'takwuling'),
(u' 流 浮 山 ', 'laufaushan'),

22. libhk0
class hk0:
def gettime(self, report):
…
def hk0current(self, report):
…

23. Challenge
● Challenge on first day of hk0weather release.
● Facebook comment to blog post: 「其實 HKO
有個年費 plan 係嘔兩個好靚仔嘅 XML 。而只
要你肯問，你件嘢又非商業用途， HKO 都會免
費俾你用」

24. Challenge
●
I replied after googling for it.
●
google 找到相關 XML schema 示範資料，但 ....
●
1. 在於 Open Data 角度，暫時看不到有網頁說明任何人以非商
業用途使用資訊，可以免費獲得登入戶口。而我本身是「已登
記」使用 HKO 網頁資料作非商業用途的使用 / 轉載者，我一直收
到來自 HKO 的更新電郵都是說明 HTML 格式的更新， " 印象中 "
HKO 並沒有發出過 XML 和邀請開 XML 使用戶口。
●
2. 而 ( 以我覺得 ) 以業餘氣象發燒友的角度來看，目前 HKO XML
schema 仍有不足之處，我已想到一些我想要的數據，仍是只能
從 HTML 版本取得。
●
Challenge: FAIL! XD

25. Open Data from Govenment
● My Chinese Blog Post 「香港政府機構開放資料
Open Data 情況」 on 2013/1/17.
● Data.One released on 2011/3/31.
● Weather at Data.One provides 7 dataset URLs,
returns RSS (XML) format (Eng/TChi/SChi)
● One word: Useless.
● Data.One dataset is different with HKO own
XML metioned in previous “challenge”.

26. Open Data from Govenment
● Example - Current local weather report:
● Plain text report in RSS.
● Difference to quote report content:
– Website: HTML tags
– Data.One: RSS description tags
● Other weather data is missing, eg. Area
temperture updates per each 12 mins.

27. Open Data from Govenment
● Weather at Data.One is 'report' but not 'data'.
● Weather RSS is already released by HKO
before launch of Data.One.
● Technically, json/xml format is better
readable by computer programs.

28. Open Data from Govenment

29. Improving Legco Vote Result Data ?
● Legcovotes.net is created by Hong Kong
netitizens(?).
● Only 20 famous vote results are included.
● It is possible to let public to input other vote
results by hand, and submissions should be
verified by legcovotes.net authoritative.
● Including other data, eg. Minutes in plain text
or paragraphs related to a counciler.

30. Thank you.
● This presentation will be uploaded to
slideshare.
● Licensed in Creative Commons BY-NC-SA.
● http://slidesha.re/11ZPKvx
● http://sammy.hk