Buzztterの裏側とその周辺技術

•

18 j'aime•4,274 vues

Yoji Shidara

Technologie

w
w TFtgt DFtgt TFref DFref

w TFtgt
DFtgt

w TFref
DFref

>> t = Time.parse(quot;2007-11-3quot;)
=> Sat Nov 03 00:00:00 +0900 2007

>> Status.count(:conditions=>[quot;created_at
BETWEEN ? AND ?quot;, t, t.tomorrow])
=> 125626

Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples
Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples
Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples
Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples
Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples
Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples
Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples
Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples
Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples
Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples
Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples
Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples
Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples
Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples
Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples
Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples
Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples
Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples
Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples
Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

i j

i, j
j
Ci,j = P (tk−1 |tk )P (tk+1 |tk )
k=i

Ci,j < 0.75
i..j

$count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]$

2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8
2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms)
2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s)
2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms)
2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms)
2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s)

2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english
2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms)
2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s)
2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms)
2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms)
2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Recommandé

Åbning Phdskole Mette Thunøhcseidelin.hum.ku.dk

Twitter4Rでつくるゆるふわ愛されTwitter botYoji Shidara

Sinatraで鼻歌混じりのWeb開発 @OSC2009-DoYoji Shidara

Den Gode Phdansøgning Peter Stray Jørgensenhcseidelin.hum.ku.dk

Erhvervsphd Morten Bovbjerghcseidelin.hum.ku.dk

Sinatraで鼻歌まじりのWeb開発Yoji Shidara

Inside BuzztterYoji Shidara

20100721 BuzztterYoji Shidara

Recommandé

Åbning Phdskole Mette Thunøhcseidelin.hum.ku.dk

Twitter4Rでつくるゆるふわ愛されTwitter botYoji Shidara

Sinatraで鼻歌混じりのWeb開発 @OSC2009-DoYoji Shidara

Den Gode Phdansøgning Peter Stray Jørgensenhcseidelin.hum.ku.dk

Erhvervsphd Morten Bovbjerghcseidelin.hum.ku.dk

Sinatraで鼻歌まじりのWeb開発Yoji Shidara

Inside BuzztterYoji Shidara

20100721 BuzztterYoji Shidara

絵文字Ruby: From Sapporo.rb with Love for Emoji.Yoji Shidara

Jpmobile: Who I Wanna Be And Who I AmYoji Shidara

Building Static Website With Github And JekyllYoji Shidara

From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...Yoji Shidara

The Way We Are Working On Our Website @とちぎRuby会議02Yoji Shidara

SAPICAの利用履歴を可視化するYoji Shidara

Ruby on Rails でつくるアタシ好みの愛され Web サービスYoji Shidara

RubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こうYoji Shidara

ガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobileYoji Shidara

Twitter分散クロールの野望Yoji Shidara

Pluginが広げるRailsの魅力Yoji Shidara

Rubyistからみたsoupcurry.infoYoji Shidara

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski

Key Features Of Token Development (1).pptxLBM Solutions

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Slack Application Development 101 Slidespraypatel2

SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited

SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

Contenu connexe

Plus de Yoji Shidara

絵文字Ruby: From Sapporo.rb with Love for Emoji.Yoji Shidara

Jpmobile: Who I Wanna Be And Who I AmYoji Shidara

Building Static Website With Github And JekyllYoji Shidara

From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...Yoji Shidara

The Way We Are Working On Our Website @とちぎRuby会議02Yoji Shidara

SAPICAの利用履歴を可視化するYoji Shidara

Ruby on Rails でつくるアタシ好みの愛され Web サービスYoji Shidara

RubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こうYoji Shidara

ガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobileYoji Shidara

Twitter分散クロールの野望Yoji Shidara

Pluginが広げるRailsの魅力Yoji Shidara

Rubyistからみたsoupcurry.infoYoji Shidara

Plus de Yoji Shidara (12)

絵文字Ruby: From Sapporo.rb with Love for Emoji.

Jpmobile: Who I Wanna Be And Who I Am

Building Static Website With Github And Jekyll

From Japanese mobile-web world, to Latin-1 developers. (a part of "East Meets...

The Way We Are Working On Our Website @とちぎRuby会議02

SAPICAの利用履歴を可視化する

Ruby on Rails でつくるアタシ好みの愛され Web サービス

RubyKaigi2008弾丸レポート / ガラパゴスに線路を敷こう

ガラパゴスに線路を敷こう: 携帯電話用RailsプラグインJpmobile

Twitter分散クロールの野望

Pluginが広げるRailsの魅力

Rubyistからみたsoupcurry.info

Dernier

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski

Key Features Of Token Development (1).pptxLBM Solutions

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Slack Application Development 101 Slidespraypatel2

SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited

SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh

How to convert PDF to text with Nanonetsnaman860154

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Install Stable Diffusion in windows machinePadma Pradeep

Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4

Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software

Dernier (20)

08448380779 Call Girls In Civil Lines Women Seeking Men

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...

Key Features Of Token Development (1).pptx

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Slack Application Development 101 Slides

SIEMENS: RAPUNZEL – A Tale About Knowledge Graph

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365

SQL Database Design For Developers at php[tek] 2024

Salesforce Community Group Quito, Salesforce 101

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

IAC 2024 - IA Fast Track to Search Focused AI Solutions

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi

How to convert PDF to text with Nanonets

Swan(sea) Song – personal research during my six years at Swansea ... and bey...

My Hashitalk Indonesia April 2024 Presentation

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

Install Stable Diffusion in windows machine

Azure Monitor & Application Insight to monitor Infrastructure & Application

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation

Buzztterの裏側とその周辺技術

8. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

10.

11.

12. >> t = Time.parse(quot;2007-11-3quot;) => Sat Nov 03 00:00:00 +0900 2007 >> Status.count(:conditions=>[quot;created_at BETWEEN ? AND ?quot;, t, t.tomorrow]) => 125626

13.

14.

15.

16.

17. Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

18.

19.

20. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

21. k

22.

23.

24. i j i, j j Ci,j = P (tk−1 |tk )P (tk+1 |tk ) k=i Ci,j < 0.75 i..j

25.

26.

27. count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]

28. 2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8 2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms) 2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s) 2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms) 2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms) 2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s) 2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english 2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms) 2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s) 2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms) 2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms) 2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)