Talk from PerlMova 2012 and YAPC::Russia "May Perl"

1. Developing apps
using Perl
Anatoly Sharifulin
YAPC::Russia 2012

2. Hello, World!

3. «Happy Perl developer»
about me by Andy Shitov

4. I program in Perl less
But it doesn't mean I love it less :-)

5. JFDI became JFTI
Just F*cking Talk about It

6. Success-story
of one app
And how Perl helped me/us with it

7. By us I mean Applifto

8. www.applifto.com

9. Our applications

10. Audience polling

11. DLTTR.com

12. «Why?» and «For what?»
in details
http://www.slideshare.net/sharifulin/ss-12313103

13. A powerful reason
for me
I was moving from LiveJournal to Posterous,
used transfer scripts and changed Twitter and Facebook
crossposting settings when started to get replies like
'Tolya! Stop it! Enough!'

14. Current analogues
are nice... not!
Slow, awkward, plain, trouble working:
TwitWipe, TwitCide, Delete My Tweets, ...

15. «Stop managering,
let’s start programming»
And have a talk at #yr2012 :-)

16. Approach and solutions

17. Main aims for me
• App must delete tweets quickly
• Appropriate deleting
• Cross-platform app: iOS, Android, Web, ...

18. Server API
— hello, Perl!
All logic — on our server API (not Twitter API),
clients is simple interface

19. Getting to know
Twitter API

20. Little experience:
• Getting user profiles after login through
Twitter on websites
• Auto-posting to Twitter with
Net::Twitter(::Lite)
• AnyEvent::Twitter::Stream for real-time
search (hey, @miyagawa!)

21. Blocking way
Task can be solved with Net::Twitter(::Lite)

22. LWP::UserAgent::POE
Transport may be changed in Net::Twitter(::Lite)

23. NO

24. Idea — 2 separate
asynchronous queues
Get timeline and delete tweets using REST Twitter API
and asynchronous requests

25. This has to work fast!

26. Authorization and OAuth

27. Net::OAuth::All
We've written our OAuth module long time ago,
supports all OAuth versions (1.0, 1.0A, 2.0)

28. Net::OAuth::All
https://github.com/likhatskiy/Net-OAuth-All

29. my $oauth = Net::OAuth::All->new(
consumer_secret => $conf->{consumer_secret},
consumer_key => $conf->{consumer_key },
token => $data->{access_token },
token_secret => $data->{access_token_secret},
);

30. $oauth
->via('GET')
->protected_resource_url('https://api.twiiter.com/...’)
->put_extra(
include_rts => 'true',
user_id => $user->{id},
count => 200,
)
->request('protected_resource')
;
$oauth->to_header;
$oauth->url; # $oauth->url_with_extra;

31. Patched the module
Unauthorized user:
limit of 150 requests per hour from one IP,
not 350 requests per hour from one user

32. Asynchronous requests

33. I use Mojolicious
Not a secret at all :-)

34. Mojo::UserAgent
Mojo::IOLoop
Good asynchronous HTTP client

35. my $delay = Mojo::IOLoop->delay;
for (@$tasks) {
...
$delay->begin;
$ua->get($url => {'Authorization' => $h} => sub {
my ($ua, $tx) = @_;
...
$delay->end( ... );
});
}
say $delay->wait;

36. Server API

37. Server API
API supports GET and POST requests,
JSON transmission format,
response codes 200 and 50x.

38. Server API
Starman + Mojolicious
Mojolicious::Plugin::ApiHelpers

39. Server API
http://api.dlttr.com

40. package App::Api;
use App::Base -controller,
with => [ 'App::Task', 'App::User' ];
sub error { shift->api_error(code => 1) }
sub any { shift->api_error(code => 5) }
...
1;

41. my $api = $r->route('/api')->to('api#', api => 1);
$api->route('/:action', action => qr/login|oauth/)->to;
my $apiu = $api->bridge->to('#auth', sign => 1);
$apiu->bridge('/task/new')->to('#task_check')
->route->to('#task_new');
$api->bridge('/')->to('#auth', maybe => 1, sign => 0)
->route->to('#hello');

42. 10 methods,123 tests
and documentation
Most methods require user authorization
and check sign (api_id + secret)

43. Test::Mojo
Test::More
Testing API methods access,
input-output data, signature check and etc.

44. Test::Mojo
Test::More
A real task from a test user

45. All methods
are covered
Without Mock-objects, no check
if a correct cluster of tweets have been deleted

46. Too laborious
and needless
A real cycle cannot be tested:
posting tweets – deleting – check

47. $t->get_ok("$url/")
->status_is(200)
->header_is('Access-Control-Allow-Origin', '*')
->json_content_is({hello => 'Hello DLTTR!'})
;
$t->get_ok(sign get => "$url/task/new")
->status_is(200)
->json_content_is({error => {msg =>
'User authorization failed', code => 2}})
;

48. Too risky to test on
your own account :-)
— You don't care about you Twitter acc, do you?
— Yep!
— So I can delete all your tweets?
— Nooooooooooooooo!!!

49. Simple
debug reguests
To check the quality of client work requests/responses
with before_dispatch + after_dispatch may be used

50. Three queues

51. 1. Queue for timelines
Search and filters on tweets ID to delete,
paging with since_id and max_id

52. 1. Queue for timelines
Max. 200 tweets per request,
getting timeline one by one per user,
Rate Limit check

53. «Limits» by Twitter
No access to all tweets,
only the latest ~3200 or starting from any exact date,
counter is not on zero and etc.

54. 2. Queue for deleting
tweets
Deleting with 'clusters' of 200 tweets, no limits.
«Buster» or «Cannon» :-)

55. 2. Queue for deleting
tweets
Error processing — Twitter API error
doesn't always mean a failure

56. 3. Queue for
Push notify
On iOS
Net::APNS::Persistent (also AnyEvent::APNS),
to check if tokens are valid — Net::APNS::Feedback

57. 3. Queue for
Push notify
On Android
WWW::Google::C2DM and WWW::Google::ClientLogin

58. Database

59. So easy — mysql
Storing and task orders,
just to relax :-)

60. 2 finer points:

61. 1. Create correct index
For queues

62. 2. Disable request
caching
select SQL_NO_CACHE * from task where ...

63. Site and web-version

64. www.dlttr.com

65. www.dlttr.com

66. Site and web-version
• Starman + Mojolicious
• Mojolicious::Plugin::I18N2
— my fork of I18N plugin
• Mojolicious::Plugin::OAuth
— login through Net::OAuth::All
• Mojolicious::Plugin::ShareHelpers
• Mojolicious::Plugin::UtilHelpers

67. Admin and statistics

68. «Big Brother»
All details on users, number of tasks, feedbacks
and real-time statistics

69. Statistics on users

71. Users wall

72. «Ordinary» apps don't
have statistics like that

73. 1M+ tweets deleted,
3K+ users,
20% make purchase
For 2 months

74. Promotion

75. Promotion on Twitter
Following, favorites, replies and search —
Net::Twitter(::Lite)

76. Promotion on Twitter
But they blocked me three times already :-)

77. Summary

78. Positive
• Easy solution using pure Perl
(work time is about 30 hours)
• Works fast, really fast
• Right delete tweets
• Any changes, updates, new services
— everything's on the server, no hard
• Real-time statistics

79. Only one negative:
Server's down — the app doesn't work,
users become mad, low rates and spell curses :-)

80. Check out DLTTR!
http://dlttr.com/app http://dlttr.com/android
www.dlttr.com

82. «Seems like a simple app
but in fact it's awesome
and in Perl»

83. P. S.

84. At #BarCampKrr
I had a talk
«Developing app
from idea to release»
(Perl wasn't mentioned)

85. But everything came
down to... Perl
«What language do you use?»,
«But why?»,
«Is it difficult to find programmers like those?» :-)

86. use Perl or die;

87. Thanks!
Anatoly Sharifulin
YAPC::Russia 2012