A quick

1. (Ab)Using the
MetaCPAN API for
Fun and Profit
Olaf Alders (OALDERS)
@wundercounter

2. Architecture
• Built on ElasticSearch
• Uses Catalyst as a thin wrapper
• You don’t need to know this

3. Real life examples

4. iCPAN - iPhone

5. iCPAN - iPad

6. Android

17. What can we build?

18. What can we build?
• Do something with Github

19. What can we build?
• Do something with Github
• Get a list of all CPAN authors who
have enabled the “hireable” flag in
their Github profiles

20. Let’s Get Started

21. Let’s Get Started
• We want to fetch some data

22. Let’s Get Started
• We want to fetch some data
• We’ll use Sawyer’s MetaCPAN::API

23. #!/usr/bin/env perl
use strict;
use warnings;
use MetaCPAN::API;
my $mcpan = MetaCPAN::API->new();
my $author = $mcpan->author('MSTROUT');

24. {
dir => "id/M/MS/MSTROUT",
email => ["perl-stuff@trout.me.uk"],
gravatar_url => "https://secure.gravatar.com/avatar/...",
name => "Matt S Trout",
pauseid => "MSTROUT",
website => ["http://www.trout.me.uk/"],
}

27. MetaCPAN Explorer

28. my $author = $mcpan->author('MSTROUT');

29. my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 1,
},
);

30. {
_shards => { failed => 0, successful => 5, total => 5 },
hits => {
hits => [
{
_id => "KHAMPTON",
_index => "cpan_v1",
_score => 1,
_source => {
city => "Los Angeles",
country => "US",
dir => "id/K/KH/KHAMPTON",
email => ["khampton@totalcinema.com", "kip.hampton@tamarou.com"],
gravatar_url => "http://www.gravatar.com/avatar/...",
name => "Kip Hampton",
pauseid => "KHAMPTON",
profile => [
{ id => "ubu", name => "coderwall" },
{ id => "ubu", name => "github" },
{ id => "kiphampton", name => "twitter" },
],
region => "CA",
updated => "2011-07-22T20:42:06",
website => ["http://totalcinema.com/"],
},
_type => "author",
},
],
max_score => 1,
total => 9780,
},
timed_out => bless(do{(my $o = 0)}, "JSON::XS::Boolean"),
took => 1,
}

31. my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 1,
},
);
# dump $result->{hits}->{hits}->[0]->{_source};

32. {
city => "Los Angeles",
country => "US",
dir => "id/K/KH/KHAMPTON",
email => ["khampton@totalcinema.com", "kip.hampton
@tamarou.com"],
gravatar_url => "http://www.gravatar.com/avatar/...",
name => "Kip Hampton",
pauseid => "KHAMPTON",
profile => [
{ id => "ubu", name => "coderwall" },
{ id => "ubu", name => "github" },
{ id => "kiphampton", name => "twitter" },
],
region => "CA",
updated => "2011-07-22T20:42:06",
website => ["http://totalcinema.com/"],
}

33. my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 100,
},
);

34. my $filter = {
{ term => { 'author.profile.name' => 'stackoverflow', } },
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

35. use Pithub;
my $p = Pithub->new;
AUTHOR:
foreach my $author ( @{ $result->{hits}->{hits} } ) {
foreach my $profile ( @{ $author->{_source}->{profile} } ) {
if ( $profile->{name} eq 'github' ) {
my $username = $profile->{id};
$username =~ s{https?://github.com/(w*)/?}{$1}i;
next AUTHOR if !$username;
if ( $p->users->get( user => $username )->content->{hireable} ) {
# do something...
}
next AUTHOR;
}
}
}

36. Getting fancy

37. my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } }
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

38. my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

39. my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);
# “missing” isn’t really helpful in this search
# just an example of how you might use it

40. my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

41. my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
fields => [ 'pauseid', 'country' ],
},
);

42. my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
sort => [ { 'author.pauseid' => 'ASC' } ],
},
);

43. Getting Help
• #metacpan or irc.perl.org
• https://metacpan.org/about/resources

44. Resources
• https://github.com/CPAN-API/cpan-
api/wiki/Beta-API-docs
• http://www.slideshare.net/
clintongormley/terms-of-endearment-
the-elasticsearch-query-dsl-explained

45. Bonus Slides

46. Base URL
• http://api.metacpan.org/v0

47. Convenience Endpoints

48. Convenience Endpoints
• /author/DOY
• /distribution/Moose
• /release/Moose
• /module/Moose
• /pod/Moose

50. Exporting Pod
• /pod/Moose?content-type=text/html (default)
• /pod/Moose?content-type=text/plain
• /pod/Moose?content-type=text/x-pod
• /pod/Moose?content-type=text/x-markdown

51. The (real) Endpoints

52. The (real) Endpoints
• /author

53. The (real) Endpoints
• /author
• /distribution

54. The (real) Endpoints
• /author
• /distribution
• /favorite

55. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating

56. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating
• /release

57. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating
• /release
• /file

58. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating
• /release
• /file

60. Using a cache
use HTTP::Tiny::Mech;
use MetaCPAN::API;
use WWW::Mechanize::Cached;
my $mcpan = MetaCPAN::API->new(
ua => HTTP::Tiny::Mech->new(
mechua => WWW::Mechanize::Cached->new()
)
);

62. Enable Compression
• use WWW::Mechanize::Gzip
• use WWW::Mechanize::Cached::Gzip
• Or set the appropriate request header

63. Use the scrolling API
• The scrolling API allows you to iterate
over an arbitrary number of results
• Be aware that when you scroll, your
docs will come back unsorted