This series of slides describes how to develop a twitter application.
This slide shows how to search tweets using Twitter Search RESTful Open API and how to implement it using Twitter4J.
Development of Twitter Application #8 - Streaming API
1. Linked Data &
Semantic Web
Technology
Development of
Twitter Applications
Part 8. Streaming API
Dr. Myungjin Lee
2. Linked Data & Semantic Web Technology
Streaming API
• Overview
– low latency access to Twitter's global stream of Tweet
data
– A proper implementation of a streaming client will be
pushed messages indicating Tweets and other events
have occurred.
• Streaming Endpoints
2
Public streams
Streams of the public data flowing through Twitter.
Suitable for following specific users or topics, and data
mining.
User streams
Single-user streams, containing roughly all of the data
corresponding with a single user's view of Twitter.
Site streams
The multi-user version of user streams. Site streams are
intended for servers which must connect to Twitter on
behalf of many users.
3. Linked Data & Semantic Web Technology
Differences between Streaming and REST
• REST API
• Streaming API
3
4. Linked Data & Semantic Web Technology
Connecting to a Streaming Endpoint
• Authentication
• Connecting
– form a HTTP request and consume the resulting
stream
– holding the connection open indefinitely, barring
server-side error, excessive client-side lag, network
hiccups, routine server maintenance or duplicate
logins
• Disconnections
– too many connections with the same credentials
– to stop reading data
– reading data too slowly
– a streaming server is restarted
– twitter’s network configuration changes
4
Auth Type Supported APIs Description
Basic Public Streams
Requests include the credentials of a
valid Twitter account.
OAuth
Public Streams
User Streams
Site Streams
Requests must be authorized according to
the OAuth specification.
5. Linked Data & Semantic Web Technology
Connecting to a Streaming Endpoint
• Stalls
– If 90 seconds pass with no data received, including
newlines, disconnect and reconnect immediately.
– The Streaming API will send a keep-alive newline
every 30 seconds to prevent your application from
timing out the connection.
• Reconnecting
– reconnecting attempts according to the type of error
experienced
• TCP/IP level network errors
– reconnect by 250ms each attempt, up to 16 seconds
• HTTP errors
– start with a 5 second wait, doubling each attempt, up to
320 seconds
• HTTP 420 errors
– start with a 1 minute wait and double each attempt
5
6. Linked Data & Semantic Web Technology
Connecting to a Streaming Endpoint
• Connection Churn
– churn: repeatedly opening and closing a connection
– keep your connections as stable and long-lived as
possible
• Rate Limiting
– Client which do not implement backoff and attempt to
reconnect as often as possible will have their
connections rate limited for a small number of
minutes.
– Rate limited clients will receive HTTP 420 responses
for all connection requests.
– If your client is rate limited frequently, it is possible
that your IP will be blocked from accessing Twitter for
an indeterminate period of time.
6
7. Linked Data & Semantic Web Technology
Connecting to a Streaming Endpoint
• HTTP Error Codes
7
Status Text Description
200 Success Self evident.
401 Unauthorized
HTTP authentication failed due to either:
•Invalid basic auth credentials, or an invalid OAuth request.
•Out-of-sync timestamp in your OAuth request (the response body will indicate
this).
•Too many incorrect passwords entered or other login rate limiting.
403 Forbidden The connecting account is not permitted to access this endpoint.
404 Unknown There is nothing at this URL, which means the resource does not exist.
406
Not
Acceptable
At least one request parameter is invalid. For example, the filter endpoint returns
this status if:
•The track keyword is too long or too short.
•An invalid bounding box is specified.
•Neither the track nor follow parameter are specified.
•The follow user ID is not valid.
413 Too Long
A parameter list is too long. For example, the filter endpoint returns this status if:
•More track values are sent than the user is allowed to use.
•More bounding boxes are sent than the user is allowed to use.
•More follow user IDs are sent than the user is allowed to follow.
416
Range
Unacceptable
For example, an endpoint returns this status if:
•A count parameter is specified but the user does not have access to use the count
parameter.
•A count parameter is specified which is outside of the maximum/minimum
allowable values.
420 Rate Limited
The client has connected too frequently. For example, an endpoint returns this status
if:
•A client makes too many login attempts in a short period of time.
•Too many copies of an application attempt to authenticate with the same
credentials.
503
Service
Unavailable
A streaming server is temporarily overloaded. Attempt to make another connection,
keeping in mind the connection attempt rate limiting and possible DNS caching in
your client.
8. Linked Data & Semantic Web Technology
Streaming API Request Parameters
• delimited
– be used on all streaming endpoints
– delimited=length
• statuses should be delimited in the stream by a length,
in bytes, a newline, and the status text
8
9. Linked Data & Semantic Web Technology
Streaming API Request Parameters
• stall_warnings
– be used on all streaming endpoints
– stall_warnings=true
• periodic messages to be delivered every 5 minutes if the
client is in danger of being disconnected
• filter_level
– be used on all streaming endpoints
– to set the minimum value of the Tweet attribute
required to be included in the stream
– one of none, low, and medium
– the default value is none.
• language
– be used on all streaming endpoints
– to return Tweets that have been detected as being
written in the specified languages
9
10. Linked Data & Semantic Web Technology
Streaming API Request Parameters
• follow
– available on POST statuses/filter
– a comma-separated list of user IDs
– to indicate the users whose Tweets should be
delivered on the stream
– the stream contain:
• Tweets created by the user.
• Tweets which are retweeted by the user.
• Replies to any Tweet created by the user.
• Retweets of any Tweet created by the user.
• Manual replies, created without pressing a reply button
(e.g. “@twitterapi I agree”).
– the stream does not contain:
• Tweets mentioning the user (e.g. “Hello @twitterapi!”).
• Manual Retweets created without pressing a Retweet
button (e.g. “RT @twitterapi The API is great”).
• Tweets by protected users.
10
11. Linked Data & Semantic Web Technology
Streaming API Request Parameters
• track
– available on POST statuses/filter
– what Tweets will be delivered on the stream
• a comma-separated list of phrases that may be one or more terms
separated by spaces
• a phrase will match if all of the terms in the phrase are present in
the Tweet, regardless of order and ignoring case
• commas as logical ORs and white spaces as logical ANDs
• each phrase must be between 1 and 60 bytes
• exact matching of phrases is not supported
• punctuation and special characters will be considered part of the
term they are adjacent to
– to check the text attribute of the Tweet, expanded_url and
display_url for links and media, text for hashtags, and
screen_name for user mentions
11
12. Linked Data & Semantic Web Technology
Streaming API Request Parameters
• locations
– available on POST statuses/filter
– to specify a set of bounding boxes to filter Tweets
– a comma-separated list of longitude, latitude pairs
– conditions
• •If the coordinates field is populated, the values there will
be tested against the bounding box.
• •If coordinates is empty but place is populated, the region
defined in place is checked .
• If none of the rules listed above match, the Tweet does not
match the location query.
• count
– to attempt to backfill missed messages which occurred
during the disconnect periot
– an integer from 1 to 150000 or from -1 to -150000
– use of this parameter should be carefully considered, as
high values increase the chance of a subsequent
disconnect
12
13. Linked Data & Semantic Web Technology
Streaming API Request Parameters
• with
– available on GET users and GET site
– to controls the types of messages delivered to User
and Site Streams clients
– the default is with=user
– The value with=followings adds messages from
accounts the user follows, equivalent to the user's
home timeline.
• replies
– available on GET users and GET site
– replies=all
• to have mimics of all Tweets returned in a streaming
connection
13
14. Linked Data & Semantic Web Technology
Processing Streaming Data
• Parsing responses
– The body of a streaming API response consists of a
series of newline-delimited messages.
• "newline" is considered to be rn (in hex, 0x0D 0x0A)
• "message" is a JSON encoded data structure or a blank
line
• JSON data
– The individual messages streamed by this API are
JSON encoded.
– the attributes of a JSON-encoded object are unordered
• Missing counts
– In very rare cases, the Streaming API may elect to
deliver an incomplete Tweet field.
14
15. Linked Data & Semantic Web Technology
Processing Streaming Data
• Transfer-Encoding: chunked
– Most streaming connections will be encoded using
chunked transfer encoding, as a Transfer-
Encoding: chunked HTTP header in the
response.
• Delimited messages
– Each message will be preceded by a string
representation of a base-10 integer indicating the
length of the message in bytes.
• Falling behind
– A way to track whether your client is falling behind is
to compare the timestamp of the Tweets you receive
with the current time.
– If the difference between the timestamps increases
over time, then the client is not processing Tweets as
fast as they are being delivered.
15
16. Linked Data & Semantic Web Technology
Processing Streaming Data
• Message ordering
– Messages from the Streaming API are not delivered in
sorted order.
– Delete messages may be delivered before the original
Tweet so implementations should be able to replay a
local cache of unrecognized deletes.
• Duplicate messages
– Duplicate messages may be delivered so
implementations should be tolerant of receiving a
Tweet more than once.
• Gzip compression
– Gzip compression may reduce the bandwidth needed
to process a stream to as small as 1/5th the size of an
uncompressed stream.
16
17. Linked Data & Semantic Web Technology
Public Streams
• Overview
– samples of the public data flowing through Twitter
• Endpoints
– GET statuses/sample
– GET statuses/firehose
– POST statuses/filter
• Using the streaming APIs
– connect to the endpoinsts
– process data returned by the streaming APIs
– There are some streaming messages which are not
returned by the REST APIs. See streaming message
types for information.
• Connections
– each account may create only one standing connection
to the public endpoints
– connecting to a public stream more than once with the
same account credentials will cause the oldest
connection to be disconnected
17
18. Linked Data & Semantic Web Technology
Public Stream Messages Types
• Blank lines
– some messages may be blank lines which serve as
"keep-alive" signals to prevent clients have stalled and
closing the connection.
• Status deletion notices (delete)
– These messages indicate that a given Tweet has been
deleted.
• Location deletion notices (scrub_geo)
– to indicate that geolocated data must be stripped from
a range of Tweets
18
19. Linked Data & Semantic Web Technology
Public Stream Messages Types
• Limit notices (limit)
– to indicate that a filtered stream has matched more
Tweets than its current rate limit allows to be
delivered
– Limit notices contain a total count of the number of
undelivered Tweets.
• Stall warnings (warning)
– When connected to a stream using the stall_warnings
parameter, you may receive status notices indicating
the current health of the connection.
19
20. Linked Data & Semantic Web Technology
Public Stream Messages Types
• Disconnect messages (disconnect)
– to deliver a message indicating why a stream was
closed
– status codes
20
Code Name Description
1 Shutdown The feed was shutdown (possibly a machine restart)
2
Duplicate
stream
The same endpoint was connected too many times.
3 Control request Control streams was used to close a stream (applies to sitestreams).
4 Stall The client was reading too slowly and was disconnected by the server.
5 Normal The client appeared to have initiated a disconnect.
6 Token revoked An oauth token was revoked for a user (applies to site and userstreams).
7 Admin logout
The same credentials were used to connect a new stream and the oldest was
disconnected.
8 Reserved for internal use. Will not be delivered to external clients.
9
Max message
limit
The stream connected with a negative count parameter and was disconnected
after all backfill was delivered.
10
Stream
exception
An internal issue disconnected the stream.
11 Broker stall An internal issue disconnected the stream.
12 Shed load
The host the stream was connected to became overloaded and streams were
disconnected to balance load. Reconnect as usual.
21. Linked Data & Semantic Web Technology
GET statuses/sample
• sample Streaming API
– Returns a small random sample of all public statuses.
The Tweets returned by the default access level are the
same, so if two different clients connect to this
endpoint, they will see the same Tweets.
• Resource URL
– https://stream.twitter.com/1.1/statuses/sample.json
• Parameters
• Other Information
– Rate Limited? Yes
– Authentication: Requires user context
– Response Object: Tweets
21
delimited
optional
Specifies whether messages should be length-delimited.
stall_warnings
optional
Specifies whether stall warnings should be delivered.
22. Linked Data & Semantic Web Technology
Twitter4J Classes for sample Streaming
• TwitterStreamFactory Class
– A factory class for TwitterFactory.
– An instance of this class is completely thread safe and
can be re-used and used concurrently.
– Methods
• TwitterStream getInstance()
– Returns a instance associated with the configuration
bound to this factory.
• TwitterStream getInstance(AccessToken accessToken)
– Returns a OAuth Authenticated instance.
– consumer key and consumer Secret must be provided by
twitter4j.properties, or system properties.
• TwitterStream Interface
– Methods
• void addListener(StatusListener listener)
• void sample()
– Starts listening on random sample of all public statuses.
• void shutdown()
– Shuts down internal dispatcher thread shared by all
TwitterStream instances.
22
23. Linked Data & Semantic Web Technology
Twitter4J Classes for sample Streaming
• StatusListener Interface
– Methods
• void onDeletionNotice(StatusDeletionNotice statusDeletionNotice)
– Called upon deletionNotice notices.
• void onScrubGeo(long userId, long upToStatusId)
– Called upon location deletion messages.
• void onStallWarning(StallWarning warning)
– Called when receiving stall warnings.
• void onStatus(Status status)
• void onTrackLimitationNotice(int numberOfLimitedStatuses)
– This notice will be sent each time a limited stream becomes
unlimited.
23
24. Linked Data & Semantic Web Technology
Example of sample Streaming API
1. import twitter4j.StallWarning;
2. import twitter4j.Status;
3. import twitter4j.StatusDeletionNotice;
4. import twitter4j.StatusListener;
5. public class StatusListenerImpl implements StatusListener {
6. @Override
7. public void onStatus(Status status) {
8. System.out.println("@" + status.getUser().getScreenName() + " - "
9. + status.getText());
10. }
11. @Override
12. public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
13. System.out.println("Got a status deletion notice id:"
14. + statusDeletionNotice.getStatusId());
15. }
16. @Override
17. public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
18. System.out.println("Got track limitation notice:"
19. + numberOfLimitedStatuses);
20. }
21. @Override
22. public void onScrubGeo(long userId, long upToStatusId) {
23. System.out.println("Got scrub_geo event userId:" + userId
24. + " upToStatusId:" + upToStatusId);
25. }
26. @Override
27. public void onStallWarning(StallWarning warning) {
28. System.out.println("Got stall warning:" + warning);
29. }
30. @Override
31. public void onException(Exception ex) {
32. ex.printStackTrace();
33. }
34. }
24
25. Linked Data & Semantic Web Technology
Example of sample Streaming API
1. import twitter4j.*;
2. public class PrintSampleStream {
3. public static void main(String[] args) throws TwitterException {
4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,
6. TwitterAccessToken.consumerSecret);
7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());
8. StatusListenerImpl listener = new StatusListenerImpl();
9. twitterStream.addListener(listener);
10. twitterStream.sample();
11. }
12. }
25
26. Linked Data & Semantic Web Technology
GET statuses/firehose
• firehose Streaming API
– This endpoint requires special permission to access.
– Returns all public statuses.
– Few applications require this level of access.
– Creative use of a combination of other resources and
various access levels can satisfy nearly every
application use case.
• Resource URL
– https://stream.twitter.com/1.1/statuses/firehose.json
• Parameters
• Other Information
– Rate Limited? Yes
– Authentication: Requires user context
– Response Object: Tweets
26
count
optional
The number of messages to backfill.
delimited
optional
Specifies whether messages should be length-delimited.
stall_warnings
optional
Specifies whether stall warnings should be delivered.
27. Linked Data & Semantic Web Technology
Twitter4J Classes for firehose Streaming
• Twitter4J Classes for firehose Streaming
– TwitterStream Interface
• Methods
– void firehose(int count)
» Starts listening on all public statuses.
• Example of firehose Streaming API
1. import twitter4j.*;
2. public class PrintFirehoseStream {
3. public static void main(String[] args) throws TwitterException {
4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,
6. TwitterAccessToken.consumerSecret);
7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());
8. StatusListenerImpl listener = new StatusListenerImpl();
9. twitterStream.addListener(listener);
10. twitterStream.firehose(0);
11. }
12. }
27
28. Linked Data & Semantic Web Technology
POST statuses/filter
• filter Streaming API
– Returns public statuses that match one or more filter
predicates.
– Both GET and POST requests are supported, but GET
requests with too many parameters may cause the request
to be rejected for excessive URL length.
• Resource URL
– https://stream.twitter.com/1.1/statuses/filter.json
• Parameters
• Other Information
– Rate Limited? Yes
– Authentication: Requires user context
– Response Object: Tweets
28
follow
A comma separated list of user IDs, indicating the users to return statuses for in the
stream.
track Keywords to track. Phrases of keywords are specified by a comma-separated list.
locations Specifies a set of bounding boxes to track.
delimited
optional
Specifies whether messages should be length-delimited.
stall_warnings
optional
Specifies whether stall warnings should be delivered.
29. Linked Data & Semantic Web Technology
Twitter4J Classes for filter Streaming
• TwitterStream Interface
– Methods
• void filter(FilterQuery query)
– Start consuming public statuses that match one or more
filter predicates.
• FilterQuery Class
– Methods
• FilterQuery count(int count)
– Sets count
• FilterQuery follow(long[] follow)
– Sets follow
• FilterQuery locations(double[][] locations)
– Sets locations
• FilterQuery track(java.lang.String[] track)
– Sets track
29
30. Linked Data & Semantic Web Technology
Example of filter Streaming API
1. import twitter4j.*;
2. public final class PrintFilterStream {
3. public static void main(String[] args) throws TwitterException {
4. StatusListenerImpl listener = new StatusListenerImpl();
5. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
6. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,
7. TwitterAccessToken.consumerSecret);
8. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());
9. twitterStream.addListener(listener);
10. FilterQuery fq = new FilterQuery();
11. // long[] follow = { 22931893, 16227629, 464020027, 299014098, 40203602,
12. // 121858259, 631481692, 173523502, 64972381, 357617589, 77689885 };
13. // fq.follow(follow);
14. // String[] track = { "psy", "#psy", "@psy_oppa", "gentleman",
15. // "#gentleman", "@ygent_official" };
16. // fq.track(track);
17. // double[][] locations = { { -74, 40 }, { -73, 41 } }; // New York City
18. // fq.locations(locations);
19. twitterStream.filter(fq);
20. }
21. }
30
31. Linked Data & Semantic Web Technology
User Streams
• Overview
– a stream of data and events specific to the
authenticated user
• Endpoint
– GET user
• Connections
– limited to only a few simultaneous User Streams
connections per OAuth application, regardless of IP
– to handle the HTTP 420 error code that indicates that
the account has been logging in to often
• User streams messages
– Types of messages
• some streaming messages which are not returned by the
REST APIs
31
32. Linked Data & Semantic Web Technology
User Stream Messages Types
• Friends lists (Friends)
– a preamble which is a list of the user’s friends before
starting regular message delivery
• Too many follows (warning)
– warning message when your application connects on
behalf of a user who follows more than 10,000
accounts
32
33. Linked Data & Semantic Web Technology
User Stream Messages Types
• Events (event)
– notifications about non-Tweet events
33
Description Event Name Source Target
Target
Object
User blocks someone block Current user Blocked user Null
User removes a block unblock Current user
Unblocked
user
Null
User favorites a Tweet favorite Current user Tweet author Tweet
User's Tweet is favorited favorite Favoriting user Current user Tweet
User unfavorites a Tweet unfavorite Current user Tweet author Tweet
User's Tweet is unfavorited unfavorite
Unfavoriting
user
Current user Tweet
User follows someone follow Current user Followed user Null
User is followed follow Following user Current user Null
User unfollows someone unfollow Current user Followed user Null
User creates a list list_created Current user Current user List
User deletes a list list_destroyed Current user Current user List
User edits a list list_updated Current user Current user List
User adds someone to a list list_member_added Current user Added user List
User is added to a list list_member_added Adding user Current user List
User removes someone
from a list
list_member_removed Current user Removed user List
User is removed from a list list_member_removed Removing user Current user List
User subscribes to a list list_user_subscribed Current user List owner List
User's list is subscribed to list_user_subscribed
Subscribing
user
Current user List
User unsubscribes from a
list
list_user_unsubscribed Current user List owner List
User's list is unsubscribed
from
list_user_unsubscribed
Unsubscribing
user
Current user List
User updates their profile user_update Current user Current user Null
34. Linked Data & Semantic Web Technology
GET user
• user Streaming API
– streams messages for a single user, as described in
User streams
• Resource URL
– https://userstream.twitter.com/1.1/user.json
• Parameters
• Other Information
– Rate Limited? Yes
– Authentication: Requires user context
34
delimited
optional
Specifies whether messages should be length-delimited.
stall_warnings
optional
Specifies whether stall warnings should be delivered.
with
optional
Specifies whether to return information for just the authenticating user, or include
messages from accounts the user follows.
replies
optional
Specifies whether to return additional @replies.
track
optional
Includes additional Tweets matching the specified keywords. Phrases of keywords
are specified by a comma-separated list.
locations
optional
Includes additional Tweets falling within the specified bounding boxes.
35. Linked Data & Semantic Web Technology
Twitter4J Classes for user Streaming
• TwitterStream Interface
– Methods
• void user()
– User Streams provides real-time updates of all data needed to update a
desktop application display.
• void user(java.lang.String[] track)
– User Streams provides real-time updates of all data needed to update a
desktop application display.
• UserStreamListener Interface
– Methods
• void onBlock(User source, User blockedUser)
• void onDeletionNotice(long directMessageId, long userId)
• void onDirectMessage(DirectMessage directMessage)
• void onFavorite(User source, User target, Status favoritedStatus)
• void onFollow(User source, User followedUser)
• void onFriendList(long[] friendIds)
• void onUnblock(User source, User unblockedUser)
• void onUnfavorite(User source, User target,
Status unfavoritedStatus)
• void onUserListCreation(User listOwner, UserList list)
• void onUserListDeletion(User listOwner, UserList list)
• void onUserListMemberAddition(User addedMember,
User listOwner, UserList list)
• void onUserListMemberDeletion(User deletedMember,
User listOwner, UserList list)
• void onUserListSubscription(User subscriber,
User listOwner, UserList list)
• void onUserListUnsubscription(User subscriber,
User listOwner, UserList list)
• void onUserListUpdate(User listOwner, UserList list)
• void onUserProfileUpdate(User updatedUser)
35
36. Linked Data & Semantic Web Technology
Example of user Streaming API
1. import twitter4j.*;
2. public class UserStreamListenerImpl implements UserStreamListener {
3. public void onStatus(Status status) {
4. System.out.println("onStatus @" + status.getUser().getScreenName()
5. + " - " + status.getText());
6. }
7. public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {
8. System.out.println("Got a status deletion notice id:"
9. + statusDeletionNotice.getStatusId());
10. }
11. public void onDeletionNotice(long directMessageId, long userId) {
12. System.out.println("Got a direct message deletion notice id:"
13. + directMessageId);
14. }
15. public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
16. System.out.println("Got a track limitation notice:"
17. + numberOfLimitedStatuses);
18. }
19. public void onScrubGeo(long userId, long upToStatusId) {
20. System.out.println("Got scrub_geo event userId:" + userId
21. + " upToStatusId:" + upToStatusId);
22. }
23. public void onStallWarning(StallWarning warning) {
24. System.out.println("Got stall warning:" + warning);
25. }
26. public void onFriendList(long[] friendIds) {
27. System.out.print("onFriendList");
28. for (long friendId : friendIds) {
29. System.out.print(" " + friendId);
30. }
31. System.out.println();
32. }
36
39. Linked Data & Semantic Web Technology
Example of user Streaming API
1. import twitter4j.*;
2. public final class PrintUserStream {
3. public static void main(String[] args) throws TwitterException {
4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,
6. TwitterAccessToken.consumerSecret);
7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());
8. UserStreamListenerImpl listener = new UserStreamListenerImpl();
9. twitterStream.addListener(listener);
10. twitterStream.user();
11. }
12. }
39
40. Linked Data & Semantic Web Technology
Site Streams
• Overview
– services to receive real-time updates for a large number of users
– for any user who has granted OAuth access to your application
– Site Streams is currently in a limited beta. Access is restricted to
whitelisted accounts.
• Endpoint
– GET site
• Types of messages
– the same types of messages streamed to User Stream
• Protected data
– to provide both public and private data in a single stream
– meticulous about what data they display
• Connecting
– to only receive messages for users who have approved OAuth
access to your application
– to consume significant (> 1 Mbit/sec) bandwith
• Limits
– must be respected during the beta period
• limit the follow count to 100 users per stream while connecting
• open no more than 25 new connections per second
40
41. Linked Data & Semantic Web Technology
Site Stream Messages Types
• Envelopes (for_user)
– an additional wrapper to identify the target of each
message except for blank keep-alive lines
• Control messages (control)
– a control message which may be used to modify the
Site Streams connection without reconnecting for new
Site Streams
41
42. Linked Data & Semantic Web Technology
Twitter4J Classes for site Streaming
• TwitterStream Interface
– Methods
• StreamController site(boolean withFollowings, long[] follow)
– Site Streams, a new feature on the Streaming API, is now available for beta testing.
• SiteStreamsListener Interface
– Methods
• void onBlock(long forUser, User source, User blockedUser)
• void onDeletionNotice(long forUser, long directMessageId, long userId)
• void onDeletionNotice(long forUser,
StatusDeletionNotice statusDeletionNotice)
• void onDirectMessage(long forUser, DirectMessage directMessage)
• void onDisconnectionNotice(java.lang.String line)
• void onException(java.lang.Exception ex)
• void onFavorite(long forUser, User source, User target, Status favoritedStatus)
• void onFollow(long forUser, User source, User followedUser)
• void onFriendList(long forUser, long[] friendIds)
• void onStatus(long forUser, Status status)
• void onUnblock(long forUser, User source, User unblockedUser)
• void onUnfavorite(long forUser, User source, User target,
Status unfavoritedStatus)
• void onUnfollow(long forUser, User source, User unfollowedUser)
• void onUserListCreation(long forUser, User listOwner, UserList list)
• void onUserListDeletion(long forUser, User listOwner, UserList list)
• void onUserListMemberAddition(long forUser, User addedMember,
User listOwner, UserList list)
• void onUserListMemberDeletion(long forUser, User deletedMember,
• User listOwner, UserList list)
• void onUserListSubscription(long forUser, User subscriber, User listOwner,
UserList list)
• void onUserListUnsubscription(long forUser, User subscriber, User listOwner,
UserList list)
• void onUserListUpdate(long forUser, User listOwner, UserList list)
• void onUserProfileUpdate(long forUser, User updatedUser)
42
43. Linked Data & Semantic Web Technology
Example of site Streaming API
1. import twitter4j.*;
2. public class SiteStreamsListenerImpl implements SiteStreamsListener {
3. public void onStatus(long forUser, Status status) {
4. System.out.println("onStatus for_user:" + forUser + " @"
5. + status.getUser().getScreenName() + " - " + status.getText());
6. }
7. public void onDeletionNotice(long forUser,
8. StatusDeletionNotice statusDeletionNotice) {
9. System.out.println("Got a status deletion notice for_user:" + forUser
10. + " id:" + statusDeletionNotice.getStatusId());
11. }
12. public void onFriendList(long forUser, long[] friendIds) {
13. System.out.print("onFriendList for_user:" + forUser);
14. for (long friendId : friendIds) {
15. System.out.print(" " + friendId);
16. }
17. System.out.println();
18. }
19. public void onFavorite(long forUser, User source, User target,
20. Status favoritedStatus) {
21. System.out.println("onFavorite for_user:" + forUser + " source:@"
22. + source.getScreenName() + " target:@" + target.getScreenName()
23. + " @" + favoritedStatus.getUser().getScreenName() + " - "
24. + favoritedStatus.getText());
25. }
26. public void onUnfavorite(long forUser, User source, User target,
27. Status unfavoritedStatus) {
28. System.out.println("onUnFavorite for_user:" + forUser + " source:@"
29. + source.getScreenName() + " target:@" + target.getScreenName()
30. + " @" + unfavoritedStatus.getUser().getScreenName() + " - "
31. + unfavoritedStatus.getText());
32. }
33. public void onFollow(long forUser, User source, User followedUser) {
34. System.out.println("onFollow for_user:" + forUser + " source:@"
35. + source.getScreenName() + " target:@"
36. + followedUser.getScreenName());
37. }
43
44. Linked Data & Semantic Web Technology
Example of site Streaming API
38. public void onUnfollow(long forUser, User source, User followedUser) {
39. System.out.println("onUnfollow for_user:" + forUser + " source:@"
40. + source.getScreenName() + " target:@"
41. + followedUser.getScreenName());
42. }
43. public void onDirectMessage(long forUser, DirectMessage directMessage) {
44. System.out.println("onDirectMessage for_user:" + forUser + " text:"
45. + directMessage.getText());
46. }
47. public void onDeletionNotice(long forUser, long directMessageId, long userId) {
48. System.out.println("Got a direct message deletion notice for_user:"
49. + forUser + " id:" + directMessageId);
50. }
51. public void onUserListMemberAddition(long forUser, User addedMember,
52. User listOwner, UserList list) {
53. System.out.println("onUserListMemberAddition for_user:" + forUser
54. + " member:@" + addedMember.getScreenName() + " listOwner:@"
55. + listOwner.getScreenName() + " list:" + list.getName());
56. }
57. public void onUserListMemberDeletion(long forUser, User deletedMember,
58. User listOwner, UserList list) {
59. System.out.println("onUserListMemberDeletion for_user:" + forUser
60. + " member:@" + deletedMember.getScreenName() + " listOwner:@"
61. + listOwner.getScreenName() + " list:" + list.getName());
62. }
63. public void onUserListSubscription(long forUser, User subscriber,
64. User listOwner, UserList list) {
65. System.out.println("onUserListSubscribed for_user:" + forUser
66. + " subscriber:@" + subscriber.getScreenName() + " listOwner:@"
67. + listOwner.getScreenName() + " list:" + list.getName());
68. }
69. public void onUserListUnsubscription(long forUser, User subscriber,
70. User listOwner, UserList list) {
71. System.out.println("onUserListUnsubscribed for_user:" + forUser
72. + " subscriber:@" + subscriber.getScreenName() + " listOwner:@"
73. + listOwner.getScreenName() + " list:" + list.getName());
74. }
44
45. Linked Data & Semantic Web Technology
Example of site Streaming API
75. public void onUserListCreation(long forUser, User listOwner, UserList list) {
76. System.out.println("onUserListCreated for_user:" + forUser
77. + " listOwner:@" + listOwner.getScreenName() + " list:"
78. + list.getName());
79. }
80. public void onUserListUpdate(long forUser, User listOwner, UserList list) {
81. System.out.println("onUserListUpdated for_user:" + forUser
82. + " listOwner:@" + listOwner.getScreenName() + " list:"
83. + list.getName());
84. }
85. public void onUserListDeletion(long forUser, User listOwner, UserList list) {
86. System.out.println("onUserListDestroyed for_user:" + forUser
87. + " listOwner:@" + listOwner.getScreenName() + " list:"
88. + list.getName());
89. }
90. public void onUserProfileUpdate(long forUser, User updatedUser) {
91. System.out.println("onUserProfileUpdated for_user:" + forUser
92. + " user:@" + updatedUser.getScreenName());
93. }
94. public void onBlock(long forUser, User source, User blockedUser) {
95. System.out.println("onBlock for_user:" + forUser + " source:@"
96. + source.getScreenName() + " target:@"
97. + blockedUser.getScreenName());
98. }
99. public void onUnblock(long forUser, User source, User unblockedUser) {
100. System.out.println("onUnblock for_user:" + forUser + " source:@"
101. + source.getScreenName() + " target:@"
102. + unblockedUser.getScreenName());
103. }
104. public void onDisconnectionNotice(String line) {
105. System.out.println("onDisconnectionNotice:" + line);
106. }
107. public void onException(Exception ex) {
108. ex.printStackTrace();
109. System.out.println("onException:" + ex.getMessage());
110. }
111. }
45
46. Linked Data & Semantic Web Technology
Example of site Streaming API
1. import twitter4j.*;
2. public class PrintSiteStream {
3. public static void main(String[] args) throws TwitterException {
4. TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
5. twitterStream.setOAuthConsumer(TwitterAccessToken.consumerKey,
6. TwitterAccessToken.consumerSecret);
7. twitterStream.setOAuthAccessToken(TwitterAccessToken.loadAccessToken());
8. SiteStreamsListenerImpl listener = new SiteStreamsListenerImpl();
9. twitterStream.addListener(listener);
10. long[] followArray = {52401019, 1476571, 822659, 20606823, 18304822};
11. twitterStream.site(true, followArray);
12. }
13. }
46