CMPE 275 Project 2 Report: RESTful social collaboration
1. CMPE 275
Project 2 Report
RESTful social collaboration through Pinterest re-enactment
Team:
Bhushan Deo (009269403)
Deven Pawar (009275994)
Dhruv Gogna (005067284)
Gaurav Bhardwaj (009297431)
Vinay Bhore (009290931)
2. Introduction
The main goal of Project 2 is to explore the design and challenges in developing RESTful web services for
a social media service such as Pinterest. Sharing content is an important pattern found in all social media
services. Providing robust and well documented web services is very important in such products as there
are several external clients which use them to access and manipulate stored user data. In this report, we
provide a detailed explanation of our team’s design and implementation.
High Level Architecture
The team’s configuration consists of the following components:
1. Web Services:
We expose our functionality to the client using RESTful web services written in Python using the
Bottle micro-framework.
2. Database:
CouchDB NoSQL database is used to persist user data. It provides data storage as JSON
documents and thus naturally maps to the requirements. Also, when client hits our web service
and some data is changed, we can verify changes in the database through the web interface
(Futon), that comes with CouchDB.
3. couchdb-python library:
This library provides interface to interact with CouchDB from python code. In our experience,
this was a very well designed library with advanced features like high-level couchDB document
to python object mapping, intuitive interfaces and good documentation.
4. Client:
For testing, we are using curl to hit our web services and see the response. Curl provides many
advanced options to make testing via HTTP requests easy.
The following sections explain the functionalities implemented and the web service and database
component in detail. These include comments about what we learnt while working on each component.
3. Functionalities
This project also required cross team interaction to come up with a stable specification for the web
services. As per the specification agreed to by the class, following functionalities were implemented by
our team.
Here, we observed how the traditional Create, Read, Update and Delete operations in databases, map
exactly to HTTP methods POST, GET, PUT and DELETE respectively.
Functionality Endpoint (fields in angular brackets are generated on the
server)
CRUD? HTTP
Met-
hod
Request
Parameters (all
are string type
unless specified)
Sign up http://localhost:8080/users/signUp Create POST firstName
lastName
emailId
password
Login http://localhost:8080/users/login/ Create POST emailId
password
Create a
board
http://localhost:8080/users/<userId>/boards/ Create POST boardName
boardDesc
category
isPrivate(Boolean)
Get all
boards
http://localhost:8080/users/<userId>/boards/ Read GET None
Get details of
a single
board
http://localhost:8080/users/<userId>/boards/<boardId>/ Read GET None
Update
Board
http://localhost:8080/users/<userId>/boards/<boardId>/ Update PUT boardName
boardDesc
category
isPrivate(Boolean)
Delete Board http://localhost:8080/users/<userId>/boards/<boardId>/ Delete DELETE None
Create Pin http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/
Create POST pinName
image
description
Get a single
pin from a
board
http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/<pinId>/
Read GET None
Get all pins
from a board
http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/
Read GET None
Update a pin http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/<pinId>/
Update PUT pinName
image
description
Delete a pin http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/<pinId>/
Delete DELETE None
Add a
comment to
a pin
http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/<pinId>/comment/
Create POST description
View
comments
on a pin
http://localhost:8080/users/<userId>/boards/<boardId>/p
ins/<pinId>/comment/
Read GET None
4. The responses were returned in JSON format and contained relevant data and endpoint information for
subsequent operations as agreed to by the class.
Web Services
Why Web Services fit in this project?
Essentially, web services help de-couple the client and the server implementations. User performs
various CRUD operations on an external system by sending HTTP requests through the client. The web
service layer, intercepts these requests on the server, performs operations on the database and sends
back appropriate response to the client. This fits well with the sharing and collaboration required by
social media products. From real world examples such as Facebook, Twitter and Pinterest, it can be seen
that a robust and well documented RESTful API is a vital contributor to these products going viral.
Bottle:
Web services were written in Python using the Bottle micro-framework. We found Bottle to be very
intuitive for creating our web services. The framework is very lightweight and in order to use it, it only
requires us to include one bottle.py file in the project directory.
Server:
Bottle runs by default on a single threaded WSGI server, but has adapters to various multi-threaded
servers such as paste and Cherrypy which give better performance for applications that need to handle
large number of concurrent requests. For testing this project, we have used the default server.
Mapping requests to appropriate methods:
Various endpoints as mentioned in the functionalities section, map easily to corresponding methods
which process the requests at those endpoint, simply by placing a @route decorator above that method.
Retrieving data from request:
In REST, the URL path determines the data that will be accessed or manipulated and the HTTP method
determines the operation that will be performed on that data.
- To get information about data that will be accessed, we use wildcards in the URL:
@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’GET’)
def getBoard(userId, boardId):
# got userId and boardId from the URL, now perform GET(read) on database
- To retrieve data from the HTTP request, bottle provides convenient methods in the request object:
@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’)
def updateBoard(userId, boardId):
# got userId and boardId from the URL, now perform PUT(update)on database
# with the fields sent in the request by the client
boardName = request.forms.get('boardName')
boardDesc = request.forms.get('boardDesc')
Returning JSON response:
JSON was the decided as the format in which response will be sent back to the client. Even in this case,
Bottle made it easy to return back response with “application/json” content type. We simply had to
5. return a python dictionary object. Bottle returns the response in JSON and the appropriate field is set in
the HTTP header.
@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’)
def updateBoard(userId, boardId):
# perform appropriate operations
return dict(status='OK', message='sample message')
If other response types such as html or xml are expected by the client, we are aware that we would have
to check for the content types which the client accepts from the HTTP request header and accordingly
format the response before sending it back (content negotiation).
Handling trailing ‘/’ slash:
If some clients put a trailing slash at the end of the endpoint URL, then that is treated as a separate
route by Bottle and might not map to the appropriate method in our web service code. Given that we
would be testing across teams, this was highly probable. To handle this, Bottle documentation suggests
to simply map two routes to the same method, one with a trailing slash and one without a slash.
@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’)
#No trailing slash
@route(‘http://localhost:8080/users/<userId>/boards/<boardId>’,method=’PUT’)
def updateBoard(userId, boardId):
# perform appropriate operations
Database
In REST web services, we can think of the URL as a path that is traversed to get to the data that is to be
accessed or manipulated. Further, the HTTP method that is specified in the HTTP request gives us the
operation (create, read, update or delete) that is to be performed on the data. Accordingly the actions
that we take in code depend on the combination of the following factors:
1. Response format accepted by the client (json, xml, html, image, etc..)
2. Data that is to be accessed (URL traversal)
3. What kind of operation on the data (HTTP method type)
Thus, for every endpoint there are several combinations that need to be handled so that we process the
user request appropriately and so that we return desired response back to the user. In this context, it is
clear that we have to organize our data based on the endpoints that we expose to the user.
We observed that this is a notable difference compared to Project 1, where there was only one endpoint
for the client, and the server queue delegated the message to the appropriate resource to handle. In
Project 2, there are several endpoints for the client and each endpoint maps to one method on the server
side.
The flat schema that we have used to store data in CouchDB is as follows. The “_id” is the userId that is
generated server side when the user signs up. This is passed around in each URL and is used to identify
user in each subsequent request.
7. Drawback:
The limitation of this schema is that for any request, we get the entire complex document associated
with a userId from the database. Then, in case of long URLs we have to employ multiple nested loops to
get to the data that we require. For example, updating information about a pin needs three nested
loops through the document that is returned. This is shown in the figure above. This would be
problematic if a user had thousands of boards and pins.
A solution to this would be to create separate tables (“databases” in CouchDB) for each step in the
endpoint URL tree, i.e. separate tables for user, board and comments, each record containing reference
to the userId. However, the tradeoff here is that this would require more queries to the database per
user request.
Note on Aerospike:
Initially we decided on using Aerospike as our database. However, its Python client seems to be under
development and there was no support for list and map data structures. Further, there was no query
support in the Python client, only full table scan was supported. Hence we decided to go with CouchDB.
Nonetheless, it was interesting to understand Aerospike’s architecture and concepts such as location
awareness of client, peer relationship among nodes, use of binary wire protocol for communication and
cross data center replication.
Conclusion
For our team, following are the major learnings from this project:
1. Programming in Python
2. RESTful web services
3. Database design for RESTful web services
4. Micro-frameworks
5. Comparison of technologies used in Project 1 and 2
6. CouchDB and Aerospike
7. Collaboration in Small and Large Software Development Teams
