The document provides an overview of the HTTP request-response cycle. It explains that an HTTP request is initiated by a user agent and includes a URL and HTTP method. The request then goes through the frontend and backend infrastructure, with the backend typically involving a host server, web server, and application server. The web server handles the request, may issue redirects, and returns a response that is delivered back to the user agent, completing the cycle.
Cool Git Tricks (That I Learn When Things Go Badly) [1/2]
Full Stack & Full Circle: What the Heck Happens In an HTTP Request-Response Cycle
1. Full-Stack & Full Circle
What The Heck Happens in an HTTP Request-Response Cycle
Carina C. Zona
http://cczona.com/
@cczona
Confident Coding
Microsoft San Francisco
October 20, 2012
@cczona
Tuesday, October 23, 12
When Estelle said she wanted to do a day of teaching "everything else" about web development, I knew right away that the HTTP request-response
cycle is something I just _had_ to talk about.
It's the topic that always gets skipped. "Welcome to web development! Let's start at HTML, CSS, and JavaScript!"
We pass it right by. And we shouldn't.
2. Hypertext Transfer Protocol (HTTP)
is the core technology of the web
@cczona
Tuesday, October 23, 12
Hypertext Transfer Protocol (HTTP) is THE CORE technology of the web.
Yes, more essential than even HTML. Assets on the web can be HTML, but don't have to be. We download software, graphics,
PDFs, Word docs, Flash...none of these is HTML. Nor is XML, or JSON. CSS, or JavaScript.
Data flowing across the web relies either on HTTP, or its secure counterpart HTTPS.
The web relies on the HTTP request-response cycle.
3. ?
?
Tuesday, October 23, 12
As a developer, you wants to take an interest in the cycle. It's where your work gets assembled. That assembly
can be slow, or it can be fast. _Which_? Well, that depends on how the request-response cycle is managed.
As a developer, you want to _understand_ what the cycle passes through so that you can arrange it to do whatever
you want.
4. Frontend
Backend
Tuesday, October 23, 12
Web development is usually regarded as two parts:
Frontend
Backend
5. Client side
Server side
Tuesday, October 23, 12
Sometimes we refer instead to
Client side
Server side
6. Frontend & Backend refer to infrastructure
Client side & server side refer to programming languages
@cczona
Tuesday, October 23, 12
It's fine, we do it all the time. But technically imprecise: [slide]
7. Client side
(JavaScript)
Server side
(Python, Ruby, PHP,
Perl, Java, etc.)
Tuesday, October 23, 12
So the cycle refers to the communication passing through our frontend & backend infrastructure. Later we'll look
at how programming languages fit into this picture.
8. Tuesday, October 23, 12
Let's add a User to this mental picture, too. Because someone's got to make the request, right? And someone's got
to be interested in what response it gets. So now we've got at least 3 potential components. They even
correspond pretty nicely to a common team configuration: a user experience designer, a frontend developer,
a backend developer.
9. Web
site/app
Tuesday, October 23, 12
Frontend and backend are different parts of a cohesive whole, the website. It's the user and site that are in
relationship with each other, want something of each other, desiring to communicate with each other.
The frontend and backend _together_ form the stack of infrastructure for the site to do that.
10. Full stack
Tuesday, October 23, 12
That's what we mean by "full stack". The combination of frontend and backend infrastructure, plus the
technologies that allow them to interact.
11. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
Voila. The request-response cycle just refers to passing a request message through the
stack, and the stack delivering a response to that.
13. URL
@cczona
Tuesday, October 23, 12
There's an important piece missing from that diagram so far, right? URL.
What is a URL anyway? Many people compare it to a street address. A series of progressive
detail that narrow it down to an exact location.
14. Anatomy of a URL
@cczona
Tuesday, October 23, 12
15. http://cczona.com/
<scheme>://<host>/
URL Basics
@cczona
Tuesday, October 23, 12
http://cczona.com/
<scheme>://<host>/
At minimum, a URL needs two parts:
16. http://cczona.com/
<scheme>://<host>/
URL Basics
@cczona
Tuesday, October 23, 12
Scheme - Browsers have lately been hiding the scheme from you. Don't be misled by that.
Scheme is _mandatory_. It is an instruction, specifying which transfer method to use.
17. http://cczona.com/
<scheme>://<host>/
URL Basics
@cczona
Tuesday, October 23, 12
Host - This part I _would_ compare to an address. It specifies where the host server is. We'll
expand on that later.
18. http://cczona.com/blog/2012/10/20/confident-coding
<scheme>://<host>/< path >
URL Common
@cczona
Tuesday, October 23, 12
It usually also has at least one more part:
Path - the name of a resource being requested
20. http://carinazona:opensesame@cczona.com:3000/
blog;year=2012?
search&term=Confident&month=October#session2
<scheme>://<user>:<password>@<host>:<port>/
<path>;<params >?
<query>#fragment
URL All the parts!
@cczona
Tuesday, October 23, 12
There are 6 optional additional parts available.
I know, crazy, right?
21. Port
Query
Fragment
URL Less Common
@cczona
Tuesday, October 23, 12
Port - is where the host server is expected to be listening for an HTTP request
Query - is one or more values intended to be used by an application
Fragment - is the name of some part within the requested resource.
22. Username
Password
Parameters
URL Rarely used for HTTP
@cczona
Tuesday, October 23, 12
23. User-Agent
@cczona
Tuesday, October 23, 12
A request is submitted via a User-Agent.
24. User-Agent examples
Browser
Crawlers/spiders/robots
Screen readers
Programs
Scripts
@cczona
Tuesday, October 23, 12
Browsers - are the most common by far. But many other user-agent types exist as well.
Crawlers/spiders/robots - automated site browser. Some examples include search engine indexers
(Googlebot), archivers (Archive.org aka "The Internet Archive"), testing tools (curl-loader), monitoring
services (uptime)
25. User-Agent examples
Browser
Crawlers/spiders/robots
Screen readers
Programs
Scripts
@cczona
Tuesday, October 23, 12
Screenreaders - assistive devices that read content aloud
Programs - such as cURL & Wget
26. User-Agent examples
Browser
Crawlers/spiders/robots
Screen readers
Programs
Scripts
@cczona
Tuesday, October 23, 12
Scripts - these can be written in any language capable of using the HTTP scheme. Which is pretty much any
common one you can imagine. Perl, PHP, Ruby, and Python have HTTP libraries that make it easy to submit
requests and accept responses.
27. Method
@cczona
Tuesday, October 23, 12
The User-Agent submits the request using one of the defined HTTP methods.
28. HTTP Methods
GET
POST
HEAD
PUT
DELETE
OPTIONS
TRACE
@cczona
Tuesday, October 23, 12
GET - is the most common method. The request is communicated primarily via the URL itself
29. HTTP Methods
GET
POST
HEAD
PUT
DELETE
OPTIONS
TRACE
@cczona
Tuesday, October 23, 12
POST - is also common, typically used via an HTML form action. The request is communicated via the URL and the form's data is
communicated via the request's headers. GETs have a length limit, so POST is a handy alternative when your form might receive a whole
lot of data.
30. HTTP Methods
GET
POST
HEAD
PUT
DELETE
OPTIONS
TRACE
@cczona
Tuesday, October 23, 12
HEAD - the request is communicated the same as a GET, so via URL. But HEAD signals that the response should only contain headers,
not body. Whereas the response to a GET is both.
PUT, DELETE, OPTIONS & TRACE have not been widely adopted.
31. Headers
@cczona
Tuesday, October 23, 12
I've been mentioning headers. The obvious question is: _what the heck is a header_?
32. Headers are metadata
@cczona
Tuesday, October 23, 12
A header is metadata about the request or response.
33. Headers Firefox Web Developer Tools
@cczona
Tuesday, October 23, 12
Browsers' developer tools can display headers. Let's take a look at these.
Notice the structure: each line is one key/value pair. The line starts with a key, then a colon,
then the value.
34. Headers Chrome Developer Tools
@cczona
Tuesday, October 23, 12
35. Headers Safari Developer Tools
@cczona
Tuesday, October 23, 12
36. Request headers
Include information about:
• request
• user's environment
• user-agent's capabilities
• user-agent's identity*
@cczona
Tuesday, October 23, 12
Request headers originate at the user-agent. Its headers include information about:
request
user's environment
user-agent's capabilities
user-agent's identity*
37. Request headers
Include information about:
• request
• user's environment
• user-agent's capabilities
• user-agent's identity*
@cczona
Tuesday, October 23, 12
There's a huge caveat here. The "user-agent" header frequently is a lie. There's a long story as to why. Never, ever
depend on the user-agent header to be accurate.
39. DNS Distributed map of domains to IPs
@cczona
Tuesday, October 23, 12
When a request is submitted by the user, the user-agent first has to determine where to
submit it. The URL's host part can be a domain name or an IP address. But the user-agent
actually can only submit to an IP address. This is where DNS comes in. A domain name
service (DNS) is a directory of domains and their IP addresses.
40. Nameservers
For example:
NS1.Dreamhost.com
NS2.Dreamhost.com
NS3.Dreamhost.com
NS4.Dreamhost.com
@cczona
Tuesday, October 23, 12
If you've ever registered a domain name, you may have noticed that the domain was either
configured with a default set of nameservers, or you had to provide the set of nameservers
for your web hosting provider.
For example (slide)
41. Nameservers
For example:
NS1.dreamhost.com
NS2.dreamhost.com
NS3.dreamhost.com
NS4.dreamhost.com
@cczona
Tuesday, October 23, 12
Nameservers are a secondary directory necessary for DNS resolution. They're responsible for
keeping track of IP addresses where each of _your domain's_ services are located.
43. Host Servers
@cczona
Tuesday, October 23, 12
The host server is not the web server. Rather, it's the hardware and operating system that
house the web server, and potentially houses other infrastructure used by the site. So for
example: OSX, Linux, Unix, Windows Server, or Novel NetWare.
44. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
The host server passes the request off to the web server.
45. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
47. Web Server
@cczona
Tuesday, October 23, 12
A web server is software that's capable of taking an HTTP request as input, and of delivering
an HTTP response as output.
The most popular web servers are Apache, Microsoft IIS, and "engine X" (nginx).
Let's take a look at the web server's log of our request.
48. 108.75.73.197 - - [20/Oct/2012:13:58:09 +0000]
"GET /blog/?search&term=Confident&month=October
HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML,
like Gecko) Chrome/22.0.1229.94 Safari/537.4"
Web Server Request Log
@cczona
Tuesday, October 23, 12
Looks pretty much like gibberish, right? Ah, but wait....
49. 108.75.73.197 - - [20/Oct/2012:13:58:09 +0000]
"GET /blog/?search&term=Confident&month=October
HTTP/1.1" 200 11633 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2) AppleWebKit/537.4 (KHTML,
like Gecko) Chrome/22.0.1229.94 Safari/537.4"
Web Server Request Log
@cczona
Tuesday, October 23, 12
Now see how many parts you can name already. What's the request method? The IP address?
The path? And query string? Any thoughts on what that purple one is?
50. Web Server Redirect
@cczona
Tuesday, October 23, 12
The web server interprets the request & determines how it should be handled.
It looks at the request path and checks whether it matches any redirection directives.
These may be temporary (e.g. "offline for maintenance"), or they may be permanent (e.g.
"blog" is now "old-blog").
51. Web Server Redirect
@cczona
Tuesday, October 23, 12
In lines 1 & 2, the web server gracefully handles files that have moved.
In lines 4 & 5, it uses a regular expression to match any files that match the pattern.
52. Web Server Redirect
@cczona
Tuesday, October 23, 12
Maybe you've seen something similar to this WordPress panel. It looks like some cool fancy
feature. Naw. It's just using a web server redirect. Regular expressions are awesome.
53. Web Server Redirect (example)
@cczona
Tuesday, October 23, 12
Here's the redirect. Every page and post of your WordPress blog is actually one script file +
an HTTP redirect. That's it. It only look like infinite pages. Why do this? Flexibility, while
enforcing consistency. You could change every URL without breaking a single link. Nice.
54. Web Server Redirect (example)
@cczona
Tuesday, October 23, 12
Once a web server has finished doing redirects, it checks whether the new path matches a
corresponding path in the web server's directory hierarchy. If it doesn't match, or there's a
problem reading in the content, error time!
55. Web Server 404 Page Not Found
@cczona
Tuesday, October 23, 12
Aww, sad panda.
Well, sad Octocat.
56. Web Server 500 Internal Server Error
@cczona
Tuesday, October 23, 12
Uh-oh. Headache Puppy says badness happened.
57. Response
@cczona
Tuesday, October 23, 12
When it does match, now the magic happens.
58. HTML
@cczona
Tuesday, October 23, 12
The web server reads in whatever content the file provides. Text, html, image, output from a
script...whatever.
59. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
Passing the request down to a script gives us opportunity to generate some (or all) content
dynamically. Often this involves using values passed in via the URL and/or POSTed data.
60. Script
@cczona
Tuesday, October 23, 12
This example script generates output, starting with a response header (line 1), then declaring
the end of headers (line 3), then outputting response body (starting at line 5).
61. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
A script may also want to call on other services, such as a database.
62. Example Response Response Message
@cczona
Tuesday, October 23, 12
Once the web server receives output suitable for delivery in an HTTP response, it send it on
its way.
63. Response headers
Mostly describe:
• Content
• How user-agent should handle it
First line is HTTP version and final status
HTTP/1.1 200 OK
@cczona
Tuesday, October 23, 12
The response likewise has headers. Most of these describe the content and how it should be
handled by the user-agent.
Notice that the first line of the response states the final status of the request. If things went
well, that's "200 OK".
64. Example Response Response Message
@cczona
Tuesday, October 23, 12
The user agent receives the response, and interprets it. In this case, the header had told the
user-agent to interpret our response's body as HTML.
65. Back to the Frontend
@cczona
Tuesday, October 23, 12
Now it's the user-agent's turn to process what it's received. It parses the content, to
determine how to render it for the user. Cycle complete!
66. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
Now it can determine what _other_ assets are needed. The user-agent fires off requests for
those as well. JavaScript, CSS, images, etc. A bunch more cycles!
67. HTTP/1.1 200 OK
response
request
GET /something
Tuesday, October 23, 12
When the user-agent has received everything it needs, its final steps are to layout the
content, and render it for the user.
69. Optimizations
@cczona
Tuesday, October 23, 12
So far we've seen that a pretty simple set of web infrastructure. But much more technology
can optionally be involved.
70. Optimizing
AJAX Minification
Caching Sprites
Expires headers Database indexes, operations,
Compression joins
Proxy server Partials
Content delivery network (CDN) REST/APIs
@cczona
Tuesday, October 23, 12
You may have noticed that there's a lot of potential here to make a site seem sluggish. So
what we learn from the HTTP request-response cycle is that there are many opportunities to
optimize for fast performance.
71. Stay tuned...
@cczona
Tuesday, October 23, 12
Luckily, Estelle will be talking next about a bunch of methods for optimizing performance!
72. Thank you!
Any questions?
@cczona
Tuesday, October 23, 12
Any questions for me?
73. Credits
• Networking icons by http://icons.iconarchive.com
• Illustrations from "HTTP: The Definitive Guide"
by David Gourley & Brian Totty
• Header screenshots from Safari Developer Tools, Chrome Developer Tools, and
Firefox Web Developer Tools
• DNS diagram from HowStuffWorks.com
• 404 Page Not Found from http://www.github.com & htttp://mozilla.org
• 500 Internal Server from http://www.tumblr.com/tagged/internal-server-error
@cczona
Tuesday, October 23, 12
74. Resources
• "HTTP: The Definitive Guide" by David Gourley & Brian Totty (O'Reilly)
• http://dev.opera.com/articles/view/http-basic-introduction/
• http://dev.opera.com/articles/view/http-lets-get-it-on/
• http://dev.opera.com/articles/view/http-response-codes/
• http://www.w3.org/community/webed/wiki/How_does_the_Internet_work
• http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
• http://en.wikipedia.org/wiki/Representational_state_transfer
• http://en.wikipedia.org/wiki/URI_scheme
• http://en.wikipedia.org/wiki/Ajax_(programming)
@cczona
Tuesday, October 23, 12