server-side, part 2 - cscie12.dce.harvard.edu 1.1 methods zget zpost zhead zput zdelete ztrace...

Server-side, Part 2 April 22, 2009

Harvard University Division of Continuing Education

Extension School

Course Web Site: http://cscie12.dce.harvard.edu/

Copyright 1998-2009 David P. Heitmeyer

Instructor email: [email protected] Course staff email: [email protected]

Table of Contents | All Slides | Link List | CSCI E-12

of 73Server-side, Part 2

4/22/2009http://localhost:8080/cocoon/projects/cscie12/slides/20090422/handout.html

Remaining Lectures: Grab BagPlease use the Lecture Feedback Form to indicate any topics you would like to hear more about in the last two lectures.

Here are some topics from my list:

Web Content Management Systems (Web CMS) Adobe Contribute, Dreamweaver, and using Templates to simplify editing Server-side systems: Drupal, MovableType, etc.

Web Analytics Security and Privacy

SSL/TLS (i.e. https) XSS (Cross Site Scripting) Phishing P3P and PICS

Hosting and Content Storage and Delivery Shared, Virtual, Dedicated, "Cloud" CDN (Content Delivery Networks)

Web 2.0 HTML 5 and XHTML 2



HyperText Transfer ProtocolGET /

view plain copy to clipboard print ?

1. morpheus% telnet www.npr.org 80 2. Trying 216.35.221.77... 3. Connected to www.npr.org. 4. Escape character is '^]'. 5. GET / HTTP/1.1 6. Host: www.npr.org 7. 8. HTTP/1.1 200 OK 9. Date: Tue, 10 Apr 2008 20:07:33 GMT

10. Server: Apache 11. Set-Cookie: Apache=140.247.197.241.289451144786054516; path=/ 12. Cache-Control: max-age=0 13. Expires: Tue, 10 Apr 2008 20:07:33 GMT 14. Transfer-Encoding: chunked 15. Content-Type: text/html 16. 17. 76c 18. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 19. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 20. <html xmlns="http://www.w3.org/1999/xhtml"> 21. <head> 22. <title>NPR - National Public Radio - News, Arts, World, US.</title> 23.  24. </html> 25. 26.

The Internet



TCP/IP: Transmission Control Protocol/Internet Protocol The TCP/IP Internet Protocol Suite

IP (Internet Protocol): provides basic communication rules

TCP (Transmission Control Protocol): provides additional facilities

IP Addresses

The name www.fas.harvard.edu resolves to the IP address of 140.247.34.66. You can use the tool host or dig to lookup the IP to name or name to IP number.

1. [dheitmey@morpheus dheitmey]$ host www.fas.harvard.edu 2. www.fas.harvard.edu has address 140.247.34.66 3. [dheitmey@morpheus dheitmey]$ host morpheus.dce.harvard.edu 4. morpheus.dce.harvard.edu has address 140.247.197.241 5. [dheitmey@morpheus dheitmey]$ host www.npr.org 6. www.npr.org has address 216.35.221.77 7.



Hostnamescscie12.dce.harvard.edu

.edu harvard.edu dce.harvard.edu cscie12.dce.harvard.edu

Numbers

As of January 2009, Internet Domain Survey reports 625,226,456 hosts in the Domain Name Service (Source: Internet Software Consortium ( http://www.isc.org/).



IP Addresses and HostnamesName: morpheus.dce.harvard.edu IP Address: 140.247.197.241 Aliases:

cscie12.dce.harvard.edu cscie153.dce.harvard.edu

Domain Name System (DNS)

A hierarchical, distributed naming system.

Resolving: cscie12.dce.harvard.edu

1. Request from User Machine to User's Primary DNS Server 2. Request from User's Primary DNS to Root Server 3. Request to Root Server for ".edu" namespace (or .com, .gov, .net, .uk, .jp, etc.) 4. Request to Primary "harvard.edu" DNS Server 5. Request to Primary "dce.harvard.edu" DNS Server



Domain Names: Top Level Domains (TLD)TLDs are managed by the Internet Assigned Numbers Authority (IANA) A sample listing is below:

.com

.org

.edu

.gov

.mil

.net 2-Letter Country Codes

.us

.de

.uk complete list

More Top Level Domains.

.aero

.biz

.coop

.info

.museum

.name

.pro

List of all Generic Top Level Domains



whois

Domain Name: HARVARD.EDU Registrant: Harvard University Network Operations Center 60 Oxford Street Cambridge, MA 02138 UNITED STATES Administrative Contact: Jay Tumas Network Operations Manager Harvard University Network Operations Center 60 Oxford Street Cambridge, MA 02138 UNITED STATES (617) 496-8500 [email protected] Technical Contact: Jay Tumas Network Operations Manager Harvard University Network Operations Center 60 Oxford Street Cambridge, MA 02138 UNITED STATES (617) 496-8500 Name Servers: NS1.HARVARD.EDU 128.103.200.101 NS2.HARVARD.EDU 128.103.1.1 NS3.HARVARD.EDU 128.119.3.170 Domain record activated: 27-Jun-1985 Domain record last updated: 17-Sep-2004 Domain expires: 31-Jul-2009



Getting Your Own Domain1. Domain Name

Buy the domain through a "registrar" Provide name servers

2. Hosting

Sidebar: Hosting Companies

DreamHost Web.com Go Daddy

See also, The "Top 10 Best Web Hosting Companies" Reviewed



Clients and Serversclient-server computing The interaction between two programs when they communicate across a network. A program at one site sends a request to a program at another site and awaits a response. The requesting program is called a client; the program satisfying the request is called the server. (definition from The Internet Book, 2nd edition by Douglas E. Comer)

Application Layer of Network

HTTP (default port 80) FTP (port 21) SMTP (port 25) telnet (port 23) ssh (port 22)



HyperText Transfer ProtocolSpecifies the grammar of a conversation between an HTTP-client (Web Browser) and an HTTP-server (Web Server) is to take place.

GET / view plain copy to clipboard print ?

1. GET / HTTP/1.1 2. Host: www.npr.org 3. 4. HTTP/1.1 200 OK 5. Date: Tue, 10 Apr 2008 20:07:33 GMT 6. Server: Apache 7. Set-Cookie: Apache=140.247.197.241.289451144786054516; path=/ 8. Cache-Control: max-age=0 9. Expires: Tue, 10 Apr 2008 20:07:33 GMT

10. Transfer-Encoding: chunked 11. Content-Type: text/html 12. 13. 76c 14. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 15. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 16. <html xmlns="http://www.w3.org/1999/xhtml"> 17. <head> 18. <title>NPR - National Public Radio - News, Arts, World, US.</title> 19.  20. </html>

HTTPExample Request and Loading of a Web Page

As an example, an XHTML page with 7 images and an external CSS file and an external Javascript file, the client would make 10 separate requests (1 request for the XHTML resource, 1 request for each of the seven images, 1 request for the CSS, and 1 request for the JS).

Client Server

HTTP Request for "example.html"

HTTP Response with "example.html" content

Parses through XHTML and determines what other requests it needs to make.

HTTP Request for CSS document

HTTP Response for CSS document

Parses through CSS and determines what other requests it needs to make.

HTTP Request for Javascript document

HTTP Response for Javascript document

HTTP Request for image 1

HTTP Response for image 1













Render page



HTTPHTTP is Stateless

Each requested resource is a separate, independent, request to the server -- it is a stateless protocol.

HTTP Versions

W3C and Internet Engineering Task Force (IETF) oversees the Hypertext Transfer Protocol.

HTTP 1.0 (1996) HTTP 1.1 (1999) Extensions to HTTP

WebDAV

An HTTP Conversation

Client Request METHOD Resource HTTP Version Client Generated Headers Request Body

Server Response Status Line Server Generated Headers Data

HTTP 1.1 Methods

GET POST HEAD PUT DELETE TRACE OPTIONS



HTTP Response Codes

HTTP 1.1 status codes

Common ones:

200 OK 301 Moved permanently 302 Moved temporarily 304 Not modified 403 Forbidden 404 Not found 500 Internal server error

The complete list:

100 Continue 101 Switching protocols 200 OK 201 Created 202 Accepted 203 Non-authoritative information 204 No content 205 Reset content 206 Partial content 300 Multiple choices 301 Moved permanently 302 Moved temporarily 303 See other 304 Not modified 305 Use proxy 400 Bad request 401 Unauthorized 402 Payment required 403 Forbidden 404 Not found 405 Method not allowed 406 Not acceptable 407 Proxy authentication required 408 Request timeout 409 Conflict 410 Gone 411 Length required 412 Precondition failed 413 Request entity too large 414 Request-URI too long 415 Unsupported media type 500 Internal server error 501 Not implemented 502 Bad gateway 503 Service unavailable 504 Gateway timeout 505 HTTP version not supported

Code Range Meaning100's Informational200's Success300's Redirected400's Request Incomplete500's Server Error



Common Request HeadersAccept Accept-Language Authorization Cookie Host If-Modified-Since Referer User-Agent



Sample Request Header ValuesView some of the Headers your browser is sending to the server

Mozilla Firefox

Opera 9

Internet Explorer 7

Amaya

Lynx

wget

HTTP_ACCEPT text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5HTTP_ACCEPT_CHARSET ISO-8859-1,utf-8;q=0.7,*;q=0.7HTTP_ACCEPT_ENCODING gzip,deflateHTTP_ACCEPT_LANGUAGE en-us,en;q=0.5HTTP_CONNECTION keep-aliveHTTP_COOKIE __utma=76898816.1019426635.1173814303.1173814303.1173814303.1;

__utmz=76898816.1173814303.1.1.utmccn=(referral)|utmcsr=localhost:8080|utmcct=/cocoon/projects/cscie12/slides/20070313/slide53.html|utmcmd=referral; nde-textsize=16px

HTTP_HOST cscie12.dce.harvard.eduHTTP_KEEP_ALIVE 300HTTP_REFERER http://localhost:8080/cocoon/projects/cscie12/slides/20070410/slide16.htmlHTTP_USER_AGENT Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3

HTTP_ACCEPT text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap,*/*;q=0.1

HTTP_ACCEPT_CHARSET iso-8859-1, utf-8, utf-16, *;q=0.1HTTP_ACCEPT_ENCODING deflate, gzip, x-gzip, identity, *;q=0HTTP_ACCEPT_LANGUAGE en-US,en;q=0.9HTTP_CONNECTION Keep-AliveHTTP_HOST cscie12.dce.harvard.eduHTTP_USER_AGENT Opera/9.01 (Windows NT 5.1; U; en)

HTTP_ACCEPT image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, */*

HTTP_ACCEPT_ENCODING gzip, deflateHTTP_ACCEPT_LANGUAGE en-usHTTP_CONNECTION Keep-AliveHTTP_HOST cscie12.dce.harvard.eduHTTP_UA_CPU x86HTTP_USER_AGENT Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727;

.NET CLR 3.0.04506.30)

HTTP_ACCEPT */*;q=0.1,image/svg+xml,application/mathml+xml,application/xhtml+xmlHTTP_ACCEPT_ENCODING *,gzipHTTP_CONNECTION TE,Keep-AliveHTTP_HOST cscie12.dce.harvard.eduHTTP_TE trailers,deflateHTTP_USER_AGENT amaya/9.51 libwww/5.4.0

HTTP_ACCEPT text/html, text/plain, audio/mod, image/*, application/msword, application/pdf, application/postscript, application/x-java-jnlp-file, text/sgml, video/mpeg, */*;q=0.01

HTTP_ACCEPT_LANGUAGE enHTTP_HOST cscie12.dce.harvard.eduHTTP_USER_AGENT Lynx/2.8.5dev.16 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.7a

HTTP_ACCEPT */*HTTP_CONNECTION Keep-AliveHTTP_HOST cscie12.dce.harvard.eduHTTP_USER_AGENT Wget/1.9+cvs-stable (Red Hat modified)



Experimenting with HTTPViewing HTTP Request and Response Headers

Firefox Extension - Firebug

Firefox Extension - Live HTTP Headers

Command Line

telnet lwp-request Documentation:

morpheus% man lwp-request morpheus% lwp-request -h



An example:

morpheus% lwp-request -USed http://www.harvard.edu/



HTTP Method: HEADhttp://cscie12.dce.harvard.edu/http/raspberry.gif cscie12.dce.harvard.edu is 140.247.197.241 we must explicitly use port 80 Note the two "returns" after we are done with the HTTP request


1. HEAD /http/raspberry.gif HTTP/1.1 2. Host: cscie12.dce.harvard.edu <return> 3. <return> 4. HTTP/1.1 200 OK 5. Date: Tue, 10 Apr 2008 20:23:14 GMT 6. Server: Apache/2.2 (Fedora) 7. Last-Modified: Wed, 06 Apr 2005 19:30:42 GMT 8. ETag: "461fb8-348c-a0f67c80" 9. Accept-Ranges: bytes

10. Content-Length: 13452 11. Connection: close 12. Content-Type: image/gif 13. 14. Connection closed by foreign host. 15.



Response 404Sometimes a file is not there...


1. HEAD /http/blueberry.gif HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. 4. HTTP/1.1 404 Not Found 5. Date: Tue, 10 Apr 2008 20:24:27 GMT 6. Server: Apache/2.2 (Fedora) 7. Connection: close 8. Content-Type: text/html; charset=iso-8859-1 9.

10. Connection closed by foreign host. 11.



HTTP Header: HostProblem: "Infinite" domain names; finite IP addresses.

Solution: "Virtual Hosts"

Example: all of the following names map to 140.247.197.241

morpheus.dce.harvard.edu cscie12.dce.harvard.edu cscie153.dce.harvard.edu cscisl.dce.harvard.edu cscis12.dce.harvard.edu csci12.dce.harvard.edu

Host Header

This is required for HTTP 1.1 requests.


1. HEAD /http/raspberry.gif HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. 4. HTTP/1.1 200 OK 5. Date: Tue, 10 Apr 2008 20:23:14 GMT 6. Server: Apache/2.2 (Fedora) 7. Last-Modified: Wed, 06 Apr 2005 19:30:42 GMT 8. ETag: "461fb8-348c-a0f67c80" 9. Accept-Ranges: bytes

10. Content-Length: 13452 11. Connection: close 12. Content-Type: image/gif 13. 14. Connection closed by foreign host. 15.



HTTP Redirecthttp://www.fas.harvard.edu/ http://www.fas.harvard.edu/home/


1. HEAD / HTTP/1.1 2. Host: www.fas.harvard.edu 3. 4. HTTP/1.1 301 Moved Permanently 5. Date: Wed, 06 Apr 2008 20:11:43 GMT 6. Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g mod_perl/1.24 7. Location: http://www.fas.harvard.edu/home/ 8. Content-Type: text/html; charset=iso-8859-1 9.

10.



Media Types (MIME Types)How a Browser Knows What Kind of File it is Getting

Multipurpose Internet Mail Extensions (media types). Server will return a media type to client. Client will handle the media appropriately. Some common media types are:

text/html text/css image/jpeg image/png image/gif application/pdf application/msword application/vnd.ms-excel All media types listed in /etc/mime.types on morpheus

More information about MIME Types is available.

Questions:

How does the server know the media type? How does the client know the media type? How does the client know "what to do with" the file?



Content NegotiationResource can be in multiple formats and languages. Client preferences can determine which resource is returned.

Content Negiation Resources

Apache Documentation: Content Negotiation ApacheWeek: Content Negotiation Explained



Content Negotiation: MIME TypesA file listing :

raspberry.gif raspberry.jpg raspberry.png raspberry

The HTTP Transaction:

1. morpheus% ls -l raspberry* 2. 16 -rw-r--r-- 1 e12 e12 13452 Apr 6 15:30 raspberry.gif 3. 16 -rw-r--r-- 1 e12 e12 16255 Apr 6 15:30 raspberry.jpg 4. 12 -rw-r--r-- 1 e12 e12 8899 Apr 6 15:30 raspberry.png


1. HEAD /http/raspberry HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. 4. HTTP/1.1 200 OK 5. Date: Tue, 10 Apr 2008 20:39:20 GMT 6. Server: Apache/2.2 (Fedora) 7. Content-Location: raspberry.png 8. Vary: negotiate,accept 9. TCN: choice

10. Last-Modified: Wed, 06 Apr 2005 19:30:42 GMT 11. ETag: "461fba-22c3-a0f67c80;4edcb400" 12. Accept-Ranges: bytes 13. Content-Length: 8899 14. Connection: close 15. Content-Type: image/png 16. 17. Connection closed by foreign host.



Content Negotiation: MIME TypesClient specifies MIME Types it accepts through HTTP "Accept" header.


1. HEAD /http/raspberry HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. Connection: close 4. Accept: image/jpeg 5. 6. HTTP/1.1 200 OK 7. Date: Tue, 10 Apr 2008 20:40:17 GMT 8. Server: Apache/2.2 (Fedora) 9. Content-Location: raspberry.jpg

10. Vary: negotiate,accept 11. TCN: choice 12. Last-Modified: Wed, 06 Apr 2005 19:30:42 GMT 13. ETag: "461fb9-3f7f-a0f67c80;4edcb400" 14. Accept-Ranges: bytes 15. Content-Length: 16255 16. Connection: close 17. Content-Type: image/jpeg



Content Negotiation: Languagelang.html English Version French Version German Version

A file listing:

HTTP Transaction:

1. morpheus% ls -l lang* 2. -rw-r--r-- 1 e12 e12 191 Apr 6 15:30 lang.html.de 3. -rw-r--r-- 1 e12 e12 193 Apr 6 15:30 lang.html.en 4. -rw-r--r-- 1 e12 e12 191 Apr 6 15:30 lang.html.fr


1. HEAD /http/lang HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. Connection: close 4. 5. HTTP/1.1 200 OK 6. Date: Tue, 10 Apr 2008 20:41:02 GMT 7. Server: Apache/2.2 (Fedora) 8. Content-Location: lang.html.en 9. Vary: negotiate,accept-language

10. TCN: choice 11. Accept-Ranges: bytes 12. Content-Length: 193 13. Connection: close 14. Content-Type: text/html; charset=UTF-8 15. Content-Language: en 16. 17. Connection closed by foreign host.



Content Negotiation: Language

Content Negotiation: Gotcha

Permissions must be set to rwxr-xr-x on directories.

Why?


1. HEAD /http/lang HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. Connection: close 4. Accept-Language: de 5. 6. HTTP/1.1 200 OK 7. Date: Tue, 10 Apr 2008 20:44:16 GMT 8. Server: Apache/2.2 (Fedora) 9. Content-Location: lang.html.de

10. Vary: negotiate,accept-language 11. TCN: choice 12. Accept-Ranges: bytes 13. Content-Length: 191 14. Connection: close 15. Content-Type: text/html; charset=UTF-8 16. Content-Language: de 17. 18. Connection closed by foreign



HTTP: trailing 'slash' for directoriesview plain copy to clipboard print ?

1. HEAD /images HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. 4. HTTP/1.1 301 Moved Permanently 5. Date: Tue, 10 Apr 2008 20:48:42 GMT 6. Server: Apache/2.2 (Fedora) 7. Location: http://cscie12.dce.harvard.edu/images/ 8. Content-Type: text/html 9.

10. HEAD /images/ HTTP/1.1 11. Host: cscie12.dce.harvard.edu 12. 13. HTTP/1.1 200 OK 14. Date: Tue, 10 Apr 2008 20:48:42 GMT 15. Server: Apache/2.2 (Fedora) 16. Connection: close 17. Content-Type: text/html; charset=UTF-8



Connection: keep-aliveAllows multiple HTTP requests to be made over the same TCP/IP connection.


1. HEAD /home/ HTTP/1.1 2. Host: www.fas.harvard.edu 3. Connection: keep-alive 4. 5. HTTP/1.1 200 OK 6. Date: Tue, 13 Apr 2008 19:24:16 GMT 7. Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g mod_perl/1.24 8. Keep-Alive: timeout=15, max=100 9. Connection: Keep-Alive

10. Content-Type: text/html 11. 12. HEAD /home/images/ HTTP/1.1 13. Host: www.fas.harvard.edu 14. Connection: close 15. 16. HTTP/1.1 200 OK 17. Date: Tue, 13 Apr 2008 19:24:34 GMT 18. Server: Apache 19. Connection: close 20. Content-Type: text/html 21. 22. Connection closed by foreign host.



Caching Related HeadersLocal cache and Proxy-server cache

If-Modified-Since Age Expires Last-Modified Cache-Control ETag



HTTP Example



If-Modified Sinceview plain copy to clipboard print ?

1. GET /images/asf_logo.gif HTTP/1.1 2. Host: www.apache.org 3. Keep-Alive: 300 4. Connection: keep-alive 5. If-Modified-Since: Wed, 15 Apr 2008 18:00:00 GMT 6. 7. HTTP/1.x 304 Not Modified 8. Cache-Control: max-age=86400 9. Date: Wed, 22 Apr 2009 19:11:16 GMT

10. ETag: "12b2a10-1c6f-3eb9a194b7b00" 11. Server: Apache/2.2.9 (Unix) 12. Expires: Thu, 23 Apr 2009 19:11:16 GMT 13. Client-Date: Wed, 22 Apr 2009 19:11:16 GMT 14. Client-Peer: 140.211.11.130:80 15. Client-Response-Num: 1



Proxy Servers



HTTP CookiesHTTP is a stateless protocol. Cookies provide a mechanism to "maintain state".

Cookie Central: The Unofficial Cookie FAQ http://www.cookiecentral.com/faq/ http://www.cookiecentral.com/

Maintaining State with Cookies

HTTP State Management Mechanism http://www.ics.uci.edu/pub/ietf/http/rfc2109.txt Cookie Central: The Unofficial Cookie FAQ http://www.cookiecentral.com/faq/ http://www.cookiecentral.com/ Persistent Client State HTTP Cookies http://www.netscape.com/newsref/std/cookie_spec.html



Cookie ExampleServer returns cookie to HTTP client ("Set-Cookie" response header) HTTP client returns cookie to server ("Cookie" request header)

ESPN Cookies

Set-Cookie: SWID=C8F9AF31-F170-42BF-9471-50A95DA24C17; path=/; expires=Tue, 10-Apr-2027 03:20:59 GMT; domain=.go.com;

Set-Cookie: DE2=dXNhO21hO2NhbWJyaWRnZTt0MTs1OzQ7NDs1MDY7MDQyLjM4MDstMDcxLjEzNTs4NDA7MjI7ODg5OzY7Cg==; path=/; expires=Tue, 17 Apr 2008 03:00:00 GMT; domain=.go.com


1. morpheus% lwp-request -USed http://www.espn.com/ 2. GET http://espn.go.com/ 3. User-Agent: lwp-request/2.07 4. 5. GET http://www.espn.com/ --> 301 Moved Permanently 6. GET http://espn.go.com/ --> 200 OK 7. Cache-Control: no-cache 8. Date: Tue, 10 Apr 2008 03:20:58 GMT 9. Pragma: no-cache

10. From: SPORTBARWEB08 11. Accept-Ranges: bytes 12. ETag: "802e571f7bc71:1762" 13. Server: Microsoft-IIS/5.0 14. Vary: Accept-Encoding 15. Content-Length: 122217 16. Content-Type: text/html; charset=iso-8859-1 17. Content-Type: text/html; charset=windows-1252 18. Last-Modified: Tue, 10 Apr 2008 03:19:21 GMT 19. Cache-Expires: Tue, 10 Apr 2008 03:24:22 GMT 20. Client-Date: Tue, 10 Apr 2008 03:21:02 GMT 21. Client-Peer: 198.105.193.43:80 22. Client-Response-Num: 1 23. P3P: CP="CAO DSP COR CURa ADMa DEVa TAIa PSAa PSDa IVAi IVDi CONi OUR SAMo OTRo BUS PHY ONL UNI PUR COM NAV INT DEM CNT STA PRE" 24. Refresh: 3600 25. Set-Cookie: SWID=C8F9AF31-F170-42BF-9471-50A95DA24C17; path=/; expires=Tue, 10-Apr-2027 03:20:59 GMT; domain=.go.com; 26. Set-

Cookie: DE2=dXNhO21hO2NhbWJyaWRnZTt0MTs1OzQ7NDs1MDY7MDQyLjM4MDstMDcxLjEzNTs4NDA7MjI7ODg5OzY7Cg==; path=/; expires=Tue, 17 Apr 2008 03:00:00 GMT; domain=



Cookie Properties/Attributesname expires domain path secure

HTTP State Management Mechanism, RFC 2965

RFC 2109, February 1997 RFC 2965, October 2000

name comment comment URL discard domain max-age path port secure version

Additional Cookie Notes

Client: 300 total cookies 4 kb per cookie 20 cookies per server or domain



Cookie Example: Server Sets a CookieForm that will set a Cookie: http://cscie12.dce.harvard.edu/http/cookie.cgi

Set-Cookie HTTP Response Header:

Set-Cookie: YourName=David%20P.%20Heitmeyer; domain=cscie12.dce.harvard.edu; path=/http/; expires=Fri, 13-May-2008 18:05:04 GMT


1. GET /http/cookie.cgi?name=David%20P.%20Heitmeyer HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. Connection: close 4. 5. HTTP/1.1 200 OK 6. Connection: close 7. Date: Wed, 13 Apr 2008 18:05:04 GMT 8. Server: Apache/2.0.49 (Fedora) 9. Content-Type: text/html; charset=ISO-8859-1

10. Client-Date: Wed, 13 Apr 2008 18:05:04 GMT 11. Client-Peer: 140.247.197.241:80 12. Client-Response-Num: 1 13. Client-Transfer-Encoding: chunked 14. Set-Cookie: YourName=David%20P.%20Heitmeyer; \ 15. domain=cscie12.dce.harvard.edu; \ 16. path=/http/; \ 17. expires=Fri, 13-May-2008 18:05:04 GMT 18. 19. <?xml version="1.0" encoding="iso-8859-1"?> 20. <!DOCTYPE html 21. PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 22. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 23. <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"><head><title>Form</title> 24. </head><body> 25. <h1>Hello, David P. Heitmeyer</h1> 26. </body></html> 27. Connection closed by foreign host. 28.



Cookie Example: Returning a CookieForm that will set a Cookie: http://cscie12.dce.harvard.edu/http/cookie.cgi


1. GET /http/cookie.cgi HTTP/1.1 2. Cookie: YourName=David%20P.%20Heitmeyer 3. Host: cscie12.dce.harvard.edu 4. Connection: close 5. 6. HTTP/1.1 200 OK 7. Connection: close 8. Date: Wed, 13 Apr 2008 18:11:40 GMT 9. Server: Apache/2.0.49 (Fedora)

10. Content-Type: text/html; charset=ISO-8859-1 11. Client-Date: Wed, 13 Apr 2008 18:11:40 GMT 12. Client-Peer: 140.247.197.241:80 13. Client-Response-Num: 1 14. Client-Transfer-Encoding: chunked 15. 16. <?xml version="1.0" encoding="iso-8859-1"?> 17. <!DOCTYPE html 18. PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 19. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 20. <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"> 21. <head><title>Form</title></head><body> 22. <h1>Hello, David P. Heitmeyer</h1> 23.



Your CookiesFirefox Webdeveloper Toolbar has a "Cookies" section.

Mozilla Cookie Manager



Cookies and Session IDsA UserID or SessionID (a long character/number string that is uniquely assigned) is often stored in cookie. The SessionID is used as the key or identifier when storing information about the user or session.

For example, a user logs in to a site. If the username and password match, the server sets a cookie ("Set-Cookie") in the browser that contains a session id; the server also makes an entry in website database that maps the session id to the username. When the cookie is returned, the session id is read and the username is looked up in the database.



Google Cookie ExampleUsing Google's "Preference" page and setting:

Search Language preference to: English, French, German SafeSearch Filtering: Strict Filtering Number of Results: 50

The Cookie name is: PREF The Value is: ID=bb504f37cd318aa9:FF=1:LR=lang_en|lang_fr|lang_de:LD=en:NR=50:TM=1113416195:LM=1113416240:S=lurnF9ALm5Wg34rs

This cookie contains a session id as well as the values of certain preferences in a colon-separated data structure.



Cookies and Ad Tracking



Method: POSTForm that will set a Cookie: http://cscie12.dce.harvard.edu/http/cookie.cgi


1. POST /http/cookie.cgi HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. Content-Length: 10 4. Content-Type: application/x-www-form-urlencoded 5. 6. name=David 7. HTTP/1.1 200 OK 8. Date: Wed, 13 Apr 2008 19:31:11 GMT 9. Server: Apache/2.0.49 (Fedora)

10. Set-Cookie: YourName=David; domain=cscie12.dce.harvard.edu; path=/http/; expires=Fri, 13-May-2008 19:31:20 GMT 11. Content-Length: 319 12. Connection: close 13. Content-Type: text/html; charset=ISO-8859-1 14. 15. <?xml version="1.0" encoding="iso-8859-1"?> 16. <!DOCTYPE html 17. PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 18. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 19. <html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US"> 20. <head><title>Form</title> 21. </head><body> 22. <h1>Hello, David</h1>



WebDAV: an extension of HTTPWeb-based Distributed Authoring and Versioning

WebDAV Resources http://www.webdav.org/ From the WebDAV Resources :

WebDAV stands for "Web-based Distributed Authoring and Versioning". It is a set of extensions to the HTTP protocol which allows users to collaboratively edit and manage files on remote web servers.



HTTP ResourcesW3C HTTP http://www.w3.org/Protocols/ HTTP Pocket Reference http://www.oreilly.com/catalog/httppr/ by Clinton Wong (O'Reilly). Illustrated Guide to HTTP http://www.manning.com/hethmon/ by Paul Hethmon (Manning Publications; ISBN 0138582262) see sample chapters and resources online.

Other Readings:

W3C Recommendations Reduce 'World Wide Wait' http://www.w3.org/Protocols/NL-PerfNote.html Apache Week: HTTP version 1.1 http://www.apacheweek.com/features/http11 WebTechniques: HTTP 1.1: What's in it for Me? http://www.webtechniques.com/archives/1997/08/webm/ Cookie Central: The Unofficial Cookie FAQ http://www.cookiecentral.com/faq/ http://www.cookiecentral.com/



Web Server SoftwareApache HTTP Server Microsoft IIS Other

Google (Google Sites) nginx lighttpd

Netcraft Web Server Survey



Apache HTTP Server

Apache Software Foundation Apache HTTP Server Project

Apache 1.3 Apache 2.x

Apache ModulesPHP Perl Python many, many others



In this Unit: Configuring Apache with .htaccess files Custom Error Documents Server Side Includes Redirect Rewrite Directory Index Setting HTTP Expires Headers Compressing content before delivery



Apache Configuration OverviewServer Configuration (httpd.conf) Unless you are the server administrator, you generally will not have access to this account. On the DCE systems, you do not have read or write access to this file. Server configuration is read at server start or restart. Per Directory (.htaccess) Certain configuration directives for Apache can be placed within per-directory .htaccess files. .htaccess file is read on a per request basis.



.htaccess File Exampledocument root: /home/e12/htdocs filename: .htaccess location: /home/e12/htdocs/apache/.htaccess contents:

filename: status404.html location: /home/e12/htdocs/apache/status404.html

http://cscie12.dce.harvard.edu/apache/ZZZ.html

Apache Default 404 document:

Custom 404 document:

1. ErrorDocument 404 status404.html



Scope of .htaccess filesDirectives within .htaccess files apply to the directory that contains the .htaccess file and all its descendants.

Directives within the file, /home/e12/htdocs/.htaccess would apply to all files within and "under" the public_html directory for the user cscie12.

Directives within the file, /home/e12/htdocs/assignments/.htaccess would apply to all files within and "under" the public_html/assignments directory for the user cscie12.



Problems You Will Have with .htaccess files Internal Server Error Can't "see" the file Incorrect Permissions



Problems You will encounter when using .htaccess files (Internal Server Error 500)500 Internal Server Error If you see begin seeing 500 Internal Server Error responses from the server after you have created or edited an .htaccess file, the most likely cause of the problem is incorrect permissions and/or an error in the directive syntax.

Permissions on the .htaccess file are not set correctly. Just like HTML and image files, the server must be able to read the .htaccess file. The simplest way to allow that is to make your .htaccess file readable by "other".

Syntax Error. An error in the syntax of a directive the .htaccess file will result in a 500 Internal Server Error. In addition, correct usage of a directive that is not allowed in the .htaccess file will result in a 500 status code. Whether or not a directive is allowed depends upon the server configuration file (httpd.conf; AllowOverride) and the directive itself.

1. morpheus% pwd 2. /home/courses/j/h/jharvard/public_html 3. morpheus% ls -l .htaccess 4. -rw------- 1 jharvard founder 349 Nov 27 00:03 .htaccess 5. morpheus% chmod o+r .htaccess 6. morpheus% ls -l ~/public_html/.htaccess 7. -rw----r-- 1 jharvard founder 349 Nov 27 00:03 .htaccess



Problems You will encounter when using .htaccess files (Can't see the .htaccess file)You can't "see" your .htaccess file.

HTTP The web server is typically configured to deny requests for .htaccess files. For example, the file corresponding to the URL, http://cscie12.dce.harvard.edu/.htaccess exists and is readable by the Web server, but if we try to follow the link, we get a 403 Forbidden response. UNIX The ls command will not list files or directories that begin with a '.' (dot). In order to see the .htaccess file when you do a directory listing, use the -a (all) option: SFTP Sometimes your SFTP program will hide the "dot" files unless explicitly told to show them.



Apache Configuration SectionsConfiguration directives can be limited by using "sections", such as

Directory Location Files VirtualHost DirectoryMatch LocationMatch FilesMatch

Within .htaccess

Note that only Files and FilesMatch can be used within .htaccess files.

Examples:

Examples:

1. <Files .htaccess> 2. Order allow,deny 3. Deny from all 4. </Files>

1. # deny access to any tilde backup files 2. <Files *~> 3. Order allow,deny 4. Deny from all 5. </Files>



Custom Error Documents.htaccess

ErrorDocument directive Custom Error Responses

1. ErrorDocument 401 /apache/status401.html 2. ErrorDocument 403 /apache/status403.html 3. ErrorDocument 404 /apache/status404.html



Server Side IncludesApache Tutorial: Introductino to Server Side Includes

To process ".shtml", put the following in an .htaccess file:

Options +Includes AddType text/html .shtml AddOutputFilter INCLUDES .shtml

To process ".html" and ".htm", put the following in an .htaccess file:

Options +Includes AddOutputFilter INCLUDES .html AddOutputFilter INCLUDES .htm



HTTP RedirectPublish "clean" URLs, and redirect Site reorganization changes URL -- redirect old to new

Redirect Rewrite Meta http-equiv refresh

Redirecting Requests

HTTP Status Codes: 301 Moved permanently 302 Moved temporarily

Redirecting client requests can be very useful:

URL moves to a new location Do your part to Fight Linkrot! (Jakob Nielson's Alertbox, http://www.useit.com/alertbox/980614.html )

resource removed site structure is reorganized

Provide "friendly" or additional URLs to access a resource



RedirectRedirect directive

.htaccess

Try it:

http://cscie12.dce.harvard.edu/apache/dce.html http://cscie12.dce.harvard.edu/apache/church_st

1. Redirect 302 /apache/dce.html http://www.dce.harvard.edu/ 2. Redirect 301 /apache/church_st http://map.harvard.edu/level3.cfm?mapname=camb_allston&tile=F7&quadrant=A&series=W


1. GET /apache/dce.html HTTP/1.1 2. Host: cscie12.dce.harvard.edu 3. Connection: close 4. 5. HTTP/1.1 302 Found 6. Date: Wed, 13 Apr 2008 20:03:10 GMT 7. Server: Apache/2.0.49 (Fedora) 8. Location: http://www.dce.harvard.edu/ 9. Content-Length: 302

10. Connection: close 11. Content-Type: text/html; charset=iso-8859-1


1. morpheus% lwp-request -USed http://cscie12.dce.harvard.edu/apache/dce.html 2. GET http://www.dce.harvard.edu/ 3. User-Agent: lwp-request/2.06 4. 5. GET http://cscie12.dce.harvard.edu/apache/dce.html --> 302 Found 6. GET http://www.dce.harvard.edu/ --> 200 OK 7. Connection: Close 8. Date: Wed, 13 Apr 2008 20:01:26 GMT 9. Accept-Ranges: bytes

10. Server: Orion/2.0.6 11. Content-Length: 3619 12. Content-Type: text/html 13. Content-Type: text/html; charset=iso-8859-1 14. Last-Modified: Wed, 27 Oct 2007 18:45:00 GMT 15. Client-Date: Wed, 13 Apr 2008 20:01:49 GMT 16. Client-Peer: 140.247.198.100:80 17. Client-Response-Num: 1



Rewritemod_rewrite uses regular expressions to match on a pattern and rewrite incoming URLs to a new URL location.

Apache mod_rewrite mod_rewrite reference



Meta RefreshNote: redirection may also be achieved on some browsers by using the http-equiv attribute of the <meta> element. The recommended method is to do it at the server level.


1.  2.  3. <meta http-equiv="Refresh" content="10; URL=http://www.harvard.edu/"/>

Directory Index and ListingsNote: Remember the difference between a directory having rwx-----x and rwx---r-x permissions?

DirectoryIndex Would you prefer main.html or overview.html to be the default files returned when a directory is requested? mod_autoindex Provides for automatic indexing of a directory.

DirectoryIndex 1. DirectoryIndex index.html main.html overview.html slide1.html



More Control over Directory Listingsmod_autoindex

Basic

Custom

The details:

Example with Course Lecture Notes

1. morpheus% pwd /home10/c/s/cscie12/public_html/autoindex/ex2 2. morpheus% ls -la 3. total 28 4. drwxr-xr-x 2 cscie12 courses 8192 Nov 27 13:28 . 5. drwxr-xr-x 6 cscie12 courses 8192 Nov 27 13:11 .. 6. -rw-r--r-- 1 cscie12 courses 207 Nov 27 13:12 .htaccess 7. -rw-r--r-- 1 cscie12 courses 147 Nov 27 13:09 HEADER.html 8. -rw-r--r-- 1 cscie12 courses 66 Nov 27 13:09 README.html 9. -rw-r--r-- 1 cscie12 courses 4168 Nov 27 12:58 client-server.gif

10. -rw-r--r-- 1 cscie12 courses 906 Nov 27 12:58 slide1.html 11. -rw-r--r-- 1 cscie12 courses 743 Nov 27 12:58 slide2.html 12. -rw-r--r-- 1 cscie12 courses 1208 Nov 27 12:58 slide3.html 13. morpheus% cat .htaccess 14. IndexOptions FancyIndexing 15. IndexOptions IconsAreLinks IconHeight=22 IconWidth=20 \ 16. NameWidth=* ScanHTMLTitles SuppressLastModified \ 17. SuppressSize SuppressColumnSorting \ 18. SuppressHTMLPreamble 19. IndexIgnore *.gif .. 20. morpheus%



Expires HTTP HeaderModule mod_expires

.htaccess

Or, expire based upon modification time of document:

1. ExpiresActive On 2. 3. ExpiresByType text/html A3600 4. # HTML expires in 1 hour 5. 6. ExpiresByType image/gif A2592000 7. # GIF expires in 30 days 8. 9. ExpiresByType image/jpeg A2592000

10. # JPEG expires in 30 days 11. 12. ExpiresByType image/png A2592000 13. # PNG expires in 30 days 14. 15. # types not specified 16. ExpiresDefault "now plus 1 day" 17. # expires in 1 day

1. ExpiresActive On 2. ExpiresByType text/html M86400 3. # HTML expires 1 day after it was last modified 4. ExpiresDefault M86400



Do not cacheIf you do not want your page cached, set these HTTP response headers:

In .htaccess in Apache, this would translate to:


1. Cache-control: no-cache 2. Pragma: no-cache 3. Expires: <set to now>

1. ExpiresDefault "now" 2. Header set Pragma "no-cache"



Compress Contentmod_deflate compresses content before sending to web browser.

Simple use:

AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml



Details about enabling .htaccess and allowed directives Context: can these directives be in .htaccess files? AllowOverride: is the server configured to allow this group of directives to be overriden in this location? Is the required module loaded?



Legal Directives I: ContextCertain Apache directives are legal within .htaccess files. Some are not. See the Apache Documentation for details. Specifically, look at the Context line that is given for the directive in question.

Apache Core Features http://www.apache.org/docs/2.2/mod/core.html Apache Module List http://www.apache.org/docs/2.2/mod/ standard Apache Directives

The following is an excerpt from the Apache HTTP Server Version 2.2 documentation

Also, the "a" indicator on the Apache Quick Reference Card indicates that the directive is valid within an .htaccess file.

ErrorDocument directive

Syntax: ErrorDocument error-code document Context: server config, virtual host, directory, .htaccess Status: core Override: FileInfo Compatibility: The directory and .htaccess contexts are only available in Apache 1.1 and later.



Legal Directives II: AllowOverrideUsers are allowed to override certain aspects of the main server configuration. The main server configuration file (httpd.conf) contains an AllowOverride directive that determines which directives within .htaccess files Apache will process. The Override line that is given for each directive in the Apache documentation indicates which configuration directive must be active in order to use that directive with an .htaccess file.

For the FAS system, the main server configuration file has the following directive in place for users' public_html directories:

1. AllowOverride FileInfo AuthConfig Limit Indexes Options 2.

ErrorDocument directive

Syntax: ErrorDocument error-code document Context: server config, virtual host, directory, .htaccess Status: core Override: FileInfo Compatibility: The directory and .htaccess contexts are only available in Apache 1.1 and later.



Legal Directives III: Apache ModulesApache is distributed with several modules. These modules may or may not be active within the Apache server with which you are working. The Core features will always be available.

For example, if the Rewrite Module (mod_rewrite) has not been activated, none of the Rewrite directives will be available to use.

Refer to the Status and Module lines in the documentation for each directive and to the documentation for the specific Apache installation you are using.

Table of Contents | All Slides | Link List | CSCI E-12

