Skip to content
Snippets Groups Projects

lab 5 · Changes

Page history
updating wiki authored by Sorrel Harriet's avatar Sorrel Harriet
Hide whitespace changes
Inline Side-by-side
Showing
with 321 additions and 0 deletions
lab-5.md 0 → 100644
View page @ 37b1ccec
# Lab 5: Let's Build a Web Server!
In this lab you will be creating a virtual server in order to serve web pages! That's right - you will actually setup your very own server in the sky!
In the process, you will learn to:
- Describe the role of a web server
- Identify components of a URL
- Identify components of an HTTP response
- Describe (in simple terms) what a CGI script is and how it relates to dynamic web applications
For Python newbies, you will also learn how to:
- Instantiate a class imported from a module
- Make a Python script self-executable
- Format a string containing \`replacement fields'
The tasks in this lab are based on a tutorial published by [Ruslan Spivak](https://ruslanspivak.com/lsbaws-part1/). While it is highly recommended you refer to the original tutorial (it has some great diagrams!), note that the code examples won't work with the version of Python we are using (Python >3).
When it comes to the task of installing software on your server, you may find you have the opportunity to practice this week's study skill: **Troubleshooting!**
Some [troubleshooting tips](http://gitlab.doc.gold.ac.uk/data-networks-web/group-work/wikis/Troubleshooting) have been posted on the group-work wiki. Please refer to these tips before asking for help! (It's good practice!) **NB:** I have put it on the group-work wiki so that you can add to it, if you think of some other useful tips!
---
## Task 1: Create a virtual server
In this task, you will create a virtual server
There is a video under week 6 on the VLE to help with this, or you may choose to refer to the written instructions below.
+ Go to the [Virtual Server Manager](https://www.doc.gold.ac.uk/dept/VirtualServer) on the department intranet
+ Click, \`New Server' to create a server
+ Select a system type:
- Webserver - installs Apache server and MySQL automatically
- Minimal - allows you to customise your stack (use this one today!)
+ Set an admin password, but leave other fields with defaults
---
## Task 2: Connect to your VS via SSH
Next you will connect to your web server using SSH. There a couple of ways to do this. If one doesn't work, it may be worth trying another (but also think about why it didn't work...for example, are you connected to eduroam?)
### Option 1: From any computer via a web browser
You can use the doc.gold.ac.uk proxy server by typing the following into your browser's address bar:
http://www.doc.gold.ac.uk/ssh/ID/
In the above, replace `ID` with your server ID. You will be prompted for a username and password.
This option launches a bash shell in the browser window.
---
### Option 2: From an SSH client on the college network
To SSH from inside the college network:
ssh USERNAME@myserver.doc.gold.ac.uk -p PORT
In the above, you should replace:
+ `USERNAME` with your username
+ `PORT` with the SSH port number of your virtual server (i.e. 2106)
You do not have to be logged into Igor for this part, but you DO need to be connected to the college network.
If you are not connected to the college network, you may either SSH to Igor first, or try the alternative procedure to connect from an outside network.
---
### Option 3: From an SSH client outside the college network
Use this option if you are on a computer that has a Unix shell, but you are not on the college network.
ssh -t USERNAME@doc.gold.ac.uk myserver ssh ID
In the above, you should replace:
+ `USERNAME` with your username
+ `ID` with the ID of your virtual server (i.e. 106)
---
## Task 2: Install a minimal software stack
Now that your server is running, you need to install some software on it! If you opted for the minimal setup, there won't be any server software or database management systems preinstalled.
Create a minimal stack by installing Python 3, mongodb and/or MySQL, and either or both of the pymongo and mysqldb API's. It will also be very advantageous for you to install git, as that will give you easy access to the example scripts.
**Note:** For the purposes of completing this lab, the only essential software requirement is Python, which is pre-installed on the minimal virtual server. Therefore, you can choose whether or not you want to go the whole way by installing a Database Management System and Python database API (it's good practice!)
---
### Highly recommended: Install git
You are also advised to install git to make accessing the lab resources easier:
sudo apt-get update
sudo apt-get install git
You then need to make an ssh key pair for your server like you did in Lab 1:
ssh-keygen -t rsa
Open it in vim or nano and copy-paste its contents in GitLab. You don't remove the other key, but make sure this one has a unique name.
vim ~/.ssh/id_rsa.pub
**NB:** If using vim, you will need to install it first!
sudo apt-get install vim
---
### Recommended: Install MongoDB
Installing mongo requires the following commands on an Ubuntu 16.04 system. For an explanation about what these commands mean, refer to this [Digital Ocean Tutorial](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04).
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo vim /etc/systemd/system/mongodb.service
You should now see a new text file has opened in the vim editor. This is a configuration file which will contain some minimal configuration details for mongo.
Paste the following inside it:
[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target
[Service]
User=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf
[Install]
WantedBy=multi-user.target
Then do `<Esc>:wq` to save the file and quit the vim editor.
Now you can start mongodb:
sudo systemctl unmask mongodb
sudo systemctl start mongodb
Check its status by doing:
sudo systemctl status mongodb
To start the mongo interactive shell, simply do:
mongo
---
### Optional: Install MySQL
If you'd rather install MySQL (or want to install it as well), you can also do:
sudo apt-get update
sudo apt-get install mysql-server-5.5
To log into mysql shell as root user, you do so like this (assuming you've set a password):
mysql -u root -p
**NB:** To get the latest version of MySQL, just replace `mysql-server-5.5` with `mysql-server`. However, since Igor runs version 5.5, it might be worth keeping it the same. That's also the reason why we're installing Python 3.5 rather than Python 3.6!
---
### Recommended: Install Pymongo API
To install pymongo (Python's mongo database API), you first need to install the Python Package Index mangager (PIP):
sudo apt-get install python3-setuptools
sudo easy_install3 pip
You can then go ahead and install Python with the pymongo API:
sudo pip3 install pymongo
**Tip:**
To make Python 3.5 the default version of Python (i.e. the version that is started when you type `python` in the terminal), you can do:
alias python=/usr/bin/python3.5
You could make this change persistent by putting the line in your server's ~/.bashrc file. That way, it will get executed every time a new terminal session is started.
---
### Extension: Knock yourself out
You can install anything else you want or think you might need in addition to the above...after all, you are a super user!
---
## Task 3: Serve a web page
To do this next step, you might wish to clone the lab-exercises repo onto your server, which will avoid having to use the vim editor to make your own web page. If you'd prefer to make your own, that's fine too!
+ Make a web page from scratch, or clone lab-exercises for a ready-made example:
mkdir dnw
cd dnw
git clone git@gitlab.doc.gold.ac.uk:data-networks-web/lab-exercises.git
cd lab-exercises/lab-5
ls
You should see an `index.html` file in the lab-5 folder.
+ Make sure the file is executable:
chmod +x index.html
+ To serve the current directory at a specified port number, do:
python3 -m http.server 8000
+ To exit and kill the process, do `<Ctrl+c>`
---
## Task 4: Access your server via HTTP
Now you may access your index web page from a browser!
+ From any network, enter a URL of this format in the address bar:
http://www.doc.gold.ac.uk/usr/ID/
- In the above, replace \`ID' with your virtual server's ID (for example, 106)
+ From the college network, you can enter a URL in the format:
http://myserver.doc.gold.ac.uk:WWW/
- In the above, replace \`WWW' with the number \`5', followed by your server's WWW port number. This will be the number 8000 + the server ID. For example, if your server ID is 199, its WWW port number will be 8199. So with the \`5', that would be 58199.
---
## Task 5: Make your server a bit smarter
In practice, there may be other options you want to set in relation to your server's configuration. For example, you might want to specify a path to a particular directory where your website files live (like public_html, for example!)
You might also want to remember your web server configuration for the future.
In such cases, it would be preferable to instantiate the server from a .py script.
A very simple version might look like this: (**NB:** if you cloned lab-exercises, the script is already written in verySimpleServer.py)
import http.server
PORT = 8000
server = http.server.HTTPServer
handler = http.server.SimpleHTTPRequestHandler
httpd = server(("", PORT), handler)
print("serving at port", PORT)
httpd.serve_forever()
To run this script, you will need to make sure the file permissions are such that the file is executable. You can do this using the chmod (change mode) command:
chmod +x verySimpleServer.py
You could then make the script self-executable by including a she-bang line at the top of the script like this:
#!/usr/bin/env python3
That will enable you to run the server like this:
./verySimpleServer.py
+ See if your server is working by accessing it from a browser! (as per task 5)
You can explore many other configurables of Python's http.server object from the [Python http.server documentation](https://docs.python.org/3.5/library/http.server.html).
---
## Task 6: Serve a dynamic page
In tasks 3 and 5, you served a static HTML page. In practice, we want the server to execute Python scripts when it receives an HTTP request. The scripts will generate the HTML that is sent back in the response body to the client.
To make your very simple server execute Python scripts, you can make it handle requests using the Common Gateway Interface (CGI) protocol. CGI is a standard protocol for web servers to execute Command-line interface programs.
+ Locate the [simpleServer.py](http://gitlab.doc.gold.ac.uk/data-networks-web/lab-exercises/blob/master/lab-5/simpleServer.py) script in lab-exercises repository.
+ Note the differences between this script and the verySimpleServer.py script. The major difference is the use of a CGIHTTPRequestHandler class, rather than the SimpleHTTPRequestHandler class.
+ Replicate these changes in your server script (or just use the simpleServer.py script, if you cloned lab-exercises). Make sure it is also executable.
+ Make a cgi-bin directory and a test python script. The test file should output HTML. You might include an element of logic too! If you cloned lab-exercises, you will find a ready-made example called [splash.py](http://gitlab.doc.gold.ac.uk/data-networks-web/lab-exercises/blob/master/lab-5/cgi-bin/splash.py) in the lab-5/cgi-bin folder!
+ Make sure the file is executable:
chmod +x cgi-bin/splash.py
+ Read the comments in the splash.py example code to try to understand what each line does. You could always adapt the code to do something different if you like.
+ Serve your new improved server. For example,
./simpleServer.py
+ Access the application from a browser. You'll do this like before, but appending the URL with the path to your cgi script. For example, from inside the college network: http://myserver.doc.gold.ac.uk:WWW/cgi-bin/splash.py
- In the above, replace \`WWW' with 5, followed by your server's WWW port ID (i.e. 58106)
+ Or from outside the college network: http://www.doc.gold.ac.uk/usr/ID/cgi-bin/splash.py
- In the above, replace \`ID' with your server's ID (i.e. 106)
---
## Task 7: Take the Lab 5 quiz!
Having done (hopefully) all of that, you are now ready for the lab 5 quiz. The quiz will test your understanding of what you just did. You can take it up to 3 times, and it will be open until 14:00 PM on Friday 17th November. We will discuss the answers in tomorrow's lecture!
Please note that **the final 3 questions require further reading** (i.e. the answers aren't necessarily covered in these instructions!) These questions are intended to push those in need of some extra challenge...imagine this is the exam and you want to score above 70%...you need to inject some of your own initiative! ;-)
---
## Sources/Resources
Note: Python code in these tutorials is incompatible with Python 3. Use them for reference purposes only (they have some helpful notes that are still relevant).
[Ruslan Spivak: Let's Build a Web Server (part 1)](https://ruslanspivak.com/lsbaws-part1/), Ruslan Spivak, 2015. Note: Code incompatible with Python 3.
[Pointless Programming: Python CGI Tutorial](https://pointlessprogramming.wordpress.com/2011/02/13/python-cgi-tutorial-1/). Note: Code incompatible with Python 3.
[Simple HTTP Server Tutorial](http://2ality.com/2014/06/simple-http-server.html), Dr. Axel Rauschmayer, 2014. Note: Code incompatible with Python 3.
[Linux Tech Tip: Simple Python HTTP Server](http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python), 2009. Note: Code incompatible with Python 3.
[How to Install Mongo on Ubuntu 16.04](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04), Digital Ocean, 2016