Have you ever wondered how it can be that when you type in a web address in a browser you content from far far away makes it onto your screen? In this article I will explain the basic building blocks of the web.
Internet versus World Wide Web
Many people use the term internet and the web interchangeably but they are not the same thing. The internet is the way one computer can connect with and interact with another computer that could be on the other side of the world. The world wide web is a specific protocol for sending and receiving documents across the internet called HTTP (Hyper Text Transfer Protocol) that was inverted by Sir Tim Berners-Lee, a Brit! That’s right folks, no matter what the Americans tell you, the web is a British invention!
There are other protocols used over the internet, for example IMAP (Internet Mail Application Protocol) and POP (Post Office Protocol) allow us to receive emails and SMTP (Send Mail Transfer Protocol) allow us to send emails. In fact there are lots of different protocols for different types of communication over a network, each using separate ports to keep the messages separated from each other.
Sending and receiving documents?
It may surprise you that I described HTTP, and thus the web, as a way to send and receive documents. Surely it’s web pages that are being sent, not documents?
Actually, a web page comprises of something called Hyper-Text Markup Language (HTML) which is just a way of describing a structured document that a web browser can understand and convert into something to display on screen (or read out for screen readers).
HTML provides the structural building blocks of a web page but we need something else to tell a browser how to draw that document within the browser window and make it look pretty. Cascading Style Sheets (css) tell the browser how the elements described in the HTML should look including things like text colour, background colour, what should happen when you hover over something etc.
Client Server Technology
So far we’ve described what the components are that make up a web page, HTML and CSS but how does that content get onto your screen? The web consists of two components, web browsers that make requests to retrieve content from somewhere on the web and web servers whose job it is to respond to a web browsers requests and send the content.
In the early years of the internet this was very simple, the web browser sends a http get request for a web page, the web server receives that web request and finds the html page that matches the request and then sends it back to the browser. Most probably, the web page had links to images, stylesheets and javascript files that are needed to display the complete web page so the browser then makes separate requests to the server for each of these related documents and uses them in conjunction with each other to display the page.
However, that process didn’t allow for a very interactive experience. To make a change to a website, the HTML would have needed to have been altered. You couldn’t personalise the experience so server side programming languages were introduced to add logic to pages. For example, you could store a list of products in a database that could be updated using a content management system and instead of just loading a html file from disk, the web server would evaluate the logic in the code before finally returning the HTML complete with products from the database to the web browser.
As far as the web browser is concerned, it still just receives a simple HTML document. It doesn’t need to know or understand how that document was created, whether it’s just simply been served up directly from a file stored on the web server or whether the web server has executed some code to generate the HTML.
How DNS Works
So, a web browser makes a request and a web server returns the requested HTML document but how does your web browser know where to send the request so that it’s received by the right web server?
Well, just as we have directories like the yellow pages to look up what phone number to ring to speak to Bob Jones, the internet has a directory system to look up which web server to contact to serve www.mystware.co.uk. This directory is called Domain Name System (DNS).
When it comes to phone numbers, it’s ok to send out a directory once a year because people don’t tend to change phone numbers very often but new domains are created all the time and it’s not uncommon for a website to move from one server to another so DNS has to respond fast to changes.
When you hit enter after typing in a web address first your browser checks to see if it already knows the IP address of the computer responsible for the domain name you entered from previous visits to the same site. If it does not know the IP address, it then forwards the request onto the default DNS server configured for your computer. This is typically your modem or router. If the router does not know the IP address, it forwards the request onto one of your Internet Service Provider’s (ISP) DNS Server.
If your ISP’s DNS server does not know the IP address, it will forward the request onto the DNS server for the top level domain the website belongs to, for example, if you type in www.bbc.co.uk, the request would hit the DNS servers responsible for the .UK top level domain, which would then look up the IP address of the DNS Server for .CO within the .UK records and then finally the .CO DNS server would send the IP address all the way back to your computer. Each of the DNS servers involved in looking up the domain name for you would store that domain so that next time it is requested, they would be able to respond without passing on to the next DNS server in the hierarchy.
However, because IP addresses may change, each DNS server will only store (cache) the lookups it has previously performed for a short period of time so that they can retrieve updated information as a web address changes IP address.
The whole DNS system is distributed so there is no one single path to lookup an IP address which means if a particular DNS server is too busy to respond or is offline, the request can be rerouted via a different DNS server. Your computer will only wait a short time for a response from the next part of the chain before trying to request the information from an alternative DNS server.
Just to make a simple request for a web page can involve a whole host of requests made to servers all over the planet in fractions of a second and all this goes on without us even noticing.
Making the Web Interactive
Modern web pages are more than just static documents. We’ve discussed how the web evolved from simple files stored on a server to web pages generated from code that is executed on a web server to generate html to be sent to your web browser but there’s still something missing. CSS rules allow a certain amount of interaction such as colour changes when hovering over a menu but it’s still not the rich immersive experience we’re accustomed to today.
Today we see pages that load lists of tweets from twitter or statuses from facebook and automatically update without us having to do anything. It all feels very instant, but how can this be if all our web browser receives is a static HTML document? How can we drag and drop elements from one part of a page to another and how do animations work?
Back in the early nineties animation was achieved by using animated gifs, gifs are just image files that actually contain multiple frames that play like a very simple bit of video. Then came along a company called Macromedia that invented something called Flash. Flash was a plugin that you could install on your web browser for rich animations and interactions.
Even now there are many websites out there that are either completely built in flash or have flash components. Flash was great but it required installing a separate piece of software to work and that meant that for users who could not install flash the websites would not work properly. The flash swf files themselves were also quite large so web pages could be slow to load too.
Just as a web server could execute code to produce dynamic web pages, web browsers started to build in a programming language called javascript that allowed code to be executed on a web browser. Javascript made it possible to manipulate the html document without the need to send a request to the server. It could be moved to add new elements to a html document or move them around, create an animation affect etc.
Another exciting technique made possible by javascript was the ability for a page to make another http request for some content after the page had been rendered on your browser. This technique was coined as AJAX (Asynchronous Javascript and XML). The Asynchronous part just meant that the javascript is being executed in the background after the page has already been loaded so your still able to interact with the web page without having to wait for the javascript to finish doing what it’s doing. XML was another language for passing information to and from a server but although we still call it AJAX, it’s more common now to use a much simpler format for sending and receiving information asynchronously called Javascript Object Notation (JSON).
Going back to our example of loading a list of tweets from twitter what would happen is the following, you type the web address, the web server sends back a static HTML page, stylesheets, images and javascript files your web browser needs to display the page. Once the web browser has downloaded all this content, it then executes the javascript which sends another HTTP request to twitter to ask for the latest tweets, once a response is received, it then displays those tweets on the page but whilst you’re waiting you can still interact with the page.
The javascript could be written to keep checking for updates without you having to reload the whole web page from the server.
A stateless system
When you visit a website like facebook today, it remembers who you are and your preferences and settings for the website but HTTP is a stateless system, all a web server does is receive a request and processes it, it’s like a game of tennis, you serve a request and the server hits the ball back, once it has put the ball back in your court as far as the server is concerned it’s finished talking to you. It’s a bit like having a phone conversation where after you’ve said something and someone replies, you put the phone down. If you want to say something else to the other person, you have to ring them again.
That might sound like a pretty strange system but if a web server had to keep open a connection until you explicitly told the server you’re done, web servers would run out of available connections pretty quickly and we’d have to wait for someone elses web browser to finish talking to the server before we got our turn. You also have the difficulty of identifying when to end that conversation between browser and server. I could spend half an hour looking at one web page then look at another page on the same site or I might just visit one page then leave. This is why each request is treated as a separate entity.
It would be really annoying if every single time you clicked on a link on a web site you had to log in again so the website you visit somehow needs to identify when the same user or web browser makes another request so it can retrieve your specific settings and apply them.
One way round this problem is to use a session token. When a web browser first makes a request to a web server, the web server generates a unique identifier that it sends back as part of the response to the web browser. When the browser makes subsequent requests, it sends the session token with the request so the server can identify that this is the same browser that made the initial request and information can be stored on the web server that relates to the choices made, either in memory on the server or in a session database. These sessions are typically short lived lasting for only a period of twenty minutes after the last request with that session was made before it is removed to conserve the amount of resources being used on the server. If you’ve ever used an e-commerce website, added items to a basket, then gone away for a while to find your items have been removed when you come back, it’s probably because your session has expired and the items in your cart were stored using the session.
Another way to store your preferences between pages on a website is to use cookies. Cookies are tiny text files stored by your web browser that web pages can use to remember settings or identifiers. Only the website that created the cookie can access the data within the cookie. There are different types of cookie, session cookies are only stored on your computer until you close your web browser. Persistent cookies are stored until a specific expiry date set when the session expires.
Over the last few years there has been a lot of controversy regarding Cookies and the EU changed the law so that websites that use cookies (which is pretty much all websites) must now explicitly tell you that they are using cookies and allow you to opt out. The reason is cookies can be used to identify you and track your movements across the internet. Although internet privacy is a concern, without these cookies most websites will not work properly and the cookie notifications are more of an irritant to the user than anything else.
Thankfully, in the UK, the Cookie law is broadly being ignored and the Information Commissioner’s Office, who are responsible for making sure companies implement such legislation are interpreting the law very differently and say they will not take action unless complaints are made, and even in that case, it will be advisory warnings rather than fines.
In fact, some of the cookie warnings I have seen themselves breach the cookie legislation because if you state that you do not want cookies to be used, they store your choice as a cookie.
One limitation of cookies is that they are limited to a maximum size of 4Kb and they are submitted to the server every time you send another request. Modern browsers now have an alternative called local storage that allows data to be stored on the browser up to 5Mb and the data is not sent to the server with each request, which would slow requests down.
The Debrief
As you can see, the process involved in rendering a web page in a browser window involves a lot of different technologies and complex considerations and in this article I’ve tried to give you a whistle-stop tour of how the web works without getting too bogged down in the technical aspects.
If you’re interested in learning more or have any feedback, leave a comment below and I may add further articles to go over things in more detail.
I hope you found this article interesting and informative!