What is this "hosting" anyway?
Wikipedia says: "Web hosting is a service that allows individuals and organizations to make their own website available via the World Wide Web". Basically if you want to have your website online you need to put it somewhere that people can get to it, this is always going to be on a server connected to the internet.
The scope of hosting service varies widely but can generally be classified into 3 groups: Shared Hosting, Virtual Private Server (VPS) Hosting, and Dedicated Server Hosting.
Shared Hosting
In a shared hosting environment there are multiple websites all sharing the same resources, often there will be 1 server hosting hundreds of simple websites. The advantage to this is that cost can be decreased because you are sharing resources with other people. The disadvantage here is that you are essentially overselling those resources again and again. If any 1 of the websites on the shared server starts using more resources (for example if they have a traffic increase) all the other sites have less available. So your site could be really slow someday because someone else is getting a lot of traffic.
Dedicated Server Hosting
With a dedicated server it is pretty much just like it sounds. You have your own physical server that is designated for your use only. You have complete utilization of all the machines resources from processing power, and memory to hard drive space, no other websites will share the server unless they are yours and you want that. Sometimes you will purchase the hardware yourself outright and other times you'll lease it from the hosting provider. Generally it is better to lease because the hosting provider will maintain the hardware and replace parts if they go bad. The advantage of a dedicated server is that you have all the power of a single machine to yourself. The disadvantage lies in hardware failure. If a hard drive or power supply goes out or some other mechanical failure happens your server could go down until the hardware is repaired, in best case scenarios this can often be 24-48 hours and hopefully the failure didn't cause any data loss. It is possible to get servers that have redundant hard drives and other parts to have a failsafe against hardware failure, but those servers are much more expensive.
Virtual Private Server (VPS) Hosting
VPS hosting is somewhat of a mix between Shared and Dedicated taking all the good parts and few if none of the bad. The process of VPS hosting is based on a technology called "Virtualization" in which the resources of a host machine are divided amongst many virtual machines which run on the host. The host machines are often very high end powerful servers that would be overkill for use as a Dedicated Server, however through virtualization you can essentially have multiple dedicated servers by dividing the resources.
VPS hosting doesn't suffer from the same disadvantages as Shared Hosting because with a VPS the resources are fixed. If a VPS is given 1 Gigabyte of memory it is there for it to use and restricted from others even if it only ever uses 1/4 of that.
Hardware upgrades are a significant advantage with a VPS. If the processing power, memory or storage space needs to be increased it only takes a few clicks and it is done. With Dedicated servers the hardware must be bought, information copied over if necessary, the server must be shut down, hardware installed and the brought back up for any post installation tasks.
Some people might be thinking right about now "don't the host machines need hardware maintenance sometimes too?" The answer is yes, but through a technology called "Live Migration" the virtual machines can be moved from one host to another without any down time. This makes maintenance on the hosts realy easy.
Another advantage of a VPS is the setup time. With a dedicated server you need to order the machine, then install all the software and lastly install the physical machine into the data center and connect it to the internet. With a VPS they can be setup from "templates" wich include all the necessary software and configurations, and because the amount of memory and disk space is just a setting a new VPS can be setup in mere minutes.