ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

I installed MySQL on my local server today and when I went to access it via mysql_secure_installation to set my root password etc., I ran into the following curious error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

While I initially thought that it was some kind of installation issue, after much scratching of pate, I realised that Occam was right and that I should be checking the simpler things first. Sure enough, after a quick check, I found that MySQL was not even running :/

In other words, if you run into this error, make sure that you are actually running the damn thing before thinking that the Gods are conspiring against you!

*facepalm*