dpkg cannot install as there's no space left on the device even though there is

I ran into this issue a couple of days ago and cannot recall the exact error message. However, the problem was effectively that aptitude could not install the new kernel update because my partition had apparently run out of space. An interrupted update to Klipper is one thing and the Linux kernel a whole 'nother kettle of fishies. Thinking that I simply needed to free up some space on my partition, I checked the current status via df. I surprisingly had 30% of free space still left lying about (my /home is mounted on a different partition). So what ... gave?

As is my usual practice in such situations, off I went to pay my penance at that sanctum sanctorum of the Interwebz which is El Goog. I was informed that even if I had 30% of my partition free, I might have run out of inodes (or index nodes). Sure enough, when I ran,

... I found that I was apparently using all 100% of my available inodes.

To put it simply, each file is linked to an index node and a partition has only so many index nodes available to it. Ergo, somewhere deep within the recesses of my now sinister partition lay at least one directory with an astronomical number of files within it. Now I had to find out which one it was.

A wee little tiny bird informed me of the following string of gobbledygook which apparently lists the directories with the most number of files within:

While the above length of pipe certainly appears to do its job, it takes a while as it is to be run from / and take about 3 light seconds to complete. In the meantime, I was unreliably informed via IRC, that the culprit in most cases is directories containing old, outdated, obsolete, orphaned, over the hill kernels that are simply occupying space. While I suppose that there are many ways to skin this feline, I chose the synaptic package manager and used its search field to locate installed packages that contained the string linux-image. Sure enough, a slew of kernels still regarded my cosy little partition to be their home. These included a bunch of 3.3.x kernels as well as 2.6.x kernels (this box is aulde). Rather than mess with any of the kernels that the box was running on, I decided to nuke all the 2.6 kernels. This took a while. But once I was done my inode count came down from ~100% to 47%.

All is well again in my world for a couple of years to come.

(P.S. If synaptic or apt fail to even run because you don't have enough inodes or because dpkg was interrupted, then there really is no other choice than to manually delete files from the partition in question. I'd suggest carefully cleaning up the usually suspects such as /var/log and /var/cache. Using the find command pasted further above is another potential avenue of pursuit. In my case, once I had around 4000 inodes free, dpkg was able to proceed.)