(Windows) Gvim: Backspace in insert mode does not delete text

I had an odd issue just now when using Gvim. Usually, backspacing when in insert mode (iwhen running the editor as an administrator.