Earlier today, I had to perform a substitution in Gvim within a 2GB file. Unfortunately, after a few thousand operations, Vim ran out of memory with the following error:
Out of memory!
Out of memory! (allocating XXX bytes)
Windows too complained about running out of virtual memory and my system slowed down to treacle-like response times.
While I could have split the file up into smaller chunks and performed the substitutions, a little nosing around revealed a quicker and neater alternative. Besides Vim storing the entire file in memory, it also stores undo data in the same place. While performing huge substitution operations on large files, this undo information can get prohibitively large leading to out of memory issues. The temporary fix is to simply turn undo off during such operations. This can be done by typing:
ul stands for undo levels)
This did the trick for me as I was able to perform my find and replace without running out of memory.
You can reset the undo levels value by simply restarting Vim or by setting it using the same command to your preferred value. The default is 10000.