I suspect it's when you have to write the swapfile to disk. Make sure that you are running vim on a computer with fast disk access; if you have to use a shared filesystem you can try moving the swap file using
It's because the 'esckeys' option is enabled (a consequence of nocompatible as I just discovered). When you press ^[O, there's a small delay as it figures out if you're using an arrow/function key or if you just meant those two keys in sequence.
One solution is to disable that option and give up on the arrow keys in insert mode.
Another is to set 'timeoutlen' to something less than 1000, maybe 100 (but be careful over slow connections).
Another is to use ^C instead of ^[ to leave insert mode.
It happens if you press "O" right after the Escape key, right? I always assumed it was because the terminal driver was waiting a bit to see if you were going to complete an escape sequence.