Question : Problem: Correct Keymapping Over SSH for vi Editing.

I am using OS X to login via the terminal to a Linux server. When I use vi, I have to Ctrl-H to delete and when I am insert mode I can't use the arrow keys. The term defaults to TERM=xterm-color. How can I change the keyboard settings?

Answer : Problem: Correct Keymapping Over SSH for vi Editing.

if the arrow keys work in vi depends on a lot of things
1. check how your vi is configured: using termcap (most commonly) or terminfo,
then you need to check the corresponding file if the arrow keys are known there.
2. If you're done with 1. you need to find out how your arrow keys "arrive" in the remote shell. This depends on a couple of things too: (OS) keyboard mapping local, shell local, (X) application (i.e. xterm) local, (X) window manager local, X server (if any) local, OS remote, shell remote, shell settings remote ... probably some more

So you see that the cause of the problem is not simply to identify.
I'd start top-down debugging, means that you first check in your remote shell how the special keys arrive (keysym, keycode). dumpkeys is a good starting point for that.
see
   man 5 keymaps
   man dumpkeys
   man loadkeys
   man termcap

IIRC the vi's termcap settings for arrow keys are
   km:kb=^H:kl=\E[D:kd=\E[B:ku=\E[A:kr=\E[C:ks=\E>\E[?1l:ke=\E>\E[?1l:
(where \E is ESC)

As I don't have any problems comming from OS X to Linux and using vi(m), I guess that it is a problem on your remote Linux box.