Typically what you do is set up a special vimrc-type file with the settings for a particular language, and then use autocommands in your main .vimrc to execute the special vimrc when necessary. Here's my configuration for Haskell (.hs, etc.) files:
autocmd! BufNewFile,BufReadPre,FileReadPre *.hs so ~/.vim/haskell.vim
autocmd! BufNewFile,BufReadPre,FileReadPre *.hsc so ~/.vim/haskell.vim
autocmd! BufNewFile,BufReadPre,FileReadPre *.lhs so ~/.vim/haskell.vim
autocmd! BufNewFile,BufReadPre,FileReadPre *.cabal so ~/.vim/haskell.vim
My ~/.vim/haskell.vim does stuff like "set expandtab" to use spaces instead of tabs, and all sorts of other magic for formatting and things like this. You can often download good versions of these for various languages from http://vim.org and other sites.
Note that you can do a lot more than just change vim settings. For example, you can run the file through a filter before and after editing:
" Edit gpg-encrypted ascii-armoured files
autocmd! BufReadPre,FileReadPre *.asc set bin
autocmd BufReadPost,FileReadPost *.asc '[,']!gpg -q -d
autocmd BufReadPost,FileReadPost *.asc set nobin
autocmd! BufWritePre,FileWritePre *.asc set bin
autocmd BufWritePre,FileWritePre *.asc '[,']!gpg -e
autocmd BufWritePost,FileWritePost *.asc undo
autocmd BufWritePost,FileWritePost *.asc set nobin
My answer is based on this tip on the VIM Wiki. This answer uses the "after" directory so you won't have to muck with the supplied plugin files for different filetypes.
For example, to specify custom settings for Python files, create a file called python.vim to hold your Python settings:
These other answers seem way too complex. Instead of messing around with yet more directories and files in your ~/.vim tree, just add the following to your ~/.vimrc.
I use the editor config plugin and add a .editorconfig file to all my projects - it will let you define different indentation settings for different projects using the same programming language which can be useful as often different projects in the same language have different coding standards.