This is not possible, and likely never will be because it would be rather heavy-handed for RubyGems to allow gem developers to require that users have a specific version control system installed to access a gem. Gems should be self-contained with a minimal number of dependencies so that people can use them in as wide an array of application as possible.
If you want to do this for your own internal projects, my suggestion would be to use Bundler which supports this quite well.
According to a commenter, this is no longer true. Prior information retained for historical context.
Duplicating the reference to a gem in Gemfile and .gemspec now appears to raise a warning message in Bundler, so this answer would appear to be no longer true.
Outdated info
This article by Yehuda Katz cleared up similar confusion for me. It says that, for use in development only, it's best to add the git stuff into the gemfile, but that bundler will still use the dependency/version info from the gemspec (seems magical to me, but I trust Yehuda).
I just was trying to figure this problem out as well. And I just came up with the following solution (which I'm not sure if your publishing your gem or have rights to redistribute that oauth2 gem).
Say your are in a project P and you want to use the self made gem tools which itself uses an OS gem oauth2.
If you made a patch within oauth2 and need that patch in your gem tools, you won't be able to fix this issue in the gem according to the accepted answer.
However, you can speficy the version you want within your projet P's Gemfile, and this will be the version used by tools on runtime:
I was facing similar issue and here is what I found. You cannot add git branch directly for some other gem, However you can acheive this another way.
You can define a private gem with repository link and branch name in gemfile of you custom gem i.e