It can use either. By default, preference is given to UDP, but depending on the firewall(s) in between the peers connecting it may only be able to connect with TCP. You can use Wireshark to capture packets and verify whether TCP or UDP is being used. In Chrome you can also see details on the selected candidate (googActiveConnection) by going to chrome://webrtc-internals.
"Reliability mode" probably refers to the reliability mode of the DataChannel, which can be configured to run in reliable or unreliable mode. DTLS refers to the currently optional, but soon to be default method of exchanging encryption keys (the other deprecated mode is SDES). Firefox only supports DTLS, so for browser interop, you'll currently need to enable it in Chrome.
The RTCPeerConnection (media) will use TCP or UDP, while the DataChannel uses SCTP. The SCTP implementation used by Firefox is implemented on top of UDP: https://code.google.com/p/sctp-refimpl/.
It's possible to filter out TCP or UDP ICE candidates before adding them with addIceCandidate. Generally, you should not try to force the transport used since WebRTC will just "do the right thing". The browser does not limit the number of TCP connections used by WebRTC beyond any limit on the RTCPeerConnection or DataChannel (i.e., if you can have 10 PeerConnections, they can each use TCP without any problem).