You have to switch to 'Terminal-Normal mode' with Ctrlw, N (that's Ctrl-w, capital N). Then you can use the usual Vim commands to move around, cut, copy and paste.
Once finished, press either i or a to resume using the terminal as before.
And then <C-b> whenever I need to scroll back up, using gg to go all the way up, then G to come back down, (or <C-F> for page by page). I press 'i' or 'a' from any page to enter commands in the shell. Simplicity itself.
If you are using neovim, press Ctrl + \ followed by Ctrl + n to enter normal mode in a terminal.
The help (:help terminal-input) says the following:
In this mode all keys except <C-\><C-N> are sent to the underlying program. Use <C-\><C-N> to return to normal-mode. CTRL-\_CTRL-N
Also, if you do this often, I recommend mapping it to something more convenient, like a double tap of Escape:
tnoremap <Esc><Esc> <C-\><C-n>
(That way you can still use programs in the command line, that require escape to retain it's original functionality, since if you don't press anything else, it will just send a regular escape event after your defined timeoutlength.)