In my answer, I'll concentrate on setting the default font size through X resources. The use of X resources has already been mentioned in the answer mentioning ~/.Xdefaults; I'll give more details (the same which I have already described in https://unix.stackexchange.com/a/426914/4319. Apart from the height of the "default" "face" in Emacs, one can similarly set other font parameters.
To set a specific default font height for Emacs, I have put into /etc/X11/Xresources-site (/etc/X11/Xresources is also OK, though can be overwritten by your distro):
Emacs.default.attributeHeight: 94
This would affect also remote X clients which are Emacs (e.g., emacs started on a remote host via ssh).
/etc/X11/Xresources-site and /etc/X11/Xresources (and probably ~/.Xresources and ~/.Xdefaults) are usually read at the start of your X session; to affect your current X resources immediately, run something like xrdb -merge /etc/X11/Xresources-site. The X resources can be viewed by xrdb -query.
Actually, in my case, /etc/X11/Xresources-site is being read thanks to a line in /etc/X11/Xresources (which is read by the start scripts):
#include "/etc/X11/Xresources-site"
so /etc/X11/Xresources is the thing that is read for sure.
There are also some files with the same syntax which are read each time an X program like emacs starts. In my case, they are: ~/.Xdefaults-MY_HOST_NAME, /etc/X11/app-defaults/Emacs (only for emacs-athena, not for emacs-gtk3), /usr/share/X11/app-defaults/Emacs etc. (But I like the idea of loaded X resources more -- shown with xrdb -query; so that remote X clients read the same X resources.)
Emacs 24.3 had a bug which made it not honor the attributes for the default face coming from the X resources, such as in my example above. This was fixed since 24.4.