This has been a feature request for some time. The short answer is right now there's no VSCode setting that allows you to customize the sidebar font size. There is a hacky workaround solution proposed by @lindesvard in the same issue thread that involves using the Custom CSS and JS Loader extension to load in a custom CSS file that modifies the sidebar styling if you're willing to experiment with it.
To get this working follow the Getting Started section in the extension details, then add URLs to your custom files to the "vscode_custom_css.imports": [""] array. (Note that on Windows mine looks like the following: "vscode_custom_css.imports": ["file:///C:\\dev\\vscode-styling.css"])
Mihai Chelaru's answer presents a way to do more than just adjust the sidebar's font size. This, however, comes with a price of a bit of fiddling, possibly more than you'd be ready for, if the only thing you were interested in was the simple act of changing the sidebar's font size. So here's a simpler solution.
Just zoom the whole VS Code's UI in or out with CTRL++ or CTRL+- respectively and than adjust font sizes we do have control over in settings.json file, i.e.:
E.g. you want to increase the sidebar font size keeping editor font size unchanged: set "window.zoomLevel": 0.5 and "editor.fontSize": 18. Look at the result. Play with both values to get the result you like.
Again, this plugin is meant to do more than just changing the font size of the sidebar. But it has some minor advantages over Custom CSS and JS Loader. Like:
It won't show Unsupported on the title-bar or won't show any error message like Your installation is corrupted.
You do not have to reload settings every time VS Code is updated.