What is the problem with having the XML files get copied into the folder on release builds? It seems to me that this is fine and that the real problem is with the code that picks up files to place them in the installer. Picking up third party dlls from your own bin\release folder is not a good practice in my opinion. I'd have my installer pick up third party dlls from their own folder in my source tree.
Visual studio project file has the same format as any msbuild file. So you can manually add the condition into corresponding section to not copy your xml files if configuration name is 'Release'.
The setting is in the Properties of the project in question, under the "Build" tab, uncheck "XML documentation file". It's also an MSBuild property called <DocumentationFile> under the applicable <PropertyGroup> for your build configuration in the .*proj file.
When you build a project the .xml/.pdb files are gathered through the ResolveAssemblyReference task. When ResolveAssemblyReference is called it is passed a list of file extensions for related files. That list of file extensions is captured in the MSBuild property AllowedReferenceRelatedFileExtensions. By default that list will contain ".pdb;.xml".
If you want to exclude all related reference files from being picked up then just override the value of the property to something which related files won't have extensions of. For example you can set AllowedReferenceRelatedFileExtensions to "-".
You can also customize the list of file which are returned by that. If you only want to find only .pdb files then you will need to pass in AllowedReferenceRelatedFileExtensions=".pdb". In that case any references which have .pdb file next to the .dll/.exe they will be copied as well. You can also use this to copy other related files which may not end in .pdb/.xml. For example if you have a referenced assembly named, MyAssembly.dll and in that same folder there exists MyAssembly.pdb and MyAssembly.foo If you set AllowedReferenceRelatedFileExtensions=".pdb;.foo" then both the .pdb and .foo file will be copied to the output directory.