Put them in your command line invocation of the java executable, like this:
java -Xms512M -Xmx1024M my.package.MainClass
Keep in mind that you may want the starting and max heap sizes to be the same, depending on the application, as it avoids resizing the heap during runtime (which can take up time in applications that need to be responsive). Resizing the heap can entail moving a lot of objects around and redoing bookkeeping.
For every-day projects, make them whatever you think is good enough. Profile for help.
The environment variable to set is MAVEN_OPTS, for example MAVEN_OPTS=-Xmx1024m. The maxmem configuration in the pom only applies when you set the compiler plugin to fork javac into a new JVM. Otherwise the plugin runs inside the same VM as Maven and thus within the memory passed on the command line via the MAVEN_OPTS.
To set MAVEN_OPTS under Windows 7:
Right click on My Computer and select Properties (keyboard shortcut press Windows + Pause/Break)
Click the Advanced System Settings link located in the left navigation of System Properties to display the Advanced System Properties
Go to the Advanced tab and click the Environment Variables button located at the bottom of the Advanced System Properties configuration window
Create a New user variable, set the Variable name to MAVEN_OPTS and set the Variable value to -Xmx1024m (or more)
If you are running out of heap space during the surefire (or failsafe) JUnit testing run, changing MAVEN_OPTS may not help you. I kept trying different configurations in MAVEN_OPTS with no luck until I found this post that fixed the problem.
Basically the JUnits fork off into their own environment and ignore the settings in MAVEN_OPTS. You need to configure surefire in your pom to add more memory for the JUnits.
Hopefully this can save someone else some time!
Edit: Copying solution from Keith Chapman's blog just in case the link breaks some day:
Update (5/31/2017):
Thanks to @johnstosh for pointing this out - surefire has evolved a bit since I put this answer out there. Here is a link to their documentation and an updated code sample (arg line is still the important part for this question):
After trying to use the MAVEN_OPTS variable with no luck, I came across this site which worked for me. So all I had to do was add -Xms128m -Xmx1024m to the default VM options and it worked.
To change those in Eclipse, go to Window -> Preferences -> Java -> Installed JREs. Select the checked JRE/JDK and click edit.
Add an environmental variable (in both system and user's variables, I have some weird problem, that it gets the var from various places, so I add them in both of them).
Name it MAVEN_OPTS.
Value will be: -Xms1024m -Xmx3000m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled
The numbers can be different, make them relative to your mem size.
I had that problem and this fixed it, nothing else!