Open the integrated solutions console
https://
Default login is
system:manager
Navigate to the JVM for the server profile you want to change
Application servers > server1 > Process Definition > Java Virtual Machine
Change these values
Initial Heap Size=512
Maximum Heap Size=512
These correspond to the Xms and Xmx params of the JVM. So make sure you do not set Max lower than Initial, nor set Max higher than available OS RAM. I am not sure if adding a K,M,or G to the number will work, it seems by default the numbers are considered MB. Be aware that your JVM is 32 bit so there is a mathematical limit (and practical limit) to how much memory it can address. 3800 MB is where you will likely top out for max heap space unless your OS limits you to 2GB, which puts your heap max down to 1600MB.
You will have to restart the WAS server profile instance to have this change take effect on the JVM.