A 32 bit JVM is going to be limited to a maximum of 2G on Windows and Linux. In practice, the limit is lower than that, because the JVM requires a contiguous address space for the heap, and some of the 2G space will be used, and possibly fragmented by other things besides the heap. I've read that Windows might give you 1.2 to 1.4 maximum. Linux maybe 1.6 (Those numbers come from a quick Google of the subject, not extensive testing or personal experience).
With a 64 bit JVM on a 64 bit OS, the 2G limit goes away, and the upper bound is probably going to be limited by the maximum size of the swap file.
I just tried running java -Xmx6g and it didn't fail. I allocated a 1g array, and that worked. I allocated a 2g array and it slowed my machine to a crawl for a couple minutes, but it did work. I only have 4g of physical, and a number of programs running (including XP running under VMWare). I'm not surprised it crawled. I'm surprised it worked.
But I have to agree with Silent. Just out of curiosity, what are you doing? There certainly are problems that require these large amounts of memory, but they are a pretty tiny subset of problems. Maybe there are better ways of doing what you are trying to do, with less memory? Or maybe not.