The Operating System and the various library functions (malloc, etc.) control the run-time stack and the heap. That software is set up to detect and prevent overlap. Or it may allow some overlap, assuming you won't be maxing out the stack and the heap at the same time. But it will keep a look-out for the sizes and locations of everything, including what may have been swapped out to disk. That's part of any well designed OS.
If you max out both at the same time, what will happen is, when you make one recursive call too many, causing the stack to overflow, your application will be crashed by the OS. On Linux, you'll get something like a Segment Violation core-dump. Or, when you make one too many "malloc" calls, attempting to get more memory than what is currently available, "malloc" will return a NULL pointer.
If your code is written correctly, it won't have too many recursive calls that might overflow the stack. In normal operations, the run-time stack doesn't need to be very large, so if you blow that stack, there's probably a bug in your code.
Also if your code is written correctly, it will test for that NULL pointer return from malloc and do something appropriate (like logging an error message somewhere then halting). If it doesn't test, it will just keep on running until it attempts to use that NULL pointer, and then the OS will crash your application with a Memory Access Violation core-dump.