Is your FO instance file being created successfully?
If not, then try switching XSLT processors. If you are using a DOM based processor, a large file will consume a very large amount of RAM. Try using a SAX processor, such as SAXON http://saxon.sourceforge.net/
If you are able to generate a valid FO instance file, then your problem is with the FO processor generating the PDF file.
It is often helpful to try other FO processors, to see if there is a bug/limitation in the one you are using.
Here are a few FO processors you could try:
XEP: http://www.renderx.com
Antennahouse: http://www.antennahouse.com
FOP: http://xml.apache.org/fop/
IBEX: http://xmlpdf.com/