Need to cut down on memory hungry Java program?

Question

Hey

I am writing a program that is supposed to take a 18MB text file, parse it and enter it into a a data structure to keep track of stats on each word (like frequency, next words and their frequencies.)

I have decided to store everything in Binary Search Tree of my own implementation but the problem is that when my program actually populates the tree I eventually get that "Java heap out of memory" error.

I think the default heap size is 128 MB so I don't understand why my program runs out of memory.

Each node of my tree contains a string, a double (the word's frequency), and an ArrayList of nodes (to represent my list of words that could possible come after each word)

My hunch is that it runs out of memory when I add nodes to the ArrayList but I am not sure. I thought that maybe it has something to do with the fact that I keep running the program over and over ...

I am not sure about this ...

Anyone have any suggestions?

Answer 1

Well i can probably say its the Strings, doubles and Collections that are causing the memory problem. and might have something to do with Garbage Collection.



the first thing i would do is change those doubles to floats, unless of course you absolutely need the precision of the double.



second if your ArrayList is being populated with new Nodes, i would do some limiting on how many it holds, 8 and under seems sufficient for me. if its being populated with addresses that point to nodes elsewhere well you can obviously get away with a few more.



if you are still running out of memory try forcing garbage collection near the end when free memory is getting rather small



Thirdly Going to a more primitve data type instead of the two objects you use rather liberally , like an array of nodes instead of an ArrayList, or an array of bytes using UTF-8 to convert to string when u need it instead of a String. those two can get rather big with their overheads.

Answer 2

I take a little stroll down memory lane just about every day. Hummmm...I wonder if that kind of walking will burn calories? It's a whole lot easier and more fun than walking outdoors in the heat! Seriously, I am glad I can think back on the good times in the good old days!

Answer 3

I assume you're loading the text file into a string buffer. Make sure you clear the buffer when you draw the strings out, otherwise it adds up quickly.



Beyond that I can't think of anything off hand.