TOPIC: Fatal Error: Out of Heap Space
#88
martin

Fatal Error: Out of Heap Space 4 Years, 5 Months ago
Hi everybody;

As i'm working with Suneido since a couple of months, I remarked a silly memory allocation problem. When Suneido executes, it always grow on used memory up to a point it crash with a fatal error.

Is there a bug in memory allocation functions of Suneido because simple code like below when executed for the 2nd times crash Suneido with the following error message: Fatal Error: Out of heap space.
Code:

function()
{
test = ''
for (i=0 ; i<4000000 ; ++i)
{
test = test $ '          '
}
test = ''
}



The example shown is a way exagerated by concatenating a lot of strings, but it effectively shows the problem when I have to concatenate big strings and it looks like the allocated memory is never released when we exit these functions.

Just check the memory used by suneido in the windows taskbar while running this function. Suneido is showing more and more used memory as it's running. But after you've run it the 1st time, you'll remark the used memory is never release. Normally when allocated memory is release, the memory should drop in the taskbar.

I remark this problem since a long times ago but I was unable to reproduce it on a regular basis until I have to make a library doing a lot of concatenate strings.

Martin
 
 
#89
Re:Fatal Error: Out of Heap Space 4 Years, 5 Months ago
Your example is quite extreme so I am not surprised it crashes. It is building a 40 mb string and the current release of Suneido has a maximum heap size of 64 mb. (I am curious how other languages like Python or Ruby would handle this test?)

We have seen a similar problem in some of our applications, although not from simple code like yours. The next release will use a different garbage collector which hopefully will help this problem. (See the Internals and Enhancements postings.)

However, memory will not always be returned to the operating system. Once the heap has grown to a certain size, it will normally not shrink. In the current release you can run Memory() and look at the total allocated and free space to see how much is actually in use. Also, the memory shown by the task manager also includes the memory mapped database.

One possible cause of the problem is fragmentation. If you allocate a large block, free it, then allocate some small blocks, the large block may get broken up into smaller blocks. If you then allocate another large block it may not be able to reuse the previous space.

Conservative garbage collection is not always "deterministic". It assumes that anything that looks like a pointer is a pointer. It may also see pointers in areas of memory (e.g. the stack) that are "old". These issues may cause memory to be kept allocated when it shouldn't. Suneido implements concatenation by making a linked list of string fragments. In combination with conservative garbage collection, this can lead to an entire linked list being kept alive when it should not.
 
 
andrew
 
#90
martin

Re:Fatal Error: Out of Heap Space 4 Years, 5 Months ago
Thank you for your fast answer Andrew.

I'm really sad to learn this kind of limitation exists in Suneido. I was on the impression Suneido was working like C++ or pascal which can use all the free ram we can give it. Today with computers having around 1 Gig of ram inside, 64 Meg isn't simply enough to play with.

I'll have to probably go back to what I was using before because I can't trust a programming language which can crash anytime without warning because I have done too much concatenate strings.

I'm curious to see the next release because with this kind of limitation, it has put a stop on what I can use Suneido for.

And again, thank you Andrew.

Martin
 
 
#93
Re:Fatal Error: Out of Heap Space 4 Years, 5 Months ago
The next release does not have a fixed limit on heap size. It should be out in the next few weeks.
 
 
andrew
 
#99
Re:Fatal Error: Out of Heap Space 4 Years, 5 Months ago
I ran your test code on the new release. After running it 4 times the heap size stops growing.
 
 
andrew
 
#101
Re:Fatal Error: Out of Heap Space 4 Years, 5 Months ago
Out of curiousity, I ran a similar test on Ruby:

Code:

t = Time.now
s = ""
4000000.times do
s = s + "          "
end
s = ""
print Time.now - t



I started it Sat. afternoon. The memory as reported by the Windows Task Manager jumps up and down a lot - I am not sure why. It had not finished when I went home. When I came back to work Monday morning (about 36 hours later), it was still running! (This test takes about 5 seconds on the new release of Suneido.) Maybe something had gone wrong, but it appeared to be still working. In any case it was taking orders of magnitude longer than Suneido. The memory was also up to about 500mb - quite a bit higher than Suneido.

I do not want to be critical of Ruby - I think it is a good language. And I am sure there is probably a more efficient way of doing this in Ruby. But it does show Suneido is not the only language to have trouble with this example.
 
 
andrew
 
#111
martin

Re:Fatal Error: Out of Heap Space 4 Years, 5 Months ago
I downloaded the new beta version and I can say I'm amazed about this new version.

This new version is almost 2 times faster than the old one for manipulating strings, wow! what an improvement.

And with the 64Mb memory limit removed, I'm in heaven.

Wow! You always find a way to impress us Andrew!

I should say thank you again for this good work!