I've been trying to find consistent method to raise a
MemoryError exception in
Python and have so far been unsuccessful. I'm mostly interested in real-world usage scenarios such has executing huge number arithmetic. Here is what my latest test looks like.
#Example; Massive memory consumption.
if __name__=="__main__":
list_obj=[]
cnt=0
while cnt<10**24:
list_obj.append(cnt**2)
cnt+=1
Here, we are simply appending progressively larger integers to a list. Executing this on my
Ubuntu laptop results a huge increase in memory consumption. Because we are constructing a massive list object, this would make sense. However, a memory error isn't raised for me. Figuring that my laptop has enough memory to not let any kind of memory mishap take place for awhile, I fired-up a virtual machine with much less memory capabilities and executed the same program. The
Python process ends up being killed. No
MemoryError exception. What gives?
If I want to handle memory errors in
Python, how can I deal with it if the process is terminated before I get a chance to?
Linux generally follows a policy of overallocation. What this means is that malloc() will almost always succeed, but pages won't actually be allocated until you attempt to write to them. By that point however, it's too late to signal an error - the malloc is long past, so the kernel has to take drastic measures, by killing a process in the hopes of freeing up memory.
ReplyDeleteIf you'd allocated it as zeroed memory, rather than attempting to write to it. eg mem = [0] * 2**24, you'd have succeeded until you attempted to write a non-zero value to the memory.
There is one case where this will return an error however, and that's when you run out of address space, rather than physical memory. On a 32 bit processor, you won't be able to address more than 2 GiB, so allocations of that amount will probably fail immediately with a MemoryError. (In fact, since memory will fragment, even much smaller ranges may fail as no contiguous block exists in that range)
I strongly suspect that the process is getting a signal of some sort, perhaps SIGSEGV. You might have greater luck in setting up a signal handler.
ReplyDeleteYour process is probably getting killed by the Out Of Memory killer (oom killer) in the kernel. Check in /var/log/messages, you should see a message from the oom killer.
ReplyDeleteSadly my montecarlo analysis app receives MemoryError exceptions all the time. :( I hit the 3GB (not 2) limit for a 32bit app under Linux. You don't say how much memory your computer has but I'm guessing that you're swapping before hitting 3G which is making something else break before you hit the limit. (Which probably means you should be worrying about other issues before this one most likely)
ReplyDeleteTo generate the exception run on a 4G machine and just create a list, copy the list to a new one, append them together, repeat. Shouldn't take more than a few minutes on a fast machine with psyc like mine. Maybe longer without psyc.
Good luck!