Memory Management

There is a new approach to memory management that claims to be able to speed up applications. Actually, it has the potential to but nothing is carved in stone. It would be interesting to contrast the performance gained versus the stability lost after implementing this method.

Trying to break Python memory consumption

I've been trying to find consistent method to raise a MemoryError exception in Python and have so far been unsuccessful. I'm mostly interested in real-world usage scenarios such has executing huge number arithmetic. Here is what my latest test looks like.

#Example; Massive memory consumption.

if __name__=="__main__":
   list_obj=[]
   cnt=0
   while cnt<10**24:
       list_obj.append(cnt**2)
       cnt+=1

Here, we are simply appending progressively larger integers to a list. Executing this on my Ubuntu laptop results a huge increase in memory consumption. Because we are constructing a massive list object, this would make sense. However, a memory error isn't raised for me. Figuring that my laptop has enough memory to not let any kind of memory mishap take place for awhile, I fired-up a virtual machine with much less memory capabilities and executed the same program. The Python process ends up being killed. No MemoryError exception. What gives?

If I want to handle memory errors in Python, how can I deal with it if the process is terminated before I get a chance to?

Python memory Usage

Here is an example in Python of how to retrieve the system memory usage. This example was adapted from an entry on stackoverflow.

#Example; Get the system memory usage.

import subprocess

class MemUsage(object):
  def __init__(self):
      self.total=0
      self.used=0
      self.free=0
      self.shared=0
      self.buffers=0
      self.cached=0
      self.init_data()

  def init_data(self):
      command="free"
      process=subprocess.Popen(command,\
                               shell=True,\
                               stdout=subprocess.PIPE)
      stdout_list=process.communicate()[0].split('\n')
      for line in stdout_list:
          data=line.split()
          try:
              print data
              if data[0]=="Mem:":
                  self.total=float(data[1])
                  self.used=float(data[2])
                  self.free=float(data[3])
                  self.shared=float(data[4])
                  self.buffers=float(data[5])
                  self.cached=float(data[6])
          except IndexError:
              continue

  def calculate(self):
      return ((self.used-self.buffers-self.cached)/self.total)*100

  def __repr__(self):
      return str(self.calculate())

if __name__=="__main__":
  print MemUsage()
             

Here we have a simple class called MemUsage. The constructor initializes the attributes of the class needed to compute the memory usage. The init_data() method is what MemUsage invokes in order to retrieve the required system data. This is done by using the subprocess module to execute the free command. The resulting data is then mapped to the corresponding attributes. We compute the memory usage as a percentage by subtracting the buffers and cache from the used memory and dividing the result by the total memory.