Added error handling for objstore not having enough memory to allocate

[Wapaul1]

#163

[mehrdadn]

Nice commit! But this fails the build on Windows. Can you do something like #if defined(WIN32) || defined(_WIN32)/#else/#endif to conditionally enable the platform-specific bits?

[mehrdadn]

By the way, there's a better way to check if you're out of memory: try allocating it, and then catch the error so you can handle it. I don't think you'd need to explicitly check how much free memory the system has at all (not just because there's a minor race condition, but also because it may be misleading too, since some memory might be used in non-critical ways by the system and still be able to be freed).

[Wapaul1]

I would do that, except that this allocate does not throw an exception, only a bus error. Thus, we have to check the available memory. This fix is just a patch until the new object store is implemented.

[mehrdadn]

That's strange, sorry I didn't realize that. In that case then just make it platform-dependent so it doesn't fail on other platforms. Note that even #include <sys/statvfs.h> fails on a system that doesn't include that header, so you have to be careful.

[mehrdadn]

I'm not sure if these are necessary for /dev/shm/, but if it behaves like typical file systems, then:

• If f_bavail also works, use that instead of f_bfree.
• If applicable, check if f_favail >= 1 as well. (Note: I only took a cursory glance at this and it may not actually be a valid check; I'm not sure.)

[robertnishihara]

I just tried the following in python scripts/shell.py (on Ubuntu)

import numpy as np
x = np.zeros(100000)
l = []
for i in range(1000):
  l.append(ray.put(x))

Eventually, I got an error saying

fatal error occured: Check failed at line 22 in /home/ubuntu/ray/src/utils.cc: (size + 100) <= (buffer.f_bsize * buffer.f_bfree) with message Not enough available memory for allocation by objstore

as expected.

However, if I then start up python scripts/shell.py again and do

import numpy as np
x = np.zeros(100000)
ray.put(x)

I immediately get the error

fatal error occured: Check failed at line 22 in /home/ubuntu/ray/src/utils.cc: (size + 100) <= (buffer.f_bsize * buffer.f_bfree) with message Not enough available memory for allocation by objstore

Do I need to clear out some files somewhere?

Also, this may be unrelated, but there were several instances where I got this error as soon as I tried anything

I0718 17:49:18.717566861   16251 subchannel.c:642]           Connect failed: {"created":"@1468864158.717542076","description":"Failed to connect to remote host","errno":107,"file":"src/core/lib/iomgr/error.c","file_line":256,"os_error":"Connection refused","target_address":"ipv4:127.0.0.1:10001"}
I0718 17:49:18.717596258   16251 subchannel.c:647]           Retry in 0.999912500 seconds

[robertnishihara]

I wasn't able to trigger the error on Mac OS X.

I ran

import numpy as np
x = np.zeros(10000000)
l = []
for i in range(1000):
    print i
    l.append(ray.put(x))

and that completed successfully.

>>> import sys
>>> sys.getsizeof(x)
80000096

So this should be storing 80GB in the object store.