Critically Low on Memory

[power1001]

I get this message whenever I load a lot of images.  My Imac has 8GB.  Does the program swap memory with the hard drive ?  I'm personally not so much concerned with speed as I am with not hanging or failing.  Is there a setting adjustment or how to know how much memory / resources are used or available ?

I know it takes more resources, but it would also be nice if there was a task completion percentage bar or something like that to know when its done and when you can move to next step, etc.

[Sander Berents]

For the following explanation please open Activity Monitor (/Applications/Utilities), and select its Memory tab. Click on the Memory column to sort the listed applications by memory usage in descending order. Put the window next to the Observatory library window.


If you select an image, Observatory will start loading it in memory. Even if this is not strictly necessary. It does this in the background, so the application remains responsive. You can see that this is happening because the "Stop Tasks" button in the toolbar is temporarily enabled.

Whenever you see this button enabled, you know that Observatory is doing something in the background. These will be tasks you initiated (e.g. image loading because of image selection, or applying adjustments, or creating a stack), but Observatory itself also initiates background tasks. For example to generate a thumbnail for an image or for a stack. Sometimes you don't want to wait for that, because you realize you made a wrong selection. In that case, just click the "Stop Tasks" toolbar button (or press ⌘.). This will stop all the active tasks, reset your selection, and even adds a 30 second delay for the automatic background tasks. The tasks are stopped in a graceful way, so it sometimes may take a few seconds.

If you select images like this and look at Observatory's memory usage in Activity Monitor, you will see its memory usage goes up as you select more images. But, while you add more and more images to your selection, at a certain point you will see the memory consumption suddenly goes down significantly. Observatory has then entered a memory preserving mode, which is slower, but allows it to keep more images in memory. When this happens depends on how much memory your Mac has. This memory preserving mode is about twice as efficient in terms of memory for images that do not have any adjustments applied to them, so you will typically be able to select twice the number of dark frames compared to light frames, if the latter have adjustments.

As you continue to add images to your selection, the memory usage may go up so far that the graph at the bottom of Activity Monitor changes from the usual green color, to yellow or even red. While memory usage increases, your Mac makes more memory available by swapping some to disk, and by the time it reaches the red color your system's performance will be impacted. Not only Observatory's, but your whole Mac. If your Mac is equipped with a hard drive, it will typically be very unresponsive in the red zone. It's not crashed or stuck, but you probably don't want it in that state. If your Mac is equiped with an SSD, even in the red zone it is typically still fairly responsive, so you can push Observatory much further on such Macs.

The yellow zone corresponds to the "The system is running low on memory" warning in Observatory, the red zone is the "The system is critically low on memory" warning in Observatory. The actual warnings shown in Observatory are triggered by macOS itself, and if it quickly goes from yellow to red, it often skips the first warning level. Activity Monitor may also already display the red level for a while until Observatory finally shows the warning.

So what do you do when you get these warnings?

The first one is there to make you realize that Observatory is limited by your system's available memory. Observatory originally was intended for organizing images obtained with monochrome astronomical cameras, and making it easy to deal with FITS and XISF files on the Mac. But many users with DSLR cameras use it for stacking and light processing too, and those cameras always have many pixels, and always generate color images. Those images take up a lot of memory.

For example an image taken with a monochrome camera with 16 megapixels expands 64 MB in RAM, while a color image with 16 megapixels expands to 192 MB (Observatory works with float pixels). When you stack 20 such color images, Observatory loads all of them in memory, consuming nearly 4 GB. As long as you don't apply any adjustment to any of them, during stacking that would be approximately what Observatory will use, but if you apply adjustments to the images it may consume closer to 8 GB because of Observatory's nondestructive behavior.

Anyway, when you see this warning, you typically just click the Continue button and observe how it goes. After a while, you may even want to select its "Do not show this message again" checkbox (if you change your mind later, click the Reset Warnings button in General Preferences). Observatory has switched to the memory preserving mode by now for sure.

If you see the "The system is critically low on memory" warning for the first time then I would suggest to click Stop Tasks, and reduce the selection or stack size up to the point it completes the task without showing this warning.

Another way is for example by creating an Observatory library with 50 images, and do some experiments with stacking 5 of them, then 10, then 15, etc, to see when it starts displaying the warnings (click the Stop Tasks button in the toolbar to interrupt processing when needed). Repeat the same experiment after applying an adjustment to the images, for example Flatten Background. This way you learn the limits for calibration frames and light frames.

For systems with a hard disk, this may be the limit beyond which you don't want to go. To let Observatory set this limit for you, click the "Automatically stop all ongoing tasks" checkbox. That is, unless you want to push Observatory beyond this limit, and let it complete the tasks overnight. On systems with an SSD, you will find that you can go well beyond this point.

How to stack 100 images in Observatory if your system becomes unresponsive beyond 50 images? Just create 2 stacks of 50 images each. Create 2 managed masters of these stacks, and then stack them. It's not ideal, but it allows you to go well beyond any limits.

The more memory your Mac has, the further you can push Observatory. But replacing the hard disk with an SSD is nearly always the best first thing to do. It makes a huge difference.

To answer your last question:

Observatory displays multiple progress indicators, but most of them indeed do not let you estimate how much longer it will take. But the one in the Editor does for stacking. Select the stack image to see it.

[power1001]

Sander,

Thank you for such a comprehensive (and interesting) answer.  This will make my work flow much more efficient.