However, fragmentation can sometimes cause Ruby programs to twice as much memory as they would otherwise, sometimes as much as four times more! Note the enormous drop after MALLOC_ARENA_MAX changed to 2. It’s a fairly common issue in C programs, not just Ruby.Īctual client graph. If we can’t move objects in memory, fragmentation is an inevitable result. Doing so would potentially break any C language extensions which are holding raw pointers to a Ruby object. In particular, Ruby cannot move objects in memory. It’s an inevitable consequence of how Ruby manages memory. All Ruby processes experience some memory fragmentation. It looks like a long curve, approaching some unseen limit. Memory fragmentation causes logarithmic growth in memory. For example, if your Markdown parser leaks 10kb every time you call it, your memory growth will continue forever at a linear rate, since you tend to call the markdown parser at a regular frequency. However, unlike a memory leak, memory growth due to fragmentation is logarithmic, while memory leaks are linear.Ī memory leak in a Ruby program is usually caused by a C-extension bug. The problem manifests itself as a slow, creeping memory growth in Ruby processes. Typical Ruby memory growth looks logarithmic. The reasons are many, but one reason in particular is extremely difficult to diagnose and debug: fragmentation. Sidekiq, the popular Ruby background job processor, has processes which can get just as large or even larger. Many Rails applications suffer from up to a gigabyte of memory use per process. Ruby is not known for being a language that’s light on memory use. You may not be suffering the problem it solves. Now, before you go copy-pasting this “magical” environment variable into all of your application environments, know this: there are drawbacks. Processes settled at about half the memory usage they had before - around 512MB each.Īctually, it’s not that simple. His processes restarted, and immediately the slow growth was eliminated. I asked him to change a single environment variable: MALLOC_ARENA_MAX. They would start at about 300MB each, then slowly grow over the course of several hours to almost a gigabyte, where they would start to level off. I had a client whose Sidekiq processes were using a lot of memory - about a gigabyte each. It’s not every day that a simple configuration change can completely solve a problem. Let's look at one source of memory fragmentation in multi-threaded CRuby programs: malloc's per-thread memory arenas. Summary: Memory fragmentation is difficult to measure and diagnose, but it can also sometimes be very easy to fix.

After effects pixel sorter memory leak