oom.md 7.4 KB

Investigating Out of Memory crashes

A large fraction of process crashes in Chromium are due to Out Of Memory (OOM) conditions. This page is meant to help Chromium developers understand stack traces, and investigate. Note that some of the documentation here will only be applicable to Google Chrome, as it is specific to the way Google’s crash reporting infrastructure aggregates and reports crashes.

Some of the following also assumes that the malloc() implementation is PartitionAlloc, which is as of 2022 the case on most platforms.

[TOC]

Identifying OOM crashes

When a process crashes due to an Out Of Memory condition, this is usually signaled by the presence of base::internal::OnNoMemoryInternal() on the stack.

Google Chrome only: crash report infrastructure tags these as «[Out of Memory]» based on this, and other function names. The full list is determined in the (internal) crash server’s code.

Since Chromium configures its memory allocators to prefer crashing rather than returning nullptr, an OOM crash can be triggered from anywhere in the code, and most commonly from within the allocator, or higher-level functions such as operator new in C++.

Distinguishing between underlying causes

Different causes

A process can reach an OOM condition for several reasons:

  • The OS is truly out of memory, regardless of how much memory the current process is using
  • Some limit inside the OS is reached. For instance, on Windows, there exists a global «commit limit», which is the amount of memory that the system can commit. Note that it is possible to commit more memory than what is actually in use. This may also happen on Linux systems configured with no or limited «overcommit», though the majority of systems don’t have a limit.
  • Virtual address space exhaustion. This is most likely to happen for relatively large allocations, on 32 bit systems, where total addressable space is typically 2GiB (most Windows systems), 3GiB (e.g. some Windows configurations, Linux) or 4GiB (e.g. WoW64). However, it may also happen on 64 bit systems, either due to:
    • Limited virtual addressable space in the CPU/OS. For instance most Android ARM64 systems have only 40 bits of address space as of 2022.
    • «Cage» exhaustion. This is most likely to happen with PartitionAlloc on 64 bit systems, where all allocations are grouped into a single contiguous virtual address space «cage».
  • Sandbox per-process memory limit. For some process types (e.g. Renderers) and on most platforms, the sandbox enforces a maximum per-process memory limit. Given that this limit is typically set at the OS level, it may not be distinguishable from e.g. commit limit exhaustion.
  • Excessive allocation size. Some allocators (notably PartitionAlloc) purposely limit the maximum allocation size.

Identifying the cause

In the case of PartitionAlloc, it is possible to distinguish some of these cases:

  • Virtual address space exhaustion. This is identified by the presence of PartitionOutOfMemoryMappingFailure() on the stack. It means that the allocator was unable to find enough address space, either for its internal memory allocation unit size, or the requested size. Since memory is not committed as this step, this signals an address space issue.
  • Commit. This is identified by the presence of PartitionOutOfMemoryCommitFailure() on the stack. This signals that either the OS or the sandbox limit has been reached.
  • Excessive allocation size. Shown by PartitionExcessiveAllocationSize() on the stack.

What to do?

Commit Limit Reached

The process is «truly» out of memory, or the system is. Some amount of these crashes is expected, and the crashing location is not necessarily the culprit. Indeed, as a rough approximation, the failing allocation is more likely to be from a component naturally allocating a lot of memory, e.g. V8 or rendering.

However, if there is a spike, and many stack traces come from an unusual location (e.g. newly added code), this may signal a memory leak in the component on the stack, or excessive temporary allocations.

Also, if PartitionAllocDirectMap() is on the stack, the memory allocation was large. It may come from a large buffer, and potentially made worse by buffer resizing. For instance, std::vector often double their size when out of capacity. In which case, reserve()-ing the right size ahead of time may help.

Excessive allocation size

Is the calling code expected to allocate more than 2GiB? Or it is an underflow somewhere in the calling code?

Virtual address space

On 32 bit systems, this is most likely to occur when overall memory usage is high, or when the allocation size request is large. Is the calling code allocating a very large buffer?

Debugging

General

On Windows, the allocation size is added into the exception record. In Google Chrome’s crash dashboard, this is shown in «Parameter[0]» of the exception info. On other operating systems, the allocation size if put on the stack before crashing, and thus visible in minidumps.

PartitionAlloc and Google specific

  1. Starting from a specific report, click on the bug icon to start a cloud lldb instance
  2. Locate the PartitionRoot<true>::OutOfMemory() frame on the stack, move to it with f 5
  3. Locate the stack addresses by printing registers re re
  4. Show the stack content with x <stack_pointer> <frame pointer>

Below is an example for a crash on x86_64:

( lizeb ) bt
* thread #1, stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x10c45912f)
  * frame #0: 0x000000010c45912f Google Chrome Framework`base::internal::OnNoMemoryInternal(unsigned long) at memory.cc:62
    frame #1: 0x000000010c459149 Google Chrome Framework`base::TerminateBecauseOutOfMemory(unsigned long) at memory.cc:69
    frame #2: 0x000000010c4f39c6 Google Chrome Framework`OnNoMemory(unsigned long) at oom.cc:17
    frame #3: 0x000000010d7e5794 Google Chrome Framework`WTF::PartitionsOutOfMemoryUsing2G(unsigned long) at partitions.cc:281
    frame #4: 0x000000010d7e4d2c Google Chrome Framework`WTF::Partitions::HandleOutOfMemory(unsigned long) at partitions.cc:415
    frame #5: 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) at partition_root.cc:521
[...]
( lizeb ) f 5
frame #5: 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) at partition_root.cc:521
( lizeb ) re re
General Purpose Registers:
       rbp = 0x00007ffee7012c50
       rsp = 0x00007ffee7012bf0
       rip = 0x000000010c4f7474  Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) + 196 at partition_root.cc:522
21 registers were unavailable.
( lizeb ) x 0x00007ffee7012bf0 0x00007ffee7012c50
0x7ffee7012bf0: 76 61 5f 73 69 7a 65 00 00 00 00 07 00 00 00 00  va_size.........
0x7ffee7012c00: 61 6c 6c 6f 63 00 20 20 00 2d 2d 01 00 00 00 00  alloc.  .--.....
0x7ffee7012c10: 63 6f 6d 6d 69 74 00 20 00 a0 9d 01 00 00 00 00  commit. ........
0x7ffee7012c20: 73 69 7a 65 00 20 20 20 00 00 20 00 00 00 00 00  size.   .. .....
0x7ffee7012c30: aa aa aa aa aa aa aa aa 00 18 b0 12 01 00 00 00  ................
0x7ffee7012c40: 00 00 20 00 00 00 00 00 48 22 b0 12 01 00 00 00  .. .....H"......

The results here can help the PartitionAlloc team to identify issues, as important metrics from PartitionAlloc are saved above. For instance virtual address space usage is (in little endian) 0x70000000.