A large fraction of process crashes in Chromium are due to Out Of Memory (OOM) conditions. This page is meant to help Chromium developers understand stack traces, and investigate. Note that some of the documentation here will only be applicable to Google Chrome, as it is specific to the way Google’s crash reporting infrastructure aggregates and reports crashes.
Some of the following also assumes that the malloc()
implementation is
PartitionAlloc, which is as of 2022 the case on most platforms.
[TOC]
When a process crashes due to an Out Of Memory condition, this is usually
signaled by the presence of base::internal::OnNoMemoryInternal()
on the stack.
Google Chrome only: crash report infrastructure tags these as «[Out of Memory]» based on this, and other function names. The full list is determined in the (internal) crash server’s code.
Since Chromium configures its memory allocators to prefer crashing rather than
returning nullptr
, an OOM crash can be triggered from anywhere in the code,
and most commonly from within the allocator, or higher-level functions such as
operator new
in C++.
A process can reach an OOM condition for several reasons:
In the case of PartitionAlloc, it is possible to distinguish some of these cases:
PartitionOutOfMemoryMappingFailure()
on the stack. It means that the
allocator was unable to find enough address space, either for its internal
memory allocation unit size, or the requested size. Since memory is not
committed as this step, this signals an address space issue.PartitionOutOfMemoryCommitFailure()
on the stack. This signals that either
the OS or the sandbox limit has been reached.PartitionExcessiveAllocationSize()
on the stack.The process is «truly» out of memory, or the system is. Some amount of these crashes is expected, and the crashing location is not necessarily the culprit. Indeed, as a rough approximation, the failing allocation is more likely to be from a component naturally allocating a lot of memory, e.g. V8 or rendering.
However, if there is a spike, and many stack traces come from an unusual location (e.g. newly added code), this may signal a memory leak in the component on the stack, or excessive temporary allocations.
Also, if PartitionAllocDirectMap()
is on the stack, the memory allocation was
large. It may come from a large buffer, and potentially made worse by buffer
resizing. For instance, std::vector
often double their size when out of
capacity. In which case, reserve()
-ing the right size ahead of time may help.
Is the calling code expected to allocate more than 2GiB? Or it is an underflow somewhere in the calling code?
On 32 bit systems, this is most likely to occur when overall memory usage is high, or when the allocation size request is large. Is the calling code allocating a very large buffer?
On Windows, the allocation size is added into the exception record. In Google Chrome’s crash dashboard, this is shown in «Parameter[0]» of the exception info. On other operating systems, the allocation size if put on the stack before crashing, and thus visible in minidumps.
PartitionRoot<true>::OutOfMemory()
frame on the stack, move to it with f 5
re re
x <stack_pointer> <frame pointer>
Below is an example for a crash on x86_64:
( lizeb ) bt
* thread #1, stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x10c45912f)
* frame #0: 0x000000010c45912f Google Chrome Framework`base::internal::OnNoMemoryInternal(unsigned long) at memory.cc:62
frame #1: 0x000000010c459149 Google Chrome Framework`base::TerminateBecauseOutOfMemory(unsigned long) at memory.cc:69
frame #2: 0x000000010c4f39c6 Google Chrome Framework`OnNoMemory(unsigned long) at oom.cc:17
frame #3: 0x000000010d7e5794 Google Chrome Framework`WTF::PartitionsOutOfMemoryUsing2G(unsigned long) at partitions.cc:281
frame #4: 0x000000010d7e4d2c Google Chrome Framework`WTF::Partitions::HandleOutOfMemory(unsigned long) at partitions.cc:415
frame #5: 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) at partition_root.cc:521
[...]
( lizeb ) f 5
frame #5: 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) at partition_root.cc:521
( lizeb ) re re
General Purpose Registers:
rbp = 0x00007ffee7012c50
rsp = 0x00007ffee7012bf0
rip = 0x000000010c4f7474 Google Chrome Framework`base::PartitionRoot<true>::OutOfMemory(unsigned long) + 196 at partition_root.cc:522
21 registers were unavailable.
( lizeb ) x 0x00007ffee7012bf0 0x00007ffee7012c50
0x7ffee7012bf0: 76 61 5f 73 69 7a 65 00 00 00 00 07 00 00 00 00 va_size.........
0x7ffee7012c00: 61 6c 6c 6f 63 00 20 20 00 2d 2d 01 00 00 00 00 alloc. .--.....
0x7ffee7012c10: 63 6f 6d 6d 69 74 00 20 00 a0 9d 01 00 00 00 00 commit. ........
0x7ffee7012c20: 73 69 7a 65 00 20 20 20 00 00 20 00 00 00 00 00 size. .. .....
0x7ffee7012c30: aa aa aa aa aa aa aa aa 00 18 b0 12 01 00 00 00 ................
0x7ffee7012c40: 00 00 20 00 00 00 00 00 48 22 b0 12 01 00 00 00 .. .....H"......
The results here can help the PartitionAlloc team to identify issues, as important metrics from PartitionAlloc are saved above. For instance virtual address space usage is (in little endian) 0x70000000.