1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174117511761177117811791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219 |
- .. include:: <isonum.txt>
- ============================================
- Reliability, Availability and Serviceability
- ============================================
- RAS concepts
- ************
- Reliability, Availability and Serviceability (RAS) is a concept used on
- servers meant to measure their robustness.
- Reliability
- is the probability that a system will produce correct outputs.
- * Generally measured as Mean Time Between Failures (MTBF)
- * Enhanced by features that help to avoid, detect and repair hardware faults
- Availability
- is the probability that a system is operational at a given time
- * Generally measured as a percentage of downtime per a period of time
- * Often uses mechanisms to detect and correct hardware faults in
- runtime;
- Serviceability (or maintainability)
- is the simplicity and speed with which a system can be repaired or
- maintained
- * Generally measured on Mean Time Between Repair (MTBR)
- Improving RAS
- -------------
- In order to reduce systems downtime, a system should be capable of detecting
- hardware errors, and, when possible correcting them in runtime. It should
- also provide mechanisms to detect hardware degradation, in order to warn
- the system administrator to take the action of replacing a component before
- it causes data loss or system downtime.
- Among the monitoring measures, the most usual ones include:
- * CPU – detect errors at instruction execution and at L1/L2/L3 caches;
- * Memory – add error correction logic (ECC) to detect and correct errors;
- * I/O – add CRC checksums for transferred data;
- * Storage – RAID, journal file systems, checksums,
- Self-Monitoring, Analysis and Reporting Technology (SMART).
- By monitoring the number of occurrences of error detections, it is possible
- to identify if the probability of hardware errors is increasing, and, on such
- case, do a preventive maintenance to replace a degraded component while
- those errors are correctable.
- Types of errors
- ---------------
- Most mechanisms used on modern systems use technologies like Hamming
- Codes that allow error correction when the number of errors on a bit packet
- is below a threshold. If the number of errors is above, those mechanisms
- can indicate with a high degree of confidence that an error happened, but
- they can't correct.
- Also, sometimes an error occur on a component that it is not used. For
- example, a part of the memory that it is not currently allocated.
- That defines some categories of errors:
- * **Correctable Error (CE)** - the error detection mechanism detected and
- corrected the error. Such errors are usually not fatal, although some
- Kernel mechanisms allow the system administrator to consider them as fatal.
- * **Uncorrected Error (UE)** - the amount of errors happened above the error
- correction threshold, and the system was unable to auto-correct.
- * **Fatal Error** - when an UE error happens on a critical component of the
- system (for example, a piece of the Kernel got corrupted by an UE), the
- only reliable way to avoid data corruption is to hang or reboot the machine.
- * **Non-fatal Error** - when an UE error happens on an unused component,
- like a CPU in power down state or an unused memory bank, the system may
- still run, eventually replacing the affected hardware by a hot spare,
- if available.
- Also, when an error happens on a userspace process, it is also possible to
- kill such process and let userspace restart it.
- The mechanism for handling non-fatal errors is usually complex and may
- require the help of some userspace application, in order to apply the
- policy desired by the system administrator.
- Identifying a bad hardware component
- ------------------------------------
- Just detecting a hardware flaw is usually not enough, as the system needs
- to pinpoint to the minimal replaceable unit (MRU) that should be exchanged
- to make the hardware reliable again.
- So, it requires not only error logging facilities, but also mechanisms that
- will translate the error message to the silkscreen or component label for
- the MRU.
- Typically, it is very complex for memory, as modern CPUs interlace memory
- from different memory modules, in order to provide a better performance. The
- DMI BIOS usually have a list of memory module labels, with can be obtained
- using the ``dmidecode`` tool. For example, on a desktop machine, it shows::
- Memory Device
- Total Width: 64 bits
- Data Width: 64 bits
- Size: 16384 MB
- Form Factor: SODIMM
- Set: None
- Locator: ChannelA-DIMM0
- Bank Locator: BANK 0
- Type: DDR4
- Type Detail: Synchronous
- Speed: 2133 MHz
- Rank: 2
- Configured Clock Speed: 2133 MHz
- On the above example, a DDR4 SO-DIMM memory module is located at the
- system's memory labeled as "BANK 0", as given by the *bank locator* field.
- Please notice that, on such system, the *total width* is equal to the
- *data width*. It means that such memory module doesn't have error
- detection/correction mechanisms.
- Unfortunately, not all systems use the same field to specify the memory
- bank. On this example, from an older server, ``dmidecode`` shows::
- Memory Device
- Array Handle: 0x1000
- Error Information Handle: Not Provided
- Total Width: 72 bits
- Data Width: 64 bits
- Size: 8192 MB
- Form Factor: DIMM
- Set: 1
- Locator: DIMM_A1
- Bank Locator: Not Specified
- Type: DDR3
- Type Detail: Synchronous Registered (Buffered)
- Speed: 1600 MHz
- Rank: 2
- Configured Clock Speed: 1600 MHz
- There, the DDR3 RDIMM memory module is located at the system's memory labeled
- as "DIMM_A1", as given by the *locator* field. Please notice that this
- memory module has 64 bits of *data width* and 72 bits of *total width*. So,
- it has 8 extra bits to be used by error detection and correction mechanisms.
- Such kind of memory is called Error-correcting code memory (ECC memory).
- To make things even worse, it is not uncommon that systems with different
- labels on their system's board to use exactly the same BIOS, meaning that
- the labels provided by the BIOS won't match the real ones.
- ECC memory
- ----------
- As mentioned in the previous section, ECC memory has extra bits to be
- used for error correction. In the above example, a memory module has
- 64 bits of *data width*, and 72 bits of *total width*. The extra 8
- bits which are used for the error detection and correction mechanisms
- are referred to as the *syndrome*\ [#f1]_\ [#f2]_.
- So, when the cpu requests the memory controller to write a word with
- *data width*, the memory controller calculates the *syndrome* in real time,
- using Hamming code, or some other error correction code, like SECDED+,
- producing a code with *total width* size. Such code is then written
- on the memory modules.
- At read, the *total width* bits code is converted back, using the same
- ECC code used on write, producing a word with *data width* and a *syndrome*.
- The word with *data width* is sent to the CPU, even when errors happen.
- The memory controller also looks at the *syndrome* in order to check if
- there was an error, and if the ECC code was able to fix such error.
- If the error was corrected, a Corrected Error (CE) happened. If not, an
- Uncorrected Error (UE) happened.
- The information about the CE/UE errors is stored on some special registers
- at the memory controller and can be accessed by reading such registers,
- either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64
- bit CPUs, such errors can also be retrieved via the Machine Check
- Architecture (MCA)\ [#f3]_.
- .. [#f1] Please notice that several memory controllers allow operation on a
- mode called "Lock-Step", where it groups two memory modules together,
- doing 128-bit reads/writes. That gives 16 bits for error correction, with
- significantly improves the error correction mechanism, at the expense
- that, when an error happens, there's no way to know what memory module is
- to blame. So, it has to blame both memory modules.
- .. [#f2] Some memory controllers also allow using memory in mirror mode.
- On such mode, the same data is written to two memory modules. At read,
- the system checks both memory modules, in order to check if both provide
- identical data. On such configuration, when an error happens, there's no
- way to know what memory module is to blame. So, it has to blame both
- memory modules (or 4 memory modules, if the system is also on Lock-step
- mode).
- .. [#f3] For more details about the Machine Check Architecture (MCA),
- please read Documentation/x86/x86_64/machinecheck.rst at the Kernel tree.
- EDAC - Error Detection And Correction
- *************************************
- .. note::
- "bluesmoke" was the name for this device driver subsystem when it
- was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net.
- That site is mostly archaic now and can be used only for historical
- purposes.
- When the subsystem was pushed upstream for the first time, on
- Kernel 2.6.16, it was renamed to ``EDAC``.
- Purpose
- -------
- The ``edac`` kernel module's goal is to detect and report hardware errors
- that occur within the computer system running under linux.
- Memory
- ------
- Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the
- primary errors being harvested. These types of errors are harvested by
- the ``edac_mc`` device.
- Detecting CE events, then harvesting those events and reporting them,
- **can** but must not necessarily be a predictor of future UE events. With
- CE events only, the system can and will continue to operate as no data
- has been damaged yet.
- However, preventive maintenance and proactive part replacement of memory
- modules exhibiting CEs can reduce the likelihood of the dreaded UE events
- and system panics.
- Other hardware elements
- -----------------------
- A new feature for EDAC, the ``edac_device`` class of device, was added in
- the 2.6.23 version of the kernel.
- This new device type allows for non-memory type of ECC hardware detectors
- to have their states harvested and presented to userspace via the sysfs
- interface.
- Some architectures have ECC detectors for L1, L2 and L3 caches,
- along with DMA engines, fabric switches, main data path switches,
- interconnections, and various other hardware data paths. If the hardware
- reports it, then a edac_device device probably can be constructed to
- harvest and present that to userspace.
- PCI bus scanning
- ----------------
- In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors
- in order to determine if errors are occurring during data transfers.
- The presence of PCI Parity errors must be examined with a grain of salt.
- There are several add-in adapters that do **not** follow the PCI specification
- with regards to Parity generation and reporting. The specification says
- the vendor should tie the parity status bits to 0 if they do not intend
- to generate parity. Some vendors do not do this, and thus the parity bit
- can "float" giving false positives.
- There is a PCI device attribute located in sysfs that is checked by
- the EDAC PCI scanning code. If that attribute is set, PCI parity/error
- scanning is skipped for that device. The attribute is::
- broken_parity_status
- and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for
- PCI devices.
- Versioning
- ----------
- EDAC is composed of a "core" module (``edac_core.ko``) and several Memory
- Controller (MC) driver modules. On a given system, the CORE is loaded
- and one MC driver will be loaded. Both the CORE and the MC driver (or
- ``edac_device`` driver) have individual versions that reflect current
- release level of their respective modules.
- Thus, to "report" on what version a system is running, one must report
- both the CORE's and the MC driver's versions.
- Loading
- -------
- If ``edac`` was statically linked with the kernel then no loading
- is necessary. If ``edac`` was built as modules then simply modprobe
- the ``edac`` pieces that you need. You should be able to modprobe
- hardware-specific modules and have the dependencies load the necessary
- core modules.
- Example::
- $ modprobe amd76x_edac
- loads both the ``amd76x_edac.ko`` memory controller module and the
- ``edac_mc.ko`` core module.
- Sysfs interface
- ---------------
- EDAC presents a ``sysfs`` interface for control and reporting purposes. It
- lives in the /sys/devices/system/edac directory.
- Within this directory there currently reside 2 components:
- ======= ==============================
- mc memory controller(s) system
- pci PCI control and status system
- ======= ==============================
- Memory Controller (mc) Model
- ----------------------------
- Each ``mc`` device controls a set of memory modules [#f4]_. These modules
- are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``).
- There can be multiple csrows and multiple channels.
- .. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely
- used to refer to a memory module, although there are other memory
- packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI
- specification (Version 2.7) defines a memory module in the Common
- Platform Error Record (CPER) section to be an SMBIOS Memory Device
- (Type 17). Along this document, and inside the EDAC subsystem, the term
- "dimm" is used for all memory modules, even when they use a
- different kind of packaging.
- Memory controllers allow for several csrows, with 8 csrows being a
- typical value. Yet, the actual number of csrows depends on the layout of
- a given motherboard, memory controller and memory module characteristics.
- Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems)
- data transfers to/from the CPU from/to memory. Some newer chipsets allow
- for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory
- controllers. The following example will assume 2 channels:
- +------------+-----------------------+
- | CS Rows | Channels |
- +------------+-----------+-----------+
- | | ``ch0`` | ``ch1`` |
- +============+===========+===========+
- | |**DIMM_A0**|**DIMM_B0**|
- +------------+-----------+-----------+
- | ``csrow0`` | rank0 | rank0 |
- +------------+-----------+-----------+
- | ``csrow1`` | rank1 | rank1 |
- +------------+-----------+-----------+
- | |**DIMM_A1**|**DIMM_B1**|
- +------------+-----------+-----------+
- | ``csrow2`` | rank0 | rank0 |
- +------------+-----------+-----------+
- | ``csrow3`` | rank1 | rank1 |
- +------------+-----------+-----------+
- In the above example, there are 4 physical slots on the motherboard
- for memory DIMMs:
- +---------+---------+
- | DIMM_A0 | DIMM_B0 |
- +---------+---------+
- | DIMM_A1 | DIMM_B1 |
- +---------+---------+
- Labels for these slots are usually silk-screened on the motherboard.
- Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are
- channel 1. Notice that there are two csrows possible on a physical DIMM.
- These csrows are allocated their csrow assignment based on the slot into
- which the memory DIMM is placed. Thus, when 1 DIMM is placed in each
- Channel, the csrows cross both DIMMs.
- Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
- In the example above 2 dual ranked DIMMs are similarly placed. Thus,
- both csrow0 and csrow1 are populated. On the other hand, when 2 single
- ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will
- have just one csrow (csrow0) and csrow1 will be empty. The pattern
- repeats itself for csrow2 and csrow3. Also note that some memory
- controllers don't have any logic to identify the memory module, see
- ``rankX`` directories below.
- The representation of the above is reflected in the directory
- tree in EDAC's sysfs interface. Starting in directory
- ``/sys/devices/system/edac/mc``, each memory controller will be
- represented by its own ``mcX`` directory, where ``X`` is the
- index of the MC::
- ..../edac/mc/
- |
- |->mc0
- |->mc1
- |->mc2
- ....
- Under each ``mcX`` directory each ``csrowX`` is again represented by a
- ``csrowX``, where ``X`` is the csrow index::
- .../mc/mc0/
- |
- |->csrow0
- |->csrow2
- |->csrow3
- ....
- Notice that there is no csrow1, which indicates that csrow0 is composed
- of a single ranked DIMMs. This should also apply in both Channels, in
- order to have dual-channel mode be operational. Since both csrow2 and
- csrow3 are populated, this indicates a dual ranked set of DIMMs for
- channels 0 and 1.
- Within each of the ``mcX`` and ``csrowX`` directories are several EDAC
- control and attribute files.
- ``mcX`` directories
- -------------------
- In ``mcX`` directories are EDAC control and attribute files for
- this ``X`` instance of the memory controllers.
- For a description of the sysfs API, please see:
- Documentation/ABI/testing/sysfs-devices-edac
- ``dimmX`` or ``rankX`` directories
- ----------------------------------
- The recommended way to use the EDAC subsystem is to look at the information
- provided by the ``dimmX`` or ``rankX`` directories [#f5]_.
- A typical EDAC system has the following structure under
- ``/sys/devices/system/edac/``\ [#f6]_::
- /sys/devices/system/edac/
- ├── mc
- │ ├── mc0
- │ │ ├── ce_count
- │ │ ├── ce_noinfo_count
- │ │ ├── dimm0
- │ │ │ ├── dimm_ce_count
- │ │ │ ├── dimm_dev_type
- │ │ │ ├── dimm_edac_mode
- │ │ │ ├── dimm_label
- │ │ │ ├── dimm_location
- │ │ │ ├── dimm_mem_type
- │ │ │ ├── dimm_ue_count
- │ │ │ ├── size
- │ │ │ └── uevent
- │ │ ├── max_location
- │ │ ├── mc_name
- │ │ ├── reset_counters
- │ │ ├── seconds_since_reset
- │ │ ├── size_mb
- │ │ ├── ue_count
- │ │ ├── ue_noinfo_count
- │ │ └── uevent
- │ ├── mc1
- │ │ ├── ce_count
- │ │ ├── ce_noinfo_count
- │ │ ├── dimm0
- │ │ │ ├── dimm_ce_count
- │ │ │ ├── dimm_dev_type
- │ │ │ ├── dimm_edac_mode
- │ │ │ ├── dimm_label
- │ │ │ ├── dimm_location
- │ │ │ ├── dimm_mem_type
- │ │ │ ├── dimm_ue_count
- │ │ │ ├── size
- │ │ │ └── uevent
- │ │ ├── max_location
- │ │ ├── mc_name
- │ │ ├── reset_counters
- │ │ ├── seconds_since_reset
- │ │ ├── size_mb
- │ │ ├── ue_count
- │ │ ├── ue_noinfo_count
- │ │ └── uevent
- │ └── uevent
- └── uevent
- In the ``dimmX`` directories are EDAC control and attribute files for
- this ``X`` memory module:
- - ``size`` - Total memory managed by this csrow attribute file
- This attribute file displays, in count of megabytes, the memory
- that this csrow contains.
- - ``dimm_ue_count`` - Uncorrectable Errors count attribute file
- This attribute file displays the total count of uncorrectable
- errors that have occurred on this DIMM. If panic_on_ue is set
- this counter will not have a chance to increment, since EDAC
- will panic the system.
- - ``dimm_ce_count`` - Correctable Errors count attribute file
- This attribute file displays the total count of correctable
- errors that have occurred on this DIMM. This count is very
- important to examine. CEs provide early indications that a
- DIMM is beginning to fail. This count field should be
- monitored for non-zero values and report such information
- to the system administrator.
- - ``dimm_dev_type`` - Device type attribute file
- This attribute file will display what type of DRAM device is
- being utilized on this DIMM.
- Examples:
- - x1
- - x2
- - x4
- - x8
- - ``dimm_edac_mode`` - EDAC Mode of operation attribute file
- This attribute file will display what type of Error detection
- and correction is being utilized.
- - ``dimm_label`` - memory module label control file
- This control file allows this DIMM to have a label assigned
- to it. With this label in the module, when errors occur
- the output can provide the DIMM label in the system log.
- This becomes vital for panic events to isolate the
- cause of the UE event.
- DIMM Labels must be assigned after booting, with information
- that correctly identifies the physical slot with its
- silk screen label. This information is currently very
- motherboard specific and determination of this information
- must occur in userland at this time.
- - ``dimm_location`` - location of the memory module
- The location can have up to 3 levels, and describe how the
- memory controller identifies the location of a memory module.
- Depending on the type of memory and memory controller, it
- can be:
- - *csrow* and *channel* - used when the memory controller
- doesn't identify a single DIMM - e. g. in ``rankX`` dir;
- - *branch*, *channel*, *slot* - typically used on FB-DIMM memory
- controllers;
- - *channel*, *slot* - used on Nehalem and newer Intel drivers.
- - ``dimm_mem_type`` - Memory Type attribute file
- This attribute file will display what type of memory is currently
- on this csrow. Normally, either buffered or unbuffered memory.
- Examples:
- - Registered-DDR
- - Unbuffered-DDR
- .. [#f5] On some systems, the memory controller doesn't have any logic
- to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories.
- On modern Intel memory controllers, the memory controller identifies the
- memory modules directly. On such systems, the directory is called ``dimmX``.
- .. [#f6] There are also some ``power`` directories and ``subsystem``
- symlinks inside the sysfs mapping that are automatically created by
- the sysfs subsystem. Currently, they serve no purpose.
- ``csrowX`` directories
- ----------------------
- When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the ``csrowX``
- directories. As this API doesn't work properly for Rambus, FB-DIMMs and
- modern Intel Memory Controllers, this is being deprecated in favor of
- ``dimmX`` directories.
- In the ``csrowX`` directories are EDAC control and attribute files for
- this ``X`` instance of csrow:
- - ``ue_count`` - Total Uncorrectable Errors count attribute file
- This attribute file displays the total count of uncorrectable
- errors that have occurred on this csrow. If panic_on_ue is set
- this counter will not have a chance to increment, since EDAC
- will panic the system.
- - ``ce_count`` - Total Correctable Errors count attribute file
- This attribute file displays the total count of correctable
- errors that have occurred on this csrow. This count is very
- important to examine. CEs provide early indications that a
- DIMM is beginning to fail. This count field should be
- monitored for non-zero values and report such information
- to the system administrator.
- - ``size_mb`` - Total memory managed by this csrow attribute file
- This attribute file displays, in count of megabytes, the memory
- that this csrow contains.
- - ``mem_type`` - Memory Type attribute file
- This attribute file will display what type of memory is currently
- on this csrow. Normally, either buffered or unbuffered memory.
- Examples:
- - Registered-DDR
- - Unbuffered-DDR
- - ``edac_mode`` - EDAC Mode of operation attribute file
- This attribute file will display what type of Error detection
- and correction is being utilized.
- - ``dev_type`` - Device type attribute file
- This attribute file will display what type of DRAM device is
- being utilized on this DIMM.
- Examples:
- - x1
- - x2
- - x4
- - x8
- - ``ch0_ce_count`` - Channel 0 CE Count attribute file
- This attribute file will display the count of CEs on this
- DIMM located in channel 0.
- - ``ch0_ue_count`` - Channel 0 UE Count attribute file
- This attribute file will display the count of UEs on this
- DIMM located in channel 0.
- - ``ch0_dimm_label`` - Channel 0 DIMM Label control file
- This control file allows this DIMM to have a label assigned
- to it. With this label in the module, when errors occur
- the output can provide the DIMM label in the system log.
- This becomes vital for panic events to isolate the
- cause of the UE event.
- DIMM Labels must be assigned after booting, with information
- that correctly identifies the physical slot with its
- silk screen label. This information is currently very
- motherboard specific and determination of this information
- must occur in userland at this time.
- - ``ch1_ce_count`` - Channel 1 CE Count attribute file
- This attribute file will display the count of CEs on this
- DIMM located in channel 1.
- - ``ch1_ue_count`` - Channel 1 UE Count attribute file
- This attribute file will display the count of UEs on this
- DIMM located in channel 0.
- - ``ch1_dimm_label`` - Channel 1 DIMM Label control file
- This control file allows this DIMM to have a label assigned
- to it. With this label in the module, when errors occur
- the output can provide the DIMM label in the system log.
- This becomes vital for panic events to isolate the
- cause of the UE event.
- DIMM Labels must be assigned after booting, with information
- that correctly identifies the physical slot with its
- silk screen label. This information is currently very
- motherboard specific and determination of this information
- must occur in userland at this time.
- System Logging
- --------------
- If logging for UEs and CEs is enabled, then system logs will contain
- information indicating that errors have been detected::
- EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac
- EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac
- The structure of the message is:
- +---------------------------------------+-------------+
- | Content | Example |
- +=======================================+=============+
- | The memory controller | MC0 |
- +---------------------------------------+-------------+
- | Error type | CE |
- +---------------------------------------+-------------+
- | Memory page | 0x283 |
- +---------------------------------------+-------------+
- | Offset in the page | 0xce0 |
- +---------------------------------------+-------------+
- | The byte granularity | grain 8 |
- | or resolution of the error | |
- +---------------------------------------+-------------+
- | The error syndrome | 0xb741 |
- +---------------------------------------+-------------+
- | Memory row | row 0 |
- +---------------------------------------+-------------+
- | Memory channel | channel 1 |
- +---------------------------------------+-------------+
- | DIMM label, if set prior | DIMM B1 |
- +---------------------------------------+-------------+
- | And then an optional, driver-specific | |
- | message that may have additional | |
- | information. | |
- +---------------------------------------+-------------+
- Both UEs and CEs with no info will lack all but memory controller, error
- type, a notice of "no info" and then an optional, driver-specific error
- message.
- PCI Bus Parity Detection
- ------------------------
- On Header Type 00 devices, the primary status is looked at for any
- parity error regardless of whether parity is enabled on the device or
- not. (The spec indicates parity is generated in some cases). On Header
- Type 01 bridges, the secondary status register is also looked at to see
- if parity occurred on the bus on the other side of the bridge.
- Sysfs configuration
- -------------------
- Under ``/sys/devices/system/edac/pci`` are control and attribute files as
- follows:
- - ``check_pci_parity`` - Enable/Disable PCI Parity checking control file
- This control file enables or disables the PCI Bus Parity scanning
- operation. Writing a 1 to this file enables the scanning. Writing
- a 0 to this file disables the scanning.
- Enable::
- echo "1" >/sys/devices/system/edac/pci/check_pci_parity
- Disable::
- echo "0" >/sys/devices/system/edac/pci/check_pci_parity
- - ``pci_parity_count`` - Parity Count
- This attribute file will display the number of parity errors that
- have been detected.
- Module parameters
- -----------------
- - ``edac_mc_panic_on_ue`` - Panic on UE control file
- An uncorrectable error will cause a machine panic. This is usually
- desirable. It is a bad idea to continue when an uncorrectable error
- occurs - it is indeterminate what was uncorrected and the operating
- system context might be so mangled that continuing will lead to further
- corruption. If the kernel has MCE configured, then EDAC will never
- notice the UE.
- LOAD TIME::
- module/kernel parameter: edac_mc_panic_on_ue=[0|1]
- RUN TIME::
- echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
- - ``edac_mc_log_ue`` - Log UE control file
- Generate kernel messages describing uncorrectable errors. These errors
- are reported through the system message log system. UE statistics
- will be accumulated even when UE logging is disabled.
- LOAD TIME::
- module/kernel parameter: edac_mc_log_ue=[0|1]
- RUN TIME::
- echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
- - ``edac_mc_log_ce`` - Log CE control file
- Generate kernel messages describing correctable errors. These
- errors are reported through the system message log system.
- CE statistics will be accumulated even when CE logging is disabled.
- LOAD TIME::
- module/kernel parameter: edac_mc_log_ce=[0|1]
- RUN TIME::
- echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
- - ``edac_mc_poll_msec`` - Polling period control file
- The time period, in milliseconds, for polling for error information.
- Too small a value wastes resources. Too large a value might delay
- necessary handling of errors and might loose valuable information for
- locating the error. 1000 milliseconds (once each second) is the current
- default. Systems which require all the bandwidth they can get, may
- increase this.
- LOAD TIME::
- module/kernel parameter: edac_mc_poll_msec=[0|1]
- RUN TIME::
- echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
- - ``panic_on_pci_parity`` - Panic on PCI PARITY Error
- This control file enables or disables panicking when a parity
- error has been detected.
- module/kernel parameter::
- edac_panic_on_pci_pe=[0|1]
- Enable::
- echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
- Disable::
- echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
- EDAC device type
- ----------------
- In the header file, edac_pci.h, there is a series of edac_device structures
- and APIs for the EDAC_DEVICE.
- User space access to an edac_device is through the sysfs interface.
- At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices
- will appear.
- There is a three level tree beneath the above ``edac`` directory. For example,
- the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net
- website) installs itself as::
- /sys/devices/system/edac/test-instance
- in this directory are various controls, a symlink and one or more ``instance``
- directories.
- The standard default controls are:
- ============== =======================================================
- log_ce boolean to log CE events
- log_ue boolean to log UE events
- panic_on_ue boolean to ``panic`` the system if an UE is encountered
- (default off, can be set true via startup script)
- poll_msec time period between POLL cycles for events
- ============== =======================================================
- The test_device_edac device adds at least one of its own custom control:
- ============== ==================================================
- test_bits which in the current test driver does nothing but
- show how it is installed. A ported driver can
- add one or more such controls and/or attributes
- for specific uses.
- One out-of-tree driver uses controls here to allow
- for ERROR INJECTION operations to hardware
- injection registers
- ============== ==================================================
- The symlink points to the 'struct dev' that is registered for this edac_device.
- Instances
- ---------
- One or more instance directories are present. For the ``test_device_edac``
- case:
- +----------------+
- | test-instance0 |
- +----------------+
- In this directory there are two default counter attributes, which are totals of
- counter in deeper subdirectories.
- ============== ====================================
- ce_count total of CE events of subdirectories
- ue_count total of UE events of subdirectories
- ============== ====================================
- Blocks
- ------
- At the lowest directory level is the ``block`` directory. There can be 0, 1
- or more blocks specified in each instance:
- +-------------+
- | test-block0 |
- +-------------+
- In this directory the default attributes are:
- ============== ================================================
- ce_count which is counter of CE events for this ``block``
- of hardware being monitored
- ue_count which is counter of UE events for this ``block``
- of hardware being monitored
- ============== ================================================
- The ``test_device_edac`` device adds 4 attributes and 1 control:
- ================== ====================================================
- test-block-bits-0 for every POLL cycle this counter
- is incremented
- test-block-bits-1 every 10 cycles, this counter is bumped once,
- and test-block-bits-0 is set to 0
- test-block-bits-2 every 100 cycles, this counter is bumped once,
- and test-block-bits-1 is set to 0
- test-block-bits-3 every 1000 cycles, this counter is bumped once,
- and test-block-bits-2 is set to 0
- ================== ====================================================
- ================== ====================================================
- reset-counters writing ANY thing to this control will
- reset all the above counters.
- ================== ====================================================
- Use of the ``test_device_edac`` driver should enable any others to create their own
- unique drivers for their hardware systems.
- The ``test_device_edac`` sample driver is located at the
- http://bluesmoke.sourceforge.net project site for EDAC.
- Usage of EDAC APIs on Nehalem and newer Intel CPUs
- --------------------------------------------------
- On older Intel architectures, the memory controller was part of the North
- Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and
- newer Intel architectures integrated an enhanced version of the memory
- controller (MC) inside the CPUs.
- This chapter will cover the differences of the enhanced memory controllers
- found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and
- ``sbx_edac`` drivers.
- .. note::
- The Xeon E7 processor families use a separate chip for the memory
- controller, called Intel Scalable Memory Buffer. This section doesn't
- apply for such families.
- 1) There is one Memory Controller per Quick Patch Interconnect
- (QPI). At the driver, the term "socket" means one QPI. This is
- associated with a physical CPU socket.
- Each MC have 3 physical read channels, 3 physical write channels and
- 3 logic channels. The driver currently sees it as just 3 channels.
- Each channel can have up to 3 DIMMs.
- The minimum known unity is DIMMs. There are no information about csrows.
- As EDAC API maps the minimum unity is csrows, the driver sequentially
- maps channel/DIMM into different csrows.
- For example, supposing the following layout::
- Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
- dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
- dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
- Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
- dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
- Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
- dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
- The driver will map it as::
- csrow0: channel 0, dimm0
- csrow1: channel 0, dimm1
- csrow2: channel 1, dimm0
- csrow3: channel 2, dimm0
- exports one DIMM per csrow.
- Each QPI is exported as a different memory controller.
- 2) The MC has the ability to inject errors to test drivers. The drivers
- implement this functionality via some error injection nodes:
- For injecting a memory error, there are some sysfs nodes, under
- ``/sys/devices/system/edac/mc/mc?/``:
- - ``inject_addrmatch/*``:
- Controls the error injection mask register. It is possible to specify
- several characteristics of the address to match an error code::
- dimm = the affected dimm. Numbers are relative to a channel;
- rank = the memory rank;
- channel = the channel that will generate an error;
- bank = the affected bank;
- page = the page address;
- column (or col) = the address column.
- each of the above values can be set to "any" to match any valid value.
- At driver init, all values are set to any.
- For example, to generate an error at rank 1 of dimm 2, for any channel,
- any bank, any page, any column::
- echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
- echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
- To return to the default behaviour of matching any, you can do::
- echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
- echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
- - ``inject_eccmask``:
- specifies what bits will have troubles,
- - ``inject_section``:
- specifies what ECC cache section will get the error::
- 3 for both
- 2 for the highest
- 1 for the lowest
- - ``inject_type``:
- specifies the type of error, being a combination of the following bits::
- bit 0 - repeat
- bit 1 - ecc
- bit 2 - parity
- - ``inject_enable``:
- starts the error generation when something different than 0 is written.
- All inject vars can be read. root permission is needed for write.
- Datasheet states that the error will only be generated after a write on an
- address that matches inject_addrmatch. It seems, however, that reading will
- also produce an error.
- For example, the following code will generate an error for any write access
- at socket 0, on any DIMM/address on channel 2::
- echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
- echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
- echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
- echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
- echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
- dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
- For socket 1, it is needed to replace "mc0" by "mc1" at the above
- commands.
- The generated error message will look like::
- EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
- 3) Corrected Error memory register counters
- Those newer MCs have some registers to count memory errors. The driver
- uses those registers to report Corrected Errors on devices with Registered
- DIMMs.
- However, those counters don't work with Unregistered DIMM. As the chipset
- offers some counters that also work with UDIMMs (but with a worse level of
- granularity than the default ones), the driver exposes those registers for
- UDIMM memories.
- They can be read by looking at the contents of ``all_channel_counts/``::
- $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
- /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
- 0
- /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
- 0
- /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
- 0
- What happens here is that errors on different csrows, but at the same
- dimm number will increment the same counter.
- So, in this memory mapping::
- csrow0: channel 0, dimm0
- csrow1: channel 0, dimm1
- csrow2: channel 1, dimm0
- csrow3: channel 2, dimm0
- The hardware will increment udimm0 for an error at the first dimm at either
- csrow0, csrow2 or csrow3;
- The hardware will increment udimm1 for an error at the second dimm at either
- csrow0, csrow2 or csrow3;
- The hardware will increment udimm2 for an error at the third dimm at either
- csrow0, csrow2 or csrow3;
- 4) Standard error counters
- The standard error counters are generated when an mcelog error is received
- by the driver. Since, with UDIMM, this is counted by software, it is
- possible that some errors could be lost. With RDIMM's, they display the
- contents of the registers
- Reference documents used on ``amd64_edac``
- ------------------------------------------
- ``amd64_edac`` module is based on the following documents
- (available from http://support.amd.com/en-us/search/tech-docs):
- 1. :Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD
- Opteron Processors
- :AMD publication #: 26094
- :Revision: 3.26
- :Link: http://support.amd.com/TechDocs/26094.PDF
- 2. :Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh
- Processors
- :AMD publication #: 32559
- :Revision: 3.00
- :Issue Date: May 2006
- :Link: http://support.amd.com/TechDocs/32559.pdf
- 3. :Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h
- Processors
- :AMD publication #: 31116
- :Revision: 3.00
- :Issue Date: September 07, 2007
- :Link: http://support.amd.com/TechDocs/31116.pdf
- 4. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
- Models 30h-3Fh Processors
- :AMD publication #: 49125
- :Revision: 3.06
- :Issue Date: 2/12/2015 (latest release)
- :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf
- 5. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
- Models 60h-6Fh Processors
- :AMD publication #: 50742
- :Revision: 3.01
- :Issue Date: 7/23/2015 (latest release)
- :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf
- 6. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h
- Models 00h-0Fh Processors
- :AMD publication #: 48751
- :Revision: 3.03
- :Issue Date: 2/23/2015 (latest release)
- :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf
- Credits
- =======
- * Written by Doug Thompson <dougthompson@xmission.com>
- - 7 Dec 2005
- - 17 Jul 2007 Updated
- * |copy| Mauro Carvalho Chehab
- - 05 Aug 2009 Nehalem interface
- - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section
- * EDAC authors/maintainers:
- - Doug Thompson, Dave Jiang, Dave Peterson et al,
- - Mauro Carvalho Chehab
- - Borislav Petkov
- - original author: Thayne Harbaugh
|