edac.txt 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589
  1. EDAC - Error Detection And Correction
  2. Written by Doug Thompson <norsk5@xmission.com>
  3. 7 Dec 2005
  4. EDAC was written by:
  5. Thayne Harbaugh,
  6. modified by Dave Peterson, Doug Thompson, et al,
  7. from the bluesmoke.sourceforge.net project.
  8. ============================================================================
  9. EDAC PURPOSE
  10. The 'edac' kernel module goal is to detect and report errors that occur
  11. within the computer system. In the initial release, memory Correctable Errors
  12. (CE) and Uncorrectable Errors (UE) are the primary errors being harvested.
  13. Detecting CE events, then harvesting those events and reporting them,
  14. CAN be a predictor of future UE events. With CE events, the system can
  15. continue to operate, but with less safety. Preventive maintenance and
  16. proactive part replacement of memory DIMMs exhibiting CEs can reduce
  17. the likelihood of the dreaded UE events and system 'panics'.
  18. In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
  19. in order to determine if errors are occurring on data transfers.
  20. The presence of PCI Parity errors must be examined with a grain of salt.
  21. There are several add-in adapters that do NOT follow the PCI specification
  22. with regards to Parity generation and reporting. The specification says
  23. the vendor should tie the parity status bits to 0 if they do not intend
  24. to generate parity. Some vendors do not do this, and thus the parity bit
  25. can "float" giving false positives.
  26. [There are patches in the kernel queue which will allow for storage of
  27. quirks of PCI devices reporting false parity positives. The 2.6.18
  28. kernel should have those patches included. When that becomes available,
  29. then EDAC will be patched to utilize that information to "skip" such
  30. devices.]
  31. EDAC will have future error detectors that will be integrated with
  32. EDAC or added to it, in the following list:
  33. MCE Machine Check Exception
  34. MCA Machine Check Architecture
  35. NMI NMI notification of ECC errors
  36. MSRs Machine Specific Register error cases
  37. and other mechanisms.
  38. These errors are usually bus errors, ECC errors, thermal throttling
  39. and the like.
  40. ============================================================================
  41. EDAC VERSIONING
  42. EDAC is composed of a "core" module (edac_mc.ko) and several Memory
  43. Controller (MC) driver modules. On a given system, the CORE
  44. is loaded and one MC driver will be loaded. Both the CORE and
  45. the MC driver have individual versions that reflect current release
  46. level of their respective modules. Thus, to "report" on what version
  47. a system is running, one must report both the CORE's and the
  48. MC driver's versions.
  49. LOADING
  50. If 'edac' was statically linked with the kernel then no loading is
  51. necessary. If 'edac' was built as modules then simply modprobe the
  52. 'edac' pieces that you need. You should be able to modprobe
  53. hardware-specific modules and have the dependencies load the necessary core
  54. modules.
  55. Example:
  56. $> modprobe amd76x_edac
  57. loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
  58. core module.
  59. ============================================================================
  60. EDAC sysfs INTERFACE
  61. EDAC presents a 'sysfs' interface for control, reporting and attribute
  62. reporting purposes.
  63. EDAC lives in the /sys/devices/system/edac directory. Within this directory
  64. there currently reside 2 'edac' components:
  65. mc memory controller(s) system
  66. pci PCI control and status system
  67. ============================================================================
  68. Memory Controller (mc) Model
  69. First a background on the memory controller's model abstracted in EDAC.
  70. Each 'mc' device controls a set of DIMM memory modules. These modules are
  71. laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can
  72. be multiple csrows and multiple channels.
  73. Memory controllers allow for several csrows, with 8 csrows being a typical value.
  74. Yet, the actual number of csrows depends on the electrical "loading"
  75. of a given motherboard, memory controller and DIMM characteristics.
  76. Dual channels allows for 128 bit data transfers to the CPU from memory.
  77. Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs
  78. (FB-DIMMs). The following example will assume 2 channels:
  79. Channel 0 Channel 1
  80. ===================================
  81. csrow0 | DIMM_A0 | DIMM_B0 |
  82. csrow1 | DIMM_A0 | DIMM_B0 |
  83. ===================================
  84. ===================================
  85. csrow2 | DIMM_A1 | DIMM_B1 |
  86. csrow3 | DIMM_A1 | DIMM_B1 |
  87. ===================================
  88. In the above example table there are 4 physical slots on the motherboard
  89. for memory DIMMs:
  90. DIMM_A0
  91. DIMM_B0
  92. DIMM_A1
  93. DIMM_B1
  94. Labels for these slots are usually silk screened on the motherboard. Slots
  95. labeled 'A' are channel 0 in this example. Slots labeled 'B'
  96. are channel 1. Notice that there are two csrows possible on a
  97. physical DIMM. These csrows are allocated their csrow assignment
  98. based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
  99. is placed in each Channel, the csrows cross both DIMMs.
  100. Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
  101. Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
  102. will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
  103. when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
  104. csrow1 will be populated. The pattern repeats itself for csrow2 and
  105. csrow3.
  106. The representation of the above is reflected in the directory tree
  107. in EDAC's sysfs interface. Starting in directory
  108. /sys/devices/system/edac/mc each memory controller will be represented
  109. by its own 'mcX' directory, where 'X" is the index of the MC.
  110. ..../edac/mc/
  111. |
  112. |->mc0
  113. |->mc1
  114. |->mc2
  115. ....
  116. Under each 'mcX' directory each 'csrowX' is again represented by a
  117. 'csrowX', where 'X" is the csrow index:
  118. .../mc/mc0/
  119. |
  120. |->csrow0
  121. |->csrow2
  122. |->csrow3
  123. ....
  124. Notice that there is no csrow1, which indicates that csrow0 is
  125. composed of a single ranked DIMMs. This should also apply in both
  126. Channels, in order to have dual-channel mode be operational. Since
  127. both csrow2 and csrow3 are populated, this indicates a dual ranked
  128. set of DIMMs for channels 0 and 1.
  129. Within each of the 'mc','mcX' and 'csrowX' directories are several
  130. EDAC control and attribute files.
  131. ============================================================================
  132. DIRECTORY 'mc'
  133. In directory 'mc' are EDAC system overall control and attribute files:
  134. Panic on UE control file:
  135. 'panic_on_ue'
  136. An uncorrectable error will cause a machine panic. This is usually
  137. desirable. It is a bad idea to continue when an uncorrectable error
  138. occurs - it is indeterminate what was uncorrected and the operating
  139. system context might be so mangled that continuing will lead to further
  140. corruption. If the kernel has MCE configured, then EDAC will never
  141. notice the UE.
  142. LOAD TIME: module/kernel parameter: panic_on_ue=[0|1]
  143. RUN TIME: echo "1" >/sys/devices/system/edac/mc/panic_on_ue
  144. Log UE control file:
  145. 'log_ue'
  146. Generate kernel messages describing uncorrectable errors. These errors
  147. are reported through the system message log system. UE statistics
  148. will be accumulated even when UE logging is disabled.
  149. LOAD TIME: module/kernel parameter: log_ue=[0|1]
  150. RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ue
  151. Log CE control file:
  152. 'log_ce'
  153. Generate kernel messages describing correctable errors. These
  154. errors are reported through the system message log system.
  155. CE statistics will be accumulated even when CE logging is disabled.
  156. LOAD TIME: module/kernel parameter: log_ce=[0|1]
  157. RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ce
  158. Polling period control file:
  159. 'poll_msec'
  160. The time period, in milliseconds, for polling for error information.
  161. Too small a value wastes resources. Too large a value might delay
  162. necessary handling of errors and might loose valuable information for
  163. locating the error. 1000 milliseconds (once each second) is the current
  164. default. Systems which require all the bandwidth they can get, may
  165. increase this.
  166. LOAD TIME: module/kernel parameter: poll_msec=[0|1]
  167. RUN TIME: echo "1000" >/sys/devices/system/edac/mc/poll_msec
  168. ============================================================================
  169. 'mcX' DIRECTORIES
  170. In 'mcX' directories are EDAC control and attribute files for
  171. this 'X" instance of the memory controllers:
  172. Counter reset control file:
  173. 'reset_counters'
  174. This write-only control file will zero all the statistical counters
  175. for UE and CE errors. Zeroing the counters will also reset the timer
  176. indicating how long since the last counter zero. This is useful
  177. for computing errors/time. Since the counters are always reset at
  178. driver initialization time, no module/kernel parameter is available.
  179. RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset
  180. This resets the counters on memory controller 0
  181. Seconds since last counter reset control file:
  182. 'seconds_since_reset'
  183. This attribute file displays how many seconds have elapsed since the
  184. last counter reset. This can be used with the error counters to
  185. measure error rates.
  186. Memory Controller name attribute file:
  187. 'mc_name'
  188. This attribute file displays the type of memory controller
  189. that is being utilized.
  190. Total memory managed by this memory controller attribute file:
  191. 'size_mb'
  192. This attribute file displays, in count of megabytes, of memory
  193. that this instance of memory controller manages.
  194. Total Uncorrectable Errors count attribute file:
  195. 'ue_count'
  196. This attribute file displays the total count of uncorrectable
  197. errors that have occurred on this memory controller. If panic_on_ue
  198. is set this counter will not have a chance to increment,
  199. since EDAC will panic the system.
  200. Total UE count that had no information attribute fileY:
  201. 'ue_noinfo_count'
  202. This attribute file displays the number of UEs that
  203. have occurred have occurred with no informations as to which DIMM
  204. slot is having errors.
  205. Total Correctable Errors count attribute file:
  206. 'ce_count'
  207. This attribute file displays the total count of correctable
  208. errors that have occurred on this memory controller. This
  209. count is very important to examine. CEs provide early
  210. indications that a DIMM is beginning to fail. This count
  211. field should be monitored for non-zero values and report
  212. such information to the system administrator.
  213. Total Correctable Errors count attribute file:
  214. 'ce_noinfo_count'
  215. This attribute file displays the number of CEs that
  216. have occurred wherewith no informations as to which DIMM slot
  217. is having errors. Memory is handicapped, but operational,
  218. yet no information is available to indicate which slot
  219. the failing memory is in. This count field should be also
  220. be monitored for non-zero values.
  221. Device Symlink:
  222. 'device'
  223. Symlink to the memory controller device.
  224. Sdram memory scrubbing rate:
  225. 'sdram_scrub_rate'
  226. Read/Write attribute file that controls memory scrubbing. The scrubbing
  227. rate is set by writing a minimum bandwith in bytes/sec to the attribute
  228. file. The rate will be translated to an internal value that gives at
  229. least the specified rate.
  230. Reading the file will return the actual scrubbing rate employed.
  231. If configuration fails or memory scrubbing is not implemented, the value
  232. of the attribute file will be -1.
  233. ============================================================================
  234. 'csrowX' DIRECTORIES
  235. In the 'csrowX' directories are EDAC control and attribute files for
  236. this 'X" instance of csrow:
  237. Total Uncorrectable Errors count attribute file:
  238. 'ue_count'
  239. This attribute file displays the total count of uncorrectable
  240. errors that have occurred on this csrow. If panic_on_ue is set
  241. this counter will not have a chance to increment, since EDAC
  242. will panic the system.
  243. Total Correctable Errors count attribute file:
  244. 'ce_count'
  245. This attribute file displays the total count of correctable
  246. errors that have occurred on this csrow. This
  247. count is very important to examine. CEs provide early
  248. indications that a DIMM is beginning to fail. This count
  249. field should be monitored for non-zero values and report
  250. such information to the system administrator.
  251. Total memory managed by this csrow attribute file:
  252. 'size_mb'
  253. This attribute file displays, in count of megabytes, of memory
  254. that this csrow contains.
  255. Memory Type attribute file:
  256. 'mem_type'
  257. This attribute file will display what type of memory is currently
  258. on this csrow. Normally, either buffered or unbuffered memory.
  259. Examples:
  260. Registered-DDR
  261. Unbuffered-DDR
  262. EDAC Mode of operation attribute file:
  263. 'edac_mode'
  264. This attribute file will display what type of Error detection
  265. and correction is being utilized.
  266. Device type attribute file:
  267. 'dev_type'
  268. This attribute file will display what type of DRAM device is
  269. being utilized on this DIMM.
  270. Examples:
  271. x1
  272. x2
  273. x4
  274. x8
  275. Channel 0 CE Count attribute file:
  276. 'ch0_ce_count'
  277. This attribute file will display the count of CEs on this
  278. DIMM located in channel 0.
  279. Channel 0 UE Count attribute file:
  280. 'ch0_ue_count'
  281. This attribute file will display the count of UEs on this
  282. DIMM located in channel 0.
  283. Channel 0 DIMM Label control file:
  284. 'ch0_dimm_label'
  285. This control file allows this DIMM to have a label assigned
  286. to it. With this label in the module, when errors occur
  287. the output can provide the DIMM label in the system log.
  288. This becomes vital for panic events to isolate the
  289. cause of the UE event.
  290. DIMM Labels must be assigned after booting, with information
  291. that correctly identifies the physical slot with its
  292. silk screen label. This information is currently very
  293. motherboard specific and determination of this information
  294. must occur in userland at this time.
  295. Channel 1 CE Count attribute file:
  296. 'ch1_ce_count'
  297. This attribute file will display the count of CEs on this
  298. DIMM located in channel 1.
  299. Channel 1 UE Count attribute file:
  300. 'ch1_ue_count'
  301. This attribute file will display the count of UEs on this
  302. DIMM located in channel 0.
  303. Channel 1 DIMM Label control file:
  304. 'ch1_dimm_label'
  305. This control file allows this DIMM to have a label assigned
  306. to it. With this label in the module, when errors occur
  307. the output can provide the DIMM label in the system log.
  308. This becomes vital for panic events to isolate the
  309. cause of the UE event.
  310. DIMM Labels must be assigned after booting, with information
  311. that correctly identifies the physical slot with its
  312. silk screen label. This information is currently very
  313. motherboard specific and determination of this information
  314. must occur in userland at this time.
  315. ============================================================================
  316. SYSTEM LOGGING
  317. If logging for UEs and CEs are enabled then system logs will have
  318. error notices indicating errors that have been detected:
  319. EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
  320. channel 1 "DIMM_B1": amd76x_edac
  321. EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
  322. channel 1 "DIMM_B1": amd76x_edac
  323. The structure of the message is:
  324. the memory controller (MC0)
  325. Error type (CE)
  326. memory page (0x283)
  327. offset in the page (0xce0)
  328. the byte granularity (grain 8)
  329. or resolution of the error
  330. the error syndrome (0xb741)
  331. memory row (row 0)
  332. memory channel (channel 1)
  333. DIMM label, if set prior (DIMM B1
  334. and then an optional, driver-specific message that may
  335. have additional information.
  336. Both UEs and CEs with no info will lack all but memory controller,
  337. error type, a notice of "no info" and then an optional,
  338. driver-specific error message.
  339. ============================================================================
  340. PCI Bus Parity Detection
  341. On Header Type 00 devices the primary status is looked at
  342. for any parity error regardless of whether Parity is enabled on the
  343. device. (The spec indicates parity is generated in some cases).
  344. On Header Type 01 bridges, the secondary status register is also
  345. looked at to see if parity occurred on the bus on the other side of
  346. the bridge.
  347. SYSFS CONFIGURATION
  348. Under /sys/devices/system/edac/pci are control and attribute files as follows:
  349. Enable/Disable PCI Parity checking control file:
  350. 'check_pci_parity'
  351. This control file enables or disables the PCI Bus Parity scanning
  352. operation. Writing a 1 to this file enables the scanning. Writing
  353. a 0 to this file disables the scanning.
  354. Enable:
  355. echo "1" >/sys/devices/system/edac/pci/check_pci_parity
  356. Disable:
  357. echo "0" >/sys/devices/system/edac/pci/check_pci_parity
  358. Panic on PCI PARITY Error:
  359. 'panic_on_pci_parity'
  360. This control files enables or disables panicking when a parity
  361. error has been detected.
  362. module/kernel parameter: panic_on_pci_parity=[0|1]
  363. Enable:
  364. echo "1" >/sys/devices/system/edac/pci/panic_on_pci_parity
  365. Disable:
  366. echo "0" >/sys/devices/system/edac/pci/panic_on_pci_parity
  367. Parity Count:
  368. 'pci_parity_count'
  369. This attribute file will display the number of parity errors that
  370. have been detected.
  371. =======================================================================