HP ProLiant DL980 G7 Server – IML Shows During POST: Memory module failure prevented bus training

SYMPTOM:Memory module failure prevented bus training
SYMPTOM:Memory bus training failed
SYMPTOM:POST Error: 207-Memory initialization error on Memory Board
SYMPTOM:Quad memory module loading order violated
Cause
CAUSE:The DIMM 1A on Memory Board 9 failed.  Details: System reported in the IML log the following error messages after two reboots:  Caution Main Memory 06/10/2013 04:18 06/10/2013 04:18 2 Unknown - Message: Memory module failure prevented bus training (Board 9, Memory Module 1)   Critical Main Memory 06/10/2013 04:18 06/10/2013 04:18 1 Unknown - Message: Memory bus training failed (Board 9)   Caution Main Memory 06/10/2013 04:18 06/10/2013 04:18 2 Unknown - Message: Quad memory module loading order violated (Board 10)

   Critical POST Message 06/10/2013 04:20 06/10/2013 04:20 1 POST Error: 207 - Message: POST Error: 207-Memory initialization error on Memory Board 9 Memory Module 1. The operating system may not have access to all of the memory installed in the system.   Critical POST Message 06/10/2013 04:20 06/10/2013 04:20 1 POST Error: 207 - Message: 

POST Error: 207-Invalid Memory Configuration - Memory Board 10 has non-uniform memory distribution. Please refer to Memory Population Rules in documentation and check memory for possible errors.   

All memory on Memory Boards 9 + 10 are disabled and not available to the OS. These boards are located on the lower CPU tray in (Slot 7 + 8 - most left slots).  The SID (System Insight Display) shows the following state:
control_panel.jpg
control_panel.jpg
  The IML message and the SID are in sync: This is a real issue with DIMM 1A on Memory Board 9.  The type DIMM can be determined from the hpdiag report: Board 9  Card 9 Status Online  DIMM 1A (DDR3) 8192 Mbytes  DIMM 2C Not Installed  DIMM 3B Not Installed  DIMM 4D Not Installed  DIMM 5D Not Installed  DIMM 6B Not Installed  DIMM 7C Not Installed  DIMM 8A (DDR3) 8192 Mbytes   DIMM Description: DIMM 8GB PC3-10600R 512Mx4 RoHS HP 8GB 2Rx4 PC3-10600R-9 Kit
Answer/Solution
FIX:Replace DIMM 1A on Memory Board 9 (Lower Tray Memory Board 7).