Sun Fire V40z: Four Opterons in a 3U

Chipsets (con't)

We have mentioned this in the past, but Opteron severely differs from Xeon due to the HyperTransport links from processor to processor, and that the memory controller is on the processor die. With each processor sharing a 6.4GB/s link to two other processors, and indirectly their memory banks, a four-way Opteron configuration does not get bottlenecked on a memory controller or Northbridge.

Most high end dual and quad Opteron solutions use two AMD 8131 PCI-X tunnels to control IO off the processors, but they go a step further by daisy-chaining a third 8131 tunnel to the secondary tunnel (which is why the V40z can utilize four 64-bit PCI-X 133MHz slots and still have enough IO for the integrated controllers). The HyperTransport specification details that five devices can be within an HT chain, so having two 8131 PCI-X tunnels daisy-chained is clearly within spec. A brief block diagram of our V40z can be found below.

Click to enlarge.

There is enough headroom on primary PCI-X tunnel for two more 100MHz PCI-X channels; and a third and slowest channel runs in tandem with the gigabit Ethernet and SCSI controller. Browsing through some online documentation revealed that the seventh, 66MHz PCI-X card shares bus resources with the LSI and Broadcom controllers, which means inserting a 33MHz PCI expansion card in the bus will reduce the operating bus of the SCSI and gigabit Ethernet controllers. Considering the 800MB/s headroom on that particular bus, it would seem like a poor choice to install a high bandwidth PCI device in the 66MHz interface anyway. Likewise, using a 66MHz and a 100MHz PCI-X device in tandem on the primary 8131 tunnel will result in both buses slowing to 66MHz.

AMD’s 8111 I/O Hub is linked off the primary 8131 PCI-X tunnel, and from there, most of the basic system functions are controlled including the XGI graphics adaptor. Even though XGI hasn’t been particularly popular on the desktop, Trident’s penetration into the server market solidified XGI’s server market share. The majority of the features on the AMD 8111 remain disabled, like AC’97 audio and the integrated 10/100 Ethernet controller.

Two Broadcom BCM703 chips control the external gigabit Ethernet for the server, but there are also two 10/100 out-of-band Ethernet ports that we will go into more depth later. Winbond provides the rest of the basic functionality of the machine not handled by the AMD 8111 I/O hub. LSI’s 53C1020 Ultra320 SCSI adaptor provides the V40z with the onboard SCSI.

Click to enlarge.

If the soul of our V40z is the Opteron 850, then the temporal lobe would be the Motorola MPC855T Service Processor. The Motorola MPC855T is a particularly interesting chip that we saved an entire page for more detail.