Most modern servers use Broadcom network cards. The module is called tg3 and is known to have issues.
You can find in my blog several posts about weird problems with tg3. This post is about another one I’ve encountered only recently.
The server: Dell PowerEdge (PE) 830. Linux version: RedHat Advanced Server 4 (RHEL4) Update 4. Kernel version 2.6.9-42.ELsmp. Version of the tg3 module: 3.52-rh.
Problem: When the card is being activated (brought up), it looses link for about 5 seconds. Later the link reappears and everything is fine. Implication: You cannot run DHCP client (dhclient) with this card, as it looses the link, and the DHCP client will fail with "link not ready" error message.
Fixed IP address works well, although the cards still looses the link when its being activated.