For anyone who is interested in my Redhat/VMware/iSCSI woes, here’s an update.  Jason and I were looking at requirements for VMware ESX and it mentioned that RHEL v4’s iSCSI initiator software was not on the approved list.  RHEL v3 was on the list.  I leveled the box and installed v3.  An amazing thing happened.  It just worked!  I don’t know what this means exactly.  On the surface, it points to me that there is a kernel memory issue with RHEL v4, iSCSI, and the VMware disk creating process.  I know that this process is CPU intensive but why would v4 handle it better than v3? 

In order to get into the virtualization game at my place of employment, we decided to implement the free version of Vmware, VMWare Server.  We decided that it would run on Redhat Enterprise Linux connected to an iSCSI SAN using Dell PowerEdge 1850s for the server hardware.  We’ve been running this environment for between 6-8 months without much of a problem.  Recently, we purchased some PowerEdge 1950’s.  One of them was to be the virtual machine host.

After installing the OS, Redhat iSCSI intiator software, and vmware bits on the 1950 I started running into problems.  The operation of creating a virtual machine whose virtual disk resides on the SAN causes the oom-killer to start taking processes out.  The only thing different from our previous production environment and this one is the hardware. 

To compound matters, if we install VMware ESX server on the 1950, it seems to function like a champ.  I’m really at a loss right now.  If anyone has any suggestions, that would be cool.  I ain’t no computer scientist.  I’m running 32bit Redhat, not 64bit.  I had thought that this was an issue with NUMA (Non-Uniform Memory Access?) but with the numa=off boot parameter passed, I still get this behavior.  Ideas?