The past one week have been busy with VMware vSphere 4 and Cisco Nexus 5000 POC and the result is really disappointed. Personally I couldn’t believed and I think I may missed out something. If anyone have any idea or suggestion, please feel free to comment here or post reply at slow Performance with 10 Gb CNA card on vSphere 4 thread.

Benchmark Test Configuration
2 x VMware vSphere 4 hosts
2 x Dell PE2950 Hardware
– Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
– 16GB RAM
– Qlogic QLE8042 10GbE Mercury Converged Network Adapter(CNA) – connected to PCIe 8x slot

VMware vSphere vCenter
– running as virtual machine
– 8vCPU
– 6GB RAM
– Microsoft Windows Server 2003 Standard edition

2 x Microsoft Windows Server 2003 Standard edition virtual machines
– 4vCPU
– 4GB RAM

Software:
Netperf

MTU 9000 setup
VMware vSphere host
esxcfg-vswitch -m 9000 vSwitch2

Windows Server 2003 standard edition virtual machine
Device Manager -> Network Adapters -> VMXNET3 Ethernet Adapter -> Properties -> Advanced -> Jumbo Packet -> Jumbo 9000 and Speed / Duplex -> 10Gbps Full Duplex

Cisco Nexus 5000K – Enable MTU 9000 and Jumbo Frame

VMware vSphere hosts
name: esx05
name: esx06

Windows Server 2003 standard edition virtual machine
name: test1 (running on esx06)
name: test2 (running on esx05)

Virtual Switch
Port Group name – test262 (Connected with single vNICs detected as Intel 82598EB 10 Gigabit AF Dual Port Network) with VLAN ID 1.

Note: vSphere auto detect as ISP8432 4Gb FCoE PCI Express HBA & Intel 82598EB 10 Gigabit

And the result:

click to enlarge.

Related posts:

1. How to remove Cisco Nexus 1000V plugin
2. Performance tuning on Virtual Infrastructure with MD3000i
3. vSphere client on windows 7

As my previous experienced, if you having a disk space full on /var, the host server will hung and all the VMs on the host will stop functioning. I had my test machine setup as default configuration, which the /var/log only allocated for 2GB in place. Prior to move the system to production, I decide to extend the /var/log from 2GB to 10GB as our current standard. Here come to the idea that I try to extend, unmount and remount as I usually did on the Linux.

I found that the partition table for vSphere 4 is totally different as ESX 3.5 did. At 1st was confusing as I only have 1 logical drive with RAID 1 configuration on my vSphere 4 machine, but when I did the cat /proc/partitions, it showed as 2 physical devices as sda and sdb.

I tested to delete the partition table from the fdisk utility as I did on usual Linux, but is not success. The system will roll back to the existing partition table before I tried to make the changes automatically. You may see the IO error happen when you try to write the partition table. At the same time, I found that there is a vmdk with similar 8GB of flat file and vmdk was created on the existing datastore which reflect the local storage on the vSphere host. It is viewable through CLI and vCenter client both. After some research, looks like the virtual disk is referring to vSphere core module and important mount point such as / & /var/log. Whenever I tried to remove the virtual disk from the ESX host, I get denied with error message of the virtual disk in use.

This 2 screen shots had helped me to understand that the new system partition method in vSphere 4, which is totally different as it did in ESX 3.5. At the mean time, I also found that to format the rest of the unclaim partiton on the local HDD for VMFS 3 file system as Datastore is compulsory during the setup too. Therefore, sizing on the partitions require to be planned prior to final deployment.

Related posts:

1. ESX Host Profiles with vSphere
2. How To Upload Files Into Datastore
3. Storage VMotion through GUI in vSphere

We see the growth in the market to be more aggressive for consolidation in the data center for both physical and virtual server from time to time. There are plenty of solutions in place allow blade to support virtualization today such as virtual connect from HP, pass through module, infiniband integration Xsigo, Cisco UCS and etc. This had significant resolved the I/O interfaces require per blade to host the virtualization host server. CPU and memory per blade and significant increase with the latest release from all the major server vendors, the CPU, memory and disk I/O are no longer the concerns for virutalization.

But I do want to point out some of weakness on the solutions proposed by the deployment team or vendors that suggest to start with 1 blade chassis to reduce the CAPEX require. To achieve the fully redundancy on the virtualization, we may require NIC teaming, redundant FC connection, redundant ISCSI HBA and etc. Some of the guys may had forgotten about the chassis issues on the blade today. If I would like to run the virtualization hosts or clustering servers on blade, I will not choose to go with only single blade chassis in this case. I will not believe there is 100 % guarantee of no failure on the chassis of the blade. I will definitely suggest to split the blade across to 2 different chassis as minimum to minimize the risk. But of course, we may still have a single point of failure on the SAN which require huge amount of capex investment to keep the production storage to be redundant. Imagine if you run 5 ESX hosts on single blade chassis today, you will easily achieve 15:1 per blade for consolidation, and if you are UNLUCKY enough and your chassis just burn or malfunction for any failure, you will have 75 VMs go down at 1 time and your HA and DRS is not worked at all. You will definitely screw and whack by your business or boss in this case. If we do not have enough capex to start with 2 blade chassis for HA in the virtualization, I will suggest to stick with 1U or 2U rack mount servers to minimize the risk of the business. I am impressed with the blade technology today, but just to start with 1 single chassis for HA and redundancy purpose, will not be my call.

Related posts:

1. Why ESX are not suitable to run on Blade
2. IBM x3950 M2 Powerful ESX Machine
3. Best Server Model for Virtualization – R900, R905

You may had experienced the snapshot problem when the snapshot delta virtual disk had become huge which is more than 20GB and is not able to be truncated by removing all the snapshot. Usually the problem occur when the snapshot is no longer available from snapshot manager, but you can see the delta files available from the command prompt or physical folder that contain the Virtual machine images. I experienced the 2nd time now which I am not able to remove the snapshot on a particular virtual machine from snapshot manager and command prompt. The production virtual machine was powered off due to disk full on the specify datastore. This virtual machine contain a previous snapshot which taken 2 weeks before and yet to be truncated. The delta disk had grown up to >20 GB and fully fill up the available space within the data store.

In order to recover the virtual machine, I had to remove all the snapshots on the virtual machine, but the process hang up after 3 to 4 times I tried. You may able to perform this with command and if you are lucky enough, you should able to remove the snapshot by using command. Alternatively, you can also clone a new virtual disk image contain the delta file changes to a whole new virtual disk by using vmkfstools command. This will require quite some times for you to recover the situation mostly the business will not able to wait and it is no guarantee that the recovery will fully work as we expected.

For my case, the quickest way to recover this is to clone the entire virtual machine to a new datastore that have the disk space you require and power it up immediate after the clone. We double checked the entire virtual machines mount point and services, everything require and last file transaction is available without any issue and back to production. Now I will have my sweet time to slowly investigate the failure to remove snapshots on the virtual machines. Well, is not necessary the method I am using here is the best, but I would like to share my personal experience in the critical production time, you may wanna consider the best option to recover the production virtual machine as soon as possible.

Related posts:

1. Manual commit snapshots delta file to vmdk flat file
2. Make a Clone from a Snapshot
3. How to recover a VM hung in the ESX Farm

A virtual network adapter is a program (instead of a physical network adapter) that allows a computer to connect to a network. A virtual network adapter can also be used to connect all the computers on a local area network (LAN) to a larger network such as the Internet or a collection of LANs.

Here, I would like to share how to change virtual machine virtual network adapter manually. In fact, you may know you can add a new virtual adapter and choose the adapter type from drop down menu.

Here you go:

1. Login to VMware ESX server
2. Type vmware-cmd -l and look for virtual machine (vmx = vm config file) which require change virtual network adapter
3. You may check VMware KB Article: 1001805 for available network adapters details.

# VMXNET — The VMXNET virtual network adapter has no physical counterpart. VMXNET is optimized for performance in a virtual machine. Because operating system vendors do not provide built-in drivers for this card, you must install VMware Tools to have a driver for the VMXNET network adapter available.

# Flexible — The Flexible network adapter identifies itself as a Vlance adapter when a virtual machine boots, but initializes itself and functions as either a Vlance or a VMXNET adapter, depending on which driver initializes it. With VMware Tools installed, the VMXNET driver changes the Vlance adapter to the higher performance VMXNET adapter.

# E1000 — An emulated version of the Intel 82545EM Gigabit Ethernet NIC, with drivers available in most newer guest operating systems, including Windows XP and later and Linux versions 2.4.19 and later.

# VMXNET 2 (Enhanced) — The VMXNET 2 adapter is based on the VMXNET adapter but provides some high-performance features commonly used on modern networks, such as jumbo frames and hardware offloads. This virtual network adapter is available only for some guest operating systems on ESX/ESXi 3.5 and later.

4. Edit Virtual Machine vmx file and choose virtual network adapter as show below:

Flexible
Nothing have to add into vmx file.

VMXNET 3
ethernetX.virtualDev = “vmxnet3″

Note: ethernetX = number of ethernet card.

5. Save the VMX file and power on your virtual machine.
6. Power on your virtual machine using virtual center or type # vmware-cmd your_virtualmachine.vmx start

Enjoy!

Related posts:

1. Tips: VM Network Problem
2. How to Enable Virtual Machine Monitoring

The Cisco Nexus 1000V switch is a pure software implementation of a Cisco Nexus switch. It resides on a server and integrates with the hypervisor to deliver VN-Link virtual machine-aware network services. The Cisco Nexus 1000V switch takes advantage of the VMware vSphere vNetwork Distributed Switch framework to offer tightly integrated network services as part of both a server virtualization strategy and a broader data center virtualization strategy. In addition, the switch provides operations and management consistency with existing Cisco Nexus and Cisco Catalyst switches.

Here I would like to share how to remove Cisco Nexus 1000V Plugin

1. To view the extension key on the vCenter Server, open an Internet Browser, and enter the URL path http:///mob

2. Authentication dialog box open, Enter the username and password and click OK

3. The Managed Object Browser (MOB) open the Service Instance page, in the value column of the Properties table, click Content

4. In the Value column of the Properties table, Click extensionManager OR you can open browser with URL http://localhost/mob/?moid=ExtensionManager

5. In the Methods table, click UnregisterExtension

6. In the Value of the Parameters table, paste “Cisco_Nexus_1000V_1850055053″ and click Invoke Method.

Note: The actual value of “Cisco_Nexus_1000v_xxxxx” will wary. It should match the extension key from the Cisco_nexus_1000v_extension.xml file.

Enjoy!

Related posts:

1. Cisco Nexus 5000 POC
2. Installing the Performance Overview Plugin issue
3. Disk Performance on ESX and VM

Microsoft had included the virtual PC with backward compatible of Windows XP to support the users who may had difficult to run their application on Windows 7. This is free and we can see more user will sure include this as part of the Windows 7 installation move forward. Bear in mind, if you have VMware Workstation running on the same machine, you may see the conflict happen if you have your Virtual PC application running and you try to start up a virtual machine from VMware Workstation. There will be a message prompt to inform you that there are existing hypervisor utilize the virtualization capabilities on your machine. The only way to get your VM start up is to end the session of your Virtual PC application. You should able to start your virtual machine up. Not very sure whether there are any workaround beside this, but at this moment, it consider a conflict and we need to choose either 1 to be run at a time.

Related posts:

1. VMware Server 2 Vs VMware Workstation
2. vSphere client on windows 7
3. VMware Workstation 6.5 RC1 Available for Download

With my last 3 years of experience, I had seen numbers of companies require significant expansion in the data center to support the business growth, just because of the additional 200 to 2000 servers or storage shelves they need to put in to the data center. With the technology today, it does help us to reduce the space we need to slot in the servers, and storage thanks to the success of virtualization and higher capacity disk drives in the market. To run a full solution of data center with less number of server racks for SME and enterprise, will no longer be a dream due to some facts below,

1. Higher consolidation ratio in virtualization with the latest CPU, memory and servers performance
2. Higher capacity disk drive which significant reduce the number of storage expansion require
3. Virtual I/O which allow the higher utilization of the available bandwidth in data center
4. UCS to provide consolidation and higher capacity of bandwidth to support network and storage utilization in the data center

I may had not included all the facts here, but these should be good enough to point out the changing happening to the current market. I had been impressed with the IBM 3950 M2 previously but now, the Unisys alliances with VMware is another choice which provide similar features for the large environment which contains large number of machines running. The unisys Enterprise Server ES7000 could be the ideal machine in this case. 16 physical cpu sockets which contain 6 core each from Intel Xeon 7400 series of processor, and maximum of 1TB of memory with 8GB memory module in a single server. If we do a calculation, you should able to have 96 cores of CPUs in a single server. This will allow more virtual machine to be consolidated into it.

You can view here for more information about the hardware details.

So, with all the technology available, we should able to have our data center running with less number of server racks for servers, storage and networks. This had significant reduce the space require to run a data center, and allow the users to expand the life cycle of the data center in a longer term. If I have to choose to invest 1 Million on DC expansion VS 1 Million of Virtualization, servers and storage, I will definitely choose the 2nd option which provide additional capacity and reduce the space require by a data center. In a longer term, the real estate cost, power & cooling always cost us a big sum of dollars to keep the data center running.

In the near future, there will be 8 cores and 12 cores CPU in the future market. When it come to the time for us to refresh the current ESX servers which run on quad cores or 6 cores base server, the new servers capacity may be double the current servers that we using, and for sure it will allow more machine to be consolidated.

Related posts:

1. Efficient Data Center – Virtualize & Consolidation
2. Virtualization – Network, Storage, Data Center
3. Consolidate Server Network Virtualization and Storage

You can easily found “how to reset the root password on ESX” articles on the web for an example VMware KB article 1317898, xtravirt, or Petri.

However, unauthorized entry into single user mode is a risk. For security enhancement, you should enable password protect GRUB and enable root password for single user mode on vSphere 4.

How To Enable Root Password for Single User Mode on vSphere 4

• Login to vSphere server as root
• Edit /etc/inittab file
• # vi /etc/inittab
• Insert “~~:S:wait:/sbin/sulogin” and reboot



What is this mean?
man inittab shows
id:runlevels:action:process
id = is a unique sequence of 1-4 characters which identifies an entry in inittab
runlevels = lists the runlevels for which the specified action should be taken
action = describes which action should be taken.
process = specifies the process to be executed.
S = Single user mode
wait = The process will be started once when the specified runlevel is entered and init will wait for its termination.


• You should be able to see password prompt during single user mode boot up

Password protecting GRUB
Note: This may NOT really helpful to anyone of you because you’re require to enter password during boot up.

• Login to vSphere server as root
• Type: # /sbin/grub-md5-crypt
• Copy the encrypted password(e.g.$1$LgZD4/$vgg6d30on18oNlUB6FVew1) and paste into /boot/grub/grub.conf as example show below
• You’re prompt to enter GRUB password as you enter just now before server boot up

Related posts:

1. How to enable ssh in vSphere 4i
2. How to force VM to enter BIOS Setup mode on next boot up
3. New vSphere Client

I am currently doing some test to simulate the real update requirement for my production VMware Farm to be upgraded from ESX 3.5 to vSphere 4. To minimize the impact to our production system, we may want to do this with as minimal impact as possible. Due to the virtual hardware version and tools upgrade require, is pretty tough for us to perform the entire upgrade at 1 time. Therefore, our plan is get the host upgraded and follow by each individual virtual machine to be upgraded follow the suitable timing for different business unit. HA and DRS will need to be disable temp during the ESX upgrade.

I am able to get the ESX 3.5 to be manage by the latest vcenter. A vmotion from ESX 3.5 to vsphere 4 had been successes too, but the latest version of virtual machine which built from vSphere 4 might not compatible to vmotion back to the ESX 3.5 hosts. At the same time, if you have different processors chipset in the environment and require EVC to be turned on, it may be a little challenge to do so. You may want to ensure the EVC to be done with no down time.  You may need to refer to my previous post about how to enable EVC with no down time.

This trick is workable in the case. The new cluster created in vSphere 4 environment, may temporally disable the HA and DRS due to the incompatibility within ESX 3.5 and vSphere 4 for VMotion, which may happen on the latest virtual hardware version which provided by vSphere 4. Anyway, my test will still going on to further analyze the best upgrade method to be used to our environment due to the challenge of different region, time zone and business function we currently serve with our VMware environment.

Related posts:

1. New experience on vCenter 4
2. vSphere client on windows 7
3. Create VMFS with High Availability and Vmotion in local hard drive