Category Archives: ESXi

How to use vBenchmark tool to measure the infrastructure performance ?

Traditionally, datacenter operators have been focused on benchmarking application performance. In recent years, there has been an emphasis on the measurement of infrastructure performance (e.g. consolidation ratio) in addition to application performance. However, there have been no tools available to measure the operational statistics calculated from actual tasks and events within a datacenter such as how long it takes to provision a new service, how often SLAs (Service Level Agreements) are being met, or even how efficiently datacenter resources are being utilized. VMware vBenchmark is the industry’s first tool designed to measure such metrics for virtualized datacenters. With this tool, you can
Measure key operational statistics/metrics of your VMware infrastructure across three categories:
Efficiency: for example, physical RAM savings by using virtualization.
Agility: for example, average time to provision a VM.
Quality of Service: for example, downtime avoided by using availability features.

Compare your operational metrics against your peer group by choosing to contribute your stats to the community repository. The stats that you submit are anonymized and encrypted for secure transmission.
Track your operational metrics over time. For instance, you can query metrics for the last 1 month, 3 month or 6 month time period and compare how your operational metrics have changed with time.

Kindly refer the article present at below link to understand its usage.

https://drive.google.com/file/d/0B7F4NEbnRvYidUJ3YXY3MzQxOEE/edit?usp=sharing

We can download the free vBenchmark tool .ova template from below location

http://labs.vmware.com/flings/vbenchmark.

Advertisements

How to use esxtop in VMware efficiently ?

In the article present at below link, I have shown the various usages of esxtop.

https://drive.google.com/file/d/0B7F4NEbnRvYibW94RnI0M082czA/edit?usp=sharing

Apart from the mentioned one. You can use the esxtop in batch mode to capture the performance statistics of particular time.
You can take that output out from ESX on your windows machine.

After that you can analyze that output with tools like Perfmon and esxplot.

Tip : By default esxtop will capture the output at 5 second sampling period while vcenter shows the real time statistics at 20 second.

Digging more on Veeam Backup technology

I have created articles on Veeam showing its stunning power in simplicity. Kindly click on the below links to access articles.

Creating new schedule backup Job:

https://drive.google.com/file/d/0B7F4NEbnRvYiRWFMOFZ0VFMzQUk/edit?usp=sharing

Creating new replication Job:

https://drive.google.com/file/d/0B7F4NEbnRvYiQnVBZXJKX0pUWWc/edit?usp=sharing

Starting backup job and recover the VM:

https://drive.google.com/file/d/0B7F4NEbnRvYiNUxmbXA2Ym9YYkU/edit?usp=sharing

How Veeam take backup in VMware ?

Recently we have started using Veeam to take the backup of VM in our environment.I am thrilled to see the amazing capabilities of veeam. Especially like vPower NFS with which we can boot the VM from backup directly without restoring it.

I have installed the Evaluation version Veeam on Windows server 2008 R2.

I am showing you how to take the back up of VM using Veeam in document which is present at given link and below procedure is showing what happens at VM level while taking the backup.

https://drive.google.com/file/d/0B7F4NEbnRvYiRHZZWWp0R3VvbWc/edit?usp=sharing

I have one very small sized VM of which I am taking the backup. I have shown the status of VM files at various level during the backup.

1) Currently my VM is powered off

/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392/Linux2-11.1 # ls -lsh
total 41992
39936 -rw——-    1 root     root       50.0M Sep  7 17:03 Linux2-11.1-flat.vmdk
1024 -rw——-    1 root     root        8.5K Sep  7 17:04 Linux2-11.1.nvram
0 -rw——-    1 root     root         519 Sep  7 16:44 Linux2-11.1.vmdk
0 -rw-r–r–    1 root     root           0 Sep  7 16:44 Linux2-11.1.vmsd
8 -rwxr-xr-x    1 root     root        2.7K Sep  7 17:04 Linux2-11.1.vmx
0 -rw-r–r–    1 root     root         266 Sep  7 16:44 Linux2-11.1.vmxf
1024 -rw-r–r–    1 root     root      121.1K Sep  7 17:04 vmware.log

2) After powering on the VM

/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392/Linux2-11.1 # ls -lsh
total 171024
16384 -rw——-    1 root     root       16.0M Sep 13 16:07 Linux2-11.1-dc9f5b1f.vswp
39936 -rw——-    1 root     root       50.0M Sep  7 17:03 Linux2-11.1-flat.vmdk
1024 -rw——-    1 root     root        8.5K Sep  7 17:04 Linux2-11.1.nvram
0 -rw——-    1 root     root         519 Sep  7 16:44 Linux2-11.1.vmdk
0 -rw-r–r–    1 root     root           0 Sep  7 16:44 Linux2-11.1.vmsd
8 -rwxr-xr-x    1 root     root        2.7K Sep 13 16:07 Linux2-11.1.vmx
0 -rw——-    1 root     root           0 Sep 13 16:07 Linux2-11.1.vmx.lck
0 -rw-r–r–    1 root     root         266 Sep  7 16:44 Linux2-11.1.vmxf
8 -rwxr-xr-x    1 root     root        2.7K Sep 13 16:07 Linux2-11.1.vmx~
1024 -rw-r–r–    1 root     root      121.1K Sep  7 17:04 vmware-1.log
1024 -rw-r–r–    1 root     root       19.3K Sep 13 16:07 vmware.log
111616 -rw——-    1 root     root      109.0M Sep 13 16:07 vmx-Linux2-11.1-3701431071-1.vswp

3) I have ran the backup from Veeam, it will create the snapshot to take the backup of VM. You will not see any process inside the VM.

/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392/Linux2-11.1 # ls -lsh
total 172072
8 -rw——-    1 root     root        3.6K Sep 13 16:11 Linux2-11.1-000001-ctk.vmdk
8 -rw——-    1 root     root        4.0K Sep 13 16:11 Linux2-11.1-000001-delta.vmdk
0 -rw——-    1 root     root         393 Sep 13 16:11 Linux2-11.1-000001.vmdk
1024 -rw——-    1 root     root       27.7K Sep 13 16:11 Linux2-11.1-Snapshot1.vmsn
8 -rw——-    1 root     root        3.6K Sep 13 16:11 Linux2-11.1-ctk.vmdk
16384 -rw——-    1 root     root       16.0M Sep 13 16:07 Linux2-11.1-dc9f5b1f.vswp
39936 -rw——-    1 root     root       50.0M Sep 13 16:08 Linux2-11.1-flat.vmdk
1024 -rw——-    1 root     root        8.5K Sep 13 16:08 Linux2-11.1.nvram
0 -rw——-    1 root     root         582 Sep 13 16:11 Linux2-11.1.vmdk
0 -rw-r–r–    1 root     root         498 Sep 13 16:11 Linux2-11.1.vmsd
8 -rwxr-xr-x    1 root     root        2.8K Sep 13 16:11 Linux2-11.1.vmx
0 -rw——-    1 root     root           0 Sep 13 16:07 Linux2-11.1.vmx.lck
0 -rw-r–r–    1 root     root         266 Sep  7 16:44 Linux2-11.1.vmxf
8 -rwxr-xr-x    1 root     root        2.8K Sep 13 16:11 Linux2-11.1.vmx~
1024 -rw-r–r–    1 root     root      121.1K Sep  7 17:04 vmware-1.log
1024 -rw-r–r–    1 root     root      106.7K Sep 13 16:11 vmware.log
111616 -rw——-    1 root     root      109.0M Sep 13 16:07 vmx-Linux2-11.1-3701431071-1.vswp

4) As soon as backup is complete snapshot will get delete automatically.

/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392/Linux2-11.1 # ls -lsh
total 171032
8 -rw——-    1 root     root        3.6K Sep 13 16:12 Linux2-11.1-ctk.vmdk
16384 -rw——-    1 root     root       16.0M Sep 13 16:07 Linux2-11.1-dc9f5b1f.vswp
39936 -rw——-    1 root     root       50.0M Sep 13 16:13 Linux2-11.1-flat.vmdk
1024 -rw——-    1 root     root        8.5K Sep 13 16:12 Linux2-11.1.nvram
0 -rw——-    1 root     root         582 Sep 13 16:12 Linux2-11.1.vmdk
0 -rw-r–r–    1 root     root          43 Sep 13 16:12 Linux2-11.1.vmsd
8 -rwxr-xr-x    1 root     root        2.8K Sep 13 16:12 Linux2-11.1.vmx
0 -rw——-    1 root     root           0 Sep 13 16:07 Linux2-11.1.vmx.lck
0 -rw-r–r–    1 root     root         266 Sep  7 16:44 Linux2-11.1.vmxf
8 -rwxr-xr-x    1 root     root        2.8K Sep 13 16:12 Linux2-11.1.vmx~
1024 -rw-r–r–    1 root     root      121.1K Sep  7 17:04 vmware-1.log
1024 -rw-r–r–    1 root     root      137.0K Sep 13 16:13 vmware.log
111616 -rw——-    1 root     root      109.0M Sep 13 16:07 vmx-Linux2-11.1-3701431071-1.vswp

5) Again I have powered off the VM. Now it look same like as in step 1

/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392/Linux2-11.1 # ls -lsh
total 43024
8 -rw——-    1 root     root        3.6K Sep 13 16:16 Linux2-11.1-ctk.vmdk
39936 -rw——-    1 root     root       50.0M Sep 13 16:14 Linux2-11.1-flat.vmdk
1024 -rw——-    1 root     root        8.5K Sep 13 16:16 Linux2-11.1.nvram
0 -rw——-    1 root     root         582 Sep 13 16:12 Linux2-11.1.vmdk
0 -rw-r–r–    1 root     root          43 Sep 13 16:12 Linux2-11.1.vmsd
8 -rwxr-xr-x    1 root     root        2.8K Sep 13 16:16 Linux2-11.1.vmx
0 -rw-r–r–    1 root     root         266 Sep  7 16:44 Linux2-11.1.vmxf
1024 -rw-r–r–    1 root     root      121.1K Sep  7 17:04 vmware-1.log
1024 -rw-r–r–    1 root     root      157.5K Sep 13 16:16 vmware.log

After that I deleted the VM from Disk to check my backup. Once I deleted the VM(Linux2-11.1) it was not showing in repository.

(unknown) # cd /vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392/
/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392 # ls -lrt
total 32
d-w-r-xr-T    1 root     root           420 Sep  8 08:45 vmkdump
drwxr-xr-x    1 root     root          1680 Sep  8 11:06 Red-Linux-2
drwxr-xr-x    1 root     root          3080 Sep 13 15:42 Red-Linux-1
drwxr-xr-x    1 root     root          3360 Sep 13 16:04 Linux-11.1

I initiated the restore operation. After the restoration is complete my VM is back.

/vmfs/volumes/54044ebd-7cada24f-a4f5-000c29903392 # ls -lrt
total 40
d-w-r-xr-T    1 root     root           420 Sep  8 08:45 vmkdump
drwxr-xr-x    1 root     root          1680 Sep  8 11:06 Red-Linux-2
drwxr-xr-x    1 root     root          3080 Sep 13 15:42 Red-Linux-1
drwxr-xr-x    1 root     root          3360 Sep 13 16:04 Linux-11.1
drwxr-xr-x    1 root     root          1120 Sep 13 16:23 Linux2-11.1

I was using Veeam free edition for testing purpose hence not able to show you the power of vPower V7 of Veeam which includes amazing capabilites.

Understanding ESXi root disk partitions.

After the installation of ESXi if we have look at partition of the disk which is used for installation of ESXi we will find 5 partitions on it. In this article I am covering some brief idea about size calculation and significance of each partition.

Take the putty session to ESXi.

We will find couple of partitions on that disk. What are these partitions ?

~ # partedUtil getptbl /dev/disks/mpx.vmhba1\:C0\:T0\:L0
gpt
261 255 63 4194304
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

Now see the output of df -h you will see that along with VMFS datastores there are vfat file systems mounted in it.

~ # df -h
Filesystem   Size   Used Available Use% Mounted on
VMFS-5      29.8G  10.7G     19.0G  36% /vmfs/volumes/ISCSIDataStore-2
VMFS-5      19.8G   4.2G     15.5G  21% /vmfs/volumes/ISCSIDatastore-1
VMFS-5      19.8G   1.0G     18.7G   5% /vmfs/volumes/LocalDataStore-1
vfat       249.7M 157.8M     91.9M  63% /vmfs/volumes/e848f979-a81f576c-bb22-bc4cffa55d54
vfat       249.7M 173.3M     76.5M  69% /vmfs/volumes/4c129273-503f4af7-8e7a-d63e665c5ea0
vfat       285.8M 192.6M     93.2M  67% /vmfs/volumes/540c641b-75203ae6-8a13-000c29bcb49e

Why I am seeing 3 partitions mounted instead of 5 as per partedUtil output ?

These vfat file systems are partition of the ESXi. Now if you compare the output with partedUtil command you will see only three partitions are mounted in output of df. These partitions are of type “linuxNative”. Now if you see paredUtil command has 3 “linuxNative” partitions which are showing mounted in df.

How to calculate the size of partition from output of paredUtil ?

If we take look at output of partedUtil you will see 5 partitions in it. Now let me take 1 partition as an example to calculate the size.

1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

Taking second and third column
Starting value = 64 and Ending Value = 8191
Formula for calculation in MB is = ( Ending Value – Starting Value )/2*1024
In this case it will become = ( 8191 – 64) / 2 * 1024 which comes approx 4 MB

What are the significance of these 5 partitions ?

We will start in same order in which they are displaying in output of partedUtil.

Partition 1 : First partition contains the boot loader which runs when the host boots up. Size is 4MB.

Partition 5 : It contains the core hypervisor code. Which is packaged into several files. Size is 249 MB(approx)

Partition 6 : It contains the copy of partition 5 Once an update is applied we are having one safe copy in another partition. Size is 249 MB(approx)

Partition 7 : It is used to hold diagnostic core dumps.Size is 110 MB.

Partition 8 : It contains the vmware tools ISO. This contains the auxiliary files. Size is 286 MB (approx)

 

VMware Interview Questions — Part 2

In my earlier post (https://ervikrant06.wordpress.com/2014/08/30/vmware-interview-questions-part-1/) I shared some interview questions but I feel need to dig more on vSphere HA topic hence came up with this post.

Q : What is FDM ?

A : FDM stands for Fault domain manager. Its having two communication with hostd and vCenter server. vCenter gets the VM state information from FDM.

Q : What is master node in cluster is responsible for ?

A : Basically master role plays all the role like the primary node in vSphere 4 cluster.
a) Restarting failed Virtual machines
b) Exchanging state with vCenter.
c) Monitor the state of slaves.

Q : What if the master node fails ?

A : If the master node fails reelection will occur. From the Hosts that is participating in the election host with highest number of datastores will be elected as master. If two or more hosts have same number of datastores the one with the highest Managed Object ID will be chosen.

Q : How many host failure types can be detected in HA ?

A : a) Host fails
b) Host becomes network isolated
c) Host loses network connectivity with master host.

Q : What is datastore heartbeating ?

A : Prior to vSphere 4.0 HA will kick off the restart of VM even if only the management network of host was isloated and VMs are still running. This was unnecessary issue. To remediate that new level of resilence is added to make distinction between failed host and an isolated or partitioned host.

Q : How many datastore by default is chosen for datastore heartbeating ?

A : Two datastores by default are chosen for datastore heart-beating. You can change this value with help of das.heartbeatDsPerHost parameter.

Q : How datastore heartbeating works ?

A : HA leverages the existing VMFS file system locking mechanism. The locking mechanism uses a region called “heartbeat region. Per host one file is created, HA will check if whether heartbeat region is updated or not.

Q : What is the difference between isolated and partition state ?

A : Isolated is not receiving any election traffic but partition is receiving it.
Isolated can’t ping the isolation address but partition can.

Similar point is :
In both cases host is not able to receive heartbeat from master

Q : What happens whey you disconnect a host from a cluster ?

A : All VMs registered to that host are unprotected by vSphere HA. Means they are no longer in control of HA.

Q : What happens when we select the disabled VM restart priority for VM in HA ?

A : If disabled setting is selected, vSphere HA is disabled for VM, which means it is not restarted on other ESXi hosts if host fails on which currently its residing. If there is an OS level issue and vSphere HA VM/Application Monitoring is enabled then reboot of OS will trigger but VM will remain on same host.

Q : What is difference between shutdown and power off the VM ?

A : Shutdown operation preserve the state of VM. Its better than powering off the VM in some cases.
Shutdown operation require vmware tools to be installed for VM.
Shutdown operation require more time than poweroff.

Q : When the split brain condition will occur in vSphere HA  cluster ?

A :  If Host isolation response is disabled and one host got isolated and it has running VMs. As the VMs are not having access to disk locks. VMs got failed over to another host even though the original instances of the VM remain running on the isolated host. Now failed over instances of VM start working by regaining disk locks.

Q : What changes occur if we enable vSAN in HA cluster ?

A : By default HA cluster uses managment network of HA traffic. If vSAN is configured it will use vSAN dedicated network for HA traffic. This change doesn’t occur automatically, HA reconfiguration on hosts require to bring it into effect.

How to capture VMware network traffic using pktcap-uw

Have you guys use the command pktcap-uw to monitor the traffic in your VMware environment? If not then start using it. Its very useful tool.

Let me show you the glimpse of its power. I have taken the putty session to ESX on which my VM is hosted for which I  want to capture the traffic.

Case 1 :

I have created two VMs manamgement vnics(Virtual NIC) of both VM is created on vmnic0 (Physical NIC).

Now I am monitoring the traffic between the VMs when I am pinging from one VM to another VM.

~ # pktcap-uw –uplink vmnic0 -c 20 vmnic_capture.pcap
The name of the uplink is vmnic0
To capture 20 packets
No server port specifed, select 59721 as the port
Output the packet info to console.

Case 2 :

Now if you want to monitor the vnic (Virtual interface) traffic that we can also do. But to do that we need to identify the vnic Port-ID becuase that is the unique
parameter associated with VNIC.

There are two methods to find Port-ID

1) One is two step.

~ # esxcli network vm list
World ID Name Num Ports Networks
——– ———- ——— —————
52666 Linux-11.1 1 dvportgroup-302

~ # esxcli network vm port list -w 52666
Port ID: 50331655
vSwitch: VDSwitch-1
Portgroup: dvportgroup-302
DVPort ID: 38
MAC Address: 00:50:56:99:86:67
IP Address: 0.0.0.0
Team Uplink: vmnic0
Uplink Port ID: 50331651
Active Filters:

2) Second method is using esxtop

Type the “esxtop” on command prompt
Press “n”
Locate your VM for which you want to trace the package.

PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PKTRX/s MbRX/s %DRPTX %DRPRX
50331655 52666:Linux-11.1.eth vmnic0 DvsPortset-0 0.99 0.00 0.99 0.00 0.00 0.00

Coming back to pktcap-uw. I used the VM port-ID collected to capture the network traffic.

~ # pktcap-uw –switchport 50331655 -c 25 -o vnic_capture1.pcap
The switch port id is 0x03000007
To capture 25 packets
The output file is vnic_capture1.pcap
No server port specifed, select 60899 as the port
Local CID 2
Listen on port 60899
Accept…Vsock connection from port 1026 cid 2
Dump: 22, broken : 0, drop: 0, file err: 0Receive thread exiting…
Dump: 25, broken : 0, drop: 0, file err: 0Dump thread exiting…
Destroying session 2

Dumped 25 packet to file vnic_capture1.pcap, dropped 0 packets.
Done.

Case 3 :

Last with feeling we can monitor the vmknic traffic as well. In this case I am narrowing down by specifying the destination IP.

~ # pktcap-uw –vmk vmk0 –dstip 192.168.13.1 -c 25 -o vnic_capture2.pcap
The name of the vmk is vmk0
The session filter destination IP address is 192.168.13.1
To capture 25 packets
The output file is vnic_capture2.pcap
No server port specifed, select 61256 as the port
Local CID 2
Listen on port 61256
Accept…Vsock connection from port 1027 cid 2
Dump: 24, broken : 0, drop: 0, file err: 0Receive thread exiting…
Dump: 25, broken : 0, drop: 0, file err: 0Dump thread exiting…
Destroying session 3

Dumped 25 packet to file vnic_capture2.pcap, dropped 0 packets.
Done.

To know about more option issue below command you will plathora of options to narrow down your network traffic capturing.
~ # pktcap-uw -h

You can copy the files generated after command completion to your local desktop using winscp or any of your favorite tool. Then you can analyse them.