Hooray, I made it through the class! Now it is time to study and take the VCP test.

What I learned:

1. If you have a VM with the vnic disconnected and you enable FT on the VM then the only way to enable the vnic is to enable the vnic, power off the VM, and power back on the VM. Just connecting the vnic or just rebooting the VM is not sufficient. This appears to be true for any setting change to the VM.
   
   
2. vCenter Server Heartbeat can work over a WAN even if the primary and secondary vCenter Server instances have different public IP addresses. This is a very cool feature and hopefully can be spun off to allow for a supported way to easily change the IP address of a vCenter Server instance.


3. vDR only supports file level restores from the command line. In addition, vDR only allows backups from the Hosts and Clusters view. This means you cannot backup based on the folders under the VMs and Templates view. One other note is that if a vDR VM were to die, another vDR could be brought up and attached to the backup disk of the previous vDR instance allowing restores. vDR is a great alternative to VCB. VCB will not longer be supported after vSphere 4.1.
   
   
4. By default ESXi partitioning creates a 4GB VFAT scratch partition for system swap. If the scratch partition is not enabled, the host will use an additional 512MB of physical memory on the host. This is different than ESX where the recommendation is to create a swap file that is twice the size of the memory allocated to the VMkernel.

Clarifications I made:

1. Instructors: If you forget the root password on ESX(i) then you need to reinstall ESX(i).
   Comments: While this is one option, it is not the only option. In the case of ESX, you can boot the server into single user mode where you can change the root password. In the case of ESXi, this is not an option. As such, for ESXi the VMware recommended solution is to perform a repair install on ESXi, which will reinstall ESXi, but preserve an existing VMFS datastores. The downside to this approach is that all VMs will need to be re-registered with the ESXi host. Alternatively, a Linux Live CD can be used to mount the ESXi partitions whether it is the embedded or installable version. It is important to note that making the root password blank (e.g. when resetting the root password) will result in all NFS datastores becoming disconnected. This is because NFS requires root authentication to work with ESX(i).
   
   
2. Instructors: VUM requires a completely separate database from vCenter Server.
   Comments: This is not true. vCenter Server and VUM can share the same datastore though VMware best practice is for each to have a separate database. This is explained in the ESX(i) installation guides (e.g. http://www.vmware.com/pdf/vsphere4/r40/vsp_40_esx_vc_installation_guide.pdf and http://www.vmware.com/pdf/vsphere4/r41/vsp_41_esx_vc_installation_guide.pdf):
   
   

   vCenter Update Manager can use the same database as vCenter Server, but VMware recommends that you have separate databases for vCenter Server and vCenter Update Manager

   
   In my opinion, just because you can does not mean you should. My recommendation is to always separate the two. Also, just as an FYI, separate databases does not mean separate SQL instances.

Questions I raised:

1. With vCenter Server Heartbeat, can you have the primary vCenter Server instance with SQL and the secondary vCenter Server instance with SQL Express? They did not know. I believe the answer is no, but even if you could this would not be a best practice. I can understand from a cost perspective, but if you need vCenter Server Heartbeat and can afford it then you can likely afford the another SQL license if required. Also, I believe two Windows licenses are needed, but a single SQL license is sufficient.
   
   
2. VUM allows you to control the schedule of patch deployment as well as force powered-off and suspended VMs to power on for remediation. What happens if a cluster has DPM enabled and an ESX host is in maintenance mode (i.e. will the host be powered on for remediation)? They said VUM will not force a host out of standby mode and as such hosts with DPM enabled are not guaranteed to remediate. With that said, a host may come out of standby mode during a remediation of a cluster because a host being remediated must be in maintenance mode, which may require additional resources in the cluster because of current load or HA requirements.

Lesson of the day: Use vCenter Linked Mode with extreme caution. Make sure you understand the potential security implications as well as the vCenter Server outages that could occur if not careful!

What I learned:

1. With vCenter Linked Mode, the search service queries AD for information about user permissions. As such, you must be logged in to a domain account in order to search all vCenter Server systems. If you log in using a local account, searches return results only for the local vCenter Server system, even if it is joined to other servers in a Linked Mode group. This comes as a surprise because if at least two linked vCenter Server instances have an identical local user account (i.e. same credentials), you can view everything that user has permissions to view in both linked vCenter Server instances. However, if you perform a search, you will only see results for the local vCenter Server instance the you logged in to.
   
   
2. When adding a vCenter Server instance to a Linked Mode group, the user running the installer must be either a domain user with permission on all vCenter Server instances to be linked or provide credentials to a local administrator on the target vCenter Server instance in addition to all vCenter Server instances currently in the linked group. If this is not the case then you get the following error message:
   
   

   The currently logged on user does not have sufficient privileges on the LDAP server <vCenter_Server_name> to be able to configure replication.
   
   By default, the currently logged on user account needs to be a member of the local administrators group on the remote server. If you are using a local administrator account on this server then the same named account with the same password needs to exist on the remote server. This account also needs to be a members of the local administrators group on the remote server.
   
   Alternatively, perform this installation using a domain account and ensure the account is a member of the local administrators group on the remote server. You can use a domain administrator account for this as normally domain administrators are automatically a member of a local administrators group.



3. If you have vCenter Servers in linked mode and you log onto a vCenter Server instance with a local account that is not identical on each vCenter Server instance (i.e. either the the local user does not exist on all vCenter Server instances or the credentials are not consistent on all vCenter Server instances) and you attempt to isolate the vCenter Server instance (i.e. remove it from the linked group) the task will fail and stop the vCenter Server progress on the vCenter Server instance you are currently logged in to. Looking at the logs you will notice:
   
   

   [2010-08-05 08:37:16 SEVERE] Operation "Isolate instance VMwareVCMSDS" failed: : Action: Prepare for Isolate Problem: Unable to reach a peer instance

   
   This means the the credentials used did not work, poor error message in my book. In addition, you will notice above this error message that the vCenter Server process stopped:
   
   

   [2010-08-05 08:37:16 INFO] Service 'VMware VirtualCenter Management Webservices' is shutdown
   [2010-08-05 08:37:16 INFO] Service 'VMware VirtualCenter Server' is shutdown

   
   You can force the isolation, but the force will only apply to the vCenter Server instance you are currently logged in to and will need to be performed on all other vCenter Server instances in the linked group. As such, the best practice would be to use a domain account that is known to exist and have permissions on all vCenter Server instances in the linked group.
   
   
4. If a VMs swap file (i.e. .vswp) is not accessible to the destination host, VMotion must be able to create a swap file accessible to the destination host before migration can begin. As an example, this means if a VM is stored on a shared datastore, but the swap file is stored locally on an ESX host and the VM is migrated to a different ESX host, the swap file must and will be created on a datastore accessible to the destination ESX host depending on the configuration of the ESX host or else the migration will fail.

Clarifications I made:

1. Student Manual: For DPM to work, the VMotion NIC on each host must support WOL.
   Comments: This is important to keep in mind when architecting the network configuration of ESX hosts. In the case of on-board and expansion cards NICs, best practice is to NIC team mixing both. Ensure the expansion card NIC allocated for VMotion traffic supports WOL.
   
   
2. Instructors: DPM + Alarms will result in notifications when hosts are put into standby mode.
   Comments: Given the default alarm definitions this is not true. A clear distinction is given between unavailable and standby ESX hosts. In addition, a default alert exists to notify when a host fails to come out of standby mode.
   
   
3. Instructors: HA works without vCenter Server.
   Comments: While it is true that vCenter Server is only needed to initially configure HA and then HA can run independently, if vCenter Server and the five primary HA nodes go down HA will not work.
   
   
4. Instructors: A host can only be removed from a cluster if in maintenance mode.
   Comments: The only way you should remove a host from a cluster is by putting it in maintenance mode. You could also disconnect and then remove the ESX host from vCenter Server and then re-add the ESX host to vCenter Server, but this can cause issue. For example, this approach will remove performance statistics, may corrupt vDS, and will move VMs out of folders.
   
   
5. Student Manual: Requirement of FT is that VMs must be provisioned with thick virtual disks.
   Instructors: They actually need to be eager zero thick virtual disks
   Comments: Actually this is not true to NAS!

Questions I raised:

1. Does DPM and HA play nice? They said yes, in fact all VMware features play nice and in the following order of priority: FT, HA, DRS, and DPM.
   
   
2. Why does the slot size set default to the maximum reservations for CPU and memory? Since multiple slots can be used in the case of large VMs with no allocation would it not make more sense to just default to a very small slot size? They said the reason was because of overhead associated with doing this.

Another great day of knowledge transfer!

What I learned:

1. For VMware ESX, the service console always runs on the first HEC and is never migrated to another one.
   
   
2. If VMware Tools cannot be installed on a VM then the allocated CPU and memory shares should be set higher than other VMs as pagefiles or swap partitions on the VM may not always be used resulting in reduced performance. VMware Tools adds the vmmemctl also know as the balloon driver to a VM, which is used during memory overcommitment. If the vmmemctl driver is not present and part of the VMs memory needs to be taken away (e.g. if another VM has a reservation) then the memory is stolen and forced onto the vswp file for the VM. If the vmmemctl driver is present, the driver overcommits the memory on the VM allowing the VM's OS to handle the overcommitment by moving less important information in memory to a pagefile or swap file.
   
   
3. Since by default up to 65% of memory can be paged out by the vmmemctl driver, the recommendation would be to set a 35% reservation on very important VMs to minimize, though not completely eliminate, the possibility of swapping.
   
   
4. If you see the error, "The distributed Virtual Switch corresponding to the proxy switches d5 6e 22 50 dd f2 94 7b-a6 1f b2 c2 e6 aa 0f bf on the host does not exist in vCenter or does not contain the host." please click here for a workaround. (FYI not actually taught, but experienced during a lab.)

Clarifications I made:

1. Student Manual: As a best practice, time synchronization with the host should always be enabled.
   Comments: According to VMware KB articles this is not true: http://kb.vmware.com/kb/1318 and http://kb.vmware.com/kb/1006427.
   
   
2. Instructors: vCenter Server permissions are always most restrictive.
   Comments: While this is true if permissions are set on a group and user basis, it is not true if only set on a user basis. In the case of user basis only, permissive override restrictive and vice versa when defined deeper in the vCenter Server hierarchy.
   
   
3. Instructors: Disks on NFS datastores are always thin provisioned.
   Comments: This is because NFS is file and not block based.
   
   
4. Instructors: If a Windows VM disk is thin provisioned and a full format is performed the disk becomes thick provisioned.
   Comments: None other than to keep this in mind. To ensure this does not happen use quick format.

Questions I raised:

1. How do you check for VMs that are not registered, but still exist on datastores? (FYI I did not ask this question). They did not know, I responded by saying this can be scripted, but VMware does not provide a way.
   
   
2. If you inflate a thin provisioned disk do the fragmented files get put back together? They said inflation attempts to do this, but if space is not available then only the inflated disk space is contagious. While fragmentation cannot be determined through VMware, Storage VMotion will remove it and actually requires contagious space to be available on the destination datastore.

Just as an FYI, these posts are not an all-inclusive list of what I learned, clarifications I made, and questions I raised. My goal is to highlight what I think were the most important items in each category.

What I learned:

1. The data plane of a distributed switch is implemented with a hidden vSwitch in the VMkernel on each ESX host. This makes sense as networking continues to function even if vCenter Server goes down.
   
   
2. HA restarts VMs first by priority and second in alphabetical order. This means if you have ten VMs set with a high restart priority they will be restarted before any other VMs and in order based on name. This makes sense, however I never realized the significance of a VM name.

Clarifications I made:

1. Instructors: If you set all three security policies on a portgroup (i.e. promiscuous mode, MAC address changes, and forced transmits) to reject then a cloned Windows VM whether sysprepped or not will not be allowed on the network.
   Comments: This is true ONLY IF the Windows VM IS NOT sysprepped. The reason for this is because the MAC address assigned to the VM does not match the actual MAC address of the cloned system unless the cloned system is sysprepped. I believe for security purposes the policy should default to rejecting all three exceptions.
   
   
2. Instructors: Traffic shaping can only be configured on a per-portgroup basis.
   Comments: This makes sense as I have only seen configuration of traffic shaping listed under portgroups. This means in order to guarantee traffic shaping on a per-VM basis each VM would need to be assigned a dedicated portgroup (multiple VMs could share the same VLAN, but different portgroups with the same VLAN would need to be configured for each VM). In addition, traffic shaping happens on the physical NICs and not on the BUS. This means two VMs on the same ESX host attached to the same portgroup cannot be restricted via VMware traffic shaping.

Questions I raised:

1. Is there any impact (e.g. performance) to increasing the number of ports per vSwitch? They said yes, but could not quantify.
   
   
2. How many vSwitches should you have? They said it should be the same physical as it is logical (i.e. one-to-one mapping). I agree!
   
   
3. How is a static binding on a distributed switch ensured? They said vCenter Server handles it, but could not elaborate how.

This week I am in the VMware vSphere: Fast Track [V4] training class so I can finally get my VCP! I figured I could take the next five days and tell you what I learned, clarifications I made, and questions I raised.

What I learned:

1. In the VMs and Templates view, you can move multiple VMs into a folder simultaneously by selecting the Virtual Machines tab and then using the shift or ctrl keys. I had always tried to do this from the left panel, which only allows one at a time. (This was not actually taught, my partner showed me this.)
   
   
2. VMware does not support a native VLAN when tagging. From what I found online (http://kb.vmware.com/kb/1266), it appears native VLANs are supported, but not recommended.
   
   Update: Native VLANs ARE supported on ESX as I thought. It is important not to add a VLAN ID to a portgroup for a native VLAN as this will not work. More information can be found at: http://kb.vmware.com/kb/1004074 andhttp://kb.vmware.com/kb/1003806.
   
   
3. The last vmnic added to a vSwitch has the highest priority. This means if you add a vmnic to a vSwitch that is not configured properly (i.e. the wrong VLANs assigned to the port) it will negatively impact your vSwitch. In the most severe case, this could result in loss of management on the ESX host. While I had experienced this before, I did not know the last added vmnic had the highest priority.

Clarifications I made:

1. Instructors: You should always use FQDNs when configuring VMware services.
   Comments: It is extremely important that your DNS server is configured properly. In addition, it should be noted that if you have multiple records for the same ESX host then the last record that is returned is used no matter what is entered. To test this, add a CNAME record to an ESX host that already has an A record. Then add the ESX host to a vCenter Server instance using the A record FQDN.
   
   
2. Instructors: The NIC you select for installation will become vmnic0.
   Comments: While true, it is important to note the difference between PXE NIC and installation NIC. The selected installation NIC, which may or may not be the same as the selected PXE NIC, will become vmnic0.
   
   
3. Instructors: Folders can be use to organize and control access the physical and virtual "worlds" of vCenter Server.
   Comments: It is important to note that resource pools should NOT be used as an organization or access control method. The reason for this is because resource contention can occur if resource pools are not configured properly. One reason why resource pools are used as folders is because folders cannot be created on a host by host basis.

Questions I raised:

1. Why does the 'Use Windows session credentials' checkbox in the vSphere client not work with ESX 3.5? They did not know.
   
   
2. Why does the ESX(i) installation not default to the maximum amount of memory for the VMkernel? They did not know, but thought it was a good idea.
   
   
3. Exporting system logs from an ESX host multiple time results in the following error message: 
   
   

   DiagnosticManager.GenerateLogBundles
   vim.fault.TaskInProgress

   
   It does not matter whether the same user is logged in twice or two different users are logged in. Why? Why are the tasks not queued? They did not know. I later found it that this works as expected from vCenter Server, whether all logs are downloaded or just the logs from a single host.