How to Upgrade from Windows vCenter 5.5 to VCSA 6.5 including SRM

In last couple of days, I heard this question many times for below scenario. Since 5.5 is about to expire this month, so IT admins are upgrading their environment to new version of vSphere. Though going directly to 6.7 is something that do not meet N-1 requirement for most of the environment. Hence most of the admin prefers to put environment on vSphere 6.5.

Very first question everyone think about is –

  • Does upgradation path from vCenter 5.5 to VCSA 6.5 is supported? Answer is YES.
  • Does upgradation path from VMware SRM 5.8.1 to 6.5 is supported? Answer is NO. You need to upgrade VMware SRM 5.8.1 to 6.0 or 6.1.2, and then upgrade SRM 6.0 to 6.5.

Here we have details of existing environment and requirement which need to meet.

Existing Environment: –

  • Windows based vCenter Server – Version 5.5
  • Site Recovery Manager – Version 5.8
  • Replication Type in SRM – vSphere Replication

Requirement: –

  • Appliance based vCenter Server (VCSA) – Version 6.5
  • Site Recovery Manager – Version 6.5
  • Replication Type in SRM – vSphere Replication

Sequence to upgrade Windows based vCenter 5.5 to VCSA 6.5: –


If you are planning to upgrade your vSphere environment, follow this order to upgrade vCenter server, SRM, and vSphere Replication.

Overview of Upgrade order: –

  • Since vCenter server 5.5 doesnt have PSC server, you need to make sure that you install PSC when prior to upgrade vCenter server.
  • Upgrade PSC and vCenter Server in Protected Site.
  • Upgrade vSphere Replication appliance in Protected Site.
  • Upgrade Site Recovery Manager in Protected Site.
  • Perform the same steps in Recovery Site.
  • If you are using array based replication in SRM, you need to upgrade SRA in both site.
  • Once up gradation sequence has been done, verify vCenter server and SRM sites status.
  • Upgrade ESXi host in both protected and recovery sites.

Step by Step Guide to Upgrade vSphere Environment: –

Prerequisites: –

  • Download vCenter Server Appliance ISO image, VMware SRM 6.5 setup, and vSphere Replication appliance from VMware download portal.
  • Ensure that you have a windows machine from where you will initiate the installation.
  • Ensure to get SSO credentials and VCDB and SRM DB database handy.
  • Note down the details of ESXi Host where you will deploy VCSA appliance.
  • If you want to use the same name of vCenter Server which you are using currently, you need to rename windows vCenter VM with alternate name.
  • If you are using VMware SRM 5.8.1, then you need to upgrade it to VMware SRM 6.0 and then you can upgrade it to Vmware SRM 6.5.

vCenter Server Upgrade: –

There are two stage process to upgrade vCenter Server.

  1. Deployment of VCSA
  2. Migration of Windows vCenter Data to newly deployed VCSA.


Stage 1:

  • Now mount VCSA image ISO in any windows machine. Explore the ISO folder and navigate to VMware-Migration-Assistant.exe. Right click and click on Run as Administrator.
  • Follow the steps and provide SSO credentials.
  • Once you will get black screen with a message – Waiting for migration to Start, Switch to ISO folder again.
  • Go to vcsa-ui-installer/win32 folder and click on Installer and Run as administrator.
  • On windows screen, you will get four options.
    • Install
    • Upgrade
    • Migrate
    • Restore
  • Click on Migrate and follow the steps to complete the deployment of VCSA.

Stage 2:

  • Once deployment gets complete, you need to switch to stage 2. Here you need to migrate existing windows vCenter server data to VCSA.
  • During the data migration the Windows vCenter will be shutdown and the VCSA will be configured with its IP address.
  • Follow the steps and complete stage 2. Now you can access vCenter server using Web client.


Upgrade vSphere Replication Appliance: –

Download vSphere replication appliance and upgrade appliance by following below VMware document.

Upgrade VMware Site Recovery Manager: –

Follow below article to upgrade VMware Site Recovery Manager.







How VMware HA Works | Deep Dive

Overview of HA (High Availability): –

  • When you creates HA cluster very first time, then Virtual Machines are configured with cluster default settings.
    • VM Restart Priority
    • Host Isolation Response
    • VM Monitoring
  • There is master host election when the cluster is first created. All other hosts are slaves.
  • Master host is responsible for monitoring the host connectivity with slave host.
  • Master host also deals with different possible issues that can happen.
    • Host get network isolated.
    • Host fails (Hardware or other problem).
    • Host loses connection to the master host.
  • For Virtual Machine monitoring, there are three options.
    • Leave running (Default)
    • Shutdown (Required VM Tools)
    • Power Off

Component of HA: –

  • FDM
  • Hostd
  • vcenter

FDM (Fault Domain Manager): –

  • Communicating host resource information, VM state, and HA properties to the other hosts.
  • It also handles heartbeat mechanism.HA deep dive
  • It provides VM placement and VM restart.
  • HA has no dependency on DNS. It works on IP Address. This is improvement in FDM.
  • FDM directly talk to hostd and vCenter
  • FDM is not dependant on VPXA.
  • You can check FDM logs – fdm.log in /var/log/

HOSTD Agent: –

  • It is agent which is installed on ESXi host.
  • Responsible for many task like power on Virtual Machine
  • If HOSTD is unavailable or not running, host will not participate in any FDM related process.
  • FDM relies on HOSTD for information about the VM that are registered to the host, and manager VM through HOSTD API.
  • FDM is dependant on HOSTD. If HOSTD is not operational, FDM halts all functions and wait for HOSTD to become operations.

Use of vCenter in HA: –

  • Deploying and configuring HA Agent.
  • Communication of cluster confiugration change
  • Protection of VM
  • Pushing out the FDM to the ESXi hosts.
  • Communicate configuratoin changes in the cluster to the host.
  • HA leverage vCenter to retrieve information about the status of VM.

Fundamental Concepts of HA: –

  • Master/Slave Agent

  • Heart-beating

  • Isolated vs Network Partitioned

  • VM Protection

  • Component Protection

Understand Master Host: –

  • HA Architecture includes the concept of Master and Slave HA agent.
  • There is only one master slave in a HA cluster, except during Network Partition scenario.master host
  • Master is responsible for monitoring the health of VM
  • Restart any VM which fails.
  • Slave pass information to master.
  • HA agent also implements the VM/App monitoring feature which allows it to restart virtual machine in case of a OS or restart service in case of application failure.
  • Master Agent keep track of the VM for which it is responsible for, and take action when appropriate.
  • Master will claim responsibility for a VM by taking ownership of the datastore on which VM configuration file is stored.
  • Master is responsible for exchanging state information with vCenter Server.
  • Send/Receive information to vCenter when required.
  • Master host initiate the restart of VM when host failed.

What if Master Fails?

master fails

HA election occurs when you enable HA on VMware Cluster and master host:-

  • Fails
  • Become Network partition or isolated.
  • Disconnect from vCenter Server.
  • Put in maintenance or standby.

HA election takes 15 seconds to elect slave as a master. It works over UDP protocol.

Make the election process on the basis of highest number of datastore.

If two or more host has some number of datastore, the highest/largest MOID will get preference. It’s done on basis of lexically MOID. Let’s take a value of MOID of two Hosts 99 and 100. Here 9(99) is greater then 1(100)(9 >1). In this example, 99 is largest MOID.

When master is elected, it will try to acquire the ownership of datastore which it can directly access by proxying request to one of the slave connected to it using the management network.

For regular storage architecture, it does this by locking a file called “Protected List”.

Naming format and location for this Protected List file is as below.

./vSphere HA/ <Cluster Specific Directory>/ProtectedList

Structure of cluster specific directory.

<UUID of VC> -<Number of the MOID of Cluster>-<Random 8 character string>-<Name of the host running VC>

Understand Slave Host: –

  • It monitors state of Virtual machine and inform Master host.
  • Monitor health of master by monitoring heatbeat.
  • Slave host sends heartbeat to master so that master can detect outage.

Local Files for HA: –

When HA is configured on a host, the host will store specific information about it’s cluster locally.

  • Cluster Config
    • It’s not human readable.
    • It contains configuration details of cluster.
  • vmmetadata
    • This file is also not human readable.
    • It contains actual compatibility information matrix for every HA protected VM and list all with which it is compatible.
    • Metadata includes the custom properties, descriptions, tags, owner, cost center, etc regarding a Virtual Machine.
  • fdm.cfg
    • Configuration setting and logging and syslog details are store in this file.
  • hostlist
    • A list of host participating in the cluster, including hostname, IP address, MAC address and heartbeat datastore.

Understand Heartbeating: –

Mechanism used by HA which check if host is alive.

There are two types of Heartbeat.

  1. Network Heartbeat
  2. Datastore Heartbeat

Network Heartbeat: –

  • It use by HA to determine if a ESXi host is alive.
  • Slave send network heartbeat to master and master to slave.
  • It send heartbeat by default every second.

Datastore Heartbeat: –

  • It add on extra level of resiliency and prevent unnecessary restart attempts.
  • Datastore heartbeat enables a master to determine the state of a host that is not reachable via management network.
  • By default there are two datastores get selected. But it can be possible to add more datastores. You can do this by following string in Advance options. Valid values can range from 2-5 and the default is 2.
  • Selection process gives preference VMFS datastores over NFS.

Network Isolated vs Partitioned Network Partitioned: –

Isolated vs partitioned


  • When it doesn’t observe any HA management traffic on management network and it cannot ping the configured isolation network address.
  • Host is isolated only when host inform the master via the datastore that is isolated.


  • When host is operational but cannot reach over management network.
  • There will be multiple masters in case of network partitioned.


How vMotion Works ?

What is vMotion?

  • vMotion  enables live migration of a running virtual machine between ESXi Hosts
  • It is transparent to the Virtual Machine’s OS and applications.
  • Invaluable tool to admin to achieve the followings;
    • Avoid Server Downtime
    • Allow Troubleshooting
    • Provide Flexibility
  • Key enabler of DRS, DPM, and FT

vMotion works

What needs to be migrated?

  • Processor and devices state  – CPU, Network, SVGA
  • Disk – Use shared storage between source and destination
  • Memory – Pre copy memory while Virtual Machine is running

How vMotion Works?

  • Quiesce Virtual Machine on source machine.
  • Transfer memory and device state(checkpoint) from source to destination.
  • Resume Virtual Machine on Destination
  • Copy remainder of memory from source to destination
  • Free Virtual Machine resources on source machine.

vMotion Step by Step

Other Interesting Facts, Problems, and Troubleshooting: –

  • Virtual  Machine remains suspended during memory transfer
  • Copying Virtual Machine with large memory size may problematic.
  • 64 GB Virtual Machine requires around 57 seconds on 10 GbE NIC.
  • VMotion will check the remote system to make sure there is enough RAM and CPU before it begins the process.

Troubleshooting: –

  • Migration ID is same on source and destination.
    • Go to VMkernel log (/var/log/vmkernel.log)
    • Grep the migration ID for all vMotion related timing and statistics.

That’s it from here. Stay connected.


Understand L1 Terminal Fault (L1TF) – Impact and Mitigation Plan for VMware Admins

Overview of L1TF Vulnerabilities: –

Intel has disclosed on 14th Aug 2018 new class of three CPU speculative-execution vulnerabilities within its server, client and workstation processors, known as “L1 Terminal Fault (L1TF)” which can occur on past and current Intel processors (from at least 2009 – 2018)


The processor vulnerability goes by L1TF, L1 Terminal Fault, and Foreshadow. Researchers who discovered the problem back in January and reported it to Intel called it “Foreshadow”. It is similar to vulnerabilities discovered in the past such as Spectre and Meltdown.

The new L1 Terminal Fault vulnerability involves a security hole in the CPU’s L1 data cache, a small pool of memory within each processor core that helps determine what instruction the core will execute next. L1 Terminal Fault vulnerability can occur when affected Intel microprocessors speculate beyond an unpermitted data access.

VMware has defined below mentioned four categories for such vulnerabilities.

  • Hypervisor-Specific Mitigations prevent information leakage from the hypervisor or guest VMs into a malicious guest VM. These mitigations require code changes for VMware products.
  • Hypervisor-Assisted Guest Mitigations virtualize new speculative-execution hardware control mechanisms for guest VMs so that Guest OSes can mitigate leakage between processes within the VM. These mitigations require code changes for VMware products.
  • Operating System-Specific Mitigations are applied to guest operating systems. These updates will be provided by a 3rd party vendor or in the case of VMware virtual appliances, by VMware.
  • Microcode Mitigations are applied to a system’s processor(s) by a microcode update from the hardware vendor. These mitigations do not require hypervisor or guest operating system updates to be effective.

What is affected by L1TF: –

This vulnerability is Intel-specific. Other processors such as AMD are not affected.

What is impacted

Intel’s previously released microcode updates are expected to lower the risk of data exposure for consumer and enterprise users running non-virtualized operating systems, Hence, there are no significant performance impacts have been noted with this particular mitigation. For virtual machines, however, the risk is higher.

Three CVEs have been assigned to this issue:

L1TF Vunerabilities

  • CVE-2018-3615 for Intel Software Guard Extensions (L1 Terminal Fault-SGX)

    • Systems with microprocessors utilizing speculative execution and Intel SGX may allow unauthorized disclosure of information residing in the L1 data cache from an enclave to an attacker with local user access via side-channel analysis.
    • Does not affect VMware products and/or services.
    • Reference Link:
  • CVE-2018-3620 for operating systems and System Management Mode (L1 Terminal Fault-OS/ SMM)

    • Systems with microprocessors utilizing speculative execution and address translations may allow unauthorized disclosure of information residing in the L1 data cache to an attacker with local user access via a terminal page fault and side-channel analysis.
    • Virtual Appliances are impacted. List of unaffected appliances can be found from here. is recommended to contact 3rd party vendors for appliance patches.
    • Products that ship as an installable windows or linux binary are not directly affected.
    • May also be applicable to customer-controlled environments running in a VMware SaaS offering. Refer to
    • Other Reference Link:
    • Requires Operating System-Specific Mitigations.
  • CVE-2018-3646 for impacts to virtualization (L1 Terminal Fault-VMM)

    • This is specific to Virtual environment and impacting your Virtual machines.
    • Systems with microprocessors utilizing speculative execution and address translations may allow unauthorized disclosure of information residing in the L1 data cache to an attacker with local user access with guest OS privilege via a terminal page fault and side-channel analysis.
    • It impacts hypervisors. It may allow a malicious VM running on a given CPU core to effectively infer contents of the hypervisor’s or another VM’s privileged information residing at the same time in the same core’s L1 Data cache.
    • Requires Hypervisor-Specific Mitigations for hosts running on Intel hardware.

Affected Products: –

The following Intel-based platforms are potentially impacted by these issues.

Intel® Core™ i3 processor (45nm and 32nm)
Intel® Core™ i5 processor (45nm and 32nm)
Intel® Core™ i7 processor (45nm and 32nm)
Intel® Core™ M processor family (45nm and 32nm)
2nd generation Intel® Core™ processors
3rd generation Intel® Core™ processors
4th generation Intel® Core™ processors
5th generation Intel® Core™ processors
6th generation Intel® Core™ processors **
7th generation Intel® Core™ processors **
8th generation Intel® Core™ processors **
Intel® Core™ X-series Processor Family for Intel® X99 platforms
Intel® Core™ X-series Processor Family for Intel® X299 platforms
Intel® Xeon® processor 3400 series
Intel® Xeon® processor 3600 series
Intel® Xeon® processor 5500 series
Intel® Xeon® processor 5600 series
Intel® Xeon® processor 6500 series
Intel® Xeon® processor 7500 series
Intel® Xeon® Processor E3 Family
Intel® Xeon® Processor E3 v2 Family
Intel® Xeon® Processor E3 v3 Family
Intel® Xeon® Processor E3 v4 Family
Intel® Xeon® Processor E3 v5 Family **
Intel® Xeon® Processor E3 v6 Family **
Intel® Xeon® Processor E5 Family
Intel® Xeon® Processor E5 v2 Family
Intel® Xeon® Processor E5 v3 Family
Intel® Xeon® Processor E5 v4 Family
Intel® Xeon® Processor E7 Family
Intel® Xeon® Processor E7 v2 Family
Intel® Xeon® Processor E7 v3 Family
Intel® Xeon® Processor E7 v4 Family
Intel® Xeon® Processor Scalable Family
Intel® Xeon® Processor D (1500, 2100)

** indicates Intel microprocessors affected by CVE-2018-3615 – L1 Terminal Fault: SGX

How to Mitigate L1TF in your VMware Environment: –

Need to ensure that all virtualized operating systems have been updated. Additional steps include turning off hyper-threading in some scenarios and enabling specific hypervisor core scheduling features.

Mitigation of CVE-2018-3615 (L1 Terminal Fault – SGX) – {No action for VMware Admins}:

  • CVE-2018-3615 does not affect VMware products and/or services. Hence no mitigation is required for Vmware admins.

Mitigation of CVE-2018-3620 (L1 Terminal Fault – OS) – {More Specific to 3rd party Vendors}:

  • Mitigation of CVE-2018-3620 requires Operating System-Specific Mitigations.  Impact may have on Virtual Appliances and VMware SaaS Offerings.
  • Products that ship as an installable windows or linux binary are not directly affected, but patches may be required from the respective operating system vendor that these products are installed on.
  • It is recommended to contact 3rd party vendors for appliances and SaaS offerings for mitigation plan of CVE-2018-3620. Like if you are using a Cisco virtual appliance, then you need to contact Cisco vendor for the mitigation plans.
  • For VMware appliances, Vmware has listed the unaffected appliance here.

Mitigation of CVE-2018-3646 (L1 Terminal Fault – VMM) – {More specific to VMware Admins}:

  • Mitigation of CVE-2018-3646 requires Hypervisor-Specific Mitigations for hosts running on Intel hardware.
  • As a Vmware admin, you need to focus more on VCE-2018-3646 as it is directly impacting Hypervisors and Virtual machines which have Intel CPU. Affected product list may be find out above.Mitigation Structure
  • Sequential-context attack vector: a malicious VM can potentially infer recently accessed L1 data of a previous context (hypervisor thread or other VM thread) on either logical processor of a processor core.
  • Concurrent-context attack vector: a malicious VM can potentially infer recently accessed L1 data of a concurrently executing context (hypervisor thread or other VM thread) on the other logical processor of the hyperthreading-enabled processor core.

There are three phases to mitigate CVE-2018-3646 as mentioned below.

  1. Update Phase
  2. Planning Phase
  3. Scheduler-Enablement Phase

L1TF Mitigation Phases

Above three mitigation phases for CVE-2018-3646 is defined below in more descriptive way:

To summarize this, here is quick step:

  • Update vCenter Server using Product listed in VMSA
  • Patch your ESXi Hosts using Product listed in VMSA.
  • Enable ESXi Side-Channel-Aware Scheduler using vSphere Web Client.
    • Set option to True of VMkernel.Boot.hyperthreadingMitigation in Advance Setting of ESXi Host.

Mitigation Plan

You can refer to VMware KB Article 55806 to see the depth mitigation plan for CVE-2018-3646.

Your Environment is secure now. Enjoy the monsoon.

That’s all from this article. If you want to add your inputs, please do share in comment box.


Share this article to others if you found it useful.





Learning Modules on Kubernetes for Beginners

Good Day All! Hope you are enjoying your weekend.

In last couple of days, I was exploring Kubernetes in my Lab environment. I explored many new things as a beginner. I tried to gather all W-H questions which may ask or come into your mind if you are thinking to start Kubernetes.


Based on that, I prepared a learning module on Kubernetes Specially for Beginners. If you are new to Kubernetes, you might have lot of queries such as How and Where to Start it from scratch. I segregated all such queries in different topics and prepared a complete learning module specially for techies who just jumped or thinking to start this hot selling technology.

Here is complete module of Kubernetes for Beginners: –


Topic 1 – What is Kubernetes? | Learn Kubernetes – Part 1

Topic 2 – Components and Architecture of Kubernetes | Learn Kubernetes – Part 2

Topic 3 – Versions of Kubernetes | Learn Kubernetes – Part 3

Topic 4 – Kubernetes Terminology every admins need to know | Learn Kubernetes – Part 4

Topics 5 – Getting Start to Setup and Configure Kubernetes | Learn Kubernetes – Part 5

Topics 6 – How to Install Kubernetes on Windows 10 with Hyper-v using Minikube| Learn Kubernetes – Part 6

In next few days, I am going to add more topics here whatever I get to know as a beginners. I will get back to you through this blog. Please stay tuned.

If you wants to add some topics here or if I did some mistake, please do not hesitate to share in commend box or email us at


If you think that it is useful, please do share with others.