Insane SCVMM - MAC address management

Like most software professionals I'm super busy and have no time to write blogs even though I like to. Two things made this blog possible. One is the long weekend and the other more important one is insanity of SCVMM (System Center Virtual Machine Manager). In my current project I have to integrate with VMWare vCenter and Microsoft SCVMM closely. Time and again, we find things about SCVMM that just make us so frustrating - we can't believe how can someone release an enterprise product with such a freaking issue or lack of thought. Anyways here I'm - deciding to start a series called 'Insane SCVMM'. This is the first one and it's about how SCVMM and Hyper-V manage MAC addresses.

In case of physical network adapters, manufacturers worry about MAC address assignment but in case of virtual network adapters (vnics) used by virtual machines, the software (typically hypervisor or hypervisor managers like vcenter, scvmm) need to do this. We found interesting things on how SCVMM/Hyper-V combo does this.

Let's start with HyperV. HyperV individually has a limit of 256 unique mac addresses by default. The range is stored in registry when you enable the hyperv role on a windows server –
registry stores the min and max mac address for the range. The first 3 octets are fixed (Microsoft organization identifier) and next 2 octets come from the last 2 components of ip address of the primary NIC on the box. That leaves only one octet which gives the number 256.

If we clone the hyperv box (or use a windows image to build multiple hyperv boxes),
the same range is used! Unfortunately sysprep doesn’t help here, although it
should have - talk about cross-group collaboration! So the recommendation is to install hyperv after os setup (post install script) or explicitly remove the relevant registry keys so they are regenerated when hyperv starts on the new box. http://blogs.technet.com/b/jhoward/archive/2008/07/15/hyper-v-mac-address-allocation-and-apparent-network-issues-mac-collisions-can-cause.aspx is a good article about this.

However when scvmm is used to manage hyperv hosts (and create vms) the mac address
management happens at scvmm level where the limit is 3million+. Ideally this is all one should have cared, IF scvmm had a good design and better coordination with Hyper-V. But there is a
catch – a big one in-fact. Even through scvmm, when we select a dynamic mac
address, scvmm leaves the mac address generation to the hyperv host - how great? This
essentially means the mac address is assigned from hyper-v mac address pool
with no scvmm intervention and hence we still need to worry about 256 mac limit. When user selects a static mac address and uses ‘generate’ button (shown below) the mac address is generated from scvmm’s global [static] mac address pool. In fact
when a new vm is created with ‘dynamic’ mac address for a vnic, that vnic’s mac
address will be all zeroes until you start the vm. Another insanity.

Clone dynamic mac case: when you clone a vm, the cloned vm will get same mac as
source until you start the clone (insanity) at which point hyper-v generates a new
address. Funny things can happen when all addresses on hyperv are filled up and
things have to recycled. Vm starts can fail with no more mac addresses or they
can start with a mac address that’s unused by any currently running vm – thus
possibly getting a currently in-use mac of a powered down vm. However Hyper-V
at-least guarantees mac address once assigned to a vnic is immutable.

Clone static mac case: the cloned vm will get the same static mac as the source and
this will not be auto-generated from the scvmm global mac pool even after
powering on the cloned vm.

Create vm is similar. Create vm starts out as all zeroes for all dynamic macs (insanity) until we
start the vm.