Insane SCVMM and Hyper-V - no API for remote access

For any enterprise product one of the fundamental facets is its public [programmatic] interface using which other products can interact with it or users can write scripting etc. 100% marks in this regard to VMWare vCenter which has a very well thought through web interface based on SOAP. Top that with client specific bindings in Java, .NET, Python etc and you are covered. There is no talk/bothering about what OS you are using on the client in order to connect and program the enterprise service.

Enter the world of 'insane scvmm' - things are not like how they should be and even worse, they are no where close to being sane. SCVMM is Microsoft's flagship product in v12n management and squarely positioned to compete with VMWare's vCenter. But guess what, there is no public API to SCVMM that's programmable from across different OSes. This is because the primary and only API available to program SCVMM is a powershell based. This has few major problems

1. Powershell is not available on non-Windows boxes. I know folks out there think powershell is universal et. al. But there is no credible and corporate-backed implementation of powershell on linux, solaris etc. Without this, no entperise wants to rely on the stack
2. The powershell documentation for scvmm is a nightmare to comprehend. Take 'New-VM' as an example - this cmdlet is used to create a new vm. It takes literally 100+ parameters across different variants of it. The parameter list is flat and available valid combinations and permutations are hard to deduce. Plus the documentation is not even available online in an html form so that it can bing-ed or google-d. It's only in MSWORD or PDF form.
3. Even in the documentation there is no place for me to just understand the entire set of properties that make up an object, say vm. There is only set-vm which has a set of properties that can be set- with usual quirks of valid combinations etc. There is a get-vm but that has no info about all the properties. Contrast this with vcenter's web-published documentation of its awesome object model.
4. Even if I accept client-can-only-be-on-windows apathy, the way to get the client SDK setup is clunky. I have to install a full-fledged admin console - which is a heavy-weight UI just to get 1 or more DLLs that have the scvmm powershell modules. Contrast that with vcenter which has a web interface to download the WSDL.

Coming to HyperV, this is even worse. The only interface to HyperV is WMI-based and that too the client MUST be local. To enable a remote client, one has to go through a laborious set of configuration changes, security anti-patterns etc so the client (again only windows based) can be on a different box. The interface is totally different and disjoint from that of SCVMMs. Contrast this with ESXi - the data and programming model for ESXi is exactly identical to that of vcenter. In fact any client that's written against the interface can work equally well when pointed to esxi or vcenter - this is holy grail!

We use .NET reflector and without this, it is simply impossible to write any non-trivial client for programming scvmm. Reflector disassembles a .NET assembly hence giving us insight into the actual model. We deal with SCVMM APIs in and out very deeply - many times I literally get the feeling of sickness looking at the api maturity, crazy constructs etc, especially relative to vmware centers APIs. God bless me and our team dealing with this insane product!

Insane SCVMM: Duplicating all BIOS identifiers on vm clone

Welcome to the second past in my 'insane scvmm' series. This one is about another insane thing that scvmm does in the context of SMBIOS UUIDs. Identity is a key foundation in any software area. Network adapters are identified by MACs. Machines are identified by an IP address within a single network. The whole web is based on URLs which are essentially identities to content on the web.

To identify computers uniquely, there is a standard called http://dmtf.org/standards/smbios. Of course not all vendors embrace and follow this but most do and all the Intel-based computer architectures and hence Windows OS based machines definitely do. The SMBIOS unique identifier is a UUID (aka GUID). Normally computer manufacturers assign this for physical machines. For virtual machines, hypervisors are responsible to assign the same.

In virtualization (v12n), 'clone vm' is a popular operation as it can be used to quickly provision an identical server including the os, apps, config etc. What we learnt to our utter surprise is cloning vms using scvmm results in the same SMBIOS UUID (and many other BIOS identifiers like serial number) assigned to the cloned vm, as that of the source vm. Totally insane. I raised this question at http://social.technet.microsoft.com/Forums/en-US/virtualmachinemanager/thread/b8cb5945-5b00-4145-b5ae-dc94fb5b68dc but received no credible response. One may wonder why do we care about unique UUID? The bad consequences are boundless - basically any (sane) software that relies on SMBIOS UUID to uniquely identify x86 based computers will be affected in strange ways.

Hyper-V by itself doesn't have a first-class clone operation. So SCVMM must be doing this somehow using available Hyper-V operations, so this is likely an issue in how scvmm and hyperv are working together. Wait a min, they never work together - isn't that the whole problem?

Insanity continues...

Insane SCVMM - MAC address management

Like most software professionals I'm super busy and have no time to write blogs even though I like to. Two things made this blog possible. One is the long weekend and the other more important one is insanity of SCVMM (System Center Virtual Machine Manager). In my current project I have to integrate with VMWare vCenter and Microsoft SCVMM closely. Time and again, we find things about SCVMM that just make us so frustrating - we can't believe how can someone release an enterprise product with such a freaking issue or lack of thought. Anyways here I'm - deciding to start a series called 'Insane SCVMM'. This is the first one and it's about how SCVMM and Hyper-V manage MAC addresses.

In case of physical network adapters, manufacturers worry about MAC address assignment but in case of virtual network adapters (vnics) used by virtual machines, the software (typically hypervisor or hypervisor managers like vcenter, scvmm) need to do this. We found interesting things on how SCVMM/Hyper-V combo does this.

Let's start with HyperV. HyperV individually has a limit of 256 unique mac addresses by default. The range is stored in registry when you enable the hyperv role on a windows server –
registry stores the min and max mac address for the range. The first 3 octets are fixed (Microsoft organization identifier) and next 2 octets come from the last 2 components of ip address of the primary NIC on the box. That leaves only one octet which gives the number 256.

If we clone the hyperv box (or use a windows image to build multiple hyperv boxes),
the same range is used! Unfortunately sysprep doesn’t help here, although it
should have - talk about cross-group collaboration! So the recommendation is to install hyperv after os setup (post install script) or explicitly remove the relevant registry keys so they are regenerated when hyperv starts on the new box. http://blogs.technet.com/b/jhoward/archive/2008/07/15/hyper-v-mac-address-allocation-and-apparent-network-issues-mac-collisions-can-cause.aspx is a good article about this.

However when scvmm is used to manage hyperv hosts (and create vms) the mac address
management happens at scvmm level where the limit is 3million+. Ideally this is all one should have cared, IF scvmm had a good design and better coordination with Hyper-V. But there is a
catch – a big one in-fact. Even through scvmm, when we select a dynamic mac
address, scvmm leaves the mac address generation to the hyperv host - how great? This
essentially means the mac address is assigned from hyper-v mac address pool
with no scvmm intervention and hence we still need to worry about 256 mac limit. When user selects a static mac address and uses ‘generate’ button (shown below) the mac address is generated from scvmm’s global [static] mac address pool. In fact
when a new vm is created with ‘dynamic’ mac address for a vnic, that vnic’s mac
address will be all zeroes until you start the vm. Another insanity.

Clone dynamic mac case: when you clone a vm, the cloned vm will get same mac as
source until you start the clone (insanity) at which point hyper-v generates a new
address. Funny things can happen when all addresses on hyperv are filled up and
things have to recycled. Vm starts can fail with no more mac addresses or they
can start with a mac address that’s unused by any currently running vm – thus
possibly getting a currently in-use mac of a powered down vm. However Hyper-V
at-least guarantees mac address once assigned to a vnic is immutable.

Clone static mac case: the cloned vm will get the same static mac as the source and
this will not be auto-generated from the scvmm global mac pool even after
powering on the cloned vm.

Create vm is similar. Create vm starts out as all zeroes for all dynamic macs (insanity) until we
start the vm.