Hypervisor insanity

I've been working with virtualization for about a decade.  I've either set up, managed, or tested the vast majority of hypervisors on the market.  The few that I will immediately discard are Xen and Hyper-v, because they put more effort in attempting to disrupt the field than they did building out a capable product.  Hyper-v is great if you are a Windows shop, but even Microsoft is moving away from being a Windows shop (seriously, look at how they accomplish containerization).  Xen is just a solid no from the schism that resulted in Xen and KVM about 9 years ago.  KVM won, it works better.  The best tools I have used are typically VMware products.

For type 2 hypervisors, the ones that run like applications on an existing operating system, I don't keep very up to date.  Virtual Box is pretty good at what it does, but it can be a complete pain to work with professionally.  Open Virtual Format (OVF) was supposed to be the standard for how the virtual machine files were stored, and Virtual Box has done an okay job with keeping up, but still present problems when migrating from their OVF to VMwares OVF or KVMs QCOW. 

Aggregate 1:
Virtual Box, an easy way to annoy the enterprise

KVM is an excellent standard for virtualization.  It is definitely in the top 2 for virtualization.  The biggest problem is that people gravitate towards it because it is free.  They ignore major shortcomings because they didn't have to pay for it.  The biggest shortcoming is that it defaults to QCOW for it's files, reducing portability to other hypervisors.  Because they did this, there is not a unified version of OVF across the hypervisors with all of the groups working towards a common goal.  It also makes porting it to Virtual Box or VMware a major obstacle.  This is not my biggest gripe with KVM.  My biggest gripe is the Qemu requirement turning a type 1 hypervisor into a type 2 hypervisor.  I know they are working to bake in the pieces needed, and may have already succeeded in doing so, but it is still a requirement in every desktop implementation of KVM that I have worked with.

Aggregate 2:
KVM, the #1 choice for linux users

VMware has a couple of type 2 products, which work well.  Player allows you to build a VM and works well for testing an application.  It's a great way to get a VM up and running before determining if you want to spend money on a full type 2 suite.  If you decide that you want to do that, then Workstation has a ton of tools.  It is built for enterprise development and can run on both Windows and Linux.  Once again, the shortcoming is the OVF format, which VMware is actively supporting on all of their hypervisors.  It just doesn't play nice with others in a similar way that Microsoft office applications don't play nice with open source office applications. 

Aggregate 3:
VMware Workstation, how much do you want to spend?

I rarely use type 2 hypervisors.  I usually am getting aggravated at someone bringing me a Virtual Box OVF that doesn't work because they forgot to unmount some virtual optical disk, or isn't compatible with the system I am on.  I run KVM on my home workstation, and have managed it on quite a few systems professionally.  What irritates me is that the desktop implementation of KVM is treated like an enterprise type 1 hypervisor on many of the systems I have managed.  It is not, and should not be treated like one.  This has led to many design flaws that I have worked hard to fix.

Type 1 hypervisors are where it's at.  In the beginning of my professional work with virtualization, there were few options.  You could run the very old version of Xen that became KVM on a minimal linux system and virtualize a few systems.  Or, you could run ESX to maximize the density of virtual machines.  This was back when ESX still used RedHat code to boot.  I started on ESX 3 without a vCenter server.  I consolidated 1 rack of 3u servers into 4u of space.  When the bosses took notice, I got funding and took 8 racks of servers down to 6x 2u servers.  I upgraded through the years to vSphere 6 in that environment and left for greener pastures. 

Since I have left, I have been able to implement vSAN and NSX to get closer to the Hyper Converged Infrastructure (HCI), which I think is really what I had in mind when I was imagining what a Software Defined Data Center (SDDC) should be.  This is when I started running into the insanity of competing products.  The biggest in management requirements being Open Stack.  An amazing product if you have a team that can work on it.  There really isn't a good way to implement this monstrosity without a few thousand documents written on how to operate it in case you ever want to take a vacation.  Trying to pull pieces out of it to make HCI work without implementing Open Stack is a fools errand.  I spent a few months messing around with Open Daylight in an attempt to avoid spending money on NSX.  Then I realized that Open Daylight is a toolset for you to build your own solution, not a solution that can be implemented. 

Aggregate 4:
Open Stack, never be allowed to take vacation again

This led me to look for solutions that are ready to be implemented elsewhere for less money.  I looked into Ceph and Gluster on Red Hat Enterprise Virtualization (RHEV), being naive and thinking that it wouldn't cost money.  Ceph was pretty much a non starter on RHEV.  Gluster costs money and was documented in a way that looked like drive mirroring across nodes.  There wouldn't have been any ideal savings in price when drives and capability were factored in.  The worst of the HCI was looking into Software Defined Networking (SDN) on RHEV.  They were beta testing Open Virtual Switch.  This immediately put them 5 to 10 years behind VMware.  The RHEV management console was the definition of janky.  So, they lost out on anything bigger than a 4 node cluster for testing, which was wiped when the systems were needed elsewhere.

This is also where I found out that Red Hat was becoming a strange creature.  They were buying up a bunch of Open Source projects and charging a significant amount of money for enterprise support.  By the time my lab started talking about containers, Red Hat became the worst company to test on.  The products that are free and available everywhere became a cost that we could not get over.  Since my professional development operations happen out of band, not connected to any other network, their implementation of Docker was a nightmare.  A few simple changes could fix it, but it was a bridge too far by the time you added up the cost and lack of turn-key features. 

Aggregate 5:
RHEV, 5 years behind competition is up to date

VMware tends to just work.  The turn-key solutions are pretty much ready to go out of the box.  The worst thing I can say about it is that up until recently, I couldn't really build my home lab out with quiet ESXi servers and not worry about an enterprise grade power bill.  It is very expensive to run VMware HCI.  But it works and is cutting edge.  They also have online Hands On Labs that can help get the staff trained quickly and effectively for free.  Before management freaks out about the price, keep in mind that it actually works and it does things that competitors can not do.

Aggregate 6:
VMware, throw money at problems to make them go away

Since I run FreeNAS at home, I had a different solution.  Welcome Bhyve.  The BSD type 1 hypervisor.  I can use my NAS storage as the virtualization datastore, kick off virtual machines, and set up a decent environment for development.  The biggest lacking feature that I have noticed is the inability to easily pass through devices on the box that I built.  I'm not certain if this is an actual issue, since my hardware doesn't really support some of the things I would like to pass through, like PCI-e cards.  But, at some point I would like to add on my z-wave dongle to a virtual container host, so I will be blogging about that in the future. 

Aggregate 7:
Bhyve, because I already have the NAS

For almost all of my needs at home, the Bhyve implementation works out perfectly.  For anything else, I use the KVM implementation on my workstation.  It is the "everything else" box.  Hopefully this shed some light on enterprise and home solutions, and how/why to make decisions on what might be a good fit for home and business labs.