Archives for March 2014


Binny Gill on your Data Centers Silent Killer

Binny Gill, is a Director of Engineering at Nutanix. Truth is I know he was promoted but I don’t remember his new fancy title 🙂

Hard drive disk toning in blue color close-upI had to chance to talk with him about storage as whole from the enterprise perspective. He mentioned how important it was to protect the data above anything else, resiliency over performance. Safe to say I think Nutanix has done that and performance will continue to climb with existing gear with our NOS 4.0 release. (Note the numbers are impressive now)

I did want to share something from Binny if your thinking of storing any amount data:

We are far superior than traditional filesystems because we are paranoid about silent corruptions and bitrot.

We keep checksum separate from data for all data and compare the checksum before returning data. Others use 520 byte sectors or more (528 bytes in IBM drives).. but those things are not commodity and not broadly applicable. When backup is taken, the checksum is compared and we are sure that there is no spreading of bitrotted data to backups.

We have disk scrubbing that makes sure that any bit flips are detected and repaired.

Our metadata is also self-checksumming and hence protected from bitrot.

Simply stated, we will never return corrupted data even if the drives are faulty. At best we will return no data.

You can go to 6 TB to 6 PB+ with Nutanix but it would be all for not if we didn’t manage your data like your retirement fund.

[UPDATE: For DRAM, we use x4 dies, we use Chipkill which is a much superior algorithm than vanilla ECC. We can lose an entire die and still be able to correct the data.]

Additional Resource:
A great article on bit rot.


Nutanix Short: The Welcome Banner

You have the option to create a welcome banner, which will be the first screen that appears when a user attempts to log into the (PRISM)web console. The content of the banner page is configurable, and it can include a custom message and graphics. Since everything is in HTML5 you can insert html tags into the welcome banner. I think this has lead to lots of funny messages within the Nutanix Engineering department since the feature has been added.

Below is a cluster that I use for the Nutanix weekly webinar’s

Nutanix welcome banner

Configuration of the welcome banner

When you first log in.

What you see before you log into the cluster

What you see before you log into the cluster

Really to keep the compliance people happy but you can have some fun with it too.


EUC Tip 93: XenDesktop & the vCenter Virtual Appliance

If you plan just to use the self-signed certificate with XenDesktop and Machine Creation Services(MCS) make sure you regenerate the certificate for the vCenter Virtual Appliance. This will allow you to configure MCS by allowing you to store the certificate from vCenter on the Desktop Controller.

XenDesktop on Nutanix

Regenerate the local cert


Data Locality: Congestion Not Latency

I love Nutanix data locality for helping with the noisy neighbour syndrome and because it helps in spreading the data evenly across the cluster. Spreading the data across the cluster has impact on rebuild times and bottlenecks.

Our Director of Engineering, Vishal Sinha brought up another good point yesterday around data locality. He mentioned that what kills network performance is not latency but congestion. Congestion can come in many forms – microbursts (200Mbps burst for 10ms, which equates to 20Gbps equivalent of traffic on the 10G port for that 10ms, resulting in lots of traffic getting dropped if the switch does not have enough buffer space), or for e.g. a mis-behaving NIC sending PAUSE frame and slowing down the network.
Our data locality feature drastically reduces chances of running into network related storage issues since we don’t rely on network for reads and we need to write to only one remote node. It’s all about reducing coupling and dependency between various components, and limiting resource consumption. Do more with less, make components independent == scalability. Data locality is core component of distributed systems, for example hadoop.

If you want to read more about microburst, here is a link:


Flash – Performance at what cost?

Flash is everywhere these days and everyone likes to throw out impressive performance numbers. If someone told you that flash is faster than a hard drive I am sure the thought that pops into your head is, “yeah… thanks for the info”. Nothing shocking about the statement, the real question is what do you do with this resource? Most vendors are using roughly the same NAND with slight variations. How does the implementation take full advantage of the most expensive part of the solution?

One Giant Pool of Flash

Nutanix is a converged distributed storage system made up of many nodes. Each node supplies locally direct attached flash and hard drives. It’s the local virtual storage controller running on each node that takes all of the resources and puts them into one storage pool.

While workloads running on Nutanix write locally for performance, the workloads are not limited to the local flash devices. Nutanix doesn’t create silos within its architecture; it’s controlled orchestration to take advantage of all of the components. The Nutanix Virtual Computing has an awareness of flash capacity on all of the nodes and when reaching capacity on a local device will allow workloads to go over the network to maintain performance. Nutanix customers don’t have to pay a premium for larger flash devices and have peace of mind if the working set spills over the available local flash.




Post by Declan Waters

Nutanix today announced that it has been granted a patent (US 8,601,473) from the United States Patent and Trademark Office recognizing the company’s fundamental storage architecture for virtualized datacenters, which not only obsoletes 20 year-old storage area network (SAN) technology, but also challenges the entire industry to design a next-generation storage architecture with comparable performance and scalability. The patent provides clarity as to how software-defined storage solutions are optimally designed and implemented, while at the same time bringing the technology into mainstream enterprises around the world.

Leveraging these documented inventions, Nutanix brings simplification to enterprise data centers by delivering scale-out storage services via software running on off-the-shelf x86 servers. Datacenter managers are now freed from rigid storage constructs and burdensome administrative tasks that have frustrated virtualization teams and slowed important business initiatives. Nutanix customers benefit from advanced web-scale technologies without sacrificing project velocity, data management or the freedom to choose the right hypervisor for their organization.

This patent demonstrates the company’s commitment to advancing the state of the art in data center technology and to simplifying how enterprise IT infrastructures are designed, built and managed. Specifically, it details how a system of distributed nodes (servers) provides high-performance shared storage to virtual machines (VMs) by utilizing a “Controller VM” that runs on each node. The Controller VMs aggregate the local storage resources across all servers, including direct-attached flash, and delivers a shared pool of data storage that is functionally equivalent to a SAN or NAS array – but with greater scalability and higher performance. The architecture natively delivers enterprise-class storage capabilities, such as snapshots, clones, compression and deduplication, to run any production workload.

“Nutanix has pioneered a radically simpler and more scalable storage architecture for all virtualized environments,” said Dr. Willy Zwaenepoel, professor and former dean at Ecole Polytechnique Federale de Lausanne (EPFL) in Switzerland. “This patent recognizes their technical vision and leadership while raising the stakes on other companies in the web-scale market.”

The Nutanix architecture also provides a unified data fabric for building hybrid clouds. Enterprise IT managers can leverage public cloud resources on-demand, while retaining the control and security of their private cloud infrastructure. This flexibility is made possible by a 100 percent software-driven architecture and implemented independent of any specific virtualization technology or vendor.

In addition to publicly disclosing important intellectual property, the patent also details Nutanix’s vision for how data centers will evolve in the cloud era. It is this powerful vision, captured nearly three years ago, which has driven the company’s leadership in the fast-growing web-scale and converged infrastructure markets. Highlights include:

● The modern data center must “…virtualize all storage hardware as one global resource pool that is high in reliability, availability and performance…” to eliminate islands of storage that are difficult to manage, and nearly impossible to scale.
● Unlike traditional network-based storage, the Nutanix architecture “…permits local storage that is within or directly attached to the server and/or appliance to be managed as part of the storage pool.” This enables enterprises to fully leverage server-attached flash technology, and avoid the end-to-end latency incurred when flash is added to centralized, network-based storage systems.
● Datacenter scalability will be driven by a “…massively-parallel storage architecture that scales as and when hypervisor hosts are added…” Such massive scalability can only be achieved if the storage infrastructure can be scaled independent of any hypervisor constraints.
● When storage resources are virtualized, they should be able to “…used in conjunction with any hypervisor from any virtualization vendor.”

“While virtualization has brought impressive flexibility to both applications and server infrastructures, it has also outpaced the capabilities of popular storage products that have dominated in the previous IT generation,” said Brian Bryne, Principal Engineer at Nutanix. “Nutanix’s research and development teams are re-imagining what storage architectures need to look like to enable the next generation of datacenters.”

Other Resources
Read a summary of the patent:


The Basics: Snapshots

In my IT career at work I have never had shared storage that didn’t have snapshots integrated. They were considered fundamental to me if I was going to put all my eggs into one basket. Performance, ease of use and reliability were all deciding factors on selecting a solution. There are lots of different solutions on the market today and some are really good but it’s hard to get all three deciding factors to line up if snapshots were not integrated from the ground up.


It’s great to have a feature but if the performance impacts of using it are such that it impacts the environment then what’s the point? Snapshots are used for a point in time copy of data so that you can roll back incase of corruption, file recovery or part of a larger business continuity plan that makes use of replication.

Snapshots should be able to be used on running applications without causing any performance impacts. Nutanix being per VM defiantly helps in this regard. If you have a LUN with 100 server workloads and take a hardware-based snapshot then you are taking a snapshot of every one of those workloads. Cache and metadata resources are wasted, as all the workloads now have to be tracked and maintained due to the snapshot process.

More and snapshots under stress video here.

Nutanix will be releasing a Tech Note on Snapshots as well in the next week or two.


W5 of Using VMware View Planner: Complete Setup

by Tony Holland Senior SE Nutanix @tonyholland00

Have you ever wanted to test a VDI workload to see what it looks like? If you have, then I would highly recommend looking at VMware View Planner.

I have been working with VMware View Planner since its inception with View Planner 1.0. It was a very primitive tool at the time, but having a tool to simulate what an actual VDI user typically does throughout the day is a very hard thing to come by. Who doesn’t have hundreds or thousands of users at your beck and call to test with? VMware View Planner provides exactly that. The primary users of VDI workload generation tools are hardware vendors. Hardware vendors do not have 100-1000’s of users to test with so they are forced into using synthetic tools to simulate the workload. The workload generated will not match your companies user workload, but it does provide us (Nutanix) and all other hardware players the exact same test to run over and over to find the breaking points / bottlenecks in our solutions. It is not a sizing tool!!

To acquire VMware View Planner you need to talk with a VMware partner. The download for the version of VMware View Planner 3.0 is only available though VMware Partner Central.

One key point that many IT administrators don’t understand about View Planner is that it should not be used as a sizing guideline for how many VDI users you can run on existing or new hardware. View Planner does a great job of simulating the workload of a VDI user but I have yet to see a single company that has a VDI image / template exactly like the one used by View Planner. I also have never seen a company that all users do the exact same thing for hours on end. If you want to size how many actual users you will be able to run, you should use a real user. Pick a handful of people to deploy a VDI desktop to and run a very small pilot. Use your corporate image/template to build the VDI desktops and have actual users work off those VDI desktops. Collect Read-Write IOPS, CPU and Memory data used per user during the pilot and extrapolate that data to figure out the correct amount of users your environment can handle. Don’t forget about N+1. View Planner should be utilized to simulate a synthetic VDI workload that can help find the bottlenecks in your hardware.

Another great thing about View Planner is that you can use it with other VDI technologies not just VMware View. If you follow my directions below you can utilize any VDI servicing apps such as XenDesktop or VMware View to create the desktop pools. I love this because I can test all the features, for example…VMware Storage accelerator, VMware VCAI, Citrix MC or PVS. The only things required for VMware View Planner are the ESXi hypervisor and vCenter. A simple Powershell script could also spin up the required number of desktops to do a View Planner run.

While View Planner does a nice job simulating VDI workloads, it does not take breaks. Think about your office, how often do you see people flirting with co-workers, chit-chatting on phone, going on a smoke breaks and impromptu meetings. With View Planner testing 100 or 1000 desktops in those 3-5 hours while you run the 5 iterations, those 100-1000 VDI desktops will be working/running 100% of the time – no potty breaks. This phenomenon of 100% of your users working simultaneously, not only will never happen but it also will cause the user density number to be skewed. Planning for 80% concurrency is probably more realistic.

Why use VMware View Planner in local only mode? To me the local only mode is very simple, it uses the least amount of infrastructure, resources, and gets IT administrators the data that they are yearning for. Again, it is not a sizing tool, but it is a great load (IOPS and CPU/Memory) generation tool. Utilizing remote mode you are testing the PCoIP protocol. PCoIP typically is not the issue when dealing in a LAN environment IOPS, CPU and memory are. So in my opinion adding extra complexity and hardware to have virtual client desktops launching a bunch of View sessions is unnecessary to get the data that most IT admin’s are looking for.

If View Planner looks like a tool that would be beneficial to you here are some prerequisites :

• View Planner 3.0 (Talk to your VMware partner)
• Microsoft Office with Volume Licensing 2007 or 2010 with no service packs and only (32-bit)
• Microsoft Internet Explorer version 8.x or 9.x.
• Mozilla Firefox version 3.6.x, version 4.0.x, version 5.0.x, version 6.0.x, or version 7.0.x
• Adobe Reader version 9.x or version 10.x (also called Adobe Reader X) on each desktop virtual machine.
You will find all of the instructions on how to install the applications in the View Planner User Guide page 78
• ESXi hosts 5.0, 5.1 or 5.5 to run View Planner appliance and the VDI Desktop VM’s
• VMware vCenter

Steps to setup View Planner

• Download the View Planner OVF from VMware Partner Central
• Download Planner Documentation from Partner Central
• Deploy OVF of View Planner on an ESXi host… Follow View Planner User Guide p.35
• Login to View Planner appliance web page (Username: root Password: abc123) and configure your vCenter IP, and credentials and
vCenter VDI Cluster Datacenter… Follow View Planner User Guide p.38
• Create a new VDI Desktop template Windows XP, 7 or 8 VM named “GoldTemplate”…Follow the View Planner Users Guide p.65
• Install template desktop OS using a VL MAK or KMS key KMS is preferred (XP, Windows 7 or 8)… Follow the View Planner Users Guide p.68 instructions for “Installing Applications in Desktop Virtual Machine”

Complete View Planner User Guide Documentation Sections
• Create the Desktop Template Virtual Machines…p. 65
• Install Microsoft Windows in the Template Virtual Machine…p. 68
• Install the VMware Tools Suite in Microsoft Windows…p. 72
• Add a Data Disk to the Desktop Virtual Machine…p. 75
• Go into Computer Management and Manage the local user accounts. You will need to enable the Administrator account and assign the password of Q1w2e3r4!
• Reboot computer and login as Administrator and password Q1w2e3r4!
• Continue to follow the View Planner guide on installing applications in Desktop Virtual Machine…. p. 78

Complete View Planner Documentation Sections
• Install Applications in the Desktop Template Virtual Machine…. p. 78
• Enable Pseudo Perfcounter for the Virtual Machine…. p. 79
• Install Windows Updates…. p. 80
• Add the Desktop Template Virtual Machine to the Dedicated Domain…. p. 82
• Install the View Agent…. p. 85
• Take a Snapshot of the Virtual Machine…. p. 85
• Download and Install the View Planner Workload Support File…. p. 86
• Shutdown

Now you can start to follow my directions as View Planner documentation and I go about things a little differently.

• Startup your View Planner Desktop template it should auto -login and you should notice a cmd window bat file running in the background.


• Browse C:\ drive of the View Planner Desktop template. Notice the IP.txt file. The only line needed is the harnessIP=.
** – If you do not see IP.txt your installation of the View Planner desktop files did not complete successfully. Manually create the file if need be, but their could be
other problems with the template install. I would attempt to uninstall and reinstall the View Planner desktop files. If all else fails go back to snapshot taken before this step.


Pre View Planner Template Checkup
• Check to make sure that the Windows Firewall is disabled
• Startup all apps, Word, Excel, Powerpoint, Adobe, Firefox, IE and make sure that no extra windows pop up when the app is running.
• Shutdown the GoldTemplate VM.
• Login to View Planner Appliance web page (username: root password: abc123)
• Under Run & Reports tab click on New under Run Profiles
• Fill out Profile Name: Test-1VM
• Number of VM’s: 1
• Desktop Name Prefix: GoldTemplate
• RampupTime:5
• Test Type: Local
• AD Group Settings: Name: Test | Workload Profile: Choose the StandardBenchmarkProfile_1i | % VMs: 100
• Click Save
• Login to the View Planner Appliance via SSH via Putty or a SSH client (username: root password: vmware)
• Change directory to /ViewPlanner
• Tail -f viewplanner.log
• Go back to the View Planner Appliance webpage
• Under Run & Reports tab click on the run profiles and click on the newly created Test-1VM
• Click on Run
• This will start the View Planner process on the VM call GoldTemplate
• Monitor the viewplanner.log via the SSH session. You should see View Planner scan all VM’s to find a match to “GoldTemplate”


• View Planner appliance will tell vCenter to start the VM “GoldTemplate” and View Planner will wait for the guest to register or (check in) with the View Planner appliance.
• Open the VM Console view of “GoldTemplate” from vCenter and watch View Planner startup and run the applications.


• Make sure that all apps open correctly and that no extra pop up windows appear.
• After around 20-30 minutes you will see that the workload was successfully uploaded to View Planner
• Look at the viewplanner.log screen on the SSH session. You should see a message indicating that the test was complete.
** Couple of things to watch for here…
If you used Office x64 View Planner will stall on MS Word and not work
If you don’t have the correct HarnessIP in the IP.txt file you will never connect to the View Planner appliance and the workload will not start.
Windows Firewall will also get in the way. Be sure that firewall is disabled after you added the computer to the domain.


• Properly Shutdown GoldTemplate VM
• Create a final snapshot of the GoldTemplate VM called TestedViewPlannerSetup (Use this snapshot with VMware View Linked Clones)
• You can also now delete the snapshot created before as you have now validated that your desktop is ready to go!

Creating Desktops via VMware View 5.3

I utilize VMware View to create the desktops for my View Planner runs. You can use whatever technology you prefer when creating these pools. Even Xendeskop will work!!! In this use case I am going to create non-persistent Dedicated pool with VMware View Linked Clones.

• In VMware View add a new Pool
• Type “Automated Pool”
• User assignment “Dedicated”
• View Composer Linked Clones
• Pool ID: Pool1
• Pool Settings: I leave the defaults PCoIP will not be used so these settings are not really relevant
• Provisioning Settings: Enable Provisioning; Disable Stop provisioning on error
• Virtual Machine Naming: Use Naming Pattern: VDI-P1-
• Pool Sizing: X amount of desktops you want to create
• Provisioning Timing: Provision All Desktops up front
• View Composer Disks: Redirect disks if you would like I typically choose not to for this test
• Storage Optimization: For Nutanix you should never separate Replica and OS disk other technologies you might want to choose this option
• vCenter Settings: Choose your Parent VM: GoldTemplate
Choose Snapshot: TestedViewPlannerSetup
VM Folder: Your Choice
Host or Cluster: Hosts running Guests
Resource Pool: Your Choice
Datastore: Choose Datastore to provision Pool desktops onto. Nutanix should typically only have one Datastore.
• Advanced Storage Settings
• Use View Storage Accelerator: Optional (Adds provisioning time)
• Use NFS Snapshots: Optional again speeds up provisioning
• Click Finish to create the pool!!

• Once provisioning has completed and all desktops are fully provisioned clicking on desktops in VMware View should like out all of the newly created VM’s. All desktops should have a status of “Provisioned” and a DNS name.


• After all desktops are successfully provisioned and DNS assigned Disable the VMware Pool.
By disabling the pool you remove VMware View from controlling the desktops. If you leave the pool enabled and shutdown all VM’s to prepare for the View Planner run, View will automatically power on the VM’s which in production is a great thing, but in this use case having vCenter control all of the power on and power off’s. From this stage forward we will use View Planner and vCenter to power on and off the desktop’s. Thanks for doing all that work View but we are done with you for now!

Prepare Desktops for View Planner Run

• Using vCenter go to the cluster object and click on the Virtual Machine tab. Sort VM’s by name. You should be able to highlight all powered on VM’s and select the Shut Down Guest OS option. This will cleanly shut down all of your desktop VM’s to prepare for View Planner run. I typically create a few powershell scripts that will do this by Name if you are working with more than 100 or so guests.

• Login to the View Planner appliance via SSH via Putty or SSH client of choice.

• Change directoy to ViewPlanner
• Cd /ViewPlanner
• Edit the adminops.cfg file
• vi adminops.cfg
• Press the i key for insert mode.


• I typically change this to 32 or a lesser number as this can create a fairly large bootstorm. If only a few hundred users you can probably leave at 64. This is also a fun parameter to play with to see how a boot storm can effect the environment.
• Edit to 32 or whatever value you would like to set
• Press esc key
• :wq! (this will write changes to the adminops.cfg file)

• Reboot View Planner appliance VM

Run a View Planner Test

• Using vCenter perform a proper shutdown of all VDI guests that you want to run View Planner on.
• Login to the web console of the View Planner Appliance
• Under the Run & Reports Tab create a New Run Profile.


• Fill out Profile Name: Production-Run
• Number of VM’s: 100 or however many VM’s you want to test
• Desktop Name Prefix: VDI-P1- (use the name scheme that you used to create your VDI Pool in View)
• Ramp up Time:(will populate automatically based on # of VM’s)
• Test Type: Local
• AD Group Settings: Name: Test | Workload Profile: Choose the StandardBenchmarkProfile_5i | % VMs: 100 click Add Group
• Save
• Login via SSH to View Planner Appliance (username: root password: vmware)
• Change directory to /ViewPlanner
• tail -f viewplanner.log
• Switch back to the VMware View Planner webconsole
• Select you newly created Run Profile “Production-Run”
• Click on Run

• Notice vCenter starting to power on all of your desktopsin the viewplanner.log. This is a great place to watch what is exactly happening with the View Planner run.


• After all desktops are powered on and all checked in View Planner will attempt to resolve IP Address for each desktop via VMware tools.


• After successful IP address resolution View Planner will take a moment to rest(Ramp Up Time) before it starts to tell the VM’s to begin the workload.


• The workload will then be started on each desktop machine. You should see CPU and memory usage start to increase on the ESXi host(s).

The test will now run though 5 iterations. Which means it will do 5 loops of the test. If you want more iterations to see how your storage handles it simply adjust that in the View Planner run profile. I have set iterations to 5000 so it would start and run for days to see how everything handled it.

This is fun time for an IT administrator. Once the test is started utilize ESXTOP take a look at IOPS (cmds) and latency. Look in vCenter at CPU and RAM consumption. Look at Datastore latency from vCenter……It is like Christmas with data.




• Once CPU, memory and IOPS drop off you can make a safe guess that the View Planner test is completed.
• Checkout the viewplanner.log via the SSH
• Also have a look in the Web Console. You should see the Run Successful


• Now that you have completed a successful View Planner run you will have all sorts of great information.
• I typically will go and shutdown all of my guests as a first step after confirmation that the test was a success. Even if the test was not a success you will want to properly shutdown the guest VM’s to get them ready for another run.
• If you want to test something thing different, delete the Pool in View or simply recompose the desktops. Just remember to disable the pool after you finish the creation as View will continually startup VM’s.

I also wanted to give a shout out to Banit Agrawal from VMware who has helped me out with View Planner from the 1.0 days. Thank you for all of the assistance. I hope this post will help a few others learn from some of the mistakes and stumbles that I had during my time working with View Planner.

Thank you for reading, I hope that this will help understand the use cases for and against synthetic workload testing. If you have questions please don’t hesitate to reach out over twitter. @tonyholland00


Scale-Out Storage – In the Hypervisor Kernel or in a VM?

A new tech note from Nutanix discussing architectural considerations with implementing a converged, scale-­‐out storage fabric that run across a cluster of nodes. This paper focuses on high availability and resiliency for virtualizing business critical applications. The paper covers running storage services embedded in the hypervisor kernel and as virtual machine in the user space.

Scale-Out Storage – In the Hypervisor Kernel or in a VM?



Nutanix Whiteboard: Technical Deep Dive with @LukasLundell & the @otherscottlowe’s Scott D. Lowe and Nutanix’ Director of Solutions Engineering Lukas Lundell dig into how Nutanix’ hyperconverged infrastructure architecture achieves massive scale in a small package.