Moby Project Summit Notes

The Moby Project was born out of the containerd / Docker Internals Summit

For components to be successful they need to be successful everywhere. which lead into SwarmKit being mentioned as not being successful because no other ecosystem was using it. Seems to be a strong commitment to make everything into a component out in the open.

Docker wants to be seen as a open source leader thru doing the hard work to support components.

All open-source development will be under the Moby project.

Upstream = components
Moby = Staging area for products to move on like containerd is in the CNF project.
– Heart of open-source activities, a place to integrate components
– Docker remains docker
– Docker is built with Moby
– You use Moby to build things like Docker
– Solomon mentions “1000 of smart people could disagree on what to do”, Docker represents it’s opinion. It’s a lot easier to agree on low level functions because there is few ways to do them.
– Moby will end up as go libraries in Docker but that will go away.

Moby is connected to Docker but it’s not Docker. Name inspired from the Fedora project.

Moby is a trade off to get it out in the open early versus completeness

GitHub should be used a support forum.

InfraKit is a toolkit for creating and managing declarative, self-healing infrastructure. It breaks infrastructure automation down into simple, pluggable components. These components work together to actively ensure the infrastructure state matches the user’s specifications. Although InfraKit emphasizes primitives for building self-healing infrastructure, it also can be used passively like conventional tools

LinuxKit, a toolkit for building custom minimal, immutable Linux distributions.

– Secure defaults without compromising usability
– Everything is replaceable and customisable
– Immutable infrastructure applied to building Linux distributions
– Completely stateless, but persistent storage can be attached
– Easy tooling, with easy iteration
– Built with containers, for running containers
– Designed for building and running clustered applications, including but not limited to container orchestration such as Docker or Kubernetes
– Designed from the experience of building Docker Editions, but redesigned as a general-purpose toolkit

No master plans to change away for go.

Breaking out the monolithic engine API will mostly likley done with gRPC. gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services.

SwarmKit Update
SwarmKit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

New Features

– Topology-Aware Scheduling
– Secrets
– Service Rollbacks
– Service Logs
– HA scheduling
– Encrypted Raft Store
– Health-Aware Orchestration
– Synchronous CLI
What is Next?
– Direct integration of containerd into SwarmKit by passes the need for Docker Engine
– Config Management to attach configuration to services
– Swarm Events to watch for state changes and gRPC Watch API
– Create a generic runtime to support new run times without changing SwarmKit
– Instrumentation

LibNetwork Update
– Quality More visibility, motioning and troubleshooting.
– Local-scoped network plugins in Swarm-mode
– Integration with containerd


Docker Datacenter: Usability For Better Security.

With the new release of Docker Datacenter 2.1 it’s clear the Docker is very serious about the enterprise and providing the tooling that is very easy to use. Docker has made the leap to supporting enterprise applications with its embedded security and ease of use. DCC 2.1 and Docker-engine-cs 1.13 give the additional control needed for operations and development teams to control their own experience.

Docker datacenter continues to build on containers as a service. In the 1.12 release of DDC it enabled agility and portability for continuous integration and started on the journey of protecting the development supply chain throughout the whole lifecycle. The new release of DDC’s focuses on security, specifically secret management.
The previous version of DDC already had wealth of security features
• LDAP/AD integration
• Role based access control for teams
• SS0 and push/pull images with Docker Trusted Registry
• Imaging signing – prevent running a container unless image signed by member of a designated
• Out of the box TLS with easy setup, including cert rotation.

With the DDC 2.1 the march on security is being made successful by allowing both operations and developers to have a usable system without having to lean into security for support and help. The native integration with the management plane allows for end to end container lifecycle management. You also inherit the model that’s independent no matter the infrastructure you’re running on it will work. It can be made to be dynamic and ephemeral like the containers it’s managing. This is why I feel PAAS is dead. With so much choice and security you don’t have to limit yourself where you deploy to, a very similar design decision to Nutanix by enabling choice. Choice gives you access to more developers and the freedom to color outside the lines of the guardrails that a PAAS solution may empose.

Docker Datacenter Secrets Architecture

1) Everything in the store is encrypted, notably that includes all of the data that is stored in the orchestration . With least privlege — only node is distributed to the containers that need them. Since the management mayor is scalable you also get that for your key management as well. Due to the management layer being so easy to set up you don’t have developers embedding secrets in Github to get a quick work around.
2) Containers and the filesystem makes secret only available to only the designated app . Docker expose secrets to the application via a file system that is stored in memory. The same rotation of certificates for the management letter also happens with the certificates for the application. In the diagram above the red service only talks of the red service and the blue service is isolated by itself even though it’s running on the same node as the red service/application.
3) If you decide that you want to integrate with a third-party application like Twitter and be easily done. Your Twitter credentials can be stored in the raft cluster which is your manager nodes. When you go to create the twitter app you give it access to the credentials and even do a “service-update” if you need swap them out without the need to touch every node in your environment.

With a simple interface for both developers and IT operations both have a pain-free way to do their jobs and provide a secure environment. By not creating road blocks and slowing down development or operations teams will get automatic by in.


Demo Time – Nutanix CE and VSA’s

In order to successfully complete your home lab, you’re going to need configure compute (the servers), networking (routers and switches etc.) and storage. For those that are solely interested in studying or testing an individual application, operating system, or the network infrastructure, you should be able to complete this with no more storage than the local hard drive in your PC.

For those who are looking to learn how cloud and data center technologies work as a whole however, you’re going to require some form of dedicated storage. A storage simulator or a Virtual Storage Appliance (VSA) or Nutanix CE is likely to be the best option for this task.

If you’re studying hypervisor technologies you’re going to have to spend on compute hardware as well as any of the network infrastructure devices that are incapable of being virtualized. Unless you have a free flowing money source, you’re most likely going to want to contain the storage costs by using virtualized storage rather than SAN or NAS hardware.

The Flackbox blog has compiled a lengthy and comprehensive list of all the available simulators and VSAs. All of the software is free but may require a customer or partner account through the vendor to be able to download. The login and system requirements for every option are included in the list as well. Thanks to Neil for putting those together.

Nutanix CE can be seen as having high requirements for a home lab but once you factor that management is included it’s not that bad. You can also you a free instance with Ravello.

If you don’t meet the requirement you can always use OpenFiler or StarWind if you have gear at home.

For those looking to mimic their organization’s production environment as closely as possible, choose the VSA or simulator from your vendor.

GUI demos are also included at the bottom of the list. These are not designed or suitable for a lab but are great for those looking to get a feel of a particular vendor’s Storage GUI.


THE WORD FROM GOSTEV – 3rd Party Backups aren’t going away.

First off the Veeam newsletter is great and you should sign up. There was one comment that I found interesting was regarding the need for backups. I’ve always said that while Nutanix has a great integrated backup story sometimes it doesn’t meet all of the requirements needed by a business. Getting it out of the storage vendor’s hands is a wise decision. While Nutanix and every other vendor does rigourous QA the fact remains is that were still human and problems can occur.

Something like this has to happen once in a while so that everyone is reminded that storage snapshots are not backups – not even if you replicate them to a secondary array, like these folks did > HPE storage crash killed Australian Tax Office. You may still remember the same issue with EMC array crash disabling multiple Swedish agencies for 5 days not so long ago. These things just happen, this is why it is extremely important to make real backups by taking the production data out of the storage vendor’s “world” – whether we’re talking about classic storage architectures, or up and coming hyper-converged vendors (one of which have not been shy marketing < 5 min "backup" windows lately).

Food for thought, in the end it will be what meets the needs of your business. AKA Can you live with the pain.


Get Ready for AOS 5.0 – Nutanix

This authentication behavior is changed in AOS 5.0. If you are using Active Directory, you must also assign roles to entities or users, especially before upgrading from a previous AOS version. If you’re not using AD, pass Go and collect $200!

For customers upgrading their clusters to AOS 5.0:

* Customers upgrading their clusters to AOS 5.0 will see a pre-upgrade check warning if user authentication is enabled for the Active Directory (AD) service and role permissions are not assigned to any user. The upgrade process will fail in this case.

Warning - no role mappings

Warning – no role mappings

* The AOS 5.0 Prism service (part of the Prism web console) will not authenticate AD users if role permissions are not configured for those users. This situation effectively locks out existing AD users that previously were allowed to access the Prism 4.x web console and other components such as the Nutanix command line (nCLI).
Add a Role mapping for your AG Groups or Users

Add a Role mapping for your AG Groups or Users

To upgrade successfully in this case and to maintain existing access, assign roles (role permissions) to entities that are allowed access to Prism before attempting to upgrade your cluster.


Integrated Single Node Backup with Nutanix

Integrated backup for remote branch offices and small to medium sized business. Single Node backup is using the NX-1155 which is quotable today . Single Node Backup is apart of AOS 5.0


Serve Files with Enterprise Cloud Agility, Security, and Availability with Acropolis File Services


Nutanix continues on its Enterprise Cloud journey at the .NEXT On-Tour event in Bangkok, Thailand. Today, we are proud to announce that we are planning to support Acropolis File Services (AFS) on our storage only nodes, the NX-6035C-G5. Acropolis File Services provides a simple and scalable solution for hosting user and shared department files across a centralized location with a single namespace. With Acropolis File Services, administrators no longer waste time with manual configuration or need Active Directory and load balancing expertise. If and when released, this will make 6035C-G5 nodes even more versatile, adding to the current capabilities of serving as a backup or replication target and running Acropolis Block Services.

[read more]


Battle Royale: View Composer VS Instant-Clones – Deploy

Horizon 7 added Instant-Clones with the ability to clone a full desktop in 4-5 secs. What is the catch? Not really a catch, but no explanation that it takes a bit of time to prep the desktops. For testing purposes, I decided to clone 100 desktops with View Composer and 100 desktops with Instant Clones.

For these tests I used NX 3460-G4, Win 10, 2 vCPU, 2 GB of RAM

Impact of cloning 100 desktops with View Composer


You can see hypervisor IOPS and disk IOPS. The impact is really shown on what is happening on the backend and CPU used to create the desktops. So roughly 16,000 IOPS to create the desktops with Composer.

Impact of cloning 100 desktops with Instant-Clones

You can see an initial bump in IOPS due to the replica that has to be copied without VAAI. The replica also has to get fingerprinted with does take some time. In my testing it took about eight minutes. The reduction in IOPS is amazing. While you still need performance for running the desktops, you don’t have to worry about provisioning destroying your performance. Disk IOPS was ~ only 1200 IOPS at its peak.

Summary VC vs Instant Clone

Deploy 100 Desktops
View Composer: 5 min
Instant Clone: 14 min —– virtual disk digest – 8.22 min
—– Clone 100 desktops 1.4 min

While the overall process took longer the impact is a lot better with Instant-Clones. With hundreds of desktops Instant-Clones is powerful tool to have in your back pocket. Once Instant-Clones gets GPU support I think they will really take off as the default choice. If you have related questions to performance I encourage you to talk to your Nutanix SE and they can get put you in touch with the Solution and Performance Team at Nutanix.

Related Articles

Tale of Two Lines


Securing the Supply Chain with Nutanix and Docker #dockercon2016

I was watching the below video from DockerCon 2016 and there was lots of striking similarities between what Nutanix and Docker is doing secure working environment for the Enterprise Cloud. There is no sense turning the alarm on for your house and then not locking the doors. You need to close all the gaps for your infrastructure and the applications that live on top of it.

The most interesting part of the session for me was the section on security scanning and gating. Docker has Security Scanning which is available as an add-on to Docker hosted private repositories on both Docker Cloud and Docker Hub. Scans run each time a build pushes a new image to your private repository. They also run when you add a new image or tag. Most scans complete within an hour, however large repositories may take up to 24 hours to scan. The scan traverses each layer of the image, identifies the software components in each layer, and indexes the SHA of each component.
The scan compares the SHA of each component against the Common Vulnerabilities and Exposures (CVE) database. The CVE is a “dictionary” of known information security vulnerabilities. When the CVE database is updated, the service reviews the indexed components for any that match the new vulnerability. If the new vulnerability is detected in an image, the service sends an email alert to the maintainers of the image.

A single component can contain multiple vulnerabilities or exposures and Docker Security Scanning reports on each one. You can click an individual vulnerability report from the scan results and navigate to the specific CVE report data to learn more about it.

On the Nutanix side of the fence all code is scanned with 2 different vulnerability scanners at every step of the development life-cycle. To top that off Nutanix already apply s an intrinsic baseline, and we already monitor and self-heal that baseline with SCMA the Security Configuration Management Automation and leverage the SaltStack framework so that your production systems can Self-Heal from any deviation and are always in compliance. Features like two factor authentication (2FA) and cluster lockdown further enhance the security posture. The cluster-wide setting can forward all logs to a central host as well. All CVEs related to the product are tracked and provide an internal turn around time of 72 hours for critical patches! There is some added time on getting a release cut but it fast and everything is tested as whole instead of a one off change that could have a domino a effect.

When evaluating infrastructure and development environments for a security-conscious environment, it’s imperative to choose one that is built with a security-first approach that continually iterate on patching new threats thereby reducing the attack surface. Docker is doing some great work on this front.


    Nutanix Acropolis File Services – Required 2 Networks

    When configuring Acropolis File Services you may be prompted with the following message:

    “File server creation requires two unique networks to be configured beforehand.”

    The reason is you two managed networks for AFS. I’ve seen this come up a lot lately so I thought I would explain the why. While it may change over time this is the current design.


    The above diagram shows one file server VM running on a node, but you can put multiple file server VMs on a node for multitenancy.

    The file server VM has two network interfaces. The first interface is a static address used for the local file server VM service that talks to the Minerva CVM service running on the Controller VM. The Minerva CVM service uses this information to manage deployment and failover; it also allows control over one-click upgrades and maintenance. Having local awareness from the CVM enables the file server VM to determine if a storage fault has occurred and, if so, if action should be taken to rectify it. The local address also lets the file server VM claim vDisks for failover and failback. The file server VM service sends a heartbeat to its local Minerva CVM service each second, indicating its state and that it’s alive.
    The second network interface on the file server VM, also referred to as the public interface, allows clients to service SMB requests. Based on the resource called, the file server VM determines whether to service the request locally or to use DFS to refer the request to the appropriate file server VM that owns the resource. This second network can be dynamically reassigned to other file server VM’s for high availability.

    If you need help setting up the two managmed networks there is KB article on portal.nutanix.com -> KB3406