HYCU v1.5 – Nutanix AHV Backup gets new features

HYCU v1.5 has been released by Comtrade.

The biggest one for me is ABS support! Know you can use cheap and deep storage and drive all of the storage controllers. Remember that ADS works with ABS so its a great solution.

The following new features and enhancements are available with Hycu version 1.5.0:

Backup window
It is now possible to set up a time frame when your backup jobs are allowed to run (a backup window). For example, this allows you to schedule your backup jobs to run on non-production hours to reduce loads during peak hours.

Concurrent backups
You can now specify the maximum number of concurrent backup jobs per target. By doing so, you can reduce the duration of backups and the amount of queued backup jobs.

Faster recovery
Hycu enables you to keep snapshots from multiple backups on the Nutanix cluster. Keeping multiple snapshots allows you to recover a virtual machine or an application quickly, reducing downtime. If your know Commvault Intelisnap, very similar benefits

iSCSI backup target
A new type of backup target for storing the protected data is available—iSCSI, which also makes it possible for you to use a Nutanix volume group as a backup target. You can use the iSCSI backup target for backing up the Hycu backup controller as well.

Improved backup and restore performance
Hycu now utilizes Nutanix volume groups for backing up and restoring data, taking advantage of the load balancing feature offered by Nutanix. Therefore, the new version of Hycu can distribute the workload between several nodes, which results in increased
performance of your backup and restore operations, and reduced I/O load on the Nutanix cluster and containers.

Support for AWS S3-compatible storage
Hycu enables you to store your protected data to AWS S3-compatible storage.

Shared location for restoring individual files
You can now restore individual files to a shared location so that recovered data can be accessed from multiple systems.

Support for Active Directory applications
In addition to SQL Server applications, Hycu can now also detect and protect Active Directory applications running on virtual machines. You can view all the discovered applications in the Applications panel.

Expiring backups manually

If there is a restore point that you do not want to use for a data restore anymore, you can mark it as expired. Expired backups are removed from the backup target within the next 24 hours, resulting in more free storage space and helping you to keep your Hycu system clean.

Support for Nutanix API changes

Hycu supports Nutanix API changes introduced with AOS


AOS 5.1.2 Security Updates

A long list of updates, one-click upgrade yourself to safety.

CVE-2017-1000364 kernel: heap or stack gap jumping occurs through unbounded stack allocations ( Stack Guard or Stack Clash)

CVE-2017-1000366 glibc: heap or stack gap jumping occurs through unbounded stack allocations (Stack Guard or Stack Clash)

CVE-2017-2628 curl: negotiate not treated as connection-oriented

CVE-2017-3509 OpenJDK: improper re-use of NTLM authenticated connections (Networking, 8163520)

CVE-2017-3511 OpenJDK: untrusted extension directories search path in Launcher (JCE, 8163528)

CVE-2017-3526 OpenJDK: incomplete XML parse tree size enforcement (JAXP, 8169011)

CVE-2017-3533 OpenJDK: newline injection in the FTP client (Networking, 8170222)

CVE-2017-3539 OpenJDK: MD5 allowed for jar verification (Security, 8171121)

CVE-2017-3544 OpenJDK: newline injection in the SMTP client (Networking, 8171533)

CVE-2016-0736 httpd: Padding Oracle in Apache mod_session_crypto

CVE-2016-1546 httpd: mod_http2 denial-of-service by thread starvation

CVE-2016-2161 httpd: DoS vulnerability in mod_auth_digest

CVE-2016-8740 httpd: Incomplete handling of LimitRequestFields directive in mod_http2

CVE-2016-8743 httpd: Apache HTTP Request Parsing Whitespace Defects

CVE-2017-8779 rpcbind, libtirpc, libntirpc: Memory leak when failing to parse XDR strings or bytearrays

CVE-2017-3139 bind: assertion failure in DNSSEC validation

CVE-2017-7502 nss: Null pointer dereference when handling empty SSLv2 messages

CVE-2017-1000367 sudo: Privilege escalation in via improper get_process_ttyname() parsing

CVE-2016-8610 SSL/TLS: Malformed plain-text ALERT packets could cause remote DoS

CVE-2017-5335 gnutls: Out of memory while parsing crafted OpenPGP certificate

CVE-2017-5336 gnutls: Stack overflow in cdk_pk_get_keyid

CVE-2017-5337 gnutls: Heap read overflow in read-packet.c

CVE-2017-1000366 glibc: heap/stack gap jumping via unbounded stack allocations

CVE-2017-1000368 sudo: Privilege escalation via improper get_process_ttyname() parsing

CVE-2017-3142 bind: An error in TSIG authentication can permit unauthorized zone transfers

CVE-2017-3143 bind: An error in TSIG authentication can permit unauthorized dynamic updates

CVE-2017-10053 OpenJDK: reading of unprocessed image data in JPEGImageReader (2D, 8169209)

CVE-2017-10067 OpenJDK: JAR verifier incorrect handling of missing digest (Security, 8169392)

CVE-2017-10074 OpenJDK: integer overflows in range check loop predicates (Hotspot, 8173770)

CVE-2017-10078 OpenJDK: Nashorn incompletely blocking access to Java APIs (Scripting, 8171539)

CVE-2017-10081 OpenJDK: incorrect bracket processing in function signature handling (Hotspot, 8170966)

CVE-2017-10087 OpenJDK: insufficient access control checks in ThreadPoolExecutor (Libraries, 8172204)

CVE-2017-10089 OpenJDK: insufficient access control checks in ServiceRegistry (ImageIO, 8172461)

CVE-2017-10090 OpenJDK: insufficient access control checks in AsynchronousChannelGroupImpl (8172465, Libraries)

CVE-2017-10096 OpenJDK: insufficient access control checks in XML transformations (JAXP, 8172469)

CVE-2017-10101 OpenJDK: unrestricted access to (JAXP, 8173286)

CVE-2017-10102 OpenJDK: incorrect handling of references in DGC (RMI, 8163958)

CVE-2017-10107 OpenJDK: insufficient access control checks in ActivationID (RMI, 8173697)

CVE-2017-10108 OpenJDK: unbounded memory allocation in BasicAttribute deserialization (Serialization, 8174105)

CVE-2017-10109 OpenJDK: unbounded memory allocation in CodeSource deserialization (Serialization, 8174113)

CVE-2017-10110 OpenJDK: insufficient access control checks in ImageWatched (AWT, 8174098)

CVE-2017-10111 OpenJDK: incorrect range checks in LambdaFormEditor (Libraries, 8184185)

CVE-2017-10115 OpenJDK: DSA implementation timing attack (JCE, 8175106)

CVE-2017-10116 OpenJDK: LDAPCertStore following referrals to non-LDAP URLs (Security, 8176067)

CVE-2017-10135 OpenJDK: PKCS#8 implementation timing attack (JCE, 8176760)

CVE-2017-10193 OpenJDK: incorrect key size constraint check (Security, 8179101)

CVE-2017-10198 OpenJDK: incorrect enforcement of certificate path restrictions (Security, 8179998)


Acropolis Dynamic Scheduler (ADS) for AHV (Compute + Memory + Storage)

The Acropolis Dynamic Scheduler (ADS) ensures that compute (CPU and RAM) and storage resources are available for VMs and volume groups (VGs) in the Nutanix cluster. ADS, enabled by default, uses real-time statistics to determine:

Initial placement of VMs and VGs, specifically which AHV host runs a particular VM at power-on or a particular VG after creation.

Required runtime optimizations, including moving particular VMs and VGs to other AHV hosts to give all workloads the best possible access to resources.

If a problem is detected, a migration plan is created and executed thereby eliminating hotspots in the cluster by migrating VMs from one host to another. This feature only detects the contentions that are currently in progress. You can monitor these tasks from the Task dashboard of the Prism Web console. You can click the VM link to view the migration information, which includes the migration path (to the destination AHV host).

The Acropolis block services feature uses the ADS feature for balancing sessions of the externally visible iSCSI targets.


Nutanix AFS and SMB 3.0

After you upgrade using the AFS (Acropolis File Services)2.2 bits you will have to manually change the max allowed protocol. This will be fixed with AFS 2.2.1 but here are the steps to get you going.

The following commands listed below will set the max protocol to the proper version

scli smbcli get --section global --param "server max protocol"
server max protocol = SMB2

scli smbcli set --section global --param "server max protocol" --value SMB3_00
smb.conf update is successful

scli smbcli get --section global --param "server max protocol"
server max protocol = SMB3_00

Also note if you want to run FSlogix and need an AFS share to have a block size of 512 bytes, this can be done. Default is 1024.


Windows Get Some Love with #Docker EE 17.06

With the new release of Docker 17.06 EE Windows containers gets lots of added features. First up is the ability to run Windows and Linux worker nodes in the same same cluster. This is great because you have centralized security and logging across your whole environment. Your .NET and Java teams can live in peace to consolidate your infrastructure instead of spinning of separate environments.

Continuously scanning for vulnerabilities in Windows images was added if your have Advanced EE license. Not only does it scan images it will also alert when new vulnerabilities are found in existing images.

Bringing everything together you can use the same overlay networks to connect your application in the case of SQL server and web servers running on Linux. Your developers can create a single compose file covering both SQL and web severs.

Other New Windows related features in Docker 17.06:

Windows Server 2016 support
Windows 10586 is marked as deprecated; it will not be supported going forward in stable releases
Integration with Docker Cloud, with the ability to control remote Swarms from the local command line interface (CLI) and view your repositories
Unified login between the Docker CLI and Docker Hub, Docker Cloud.
Sharing a drive can be done on demand, the first time a mount is requested
Add an experimental DNS name for the host:
Support for client (i.e. “login”) certificates for authenticating registry access (fixes docker/for-win#569)
New installer experience


Nutanix Native File Services (AFS) Now Supports AV Offload Scanning

With AFS 2.2 and AOS 5.1.2 now supports ICAP(Internet Content Adaptation Protocol), which is supported by a wide range of security vendors and products, is a standard protocol that allows file and web servers to be integrated with security products. Nutanix chose this method to give customers the ability to choose the antivirus solution that works best for their specific environment.

Following is the workflow for an ICAP-supported antivirus solution:
An SMB client submits a request to open or close a file.
The file server determines if the file needs to be scanned, based on the metadata and virus scan policies. If a scan is needed, the file server sends the file to the ICAP server and issues a scan request.
The ICAP server scans the file and reports the scan results back to the file server.
The file server takes an action based on the scan results:
If the file is infected, the file server quarantines it and returns an “access denied” message to the SMB client.
If the file is clean, it returns the file handle to the SMB client.

The ICAP service runs on each AFS file server and can interact with more than one ICAP server in parallel to support horizontal scale-out of the antivirus server. We recommend configuring two or more ICAP servers for production. The scale-out nature of AFS and one-click optimization greatly mitigates any antivirus scanning performance overhead. If the scanning affects AFS file server VM performance, one-click optimization recommends increasing the virtual CPU resources or scaling out the file server VMs. This feature also allows both the ICAP server and AFS to scale out, ensuring fast responses from the customer’s antivirus vendor.

AFS sets scanning defaults across the entire file server, but they are disabled by default per share when you enable file scanning. You can enable scan on write and scan on read. Scan on write begins when the file is closed, and scan on read occurs when the file is opened. You can also exclude certain file types and files over a certain size. Share scan polices can override any defaults set for the file server.

For each ICAP server, we spin up no more than 10 parallel connections per FSVM and randomly dispatch the file scanning among all the ICAP servers. With heavier workloads, which may encounter many scan requests and use all connections, the scan servers with more processing power scan more files. As soon as the current scan finishes, the next file is picked up from the queue, which keeps the number of active connections at 10.

Once AFS quarantines a file, the admin can rescan, unquarantine, or delete the file. Quarantined files can be searched if it is necessary to restore a file quickly.
If your antivirus vendor doesn’t support ICAP, you can scan the shares by installing an antivirus agent onto a Windows machine and then mounting all the shares from the file server. This approach allows you to schedule scans during periods of low usage. At the desktop or client level, you can set your antivirus solution to scan on write or scan only when files are modified. You can configure high-security environments to scan inline for both reads and writes.


Multi-stage build support in #Docker EE 17.06

New in Docker EE 17.06 is the ability to have multi-stage builds. This is important because you can now just grab the files(artifacts) you need for the next stage of your build and keep your builds small which leads to faster build times. This change allows you to have mutiple from arguments in your docker file.

Devs have optional give a name the build stage. Then afterward this name can be used in COPY –from=name src dest and FROM name. If a build stage is defined with that name it takes precedence in these commands, if it is not found, an image with that name is attempted to be used instead.

FROM node AS test-env
ADD ./ /app
RUN npm install
RUN npm run build

FROM nginx AS prod
COPY --from=test-env /app /var/www/html

You can run subsets of the dockerfile to get more use of out your work. If you wanted to only run the test-env section you can add a target to the docker build command with –target test-env


VMworld attendees get to the Docker booth to save money & time like Visa.

The Docker booth is right beside the Nutanix booth at VMworld this year so I have seen lots of people there but not 23,000 but there should be. Docker had been apart of all the announcements if you realized it our not. Lots of talk about Google with Kubernetes. Kubernetes still requires Docker as the container engine so whether it’s Swarm or Kubernetes you’re going to be using Docker. If you want Enterprise support Docker is both you want to be visiting and learning what they can do to develop better end to end software while saving you money.

With Docker EE has been in production at Visa for over 6 months and is seeing improvements in a number of ways:

Provisioning time: Visa can now provision in seconds rather than days even while more application teams join the effort. They can also deliver just-in-time infrastructure across multiple datacenters around the world with a standardized format that works across their diverse set of applications.
Patching & maintenance: With Docker, Visa can simply redeploy an application with a new image. This also allows Visa to respond quickly to new threats as they can deploy patches across their entire environment at one time.
Tech Refresh: Once applications are containerized with Docker, developers do not have to worry about the underlying infrastructure; the infrastructure is invisible.
Multi-tenancy: Docker containers provides both space and time division multiplexing by allowing Visa to provision and deprovision microservices quickly as needed. This allows them to strategically place new services into the available infrastructure which has allowed the team to support 10x the scale they could previously.

Visa moved a VM-based environment to containers running on bare metal and saved the time to provision and decommissioned its first containerized app by 50%.By saving time and money on the existing infrastructure and applications, organizations can reinvest the savings — both the time and money — in transforming the business.

BTW Nutanix can do bare-metal or run AHV to provide great experience for containers with our own Docker Volume plugin.


#Nutanix #AFS – Cannot access folders/files created with a dot “.”

You may experience issues with third party application when accessing folders/files with a dot “.” prefix in the name such as “.profile” or “.build”.

One of the error messages may look like the one below.

Error mesessage
The “.profile” has different permissions.

This is pertaining folders like “.profile” or “.build”. If can create the folders and see them, but when trying to access them with programs, it throws access errors. The “dot files” have different permissions.

There is a configuration option called “hide dot files” in AFS. This “hide dot files” configuration option is enabled by default in AFS.

You can run the following command on one of the FSVMs to turn off the “hide dot files” option globally.

scli smbcli global "hide dot files" No

Example of the command run successfully below.
nutanix@NTNX-x-x-x-x-A-FSVM:~$ scli smbcli global "hide dot files" no
No existing config 'hide dot files' in smb.conf. Creating a new entry now..!
smb.conf update is successful


Nutanix Docker volume plugin updated

Last week Nutanix published an update to the Nutanix docker volume plugin with support ubuntu, docker datacenter compatibility and improved logging among other improvements.

Supported scopes for the plugin are global and local. Any other value in Scope will be ignored, and local is used. Scope allows cluster managers to handle the volume in different ways. For instance, a scope of global, signals to the cluster manager that it only needs to create the volume once instead of on each Docker host.