Exploration and Practices of Edge Computing: Chapter 2 Build and Installation of Virtualization Equipment

Exploration and Practices of Edge Computing

Chapter 2 Build Image and Install Virtualized Devices

The previous article outlined the direction of cloud-based collaboration and device management. This article will describe how to use VMware's free software to build image and install virtualized devices.

Introduction to vSphere Hypervisor

VMware is the global leading provider of virtualization software, hybrid cloud computing and enterprise user computing solutions, and vSphere is its core flagship product. In April 2020, VMware released vSphere 7.0 to the world, providing more new features, especially the revolutionary Tanzu platform that fully supports Kubernetes.

vSphere is a widely used virtualization platform. Not only in data centers, the core functions required to run on devices have been included in the free product vSphere Hypervisor.

Verified device specifications

Here is the technical specifications of free vSphere Hypervisor to ru on devices, NOT the Hardware Compatibility List (HCL) required to provide commercial product support.

About 20 models of devices from multiple manufacturers that meet at least the following hardware specifications
have been verfied technically to run vSphere Hypervisor.
  • CPU: x86-64 2+ core (1GHz +), support VT-x / AMD-V
  • Memory: 4GB (ECC or non-ECC)
  • Storage: SSD / HDD / Flash 16GB
  • Network: 1Gbps wired NIC

Download related software

Mandetory software

Note: You need to register an account at VMware.com before download the free vSphere Hypervisor and get the serial number. An ISO image is necessary. If you want to customize the image, you must download the offline package too.
  • vSphere Hypervisor

Optional drivers

Note: Because some devices carry Realtek or other NIC chips unsupported officially by vSphere 6.x / 7.0, you need to download them from the community website. Such as:

Optional tools

Note: You may choose either CLI or GUI approach to customize ESXi image. If you want to thoroughly customize the image, it is recommended to leverage vCenter Server.

Customize ESXi image for devices

CLI approach

The steps to install VMware PowerCLI are skipped. 

The following is an example to add a USB NIC driver into ESXi 7.0 image.
  • Add vSphere 7.0 depot
Add-EsxSoftwareDepot. \ VMware-ESXi-7.0.0-15843807-depot.zip
  • Add USB NIC driver depot
Add-EsxSoftwareDepot. \ ESXi700-VMKUSB-NIC-FLING-34491022-component-15873236.zip

  • Obtain the image profile name in ESXi 7.0
Get-EsxImageProfile | ft name
  • Copy a new image profile ESXi7-USB from the image profile ESXi-7.0.0-15843807-no-tools without vmtools
New-EsxImageProfile -CloneProfile ESXi-7.0.0-15843807-no-tools -Name ESXi7-USBNIC -vendor VMware
  • Add USB NIC driver vmkusb-nic-fling to the newly created image profile ESXi7-USB
Add-EsxSoftwarePackage -ImageProfile ESXi7-USBNIC -SoftwarePackage vmkusb-nic-fling --Force
  • Set AcceptanceLevel to VMwareCertified for the new image profile ESXi7-USBNIC (this step is optional)
Set-EsxImageProfile -AcceptanceLevel VMwareCertified -ImageProfile ESXi7-USBNIC
  • Export the new image profile ESXi7-USBNIC as ISO image ESXi7-USBNIC.iso
Export-EsxImageProfile -ImageProfile ESXi7-USBNIC -FilePath ESXi7-USBNIC.iso -ExportToIso --Force

This exported ISO image ESXi7-USBNIC.iso can be used to install ESXi 7.0 on devices after burned to a USB thumb disk.


Note: The Realtek PCIe NIC driver provided by the community does not yet support ESXi 7.0, and you can only customize ESXi 6.7 image with it. The steps are similar to the above example.

GUI approach

You can also leverage Auto Deploy in vCenter Server to customize ESXi image. More benefits are easily choosing to add or remove other integration packages (VIB), further reducing the size of the image, or adding certain Kickstart scripts. 

Here is another example to customize ESXi 7.0 image.

 
  • Add vSphere 7.0 software depot
 
 
  • Add USBNIC software depot
 
 
  • Create new software depot for customized ESXi 7.0 image
  • Create a new image profile, add USB NIC driver vmkusb-nic-fling, delete vmtools, and set  Acceptance Level to VMware Certified (optional) .
 
 
 

Note: In this step, you can delete more integrated packages (VIBs) to compress the size of the image further, such as unused NIC and HBA drivers on the device. If you add PCIe Realtek NIC driver to ESXi 6.7 image, remember to select Community Support for Acceptance Level. 
  •  Export new image
This exported ISO image can be used to install ESXi 7.0 after burned to a USB thumb disk. 

Install ESXi on devices

Connect hardware

Before installing the ESXi image on devices, make sure that the device has sufficient hardware resources, especially the memory listed above (at least 4GB), hard disk (non-eMMC, at least 16GB) and network (wired NIC on 1Gbps). 
Connect the keyboard, monitor, USB thumb disk, network cable, etc. according to the user manual of the device. If that is a headless device, you need to connect the appropriate serial port to a display in debug mode according to the device manual.

BIOS settings

In some devices, even if 4GB memory is configured, the video memory is shared within the main memory. In this case, it is recommended to set the shared video memory to the minimum (such as 32MB) in the BIOS before the official installation.

In addition, make sure that the hardware virtualization instructions in the BIOS, either Intel Virtualization Technology or AMD Secure Virtual Machine, is set to Enabled, and Secure Boot or TPM is set to disabled.

Start the installation

Boot from the “burned” USB flash drive according to the BIOS settings. When you see the following ESXi loading screen, quickly press Shift + O keys and add the kernel parameter ignoreHeadless = TRUE. This is to support Headless mode.
 
After that, ESXi should be able to be installed normally as on the server. After the installation is complete, you will be prompted to restart.

Restart settings

After restarting, remember to quickly press Shift + O keys again when you see the loading screen above, and also add the kernel parameter ignoreHeadless = TRUE. This is because the kernel parameter has not been saved yet, but will be saved in the next step. After this restart, in addition to setting the NIC address (if not DHCP), it is recommended to open the SSH and ESXi Shell services for remote debugging. 
 
After logging into the device with ssh root@192.168.0.10 for example, change and save the kernel parameters as follows.
 
esxcfg-advcfg --set-kernel "TRUE" ignoreHeadless
esxcfg-advcfg --- get-kernel ignoreHeadless

Verify successful installation

Restart the device again, this time there is no need to type Shift + O keys to adjust the kernel parameter. You can now log in on ESXi web UI too.
  • GUI: https://192.168.0.10/
 
After logging in, you can apply the serial number which you obtained when you downloaded free vSphere Hypervisor before. 
 

Create datastore manually (optional)

For large computers such as servers, there is generally enough storage space. After installing vSphere Hypervisor or ESXi, usually at least one available datastore is automatically generated. Only with datastore can virtual machines be created and actual edge applications can be deployed. However, for a device with only a small amount of storage space (such as a USB flash disk less than 32GB), a default datastore may be be created automatically.
For example: on ESXi 6.7, the remaining space after installing the system will not be partitioned; on ESXi 7.0, in the left space after installing the base system a VMFSL partition will be created as follows. This VMFSL partition does not actually store any data, nor can it be used as a normal VMFS partition. On virtualized devices, this VMFSL partition will not be used in most cases. We will delete it and free up space to create a VMFS partition manually. 
 
  • Log in to the virtualized device
ssh root@192.168.0.10
  • Get the target storage partition table
partedUtil getptbl /dev/disks/t10.SanDisk00Ultra_Fit000000000000004C530001131021105082
 
The key point following is to replace AA31E02A400F11DB9590000C2911D1B8 with the VMFS UUID AA31E02A400F11DB9590000C2911D1B8 in the 7th partition. Do not modify anything else.
  • Get coredump partition status
esxcli system coredump partition get
  • Unlock coredump partition
esxcli system coredump partition set --enable false
  • Refresh the partition table of the target storage
partedUtil setptbl /dev/disks/t10.SanDisk00Ultra_Fit000000000000004C530001131021105082 gpt \
"1 64 204863 C12A7328F81F11D2BA4B00A0C93EC93B 128" \
"5 208896 2306047 EBD0A0A2B9E5443387C068B6B72699C7 0" \
"6 2308096 4405247 EBD0A0A2B9E5443387C068B6B72699C7 0" \
"7 4407296 20971486 AA31E02A400F11DB9590000C2911D1B8 0"

  • Create a datastore on the new VMFS partition
vmkfstools /dev/disks/t10.SanDisk00Ultra_Fit000000000000004C530001131021105082:7
 
Now you can find that a new datastore has been created, and you can rename it to "datastore1". 

Deploy edge applications

Next, virtualized devices can be used to deploy edge applications. For example, the EdgeX Foundry project under LF Edge can be successfully deployed to virtualized devices with a Ubuntu virtual machine image. In the workshop on EdgeX Foundry China Day in December 2019, we have shown examples of EdgeX Foundry deployed on virtualized devices, working properly. More than 10 container instances in EdgeX Foundry microservice architecture are all running in the same Ubuntu virtual machine instance on virtualized devices. 
 If the same virtualized device has considerable left resources after deploying an EdgeX Foundry instance, you can deploy several other edge applications supported by any other OSes. For example, Azure IoT Edge based on Windows IoT Core, “real-time” control software based on Zephyr, Wind River or QNX, legacy systems based on DOS, signage applications based on Android x86, modern edge applications based on Raspbian or Ubuntu Core, etc.
 
There are two common ways to deploy virtual machine images: GUI approach via ESXi Web UI or CLI approach via OVF Tools. Because it is the same as the deployment process of other virtual machine images, the detailed steps are not repeated here.

Passthrough PCIe and USB devices

When deploying edge applications on virtualized devices, you may need to set up pass-through for certain PCIe or USB devices, especially to connect wireless sensors in the field, accelerators in the device, Wifi or LTE/5G NICs to the cloud. These settings are not different from the methods in the data center and will not be repeated here.

Next

In the manner described above, the entire process of installing a virtualized device and deploying edge applications on it can be completed. This is acceptable for developers to conduct development, testing, and small-scale experiments. 
But it's far from desireable if considering the deployment of edge computing in a production environment, such as large-scale, unattended, remote monitoring, automatic update, etc. The next article will introduce how to manage large-scale virtualized devices in the cloud.