As VxRail is based on VSAN technology, VxRail has common network concept with vSphere Network. So I am going to write this post about very basic vSphere/VxRail network environment for the record.
(In this post, some images are used which I just googled and found. I don't know about blog image usage policy well. So if there is copyright breach in this post, let me know. I will remove the images from this post.)
VLAN Tagging Methods in vSphere
In vSphere virtual network system, it is recommended to use VLAN. By enabling VLAN, a physical LAN can be segmented into several isolated virtual LANs. As a result of appropriate VLAN configurations, secure network traffic can be deployed. Also, as VLAN is largely used in various environment other than vSphere environments, with proper VLAN settings, vSphere network can be easily integrated into existing environment.
In vSphere environment, there are 3 types of VLAN tagging method. There is no big difference among these tagging method for "how" a VLAN tagg is added to a packet. But what makes difference is "when" a VLAN tag is added to a packet. In other words, there are 3 timing "when" a VLAN tag is added to a packet in case you need to consider vSphere network environment.
If you want to know how to configure the VLAN tagging methods described below, please refer the following link:
https://kb.vmware.com/s/article/1003806
1. External Switch Tagging
With this method, all VLAN tagging is performed on the physical switch. It means, the tagging process is completely out of vSphere control. Also, because the physical switch assigns VLAN tag to the packets which come out of ESX hosts, no VLAN ID must be set to the portgroups in order to allow the physical switch to accept the packets.
Also you need to be sure that, in case you use this configuration, the ports on the physical switch which are connected to ESX host should be set as access port. When you use Virtual Guest Tagging or Virtual Switch Tagging (explained below), the physical switch ports should be trunk port. In a very brief explanation, access port is for the ports in which packets without a VLAN tag pass through. On the other hand, trunk port is for the ports in which various packets with various VLAN tags pass through.
2. Virtual Guest Tagging
All VLAN tagging is performed by the virtual machines. I don't know if it is technically correct, but in my image, VLAN ID is tagged to the packet at vNIC when the packets go out from the virtual machines.
To tell the truth, I didn't know about this configuration so much. So I read the document: https://www.vmware.com/pdf/esx3_vlan_wp.pdf
One of the advantages of using this mechanism, according to the document, is "the number of VLANs per virtual machine is not limited to the number of virtual adapters". As VLAN ID can be changed by the virtual machines themselves, they can change their VLAN ID whatever they want according to existing network environments. If the virtual machines exist in the network environment where multiple VLAN ID traffics are passing through and have to communicate with different networks occasionally, this configuration must be useful. However, as the VLAN tagging process are performed by the virtual machines, it cost their CPU usage for tagging outbound frames and removing tags from inbound frame in case without VLAN hardware acceleration.
3. Virtual Switch Tagging
All VLAN tagging is performed by the virtual switch when going out of ESX hosts. And this is the VLAN tagging method used by VxRail.
When you use this configuration, VLAN IDs are assigned to the portgroups which are connected to the virtual switch like in the image below.
With appropreate VLAN configuration, the traffics of Server-PG-1 and Server-PG-2 (examples in the above picture) are logically isolated and secured.
Memo
There is something you should be careful when deploying a new VxRail cluster. In the VMware KB document, you can find the statement below:
The Native VLAN is not tagged and thus requires no VLAN ID to be set on the ESXi/ESX portgroup.
When you are configuring VxRail cluster, you need to set VLAN IDs to portgroups. Native VLAN segment is often used for such as a management network portgroup. What is confusing here is, Native VLAN means non-tagged traffic however it still has VLAN ID. If you are using Cisco switchs, their default Native VLAN ID is 1 but it can be changed. So in some environment, just normal digit such as 52, 199 or whatever less than 4096 with some exceptions are used as Native VLAN ID. In that case, let's say Native VLAN ID of the physical switch is 52, people tend to make a mistakes to assign VLAN ID 52 (because it seems just a normal VLAN ID) to the management portgroup even though it must be non-tagged as it is Native VLAN ID. In this case, because VLAN is already tagged to the traffic from the management portgroup, the physical switch won't accept the traffic as a Native VLAN traffic, as a result, the VxRail Cluster fails to be deployed.
VxRail Cluster Network
As explained above, VxRail uses Virtual Switch Tagging as its tagging method. I used an image to explain the tagging method, however, there is only one virtual switch, or Starndard Switch, in the picture. Actually, VxRail Cluster uses Distributed Switch. In case using Starndard Switch, virtual switches can be connected only to the portgroups in the one ESX host. On the other hand, with Distributed Switch, you can connect the distributed virtual switch with the portgroups on multiple ESX hosts under the cluster.
However, VxRail uses Distributed Switch only after a VxRail cluster is successfully deployed. This is the image of VxRail nodes (It is different by its model). As of the default factory shipped state, each VxRail node is just an initialized ESX host (some basic and important compornents are already installed though). Thus, at this point, each VxRail has its ownStandard Switch inside.
These nodes are not connected internally. Therefore, without external physical switches, they can not communicate with each other so that network environment like below is necessary (if I could modify the cable connection, for ensuring redundancy and avoiding single point of failure, each node should be connected to 2 physical switches. It seems to be connected to 2 switches but the other one is just for BMC interface).
Each VxRail node is connected to physical switches and communicate each other via the switches. managenment network portgroup is used for allowing ESX hosts on each VxRail node to communicate via the physical switch during (and after) initial cluster deployment. On the other hand, VM network portgroup is used for allowing basic VxRail components such as vCenter Server Appliance and VxRail Manager to communicate via the physical switch during (and after) initial cluster deployment.
In most case, however, pre-existing switches are used for VxRail Cluster so that VLAN ID and other networking configuration are already set in the pre-existing switches. If VLAN ID configurations do not much between VxRail nodes and physical swiches, they can not forward packets. As a result, when initially configuring VxRail Cluster, you need to manually assign appropreate VLAN IDs to managenment network portgroup and VM networkportgroup which exist in VxRail node as factory shipped default configuration but with no VLAN tag setting. (When configuring VxRail Cluster, you need to consider portgroups for vMotion and VSAN as well. But VLAN tagging configuration for the 2 portgroups is automatically set by VxRail itself during initial cluster deployment. So you don't need to assign VLAN ID by yourself manually but need to specify which VLAN ID should be used for each before the deployment.)