Configuring 802.1Q tunnneling (Q-in-Q) on Cisco switches
The 802.1Q tunnneling technology also known as Q-in-Q is an extension to the well known 802.1Q standard which allows service providers to transport customers VLANs by simply adding another layer of IEEE 802.1Q tag to the original 802.1Q tagged packets that enter the ISP network. Customer VLAN IDs are preserved and traffic from different customers is segregated within the service-provider infrastructure even when they appear to be on the same VLAN. The primary benefit for the service provider is reduced number of VLANs supported for the same number of customers. By using 802.1Q tunneling the layer 2 domain of a customer can be extended across multiple sites. A Q-in-Q frame can be identified by the Ethertype field 0x8100 in the Ethernet header and it’s called a double-tagged frame. One outer ISP VLAN tag can carry 4096 customer VLAN tags and this brings the total number of available VLANs to approximately 16.8 million.
In my scenario there are two customers each having 2 sites and the requirement is to extend the layer 2 domain between SITE A and SITE B for each customer. The service provider network is composed of switches SW1, SW2 and SW3. For each customer the ISP will allocate a VLAN tag, CUSTOMER 1 will be assigned VLAN 111 and CUSTOMER 2 will be assigned VLAN 222. Both sites for CUSTOMER 1 will use 10.10.10.0/24 subnet and they will be placed in VLAN 100 internally. For CUSTOMER 2 both sites will use 18.104.22.168/24 subnet and VLAN 200.
By default the maximum transmission unit (MTU) of an interface is 1500 bytes. Using Q-in-Q an outer VLAN tag is attached to an Ethernet frame, and the packet size is increased with 4 bytes. Therefore, it is recommended to adjust the MTU of each interface within the service provider network to a higher value. The recommended minimum MTU value is 1504 bytes. In order to support frames larger than 1500 bytes on Ethernet ports we can use the command system mtu <value> in global configuration mode. After changing the MTU size a reload will be required for the new MTU to take effect.
SW1(config)# system mtu 1504 SW2(config)# system mtu 1504 SW3(config)# system mtu 1504
The increase of MTU size is needed only on the service provider switches, no modifications are required on the customer equipment. By default, IEEE 802.1Q tunneling is disabled because the default switchport mode is dynamic auto. On the service provider network first we need to configure the 802.1Q trunks between all 3 switches.
SW1(config)# interface Gi0/1 SW1(config-if)# switchport trunk encapsulation dot1q SW1(config-if)# switchport mode trunk
SW2(config)# interface Gi0/0 SW2(config-if)# switchport trunk encapsulation dot1q SW2(config-if)# switchport mode trunk
SW2(config)# interface Gi0/1 SW2(config-if)# switchport trunk encapsulation dot1q SW2(config-if)# switchport mode trunk
SW3(config)# interface Gi0/0 SW3(config-if)# switchport trunk encapsulation dot1q SW3(config-if)# switchport mode trunk
Optionally we can restrict the allowed VLANs to be carried on the trunk links (only VLANs 111 and 222) by using the following command on all trunk interfaces configured above.
switchport trunk allowed vlan 111,222
Before we setup the tunnel ports we need to make sure that both VLANs 111 and 222 (also called metro tags) are configured on all ISP switches. Below is the VLAN configuration for switch SW1.
SW1(config)# vlan 111 SW1(config-vlan)#name CUSTOMER-1 SW1(config)# vlan 222 SW1(config-vlan)#name CUSTOMER-2
Since we are not using VTP we need to apply this VLAN configuration manually also on the switches SW2 and SW3.
Next we need to enable Layer 2 protocol tunneling on the edge interfaces that are connected to the customer. Based on the topology shown above these interfaces are: Gi0/0 and Gi0/2 for switch SW1 and Gi0/1 and Gi0/2 for switch SW3. The customer facing interfaces must be configured as access ports since Layer 2 protocol tunneling cannot be enabled on ports configured in dynamic auto or dynamic desirable mode.
The customer facing ports, need to be assigned to the appropriate VLANs and configured to be in 802.1Q tunnel mode. When you enter dot1q-tunnel, the port is set unconditionally as an IEEE 802.1Q tunnel port. Below is the dot1q-tunnel configuration for all switch ports which are connected to the customer switches.
SW1(config)# interface Gi0/0 SW1(config-if)# switchport mode dot1q-tunnel SW1(config-if)# switchport access vlan 111
SW1(config)# interface Gi0/2 SW1(config-if)# switchport mode dot1q-tunnel SW1(config-if)# switchport access vlan 222
SW3(config)# interface Gi0/1 SW3(config-if)# switchport mode dot1q-tunnel SW3(config-if)# switchport access vlan 111
SW3(config)# interface Gi0/2 SW3(config-if)# switchport mode dot1q-tunnel SW3(config-if)# switchport access vlan 222
We stated that both sites for each customer which are connected across the service-provider network need to have Layer 2 connectivity. By default switches intercept and process a number of layer two protocols, like CDP, STP, VTP, and others. Our goal is to tunnel these Layer 2 protocols so the service provider switches become transparent and forward CDP, VTP, STP instead of processing them. This can be achieved using layer 2 protocol tunneling. When L2 protocol tunneling is enabled all these layer 2 protocol will behave as there were part of the same broadcast network. In order to enable L2 tunneling we need to use l2protocol-tunnel command in interface configuration mode.
SW1(config)# interface Gi0/0 SW1(config-if)# l2protocol-tunnel SW1(config-if)# no cdp enable
SW1(config)# interface Gi0/2 SW1(config-if)# l2protocol-tunnel SW1(config-if)# no cdp enable
SW3(config)# interface Gi0/1 SW3(config-if)# l2protocol-tunnel SW3(config-if)# no cdp enable
SW3(config)# interface Gi0/2 SW3(config-if)# l2protocol-tunnel SW3(config-if)# no cdp enable
When protocol tunneling is enabled, protocol packets are encapsulated with a well-known Cisco multicast address for transmission across the network. When the packets reach their destination, the well-known MAC address is replaced by the Layer 2 protocol MAC address. Using the command l2protocol-tunnel without any options we enable L2 tunneling for all layer 2 supported protocols. If you want to enable only specific layer 2 protocol you can use the following command with one of the values available between the vertical bars.
SW(config-if)# l2protocol-tunnel <cdp|lldp|stp|vtp|cos>
Now the that the ISP side configuration is complete let’s focus on the client side and configure the interfaces facing the ISP switches. On the customer side we need to define the required VLANs and enable 802.1Q trunking in order to allow all customer VLAN traffic to be transported to the other site. On customer switches CST1-A and CST1-B we need to create vlan 100 and on switches CST2-A and CST2-B we need to create vlan 200
CST1-A(config)# vlan 100 CST1-B(config)# vlan 100 CST2-A(config)# vlan 200 CST2-B(config)# vlan 200
The interfaces on the customer which connect to ISP switches (Gi0/0) will be configured as trunk ports. We need to apply the configuration below on interface Gi0/0 for all customer switches.
interface Gi0/0 switchport trunk encapsulation dot1q switchport mode trunk
Lastly let’s configure customer interfaces which are connected to the PCs (Gi0/1) as static access ports and place them in the corresponding vlans.
CST1-A(config)# interface Gi0/1 CST1-A(config-if)# switchport mode access CST1-A(config-if)# switchport access vlan 100
CST1-B(config)# interface Gi0/1 CST1-B(config-if)# switchport mode access CST1-B(config-if)# switchport access vlan 100
CST2-A(config)# interface Gi0/1 CST2-A(config-if)# switchport mode access CST2-A(config-if)# switchport access vlan 200
CST2-B(config)# interface Gi0/1 CST2-B(config-if)# switchport mode access CST2-B(config-if)# switchport access vlan 200
In order to see the list of interfaces which are configured as 802.1Q tunnels we can use the command show dot1q-tunnel on the ISP switches. Below is an example from switch SW1.
SW1#show dot1q-tunnel dot1q-tunnel mode LAN Port(s) ----------------------------- Gi0/0 Gi0/2
Let’s proceed and do some tests to check if Q-in-Q configuration works properly. First from PC-1 we’ll try to ping PC-3 . We have assigned for PC-1 ip address 10.10.10.101 and for PC-3 he have used 10.10.10.103 ip address.
root@PC-1:~# ping -c 2 10.10.10.103 PING 10.10.10.103 (10.10.10.103): 56 data bytes 64 bytes from 10.10.10.103: icmp_seq=0 ttl=64 time=7.237 ms 64 bytes from 10.10.10.103: icmp_seq=1 ttl=64 time=7.634 ms --- 10.10.10.103 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 7.237/7.436/7.634/0.199 ms
As we can see ping was successful. Now let’s run arp command and check if PC-1 can see PC-3 MAC address in its local ARP table
root@PC-1:~# arp Address HWtype HWaddress Flags Mask Iface 10.10.10.103 ether 46:a9:69:1a:12:8e C eth0
MAC address of PC-3 is present in the local ARP table of PC-1 which means PC-1 and PC-3 can communicate al layer 2. Next let’s check if layer 2 protocol tunneling is working. We can use the show l2protocol-tunnel command to display information about Layer 2 protocol tunnel ports, including the protocols configured, the thresholds, and the counters.
SW1# show l2protocol-tunnel COS for Encapsulated Packets: 5 Drop Threshold for Encapsulated Packets: 0 Port Protocol Shutdown Drop Encaps Decaps Drop Threshold Threshold Counter Counter Counter ------------------- ----------- --------- --------- --------- --------- --------- Gi0/0 cdp ---- ---- 223 1381 0 lldp ---- ---- 0 0 0 stp ---- ---- 38 12036 0 vtp ---- ---- 0 0 0
From the output above we can see that interface Gi0/0 is configured as a tunnel port. By looking at the Encaps/Decaps Counter columns we also can observe that CDP and STP traffic is being tunneled through this interface. Finally let’s check the output of show cdp neighbors and show spanning-tree commands from one of the CUSTOMER-1 switches.
CST1-A# show cdp neighbors Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone, D - Remote, C - CVTA, M - Two-port Mac Relay Device ID Local Intrfce Holdtme Capability Platform Port ID CST1-B Gig 0/0 168 R S I Gig 0/0 Total cdp entries displayed : 1
From the output we can deduce that switch CST1-A sees switch CST1-B from the remote site as his CDP neighbor so from customer perspective ISP switches are transparent.
CST1-A#show spanning-tree vlan 100 VLAN0100 Spanning tree enabled protocol ieee Root ID Priority 32868 Address 008c.3729.dd00 Cost 4 Port 1 (GigabitEthernet0/0) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32868 (priority 32768 sys-id-ext 100) Address 008c.3784.3e00 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Aging Time 300 sec Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- Gi0/0 Root FWD 4 128.1 P2p Gi0/1 Desg FWD 4 128.2 P2p
By looking at the output above we can conclude that switch CST1-A has spanning-tree for vlan 100 enabled. Also if we look at the MAC address above highlighted in red (008c.3729.dd00) we can see that switch CST1-B from the remote site is the root bridge for VLAN 100.
In this article we have demonstrated how to configure Q-in-Q technology and I hope this was helpful to anyone trying to better familiarize themselves with the concept.