Private vlans on a shared network not working correctly

[jms345]

problem

Trying to use a shared network with private vlans and the VMs were unable to ping their default gateway in testing. On investigation this turns out to be an issue with the OVS tables.

Primary vlan - 301
Isolated vlan - 302

The hypervisors use cloudbr1 which is trunk link for all guest + public traffic. All hypervisors have both vlans (301 and 302) allowed on that trunk link.

The primary vlan's L3 interface is an SVI on a Cisco switch. The Cisco switch has the correct private vlan configuration. The hypervisors can ping the L3 interface IP.

When an instance is created in this shared network it successfully gets an IP but is unable to ping the L3 interface IP. The Cisco switch shows the correct mac address of the VM NIC in it's arp table so communication to the switch is working but the return traffic is getting dropped by the OVS switch.

Looking at the OVS flow rules on the hypervisor running the VM (with some help from ai) -

root@hv2:~# ovs-ofctl dump-flows cloudbr1 | grep "dl_vlan=301"
cookie=0x0, duration=680.753s, table=0, n_packets=635, n_bytes=40640, idle_age=0, priority=200,arp,dl_vlan=301,arp_tpa=172.30.12.70 actions=strip_vlan,resubmit(,1)
cookie=0x0, duration=680.739s, table=0, n_packets=2, n_bytes=692, idle_age=659, priority=100,udp,dl_vlan=301,nw_dst=255.255.255.255,tp_dst=67 actions=strip_vlan,resubmit(,1)
cookie=0x0, duration=680.667s, table=0, n_packets=0, n_bytes=0, idle_age=680, priority=70,dl_vlan=301,dl_dst=1e:01:26:00:00:c2 actions=strip_vlan,resubmit(,1)
cookie=0x0, duration=674.094s, table=0, n_packets=105, n_bytes=6720, idle_age=21, priority=70,dl_vlan=301,dl_dst=1e:01:9d:00:00:e8 actions=strip_vlan,resubmit(,1)
cookie=0x0, duration=673.872s, table=0, n_packets=731, n_bytes=46784, idle_age=0, priority=70,dl_vlan=301,dl_dst=ff:ff:ff:ff:ff:ff actions=strip_vlan,group:301
cookie=0x0, duration=680.660s, table=1, n_packets=0, n_bytes=0, idle_age=680, priority=70,dl_vlan=301,dl_dst=1e:01:26:00:00:c2 actions=strip_vlan,output:42
cookie=0x0, duration=674.088s, table=1, n_packets=0, n_bytes=0, idle_age=674, priority=70,dl_vlan=301,dl_dst=1e:01:9d:00:00:e8 actions=strip_vlan,output:43
cookie=0x0, duration=673.865s, table=1, n_packets=0, n_bytes=0, idle_age=680, priority=70,dl_vlan=301,dl_dst=ff:ff:ff:ff:ff:ff actions=strip_vlan,group:4397

relevant lines -

cookie=0x0, duration=674.094s, table=0, n_packets=105, n_bytes=6720, idle_age=21, priority=70,dl_vlan=301,dl_dst=1e:01:9d:00:00:e8 actions=strip_vlan,resubmit(,1)
cookie=0x0, duration=674.088s, table=1, n_packets=0, n_bytes=0, idle_age=674, priority=70,dl_vlan=301,dl_dst=1e:01:9d:00:00:e8 actions=strip_vlan,output:43

table 0 has 105 matches but it is stripping the vlan 301 so when it gets to table 1 there are no matches and the default action at the end is to drop.

The solution was to add this to test -

ovs-ofctl add-flow cloudbr1 "table=1,priority=80,dl_dst=1e:01:9d:00:00:e8,actions=output:43" to match the untagged packet and send it to the VM.

Once this was added the VM could now ping it's default gateway on the Cisco switch.

But this was just a quick fix to test if that was the issue. The suggestion is that there is an issue with the matching or rewrite rules that need modifying in the code.

versions

Cloudstack 4.22.1.0, hypervisors running Ubuntu 22.04, NFS primary and secondary storage

OVS switch version - OVS version : 2.17.9

The steps to reproduce the bug

1. Create a shared network with a primary and isolated vlan.
2. Create an instance in the new shared network
3. Try to ping the default gateway
   ...

What to do about it?

No response

[weizhouapache]

@jms345
can you test the changes below with /usr/share/cloudstack-common/scripts/vm/network/ovs-pvlan-kvm-vm.sh on the kvm hosts ?

diff --git a/scripts/vm/network/ovs-pvlan-kvm-vm.sh b/scripts/vm/network/ovs-pvlan-kvm-vm.sh
index b28dbda065e..c6db7ee4f1d 100755
--- a/scripts/vm/network/ovs-pvlan-kvm-vm.sh
+++ b/scripts/vm/network/ovs-pvlan-kvm-vm.sh
@@ -189,7 +189,8 @@ then
   fi
 
   # Accept from promiscuous
-  ovs-ofctl add-flow $br table=1,priority=70,dl_vlan=$pri_vlan,dl_dst=$vm_mac,actions=strip_vlan,output:$vm_port
+  ovs-ofctl add-flow $br table=1,priority=70,vlan_tci=0x0000,dl_dst=$vm_mac,actions=output:$vm_port
+
   # From others in our own community
   if [ "$type" == "C" ]
   then

[jms345]

Tried that but no luck -

root@hv2:/usr/share/cloudstack-common/scripts/vm/network# ovs-ofctl dump-flows cloudbr1 | grep "dl_vlan=301"
 cookie=0x0, duration=339.408s, table=0, n_packets=18, n_bytes=1152, idle_age=24, priority=200,arp,dl_vlan=301,arp_tpa=172.30.12.70 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=339.395s, table=0, n_packets=2, n_bytes=692, idle_age=282, priority=100,udp,dl_vlan=301,nw_dst=255.255.255.255,tp_dst=67 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=339.322s, table=0, n_packets=0, n_bytes=0, idle_age=339, priority=70,dl_vlan=301,dl_dst=1e:01:26:00:00:c2 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=296.191s, table=0, n_packets=6, n_bytes=384, idle_age=21, priority=70,dl_vlan=301,dl_dst=1e:01:ac:00:00:c3 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=295.965s, table=0, n_packets=1, n_bytes=64, idle_age=277, priority=70,dl_vlan=301,dl_dst=ff:ff:ff:ff:ff:ff actions=strip_vlan,group:301
 cookie=0x0, duration=295.958s, table=1, n_packets=0, n_bytes=0, idle_age=339, priority=70,dl_vlan=301,dl_dst=ff:ff:ff:ff:ff:ff actions=strip_vlan,group:4397
root@hv2:/usr/share/cloudstack-common/scripts/vm/network# ovs-ofctl dump-flows cloudbr1 | grep "vlan_tci=0x0000"
 cookie=0x0, duration=406.444s, table=1, n_packets=16, n_bytes=1024, idle_age=12, priority=70,vlan_tci=0x0000,dl_dst=1e:01:26:00:00:c2 actions=output:3
 cookie=0x0, duration=363.312s, table=1, n_packets=7, n_bytes=448, idle_age=30, priority=70,vlan_tci=0x0000,dl_dst=1e:01:ac:00:00:c3 actions=output:7
root@hv2:/usr/share/cloudstack-common/scripts/vm/network# ovs-ofctl add-flow cloudbr1 "table=1,priority=80,dl_dst=1e:01:ac:00:00:c3,actions=output:7"

the mac address of the VM is 1e:01:ac:00:00:c3 so relevant lines -

cookie=0x0, duration=296.191s, table=0, n_packets=6, n_bytes=384, idle_age=21, priority=70,dl_vlan=301,dl_dst=1e:01:ac:00:00:c3 actions=strip_vlan,resubmit(,1)
cookie=0x0, duration=363.312s, table=1, n_packets=7, n_bytes=448, idle_age=30, priority=70,vlan_tci=0x0000,dl_dst=1e:01:ac:00:00:c3 actions=output:7

which looks better to me but still unable to ping gateway IP.

[jms345]

Actually this could be down to stale arp entries on my L3 switch.

Let me check and i'll get back to you.

[jms345]

Good news and bad news -

1. the VMs can now ping their default gateway after the change to the script but

2. the VMs can now ping each other as well whether they are on the same hypervisor or a different hypervisor.

Do you want to close this and me open up another bug or is it related to this one ?

[weizhouapache]

Good news and bad news -

1. the VMs can now ping their default gateway after the change to the script but
2. the VMs can now ping each other as well whether they are on the same hypervisor or a different hypervisor.

Do you want to close this and me open up another bug or is it related to this one ?

@jms345
let's use this ticket for tracking

can you share the full output of ovs-ofctl dump-flows cloudbr1 ?

[jms345]

two vms on the same hypervisor - IP/mac address -

[root@pvm3 ~]# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1e:01:41:00:00:c3 brd ff:ff:ff:ff:ff:ff
    inet 172.30.12.71/26 brd 172.30.12.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::bf37:b29b:5541:d7db/64 scope link
       valid_lft forever preferred_lft forever


[root@pvm4 ~]# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1e:01:4d:00:00:c4 brd ff:ff:ff:ff:ff:ff
    inet 172.30.12.72/26 brd 172.30.12.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3e1e:710b:506c:7f44/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::bf37:b29b:5541:d7db/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever

output of "ovs-ofctl dump-flows cloudbr1" -

root@hv2:~# ovs-ofctl dump-flows cloudbr1
 cookie=0x0, duration=2271.829s, table=0, n_packets=87, n_bytes=5568, priority=200,arp,dl_vlan=301,arp_tpa=172.30.12.70 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=2271.818s, table=0, n_packets=26, n_bytes=8996, priority=100,udp,dl_vlan=301,nw_dst=255.255.255.255,tp_dst=67 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=2271.763s, table=0, n_packets=556, n_bytes=54338, priority=70,dl_vlan=301,dl_dst=1e:01:26:00:00:c2 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=2201.942s, table=0, n_packets=211, n_bytes=20666, priority=70,dl_vlan=301,dl_dst=1e:01:41:00:00:c3 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=2114.210s, table=0, n_packets=165, n_bytes=16144, priority=70,dl_vlan=301,dl_dst=1e:01:4d:00:00:c4 actions=strip_vlan,resubmit(,1)
 cookie=0x0, duration=1485.735s, table=0, n_packets=3, n_bytes=192, priority=70,dl_vlan=301,dl_dst=ff:ff:ff:ff:ff:ff actions=strip_vlan,group:301
 cookie=0x0, duration=349.734s, table=0, n_packets=3175, n_bytes=228154, priority=0 actions=NORMAL
 cookie=0x0, duration=2271.825s, table=1, n_packets=0, n_bytes=0, priority=200,arp,dl_vlan=302,arp_tpa=172.30.12.70 actions=strip_vlan,output:vnet0
 cookie=0x0, duration=2271.814s, table=1, n_packets=0, n_bytes=0, priority=100,udp,dl_vlan=302,nw_dst=255.255.255.255,tp_dst=67 actions=strip_vlan,output:vnet0
 cookie=0x0, duration=2271.757s, table=1, n_packets=618, n_bytes=58306, priority=70,vlan_tci=0x0000,dl_dst=1e:01:26:00:00:c2 actions=output:vnet0
 cookie=0x0, duration=2201.935s, table=1, n_packets=211, n_bytes=20666, priority=70,vlan_tci=0x0000,dl_dst=1e:01:41:00:00:c3 actions=output:vnet6
 cookie=0x0, duration=2114.203s, table=1, n_packets=165, n_bytes=16144, priority=70,vlan_tci=0x0000,dl_dst=1e:01:4d:00:00:c4 actions=output:vnet7
 cookie=0x0, duration=1485.950s, table=1, n_packets=96, n_bytes=8500, priority=70,vlan_tci=0x0000,dl_dst=1e:01:9d:00:00:c5 actions=output:9
 cookie=0x0, duration=2271.750s, table=1, n_packets=0, n_bytes=0, priority=70,udp,dl_vlan=302,dl_dst=1e:01:26:00:00:c2,tp_src=67 actions=strip_vlan,output:vnet0
 cookie=0x0, duration=2201.928s, table=1, n_packets=0, n_bytes=0, priority=70,udp,dl_vlan=302,dl_dst=1e:01:41:00:00:c3,tp_src=67 actions=strip_vlan,output:vnet6
 cookie=0x0, duration=2114.196s, table=1, n_packets=0, n_bytes=0, priority=70,udp,dl_vlan=302,dl_dst=1e:01:4d:00:00:c4,tp_src=67 actions=strip_vlan,output:vnet7
 cookie=0x0, duration=1485.722s, table=1, n_packets=0, n_bytes=0, priority=70,udp,dl_vlan=302,dl_dst=ff:ff:ff:ff:ff:ff,tp_src=67 actions=strip_vlan,group:302
 cookie=0x0, duration=1485.729s, table=1, n_packets=0, n_bytes=0, priority=70,dl_vlan=301,dl_dst=ff:ff:ff:ff:ff:ff actions=strip_vlan,group:4397
 cookie=0x0, duration=1485.936s, table=1, n_packets=13, n_bytes=1396, priority=0 actions=drop