1

The setup:
3 Linux Ubuntu 18.04 virtualized in GNS3 all connected to a Ethernet hub, lets call them VM1, VM2 and VM3, each VM has a physical interface called ens3 and a IP Address from DHCP Server running on main router.

On VM1 I have created two new virtual interfaces called macsec1 and macsec2, on VM2 - macsec1 and on VM3 - macsec1. They are created with the following commands:

For VM1:

# Creating the virtual macsec1 interface and its Rx channels
sudo ip link add link ens3 macsec1 type macsec port 1 encrypt on validate strict
sudo ip macsec add macsec1 tx sa 0 pn 1 on key 11 11111111111111111111111111111111
sudo ip macsec add macsec1 rx address 0c:a0:95:25:00:00 port 1
sudo ip macsec add macsec1 rx address 0c:a0:95:25:00:00 port 1 sa 0 pn 1 on key 22 22222222222222222222222222222222

Creating the virtual macsec2 interface and its Rx channels


sudo ip link add link ens3 macsec2 type macsec port 2 encrypt on validate check
sudo ip macsec add macsec2 tx sa 0 pn 1 on key 44 44444444444444444444444444444444
sudo ip macsec add macsec2 rx address 0c:99:22:ee:00:00 port 1
sudo ip macsec add macsec2 rx address 0c:99:22:ee:00:00 port 1 sa 0 pn 1 on key 33 33333333333333333333333333333333


Set the IP and bring the interface UP


sudo ip link set dev macsec1 up
sudo ip link set dev macsec2 up
sudo ifconfig macsec1 10.1.0.1/16
sudo ifconfig macsec2 10.2.0.1/16

For VM2:

# Creating the virtual macsec1 interface and its Rx channels
sudo ip link add link ens3 macsec1 type macsec port 1 encrypt on validate strict
sudo ip macsec add macsec1 tx sa 0 pn 1 on key 22 22222222222222222222222222222222
sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 1
sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 1 sa 0 pn 1 on key 11 11111111111111111111111111111111

Set the IP and bring the interface UP


sudo ip link set dev macsec1 up
sudo ifconfig macsec1 10.1.0.2/16

And for VM3:

# Creating the virtual macsec1 interface and its Rx channels
sudo ip link add link ens3 macsec1 type macsec port 1 encrypt on validate check
sudo ip macsec add macsec1 tx sa 0 pn 1 on key 33 33333333333333333333333333333333
sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 1
sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 1 sa 0 pn 1 on key 44 44444444444444444444444444444444

Set the IP and bring the interface UP


sudo ip link set dev macsec1 up
sudo ifconfig macsec1 10.2.0.2/16

The expected outcome:
Based on the configuration I should be able to ping VM2 and VM3 from VM1 and the packets should be routed from the respective interfaces, for example. A ping request from VM1 to VM2 should come from 10.1.0.1 source and 10.1.0.2 should be the destination resulting in the correct MACsec configuration being applied and this is the case for communications between VM1 and VM2, however it does not work for VM1 to VM3, now we get to the problem.

The problem and troubleshooting process:
VM1 cannot reach VM3 using the created virtual interfaces, after checking the routes, everything seems correct to me, here are they for VM1 and VM3:

VM1 Routes:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         homerouter.cpe  0.0.0.0         UG    100    0        0 ens3
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens3
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 macsec1
10.2.0.0        0.0.0.0         255.255.0.0     U     0      0        0 macsec2

VM3 Routes:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         homerouter.cpe  0.0.0.0         UG    100    0        0 ens3
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens3
10.2.0.0        0.0.0.0         255.255.0.0     U     0      0        0 macsec1

After inspecting the tcpdumps I found out the following:
When VM3 pings VM1, the macsec1 interface on VM3 sends the ICMP echo request to 10.2.0.1 however no reply is received.

IP vm3 > 10.2.0.1: ICMP echo request, id 3387, seq 1, length 64
IP vm3 > 10.2.0.1: ICMP echo request, id 3387, seq 2, length 64

On the VM1 I can see the the incoming echo request, and also the response.

IP 10.2.0.2 > vm1: ICMP echo request, id 3387, seq 1, length 64
IP vm1 > 10.2.0.2: ICMP echo reply, id 3387, seq 2, length 64

After inspecting the traffic on the ens3 interface on VM3, we can see that both the request and the reply has gone trough it, however the macsec1 interface has not received the reply.

0c:99:22:ee:00:00 > 0c:63:58:d6:00:00, ethertype Uknown (0x88e5), length 130:
0c:63:58:d6:00:00 > 0c:99:22:ee:00:00, ethertype Uknown (0x88e5), length 130:

So now back to the question: Why macsec1 interface on VM3 is not receiving reply packets from VM1 even though the ens3 interface receives them? And also, why is this problem not happening with communication between VM1 and VM2?

Reinis
  • 13

1 Answers1

0

Note: to reproduce OP's setup (and display correct MAC addresses and SCIs below), one has to infer the MAC addresses on the 3 involved interfaces from MACsec configuration.

VM1's ens3: 0c:63:58:d6:00:00
VM2's ens3: 0c:a0:95:25:00:00
VM3's ens3: 0c:99:22:ee:00:00

When creating additional MACsec interfaces:

# Creating the virtual macsec2 interface and its Rx channels
sudo ip link add link ens3 macsec2 type macsec port 2 encrypt on validate check

the port's unique value chosen to create the 2nd virtual MACsec interface from the same "physical" interface isn't just a local identifier: it's part of the transmitted on-wire frame in the "Secure Channel Identifier" (SCI) field which is the concatenation of the local NIC's MAC address and the port as a 2-bytes value for a total of 8 bytes:

# ip -details link show dev macsec2
3: macsec2@ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1468 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 0c:63:58:d6:00:00 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 0 maxmtu 65535 
    macsec sci 0c6358d600000002 protect on cipher GCM-AES-128 icvlen 16 encodingsa 0 validate check offload off encrypt on send_sci on end_station off scb off replay off numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536

The peer (VM3) has to be configured to recognize this SCI with the correct port value for the matching RX configuration.

So on VM3's configuration, instead of:

sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 1
sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 1 sa 0 pn 1 on key 44 44444444444444444444444444444444

use:

sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 2
sudo ip macsec add macsec1 rx address 0c:63:58:d6:00:00 port 2 sa 0 pn 1 on key 44 44444444444444444444444444444444

to actually configure for receiving VM1's SCI 0c6358d60000002 and allow proper decryption of traffic from VM1 to VM3.

A.B
  • 13,968