tuxWhile upgrading our VMGuru environment I ran into an issue with the network connectivity of a cloned Linux VM. Erik wrote about it earlier when he was restoring a VCSA from backup. I had a similar issue after cloning the VM.

On of the first things I did was logging into the server from the console and check if, and which, network interfaces were detected. When running ‘ifconfig’ I noticed that I only had a loopback device (lo), but no other network devices (eth0 or something like that). How come?

The problem wasn’t that I didn’t have any network connection, but the wrong network connection. Somehow the Linux device manager udev created more network interfaces than the VM had. To understand what happened I needed to know what was involved in the process.

Knowledge base article 2002767 describes the symptoms as:

  • The network configuration in the Linux Guest operating system still refers to the original MAC addresses.
  • Mismatch between the MAC addresses in the virtual machine settings and the Linux operating system.
  • Networking does not work in a cloned Linux virtual machine.

That seems to be the case with my server. So what is causing it?

When a virtual machine is cloned, the network adapter(s) are given new MAC addresses.

Yeah, ok, anything else? Why?

Hardware detection

The main cause of the new MAC address is that your VM got a new MAC address because you cloned the VM. Every new device you plug into your server is detected by the kernel through a system called udev.

From Wikipedia:

udev is a generic kernel device manager. It runs as a daemon on a Linux system and listens (via netlink socket) to uevents the kernel sends out if a new device is initialized or a device is removed from the system. The system provides a set of rules that match against exported values of the event and properties of the discovered device. A matching rule will possibly name and create a device node and run configured programs to set up and configure the device.

udev rules can match on properties like the kernel subsystem, the kernel device name, the physical location of the device, or properties like the device’s serial number. Rules can also request information from external programs to name a device or specify a custom name that will always be the same, regardless of the order devices are discovered by the system.

So, Udev is the device manager for the Linux 2.6 kernel and later that creates/removes device nodes in the /dev directory dynamically. It runs in userspace and the user can change device names using Udev rules. Since we cloned the VM it was as if the VM got new hardware, at least the network card with a new MAC address.

The udev rules

udev uses rules to know what to do when a new hardware item is inserted into the system. These rules are placed in /etc/udev/rules.d

In this directory some rules have been placed already. With these rules you can match  against exported values of the event and properties of the discovered device. When a rule matches it will somethimes name the device and create a device node onder /dev. It also allows programs to start upon inserting the device.

Again from Wikimedia:

udev rules can match on properties like the kernel subsystem, kernel device name, physical location of the device, or properties like the device’s serial number. Rules can also request information from external programs to name a device or specify a custom name that will always be the same, regardless of the order devices are discovered by the system.

After looking through the udev rules in /etc/udev/rules.d I noticed that one of the rules,70-persistent-net.rules, had my old network devices (eth0 and eth1), but also some new devices (eth2 and eth3).

But none of them were showing up with ifconfig. What’s that all about?

The entries looked something like

SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?”, ATTR{address}==”00:50:56:aa:xx:39″, ATTR{type}==”1″, KERNEL==”eth”, NAME=”eth0″ SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?”, ATTR{address}==”00:50:56:aa:xx:xx″, ATTR{type}==”1″, KERNEL==”eth”, NAME=”eth1″ SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?”, ATTR{address}==”00:50:56:aa:xx:xx″, ATTR{type}==”1″, KERNEL==”eth”, NAME=”eth2″ SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?”, ATTR{address}==”00:50:56:aa:xx:xx″, ATTR{type}==”1″, KERNEL==”eth”, NAME=”eth3″

On inspection of the VM properties I found out that eth2 and eth3 were the devices that were actually  present in the VM.

So I deleted the first to lines from the rules file (of course after making a copy of the file in another location). Since it was a firewall VM I also changed the eth2 and eth3 to eth0 and eth1 and rebooted the VM. I was expecting everything to work.

Changing more

I was a little bit baffled when the VM was rebooted, because I still had no network connection. If I read the whole article I probably noticed that I also needed to change the if-eth0 and if-eth1 scripts. These scripts configure the network devices after they are recognized by the kernel. Where the scripts are placed depends on the distribution you are using. In my case they  were in /etc/sysconfig/network-scripts.

After editing ifcfg-eth0 and ifcfg-eth1 and changing the MAC address in it to the same values as in the udev rules  I rebooted the whole server again (yes, not very Linux-like, I know) to make sure that everything started the correct way and we were back in business.

So, now you know, if your VM doesn’t show any ethernet devices after cloning you probably need to change the udev rules and the if-eth* files.