Determining Network Online Status of Dualstack Cloud VMs

BLOG / August 29, 2025

When a Debian cloud VM boots, it typically runs cloud-init at various points in the boot process. Each invocation can perform certain operations based on the host’s static configuration passed by the user, typically either through a well known link-local network service or an attached iso9660 drive image. Some of the cloud-init steps execute before the network comes up, and others at a couple of different points after the network is up.

I recently encountered an unexpected issue when configuring a dualstack (uses both IPv6 and legacy IPv4 networking) VM to use a custom apt server accessible only via IPv6. VM provisioning failed because it was unable to access the server in question, yet when I logged in to investigate, it was able to access the server without any problem. The boot had apparently gone smoothly right up until cloud-init’s Package Update Upgrade Install module called apt-get update, which failed and broke subsequent provisioning steps. The errors reported by apt-get indicated that there was no route to the service in question, which more accurately probably meant that there was not yet a route to the service. But there was shortly after, when I investigated.

This was surprising because the apt-get invocations occur in a cloud-init sequence that’s explicitly ordered after the network is configured according to systemd-networkd-wait-online. Investigation eventually led to similar issues encountered in other environments reported in Debian bug #1111791, “systemd: network-online.target reached before IPv6 address is ready”. The issue described in that bug is identical to mine, but the bug is tagged wontfix. The behavior is considered correct.

Why the default behavior is the correct one

While it’s a bit counterintuitive, the systemd-networkd behavior is correct, and it’s also not something we’d want to override in the cloud images. Without explicit configuration, systemd can’t accurately infer the intended network configuration of a given system. If a system is IPv6-only, systemd-networkd-wait-online will introduce unexpected delays in the boot process if it waits for IPv4, and vice-versa. If it assumes dualstack, things are even worse because it would block for a long time (approximately two minutes) in any single stack network before failing, leaving the host in degraded state. So the most reasonable default behavior is to block until any protocol is configured.

For these same reasons, we can’t change the systemd-networkd-wait-online configuration in our cloud images. All of the cloud environments we support offer both single stack and dual stack networking, so we preserve systemd’s default behavior.

What’s causing problems here is that IPv6 takes significantly longer to configure due to its more complex router solicitation + router advertisement + DHCPv6 setup process. So in this particular case, where I’ve got a dualstack VM that needs to access a v6-only apt server during the provisioning process, I need to find some mechanism to override systemd’s default behavior and wait for IPv6 connectivity specifically.

What won’t work

Cloud-init offers the ability to write out arbitrary files during provisioning. So writing a drop-in for systemd-networkd-wait-online.service is trivial. Unfortunately, this doesn’t give us everything we actually need. We still need to invoke systemctl daemon-reload to get systemd to actually apply the changes after we’ve written them, and of course we need to do that before the service actually runs. Cloud-init provides a bootcmd module that lets us run shell commands “very early in the boot process”, but it runs too early: it runs before we’ve written out our configuration files. Similarly, it provides a runcmd module, but scripts there are towards the end of the boot process, far too late to be useful.

Instead of using the bootcmd facility, to simply reload systemd’s config, it seemed possible that we could both write the config and trigger the reload, similar to the following:

 bootcmd:
  - mkdir -p /etc/systemd/system/systemd-networkd-wait-online.service.d
  - echo "[Service]" > /etc/systemd/system/systemd-networkd-wait-online.service.d/10-netplan.conf
  - echo "ExecStart=" >> /etc/systemd/system/systemd-networkd-wait-online.service.d/10-netplan.conf
  - echo "ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --operational-state=routable --any --ipv6" >> /etc/systemd/system/systemd-networkd-wait-online.service.d/10-netplan.conf
  - systemctl daemon-reload

But even that runs too late, as we can see in the logs that systemd-networkd-wait-online.service has completed before bootcmd is executed:

root@sid-tmp2:~# journalctl --no-pager -l -u systemd-networkd-wait-online.service
Aug 29 17:02:12 sid-tmp2 systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Aug 29 17:02:13 sid-tmp2 systemd[1]: Finished systemd-networkd-wait-online.service - Wait for Network to be Configured
.

root@sid-tmp2:~# grep -F 'config-bootcmd ran' /var/log/cloud-init.log 
2025-08-29 17:02:14,766 - handlers.py[DEBUG]: finish: init-network/config-bootcmd: SUCCESS: config-bootcmd ran successfully and took 0.467 seconds

At this point, it’s looking like there are few options left!

What eventually worked

I ended up identifying two solutions to the issue, both of which involve getting some other component of the provisioning process to run systemd-networkd-wait-online.

Solution 1

The first involves getting apt-get itself to wait for IPv6 configuration. The apt.conf configuration interface allows the definition of an APT::Update::Pre-Invoke hook that’s executed just before apt’s update operation. By writing the following to a file in /etc/apt/apt.conf.d/, we’re able to ensure that we have IPv6 connectivity before apt-get tries accessing the network. This cloud-config snippet accomplishes that:

  - path: /etc/apt/apt.conf.d/99-wait-for-ipv6
    content: |
      APT::Update::Pre-Invoke { "/usr/lib/systemd/systemd-networkd-wait-online --operational-state=routable --any --ipv6"; }

This is safe to leave in place after provisioning, because the delay will be negligible once IPv6 connectivity is established. It’s only during address configuration that it’ll block for a noticeable amount of time, but that’s what we want.

This solution isn’t entirely correct, though, because it’s only apt-get that’s actually affected by it. Other service that start after the system is ostensibly “online” might only see IPv4 connectivity when they start. This seems acceptable at the moment, though.

Solution 2

The second solution is to simply invoke systemd-networkd-wait-online directly from a cloud-init bootcmd. Similar to the first solution, it’s not exactly correct because the host has already reached network-online.target, but it does block enough of cloud-init that package installation happens only after it completes. The cloud-config snippet for this is

bootcmd:
  - [/usr/lib/systemd/systemd-networkd-wait-online, --operational-state=routable, --any, --ipv6]

In either case, we still want to write out a snippet to configure systemd-networkd-wait-online to wait for IPv6 connectivity for future reboots. Even though cloud-init won’t necessarily run in those cases, and many cloud VMs never reboot at all, it does complete the solution. Additionally, it solves the problem for any derivative images that may be created based on the running VM’s state. (At least if we can be certain that instances of those derivative images will never run in an IPv4-only network!)

write_files:
  - path: /run/systemd/system/systemd-networkd-wait-online.service.d/99-ipv6-wait.conf
    content: |
      [Service]
      ExecStart=
      ExecStart=/lib/systemd/systemd-networkd-wait-online --any --operational-state=routable --ipv6

How to properly solve it

One possible improvement would be for cloud-init to support a configuration key allowing the admin to specify the required protocols. Based on the presence of this key, cloud-init could reconfigure systemd-networkd-wait-online.service accordingly. Alternatively it could set the appropriate RequiredFamilyForOnline= value in the generated .network file. cloud-init supports multiple network configuration backends, so each of those would need to be updated. If using the systemd-networkd configuration renderer, this should be straightforward, but Debian uses the netplan renderer, so that tool might also need to be taught to pass such a configuration along to systemd-networkd.

Tags:

debian cloud computing