wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host by wido · Pull Request #13758 · apache/cloudstack

wido · 2026-07-31T12:04:12Z

Description

This Pull Request adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack.

Management server

GuestType.L3; the guest_type column is char(32), so no schema change.
Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared), Dhcp rejected as not supported and not needed. Network mode, specifyVlan and VPC use are rejected.
DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. canHandle() selects on the offering's guest type alone; design() produces a Native broadcast domain with no isolation id. After allocation the NicProfile is forced into host-route form, which is also the signature by which the agent and ConfigDrive recognise these NICs.
createNetwork treats L3 like Shared for the subnet: explicit IP range mandatory, vlan/IP-range row created at network creation, IPv6 accepted without the /64 restriction, aclType Account.
Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide.

ConfigDrive

Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR but would leave these NICs with no addressing at all. Route generation itself is unchanged: cloud-init detects an IPv4 gateway inside 169.254.0.0/16 and sets on-link on the rendered route by itself.

KVM agent

One uplink-less bridge per network, brdr-, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding and strict rp_filter; separate bridges make isolation between networks topological rather than a filtering concern.
BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host route, regardless of the host-wide EVPN property, whose meaning is unchanged.

The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.

This implements issue #12210

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Testing is still ongoing on real hardware and this PR currently (July 2026) exists to gain initial feedback.

… Instances Adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack. Management server: - GuestType.L3; the guest_type column is char(32), so no schema change. - Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared), Dhcp rejected as not supported and not needed. Network mode, specifyVlan and VPC use are rejected. - DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. canHandle() selects on the offering's guest type alone; design() produces a Native broadcast domain with no isolation id. After allocation the NicProfile is forced into host-route form, which is also the signature by which the agent and ConfigDrive recognise these NICs. - createNetwork treats L3 like Shared for the subnet: explicit IP range mandatory, vlan/IP-range row created at network creation, IPv6 accepted without the /64 restriction, aclType Account. - Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide. ConfigDrive: - Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR but would leave these NICs with no addressing at all. Route generation itself is unchanged: cloud-init detects an IPv4 gateway inside 169.254.0.0/16 and sets on-link on the rendered route by itself. KVM agent: - One uplink-less bridge per network, brdr-<network id>, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding and strict rp_filter; separate bridges make isolation between networks topological rather than a filtering concern. - BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host route, regardless of the host-wide EVPN property, whose meaning is unchanged. The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.

codecov · 2026-07-31T12:13:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.40%. Comparing base (4f11707) to head (bf3d8a3).

❗ There is a different number of reports uploaded between BASE (4f11707) and HEAD (bf3d8a3). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (4f11707) HEAD (bf3d8a3)

unittests 1 0

Additional details and impacted files

@@              Coverage Diff              @@
##               main   #13758       +/-   ##
=============================================
- Coverage     19.65%    3.40%   -16.25%     
=============================================
  Files          6368      488     -5880     
  Lines        574881    41958   -532923     
  Branches      70351     7927    -62424     
=============================================
- Hits         112970     1429   -111541     
+ Misses       449639    40329   -409310     
+ Partials      12272      200    -12072

Flag	Coverage Δ
uitests	`3.40% <ø> (-0.01%)`	⬇️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR introduces Direct Routed (L3) guest networks where KVM hosts perform L3 routing directly to instances (no Virtual Router, no NAT, no DHCP), using ConfigDrive-only addressing and a shared link-local gateway (169.254.0.1 / fe80::1) on per-network host bridges.

Changes:

Adds GuestType.L3 with server-side offering/network validation and a new DirectRoutedNetworkGuru that forces NICs into host-route form (/32, /128) with link-local gateways.
Updates ConfigDrive generation to always emit network_data.json for direct-routed NICs even when DHCP/DNS services are not present.
Extends the KVM agent to create/manage per-network bridges (brdr-<networkId>), program host routes/neighbour entries, and adapt security-group rule plumbing for routed traffic (including secondary IP handling).

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
utils/src/main/java/com/cloud/utils/net/NetUtils.java	Centralizes link-local gateway/CIDR constants and adds IPv6 link-local gateway getter.
ui/src/views/offering/AddNetworkOffering.vue	Adds UI support for L3 offerings and enforces L3-specific service/provider composition.
ui/src/views/network/CreateNetwork.vue	Adds an L3 network creation tab.
ui/src/views/network/CreateL3NetworkForm.vue	New UI form for creating L3 (Direct Routed) networks.
ui/public/locales/en.json	Adds UI strings for L3 offering/network creation messaging.
tools/marvin/marvin/config/test_data.py	Adds Marvin test data for L3 offerings and L3 networks.
test/integration/smoke/test_l3_networks.py	Adds smoke/integration tests covering L3 network lifecycle and validations.
server/src/test/java/com/cloud/network/guru/DirectRoutedNetworkGuruTest.java	Unit tests for the new guru selection/design and NIC host-route forcing.
server/src/test/java/com/cloud/configuration/ConfigurationManagerImplTest.java	Unit tests for L3 offering validation rules.
server/src/main/resources/META-INF/cloudstack/server-network/spring-server-network-context.xml	Registers the new `DirectRoutedNetworkGuru` bean.
server/src/main/java/com/cloud/network/security/SecurityGroupManagerImpl.java	Ensures agent commands are sent for secondary IPs on L3 networks even when SG rules aren’t applied.
server/src/main/java/com/cloud/network/NetworkServiceImpl.java	Extends network creation/IPv6 checks/VLAN handling and secondary-IP orchestration for L3.
server/src/main/java/com/cloud/network/guru/DirectRoutedNetworkGuru.java	New guru for L3 direct routed guest networks; forces /32 + /128 form and link-local gateways.
server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java	Adds L3 offering validation + zone-wide IPv4 overlap checks for L3 ranges.
scripts/vm/network/vnet/modifymacip.sh	Adds targeted delete support for specific -4/-6 addresses (secondary IP remove).
scripts/vm/network/vnet/modifybrdr.sh	New script to create/delete per-network `brdr-*` bridges with sysctls + gateway addresses.
scripts/vm/network/tests/test_security_group.py	New unit tests for classic-vs-L3 security_group.py rule streams and `--directrouted` plumbing.
scripts/vm/network/tests/golden_default_network_rules.txt	Golden output for classic rule generation (used to ensure no regression).
scripts/vm/network/tests/golden_add_fw_framework.txt	Golden output for classic FW framework creation (ensures byte-identical behavior).
scripts/vm/network/security_group.py	Adds routed-aware framework/hooks and rule generation, plus `--directrouted` CLI plumbing.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtSecurityGroupRulesCommandWrapper.java	Passes direct-routed flag when applying SG rules for the VM’s first NIC.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtNetworkRulesVmSecondaryIpCommandWrapper.java	Threads direct-routed + apply-SG booleans into secondary-IP handling.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java	Adds `--directrouted` plumbing and direct-routed secondary-IP route/neigh programming via modifymacip.sh.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/BridgeVifDriver.java	Creates per-network bridges for direct-routed NICs and ensures MAC/IP script runs for those NICs.
engine/storage/configdrive/src/test/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilderTest.java	Adds tests for direct-routed NIC detection and network_data.json generation behavior.
engine/storage/configdrive/src/main/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilder.java	Always generates network_data.json for direct-routed NICs; adds NIC signature detection.
docs/design/direct-routed-networks.md	Adds comprehensive design document for Direct Routed networks.
core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java	Adds `directRouted` and `applySecurityGroupRules` flags for secondary-IP commands.
api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java	Sends agent update for secondary IP removal on L3 even when zone SG is disabled.
api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java	Allows specifying physical network ID for L3 networks.
api/src/main/java/com/cloud/network/Network.java	Adds `GuestType.L3` and parsing support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        // Host-wide property (EVPN use case) runs the MAC/IP script for every NIC; a Direct
+        // Routed NIC needs it regardless, since the host route and static neighbour entry are
+        // what deliver its traffic.
+        if (_macIpStaticEnabled || isDirectRoutedNic(nic)) {
+            executeMacIpScript(intf.getBrName(), nic.getMac(), nic.getIp(), nic.getIp6Address(), nic.getNicSecIps());
+        }


+        Network network = _networkModel.getNetwork(nic.getNetworkId());
+
+        // On a Direct Routed network the host needs a route and a static neighbour entry for the
+        // secondary IP before it is reachable at all. That is independent of security groups,
+        // which are optional there and which the Instance may not be using, so the agent is told
+        // either way - otherwise the address would stay dark until the Instance was restarted.
+        boolean directRouted = Network.GuestType.L3.equals(network.getGuestType());
+


kiranchavala · 2026-08-03T05:18:41Z

@blueorangutan package

blueorangutan · 2026-08-03T05:20:04Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-08-03T05:27:52Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18739

wido added component:kvm component:networking component:advanced-networking labels Jul 31, 2026

boring-cyborg Bot added component:api component:integration-test component:marvin component:security-group component:storage component:UI Python Warning... Python code Ahead! labels Jul 31, 2026

wido mentioned this pull request Jul 31, 2026

network: L3 routed networking without shared L2 domain #12210

Open

wido requested review from kiranchavala and weizhouapache July 31, 2026 12:38

DaanHoogland requested a review from Copilot August 3, 2026 04:54

Copilot started reviewing on behalf of DaanHoogland August 3, 2026 04:55 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host - #13758

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host#13758
wido wants to merge 1 commit into
apache:mainfrom
wido:direct-routed-network

wido commented Jul 31, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

kiranchavala commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

wido commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Management server

ConfigDrive

KVM agent

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov Bot commented Jul 31, 2026

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

kiranchavala commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

blueorangutan commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wido commented Jul 31, 2026 •

edited

Loading