Designing Dynamic BGP and Transit Gateway for Stabilizing Multi-Cloud Connectivity

Wed, 07 Oct 2026 10:25:43 +0900

As multi-cloud environments expand, traditional methods of interconnecting AWS, Azure, GCP, and on-premises data centers using static routing or individual IPSec VPNs are reaching their operational limits. Bloated route tables due to an increasing number of nodes and VPCs/VNets, human error from manual configurations, and failover delays during outages significantly degrade system availability. In this article, we design and verify a multi-cloud network architecture that balances fault tolerance and scalability by integrating dynamic Border Gateway Protocol (BGP) routing and dedicated connections (Direct Connect, ExpressRoute) into a hub-and-spoke topology centered on AWS Transit Gateway (TGW).

1. Challenges and Solutions in Multi-Cloud Networking

A. Network Management Complexity

Challenge: As cloud footprints expand, managing individual route tables across numerous AWS VPCs and Azure VNets becomes fragmented. This “routing spaghetti” leads to issues in centralized policy enforcement, security auditing, and increased Mean Time to Recovery (MTTR) during outages.
Solution: Introduce a hub-and-spoke network topology. Leveraging a centralized transit hub such as AWS Transit Gateway consolidates routing logic and centralizes cross-platform traffic control.

B. Integration of Dynamic BGP Routing and Transit Gateways

Challenge: Static routing cannot adapt dynamically to link failures, requiring manual intervention to reroute traffic and causing significant downtime.
Solution: Integrate dynamic BGP routing with transit gateways. BGP enables real-time route propagation and path selection across multi-cloud boundaries. Establishing BGP sessions between AWS TGW, Azure ExpressRoute Gateway, and GCP Cloud Router enables dynamic learning of optimal paths, achieving automatic path redundancy and rapid failover.

C. Utilization of Dedicated Connections (Cross-Cloud Direct Connect)

Challenge: VPN connections over the public internet are susceptible to internet congestion, packet loss, jitter, and security vulnerabilities.
Solution: Interconnect AWS Direct Connect, Azure ExpressRoute, and GCP Dedicated Interconnect through colocation facilities (e.g., Equinix, Megaport), completely bypassing the public internet. This ensures high bandwidth (1 Gbps to 100 Gbps), low latency, and enhanced data privacy.

2. Comparative Analysis: Legacy Configuration (AS-IS) vs. Next-Generation Configuration (TO-BE)

Comparison Item	Legacy Static Routing & VPN (AS-IS)	Modern Multi-Cloud Design (TO-BE)
Network Topology	Complex full-mesh connections (inter-VPC/VNet). Operational costs grow exponentially as node count increases.	Simple hub-and-spoke configuration centered around cloud-native hubs such as AWS Transit Gateway.
Routing Flexibility	Requires manual updates to static route tables. Switchover delays occur during link failures.	Real-time path computation, route propagation, and automated failover powered by dynamic BGP routing.
Bandwidth & Stability	Relies on IPSec VPN over the public internet. Packet loss and latency spikes occur during traffic surges.	Dedicated private lines (Direct Connect / ExpressRoute) bypass the internet to guarantee stable bandwidth.
Scalability	Adding new regions or cloud providers requires redesigning network topology and security groups.	Modular design. New VPCs/VNets can be attached to the existing transit hub without affecting current traffic.

3. Technical Specifications and Example Implementation

The following configuration establishes dynamic BGP peering with AWS Transit Gateway and Azure ExpressRoute on an on-premises or colocation router using FRRouting (FRR).

! FRRouting Configuration for Multi-Cloud BGP Peering
router bgp 65001
 bgp router-id 192.168.1.1
 no bgp default ipv4-unicast
 coalesce-time 1000
 !
 ! AWS Transit Gateway Peers
 neighbor 169.254.100.1 remote-as 64512
 neighbor 169.254.100.1 description AWS-TGW-Primary
 neighbor 169.254.100.5 remote-as 64512
 neighbor 169.254.100.5 description AWS-TGW-Secondary
 !
 ! Azure ExpressRoute Gateway Peer
 neighbor 10.0.0.2 remote-as 12076
 neighbor 10.0.0.2 description Azure-ExpressRoute-Gateway
 !
 address-family ipv4 unicast
 network 10.100.0.0/16
 !
 neighbor 169.254.100.1 activate
 neighbor 169.254.100.1 route-map AWS-IN in
 neighbor 169.254.100.1 route-map AWS-OUT out
 !
 neighbor 169.254.100.5 activate
 neighbor 169.254.100.5 route-map AWS-IN in
 neighbor 169.254.100.5 route-map AWS-OUT out
 !
 neighbor 10.0.0.2 activate
 neighbor 10.0.0.2 route-map AZURE-IN in
 neighbor 10.0.0.2 route-map AZURE-OUT out
 exit-address-family
!
ip prefix-list LOCAL-SUBNETS permit 10.100.0.0/16
ip prefix-list AWS-ALLOWED-IN permit 172.16.0.0/12 ge 12 le 24
ip prefix-list AZURE-ALLOWED-IN permit 10.200.0.0/16 ge 16 le 24
!
route-map AWS-OUT permit 10
 match ip address prefix-list LOCAL-SUBNETS
!
route-map AWS-IN permit 10
 match ip address prefix-list AWS-ALLOWED-IN
!
route-map AZURE-OUT permit 10
 match ip address prefix-list LOCAL-SUBNETS
 set as-path prepend 65001 65001
!
route-map AZURE-IN permit 10
 match ip address prefix-list AZURE-ALLOWED-IN
!

4. Troubleshooting

A. BGP ASN Conflicts and Overlaps

Issue: When designing private ASNs (64512–65534) in a multi-cloud environment, assigning duplicate ASNs across different cloud providers or on-premises sites causes routes to be rejected due to BGP loop prevention mechanism (AS-Path loop detection), leading to connectivity loss.
Mitigation: 💡 Create a centralized ASN management registry spanning all clouds and on-premises environments to eliminate duplicates. Clearly separate ASNs between AWS TGW (default: 64512) and on-premises (e.g., 65001).

B. Route Leaks and Routing Loops

Issue: Re-advertising routes learned from AWS directly to Azure without filtering causes unintended inter-cloud transit traffic, leading to line congestion and routing loops.
Mitigation: ⚠️ Strictly apply prefix lists (prefix-list) and route maps (route-map) on edge routers. Implement rigorous inbound and outbound filtering so that only local organization-owned prefixes are advertised and routes learned from other clouds are not re-advertised.

C. Asymmetric Routing

Issue: When asymmetric routing occurs—such as outbound traffic traveling via AWS Direct Connect while return traffic travels via Azure ExpressRoute—packets are dropped by stateful firewalls.
Mitigation: 🛠️ Use BGP AS-Path Prepending to intentionally extend the AS path length of backup routes, explicitly controlling the primary path. Additionally, adjust Local Preference as necessary to pin the outbound traffic path.

5. Operational Verification Commands and Status Checks

Execution of verification commands validates whether dynamic BGP routing functions correctly across the interconnected network.

Verifying BGP Neighbor Connection Status

# show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 192.168.1.1, local AS number 65001 vrf default
BGP table version 4
RIB entries 7, using 1344 bytes
Peers 3, using 61 KiB

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
169.254.100.1 4 64512 1420 1425 0 0 0 23:14:05 5 1
169.254.100.5 4 64512 1418 1422 0 0 0 23:12:10 5 1
10.0.0.2 4 12076 980 985 0 0 0 08:45:12 8 1

Verifying Learned BGP Routes

# show ip route bgp

Codes: K - kernel route, C - connected, S - static, R - RIP,
 B - BGP, O - OSPF, IA - OSPF inter area,
 V - VPNv4, NHRP - Next Hop Resolution Protocol

B&gt;* 172.16.0.0/16 [20/0] via 169.254.100.1, eth1, weight 1, 23:14:10
 * via 169.254.100.5, eth2, weight 1, 23:12:15
B&gt;* 10.200.0.0/16 [20/0] via 10.0.0.2, eth3, weight 1, 08:45:17

Verifying Symmetry via Path Tracing

$ traceroute 172.16.10.100
traceroute to 172.16.10.100 (172.16.10.100), 30 hops max, 60 byte packets
 1 192.168.1.254 (192.168.1.254) 0.421 ms 0.388 ms 0.352 ms
 2 169.254.100.1 (169.254.100.1) 2.114 ms 2.085 ms 2.051 ms
 3 172.16.10.100 (172.16.10.100) 3.452 ms 3.411 ms 3.389 ms

6. Operational Notes

Implementing dynamic routing in a multi-cloud environment serves not only to ensure connectivity, but also acts as the foundation for operational simplification and automated failure recovery. Optimizing BGP keepalive timers and hold times (e.g., in conjunction with Bidirectional Forwarding Detection (BFD)) enables sub-second fast failover in the event of a physical link failure. Standardizing prefix filtering and enforcing strict AS path control to maintain a predictable, resilient multi-cloud network is the key to ensuring the reliability of the entire infrastructure.

Hybrid-Cloud on K-Life Hack | Systems Architecture & DevOps