As multi-cloud environments expand, traditional methods of interconnecting AWS, Azure, GCP, and on-premises data centers using static routing or individual IPSec VPNs are reaching their operational limits. Bloated route tables due to an increasing number of nodes and VPCs/VNets, human error from manual configurations, and failover delays during outages significantly degrade system availability. In this article, we design and verify a multi-cloud network architecture that balances fault tolerance and scalability by integrating dynamic Border Gateway Protocol (BGP) routing and dedicated connections (Direct Connect, ExpressRoute) into a hub-and-spoke topology centered on AWS Transit Gateway (TGW).

1. Challenges and Solutions in Multi-Cloud Networking

A. Network Management Complexity

• Challenge: As cloud footprints expand, managing individual route tables across numerous AWS VPCs and Azure VNets becomes fragmented. This “routing spaghetti” leads to issues in centralized policy enforcement, security auditing, and increased Mean Time to Recovery (MTTR) during outages.
• Solution: Introduce a hub-and-spoke network topology. Leveraging a centralized transit hub such as AWS Transit Gateway consolidates routing logic and centralizes cross-platform traffic control.

B. Integration of Dynamic BGP Routing and Transit Gateways

• Challenge: Static routing cannot adapt dynamically to link failures, requiring manual intervention to reroute traffic and causing significant downtime.
• Solution: Integrate dynamic BGP routing with transit gateways. BGP enables real-time route propagation and path selection across multi-cloud boundaries. Establishing BGP sessions between AWS TGW, Azure ExpressRoute Gateway, and GCP Cloud Router enables dynamic learning of optimal paths, achieving automatic path redundancy and rapid failover.

C. Utilization of Dedicated Connections (Cross-Cloud Direct Connect)

• Challenge: VPN connections over the public internet are susceptible to internet congestion, packet loss, jitter, and security vulnerabilities.
• Solution: Interconnect AWS Direct Connect, Azure ExpressRoute, and GCP Dedicated Interconnect through colocation facilities (e.g., Equinix, Megaport), completely bypassing the public internet. This ensures high bandwidth (1 Gbps to 100 Gbps), low latency, and enhanced data privacy.

2. Comparative Analysis: Legacy Configuration (AS-IS) vs. Next-Generation Configuration (TO-BE)

3. Technical Specifications and Example Implementation

The following configuration establishes dynamic BGP peering with AWS Transit Gateway and Azure ExpressRoute on an on-premises or colocation router using FRRouting (FRR).

! FRRouting Configuration for Multi-Cloud BGP Peering
router bgp 65001
  bgp router-id 192.168.1.1
  no bgp default ipv4-unicast
  coalesce-time 1000
  !
  ! AWS Transit Gateway Peers
  neighbor 169.254.100.1 remote-as 64512
  neighbor 169.254.100.1 description AWS-TGW-Primary
  neighbor 169.254.100.5 remote-as 64512
  neighbor 169.254.100.5 description AWS-TGW-Secondary
  !
  ! Azure ExpressRoute Gateway Peer
  neighbor 10.0.0.2 remote-as 12076
  neighbor 10.0.0.2 description Azure-ExpressRoute-Gateway
  !
  address-family ipv4 unicast
    network 10.100.0.0/16
    !
    neighbor 169.254.100.1 activate
    neighbor 169.254.100.1 route-map AWS-IN in
    neighbor 169.254.100.1 route-map AWS-OUT out
    !
    neighbor 169.254.100.5 activate
    neighbor 169.254.100.5 route-map AWS-IN in
    neighbor 169.254.100.5 route-map AWS-OUT out
    !
    neighbor 10.0.0.2 activate
    neighbor 10.0.0.2 route-map AZURE-IN in
    neighbor 10.0.0.2 route-map AZURE-OUT out
  exit-address-family
!
ip prefix-list LOCAL-SUBNETS permit 10.100.0.0/16
ip prefix-list AWS-ALLOWED-IN permit 172.16.0.0/12 ge 12 le 24
ip prefix-list AZURE-ALLOWED-IN permit 10.200.0.0/16 ge 16 le 24
!
route-map AWS-OUT permit 10
  match ip address prefix-list LOCAL-SUBNETS
!
route-map AWS-IN permit 10
  match ip address prefix-list AWS-ALLOWED-IN
!
route-map AZURE-OUT permit 10
  match ip address prefix-list LOCAL-SUBNETS
  set as-path prepend 65001 65001
!
route-map AZURE-IN permit 10
  match ip address prefix-list AZURE-ALLOWED-IN
!

4. Troubleshooting

A. BGP ASN Conflicts and Overlaps

• Issue: When designing private ASNs (64512–65534) in a multi-cloud environment, assigning duplicate ASNs across different cloud providers or on-premises sites causes routes to be rejected due to BGP loop prevention mechanism (AS-Path loop detection), leading to connectivity loss.
• Mitigation: 💡 Create a centralized ASN management registry spanning all clouds and on-premises environments to eliminate duplicates. Clearly separate ASNs between AWS TGW (default: 64512) and on-premises (e.g., 65001).

B. Route Leaks and Routing Loops

• Issue: Re-advertising routes learned from AWS directly to Azure without filtering causes unintended inter-cloud transit traffic, leading to line congestion and routing loops.
• Mitigation: ⚠️ Strictly apply prefix lists (prefix-list) and route maps (route-map) on edge routers. Implement rigorous inbound and outbound filtering so that only local organization-owned prefixes are advertised and routes learned from other clouds are not re-advertised.

C. Asymmetric Routing

• Issue: When asymmetric routing occurs—such as outbound traffic traveling via AWS Direct Connect while return traffic travels via Azure ExpressRoute—packets are dropped by stateful firewalls.
• Mitigation: 🛠️ Use BGP AS-Path Prepending to intentionally extend the AS path length of backup routes, explicitly controlling the primary path. Additionally, adjust Local Preference as necessary to pin the outbound traffic path.

5. Operational Verification Commands and Status Checks

Execution of verification commands validates whether dynamic BGP routing functions correctly across the interconnected network.

Verifying BGP Neighbor Connection Status

# show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 192.168.1.1, local AS number 65001 vrf default
BGP table version 4
RIB entries 7, using 1344 bytes
Peers 3, using 61 KiB

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down     State/PfxRcd   PfxSnt
169.254.100.1   4      64512    1420    1425        0    0    0 23:14:05                5        1
169.254.100.5   4      64512    1418    1422        0    0    0 23:12:10                5        1
10.0.0.2        4      12076     980     985        0    0    0 08:45:12                8        1

Verifying Learned BGP Routes

# show ip route bgp

Codes: K - kernel route, C - connected, S - static, R - RIP,
       B - BGP, O - OSPF, IA - OSPF inter area,
       V - VPNv4, NHRP - Next Hop Resolution Protocol

B>* 172.16.0.0/16 [20/0] via 169.254.100.1, eth1, weight 1, 23:14:10
  *                     via 169.254.100.5, eth2, weight 1, 23:12:15
B>* 10.200.0.0/16 [20/0] via 10.0.0.2, eth3, weight 1, 08:45:17

Verifying Symmetry via Path Tracing

$ traceroute 172.16.10.100
traceroute to 172.16.10.100 (172.16.10.100), 30 hops max, 60 byte packets
 1  192.168.1.254 (192.168.1.254)  0.421 ms  0.388 ms  0.352 ms
 2  169.254.100.1 (169.254.100.1)  2.114 ms  2.085 ms  2.051 ms
 3  172.16.10.100 (172.16.10.100)  3.452 ms  3.411 ms  3.389 ms

6. Operational Notes

Implementing dynamic routing in a multi-cloud environment serves not only to ensure connectivity, but also acts as the foundation for operational simplification and automated failure recovery. Optimizing BGP keepalive timers and hold times (e.g., in conjunction with Bidirectional Forwarding Detection (BFD)) enables sub-second fast failover in the event of a physical link failure. Standardizing prefix filtering and enforcing strict AS path control to maintain a predictable, resilient multi-cloud network is the key to ensuring the reliability of the entire infrastructure.