multi-tenant VDOMs across a transparent routing VDOM with unique WAN VLANs per tenant...

[bkaplun]

Hi,

Here is the scenario -

I have a pair of 900D's I'm building out (active/passive) to replace a pair of 100D's. The Fortigates are connected upstream through a pair of Nexus switches and up to a Cisco ASR. This is a multitenant environment, with tenants having their own VDOMs on the Fortigates and their own Internet-facing gateway interfaces on the ASR w/ unique VLANs per tenant trunked down to the Fortigates. Currently on the 100D's, each tenant VDOM either has multiple physical interfaces (LAN and WAN) or a single physical interface with LAN and WAN traffic passing through it. I want to change this. I'd like to consolidate all WAN traffic into a single 10gig port on the 900D, thereby eliminating the physical WAN interfaces in each VDOM and breaking apart LAN/WAN traffic in VDOMs where it's currently all ingressing into and egressing from the same interface. So basically each tenant VDOM would only be left with its LAN facing physical interface(s).

So far I've had some trouble finding a good way to make this work -

Initially I tried to use a transparent routing VDOM (connected to our upstream infrastructure) in front of the tenant VDOMs (running in NAT mode) along with manually created inter-VDOM links to pass WAN traffic. Specifically, I tried to configure WAN IP addresses on the tenant side of each inter-VDOM link and static route WAN traffic across the link to each tenant's gateway on the upstream ASR. This went south when I realized that there was seemingly no way for me to use VLAN tags across inter-VDOM links.

Then I learned about EMAC VLANs and the npu-vlink interfaces. So I updated the 900D's to 6.0.4 and reconfigured things to use EMAC VLANs with inter-VDOM links for each tenant VDOM hanging off of the npu-vlink interfaces. This allowed me to tag both ends of the inter-VDOM link, put a WAN-facing public IP address on the tenant side of each link, and presumably pass tagged traffic down to the Fortigate, across the transparent routing VDOM, across the inter-VDOM links, and down to the appropriate tenant VDOMs. However, I couldn't get this to work either - with both sides of the inter-VDOM links tagged and the appropriate VLAN subinterfaces hanging off of the physical WAN uplink in the transparent routing VDOM, I still had no connectivity upstream and no layer 2 MACs learned in the arp table of either VDOM.

Most recently I reverted to normal (non npu-vlink) inter-VDOM links and tried to use forward domains to make this work. However, I'm not sure I entirely understand how to properly use forward domains with inter-VDOM links when one side of the link is in a transparent VDOM and the other side is in a NAT VDOM. How do I match the forward domain on the NAT side of the link? I still saw no MACs learned in the arp tables when I tried this.

--

I'm still testing and trying different things but I'm wondeing if anybody around here knows the proper way to approach this kind of design? I feel like I'm kind of stubmling around in the dark at this point.

Thanks.

[bkaplun]

After some more trial and error, seems like I have this working the way I want now - thanks for reading.