I not long ago posted a website about the prior blogs I’d created about SD-Accessibility/DNA Heart structure and some implementation information. My intent there and in this website is to update y’all with some more modern discussions I and other people have been having.

Extra recently, I/we also posted an update about SD-Obtain (“SDA”) Web pages.

One particular other matter appears to be to retain coming up, both equally in an SDA context and in a general context: the Online relationship, which all people appears to be to do in another way. And at times improperly if the intent is Superior Availability (“HA”). It’s possible we need to call it “The Kluge Zone” (with eerie tunes accompanying it).

For SDA, this can be a deep structure matter if you’re using SDA Transit.

One exit is relatively easy. It has to be. LISP will supply what quantities to a default route expressing, in result, “to depart SD-Obtain land, tunnel website traffic to this Border Node (or HA pair).” That tunnel generally terminates in the facts heart in relatively near proximity to your Internet exit path. If your information centre has other things (like a web-site or two) between it and the Net exit, that might need to have further assumed. That circumstance tends to be exceptional (or I’d like to think so). And the factors described down below even now utilize. With SDA Transit, you even now get to decide a person or extra default exits, which may well be your World-wide-web blocks or your facts heart blocks, but it simply cannot be equally.

When you have two web pages delivering Online exit, issues get extra challenging. If you want failover, there are some basic but non-obvious means to make it.

A function I’d extensive awaited from Cisco referred to as “border prioritization” was crafted but reportedly has been buggy for a even though. The strategy (as I understood it) was to be able to specify the border website you favor for Net egress, with failover to the other internet site. The very last I listened to was do the job is becoming carried out to repair that and that a aspect referred to as regional above-ride may well be coming sooner that could possibly be applicable. The two features do not come up in Google or Cisco look for, presumably given that not officially supported however. You will will need to speak to Cisco SDA experts if possibly of these options audio interesting or vital for your style. I’m certainly in no position to chat about internal Cisco get the job done in development.

Late update: Cisco HAS been chatting, e.g. at Amsterdam, about a precedence element: think all round principal and fallback exits.

Since this was initial drafted, I have discovered out about a brand-new Cisco element for SDA referred to as “Affinity”. It was apparently introduced in October 2022 and was seemingly also new for CiscoLive Amsterdam. I’ll submit a separate blog site about it as a abide by-on to this a person. If this is urgent for you, talk to your Cisco TME.

So, what can you do with properly-founded and common, reliable technology?

How Do I Support Dual Exits?

All is not misplaced!

With a modicum of large-speed interconnection, you can get a substantial degree of multi-web-site large availability relatively merely.

The most straightforward reply is arguably to run IP Transit. AKA “VRF-Lite on all underlay links”. Not much too bad if you have only 2-4 VNs/VRFs. Messy if you have much more.

What that purchases you is the ability to opt for which prefixes and metrics to use to direct visitors to in which you want it. I like BGP for that, myself. Link point out protocols like OSPF and IS-IS make website traffic engineering a bit more challenging.

Dual Exit Diagram

Here is a new, improved edition of a diagram I have utilized ahead of, built in particular to explore this scenario.

Note: I drew one strains to reduce litter. In actuality, the vertical pairs at the two web pages would be linked by two or four cross-backlinks (“rectangle or bowtie”). For that matter, the horizontal back links (strains in the diagram) would also be two or four inbound links.

The plan is to operate SDA Transit into two pairs of “External Border Nodes”. A single pair in every of the two info centers: proven on the remaining over. The purple highlights where VRF-Lite is jogging (or “IP transit” – I stay away from that term for this localized circumstance considering the fact that it appears to add unneeded obscurity). The VRF-Lite operates into the left facet of the Fusion Firewalls (“FFWs”). All the things in the FFWs and to their suitable is worldwide routing only.

The purple main switches are just VRFs on the very same actual physical switches that are revealed with environmentally friendly highlights. Or they can be separate switches if that helps make you truly feel much better, you want simplicity, or you like contributing to Cisco’s bottom line.

I attract the diagram this way given that every little thing else I have experimented with finishes up with triangles in it or just appears to be unusual. (E.g., stacking the purple and eco-friendly core switches in the diagram.)

Why do it this way?

Well, the vital stage is that it de-partners the SDA network from the firewalls and edge. Fundamentally, SDA Transit receives your website traffic to one of the data centers, and if there is a firewall issue, you can shunt the website traffic to the other information center, and again if essential/sought after.

The style and design concern at perform listed here is preserving firewall state for flows and return site visitors. Especially prolonged-lived flows.

For this structure, I would operate BGP on the FFWs, so that if there is a trouble, routing on the two left and suitable sides will shift stateful targeted visitors to the other web page. (Information still left to the reader.)

You then have the choice of operating with a desired web site or not. Considering that the FFWs can call for a great deal of capacity, they may perhaps be expensive, which will develop the urge to use them. If you select to do that, you have all the common routing equipment to operate with, instead than having to count on SDA features that may not exist or are new and quite possibly buggy.

LISP Pub/Sub

My being familiar with is that SDA LISP Pub/Sub at present offers you the capacity to designate Internet exits, and spherical-robins if there are multiple these kinds of web-sites.

The higher than style with its decoupling allows you have twin web-site HA and “fix” the routing to do what you experience you need to have. Solitary most well-liked exit for simplicity, or dual with extra complexity.

About the FFWs

It seems like I must contain a reminder about what the FFWs are there for.

Their main goal is to control targeted visitors heading among VNs/VRFs. That’s presumably why you created the VRFs in the first position: you wished to isolate friends, IOT, PCI targeted visitors, whichever.

The 2nd objective is as one feasible point to regulate person to server access, a typical have to have unless you’re probably performing ACLs in some kind in VMware NSX. Or less usually, ACI. That is also likely how you filter server to server flows.

Typically, sites I have found currently have an outdoors firewall pair, controlling server to World wide web site visitors, and commonly user to World wide web as effectively.

Failure Modes

Assuming fair routing, etcetera., a solitary system failure or one url failure shouldn’t be a difficulty. There is a Good deal of redundancy there!

Let us study some failure modes.

  • Exterior border pair failure or cut off from underlay at one web page.

IP Transit or LISP Failover would be necessary. IP Transit would be your common IGP or BGP failover.

LISP with out Pub/Sub appeared to NOT are unsuccessful over, or at minimum only pretty slowly but surely, the final time I tested it. LISP Pub/Sub is new considering that the past time I experienced the prospect to do screening.

  • Border pair to FFW, FFW, or FFW to core switches failure.

With dynamic routing (with the FFW participating), site visitors should really go by using the other details center. If you have a desired exit via the Web at the details center with the failure(s), visitors should go back again across the crosslink, and so on. (Sub-optimum, but beats participating in games with, or waiting around for Web failover?)

And if you seem carefully, the total FFW block tale turns into the similar as the core switch/World wide web firewalls/outer switches (or routers) story, as significantly as routing and failover.

In limited, this method de-partners how the packets get to one particular of the facts facilities (IP Transit or LISP-based mostly SDA Transit) from the total firewalling and Internet routing problem. Which is a excellent factor. Modular design, minimized complexity.


Be aware that the higher than assumes the two exits are reasonably nearby, so that latency is a non-problem. If they are significantly aside, then you will in all probability want location-aware exit priorities, which is a summary of what Affinity can do.

What I simply cannot very easily convey to you how to do is obtain load-sharing across exit complexes. The challenge to load-sharing is preserving firewall point out. Policy routing based on source IP block appears to be like it may be turned into a workable alternative.

If you’re inclined to do cross-website firewall clustering, then stateful return paths is no huge issue. I’m recreation for localized firewall clustering, but cross-web-site clustering strikes me as including complexity and hazard (new failure modes) fairly than growing availability.

Yeah, it would let use of the secondary web site units and back links, neither of which is very likely affordable.

Disadvantage: if the cross-inbound links fail in between internet sites, you nevertheless have a dilemma. Set differently, incorporating cross-site failover to a routing scheme that takes advantage of both equally Net exits statefully provides a further increment of complexity. Do you definitely want to go there?

Late Be aware: Affinity appears to be to assist handle this!

Versions on This Concept

Static routes – just say no.

If you use community addressing internally and do NAT on the firewalls, to two distinct community blocks that are externally advertised (i.e. 2 x /24 at a minimum amount), then that possibly simplifies state preservation. Marketing a /23 and one particular of the /24s out of each internet site could possibly be useful in that circumstance.

I really don’t have any other thoughts at the second. In particular kinds that stay away from complexity.

Require A lot more Control?

It is significantly less tasteful, and most likely more configuration operate, but IP Transit could be a severe  consideration here. In particular if you don’t be expecting to have more than a few of VNs (VRFs) in your SD-Accessibility network. (Be certain, for the reason that later on altering from IP Transit to SDA Transit appears like it would be agonizing?)

What that buys you is a more substantial set of routing instruments for which targeted traffic goes where.


I hope this receives you pondering about your Web exits and redundancy, and what your failover method is.

I’ve observed a good deal of models in which Web dual-site automatic failover hasn’t been implemented, or only operates for some failure modes. Also, I and others utilized to not believe in dynamic routing on firewalls (or not belief the firewall admins), maybe thanks to unpleasant CheckPoint encounters. But all that might have modified with out us noticing!

Many thanks to David Yarashus and Mike Kelsen (and J.T.) for conversations and insights all around this matter.


Disclosure Assertion


Supply link