‘Site’ propery becomes “path affinity group” (“PAG”) property
We initially described a “site” as a group of nodes on the network that are within the same metropolitan area as each other. As discussed in the preamble, this is a relatively arbitrary distinction, and one that largely boils down to nodes that are “close enough” that we prefer traffic between them stay within that “site.” From a network perspective though, what actually matters is the latency of the link (edge) connecting the nodes. In our collective practical experience, there is a direct relationship between distance and latency, but ultimately the latency of the transit path is the characteristic that matters. That is why we subseuqently defined “site” as a property of edges rather than a property of nodes.
How “site” was relevant to the scenario in this article’s preamble
The concept of “site”, in the scenario described in the preamble of this article, captured more than one characteristic of the real-world properties that it carried into the ntework architecture. It also quantized those properties in a convenient (if not immediately apparent) manner.
Link (edge) latency
As described in the initial paragraph of this page, all of the nodes in the same real-world physical location (metropolitan area) were classified as being part of the same “site.” This was roughly equivalent to <3ms of latency on the links between any two nodes in that same site. The enterprise in question wasn’t concerned with whether a potential path between two nodes was 1.5ms or 2.3ms when selecting which paths to use; and the “site” implicitly quantized all values between 0 and 3ms in a single “site value.”
Is this generalizable?
The “site” concept from the case-study in the preamble of this paper provides a useful mechanism to address the the real world network design incentives to avoid un-needed high-latency hops between network nodes. It’s worth considering, though, whether there are other properties of nodes (or the edges connecting them) that we might want to express a “locality” preference for? If yes, are those those properties innately hierarchical? Is there value to modeling that hierarchical structure in our graph?
Distance/locality can, obviously, be sub-divided in a hierarchical manner. “Degree of resiliency” is another property that we might want to take into consideration when selecting paths cross the network/graph. Degree of resiliencydoes not suggest an obviously recursive structure, but the related topic of failure domains does. So, the mechanism we’re using for expressing the “site”/distance/latency property of an edge might be adaptable to additional properties (some of which may be structured hierarchically) and the real-world phenomena of site/distance/latency definitely can be structured hierarchically.
Moving forward, we will use the label “path affinity group” rather than “site”.
Two key questions suggest themselves at this point.
How would we represent the “path affinity group” as a hierarchical structure, if we wanted to. And (of course) do we want to?
Hierarchical relationship of PAG “0” to other (0.1, 0.2, 0.3) PAG instances
The topology that we’ve described up to this point used a distinctly hierarchical model for “site” (now path-affinity group), which it inherits from the real-world topology our model is describing. Each of our “workload-hosting” PAGs requires at least one connection to the WAN PAG.
That is why we used “0” as the value for the (parent) “WAN” path-affinity-group”, and “0.n” as the value for the (child) “workload hosting” PAGs. The value represents a “stack” of both the parent’s identifier (“0”) and the child’s identifier (“n”). We now further specify that edges connecting two nodes in PAG 0.n are labeled as “0.n.0”. This lets us indicate with the edge’s label, whether it is connected to two nodes in the same PAG, or to two nodes in different PAGs, and which PAGs the nodes are “located in” (are “part of”).
That structure is trivially extensible if we want to recursively create further subdivisions of the network/graph. If, for instance, our “workload” PAGs consisted of multiple “buildings” in a single metro region, we might prefer to have traffic stay local to a building, when possible, rather than bounce between buildings. (But, that preference might be 10x or 100x less significant than the distinction between “WAN” and site.) Likewise, if a given site had a workload cluster with its own overlay-network with routing gateways “connecting” it to the underlay network, similar dynamics might be applied.
Creating hierarchical path-affinity-group structures in the graph
In the WAN > site > building hierarchy described above, the original (WAN > site) hierarchy is extended by one additional level. The relation of “WAN” to “site” is the same as that of “site” to “building.” We will extend the labeling convention for the “site” property of edges by appending an additional “section” to the label (representing the “building” identifier, if there is one.)
The following figure depicts a modified version of the reference topology graph, incorporating two additional layers of recursion in the “site” property. Site “0.1” (IED=1) is “split” into two sub-sites (0.1.1 and 0.1.2, each with IED=2), one of which (0.1.1) is further divided into two sub-sub-sites (0.1.1.1 and 0.1.1.2, each with IEDs of 3.)
/grphth-13.svg)
The preceding figure also illustrates a new rule of inference for edges. Specifically:
Edges can only connect two nodes at the same layer of the site hierarchy or in immediately adjacent layers of the site hierarchy
That rule only “makes sense” if the “site hierarchy” really is strictly hierarchical. It’s certainly easy to imagine scenarios in which “site” (or any other property we want to model this way) might be strictly hiearchical, or might be much “looser” (allowing edges between nodes in non-neighboring levels of the site hierarchy.)
Better Illustrating the Recursion
The following figure presents an alternate arrangement of the same graph, which more intuitively illustrates the recursive nature of the site hierarchy.
Refining the PAG property
PAG property values provide a unique identifier for each branch of the site hierarchy, as well as an indicator of whether an edge is present in one, or in two different levels PAG hierarchy.
These are the rules we will follow in constructing the PAG values for our graphs
- We introduce an “implicit”/non-extant “PAG root” node to each graph, representing the root of the PAG tree/hierarchy
- We introduce this “implicit” PAG root in order to allow our graph to represent fully disjoint networks
- We introduce “implicit”/non-extant “PAG root” edges to each graph, representing the parent/child relationship between the PAG-root, and nodes that it shares edges with
- Each of these implicit egdges (and the attached nodes) represents a completely disjoint PAG hierarchy (hence, a completely disjoint graph/network)
- These implicit edges are not eligible for use in paths across the graph
- Nodes that are connected to (share an implicit edge with) the implicit “PAG root” node are assigned a PAG value of “0.n”, where n is an index value of such nodes.
- They are regarded as having an integral path depth (IPD) of 1.
-
Nodes that are attached to nodes with an IPD of 1 and are not attached to the implicit root node
- Each edge property value is expressed as a decimal-point delimited sequence of octets in decimal notation.
- E.g.: “0.n.n.n.0”, or “0.n” (where “n”s are non-zero integers )
- The first octet is always “0”, representing the nominal/implicit “root” of the SP zone hierarchy.
- Subsequent octets represent child/branch layers of the PAG hierarchy.
- A non-zero integer value in these octets represents a distinct “branch” in the PAG tree, with no shared edges.
- A zero value in any octet (other than the initial/root octet) represents
- Labels are formatted as decimal-point delimited decimal octets, each beginning with “0.”
- Each non-zero octet (read left-to-right) represents a level of the site hierarchy
- E.g.: In “0.n.m.0”, the first “0” represents the implicit root of the site hierarchy, and
- “n” represents an instance of the graph-local “top”/root level of the site hierarchy (the WAN, in this exercise)
- “m” represents an instance of the graph-local 2nd-level of the site hierarchy (one of three workload-hosting sites, in this exercise.)
- E.g.: In “0.n.m.0”, the first “0” represents the implicit root of the site hierarchy, and
- All edge labels begin with “0.” ()
- Edge
)
At this point, we will adjust our model and begin treating the path-affinity-group (PAG)property as a property of the nodes in our graph, rather then a property of the edges. The reasons for doing so relate to the fact that what we are modeling in our graph is the behavior of a network of eBGP routers, and the nuances that we want to capture are going to end up being expressed as properties of the nodes themselves in BGP updates.
Nope – don’t do that (^). Keep the PAG property as a property of the edges, where it belongs. Establish a mechanism for establishing a “derived PAG value” for nodes (but it’s still ultimately a function of their edges.