ISP Network Design

I have been reading recently quite a bit in regards to ISP core network designs. In this blog, I wanted to add some structure to notes I have been taking for the past few days on this topic.

This blog is not to be comprehensively outline everything that goes on into designing an ISP network but I will be keeping this blog up to date for future reference.


CORE NETWORK HIERARCHY

  1. Core – this is what is most commonly known as the backbone network. It should be populated with high speed links and high level of link resiliency. The purpose of the core is to deliver data packets quickly and reliably.
  2. The Distribution layer is, putting it in simple terms, the PoP* aggregation. This layer would normally require a high port density for here all other modules connect to.
  3. Aggregation – I feel that here, better naming would have helped as this is in fact the Access layer within the ISP core network. Typically, border routers would sit within this layer and connect upstream to the Distribution layer.

POP DESIGN

As with any network design, a modular approach is recommended – this promotes manageability, scalability, fault isolation, performance and security.Each module would be define base on different criteria such as:

  • connectivity requirements
  • services availability
  • security
  • scalability
  • oversubscription ratios – examples:
    • 2:1 ratio for high value business customers
    • 6:1 normal business customer
    • 30:1 consumers
Oversubscription ratios in the enterprise should be:

  • 20:1 – Access to Distribution
  • 4:1 Distribution to Core

However, by deploying a collapsed core design, the contention ratio would be 20:1 instead. Though within ISP world, different ratios could be applicable and therefore acceptable. It would really depend on what type of customers are expected to connect to the core, what services would they be using, etc.

With the proliferation of  1G and 10G links (servers, appliances, …) into the access layer, becomes increasingly difficult to define what ratios are acceptable – hence proper network monitoring is required to backup capacity planning.

  • Furthermore, one should consider spreading services across multiple POPs for resiliency purposes.
  • ISPs should not use firewalls in the core. The sole purpose of the core is to serve as a fast and reliable transit network for all packets. For security purposes, dedicated core infrastructure provides built-in security features, including packet filtering capabilities – these should be used instead.

BACKBONE DESIGN

  • A routed backbone should be deployed
  • Each POP should have two core routers for resiliency purposes
  • It is recommendable that MPLS is deployed
  • Plan for at least 20% extra link bandwidth capacity; ideally, 50% – 60% is recommendable
  • Deploy link and device redundancy

ROUTING PROTOCOL DESIGN

Modularity should also be considered when designing the ISP internal routing protocols. Hierarchic protocols such as OSPF and ISIS provide built-in feature to achieve this. For example:

  • Consider routing each POP within its own dedicated routing area (for example – Area 0, if using OSPF as an IGP)
  • Aggregate or summarise where possible
    • DO NOT summarise individual customer assigned ranges
  • In most cases, accepting a full routing table will be preferred to partial or default routing. A full Internet routing table provides more flexibility, routing predictability and control.
  • In large environments, consider using iBGP route reflectors. Route reflectors could be used in the Core, Distribution or both. Routers part of the access layer will be route reflector clients – ie. assuming the distribution layer routers are acting as RR servers. This is illustrated below:

  • Do not aggregate prefixes from different customers
  • In the core, we should aim to minimise IGP prefixes to promote scalability and rapid convergence. Therefore:
    • Do not carry Internet prefixes within the IGP (do not distribute BGP into IGP)
    • Do not carry Customer prefixes within the IGP (do not distribute IGP into BGP either!)
    • Customer prefixes should be carried in iBGP
    • IGP should only carry infrastructure IPs only, including loopbacks
  • Configuration notes:
    • Routers are configured with loopback addresses
    • iBGP sessions must be configured between loopbacks and NOT “real” interface IPs
    • IGP should carry loopbacks as host IPs (/32)
    • Use next-hop-self
    • Consider limiting AS_PATH length to no longer than 20 AS-es
    • Implement TTL “hack” – this will add an extra layer of security to iBGP sessions
    • Hardwire to BGP version 4
    • Always send communities in BGP updates
    • Make use of communities – they provide increased scalability and BGP policies are easier to apply and maintain
    • Unless you know what you are doing – do not use BGP damping
    • Remove private AS-es from announcement
    • Use AS_PATH based filters as a backup to prefix filters
    • Implement maximum-prefix tracking
    • Log neighbour state changes
    • Filter BOGONS – though keep in mind this is a dynamic list and therefore, must be maintained
    • Enable unicast reverse-path check at all customer facing interfaces
    • Check MANRS website for further security best practices

TRANSIT / PEERING

Whilst clear distinctions exist between these two concepts, in my opinion, they are very similar – i.e. both are IP interconnections between two ISPs. And in fact, I would go further to say that, three types of peering agreements exist:

Transit agreement

  • One smaller ISP uses a larger ISP as a transit network to reach the Internet
  • Cross connect within the same building (Datacenter) or over long distance circuits (far more costly)
  • Bound to an SLA contract
  • A minimum of two; maximum of three transit providers is best practice
  • Routing policy:
    • receive default route (or range to be used as default route)
    • or, receive Internet routing table – particularly when multi-homing

Private peering

  • Makes sense for transferring large amounts of data
  • Typically, it is a cross-connect between two entities which are both located within the same DC. In theory, private peering is also possible over long distance circuits – though circuit costs would quickly escalate
  • Not free; the two parties agree to share the cost of the circuit
  • The routing policy would be agreed between the two entities

Public peering 

  • It is free in most cases
  • Occurs at IXP (Internet exchange points)
  • Improves Internet connectivity reachability
  • Routing policy:
    • Only domestic BGP prefixes are exchanged
    • default routes should not be accepted or sent out in BGP updates

NOTE: PeeringDB is a recommended reference for peering.

Additionally, DO NOT …

  • … accept / advertise RFC1918 prefixes due to …
    • adverse effects on traceroute functionality
    • issues with PATTH MTU discovery
    • issues anti-spoofing security techniques
    • troubleshooting concerns
  • … accept own prefixes
  • … accept prefixes longer than /24
  • … accept default routes from the upstream peering partner. Here there are some exceptions though generally speaking, if routing is to be done by means of default routing, using a static default-route provides better control

Ok … that’s it for now. If this helps you in anyway, please share and let me know your thoughts.

Thank you,

Rafael A. Couto Cabral • LinkedIn Profile
Cisco​ | F5 | VMware Certified • PRINCE2 Practitioner

Originally posted 2019-04-15 15:47:23.

Related Post

One Comment

  1. Pingback: ISP POP Design | samuelnotes

Comments are closed