Network Topologies- In context of DC

Data centers are crucial components in the functioning of modern businesses. These facilities house large amounts of data and computing resources that are critical for organizations to deliver their services. The architecture of a data center is of utmost importance in ensuring that these resources are accessible, secure, and reliable. One key aspect of data center architecture is network topology, which refers to the way in which the components of a network are arranged.

There are several network topologies that can be used in a data center, each with its own advantages and disadvantages. In this blog post, we'll explore some of the most common topologies and their applications.

Bus Topology

A bus topology consists of a single cable that connects all the devices in a network. Each device is connected to the cable through a T-connector, which splits the cable's signal. This topology is easy to implement and is cost-effective, but it can become slow and unreliable as the number of devices on the network increases. In a data center context, a bus topology is not typically used as it does not scale well.

Star Topology

A star topology consists of a central device, usually a switch or a hub, that connects to all the devices in the network through individual cables. This topology is easy to install, and faults are easy to isolate, making it a popular choice in data center architecture. However, it can be expensive to implement as it requires a large number of cables.

Ring Topology

A ring topology consists of devices that are connected to one another in a circular fashion. Each device is connected to two other devices, creating a ring. Data is transmitted in one direction around the ring, and each device repeats the signal to the next device. This topology is reliable and efficient but can be expensive to implement, and faults can be difficult to isolate.

Mesh Topology

A mesh topology consists of devices that are connected to one another through multiple paths. This topology is highly resilient and fault-tolerant as data can be rerouted if a path fails. It is also highly scalable and can accommodate a large number of devices. However, it can be expensive to implement, and the complexity of the network can make troubleshooting difficult.

Hybrid Topology

A hybrid topology is a combination of two or more topologies. For example, a data center might use a star topology for its access layer and a mesh topology for its core layer. This approach allows for greater flexibility in designing the network to meet specific requirements.

In conclusion, selecting the right network topology is critical for a data center's success. Each topology has its own strengths and weaknesses, and choosing the appropriate topology depends on factors such as scalability, reliability, cost, and ease of maintenance. Ultimately, a well-designed network topology can ensure that a data center functions efficiently, securely, and reliably, meeting the demands of modern businesses.

Copper Cables and Connectors

Active vs Passive cables

• Active cables are copper cables for data transmission that use a silicon chip (semiconductor) to boost the performance of the cable. Without a chip, a cable is considered a 'passive' cable. 
• Passive cables are liable to degrade the data they carry, due to such "channel impairments" as attenuation, crosstalk and group velocity distortion. 
• In active cables, one or several semiconductor chips are embedded in the cable to compensate for some or all of these impairments. 
• This active boosting allows cables to be more compact, thinner, longer and transmit data faster than their passive equivalents. 
• While passive cables are always copper-based, active cables can use either copper wire or fiber optics to provide the link between the cable ends.

Why Twinax is used instead of RJ-45?

• There are two types of Copper Cabling- Twinax and RJ45 Ethernet (Cat6 or higher) 
• For Twinax cabling, the connector is SFP+ and for UTP cat6 cables, the connector used is RJ-45 
• The twinax cable uses a different signal propagation method, my understanding is that it’s more like a radio wave than an electrical signal. A UTP signal needs a lot of electronics to drive generate and receive the signal. 
• The end result is that Twinax uses much less power (something around 1 – 1.5W per port) compared to UTP (4 – 6 W per port). When you have a lot of ports this power consumption can be significant factor in design. 
• Twinax is an ideal 10GbE solution for server to Top of Rack switch connection. Since it uses less power and is more reliable that UTP solution as the cable is physically more robust and not subject to physical damage, an important consideration for 10GbE over copper. Any damage to the copper cable due to crushing or bending can cause intermittent failures on Cat6 cabling (although less likely on Cat6A). 
• But the main reason for choosing twinax for 10GbE is lower power consumption for server to top of rack connections for integrated storage and data networks.

TYPES OF COPPER CABLES:

UTP Cable:
- As the name suggests, this type of cable includes an unshielded media that includes twisted pairs.
- Specifically, UTP includes four different pairs of copper cables that are each twisted together
- The twisted rate depends on the specific category of cable- can be cat5e, cat6, cat6a and better the quality, higher the distance

- It is the most used copper cable which has 4 pairs of copper cables twisted together

STP Cable
- Another type of twisted pair cabling includes a shield that is used to both contain and protect the different twisted pairs from interference. 

- This type of cabling is not commonly seen in modern LAN networks.

- Similar to UTP, but, the pairs are shielded to prevent interference. This is very rare in modern lan environments, but, is still used where high speed low-interference is needed.

- In your daily life, where have you seen any shielded cable? Yes, the one which is used for your cable TV- coaxial cables are shielded

TYPES OF CONNECTORS FOR COPPER:

RJ45-The cable connector that is found on almost all UTP and STP cables is a Registered Jack 45 which is mostly commonly referred to as RJ45.
-RJ45 cables have 8 color-coded wires, and the plugs have 8 pins and conductors. Eight wires are used as 4 pairs, each representing positive and negative polarity.

RJ-45 Straight-Through and Cross-Over connections:

We have seen about what straight-through and cross-over means in this post: http://heartofnetworking.blogspot.com/2018/05/straight-through-and-cross-over-cables.html 

We can make these cables in copper with RJ-45 connectors as shown below:

TRANSCEIVERS:

- We can usually connect the RJ-45 ports into regular RJ-45 jacks on our laptops

- But, on most new switches, it is not possible since the switches do not come with RJ-45 ports

- In such scenarios, we need to use something called as 'Transceivers' which convert the signals from RJ-45 cables into a form that the switch's ports will understand...Think of it like a translator which translates from one language to another.

- Also, most modern switches are designed for both optical and copper. So, they will not be restricted to onlyRJ-45. All modern switches and routers will have something called as SFP/QSFP ports. So, if we want to connect a RJ-45 copper cable to that port, we need to use a SFP transceiver called SFP-1G-T which has SFP ((male) port on one side and RJ-45 jack (female) on another side. We could also use the same SFP port to connect an optical fiber, just by changing the transceiver to one which supports optical connector like LC or SC/ST

SFP-1G-T

Do cables and ports speak the same language?

Transceivers essentially translate the signals from the cable into a form which the switch can understand. Mostly, switches will have SFP/QSFP ports on them. So, for a optical cable, if we use a transceiver, it will translate the light signals to electrical signs for the switch. Similarly, if we use a copper cable, we can use a transceiver for that and connect it to the same port on the switch.

TWINAX CABLES:

- Also, known as DAC cables (Directly attached Copper)

- In datacenter environments, they are used more often than RJ-45 or other UTP cables.

Reasons for DAC cables are used in datacenters much more than UTP/RJ-45?

• It has SFP connectors attached to copper, so no need to use transceivers...you can save cost of transceivers. 
• Second, UTP cables consume lot of power (around 4-6 W per port) whereas twinax cables use much less power (1-1.5 W per port). When you consider hundreds of thousands of ports, this cost saving is significant 
• Also, this cable is more robust than UTP cables and is not subject to physical damage upon bending which prevents loss

10G SFP+ Copper Twinax cable

Optical Fibers and Transceivers

What is physical difference between 10Mbps and 100Mbps cable? Will higher value of this lead to faster data transfer?

Ans- For two cables that are made of identical materials and same length,

• If you think of the Internet as a series of tubes, latency is the length of the tube between two points. Bandwidth is how wide the tube is. Indeed, it’s named bandwidth because it describes the width of the communications band. The wider the tube the more data you can send in parallel.
• The key point here that gets missed is that, regardless of how much data you are sending, you still have to move it the distance from point A to point B. That takes time and that is the latency.
• Having 10 Mbps and 100 Mbps connections does not somehow allow a single bit data of data to travel that a distance any faster.
• A large bandwidth connection simply allows you to send or receive more data in parallel. The data still needs to travel to and from your computer.
• For example,

10 Mbps line:

100 Mbps line:

From above example, we see that the high bandwidth connection downloads the file faster than the low bandwidth connection because more data can travel in parallel. So faster transmission, latency still there.

MULTIMODE (MM) FIBER:

• MMF cables use a larger internal core diameter and can utilize lower cost LEDs for transmission
• Has larger core diameter- allows multiple modes (.ie. allows light to come in at different angles) and multiple wavelengths
• But, since this might cause loss and interference between the different modes over longer distance, it is used only for short distances (upto 2 km)- common in LAN deployments
• MMF cables are usually marked with “aqua blue” or “orange” color jackets

SINGLEMODE FIBER:

• The difference between SMF and MMF is in their physical characteristics
• SMF has a much smaller core diameter (typically 8-10 µm) and accepts signals coming in from a specific angle and on a specific mode only.
• SMF can be used for cable runs of very long distances (typically up to ~40 miles without repeaters depending on wavelength).
• SM cables are usually marked with ‘Yellow’ jackets

Single Mode and Multimode Optical Fibres?

SM	MM
Small diameter (9/125 um)	Large diameter (50/125 um)
Jacket is “yellow” color	Jacket is “aqua” or “orange” color
Long distance	Short distance
Avg distance is 30 km	Avg distance is 550 m
SIngle wavelength	Multiple rays (of same wavelength)
Very costly	Cheaper

• 

QUALITY RATING OF OPTICAL FIBERS:

 Just like we have category of cables in copper that tells us the distance for which we can use those cables, even for optical cables, there are categories

SR stands for short rage. So, shorter the wavelength supported, shorter the range. So, SR supports short wavelenths and have short range. We also see that Singlemode cables have longer range than multimode ones

interconnect	AKA	Wavelenth	Fiber Mode	Max range	Transceiver	Medium
SR	Short Reach	850nm	Multimode	300m	SFP+	Fiber
LR	Long Reach	1310nm	Multimode	10km	SFP+	Fiber
LRM	Long Reach Multimode	1310nm	Multimode	220m	SFP+	Fiber
ER	Extended Reach	1550nm	Singlemode	40km	SFP+	Fiber
ZR	Extended Reach +	1550nm	Singlemode	80km	SFP+	Fiber

How does Optical cable work?

• Fiber Optics is sending signals down hair-thin strands of glass or plastic fiber.
• The light is “guided” down the center of the fiber called the “core”.
• The core is surrounded by a optical material called the “cladding” that traps the light in the core using an optical technique called “total internal reflection.”

• The refractive index of core is greater than cladding
• It can be step-index or graded-index
• It can be OM3 or OM4

TYPES OF CONNECTORS:

Straight Tip (ST):

• The Straight Tip (ST) connector is often seen on the end of a multi-mode cable;
• It is being slowly replaced by multi-fiber connectors (LC and MTP).

SC (Subscriber Connector):

• 
• The Subscriber Connector (SC) can be seen commonly on MMF or SMF
• As with SC connectors, the ST connector is slowly being replaced by multi-fiber connectors. 

What is MTP- 12 and LC cables?

MTP:

• MPO is a 12 fiber connector for ribbon cable. It's main use is for pre-terminated cable assemblies and cabling systems. Here is a 12 fiber MT broken out into 12 STs (ST is the most popular connector for multimode networks). MPO means multi-position optical. This connector is sometimes called a MTP which is a commercial name.  (http://www.thefoa.org/tech/connID.htm)
• MTP or MPO 12 Strand Fiber Assemblies are a high density cables designed to carry one signal which increases bandwidth and reduces space. This plug and play fiber optic assembly is made with Corning fibre.
• The standard specifies MPO12 (or MTP12) as connector to the SR4 QSFP, which employs traditionally 12 fibres, but 40G only need 8 (4 pairs) to carry the 4 parallels bidirectional paths. You might know that QSFPs can be programmed to operate as 4 x 10G.
• When facing an SR4 QSFP or an MTP cable with their key up, those will always receive on their left (your right if you face them).
• They will always send on their right (your left if you face them).   
• It is larger than the other connectors but for good reason - it can support up to 24 fibers in a single ferrule.
• In high density patch environments such as datacenters they are used extensively, both at single mode and multi-mode wavelengths.
• Because the sequence of the fibers cannot physically be changed after termination, the connector is often supplied with a fan-out assembly at the opposing end (such as LC, SC FC etc.). This allows the operator to change channels simply by re-patching the fanned-out side of the cable. The consequence of this is that the small form high density design of the MTP/MPO will only benefit one side of the assembly.

LC:

• LC (Lucent Connector) fiber optic patch cables are used to send high-speed data transmissions throughout your network.
• LC fiber optic cables connect two components with fiber optic connectors. A light signal is transmitted so there is no outside electrical interference.
• Good performance, highly favored for singlemode.
• Its small size gives it huge popularity in datacoms and other high-density patch applications, as its combination of small size and latch feature make it ideal for densely populated racks/panels.
• 

HENCE,

In practice, one end is a MTP end and other end, we have many LC cables and they are called MTP- LC cables.

How is 10 Gbps SFP different from 40 Gbps SFP?

Note: SFP means Small form-factor pluggable (SFP) is a specification for a new generation of optical modular transceivers. The devices are designed for use with small form factor (SFF) connectors, and offer high speed and physical compactness. They are hot-swappable.

• SFP -> 1G and SFP+ -> 10G
• But, QSFP -> 40G (technically, it is QSFP+)

10G:

• The transceivers support 10 Gbps bandwidth
• It uses a single strand of line
• We use LC cables as it supports 10G transmission and reception on single cable

40G:

• It operates as 4x10G or 40G- both methods
• If 4x10G, each pair acts as separate 10G cables and can be used individually as 10G links
• If 40G, the entire link together acts as 40G link.
• We use MTP- 12 for 40G

Universal Transceivers?

• Uses the same port and cable, if we only change the transceiver, we can get universal transceiver which supports both 10G and 40G
• Supports both single mode and multimode depending on our requirements of distance

Console vs Management:

Console	Management
No ip address	Has ip address
Needs physical access	Allows remote access using SSH, telnet,..
Serial	Ethernet (RJ-45)
Very slow (128 kbps). But, when running in command line we won’t notice	Fast (1 Gbps bandwidth)
We use rollover cables	We use normal ethernet cables
Blue and grey flat cables are used for console	Yellow flat cables are used for management port

Breakout Cables:

• Each 40G MTP-12 is split into 4x10G LC cables. Thus, it can operate only at 10G
• But, it gives higher port density and takes less space
• Thus, by using only one physical port, we get 4 ports of 10G each

Note:

• Cat6 and Cat7 can support 10G .ie. we can get 10G speeds on normal ethernet cables
• If a transceiver is attached to a wire, only then, it is called cable

Difference between OM3 and OM4

OM stands for optical multi-mode. Both OM3 and OM4 are  50/125 core fiber but they have different internal construction that allows the OM4 fiber to provide the same performance as OM3 but for longer distances. The reason for this is the difference in bandwidth, OM3 has 2500 megahertz bandwidth , OM4 has 4700 megahertz bandwidth. What this translates to is longer transmission distances for the OM4 fiber. OM4 cable has a lower attenuation and operates at a higher modal bandwidth. Both these factors mean that less power is lost in the transmission of the signal and that means the signal can be transmitted further or through more connector pairs (which add to the losses).

Universal transceiver- how does it support different speeds or bandwidth?

https://www.rp-photonics.com/bandwidth.html

Bandwidth is given by:

In SM fibres, the wavelength is varied from 1490 - 1625 nm

In SM fibres, the wavelength is varied from 850 - 1300 nm

Why is SM fibre expensive than MM fibre? Which of these should be used inside a data center?

SM fibre is more expensive as it is made of pure glass.

Inside a datacenter, MM fibres can be used and in between two data centers, we can use SM. This is because, MMF fiber cabling has a shorter distance, thus easier to manufacture, more commonly used and less expensive.