Arista 7050X series switches have some undocumented commands, which allow you to unlock more bandwidth for recirculation without disabling physical ports. Read about how and why.
If you already know those switches and about the VXLAN limitations of Trident-II ASICs, skip this section and head straight to the Arista’s implementation of Recirculation on 7050X series and unlocking more recirculation bandwidth section or if you’re in a hurry, head to TL;DR
I have not yet verified that unlocking more recirculation bandwidth actually provides a benefit
I don’t see why it wouldn’t, but I simply haven’t had the time to set up a VXLAN routing setup and monitor proper traffic distribution via
show recirc-channel traffic yet.Arista 7050X series, Trident II and Recirculation
The Arista 7050X Series
Overview
The Arista 7050X series is a series of switches from Arista introduced in November 2013 based on the Broadcom Trident-II ASIC (Broadcom Trident-II BCM56850 “Datasheet”). Arista offers various models with different port configurations, the most common you will find are the classic core / spine 32x40G QSFP+ switches (7050QX-32 / 7050QX-32S), the classic ToR 48x10G SFP+ + 4-6x 40G QSFP+ switches (7050SX-64 / 7050SX-72Q) and the RJ45 manifestation of the ToR switches with 48x10G SFP+ + 4-6x 40G QSFP+ (7050TX-64 / 7050TX-72Q). Some more exotic models are the 7050TX-96 or even the 7050TX-128 / 7050SX-128. There is also the 7250X series which is also based on the Trident II, but offers even more ports at the expensive of massive power consumption, as they use not one but six Trident-II ASICs in a crossbar configuration.
Why is the 7050X series “relevant”?
So, why do I even care about the 7050X series? After all, those switches have been introduced over 10 years ago (at time of writing), in November 2013. That’s pretty old for any kind of hardware, isn’t it?
Yes, the switches are old. But they are still good! For many applications, 40G / 10G is still plenty of bandwidth. Additionally, Arista has done a great job of supporting this product. They are supported up to the EOS 4.28 release train (we are now at 4.33), however even though they don’t get any new features, they still get security updates (e.g. EOS 4.28.12M!). Those switches are also quite powerful feature-wise, although there are some limitations due to their age (this is the reason for this blog post).
Furthermore, the Arista 7050X series is readily available on the second hand market. A good amount exists on the market, making them easy to find and driving down their price (aka: they are cheap!). All those points make the Arista 7050X series quite popular amongst small & medium business and homelabbers.
Surprisingly, the fan control is also really good compared to other vendors. They are really, really silent for a datacenter switch (or can be asked to be silent by overriding the fan value - no need for sketchy fan swaps!)
The downside is of course that some of their features have limitations due to the age of the used Broadcom Trident-II ASIC and their power consumption is not great (which can REALLY hurt if you’re from Western Europe).
I for myself prefer them over another popular alternative, the Mellanox SwitchX series.
Why I prefer Arista 7050X series over Mellanox SwitchX
Why I prefer Arista 7050X series over Mellanox SwitchX
The Mellanox SwitchX series
Mellanox SwitchX is another popular alternative to the 7050X series, especially in homelabs.
Mellanox SwitchX series primarily impress with their extremely low power consumption (35W in IDLE for 36x40G / 36x56!). Together with Mellanox NICs, they even offer 56GBit/s instead of 40GBit/s over QSFP+ ports, a proprietary Extension from Mellanox. Moreover they come in interesting form factors, like half-width SX6012, where you can place two half-width switches in a single rack unit!
Additionally, Mellanox SwitchX switches offer a unique capability called “VPI” (Virtual Protocol Interconnect): Mellanox SwitchX is able to run Ethernet AND Infiniband at the same time on the same switch even route between them. That immediately nerdsniped me back in the days.
Finally, those switches are quite hackable, allowing different license unlock, crossflash or conversion procedures.
Mellanox SwitchX switches come in many different models (although many of them are interconvertible and differences are only software).
- Available Infiniband with optional Ethernet (SX60XX) or Ethernet (SX10XX)
- Available with various port configurations, such as
- 36x 40G/56G QSFP+ in 1RU
- 18x 40G/56G in 1RU
- 12x 40G/56G QSFP+ in 1RU half width
- 48x 10G SFP+ + 12x 40G/56G QSFP+ in 1RU
- 64x 10G SFP+ in 1RU
- Available with PowerPC, x86 control plane or completely without control plane (Infiniband only)
- Available with long and short chassis
- Available with different airflow directions
The most popular models
- SX6036 / SX1036: 36x 40G/56G QSFP+ in 1RU, PowerPC control plane, Infiniband & Ethernet possible - Some artificially limited versions like SX6036T exist.
- SX6012 / SX1012: 12x 40G/56G QSFP+ in 1RU half-width, PowerPC control plane, Infiniband & Ethernet possible
- Warning: EMC OEM models exist which require a conversion procedure
Some less popular models
- SX1024: 48x 10G SFP+ + 12x 40G/56G QSFP+ in 1RU, PowerPC control plane, Ethernet only
- SX1016: 64x 10G SFP+ in 1RU, PowerPC control plane, Ethernet only
- SX6710: 36x 40G/56G QSFP+ in 1RU, x86 control plane, new style chassis, Infiniband & Ethernet possible
- SX6720: 36x 40G/56G QSFP+ in 1RU, x86 control plane, new style chassis, weird special PSUs with integrated battery backup unit, Infiniband & Ethernet possible
- SX1710: 36x 40G/56G QSFP+ in 1RU, x86 control plane, new style chassis, Infiniband & Ethernet possible
- SX6790: 36x 40G/56G QSFP+ in 1RU, unmanaged / no control plane, Infiniband only
- SX6018: 18x 40G/56G QSFP+ in 1RU, PowerPC control plane, Infiniband & Ethernet
- SX6025: 36x 40G/56G QSFP+ in 1RU, unmanaged / no control plane, Infiniband only
- SX6015: 18x 40G/56G QSFP+ in 1RU, unmanaged / no control plane, Infiniband only
- SX6005: 12x 40G/56G QSFP+ in 1RU, unmanaged / no control plane, Infiniband only
- SX1700: 36x 40G/56G in 1RU, ??? - only exists as Engineering Sample apparently
The products can have either the SwitchX ASIC or the SwitchX-2 ASIC - only a slight change in the PN indicates which is which. The SwitchX-2 ASIC came later obviously, however nobody really knows what changed and I couldn’t find any benefits.
I’m not going to cover any licensing intricacies, crossflash or conversion procedures here. That’s something for a separate blog post.
The actual reason why I prefer Arista 7050X over Mellanox SwitchX
- Arista 7050X has far more features
- Arista 7050X can do VXLAN, no chance on the Mellanox
- Mellanox lacks some capabilities like setting ACLs on SVIs…
- Arista 7050X has far newer software
- Highest MLNX-OS / Onyx for SwitchX / SwitchX-2 based switches is 3.6.8012, which was released in early 2019 but this was only a bug fix release
- I wouldn’t exactly trust the BGP implementation in this old software, for example
- Mellanox has known bugs in their switches / old software
- Using MLAG with MAGP can cause random datacenter meltdowns, passing traffic only at below 1MBit/sec (classic MLAG meltdown) - Ask me how I know
- The PowerPC control plane of the classic SX60XX / SX10XX switches is extremely annoying
- The control plane is slooooooow
- I wouldn’t trust the control plane to be fast enough for more than a handful of BGP sessions
The 7050X series is a clear winner for my - or rather: My use cases.
Don’t get me wrong: Mellanox SwitchX are great switches if you’re a homelabber or your use case is very basic. They can do VLANs, wirespeed Inter-VLAN routing, basic ACLs and MLAG (just without MAGP please..) while using a minimum amount of power and making very little noise.
The Trident-II ASIC
Trident-II is an ASIC from Broadcom, announced in 2012. It is built on a 40nm process and offers 12MBytes of packet buffer. The Trident-II has 128 SERDES running at 10GBit/s, resulting in a total maximum bandwidth of 1.28TBit/s (or 2.56TBit/s if you’re Juniper :D ). The Trident-II supports an absolute maximum of 104 ports (typically 96x 10G + 8x40G, resulting in 96 + 8x4 = 128 Lanes).
The Trident sits somewhere in the middle ground of Broadcom’s Merchant Silicon lineup. On the one side we have the Tomahawk series. The Tomahawk series was built for Hyperscalers and has only one job: Push as many packet as possible for as little money as possible. This results in generally less features (like smaller tables, less programmability, etc). On the other side we have the Jericho series (with its sibling Qumran). The Jericho series is built for service providers and offers advanced features like very deep buffers (in the Multi-Gigabyte range), very large tables for e.g. routes and a high degree of programmability. Trident is somewhere in the middle, with decently sized tables (not as small as Tomahawk’s, but also not as big as Jericho’s), decently programmability, etc…
There is not a lot of information available on the Trident-II ASIC, which is typical for network switch vendors and especially for Broadcom. However there is an interesting Whitepaper from Arista (Arista 7050X & 7050X2 Switch Architecture Whitepaper (‘A day in the life of a packet’)), that reveals some interesting details.
Trident-II has three modes of operation: Performance, 10GbE Latency and Recirculation. Those are marketing terms though. Performance mode is not really a Performance mode. It is more like a “Maximum-Scale mode”, allowing all 128 lanes to be used as data ports at the cost of converting 8 of the interfaces into fixed 40G interfaces (no breakout possible) to stay within the 104-port limit and falling back to the higher latency Store and Forward for 10G to 10G forwarding. 10GbE Latency mode goes an even more extreme route. In 10GbE you sacrifice 8 full QSFP+ interfaces (32 lanes!) in exchange for 96 lanes with flexible configuration (you could break out all of them and have 96 separate ports, don’t break them out and have 24 40G QSFP+ ports or any mixture) and Cut-through forwarding for 10GbE to 10GbE.
There is a third mode which I have not covered yet: Recirculation. This is where the fun begins
Recirculation
When the Broadcom Trident-II ASIC was announced in 2012, the VXLAN boom was just picking up pace. Broadcom thus added half-baked early / partial support for VXLAN / NVGRE. There are a lot of possible reasons why the VXLAN capability in Trident-II is not complete. It could be that it was simply too early - Trident-II predates the final VXLAN specification. It could be that they could not reasonably implement it due to chip size limitations and the 40nm process, or power limitations and the 40nm process, or maybe they did not think it would take off - we don’t know.
But why am I saying that the support is only early / partial? The Trident-II does not support VXLAN Routing (or RIOT / Routing In and Out of Tunnels) without a workaround. That means you cannot deploy EVPN-VXLAN with Symmetric Routing, and have to fall back to either Centralized Routing or even worse, Asymmetric Routing. Both alternatives are not ideal. Note: Mellanox Spectrum A0 suffers from the same issue, but there’s not even a workaround available! Very annoying. This was fixed in Spectrum A1.
So what is this workaround I mentioned? It is called recirculation. Recirculation means that the packet is sent through the packet processing pipeline twice. This second pass allows the Trident-II to implement VXLAN Routing.
This comes at a cost: We cannot magically summon more computing power. We have to sacrifice physical ports (similar to 10GbE Latency mode). Their resources will be used to implement recirculation. The only glimmer of light is that we don’t have to physically create any kind of loopback connection with cables or the like. That is done internally in the ASIC. It also comes at the cost of falling back to Store and Forward for 10GbE to 10GbE forwarding (so we get the disadvantages of both Performance mode with Store and Forward for 10GbE to 10GbE forwarding and 10GbE Latency mode with 8 disabled interfaces)
Trident-II’s successors, like Trident-II+ [Note: apparently Trident-II+ has some other new minot limitations, which I don’t really understand] or Trident 3 don’t suffer from this issue, they have enough resources in the processing pipeline to not require recirculation.
Arista’s implementation of Recirculation on 7050X series and unlocking more recirculation bandwidth
For switches utilizing all 128 SERDES lanes of the Trident-II the situation is clear: If you enable recirculation, you have to sacrifice ports. This is the case for 7050QX-32 / 7050QX-32S (32x QSFP+ = 32x 4 lanes = 128 lanes -> all lanes used) and 7050TX-128 / 7050SX-128 (96x 10G SFP+ or RJ45 + 8x 40G QSFP+ = 96 lanes + 8x4 lanes = 96 lanes + 32 lanes = 128 lanes -> all lanes used).
Other models like 7050TX-48, 7050TX-64, 7050TX-72, 7050TX-72Q, 7050TX-96, 7050SX-64, 7050SX-72, 7050SX-72Q and the 7050SX-96 do not utilize all 128 lanes. As you may have noticed (except for the 7050QX-32 / 7050QX-32S), the number after the dash (like 48, 64, 72, 96) indicates the number of used lanes / number of lanes available to the network admin as data ports.
Those models have unused lanes, so we can use those first for recirculation, right? Well: Yes and No. Yes we can use those lanes for recirculation, but Arista’s implementation makes it a bit cumbersome to configure.
We will investigate this using 7050TX-72Q (uses BCM56850A0 Trident-II) and 7050SX-64 (also uses BCM56850A0 Trident-II) switches, both running Software image version: 4.28.12M. You can verify what ASIC is being used with the show hardware system forwarding-chips command.
The default approach to recirculation on Arista 7050X series
We will start with a complete vanilla configuration on both switches.
A show inventory reveals that the switch knows about unconnected lanes / interfaces
7050SX-64(config)#show inventory
[...]
System has 81 ports
Type Count
------------------ ----
Management 1
Switched 62
Unconnected 16
SwitchedBootstrap 2 We see Unconnected 16 - the math to confirm is simple: 128 lanes - 48x1 lane for SFP+ - 4x4 lanes for QSFP+ = 64 lanes = 16 Unconnected ports with 4 lanes each
7050TX-72Q(config)#show inventory
[...]
System has 87 ports
Type Count
------------------ ----
Management 1
Switched 70
Unconnected 14
SwitchedBootstrap 2 We see Unconnected 14 - the math to confirm is again simple: 128 lanes - 48x1 lane for SFP+ - 6x4 lanes for QSFP+ = 56 lanes = 14 Unconnected ports with 4 lanes each
Great. But the interfaces do not show up when you do a show int all status:
7050SX-64(config)#show int all status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et1 notconnect 1 full 10G Not Present
Et2 notconnect 1 full 10G Not Present
[...]
Et52/1 notconnect 1 full 10G Not Present
Et52/2 notconnect 1 full 10G Not Present
Et52/3 notconnect 1 full 10G Not Present
Et52/4 notconnect 1 full 10G Not Present
Ma1 notconnect routed auto auto 10/100/1000
7050SX-64(config)#the output will look slightly different on the 7050TX-72Q, but the result is the same.
In order to get going with the Unconnected Interfaces and utilize them for Recirculation, they must be exposed first via
service interface unconnected exposeOnce you do that, the interfaces will show up in the show int all status:
7050SX-64(config)#show int all status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et1 notconnect 1 full 10G Not Present
Et2 notconnect 1 full 10G Not Present
[...]
Ma1 notconnect routed auto auto 10/100/1000
Ue1 notconnect 1 full 40G Unknown
Ue2 notconnect 1 full 40G Unknown
Ue3 notconnect 1 full 40G Unknown
Ue4 notconnect 1 full 40G Unknown
Ue5 notconnect 1 full 40G Unknown
Ue6 notconnect 1 full 40G Unknown
Ue7 notconnect 1 full 40G Unknown
Ue8 notconnect 1 full 40G Unknown
Ue9 errdisabled 1 unconf unconf Unknown
Ue10 errdisabled 1 unconf unconf Unknown
Ue11 errdisabled 1 unconf unconf Unknown
Ue12 errdisabled 1 unconf unconf Unknown
Ue13 errdisabled 1 unconf unconf Unknown
Ue14 errdisabled 1 unconf unconf Unknown
Ue15 errdisabled 1 unconf unconf Unknown
Ue16 errdisabled 1 unconf unconf Unknown
7050SX-64(config)#the output will look slightly different on the 7050TX-72Q, but the result is the same.
Great! But why are are 8 of the Unconnected Ethernet interfaces showing as errdisabled? That’s already not a good sign.
The next step would be to create a Recirculation Channel interface, but let’s first inspect the output of show recirc-channel limits
7050SX-64(config)#show recirc-channel limits
LAG Group: strata
--------------------------------------------------------------------------
Max recirc-channels per group: 1024, Max ports per recirc-channel: 64
72 compatible ports: Ethernet1 Ethernet2
Ethernet3 Ethernet4
[...]
Ethernet51/1 Ethernet51/2
Ethernet51/3 Ethernet51/4
Ethernet52/1 Ethernet52/2
Ethernet52/3 Ethernet52/4
UnconnectedEthernet1 UnconnectedEthernet2
UnconnectedEthernet3 UnconnectedEthernet4
UnconnectedEthernet5 UnconnectedEthernet6
UnconnectedEthernet7 UnconnectedEthernet8
--------------------------------------------------------------------------
7050SX-64(config)#on the 7050SX-72Q, the output is even more drastic:
7050TX-72Q(config)#show recirc-channel limits
LAG Group: strata
--------------------------------------------------------------------------
Max recirc-channels per group: 1024, Max ports per recirc-channel: 64
30 compatible ports: Ethernet49/1 Ethernet49/2
Ethernet49/3 Ethernet49/4
[...]
Ethernet53/1 Ethernet53/2
Ethernet53/3 Ethernet53/4
Ethernet54/1 Ethernet54/2
Ethernet54/3 Ethernet54/4
UnconnectedEthernet1 UnconnectedEthernet2
UnconnectedEthernet3 UnconnectedEthernet4
UnconnectedEthernet5 UnconnectedEthernet6
--------------------------------------------------------------------------
7050TX-72Q(config)#Uh oh! Our errdisabled Unconnected Ethernet interfaces are not showing up as suitable for recirculation On the 7050TX-64 we can only use 8 of the 16 Unconnected Ethernet interfaces for recirculation without sacrificing physical ports. On the 7050TX-72Q we can only use 6 of the 14 Unconnected Ethernet interfaces for recirculation without sacrificing physical ports.
And if you are like me and only have the slightest touch of OCD, this will annoy you a lot. But salvation is near!
The approach to recirculation using undocumented commands - Unlocking more recirculation bandwidth!
The key is an entirely undocumented command: switch scheduler oversubscribed - I couldn’t find any public reference to it. Maybe Arista TAC would give you this command if you raise a support ticket, but it’s definitely not documented.
Warning: This command will immediately restart the Strata agent leading to potential traffic loss
Once we apply this command, the previously errdisabled Unconnected Ethernet interfaces will be notconnect just like the other Unconnected Ethernet interfaces. Additionally, the preivously errdisabled Unconnected Ethernet interfaces will show up under show recirc-channel limits
7050SX-64(config)#show int all status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et1 notconnect 1 full 10G Not Present
Et2 notconnect 1 full 10G Not Present
[...]
Et51/4 notconnect 1 full 10G Not Present
Et52/1 notconnect 1 full 10G Not Present
Et52/2 notconnect 1 full 10G Not Present
Et52/3 notconnect 1 full 10G Not Present
Et52/4 notconnect 1 full 10G Not Present
Ma1 notconnect routed auto auto 10/100/1000
Ue1 notconnect 1 full 40G Unknown
Ue2 notconnect 1 full 40G Unknown
Ue3 notconnect 1 full 40G Unknown
Ue4 notconnect 1 full 40G Unknown
Ue5 notconnect 1 full 40G Unknown
Ue6 notconnect 1 full 40G Unknown
Ue7 notconnect 1 full 40G Unknown
Ue8 notconnect 1 full 40G Unknown
Ue9 notconnect 1 full 40G Unknown
Ue10 notconnect 1 full 40G Unknown
Ue11 notconnect 1 full 40G Unknown
Ue12 notconnect 1 full 40G Unknown
Ue13 notconnect 1 full 40G Unknown
Ue14 notconnect 1 full 40G Unknown
Ue15 notconnect 1 full 40G Unknown
Ue16 notconnect 1 full 40G Unknown
7050SX-64(config)#show recirc-channel limits
LAG Group: strata
--------------------------------------------------------------------------
Max recirc-channels per group: 1024, Max ports per recirc-channel: 64
80 compatible ports: Ethernet1 Ethernet2
[...]
Ethernet51/3 Ethernet51/4
Ethernet52/1 Ethernet52/2
Ethernet52/3 Ethernet52/4
UnconnectedEthernet1 UnconnectedEthernet2
UnconnectedEthernet3 UnconnectedEthernet4
UnconnectedEthernet5 UnconnectedEthernet6
UnconnectedEthernet7 UnconnectedEthernet8
UnconnectedEthernet9 UnconnectedEthernet10
UnconnectedEthernet11 UnconnectedEthernet12
UnconnectedEthernet13 UnconnectedEthernet14
UnconnectedEthernet15 UnconnectedEthernet16
--------------------------------------------------------------------------
7050SX-64(config)#7050TX-72Q(config)#show int all status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et1 notconnect 1 auto auto 10GBASE-T
Et2 notconnect 1 auto auto 10GBASE-T
[...]
Et54/1 notconnect 1 full 10G Not Present
Et54/2 notconnect 1 full 10G Not Present
Et54/3 notconnect 1 full 10G Not Present
Et54/4 notconnect 1 full 10G Not Present
Ma1 notconnect routed auto auto 10/100/1000
Ue1 notconnect 1 full 40G Unknown
Ue2 notconnect 1 full 40G Unknown
Ue3 notconnect 1 full 40G Unknown
Ue4 notconnect 1 full 40G Unknown
Ue5 notconnect 1 full 40G Unknown
Ue6 notconnect 1 full 40G Unknown
Ue7 notconnect 1 full 40G Unknown
Ue8 notconnect 1 full 40G Unknown
Ue9 notconnect 1 full 40G Unknown
Ue10 notconnect 1 full 40G Unknown
Ue11 notconnect 1 full 40G Unknown
Ue12 notconnect 1 full 40G Unknown
Ue13 notconnect 1 full 40G Unknown
Ue14 notconnect 1 full 40G Unknown
7050TX-72Q(config)#show recirc-channel limits
LAG Group: strata
--------------------------------------------------------------------------
Max recirc-channels per group: 1024, Max ports per recirc-channel: 64
38 compatible ports: Ethernet49/1 Ethernet49/2
Ethernet49/3 Ethernet49/4
[...]
Ethernet53/1 Ethernet53/2
Ethernet53/3 Ethernet53/4
Ethernet54/1 Ethernet54/2
Ethernet54/3 Ethernet54/4
UnconnectedEthernet1 UnconnectedEthernet2
UnconnectedEthernet3 UnconnectedEthernet4
UnconnectedEthernet5 UnconnectedEthernet6
UnconnectedEthernet7 UnconnectedEthernet8
UnconnectedEthernet9 UnconnectedEthernet10
UnconnectedEthernet11 UnconnectedEthernet12
UnconnectedEthernet13 UnconnectedEthernet14
--------------------------------------------------------------------------
7050TX-72Q(config)#From there we can happily go ahead and create our recirculation channel
7050SX-64(config)#interface Recirc-Channel 99
7050SX-64(config-if-Re99)#switchport recirculation features vxlan
7050SX-64(config-if-Re99)#interface UnconnectedEthernet 1-16
7050SX-64(config-if-Ue1-16)#traffic-loopback source system device mac
7050SX-64(config-if-Ue1-16)#channel-group recirculation 99
7050SX-64(config-if-Ue1-16)#show recirc-channel all-ports
Port Channel Recirc-Channel99:
Features: vxlan
Active Ports: UnconnectedEthernet5 UnconnectedEthernet4
UnconnectedEthernet7 UnconnectedEthernet6
UnconnectedEthernet1 UnconnectedEthernet3
UnconnectedEthernet2 UnconnectedEthernet9
UnconnectedEthernet8 UnconnectedEthernet15
UnconnectedEthernet14 UnconnectedEthernet16
UnconnectedEthernet11 UnconnectedEthernet10
UnconnectedEthernet13 UnconnectedEthernet12
7050SX-64(config-if-Ue1-16)#7050TX-72Q(config)#interface Recirc-Channel 99
7050TX-72Q(config-if-Re99)#switchport recirculation features vxlan
7050TX-72Q(config-if-Re99)#interface UnconnectedEthernet 1-14
7050TX-72Q(config-if-Ue1-14)#traffic-loopback source system device mac
7050TX-72Q(config-if-Ue1-14)#channel-group recirculation 99
7050TX-72Q(config-if-Ue1-14)#show recirc-channel all-ports
Port Channel Recirc-Channel99:
Features: vxlan
Active Ports: UnconnectedEthernet5 UnconnectedEthernet4
UnconnectedEthernet7 UnconnectedEthernet6
UnconnectedEthernet1 UnconnectedEthernet3
UnconnectedEthernet2 UnconnectedEthernet9
UnconnectedEthernet8 UnconnectedEthernet14
UnconnectedEthernet11 UnconnectedEthernet10
UnconnectedEthernet13 UnconnectedEthernet12
7050TX-72Q(config-if-Ue1-14)#And we are done! More recirculation bandwidth without disabling any physical interface.
You could now setup up your EVPN-VXLAN routing and monitor those recirculation channels with show recirc-channel traffic
For good measure, I’m repeating the disclaimer from above:
I have not yet verified that unlocking more recirculation bandwidth actually provides a benefit
I don’t see why it wouldn’t, but I simply haven’t had the time to set up a VXLAN routing setup and monitor proper traffic distribution via
show recirc-channel traffic yet.Side note:
7050TX-72Q(config-if-Ue1-14)#show switch forwarding-mode
Current switching mode: cut through
Available switching modes: cut through, store and forward
7050TX-72Q(config-if-Ue1-14)#show switch scheduler
Current scheduler mode: oversubscribed
Available scheduler modes: oversubscribed, linerate
7050TX-72Q(config-if-Ue1-14)#Apparently, we are oversubscribing the switch somehow. I can’t go into detail here, simply because no public information is available on this (as always for Broadcom). What are the consequences of oversubscribing? I don’t know.
Oh and when they say Current switching mode: cut through (and not store and forward), this only applies to 40GbE to 40GbE forwarding. According to the table in the Arista 7050X & 7050X2 Switch Architecture Whitepaper, using Recirculation always drops the switch into store and forward mode for 10GbE to 10GbE forwarding. It might be possible to force the switch to also do store and forward for 40GbE to 40GbE forwarding by using switch forwarding-mode store-and-forward if you want the same latency for everything for some reason.
TL;DR
Entering the undocumented command switch scheduler oversubscribed will enable you to use more recirculation bandwidth without sacrificing physical ports on Arista 7050X series switches that have Unconnected Interfaces (7050TX-48, 7050TX-64, 7050TX-72, 7050TX-72Q, 7050TX-96, 7050SX-64, 7050SX-72, 7050SX-72Q or 7050SX-96) .
Full commands for 7050SX-64 | |