以太网交换机EtherSwitch

Why call Ethernet 802 standard?
Because IEEE began start Ethernet standard on 1980-2.
EtherSwitch: Bhardwaj's wife, Kalpana
UGREEN CM633: MT7530BU, it will buffer AVB packets to internal SRAM to cause large latency.

1 Switch基本知识
1.1 售价
PCIe交换机售价大约是100USD。
Ethernet交换机售价大约是20到25USD。

1.2 switch和SoC EMAC的接口
switch和SoC EMAC的接口一般采用MAC-to-MAC。譬如I210网卡与bcm89230连接使用P8 SerDes接口。
[27th-Apr-2022]
switch和SoC EMAC连接后，如果SoC EMAC的静态路由表中有linkdown关键字，那么说明switch初始化失败，需要对switch进行复位以重新初始化。

1.3 无线路由器当无线交换机使用方法
- 外部网线连到4个LAN口中的一个，IE登陆路由器后，禁止DHCP服务
- 其它设备搜索WLAN信号，输入用户名密码

1.4 TP-Link TL-2420PB
TP-Link: Twisted Pair Link
RTL8224: 4port 2.5G PHY

1.5 Port Mirror
配置switch的一个port作为mirror口，监控另外一个port，将mirror port口用一根网线连接到PC以太网口，在电脑上打开Wireshark。

port mirror某个口是否成功，可以查看gPTP SYNC报文中的sourcePortIdentity字段，这个字段表示的是switch的哪个port口发出的SYNC报文。

2 Switch VLAN原理
2.1 VLAN Mode
Tag-based VLANs are the industry standard 802.1Q VLANs (dot1q), while the port-based VLANs are more akin to private VLANs.

trunk：VLAN ID多于一个的port口，基于Tag-based，使用IEEE 802.1Q VID
access：untagged port，基于Tag-based，使用IEEE 802.1Q PVID，一个port最多有一个untagged的VLAN

2.2 VLAN entry CAM
VLAN entry有4096个，每个entry占用4个字节，开机后需要将使能的VLAN信息配置在交换机的CAM中，有效的VLAN entry就以12bit的VLAN ID为索引，定位到CAM中对应的位置。

2.3 PVID默认值
1）PVID是Port VLAN ID的缩写，特别注意的是PVID属于IEEE 802.1Q，不属于Port-based VLAN。假如一个port属于多个VLAN，那么PVID就是这个port的默认VLAN ID，也就是说，如果一个没有4字节VLAN TAG的报文进入了交换机，可以配置交换机给这个报文添加上默认的VLAN ID，这个VLAN ID就是PVID
2）bcm89230 PVID寄存器IEEE 802.1Q Default Tag Register Block 0x0B00, Page 34h: Address 10h-21h；默认值是1
3）当端口被分配到多个VLAN中时，PVID通过寄存器设置为最小的VLAN ID
4）AN-135
802.1Q Tag Based and Port Based VLAN Function and Setting in KSZ8995M/MA
5）例子
VLAN ID   CPU (eth0)  LAN1 LAN2 LAN3 WAN
1                    T               U        U        U      X
2                    T               X        X        X      U
51                  X               T        T        T       T
85                  X               T        T        T       T

2.4 IEEE 802.1p
在该规范中定义了PCP的值。
PCP     Priority         Acronym     Traffic types
1          0 (lowest)        BK           Background
0          1 (default)       BE            Best effort
2          2                     EE            Excellent effort
3          3                     CA            Critical applications
4          4                     VI             Video, < 100 ms latency and jitter
5          5                    VO            Voice, < 10 ms latency and jitter
6          6                    IC              Internetwork control
7          7 (highest)      NC           Network control

2.5 QinQ
1）802.1Q in 802.1Q：VLAN双Tag，是基于IEEE 802.1Q技术的一种比较简单的二层虚拟专用网络协议
2）3个重要寄存器
PVID寄存器：bcm89230 IEEE 802.1Q Default Tag Register，16位寄存器，默认值是1，这个是多用途寄存器，untagged Port口的内部和外部的VID都用PVID
ISP TPID寄存器：16位TPID值，常规的customer TPID的值为0x8100
ISP Tag Port映射寄存器：指明哪些Port口支持双Tag
3) AVB CRF may use QinQ by mistake, 88Q5050 could use TCAM rule to filter QinQ packets

3 28nm bcm89230
3.1 bcm switch codename
bcm89501: Polar
bcm89230/bcm89531: Leo
bcm89541: Simba, BCP21
bcm89572: Scorpio

3.2 基本知识
BMW MGU21 uses APL and bcm89551, bcm89551 firmware uses Erika OS v2.1.
IMP：In-band Management Port，寄存器在I/O Registers
bcm89230 CBP：Cell Buffer Pool 192 KB，每个port有8个queue，从queue 0到queue 7
MAC流控帧：PAUSE
SPI slave：最高速度62.5MHz，Data Format：1-byte command, 4-byte address, 1 ~ 8 bytes data；地址和数据都是大端格式

3.3 ARL TCAM
通过48bit的MAC地址和12bit的VLAN ID（如果使能）产生hash索引去定位ARL TCAM中对应的entry。
交换机对多播MAC不会自动学习，所以需要用户手工配置到交换芯片的寄存器中。
MAC地址byte5.bit0：1表示多播，0表示单播。

举例如下：
SOMEIP多播MAC：01-00-5E-40-FF-FB
需要配置到交换机IC中的ARL Entry 0

3.4 CBP
如果禁止802.1Q TC（Traffic Control）功能，那么所有port共享192KB CBP（Cell Buffer Pool），并且每个port只使用queue 0（因为没有分类，所有包都映射到DSCP为0，对应到queue 0），queue 1到queue 7不使用。

每个cell大小是256 Bytes，假如一个packet的大小是514个字节，那么会占用3个cell，最后一个cell剩余的254个字节不能被其它进来的packet利用，浪费了这个cell的剩余空间。只有等这个packet被交换机转发出去，才会释放3个cell给后续进来的packet保存数据。

CBP design comes from NE2000 (Novell Ethernet). NE2000 16KB SRAM is divided into 64 pages, page size is 256 Bytes, page address range [0x40, 0x7F], 12 pages [0x40, 0x4B] for TX, 52 pages [0x4C, 0x7F] for RX.

3.5 流控原理
SerDes流控配置：
- PCS_ANADV：Auto-Negotiation Advertisement
- PCS_LPAB：Link Partner Ability
- SerDes模式时需要设置对应寄存器的AN enable位，否则双方无法进行流控协商

Congestion Management：拥塞管理
MAC流控帧目的地址：特定组播MAC地址（01-80-C2-00-00-01）或者需要暂停发送的设备的MAC地址
MAC流控帧源地址：switch提供寄存器配置一个MAC地址

Note：I210网卡在使能Qav（AVB）模式后，会禁止flow control功能

3.6 QoS
RFC791: IP Type of Service (ToS.b2...b0)
RFC2474: IP Differentiated Services CodePoint (DSCP.b5...b0), DSCP.b2...b0 is called Class Selector (CS), and compatible with ToS.b2...b0, released in December 1998
https://datatracker.ietf.org/doc/html/rfc2474

MGU18 bcm8953X firmware uses Erika OS v2.1, and uses PCP2TC mapping for traffic prioritizing by default.

3.7 ACL
There are 256-entry TCAM in bcm89230. Every CFP rule searching key is 232 bits wide, CFP means Compact Field Processor. There are 8 CFP TCAM Data Registers and 8 CFP TCAM Mask Registers. When the packet hits a particular entry due to a match, a classification ID from 0 to 255 will be generated, the classification ID will reflect which TCAM entry index was hit.

Each UDF is specified with 3-bit Offset Base value and 5-bit Offset value relative to the specified Offset Base to indicate the start of the UDF fields. 5-bit Offset value indicates the UDF starts from the location 2N bytes after the location implied by the Offset Base.
Offset Base:
000: Start of Packet
010: End of L2 Header
011: End of L3 Header
For example:
MAC DA: It takes three UDFs, defined as {000, 00000}, {000, 00001} and {000, 00010}
MAC SA: It takes three UDFs, defined as {000, 00011}, {000, 00100} and {000, 00101}
IPv4 SIP: It takes two UDFs, defined as {010, 00110} and {010, 00111}

4 switch IC
4.1 TCAM IC
TCAM (Ternary Content Addressable Memory) is a memory device with fixed width array, external DBUS interface is 72/80 bit or Quad Data Rate SRAM, it uses search broadcast to search entry in parallel, actually TCAM is a hardware accelerated database search engine.

Cypress, IDT, Netlogic provide TCAM IC, for example Cypress NSE10K (Network Search Engine) and Netlogic NL101024.  In 2006 Netlogic acquired Cypress TCAM IC, in 2009 Netlogic continued to acquire IDT TCAM IC, in 2011 Netlogic is acquired by Broadcom. Broadcom names TCAM as KBP (Knowledge-based Processor). Corigine acquired Marvell TCAM IC in 2016.

4.2 2018 55nm 88Q5050
4.2.1 AVB
SPI data format: 1-byte CMD + 2-byte addr + 2-byte data
2-byte addr: bit[9:5] for 32 addressable devices (GLB1 and GLB2), bit[4:0] for 32 register addressable per device. GLB2 offset 0x16 and 0x17 for AVB Command and AVB Data, GLB2 offset 0x18 and 0x19 for SMI PHY Command and SMI PHY Data.

Generate the firmware and configure port mirroing via GUI.
交换机的每个port都有PHY地址，并且port号等于PHY地址。88Q5050支持透明时钟（P2P TC，relay），并且集成cortex M7和一个MAC，交换机内部P0连接到该MAC上，CM7运行gPTP协议栈，这也是交换机有MAC地址的由来。AVB交换机配置静态AVB组播MAC地址到ARL CAM（Address Resolution Logic）时，只需要配置接收该组播流的port mask（egress），而发送组播流的port mask（ingress）不需要配置。

88Q5050 firmware supports AVnu AutoCDS v1.4 profile (Automotive Certification Development Subgroup).
Need configure gPTP mc MAC to ARL for switch P0, P0 is connected to Cortex M7, otherwise switch gPTP stack does not handle Pdelay_Req.

4.2.2 TCAM
[10-Jan-2022]
88Q5050 has 256 TCAM entry for ACL, it can match 48-byte (MAC header, IPv4 header) or 96-byte (MAC header, IPv6 header) from ethernet frame DA start. Each entry includes 48-byte data and 48-byte mask, 48-byte data or mask is DA, SA, Tag, EtherType, IP header. Each entry is made up of page 2, page 1, page 0, page 0 contains frame matching data from byte 1 - byte 22, page 1 contains frame matching data from byte 23 to byte 48, page 2 contains TCAM action, and memory type of page 2 is SRAM. Each TCAM register is 16 bits wide, refer to Register Specification Page 290. TCAM rule can be used to prioritize SOME/IP UDP packets and filter DLT log. Use show tcam from shell to check TCAM configurations.
mask寄存器某位为1表示接收到的以太网报文对应的位必须与数据寄存器对应的位相同，mask寄存器某位为0表示不关心接收到的以太网报文对应的位。
Mask Data Meaning
0        0        Don't care. The data bit can be a one or a zero.
1        0        Hit on 0.
1        1        Hit on 1.
0        1        Never Hit.

4.2.3 2021 28nm 88Q5151
Port7: RGMII
Port8: SerDes

4.2.4 port-mirror
set port-mirror enable 5 0x80 0
mirror P7 to P5, 0 means for both ingress and egress.

4.3 28nm RTL9068 and RTL907x
4.3.1 TCAM
RTL9068 integrated Real-M500 CPU, this CPU runs firmware for IGMP snooping, IEEE 802.1X authentication, Grandmaster, E2E TC, P2P TC.

RTL9068 ACL includes 512 TCAM entry, each TCAM entry is composed of 24 bytes data, 24 bytes mask and 1 bit valid. Each 24 bytes data is divided into 12 fields, the size of each field is 2 bytes. Each 2-byte field may be DMAC[15:0], DMAC[31:16], DMAC[47:32], SMAC[15:0], SMAC[31:16], SMAC[47:32], DIP[15:0], DIP[31:16], SIP[15:0], SIP[31:0], L4 DPORT, L4 SPORT, EtherType, UDF. Each 2-byte UDF is specified with 3-bit Offset Type value and 10-bit Offset value.

For example, TCAM Entry 0:
DA = 01:02:03:04:05:06
SA = 11:12:13:14:15:16
Field 0: DMAC[0:15] = 0x0506
Field 1: DMAC[16:31] = 0x0304
Field 2: DMAC[32:47] = 0x0102
Field 3: SMAC[0:15] = 0x1516
Field 4: SMAC[16:31] = 0x1314
Field 5: SMAC[32:47] = 0x1112

4.3.2 OPEN TC10
Sleep request
ACK
Send_Z
Send_Z

4.4 SJA1105Q
4.4.1 SPI
SPI mode: transfer mode 1, CPOL = 0 and CPHA = 1
SJA1105Q does not have an internal CPU core, hence, gPTP has to be run from an external host for the switch.

4.4.2 ARL
1-byte block ID means which table is this data block belonged to, for example, block ID of ARL table is 0x05, block ID of VLAN table is 0x07, block ID of AVB Parameters is 0x10.
3-byte block length of every data block means number of ARL or VLAN entries. A block length of 0 signals the end of the configuration file and a global 4-byte CRC is expected to follow. 

eTSEC: Enhanced Three-Speed Ethernet Controller
FCI: PFE Fast Control Interface
LS1028A: QorIQ LayerScape
PFE: Packet Forwarding Engine

5 Abbreviations
altcam: ALTERA APEX20KE
ARL：Address Resolution Logic，交换机对单播MAC会自动学习，保存到TCAM中；多播MAC不会自动学习，所以需要用户手工写进TCAM中
BMC：Baseboard Management Controller
CPSW: TI Common Platform Ethernet Switch
ESM: Ethernet Switch Manager
FIDO: ADI ethernet switch Flexible Input, Deterministic Output, Real-Time Ethernet Multiprotocol
KSZ8565R：Kendin（肯定科技）Switch，Z表示KS的无铅版本，最后的数字5表示有5个端口；被Micrel收购
LDS：Link Discovery Signal
LRE：Long Range Ethernet，长距离以太网
MIB：Management Information Base
pcap：packet capture
PVID：Port default VID，PVID属于IEEE 802.1Q，不属于Port-based VLAN
RJ45：Registered Jack
SFP: MAX3735, MAX3748
SOAD：AUTOSAR Socket Adaptor
10Base-T1S：S表示Short Reach