SONiC Architecture: Trace a Route on a Running Switch

SONiC NOS is less a single program than a set of services that talk through a Redis database. That one idea explains most of the architecture, and it means you can read the state of the switch directly.

SONiC, Software for Open Networking in the Cloud, is an open source network operating system based on Linux that runs on switches from multiple vendors and ASICs. Microsoft open sourced it in 2016, and in 2022 it ceded oversight to the Linux Foundation.

Most pages on SONiC architecture stop at a diagram. Here we follow one BGP route on a running switch instead. All output below comes from leaf1 in a two spine, two leaf fabric on the community docker-sonic-vs image from the 202511 branch.

To follow along, ask the NetPilot agent for a two spine, two leaf SONiC fabric with eBGP, or deploy the public lab on your own host. Every command here is a read, apart from one test VLAN you can remove with config vlan del 300.

The parts

The SONiC wiki lists these containers on a switch:

The virtual switch image folds these into one container under supervisord, so supervisorctl status shows the same services side by side (filtered here, with process IDs trimmed):

bgpd                             RUNNING
fpmsyncd                         RUNNING
orchagent                        RUNNING
redis-server                     RUNNING
syncd                            RUNNING
zebra                            RUNNING

The databases

Everything those services share sits in Redis, in numbered databases. The map on this switch:

{'APPL_DB': 0, 'ASIC_DB': 1, 'COUNTERS_DB': 2, 'CONFIG_DB': 4, 'PFC_WD_DB': 5,
 'FLEX_COUNTER_DB': 5, 'STATE_DB': 6, 'SNMP_OVERLAY_DB': 7, ...}

Four of them carry the main story:

sonic-db-cli reads any of them by name, so you never need the numbers.

Follow a config change

A config command writes intent to CONFIG_DB, and the rest of the switch reacts. Add a VLAN and read it in each database:

config vlan add 300
sonic-db-cli CONFIG_DB hgetall 'VLAN|Vlan300'
sonic-db-cli APPL_DB hgetall 'VLAN_TABLE:Vlan300'
sonic-db-cli STATE_DB hgetall 'VLAN_TABLE|Vlan300'

{'vlanid': '300'}
{'admin_status': 'up', 'mtu': '9100', 'mac': '3a:b2:1c:6b:fc:56', 'host_ifname': ''}
{'state': 'ok'}

Then the ASIC object:

ASIC_STATE:SAI_OBJECT_TYPE_VLAN:oid:0x260000000005eb
{'SAI_VLAN_ATTR_VLAN_ID': '300'}

One command, four records. CONFIG_DB has your intent, APPL_DB has the service's fuller view, STATE_DB says it worked and ASIC_DB has the object for the SAI. The key separators change too, | in CONFIG_DB and STATE_DB, : in APPL_DB, which trips up everyone once.

Follow a BGP route

The SONiC wiki spells out this path: bgpd tells zebra, zebra installs the route in the kernel and passes it to fpmsyncd, fpmsyncd writes APPL_DB, orchagent writes ASIC_DB, and syncd calls the SAI. Here is that path for 192.168.2.0/24, the subnet behind leaf2, on the switch.

1. FRR picks it. Two equal paths through two spines.

vtysh -c 'show ip route 192.168.2.0/24'

Routing entry for 192.168.2.0/24
  Known via "bgp", distance 20, metric 0, best
  Last update 00:00:41 ago
  * 10.0.1.0, via Ethernet0, weight 1
  * 10.0.3.0, via Ethernet4, weight 1

2. The Linux kernel has it. SONiC is Linux, and the front panel ports are kernel interfaces.

ip route show 192.168.2.0/24

192.168.2.0/24 nhid 36 proto bgp metric 20
	nexthop via 10.0.3.0 dev Ethernet4 weight 1
	nexthop via 10.0.1.0 dev Ethernet0 weight 1

3. fpmsyncd writes APPL_DB. Same route, now in Redis.

sonic-db-cli APPL_DB hgetall 'ROUTE_TABLE:192.168.2.0/24'

{'protocol': 'bgp', 'nexthop': '10.0.1.0,10.0.3.0', 'ifname': 'Ethernet0,Ethernet4', 'weight': '1,1'}

4. orchagent writes ASIC_DB. The route is now a SAI object pointing at a next hop group.

sonic-db-cli ASIC_DB keys '*ROUTE_ENTRY*192.168.2*'

ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"192.168.2.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000005"}

{'SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID': 'oid:0x50000000005e8'}

Follow that ID and you find the ECMP group:

{'SAI_NEXT_HOP_GROUP_ATTR_TYPE': 'SAI_NEXT_HOP_GROUP_TYPE_DYNAMIC_UNORDERED_ECMP'}

5. syncd calls the SAI. On a hardware switch this is where the ASIC vendor's SAI programs the chip. On the virtual switch, sai.profile names a simulated one:

SAI_VS_SWITCH_TYPE=SAI_VS_SWITCH_TYPE_BCM56850

That last step is where the virtual switch differs from hardware. The steps above it run the same SONiC services, which is why a virtual switch is a good way to learn the architecture, and why its gaps sit near the ASIC.

The whole path is also a troubleshooting map. A route in FRR but not APPL_DB points at fpmsyncd. A route in APPL_DB but not ASIC_DB points at orchagent.

Let the agent read it

Reading five databases by hand is good for learning and slow for a fabric. Ask:

On every leaf, check that the route to the other leaf's host subnet is in FRR, APPL_DB and ASIC_DB with two next hops, and flag any switch where they disagree.

The agent runs the same commands on each switch and returns one table. Direct CLI is always available too. Open a terminal on any switch and run them yourself.

FAQ

What is SONiC NOS?

SONiC, Software for Open Networking in the Cloud, is an open source network operating system based on Linux that runs on switches from multiple vendors and ASICs. Microsoft developed and open sourced it in 2016, and in 2022 Microsoft ceded oversight of the project to the Linux Foundation.

What are the main parts of the SONiC architecture?

A Redis database at the center, and services that read and write it. The SONiC wiki lists the Database, Swss, Syncd, Bgp, Teamd, Lldp, Snmp, Pmon and Dhcp-relay containers. FRR computes routes, orchagent turns them into ASIC objects, and syncd programs them through SAI.

What does SAI do in SONiC?

SAI is the Switch Abstraction Interface. The SONiC wiki defines it as an API that gives a vendor independent way of controlling forwarding elements such as a switching ASIC. It is why SONiC runs on more than one ASIC vendor.

Which Redis databases does SONiC use?

On the 202511 virtual switch, sonic-db-cli maps APPL_DB to Redis database 0, ASIC_DB to 1, COUNTERS_DB to 2, CONFIG_DB to 4 and STATE_DB to 6, among others. CONFIG_DB holds intent, APPL_DB holds application state, ASIC_DB holds what is programmed and STATE_DB holds what each service reports.

Next steps

• SONiC NOS lab online: a running switch to read, from a prompt
• SONiC FRR: configure BGP: build the fabric used here
• Run SONiC in containerlab: the same image on your own host