Theoretical foundations and architecture
Lab: locating and fixing failures (troubleshooting and fault isolation) in an MPLS L3VPN core. The method is layer by layer from the bottom: physical link and interface errors, IGP and loopbacks, LDP and bindings, LFIB/CEF and MTU, MP-BGP and vpnv4, Route Target and VRF import, the VRF table and the PE-CE edge; until a lower layer is confirmed with evidence (command output, a reply code), the upper one is not diagnosed. At each layer two questions differ: control plane ("does the node know the route and the label?") and data plane ("does it forward by them?": LFIB, ping mpls, traceroute mpls). The lab walks through three typical failures: 1. LDP disabled on one interface: IGP and BGP stay up but the neighbour loopback is marked Untagged/No Label in the label table and the VPN packet is dropped at the P router. 2. Import Route Target changed: the LSP and sessions are fine but the VRF does not import routes; comparing the RT in show vrf detail with the vpnv4 table settles it. 3. MTU lowered: every label adds 4 bytes, an L3VPN packet carries two, so a 1500-byte IP packet needs 1508 bytes in the core; small pings pass while ping vrf ... size 1500 df-bit fails.
The method: layer by layer, bottom to top
A failure in an MPLS VPN looks the same ("the customer ping does not work"), while its causes can be a dozen at different levels. The isolation method divides the path into layers and checks each, starting from the bottom, until the first faulty one is found: the physical link and interface errors, the IGP adjacency and loopback reachability, the LDP session and bindings (LIB), the programmed forwarding table (LFIB/CEF) and MTU, the MP-BGP session and VPNv4 routes, the Route Target and import into the VRF, the VRF table and the PE-CE link, and finally the customer network. The rule: until a lower layer is confirmed, do not troubleshoot the upper one, because its symptoms may be a consequence. Each layer needs evidence, not an assumption: a command output, a number, a reply code.
Control plane and data plane: two kinds of question
Two different questions are asked at every layer. Control plane: "does the node know the route and the binding?" (show ip route, show mpls ldp bindings, show bgp vpnv4 unicast all). Data plane: "does the node forward the packet based on that knowledge?" (show mpls forwarding-table detail, show ip cef detail, ping mpls). A typical trap is a binding present in the LIB while the LFIB entry is marked No Label or Untagged, or an LFIB that is correct while the hardware table is programmed wrongly. So diagnostics combine both kinds of checks: LSP ping/traceroute and BFD from the previous lessons show the state of the data plane, while the tables show the control plane.
What the tools show (the layer map)
IGP layer: ping loopback to loopback with source, show ip ospf neighbor, the BFD state. LDP layer: show mpls ldp neighbor (session Operational, transport address is the loopback), show mpls ldp discovery (hellos arrive on the right interfaces), show mpls ldp bindings 10.255.0.2 32 (the binding was received from the next hop). LFIB layer: show mpls forwarding-table 10.255.0.2 32 detail (outgoing label, interface, next hop), traceroute mpls (MRU and labels per hop). MP-BGP layer: show bgp vpnv4 unicast all summary (session Established, number of prefixes). VRF layer: show vrf detail A (RD, import/export RT), show ip route vrf A, show ip cef vrf A <prefix> detail (a stack of two labels: transport and VPN).
Three typical failures and their signs
LDP was disabled on one interface. The IGP adjacency and the BGP session to the RR stay Up because they run as IP; in the label table the neighbor's loopback prefix is shown as Untagged/No Label. A VPN packet gets only the VPN label and goes to P, which has no binding for the transport label, so the packet is dropped. Expected signs: show mpls ldp neighbor without the neighbor, ping mpls with code Q or N, ping vrf A is lost while the IGP ping works. The decisive evidence is show mpls forwarding-table <PE2 loopback> detail.
The import RT was changed. The transport works: LSP ping succeeds, LDP and BGP are fine. But the VRF does not import routes: a PE does not keep VPNv4 routes whose RT matches none of its imports, so the sought prefix is absent from show ip route vrf A and show bgp vpnv4 unicast all does not show it. Diagnosis: compare the import RT in show vrf detail A on PE1 with the RT exported by PE2 (visible on the RR with show bgp vpnv4 unicast rd <rd> <prefix> in the Extended Community field). The symptom "the LSP is fine but the route is missing" unambiguously points to the VPN policy layer.
The MTU was reduced. Small packets pass and large ones are lost: each label adds 4 bytes, and a customer packet in an L3VPN carries two (transport and VPN), so a 1500-byte IP packet needs 1508 bytes in the core. If the link MTU is 1500, the packet is fragmented (with DF=0) or dropped (with DF=1). Signs: ping vrf A <dst> size 1500 df-bit fails while size 1400 passes; ping mpls with the default size succeeds because the echo is small; traceroute mpls shows an understated MRU at a specific hop. If the ICMP "fragmentation needed" message is filtered, we get an MTU black hole: TCP sessions "hang" after the handshake. Fix: increase the MTU (or mpls mtu) on all core links to at least 1508-1520, or configure tcp adjust-mss on the PE as a temporary measure.
Timeline and events
The log and notifications give the order of events, and the order points to the cause: if BFD went down first, then OSPF, then LDP and only then BGP, the cause lies at a lower level; if only the LDP session dropped without the IGP, the culprit is configuration or MTU at the LDP level (hellos work, but the TCP session on port 646 with large messages cannot be established). A typical sign of an MTU problem for LDP/BGP: the adjacency comes up but the session drops when large updates are exchanged (Keepalive and Update). Always ask "what changed": compare the current configuration with the baseline saved before the incident.
Verification discipline
Before testing, record the baseline of the healthy network: the output of show mpls forwarding-table, the MRU in LSP traceroute, the RTT of LSP ping, the BFD counters. Then make one change, collect a minimal set of commands, form a hypothesis of the layer, confirm it and restore the state. Changing several parameters at once makes the conclusions unreliable. A missing OAM reply does not prove a break: the reply may be lost on the return path or filtered, so independent confirmation by another method is required (the LFIB table, interface counters, traceroute mpls). Conversely, a successful ping does not prove the service is healthy: it tests the path, but not the RT or the MTU of a customer packet.
Architecture Diagram & Topology
+-----------+ Broken Core Link / Label Mismatch +-----------+
| PE-1 | =====================X========================> | PE-2 |
| Diagnostics: LSP Ping, Traceroute, LFIB Inspection | Egress |
| Isolates: MTU Black Hole, LDP Sync Mismatch, Broken Binding | Target |
+-----------+ +-----------+Traffic Flow & Label Stack Scheme
Systematic MPLS Troubleshooting Pipeline:
1. Verify Control Plane: Are IGP adjacencies UP? Is LDP session Operational? Is BGP VPNv4 established?
2. Verify Label Bindings: Does local LIB have a binding for the destination FEC?
3. Verify Data Plane (LFIB): Does incoming label map to correct outgoing label and interface?
4. Run LSP Ping / Traceroute: Is packet dropped due to MTU mismatch or unprogrammed penultimate hop?
5. Remediate: Fix LDP-IGP sync, MTU mismatch, or VRF Route Target policy.