Interface input error

Introduction

This document describes how to determine why a port or interface experiences problems.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document applies to Catalyst switches that run on Cisco IOS® System Software.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Conventions

Refer toCisco Technical Tips Conventionsfor more information on document conventions.

Note: To access tools and websites, you must be a registered Cisco client.

Troubleshoot the Physical Layer 

Use the LEDs to Troubleshoot

If you have physical access to the switch, it can save time to look at the port LEDs which give you the link status or can indicate an error condition (if red or orange). The table describes the LED status indicators for Ethernet modules or fixed-configuration switches:

Platform	URL
Catalyst 6000 Series Switches	Ethernet Module LEDs
Catalyst 4000 Series Switches	Ethernet Module LEDs
Catalyst 3750 Series Switches	Front Panel LEDs
Catalyst 3550 Series Switches	Front Panel LEDs
Catalyst 2950/2955 Series Switches	Front Panel LEDs
Catalyst 2900/3500XL Series Switches	Front Panel LEDs
Catalyst 1900 and 2820 Series Switches	Front Panel LEDs

Ensure that both sides have a link. A single broken wire or one shutdown port can cause the problem where one side has a link light, but the other side does not.

A link light does not guarantee that the cable is fully functional. The cable can have encountered physical stress that causes it to be functional at a marginal level. Normally you can identify this situation if the port has many packet errors, or the port constantly flaps (loses and regains link).

Check the Cable and Both Sides of the Connection

If the link light for the port does not come on, you can consider these possibilities:

Possible Cause	Corrective Action
No cable connected	Connect cable from switch to a known good device.
Wrong Port	Make sure that both ends of the cable are plugged into the correct ports.
Device has no power	Ensure that both devices have power.
Wrong cable type	Verify the cable selection. Refer to theCatalyst Switch Cable Guide.
Bad cable	Swap suspect cable with known good cable. Look for broken or lost pins on connectors.
Loose connections	Check for loose connections. Sometimes a cable appears to be seated in the jack but is not. Unplug the cable and reinsert it.
Patch Panels	Eliminate faulty patch panel connections. Bypass the patch panel if possible to rule it out.
Media Convertors	Eliminate faulty media convertors: fiber-to-copper, and so on. Bypass the media convertor if possible to rule it out.
Bad or wrong Gigabit Interface Convertor (GBIC)	Swap suspect GBIC with known good GBIC. Verify Hw and Sw support for this type of GBIC.
Bad Port or Module Port or Interface or Module not enabled	Move the cable to a known good port to troubleshoot a suspect port or module. Use the show interface command for Cisco IOS to look for errdisable, disable or shutdown status. The show module command can indicate faulty, which can indicate a hardware problem. See the Common Port and Interface Problems section of this document for more information.

Ethernet Copper and Fiber Cables

Ensure that you have the correct cable for the type of connection you want to make. Category 3 copper cable can be used for 10 Mbps unshielded twisted pair (UTP) connections but must never be used for 10/100 or 10/100/1000Mbps UTP connections. Always use either Category 5, Category 5e, or Category 6 UTP for 10/100 or 10/100/1000Mbps connections.

Warning: Category 5e and Category 6 cables can store high levels of static electricity because of the dielectric properties of the materials used in their construction. Always ground the cables (especially in new cable runs) to a suitable and safe earth ground before you connect them to the module.

For fiber, make sure you have the correct cable for the distances involved and the type of fiber ports that are used. The two options are single mode fiber (SMF) or multimode fiber (MMF). Make sure the ports on the devices that are connected together are both SMF, or both are MMF ports.

Note: For fiber connections, make sure the transmit lead of one port is connected to the receive lead of the other port. Connections for transmit-to-transmit and receive-to-receive do not work.

Ethernet and Fast Ethernet Maximum Transmission Distances

Transceiver Speed	Cable Type	Duplex Mode	Maximum Distance between Station
10 Mbps	Category 3 UTP	Full and half	328 ft (100 m)
10 Mbps	MMF	Full and half	1.2 mi (2 km)
100 Mbps	Category 5 UTP Category 5e UTP	Full and half	328 ft (100 m)
100 Mbps	Category 6 UTP	Full and half	328 ft (100 m)
100 Mbps	MMF	Half	1312 ft (400 m)
		Full	1.2 mi (2 km)
100 Mbps	SMF	Half	1312 ft (400 m)
		Full	6.2 mi (10 km)

For more details on the different types of cables/connectors, cable requirements, optical requirements (distance, type, patch cables, and so on.), how to connect the different cables, and which cables are used by most Cisco switches and modules, refer to Catalyst Switch Cable Guide .

Troubleshoot the Gigabit Ethernet

If you have device A connected to device B over a Gigabit link, and the link does not come up, perform this procedure.

Step-by-Step Procedure

1. Verify device A and B use the same GBIC, short wavelength (SX), long wavelength (LX), long haul (LH), extended wavelength (ZX), or copper UTP (TX). Both devices must use the same type of GBIC to establish link. An SX GBIC needs to connect with an SX GBIC. An SX GBIC does not link with an LX GBIC. Refer to Mode-Conditioning Patch Cord Installation Note for more information.

2. Verify distance and cable used per GBIC as defined in this table.

   1000BASE-T and 1000BASE-X Port Cabling Specifications

GBIC	Wavelength (nm)	Copper/Fiber Type	Core Size1(Microns)	Modal Bandwidth (MHz / km)	Cable Distance2
WS-G54831000Base — T (copper)		Category 5 UTP Category 5e UTP Category 6 UTP			328 ft (100 m)
WS-G54841000BASE-SX3	850	MMF	62.5 62.5 50.0 50.0	160 200 400 500	722 ft (220 m) 902 ft (275 m) 1640 ft (500 m) 1804 ft (550 m)
WS-G54861000BASE-LX/LH	1310	MMF4SMF	62.5 50.0 50.0 8.3/9/10	500 400 500 —	1804 ft (550 m) 1804 ft (550 m) 1804 ft (550 m) 6.2 miles (10 km)
WS-G54871000BASE-ZX5	1550	MMF SMF6	8.3/9/10 8.3/9/10		43.5 miles (70 km)762.1 miles (100 km)

1. The numbers given for multimode fiber-optic cable refer to the core diameter. For single-mode fiber-optic cable, 8.3 microns refers to the core diameter. The 9-micron and 10-micron values refer to the mode-field diameter (MFD), which is the diameter of the portion of the fiber that is light-carrying. This area consists of the fiber core plus a small portion that covers the cladding. The MFD is a function of the core diameter, the wavelength of the laser, and the refractive index difference between the core and the cladding.

2. Distances are based on fiber loss. Multiple splices and substandard fiber-optic cable reduce the cable distances.

3. Use with MMF only.

4. When you use an LX/LH GBIC with 62.5-micron diameter MMF, you must install a mode-conditioning patch cord (CAB-GELX-625 or equivalent) between the GBIC and the MMF cable on both the transmit and receive ends of the link. The mode-conditioning patch cord is required for link distances less than 328 feet (100 m) or greater than 984 feet (300 m). The mode-conditioning patch cord prevents the over use of the receiver for short lengths of MMF and reduces differential mode delay for long lengths of MMF. Refer to Mode-Conditioning Patch Cord Installation Note for more information.

5. Use with SMF only.

6. Dispersion-shifted single-mode fiber-optic cable.

7. The minimum link distance for ZX GBICs is 6.2 miles (10 km) with an 8-dB attenuator installed at each end of the link. Without attenuators, the minimum link distance is 24.9 miles (40 km).

3. If either device has multiple Gigabit ports, connect the ports to each other. This tests each device and verifies that the Gigabit interface functions correctly. For example, you have a switch that has two Gigabit ports. Wire Gigabit port one to Gigabit port two. Does the link come up? If so, the port is good. STP blocks on the port and prevents any loops (port one receive (RX) goes to port two transmit (TX), and port one TX goes to port two RX).

4. If single connection or Step 3 fails with SC connectors, loop the port back to itself (port one RX goes to port one TX). Does the port come up? If not, contact the TAC, as this can be a faulty port.

5. If steps 3 and 4 are successful, but a connection between device A and B cannot be established, loop ports with the cable that adjoins the two devices. Verify that there is not a faulty cable.

6. Verify that each device supports 802.3z specification for Gigabit auto-negotiation. Gigabit Ethernet has an auto-negotiation procedure that is more extensive than the one used for 10/100 Ethernet (Gigabit auto-negotiation spec: IEEE Std 802.3z-1998). When you enable link negotiation, the system auto-negotiates flow control, duplex mode, and remote fault information. You must either enable or disable link negotiation on both ends of the link. Both ends of the link must be set to the same value or the link cannot connect. Problems have been seen when you connect to devices manufactured before the IEEE 802.3z standard was ratified. If either device does not support Gigabit auto-negotiation, disable the Gigabit auto-negotiation, and it forces the link up. It takes 300msec for the card firmware to notify the software that a 10/100/1000BASE-TX link/port is down. The 300msec default debounce timer comes from the firmware polling timer to the linecards, which occurs every 300 msec. If this link is run in 1G (1000BASE-TX) mode, Gigabit sync, which occurs every 10msec, must be able to detect the link down faster. There is a difference in the link failure detection times when you run GigabitEthenet on copper versus GigabitEthernet over fiber. This difference in detection time is based on the IEEE standards.

Warning: Disable auto-negotiation and this hides link drops or physical layer problems. This is only required if end-devices such as older Gigabit NICs are used which cannot support IEEE 802.3z. Do not disable auto-negotiation between switches unless absolutely required to do so, as physical layer problems can go undetected, which results in STP loops. The alternative is to contact the vendor for software/hardware upgrade for IEEE 802.3z Gigabit auto-negotiation support.

For GigabitEthernet system requirements as well as Gigabit Interface Converters (GBICs), Coarse Wavelength Division Multiplexing (CWDM), and Small Form-Factor Pluggable (SFP) system requirements, refer to these:

• System Requirements to Implement Gigabit Ethernet on Catalyst Switches

• Catalyst GigaStack Gigabit Interface Converter Switch Compatibility Matrix

• Cisco Gigabit Ethernet Transceiver Modules Compatibility Matrix

• Cisco 10-Gigabit Ethernet Transceiver Modules Compatibility Matrix

For general configuration information and additional information on how to troubleshoot, refer to Configuring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto-Negotiation .

Connected vs Notconnected

Most Cisco switches have a port in the notconnect state. This means it is currently not connected to anything, but it can connect if it has a good connection to another operational device. If you connect a good cable to two switch ports in the notconnect state, the link light must become green for both ports, and the port status must indicate connected. This means that the port is up as far as Layer 1 (L1) is concerned.

For Cisco IOS, you can use the show interfaces command to verify whether the interface is up, line protocol is up (connected) . The first up refers to the physical layer status of the interface. The line protocol up message shows the data link layer status of the interface and says that the interface can send and receive keepalives.

Router#show interfaces fastEthernet 6/1
FastEthernet6/1 is down, line protocol is down (notconnect)

!--- The interface is down and line protocol is down.
!--- Reasons: In this case,
!--- 1) A cable is not properly connected or not connected at all to this port.
!--- 2) The connected cable is faulty.
!--- 3) Other end of the cable is not connected to an active port or device.

!--- Note: For gigabit connections, GBICs need to be matched on each
!--- side of the connection.
!--- There are different types of GBICs, depends on the cable and
!--- distances involved: short wavelength (SX),
!--- long-wavelength/long-haul (LX/LH) and extended distance (ZX).
!--- An SX GBIC needs to connect with an SX GBIC;
!--- an SX GBIC does not link with an LX GBIC. Also, some gigabit
!--- connections require conditioning cables,
!--- that depend on the lengths involved.

Router#show interfaces fastEthernet 6/1
FastEthernet6/1 is up, line protocol is down (notconnect)

!--- The interface is up (or not in a shutdown state), but line protocol down.
!--- Reason: In this case, the device on the other side of the wire is a
!--- CatOS switch with its port disabled.

Router#show interfaces fastEthernet 6/1 status
Port   Name    Status       Vlan    Duplex   Speed  Type
Fa6/1          notconnect    1       auto     auto   10/100BaseTX

Ifshow interfacesshows up/ line protocol up (connected) but you see errors increment in the output of either command, refer to the Common Port and Interface Problems section of this document for advice.

Troubleshoot the Most Common Port and Interface Commands for Cisco IOS

This table shows the most common commands used to troubleshoot the port or interface problems on switches that run Cisco IOS System Software on the Supervisor.

Note: The right hand column on the next table gives a brief description of what the command does and lists any exceptions to the use per platform.

If you have the output of the supported commands from your Cisco device, you can use Cisco CLI Analyzer to display potential issues and fixes.

Cisco IOS Commands	Description
show version	This command displays output similar to a Cisco router, like software image name and version information and system memory sizes. Helpful with the search for software/hardware incompatibilities (with theRelease NotesorSoftware Advisor) and bugs (with theSoftware Bug Toolkit).
show module	This command displays what cards are present in the switch, the version of software they are that run, and what state the modules are in: ok, faulty, and so on. This is helpful to diagnose a hardware problem on a module or port. For more information about how to troubleshoot hardware problems with theshow module command, see the Port or Interface Status is disabled or shutdown or the Hardware Problems sections of this document.
show run-config	This command displays the current configuration file of the switch. Changes are saved to the config in Cisco IOS with the write memory command. This is helpful to use to determine whether a misconfiguration of the mod/port or interface can cause a problem.
show interfaces	The show interface command displays the administrative and operational status of a switch port, input and output packets, buffer failures, errors, and so on.
clear counters	Use theclear counters command to zero the traffic and error counters so that you can see if the problem is only temporary, or if the counters continue to increment. Note: The Catalyst 6500/6000 series switches do not clear the bit counters of an interface with theclear counterscommand. The only way to clear the bit counters in these switches is to reload.
show interfaces counters	This is the command to use on the Catalyst 6000, 4000, 3550, 2950, and 3750 series.
show counters interface  show controllers ethernet-controller	Theshow counters interface command was introduced in software version 12.1(13)E for the Catalyst 6000 series only and displays 32-bit and 64-bit error counters. For Cisco IOS on 2900/3500XL, 2950/2955, 3550, 2970 and 3750 series switches, theshow controllers Ethernet-controller command displays discarded frames, deferred frames, alignment errors, collisions, and so on.
show interfaces counters	This is the command to use on the Catalyst 6000, 4000, 3550, 2950, and and 3750 series.
show diagnostic(s) show post	The command show diagnostic was introduced in 12.1(11b)E for the Catalyst 6000 series and show diagnostics (with an s ) was introduced in for Catalyst 4000 Series. On the 2900/3500XL, 2950/2955, 3550, 2970 and 3750 series switches the equivalent command is show post which displays the results of the switch POST. For more information on troubleshoot hardware related errors on Catalyst switches, see the Hardware Problems section of this document.

Understand the Specific Port and Interface Counter Output for Cisco IOS

Most switches have some way to track the packets and errors that occur on a port or interface. The common commands used to find this type of information are described in the Most Common Port and Interface Troubleshooting Commands for Cisco IOS section of this document.

Note: There can be differences in the implementation of the counters across various platforms and releases. Although the values of the counters are largely accurate, they are not very precise by design. In order to pull the exact statistics of the traffic, it is suggested that you use a sniffer to monitor the necessary ingress and egress interfaces.

Excessive errors for certain counters usually indicate a problem. When you operate at half-duplex setup, some data link errors increment in Frame Check Sequence (FCS), alignment, runts, and collision counters are normal. Generally, a one percent ratio of errors to total traffic is acceptable for half-duplex connections. If the ratio of errors to input packets is greater than two or three percent, performance degradation can be noticed.

In half-duplex environments, it is possible for both the switch and the connected device to sense the wire and transmit at exactly the same time and result in a collision. Collisions can cause runts, FCS, and alignment errors due to the frame not completely copied to the wire, which results in fragmented frames.

When you operate at full-duplex, errors in FCS, Cyclic Redundancy Checks (CRC), alignment, and runt counters must be minimal. If the link operates at full-duplex, the collision counter is not active. If the FCS, CRC, alignment, or runt counters increment, check for a duplex mismatch. Duplex mismatch is a situation where the switch operates at full-duplex and the connected device operates at half-duplex, or vice versa. The results of a duplex mismatch are extremely slow performance, intermittent connectivity, and loss of connection. Other possible causes of data link errors at full-duplex are bad cables, faulty switch ports, or NIC software/hardware issues. See the Common Port and Interface Problems section of this document for more information.

Show Interfaces for Cisco IOS

The show interfaces card-type {slot/port}command is the used command for Cisco IOS on the Supervisor to display error counters and statistics. An alternative to this command (for Catalyst 6000, 4000, 3550, 2970 2950/2955, and 3750 series switches) is theshow interfacescard-type <slot/port>counters errors command which only displays the interface error counters. Refer to Table 1 for explanations of the error counter output.

Note: For 2900/3500XL Series switches use theshow interfacescard-type {slot/port}command with theshow controllers Ethernet-controllercommand.

Router#sh interfaces fastEthernet 6/1
FastEthernet6/1 is up, line protocol is up (connected)
   Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
   MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
      reliability 255/255, txload 1/255, rxload 1/255
   Encapsulation ARPA, loopback not set
   Full-duplex, 100Mb/s
   input flow-control is off, output flow-control is off
   ARP type: ARPA, ARP Timeout 04:00:00
   Last input 00:00:14, output 00:00:36, output hang never
   Last clearing of "show interface" counters never
   Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
   Queueing strategy: fifo
   Output queue :0/40 (size/max)
   5 minute input rate 0 bits/sec, 0 packets/sec
   5 minute output rate 0 bits/sec, 0 packets/sec

Theshow interfacescommand output up to this point is explained here (in order) :

• up, line protocol is up (connected) — The first up refers to the physical layer status of the interface. The line protocol up message shows the data link layer status of the interface and says that the interface can send and receive keepalives.

• MTU — The Maximum Transmission Unit (MTU) is 1500 bytes for Ethernet by default (for the max data portion of the frame).

• Full-duplex, 100Mb/s — Full-duplex and 100Mbps is the current speed and duplex setup of the interface. This does not tell you whether autoneg was used to achieve this. Use theshow interfaces fastEthernet 6/1 statuscommand to display this:

Router#show interfaces fastEthernet 6/1 status
Port    Name               Status       Vlan       Duplex  Speed Type
Fa6/1                      connected    1          a-full  a-100 10/100BaseTX

!--- Autonegotiation was used to achieve full-duplex and 100Mbps.

• Last input, output — The number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface. This is useful to know when a dead interface failed.

• Last clearing of «show interface» counters — The last time the clear counters command was issued since the last time the switch was rebooted. The clear counters command is used to reset interface statistics.

Note: Variables that can affect routing (for example, load and reliability) are not cleared when the counters are cleared.

• Input queue — The number of packets in the input queue.Size/max/drops= the current number of frames in the queue / the max number of frames the queue can hold before it must start to drop frames / the actual number of frames dropped because the max queue size was exceeded.Flushesis used to count Selective Packet Discard (SPD) drops on the Catalyst 6000 Series that run Cisco IOS. (The flushes counter can be used but never increments on the Catalyst 4000 Series that run Cisco IOS.) SPD is a mechanism that quickly drops low priority packets when the CPU is overloaded in order to save some process capacity for high priority packets. The flushes counter in the show interface command output increments as part of selective packet discard (SPD), which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies to only process switched traffic.

  The purpose of SPD is to ensure that important control packets, such as routing updates and keepalives, are not dropped when the IP input queue is full. When the size of the IP input queue is between the minimum and maximum thresholds, normal IP packets are dropped based on a certain drop probability. These random drops are called SPD flushes.

• Total output drops — The number of packets dropped because the output queue is full. A common cause is traffic from a high bandwidth link that is switched to a lower bandwidth link or traffic from multiple inbound links that are switched to a single outbound link. For example, if a large amount of traffic flow comes in on a gigabit interface and is switched out to a 100Mbps interface, this can cause output drops to increment on the 100Mbps interface. This is because the output queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.

• Output queue — The number of packets in the output queue. Size/max means the current number of frames in the queue/the max number of frames the queue can hold before it is full and must start to drop the frames.

• 5 minute input/output rate — The average input and output rate seen by the interface in the last five minutes. Specify a shorter period of time to get an accurate read (to better detect traffic bursts for example and issue theload-interval <seconds>interface command.

    SeeTable 1for explanations of the error counter output.

!--- ...show interfaces command output continues.
     1117058 packets input, 78283238 bytes, 0 no buffer
      Received 1117035 broadcasts, 0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 watchdog, 0 multicast, 0 pause input
      0 input packets with dribble condition detected
      285811 packets output, 27449284 bytes, 0 underruns
      0 output errors, 0 collisions, 2 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier
      0 output buffer failures, 0 output buffers swapped out

Note: There is a difference between the counter of show interface command output for a physical interface and a VLAN interface.The input packet counters increment in the output ofshow interfacefor a VLAN interface when that packet is Layer 3 (L3) processed by the CPU. Traffic that is Layer 2 (L2) switched never makes it to the CPU and is not counted in theshow interfacecounters for the VLAN interface. It would be counted on theshow interfaceoutput for the appropriate physical interface.

Theshow interfaces <card-type> <slot/port> counters errorscommand is used in Cisco IOS to display the output of the interface errors only. SeeTable 1for explanations of the error counter output.

Router#sh interfaces fastEthernet 6/1 counters errors

 Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards
 Fa6/1               0          0          0          0         0           0

 Port      Single-Col Multi-Col  Late-Col Excess-Col Carri-Sen     Runts    Giants
 Fa6/1              0         0         0          0         0         0         0

Table 1. Cisco IOS error counter output forshow interfacesorshow interfaces<card-type> <x/y> counters errorsfor the Catalyst 6000 and 4000 Series.

Counters (in alphabetical order)	Issues and Common Causes that Increase Error Counters
Align-Err	Description:show interfaces counters errors. Alignment errors are a count of the number of frames received that do not end with an even number of octets and have a bad Cyclic Redundancy Check (CRC).Common Causes:These are usually the result of a duplex mismatch or a physical problem (such as cabling, a bad port, or a bad NIC). When the cable is first connected to the port, some of these errors can occur. Also, if there is a hub connected to the port, collisions between other devices on the hub can cause these errors.Platform Exceptions:Alignment errors are not counted on the Catalyst 4000 Series Supervisor I (WS-X4012) or Supervisor II (WS-X4013).
babbles	Description:show interfaces counter indicates that the transmit jabber timer expired. A jabber is a frame longer than 1518 octets (which exclude frame bits, but include FCS octets), which does not end with an even number of octets (alignment error) or has a bad FCS error.
Carri-Sen	Description:show interfaces counters errors. The Carri-Sen (carrier sense) counter increments every time an Ethernet controller wants to send data on a half-duplex connection. The controller senses the wire and checks if it is not busy before it transmits.Common Causes:This is normal on an half-duplex Ethernet segment.
collisions	Descriptions:show interfacescounter. The number of times a collision occurred before the interface transmitted a frame to the media successfully.Common Causes:Collisions are normal for interfaces configured as half-duplex but must not be seen on full duplex interfaces. If collisions increase dramatically, this points to a highly utilized link or possibly a duplex mismatch with the attached device.
CRC	Description:show interfacescounter. This increments when the CRC generated by the LAN station or far-end device that originates the traffic does not match the checksum calculated from the data received.Common Causes:This usually indicates noise or transmission problems on the LAN interface or the LAN itself. A high number of CRCs is usually the result of collisions but can also indicate a physical issue (such as cabling, bad interface or NIC) or a duplex mismatch.
deferred	Description:show interfacescounter. The number of frames that have been transmitted successfully after they wait because the media was busy.Common Causes:This is usually seen in half-duplex environments where the carrier is already in use when it tries to transmit a frame.
pause input	Description:show interfacescounter. An increment in pause input counter means that the connected device requests for a traffic pause when its receive buffer is almost full.Common Causes:This counter is incremented for informational purposes since the switch accepts the frame. The pause packets stop when the connected device is able to receive the traffic.
input packets with dribble condition	Description:show interfacescounter. A dribble bit error indicates that a frame is slightly too long.Common Causes:This frame error counter is incremented for informational purposes, since the switch accepts the frame.
Excess-Col	Description: show interfaces counters errors. A count of frames for which transmission on a particular interface fails due to excessive collisions. An excessive collision happens when a packet has a collision 16 times in a row. The packet is then dropped. Common Causes: Excessive collisions are typically an indication that the load on the segment needs to be split across multiple segments but can also point to a duplex mismatch with the attached device. Collisions must not be seen on interfaces configured as full duplex.
FCS-Err	Description: show interfaces counters errors. The number of valid size frames with Frame Check Sequence (FCS) errors but no frame errors. Common Causes: This is typically a physical issue (such as cabling, a bad port, or a bad Network Interface Card (NIC)) but can also indicate a duplex mismatch.
frame	Description: show interfaces counter. The number of packets received incorrectly that has a CRC error and a non-integer number of octets (alignment error). Common Causes: This is usually the result of collisions or a physical problem (such as cabling, bad port or NIC) but can also indicate a duplex mismatch.
Giants	Description: show interfaces and show interfaces counters errors. Frames received that exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet) and have a bad Frame Check Sequence (FCS). Common Causes: In many cases, this is the result of a bad NIC. Try to find the offending device and remove it from the network. Platform Exceptions: Catalyst Cat4000 Series that run Cisco IOS Previous to software Version 12.1(19)EW, the giants counter incremented for a frame > 1518bytes. After 12.1(19)EW, a giant in show interfaces increments only when a frame is received >1518bytes with a bad FCS.
ignored	Description: sh interfaces counter. The number of received packets ignored by the interface because the interface hardware ran low on internal buffers. Common Causes: Broadcast storms and bursts of noise can cause the ignored count to be increased.
Input errors	Description: show interfaces counter. Common Causes: This includes runts, giants, no buffer, CRC, frame, overrun, and ignored counts. Other input-related errors can also cause the input errors count to be increased, and some datagrams can have more than one error. Therefore, this sum cannot balance with the sum of enumerated input error counts. Also refer to the section Input Errors on a Layer 3 Interface Connected to a Layer 2 Switchport.
Late-Col	Description: show interfaces and show interfaces counters errors.The number of times a collision is detected on a particular interface late in the transmission process. For a 10 Mbit/s port this is later than 512 bit-times into the transmission of a packet. Five hundred and twelve bit-times corresponds to 51.2 microseconds on a 10 Mbit/s system. Common Causes: This error can indicate a duplex mismatch among other things. For the duplex mismatch scenario, the late collision is seen on the half-duplex side. As the half-duplex side transmits, the full duplex side does not wait its turn and transmits simultaneously which causes a late collision. Late collisions can also indicate an Ethernet cable or segment that is too long. Collisions must not be seen on interfaces configured as full duplex.
lost carrier	Description: show interfaces counter. The number of times the carrier was lost in transmission. Common Causes: Check for a bad cable. Check the physical connection on both sides.
Multi-Col	Description: show interfaces counters errors. The number of times multiple collisions occurred before the interface transmitted a frame to the media successfully. Common Causes: Collisions are normal for interfaces configured as half-duplex but must not be seen on full duplex interfaces. If collisions increase dramatically, this points to a highly utilized link or possibly a duplex mismatch with the attached device.
no buffer	Description:show interfaces counter. The number of received packets discarded because there is no buffer space.Common Causes:Compare with ignored count. Broadcast storms can often be responsible for these events.
no carrier	Description:show interfacescounter. The number of times the carrier was not present in the transmission.Common Causes:Check for a bad cable. Check the physical connection on both sides.
Out-Discard	Description:The number of outbound packets chosen to be discarded even though no errors have been detected.Common Causes:One possible reason to discard such a packet can be to free up buffer space.
output buffer failures output buffers swapped out	Description:show interfacescounter. The number of failed buffers and the number of buffers swapped out.Common Causes:A port buffers the packets to the Tx buffer when the rate of traffic switched to the port is high and it cannot handle the amount of traffic. The port starts to drop the packets when the Tx buffer is full and thus increases the underruns and the output buffer failure counters. The increase in the output buffer failure counters can be a sign that the ports are run at an inferior speed and/or duplex, or there is too much traffic that goes through the port. As an example, consider a scenario where a 1gig multicast stream is forwarded to 24 100 Mbps ports. If an egress interface is over-subscribed, it is normal to see output buffer failures that increment along with Out-Discards. For troubleshoot information, see theDeferred Frames (Out-Lost or Out-Discard)section of this document.
output errors	Description:show interfacescounter. The sum of all errors that prevented the final transmission of datagrams out of the interface.Common Cause:This issue is due to the low Output Queue size.
overrun	Description:The number of times the receiver hardware was unable to hand received data to a hardware buffer.Common Cause:The input rate of traffic exceeded the ability of the receiver to handle the data.
packets input/output	Description:show interfacescounter. The total error free packets received and transmitted on the interface. Monitor these counters for increments as it is useful to determine whether traffic flows properly through the interface. The bytes counter includes both the data and MAC encapsulation in the error free packets received and transmitted by the system.
Rcv-Err	Description: For the Catalyst 6000 Series only — show interfaces counters error.Common Causes:See Platform Exceptions.Platform Exceptions:Catalyst 5000 Seriesrcv-err = receive buffer failures. For example, a runt, giant, or an FCS-Err does not increment the rcv-err counter. The rcv-err counter on a 5K only increments as a result of excessive traffic. OnCatalyst 4000 Seriesrcv-err = the sum of all receive errors, which means, in contrast to the Catalyst 5000, that the rcv-err counter increments when the interface receives an error like a runt, giant or FCS-Err.
Runts	Description:show interfacesandshow interfaces counters errors. The frames received that are smaller than the minimum IEEE 802.3 frame size (64 bytes for Ethernet), and with a bad CRC.Common Causes:This can be caused by a duplex mismatch and physical problems, such as a bad cable, port, or NIC on the attached device.Platform Exceptions:Catalyst 4000 Series that run Cisco IOSPrevious to software Version 12.1(19)EW, a runt = undersize. Undersize = frame < 64bytes. The runt counter only incremented when a frame less than 64 bytes was received. After 12.1(19EW, a runt = a fragment. A fragment is a frame < 64 bytes but with a bad CRC. The result is the runt counter now increments inshow interfaces, along with the fragments counter inshow interfaces counters errorswhen a frame <64 bytes with a bad CRC is received.Cisco Catalyst 3750 Series SwitchesIn releases prior to Cisco IOS 12.1(19)EA1, when dot1q is used on the trunk interface on the Catalyst 3750, runts can be seen onshow interfacesoutput because valid dot1q encapsulated packets, which are 61 to 64 bytes and include the q-tag, are counted by the Catalyst 3750 as undersized frames, even though these packets are forwarded correctly. In addition, these packets are not reported in the appropriate category (unicast, multicast, or broadcast) in receive statistics. This issue is resolved in Cisco IOS release 12.1(19)EA1 or 12.2(18)SE or later.
Single-Col	Description:show interfaces counters errors. The number of times one collision occurred before the interface transmitted a frame to the media successfully.Common Causes:Collisions are normal for interfaces configured as half-duplex but must not be seen on full duplex interfaces. If collisions increase dramatically, this points to a highly utilized link or possibly a duplex mismatch with the attached device.
throttles	Description:show interfaces. The number of times the receiver on the port is disabled, possibly because of buffer or processor overload. If an asterisk (*) appears after the throttles counter value, it means that the interface is throttled at the time the command is run.Common Causes:Packets which can increase the processor overload include IP packets with options, expired TTL, non-ARPA encapsulation, fragmentation, tunnels, ICMP packets, packets with MTU checksum failure, RPF failure, IP checksum and length errors.
underruns	Description:The number of times that the transmitter has been that run faster than the switch can handle.Common Causes:This can occur in a high throughput situation where an interface is hit with a high volume of traffic bursts from many other interfaces all at once. Interface resets can occur along with the underruns.
Undersize	Description:show interfaces counters errors . The frames received that are smaller than the minimum IEEE 802.3 frame size of 64 bytes (which excludes frame bits but includes FCS octets) that are otherwise well formed.Common Causes:Check the device that sends out these frames.
Xmit-Err	Description:show interfaces counters errors. This is an indication that the internal send (Tx) buffer is full.Common Causes:A common cause of Xmit-Err can be traffic from a high bandwidth link that is switched to a lower bandwidth link, or traffic from multiple inbound links that are switched to a single outbound link. For example, if a large amount of traffic bursts comes in on a gigabit interface and is switched out to a 100Mbps interface, this can cause Xmit-Err to increment on the 100Mbps interface. This is because the output buffer of the interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.

Show Interfaces Counters for Cisco IOS

To monitor inbound and outbound traffic on the port as displayed by the next output, for unicast, multicast, and broadcast traffic. Theshow interfacescard-type {slot/port}counterscommand is used when you run Cisco IOS on the Supervisor.

Note: There is, an Out-Discard counter in the Cisco IOSshow interfaces counters errorscommand which is explained inTable 1.

Router#sh interfaces fas 6/1 counters

  Port            InOctets   InUcastPkts   InMcastPkts   InBcastPkts
  Fa6/1           47856076            23        673028           149

  Port           OutOctets  OutUcastPkts  OutMcastPkts  OutBcastPkts
  Fa6/1           22103793            17        255877          3280
  Router#
  
!--- Cisco IOS counters used to monitor inbound and outbound unicast, multicast
!--- and broadcast packets on the interface.

Show Counters Interface for Cisco IOS

Theshow counters interfacecard-type {slot/port}command was introduced in Cisco IOS software version 12.1(13)E for the Catalyst 6000 series only, it offers even more detailed statistics for ports and interfaces. This commanda display the 32-bit and 64-bit error counters per port or interface.

Show Controller Ethernet-Controller for Cisco IOS

For Catalyst 3750, 3550, 2970, 2950/2955, 2940, and 2900/3500XL switches use the command show controller ethernet-controller to display traffic counter and error counter output that is similar to theoutput for Catalyst 6000 series switches.

3550-1#show controller ethernet-controller fastEthernet 0/1
  !--- Output from a Catalyst 3550.

    Transmit FastEthernet0/1           Receive
          0 Bytes                        0 Bytes
          0 Unicast frames               0 Unicast frames
          0 Multicast frames             0 Multicast frames
          0 Broadcast frames             0 Broadcast frames
          0 Discarded frames             0 No dest, unicast
          0 Too old frames               0 No dest, multicast
          0 Deferred frames              0 No dest, broadcast
          0  1 collision frames
          0  2 collision frames          0 FCS errors
          0  3 collision frames          0 Oversize frames
          0  4 collision frames          0 Undersize frames
          0  5 collision frames          0 Collision fragments
          0  6 collision frames
          0  7 collision frames          0 Minimum size frames
          0  8 collision frames          0 65 to 127 byte frames
          0  9 collision frames          0 128 to 255 byte frames
          0 10 collision frames          0 256 to 511 byte frames
          0 11 collision frames          0 512 to 1023 byte frames
          0 12 collision frames          0 1024 to 1518 byte frames
          0 13 collision frames
          0 14 collision frames          0 Flooded frames
          0 15 collision frames          0 Overrun frames
          0 Excessive collisions         0 VLAN filtered frames
          0 Late collisions              0 Source routed frames
          0 Good (1 coll) frames         0 Valid oversize frames
          0 Good(>1 coll) frames         0 Pause frames
          0 Pause frames                 0 Symbol error frames
          0 VLAN discard frames          0 Invalid frames, too large
          0 Excess defer frames          0 Valid frames, too large
          0 Too large frames             0 Invalid frames, too small
          0 64 byte frames               0 Valid frames, too small
          0 127 byte frames
          0 255 byte frames
          0 511 byte frames
          0 1023 byte frames
          0 1518 byte frames

 3550-1#
 
!--- See the next table for additional counter output for 2900/3500XL Series switches.

Counter	Description	Possible Causes
Transmitted Frames
Discarded frames	The total number of frames whose transmission attempt is abandoned due to insufficient resources. This total includes frames of all destination types.	The traffic load on the interface is excessive and causes the frames to be discarded. Reduce the traffic load on the interface if there are increments in the number of packets in this field.
Too old frames	Number of frames that took longer than two seconds to travel through the switch. For this reason, they were discarded by the switch. This only happens under extreme, high stress conditions.	The traffic load for this switch is excessive and causes the frames to be discarded. Reduce the switch load if the number of packets in this field increase. You can need to modify your network topology to reduce the traffic load for this switch.
Deferred frames	The total number of frames whose first transmission attempt was delayed, due to traffic on the network media. This total includes only those frames that are subsequently transmitted without error and not affected by collisions.	The traffic load destined for this switch is excessive and causes the frames to be discarded. Reduce the switch load if the number of packets in this field increase. You can need to modify your network topology to reduce the traffic load for this switch.
Collision frames	The collision frames counters are the number of times a packet was attempted to be transmitted but was not successful but was successful on its next attempt. This means that if the 2 collision frames counter incremented, the switch attempted to send the packet twice and failed but was successful on its third attempt.	The traffic load on the interface is excessive and causes the frames to be discarded. Reduce the traffic load on the interface if you see the number of packets increase in these fields.
Excessive collisions	The excessive collisions counter increases after 16 consecutive late collisions have occurred in a row. After 16 attempts have been made to send the packet the packet is dropped, and the counter increments.	If this counter increments, it is an indication of a wiring problem, an excessively loaded network, or a duplex mismatch. An excessively loaded network can be caused by too many devices on a shared Ethernet.
Late collisions	A late collision occurs when two devices transmit at the same time, and neither side of the connection detects a collision. The reason for this occurrence is because the time to propagate the signal from one end of the network to another is longer than the time to put the entire packet on the network. The two devices that cause the late collision never see that each sends until after it puts the entire packet on the network. Late collisions are not detected by the transmitter until after the first 64 byte slot time. This is because they are only detected in transmissions of packets longer than 64 bytes.	Late collisions are a result of incorrect cabling or a non-compliant number of hubs in the network. Bad NICs can also cause late collisions.
Good (1 coll) frames	The total number of frames which experience exactly one collision and are then successfully transmitted.	Collisions in a half-duplex environment are normal expected behavior.
Good (>1 coll) frames	The total number of frames which experience between 2 and 15 collisions, inclusive, and are then successfully transmitted.	Collisions in a half-duplex environment are normal expected behavior. Frames that increment at the upper end of this counter can exceed the 15 collisions and can be counted as Excessive collisions.
VLAN discardframes	The number of frames dropped on an interface because the CFI bit is set.	The Canonical Format Indicator (CFI) bit in the TCI of an 802.1q frame is is set to 0 for the ethernet canonical frame format. If the CFI bit is set to 1, this indicates the presence of a RIF (Routing Information Field) or Token Ring noncanonical frame which is discarded.
Received Frames
No bandwidth frames	2900/3500XL only.The number of times that a port received a packet from the network, but the switch did not have the resources to receive it. This only happens under stress conditions but can happen with bursts of traffic on several ports. So, a small number of No bandwidth frames is not a cause for concern. (It still must be far less than one percent of the frames received.)	The traffic load on the interface is excessive and causes the frames to be discarded. Reduce the traffic load on the interface if you see the number of packets increase in these fields.
No buffers frames	2900/3500XL only.The number of times that a port received a packet from the network, but the switch did not have the resources to receive it. This only happens under stress conditions but can happen with bursts of traffic on several ports. So, a small number of No buffers frames is not a cause for concern. (It still must be far less than one percent of the frames received.)	The traffic load on the interface is excessive and causes the frames to be discarded. Reduce the traffic load on the interface if you see the number of packets increase in these fields.
No dest, unicast	No destination unicast are the number of unicast packets that the port did not forward to any other ports.	These are brief descriptions of when the No dest, (unicast, multicast, and broadcast) counters can increment: If a port is an access port, and the port is connected to an Inter-Switch Link Protocol (ISL) trunk port, the No dest counter is very large since all inbound ISL packets are not forwarded. This is an invalid configuration. If a port is blocked by Spanning Tree Protocol (STP), most packets are not orwarded, which results in No dest packets. If a port just acquired a link, there is a very brief (less than one second) period where inbound packets are not forwarded. If the port is in a VLAN by itself, and no other ports on the switch belong to that VLAN, all inbound packets are dropped and the counter increments. The counter also increments when the destination address of the packet is learned on the port that the packet was received on. If a packet was received on port 0/1, with destination MAC address X, and the switch has already learned that MAC address X resides on port 0/1, it increments the counter and discards the packet. This can happen in these situations: If a hub is connected to port 0/1, and a workstation connected to the hub transmits a packets to another workstation connected to the hub, port 0/1 does not forward this packet anywhere because the destination MAC resides on the same port. This can also occur if a switch is connected to port 0/1 and starts to flood packets to all of its ports to learn MAC addresses. If a static address has been set up on another port in the same VLAN, and no static address was set up for the receiving port, the packet is dropped. For example, if a static map for MAC address X was configured on port 0/2 to forward traffic to port 0/3, the packet must be received on port 0/2 otherwise the packet is dropped. If a packet is sent from any other port, in the same VLAN as port 0/2, the packet is dropped. If the port is a secure port, packets with disallowed source MAC addresses are not forwarded and increment the counter.
No dest, multicast	No destination multicast are the number of multicast packets that the port did not forward to any other ports.
No dest, broadcast	No destination broadcast are the number of broadcast packets that the port did not forward to any other ports.
Alignment errors	Alignment errors are the number of frames received that do not end with an even number of octets and have a bad CRC.	Alignment errors are due to the frame that is not completely copied to the wire, which results in fragmented frames. Alignment errors are the result of collisions at half-duplex, a duplex mismatch, bad hardware (NIC, cable, or port), or connected devices that generate frames that do not end with an octet and have a bad FCS.
FCS errors	FCS error count is the number of frames that were received with a bad checksum (CRC value) in the Ethernet frame. These frames are dropped and not propagated onto other ports.	FCS errors are the result of collisions at half-duplex, a duplex mismatch, bad hardware (NIC, cable, or port), or connected devices that generate frames that do not end with an octet and have a bad FCS.
Undersize frames	These are the total number of packets received that were less than 64 octets long (which excludes frame bits but includes FCS) and have a good FCS value.	This is an indication of a bad frame generated by the connected device. Verify that the connected device operates correctly.
Oversize frames	Number of packets received by the port from the network, where the packets were more than 1514 bytes.	This can be an indication of faulty hardware, dot1q or ISL trunking configuration issues.
Collision fragments	The total number of frames whose length is less than 64 octets (which excludes frame bits, but includes FCS) and have a bad FCS value.	If this counter increments, this is an indication that the ports are configured at half-duplex. Set the duplex to full-duplex.
Overrun frames	The number of times the receiver hardware was unable to hand received data to a hardware buffer.	The input rate of traffic exceeded the ability of the receiver to handle the data.
VLAN filtered frames	The total number of frames which are filtered because of the type of VLAN information contained in the frame.	The port can be configured to filter 802.1Q tagged frames. When a frame is received which contains an 802.1Q tag the frame is filtered and this statistic is incremented.
Source routed frames	The total number of receive frames that are discarded due to situation that the source route bit is set in the source address of the native frame.	This kind of source routing is only defined for Token Ring and FDDI. The IEEE ethernet specification forbids this bit to be set in any Ethernet frame. Therefore, the switch discards such frames.
Valid oversize frames	The total number of frames received whose length exceeds the System MTU, yet which have good FCS values.	This statistic counts frames that exceed the configured System MTU, but which can have been increased from 1518 bytes to allow for Q-in-Q or MPLS encapsulations.
Symbol error frames	Gigabit Ethernet (1000 Base-X) uses 8B/10B Encoding to translate 8bit data from the MAC sublayer(layer 2) to a 10bit Symbol to send over the wire. When a port receives a Symbol, it extracts the 8 bit data from the Symbol (10 bits).	A Symbol error means the interface detects an undefined (invalid) Symbol received. Small amounts of symbol errors can be ignored. Large amounts of symbol errors can indicate a bad device, cable, or hardware.
Invalid frames, too large	Giant frames or frames received that exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet) and have a bad Frame Check Sequence (FCS).	In many cases, this is the result of a bad NIC. Try to find the offending device and remove it from the network.
Invalid frames, too small	Runt frames or frames received that are less than 64 bytes (which includes the FCS bits and excludes the frame header) and have either an FCS error or an alignment error.	This can be caused by a duplex mismatch and physical problems, such as a bad cable, port, or NIC on the attached device.

Common System Error Messages

For the Cisco IOS system messages format, you can refer to theMessages and Recovery Procedures Guidefor the release of software you run. For example, you can look at theMessages and Recovery Proceduresfor Cisco IOS Releases.

%AMDP2_FE-3-UNDERFLO

This error message is caused when a frame is transmitted, and the local buffer of the controller chip local buffer receives insufficient data. The data cannot be transferred to the chip fast enough to keep pace with output rate. Normally, such a condition is temporary, dependent upon transient peak loads within the system. The issue occurs when an excessive amount of traffic is processed by the Fast Ethernet interface. The error message is received when the traffic level reaches about 2.5 Mb. This traffic level constrain is due to hardware limitation. Because of this, a chance exists for the device connected to the catalyst switch to drop packets.

The resolution is that ordinarily the system recovers automatically. No action is required. If the switch overwhelms the Ethernet interface, check the speed and duplex setup. Also, use a sniffer program to analyze packets that come in and out of the router fast Ethernet interface. In order to avoid packet drops on the device connected to the catalyst switch, issue theip cefcommand on the fast Ethernet interface of the device connected to the switch.

%INTR_MGR-DFC1-3-INTR: Queueing Engine (Blackwater) [1]: FIC Fabric-A Received Unexpected Control Code

The reason for this error message is the receipt of a packet from the switch fabric, where the CRC value in the fabric header on that packet did not match the CRC value calculated by the Fabric Interface Controller (FIC) subblock of the Blackwater ASIC. This indicates that a corruption of the packet occurred within transfer, and Blackwater received the corrupted packet.

Command Rejected: [Interface] not a Switching Port

In switches that support both L3 interfaces and L2 switchport, the message «Command rejected: [interface] not a switching port»displays when you try to enter a command related to layer 2 on a port that is configured as a layer 3 interface.

In order to convert the interface from layer 3 mode to layer 2 mode, issue the interface configuration commandswitchport. After you issue this command, configure the port for any layer 2 properties.

Common Port and Interface Problems

Port or Interface Status is Disable or Shutdown

An obvious but sometimes overlooked cause of port connectivity failure is an incorrect configuration on the switch. If a port has a solid orange light, this means the software inside the switch shut down the port, either by way of the user interface or by internal processes.

Note: Some port LEDs of the platform work differently in regard to STP. For example, the Catalyst 1900/2820 turns ports orange when they are in STP block mode. In this case, an orange light can indicate the normal functions of the STP. The Catalyst 6000/4000 does not turn the port light orange when it blocks for STP.

Make sure the port or module has not been disabled or powered down for some reason. If a port or module is manually shut down on one side of the link or the other, the link does not come up until you re-enable the port. Check the port status on both sides. Use theshow run interfacecommand and check to see if the interface is in ashutdownstate:

Switch#show run interface fastEthernet 4/2
!
interface FastEthernet4/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 shutdown
 duplex full
 speed 100
end

!--- Use the no shut command in config-if mode to re-enable this interface.  

If the port goes into shutdown mode immediately after a reboot of the switch, the probable cause is the port security setup. If unicast flooding is enabled on that port, it can cause the port to shut down after a reboot. Cisco recommends that you disable the unicast flooding because it also ensure that no flooding occurs on the port once the MAC address limit is reached.

Port or Interface Status is errDisable

By default, software processes inside the switch can shut down a port or interface if certain errors are detected.

When you look at show interfacecard-type {slot/port}statuscommand for Cisco IOS:

Router#show interface fastethernet 2/4 status

  Port    Name               Status       Vlan       Duplex  Speed Type
  Gi2/4                      err-disabled 1            full   1000 1000BaseSX

  !--- The show interfaces card-type {slot/port} status command for Cisco IOS
  !--- displays a status of errdisabled.
  !--- The show interfaces status errdisabled command shows all the interfaces
  !--- in this status.

Theshow loggingcommand for Cisco IOS also display the error messages (exact message format varies) that relate to the errdisable state.

Wheb ports or interlaces are shut down as a result of errdisable are referred to as causes in Cisco IOS. The causes for this range from EtherChannel misconfiguration that causes a PAgP flap, duplex mismatch, BPDU port-guard and portfast configured at the same time, UDLD that detects a one-way link, and so on.

You have to manually re-enable the port or interface to take it out the errdisable state unless you configure an errdisable recovery option. InCisco IOS software you have the ability to automatically re-enable a port after a configurable amount of time spent in the errdisable state. The bottom line is that even if you configure the interface to recover from errdisable the problem reoccurs until the root cause is determined.

Note: Use this Recover Errdisable Port State on Cisco IOS Platforms for more information on errdisable status on switches that run Cisco IOS.

This table shows an example of the commands used to configure verify and troubleshoot the errdisable status on switches. Navigate to the link for more information about the commands Recover Errdisable Port State on Cisco IOS Platforms:

Action	Cisco IOS errdisable Commands
Configure	errdisable detect cause
Configure	errdisable recovery cause
Configure	errdisable recovery interval <timer_interval_in_seconds>
Verify & Troubleshoot	show errdisable detect
Verify & Troubleshoot	show interfaces status err-disabled

Port or Interface Status is Inactive

One common cause of inactive ports on switches that run Cisco IOS is when the VLAN they belong to disappears. This can occur when interfaces are configured as layer 2 switchports that use theswitchportcommand.

Every port in a Layer 2 switch belongs to a VLAN. Every port on a Layer 3 switch configured to be a L2 switchport must also belong to a VLAN. If that VLAN is deleted, then the port or interface becomes inactive.

Note: Some switches show a steady orange (amber) light on each port when this happens.

Use theshow interfacescard-type {slot/port}switchportcommand along withshow vlanto verify.

Router#show interfaces fastEthernet 4/47 switchport
  Name: Fa4/47Switchport: Enabled
  Administrative Mode: static access
  Operational Mode: static access
  Administrative Trunking Encapsulation: negotiate
  Operational Trunking Encapsulation: native
  Negotiation of Trunking: Off
  Access Mode VLAN: 11 ((Inactive))

  !--- FastEth 4/47 is inactive.

 Router#show vlan

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 1    default                          active    Gi1/1, Gi2/1, Fa6/6
 10   UplinkToGSR's                    active    Gi1/2, Gi2/2
 
 !--- VLANs are displayed in order and VLAN 11 is not available.
 
 30   SDTsw-1ToSDTsw-2Link             active	Fa6/45

If the switch that deleted the VLAN is a VTP server for the VTP domain, every server and client switch in the domain has the VLAN removed from their VLAN table as well. When you add the VLAN back into the VLAN table from a VTP server switch, the ports of the switches in the domain that belong to that restored VLAN become active again. A port remembers what VLAN it is assigned to, even if the VLAN itself is deleted. Refer toUnderstanding and Configuring VLAN Trunk Protocol (VTP)for more information on VTP.

Note: If the output of theshow interface <interface> switchportcommand displays the port as a trunk port even after you configure the port as an access port with theswitchport access vlan <vlan>command, issue theswitchport mode accesscommand in order to make the port an access port.

Uplink Port or Interface Status is Inactive

On a Catalyst 4510R series switch, in order to enable both the 10-Gigabit Ethernet and the Gigabit Ethernet SFP uplink ports, there is an optional configuration. In order to enable the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces, issue thehw-module uplink select allcommand. After you issue the command, re-boot the switch or else the output of theshow interface status module <module number>command shows the uplink port as inactive.

Cisco IOS Software Release 12.2(25)SG supports the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces on Catalyst 4500 switches.

Note: On the Catalyst 4503, 4506, and 4507R series switches, this capability is automatically enabled.

Deferred Counter on the Catalyst Switch Interface Increments

The issue is because the traffic load destined for the switch is excessive and causes the frames to be discarded. Normally the deferred frames are the number of frames that have been transmitted successfully after waiting for the media, because the media was busy. This is usually seen in half-duplex environments where the carrier is already in use when it tries to transmit a frame. But in full duplex environments the issue occurs when the excessive load is destined for the switch.

This is the workaround:

• Hardcode both ends of the link to full duplex so that the negotiation mismatch can be avoided.

• Change the cable and patch panel cord to ensure that the cable and patch cords are not defective.

Note: If the Deferred Counter error increments on a GigabitEthernet of a Supervisor 720, turn on speed negotiation on the interface as a workaround.

Intermittent Failure to set timer [value] from vlan [vlan no]

The issue occurs when Encoded Address Recognition Logic (EARL) is unable to set the CAM aging time for the VLAN to the required number of seconds. Here, the VLAN aging time is already set to fast aging.

When the VLAN is already in fast aging, EARL cannot set the VLAN to fast aging, and aging timer set process is blocked. The default CAM aging time is five minutes, which means that the switch flushes the table of learned MAC addresses every five minutes. This ensures that the MAC address table (the CAM table) contains the most recent entries.

Fast aging temporarily sets the CAM aging time to the number of seconds that the user specifies, and is used in conjunction with the Topology Change Notification (TCN) process. The idea is that when a topology change occurs, this value is necessary to flush the CAM table faster, to compensate for the topology change.

Issue theshow cam agingcommand to check the CAM aging time on the switch. TCNs and fast aging are fairly rare. As a result, the message has a severity level of 3. If the VLANs are frequently in fast aging, check the reason for fast aging.

The most common reason for TCNs is client PCs connected directly to a switch. When you power up or down the PC, the switch port changes state, and the switch starts the TCN process. This is because the switch does not know that the connected device is a PC; the switch only knows that the port has changed the state.

In order to resolve this issue, Cisco has developed the PortFast feature for host ports. An advantage of PortFast is that this feature suppresses TCNs for a host port.

Note: PortFast also bypasses spanning-tree calculations on the port, and is therefore only suitable for use on a host port.

Trunking Mode Mismatch

Check the trunking mode on each side of the link. Make sure both sides are in the same mode (both trunking with the same method: ISL or 802.1q, or both not trunking). If you turn the trunking mode to on (as opposed to auto or desirable) for one port and the other port has the trunking mode set to off, they are not able to communicate. Trunking changes the formatting of the packet. The ports need to be in agreement as to what format they use on the link, or they do not understand each other.

For Cisco IOS, use theshow interfacescard-type {mod/port}trunkcommand to verify the trunking configuration and Native VLAN.

Router#sh interfaces fastEthernet 6/1 trunk

  Port      Mode         Encapsulation  Status        Native vlan
  Fa6/1     desirable    802.1q         trunking      1

  Port      Vlans allowed on trunk
  Fa6/1     1-4094
!--- Output truncated.

Refer to these documents for more information on the different trunking modes, guidelines, and restrictions:

• System Requirements to Implement Trunking

• Trunking Technology Support Page

Jumbos, Giants, and Baby Giants

The Maximum Transmission Unit (MTU) of the data portion of an ethernet frame is 1500 bytes by default. If the transmitted traffic MTU exceeds the supported MTU the switch does not forward the packet. Also, dependent upon the hardware and software, some switch platforms increment port and interface error counters as a result.

• Jumbo frames are not defined as part of the IEEE Ethernet standard and are vendor-dependent. They can be defined as any frame bigger than the standard ethernet frame of 1518 bytes (which includes the L2 header and Cyclic Redundancy Check (CRC)). Jumbos have larger frame sizes, typically > 9000 bytes.

• Giant frames are defined as any frame over the maximum size of an ethernet frame (larger than 1518 bytes) that has a bad FCS.

• Baby Giant frames are just slightly larger than the maximum size of an ethernet frame. Typically this means frames up to 1600 bytes in size.

Support for jumbo and baby giants on Catalyst switches varies by switch platform, sometimes even by modules within the switch. The software version is also a factor.

Refer toConfiguring Jumbo/Giant Frame Support on Catalyst Switchesfor more information on system requirements, configure and troubleshoot for jumbo and baby giant issues.

Cannot Ping End Device

Check the end device with a ping sent from the directly connected switch first, then work your way back port by port, interface by interface, trunk by trunk until you find the source of the connectivity issue. Make sure each switch can see the end device MAC address in its Content-Addressable Memory (CAM) table.

Use theshow mac address-table dynamiccommand or substitute theinterfacekeyword.

Router# show mac-address-table interface fastEthernet 6/3
Codes: * - primary entry

  vlan   mac address     type    learn qos            ports
------+----------------+--------+-----+---+--------------------------
*    2  0040.ca14.0ab1   dynamic  No    --  Fa6/3

!--- A workstation on VLAN 2 with MAC address 0040.ca14.0ab1 is directly connected
!--- to interface fastEthernet 6/3 on a switch running Cisco IOS.

Once you know the switch actually has the MAC address of the device in the CAM table, determine whether this device is on the same or different VLAN from where you try to ping.

If the end device is on a different VLAN from where you try to ping, a L3 switch or router must be configured to allow the devices to communicate. Make sure your L3 addressing on the end device and on the router/ L3 switch is correctly configured. Check the IP address, subnet mask, default gateway, dynamic routing protocol configuration, static routes, and so on.

Use of Switchport Host to Fix Startup Delays

If stations are not able to talk to their primary servers when they connect through the switch, the problem can involve delays on the switch port when it tries to become active after the physical layer link comes up. In some cases, these delays can be up to 50 seconds. Some workstations simply cannot wait this long to find their server and then they give up. These delays are caused by STP, trunking negotiations (DTP), and EtherChannel negotiations (PAgP). All of these protocols can be disabled for access ports where they are not needed, so the switch port or interface starts forwarding packets a few seconds after it establishes a link with its neighbor device.

In Cisco IOS, you can use theswitchport host command to disable channeling and to enable spanning-tree portfast and theswitchport nonegotiatecommand to turn off DTP negotiation packets. Use theinterface-range command to do this on multiple interfaces at once.

Router6k-1(config)#interface range fastEthernet 6/13 - 18
Router6k-1(config-if-range)#switchport
Router6k-1(config-if-range)#switchport host
switchport mode can be set to access
spanning-tree portfast can be enabled
channel group can be disabled
!--- Etherchannel is disabled and portfast is enabled on interfaces 6/13 - 6/18.
Router6k-1(config-if-range)#switchport nonegotiate
!--- Trunking negotiation is disabled on interfaces 6/13 - 6/18.
Router6k-1(config-if-range)#end
Router6k-1#

Cisco IOS has the option to use theglobal spanning-tree portfast defaultcommand to automatically apply portfast to any interface configured as a layer 2 access switchport. Check the Command Reference for your release of software to verify the availability of this command. You can also use thespanning-tree portfastcommand per interface, but this requires that you turn off trunking and etherchannel separately to help fix workstation startup delays.

Note: Refer toUsing Portfast and Other Commands to Fix Workstation Startup Connectivity Delaysfor more information how to fix startup delays.

Speed/Duplex, auto-negotiation, or NIC Issues

If you have a large amount of alignment errors, FCS errors, or late collisions, this can indicate one of these:

• Duplex Mismatch

• Bad or Damaged Cable

• NIC Card Issues

Duplex Mismatch

A common issue with speed/duplex is when the duplex setup are mismatched between two switches, between a switch and a router or between the switch and a workstation or server. This can occur when you manually hardcode the speed and duplex or from auto-negotiation issues between the two devices.

If the mismatch occurs between two Cisco devices with the Cisco Discovery Protocol (CDP) enabled, you see the CDP error messages on the console or in the logging buffer of both devices. CDP is useful to detect errors, as well as port and system statistics on nearby Cisco devices. CDP is Cisco proprietary and works when you send packets to a well-known MAC address 01-00-0C-CC-CC-CC.

The example shows the log messages that result from a duplex mismatch between two Catalyst 6000 series switches: one that runs CatOS, and the other that runs Cisco IOS. These messages generally tell you what the mismatch is and where it occurs.

2003 Jun 02 11:16:02 %CDP-4-DUPLEXMISMATCH:Full/half duplex mismatch detected on port 3/2
!--- CatOS switch sees duplex mismatch.

Jun  2 11:16:45 %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet6/2
(not half duplex), with TBA04251336 3/2 (half duplex).
!--- Cisco IOS switch sees duplex mismatch.

Use theshow cdp neighborscard-type <slot/port>detailcommand to display CDP information for Cisco neighbor devices.

Router#show cdp neighbors fastEthernet 6/1 detail
-------------------------
Device ID: TBA04251336
Entry address(es):
  IP address: 10.1.1.1
Platform: WS-C6006,  Capabilities: Trans-Bridge Switch IGMP
Interface: FastEthernet6/1,  Port ID (outgoing port): 3/1
Holdtime : 152 sec
Version :
WS-C6006 Software, Version McpSW: 6.3(3) NmpSW: 6.3(3)
Copyright (c) 1995-2001 by Cisco Systems
!--- Neighbor device to FastEth 6/1 is a Cisco Catalyst 6000 Switch
!--- on port 3/1 running CatOS.
advertisement version: 2
VTP Management Domain: 'test1'
Native VLAN: 1
Duplex: full
!--- Duplex is full.
Router#

setup auto speed/duplex on one side and 100/Full-duplex on the other side is also a misconfiguration and can result in a duplex mismatch. If the switch port receives a lot of late collisions, this usually indicates a duplex mismatch problem and can place the port in  an errdisable status in a result. The half-duplex side only expects packets at certain times, not at any time, and therefore counts packets received at the wrong time as collisions. There are other causes for late collisions besides duplex mismatch, but this is one of the most common reasons. Always set both sides of the connection to auto-negotiate speed/duplex or set the speed/duplex manually on both sides.

Use theshow interfaces <card-type> <slot/port>statuscommand to display speed and duplex setup as well as other information. Use thespeedandduplexcommands from interface configuration mode to hardcode both sides to 10 or 100 and half or full as necessary.

Router#show interfaces fasstEthernet 6/1 status
Port    Name               Status       Vlan       Duplex  Speed Type
Fa6/1                      connected    1          a-full  a-100 10/100BaseTX

If you use theshow interfacescommand without thestatusoption, you see a setup for speed and duplex, but you do not know whether this speed and duplex was achieved through auto-negotiation or not.

Router#sh int fas 6/1
FastEthernet6/1 is up, line protocol is up (connected)
  Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Full-duplex, 100Mb/s

!--- Full-duplex and 100Mbps does not tell you whether autoneg was used to achieve this.
!--- Use the sh interfaces fas 6/1 status command to display this.

Bad or damaged cable

Always check the cable for marginal damage or failure. A cable can be just good enough to connect at the physical layer, but it corrupts packets as a result of subtle damage to the wiring or connectors. Check or swap the copper or fiber cable. Swap the GBIC (if removable) for fiber connections. Rule out any bad patch panel connections or media convertors between source and destination. Try the cable in another port or interface if one is available and see if the problem continues.

Auto negotiation and NIC Card Issues

Problems sometimes occur between Cisco switches and certain third-party NIC cards. By default, Catalyst switch ports and interfaces are set to autonegotiate. It is common for devices like laptops or other devices to be set to autonegotiate as well, yet sometimes autonegotation issues occur.

In order to troubleshoot auto-negotiation problems it is often recommended to try and hardcode both sides. If neither auto-negotiation or hardcode setup seem to work, there can be a problem with the firmware or software on your NIC card. Upgrade the NIC card driver to the latest version available on the web site of the manufacture to resolve this.

Refer toConfiguring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto-Negotiationfor details on how to resolve speed/duplex and auto-negotiation issues.

Refer toTroubleshooting Cisco Catalyst Switches to NIC Compatibility Issuesfor details on how to resolve third-party NIC issues.

Spanning Tree Loops

Spanning Tree Protocol (STP) loops can cause serious performance issues that masquerade as port or interface problems. In this situation, your bandwidth is used by the same frames over and over again, which leaves little room for legitimate traffic.

The STP loop guard feature provides additional protection against Layer 2 forwarding loops (STP loops). An STP loop is created when an STP block port in a redundant topology erroneously transitions to the forwarding state. This usually happens because one of the ports of a physically redundant topology (not necessarily the STP block port) no longer receives STP BPDUs. In its operation, STP relies on continuous reception or transmission of BPDUs based on the port role. The designated port transmits BPDUs, and the non-designated port receives BPDUs.

When one of the ports in a physically redundant topology no longer receives BPDUs, the STP conceives that the topology is loop free. Eventually, the block port from the alternate or backup port becomes designated and moves to a forwarding state. This situation creates a loop.

The loop guard feature makes additional checks. If BPDUs are not received on a non-designated port, and loop guard is enabled, that port is moved into the STP loop-inconsistent block state, instead of the listening / learning / forwarding state. Without the loop guard feature, the port assumes the designated port role. The port moves to the STP forwarding state and creates a loop. Refer toSpanning-Tree Protocol Enhancements using Loop Guard and BPDU Skew Detection Featuresfor more information on the loop guard feature.

This document covers reasons that STP can fail, what information to look for to identify the source of the problem, and what kind of design minimizes STP risks.

Loops can also be caused by a uni-directional link. For more information, refer to the UDLD: One-Way link problems section of this document.

UDLD: One-Way Link

A unidirectional link is a link where traffic goes out one way, but no traffic is received in the ingress direction. The switch does not know that the link ingress direction is bad (the port thinks that the link is up and works).

A broken fiber cable or other cabling/port issues can cause this one-way only communication. These partially functional links can cause problems such as STP loops when the switches involved do not know that the link is partially broken. UDLD can put a port in errdisable state when it detects a unidirectional link. The command udld aggressive-mode can be configured on switches that run Cisco IOS (check release notes for command availability) for point-to-point connections between switches where unidirectional links cannot be tolerated. The use of this feature can help you identify difficult to find unidirectional link problems

Refer toUnderstand and Configure the Unidirectional Link Detection Protocol (UDLD) Featurefor configuration information on UDLD.

Deferred Frames (Out-Lost or Out-Discard)

If you have a large number of deferred frames, or Out-Discard (also referred to as Out-Lost on some platforms), it means that the switch output buffers have filled up and the switch had to drop these packets. This can be a sign that this segment is run at an inferior speed and/or duplex, or there is too much traffic that goes through this port.

Use theshow interfaces counters errorcommand to look at OutDiscards.

Router#show interfaces counters error
Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards
Fa7/47              0          0          0          0         0           0
Fa7/48              0          0          0          0         0     2871800
Fa8/1               0          0          0          0         0     2874203
Fa8/2             103          0          0        103         0     2878032
Fa8/3             147          0          0        185         0           0
Fa8/4             100          0          0        141         0     2876405
Fa8/5               0          0          0          0         0     2873671
Fa8/6               0          0          0          0         0           2
Fa8/7               0          0          0          0         0           0

!--- The show interfaces counters errors command shows certain interfaces
!--- that increment in large amounts OutDiscards while others run clean.

Investigate these common causes of output buffer failures:

Inferior Speed/Duplex for the Amount of Traffic

Your network can send too many packets through this port for the port to handle at its current speed/duplex setup. This can happen where you have multiple high-speed ports flowing to a single (usually slower) port. You can move the device that hangs off this port to faster media. For example, if the port is 10 Mbps, move this device to a 100 Mbps or Gigabit port. You can change the topology to route frames differently.

Congestion Issues: Segment Too Busy

If the segment is shared, other devices on this segment can transmit so much that the switch has no opportunity to transmit. Avoid daisy-chained hubs whenever possible. Congestion can lead to packet loss. Packet loss causes retransmissions at the transport layer which in turn causes users to experience latency at the application level. You can upgrade10Mbps links to 100Mbps or Gigabit Ethernet links when possible. You can remove some devices from crowded segments to other less populated segments. Make congestion avoidance a priority on your network.

Applications

At times the traffic transmission characteristics of the applications used can lead to output buffer problems. NFS file transfers that come from a Gigabit attached server that uses user datagram protocol (UDP) with a 32K window size is one example of an application setup that can bring out this type of problem. If you have checked or tried the other suggestions in this document (checked speed/duplex, no physical errors on the link, all the traffic is normal valid traffic, and so on), then reduce the unit size that is sent by the application which can help to alleviate this problem.

Software Problems

If you see behavior that can only be considered strange, you can isolate the behavior to a specific box, and you have looked at everything suggested so far, this can indicate software or hardware problems. It is usually easier to upgrade the software than it is to upgrade hardware. Change the software first.

Use theshow versioncommand to verify the current software version along with thedir flash: ordir bootflash: (dependent upon the platform) command to verify the available flash memory for the upgrade:

Router#show version
Cisco Internetwork Operating System Software
IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-IS-M), Version 12.1(13)EW, EA
RLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Fri 20-Dec-02 13:52 by eaarmas
Image text-base: 0x00000000, data-base: 0x00E638AC
ROM: 12.1(12r)EW
Dagobah Revision 71, Swamp Revision 24
trunk-4500 uptime is 2 weeks, 2 days, 6 hours, 27 minutes
System returned to ROM by redundancy reset
System image file is "bootflash:cat4000-is-mz.121-13.EW.bin"

!--- Typical Cisco IOS show version output.

Router#dir bootflash:
Directory of bootflash:/
1  -rw-     8620144   Mar 22 2002 08:26:21  cat4000-is-mz.121-13.EW.bin
61341696 bytes total (52721424 bytes free)

!--- Verify available flash memory on switch running Cisco IOS.

How to Upgrade Software

For information on how to upgrade software for your Cisco Switches, navigate to link, choose your platform and look at the Software Configuration section.

Hardware Software Incompatibility

There can be a situation where the software is not compatible with the hardware. This happens when new hardware comes out and requires special support from the software. For more information on software compatibility, use the Software Advisor tool.

Software Bugs

The operating system can have a bug. If you load a newer software version, it can often fix this. You can search known software bugs with the Software Bug Toolkit.

Corrupt Images

An image can have become corrupted. For information in regard to the recovery from corrupted images, choose your platform Switch and look at the Troubleshoot section.

Hardware Problems

Check the results ofshow modulefor Catalyst 6000 and 4000 series switches that run Cisco IOS.

Check the results of the POST results from the switch to see if there were any failures indicated for any part of the switch. Failures of any test of a module or port show an ‘F’ in the test results.

For Cisco IOS, on modular switches like the Cat6000, use the commandshow diagnostics. In order to see POST results per module, use theshow diagnostics module<module> command.

ecsj-6506-d2#sh diagnostic module 3
  Current Online Diagnostic Level = Minimal
  !--- The diagnostic level is set to minimal which is a shorter,
  !--- but also less thorough test result.
  !--- You may wish to configure diagnostic level complete to get more test results.
  Online Diagnostic Result for Module 3 : MINOR ERROR
  Online Diagnostic Level when Line Card came up = Minimal
  Test Results: (. = Pass, F = Fail, U = Unknown)
  1 . TestLoopback :
  Port  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
  ----------------------------------------------------------------------------
        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  F  F  F  F F  F
 
  !--- Notice the MINOR ERROR test result and failed loopback test which means
  !--- these ports are currently unusable.
  !--- Use the hw-module{mod}reset command or, if necessary, physically reseat the
  !--- module to try and fix this problem.

  !--- If these steps fail, open a case with Cisco Technical Support.

Note: For Catalyst 3750, 3550, 2970 , 2950/2955, and 2900/3500XL Series switches use theshow postcommand, which indicates a simple pass or fail for the hw status. Use the LEDs on these switches to help you understand the POST results.

For further information on how to troubleshoot hardware problems on Catalyst switches that run Cisco IOS, navigate to the Cisco Switches support pages, choose your platform and look at the Troubleshooting > Hardwaresection. For possible issues related to Field Notices, refer toField Noticesfor LAN and ATM Switches.

Input Errors on a Layer 3 Interface Connected to a Layer 2 Switchport

By default, all layer 2 ports are indynamic desirablemode, so the layer 2 port tries to form a trunk link and sends out DTP packets to the remote device. When a layer 3 interface is connected to a layer 2 switchport, it is not able to interpret these frames, which results in Input errors, WrongEncap errors, and Input queue drops.

In order to resolve this, change the mode of the switch port tostatic accessortrunkas per your requirement.

Switch2(config)#interface fastEthernet1/0/12
Switch2(config-if)#switchport mode access

Or

Switch2(config)#interface fastEthernet1/0/12
Switch2(config-if)#switchport trunk encapsulation dot1q
Switch2(config-if)#switchport mode trunk

Rapidly Increment Rx-No-Pkt-Buff Counter and Input Errors

The Rx-No-Pkt-Buff counter can increase on ports when it has blades, such as WS-X4448-GB-RJ45, WS-X4548-GB-RJ45, and WS-X4548-GB-RJ45V. Also, some packet drop incrementation is normal and is the result of traffic bursts traffic.

These types of errors increase rapidly, especially when the traffic that passes through that link is high or when it has devices such as servers connected to that interface. This high load of traffic oversubscribes the ports, which exhausts the input buffers and causes the Rx-No-Pkt-Buff counter and input errors to increase rapidly.

If a packet cannot be completely received because the switch is out of packet buffers, this counter is incremented once for every dropped packet. This counter indicates the internal state of the Switching ASICs on the Supervisor and does not necessarily indicate an error condition.

Pause Frames

When the receive part (Rx) of the port has its Rx FIFO queue filled and reaches the high water mark, the transmit part (Tx) of the port starts to generate pause frames with an interval value mentioned in it. The remote device is expected to stop / reduce the transmission of packets for the interval time mentioned in the pause frame.

If the Rx is able to clear the Rx queue or reach low water mark within this interval, Tx sends out a special pause frame that mentions the interval as zero (0x0). This enables the remote device to start to transmit packets.

If the Rx still works on the queue, once the interval time expires, the Tx sends a new pause frame again with a new interval value.

If Rx-No-Pkt-Buff is zero or does not increment and the TxPauseFrames counter increments, it indicates that our switch generates pause frames and the remote end obeys, hence Rx FIFO queue depletes.

If Rx-No-Pkt-Buff increments and TxPauseFrames also increments, it means that the remote end disregards the pause frames (does not support flow control) and continues to send traffic despite the pause frames. In order to overcome this situation, manually configure the speed and duplex, as well as disable the flow control, if required.

These types of errors on the interface are related to a traffic problem with the ports oversubscribed. The WS-X4448-GB-RJ45, WS-X4548-GB-RJ45, and WS-X4548-GB-RJ45V switching modules have 48 oversubscribed ports in six groups of eight ports each:

• Ports 1, 2, 3, 4, 5, 6, 7, 8

• Ports 9, 10, 11, 12, 13, 14, 15, 16

• Ports 17, 18, 19, 20, 21, 22, 23, 24

• Ports 25, 26, 27, 28, 29, 30, 31, 32

• Ports 33, 34, 35, 36, 37, 38, 39, 40

• Ports 41, 42, 43, 44, 45, 46, 47, 48

The eight ports within each group use common circuitry that effectively multiplexes the group into a single, non-block, full-duplex Gigabit Ethernet connection to the internal switch fabric. For each group of eight ports, the frames that are received are buffered and sent to the common Gigabit Ethernet link to the internal switch fabric. If the amount of data received for a port begins to exceed buffer capacity, flow control sends pause frames to the remote port to temporarily stop traffic and prevent frame loss.

If the frames received on any group exceeds the bandwidth of 1 Gbps, the device starts to drop the frames. These drops are not obvious as they are dropped at the internal ASIC rather than the actual interfaces. This can lead to slow throughput of packets across the device.

The Rx-No-Pkt-Buff does not depend on the total traffic rate. It depends on the amount of the packets that are stored in the Rx FIFO buffer of the module ASIC. The size of this buffer is only 16 KB. It is counted with short traffic bursts flow when some packets fill this buffer. Thus, Rx-No-Pkt-Buff on each port can be counted when the total traffic rate of this ASIC port group exceeds 1 Gbps, since WS-X4548-GB-RJ45 is 8:1 oversubscribed module.

When you have devices that need to carry a large amount of traffic through that interface, consider the use of one port of each group so that the common circuitry that shares a single group is not affected by this amount of traffic. When the Gigabit Ethernet switching module is not fully utilized, you can balancee the port connections across port groupings to maximize available bandwidth. For example, with the WS-X4448-GB-RJ45 10/100/1000 switching module, you can connect ports from different groups, such as ports 4, 12, 20, or 30 (in any order), before you connect ports from the same group, such as ports 1, 2, 3, 4, 5, 6, 7, and 8. If this does not solve the issue, you need to consider a module without any oversubscription of ports.

Understand Unknown Protocol Drops

Unknown protocol dropsis a counter on the interface. It is caused by protocols that are not understood by the router/switch. This example of theshow run interfacecommand shows the unknown protocol drops on the GigabitEthernet 0/1 interface.

Switch#show run interface GigabitEthernet0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is BCM1125 Internal MAC, address is 0000.0000.0000 (via 0000.0000)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:05, output 00:00:03, output hang never
  Last clearing of "show interface" counters 16:47:42
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3031 packets input, 488320 bytes, 0 no buffer
     Received 3023 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 63107 multicast, 0 pause input
     0 input packets with dribble condition detected
     7062 packets output, 756368 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2015 unknown protocol drops
     4762 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Unknown protocol drops are normally dropped because the interface where these packets are received is not configured for this type of protocol, or it can be any protocol that the router does not recognize. For example, if you have two routers connected and you disable CDP on one router interface, this results in unknown protocol drops on that interface. The CDP packets are no longer recognized, and they are dropped.

Trunking between a Switch and a Router

Trunk links between a switch and a router can make the switchport go down. Trunk can come up after you disable and enable the switchport, but eventually the switchport can go down again.

In order to resolve this issue, complete these steps:

1. Make sure Cisco Discovery Protocol (CDP) runs between the switch and router and both can see each other.

2. Disable theKeepaliveson the interface of the router.

3. Reconfigure the trunk encapsulation on both devices.

When the keepalives are disabled, the CDP enables link to operate normally.

Connectivity Issues due to Oversubscription

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Refer toInterface/Module Connectivity Problemsfor more information on oversubscription.

Sub Interfaces in SPA Modules

In SPA modules, after you create a sub interface with 802.1Q, the same VLAN is not usable on the switch. Once you have encapsulation dot1q on a subinterface, you can no longer use that VLAN in the system because the 6500 or 7600 internally allocates the VLAN and makes that sub interface its only member. In order to resolve this issue, create trunk ports instead of sub interfaces. That way, the VLAN can be seen in all interfaces.

Troubleshoot Output Drops

Typically, the output drops can occur if QoS is configured and does not provide enough bandwidth to certain class of packets. It also occurs when the hardware hits an oversubscription.

For example, here you see a high amount of output drops on the interface GigabitEthernet 8/9 on a Catalyst 6500 Series Switch:

Switch#show interface GigabitEthernet8/9
GigabitEthernet8/9 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0013.8051.5950 (bia 0013.8051.5950)
  Description: Connection To Bedok_Core_R1 Ge0/1
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 18/255, rxload 23/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:28, output 00:00:10, output hang never
  Last clearing of "show interface" counters never
Input queue: 0/2000/3/0 (size/max/drops/flushes); Total output drops: 95523364
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 94024000 bits/sec, 25386 packets/sec
  5 minute output rate 71532000 bits/sec, 24672 packets/sec
     781388046974 packets input, 406568909591669 bytes, 0 no buffer
     Received 274483017 broadcasts (257355557 multicasts)
     0 runts, 0 giants, 0 throttles
     3 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     749074165531 packets output, 324748855514195 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

In order to analyze the problem, collect the output of these commands:

• show fabric utilization detail

• show fabric errors

• show platform hardware capacity

• show catalyst6000 traffic-meter

• show platform hardware capacity rewrite-engine drop

Last Input Never from the Output of Show interface Command

This example of the show interface command shows theLast input neveron the TenGigabitEthernet1/15 interface.

Switch#show interface TenGigabitEthernet1/15
TenGigabitEthernet1/15 is up, line protocol is up (connected)
  Hardware is C6k 10000Mb 802.3, address is 0025.84f0.ab16 (bia 0025.84f0.ab16)
  Description: lsnbuprod1 solaris
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:17, output hang never
  Last clearing of "show interface" counters 2d22h
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 46000 bits/sec, 32 packets/sec
     52499121 packets input, 3402971275 bytes, 0 no buffer
     Received 919 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     118762062 packets output, 172364893339 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

This shows the number of hours, minutes, and seconds since the last packet was successfully received by an interface and processed locally on the router. This is useful to know when a dead interface has failed. This counter is updated only when packets are process switched, not when packets are fast switched. Last input nevermeans there was no successful interface packet transfer to other end point or terminal. Usually this means there was no packet transfer relative to that entity.

Related Information

• Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues
• Using PortFast and Other Commands to Fix Workstation Startup Connectivity Delays
• Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation
• Upgrade Software Images and Working with Configuration Files on Catalyst Switches
• Technical Support & Documentation — Cisco Systems

Introduction

This chapter presents general troubleshooting information and a discussion of tools and techniques for troubleshooting serial connections. The chapter consists of the following sections:

• Troubleshooting Using the show interfaces serial Command

• Using the show controllers Command

• Using debug Commands

• Using Extended ping Tests

• Troubleshooting Clocking Problems

• Adjusting Buffers

• Special Serial Line Tests

• Detailed Information on the show interfaces serial Command

• Troubleshooting T1 Problems

• Troubleshooting E1 Problems

Prerequisites

Requirements

Readers of this document should be knowledgeable of the following definitions.

• DTE = data terminal equipment

• CD = Carrier Detect

• CSU = channel service unit

• DSU = digital service unit

• SCTE = serial clock transmit external

• DCE = data circuit-terminating equipment

• CTS = clear-to-send

• DSR = data-set ready

• SAP = Service Advertising Protocol

• IPX = Internetwork Packet Exchange

• FDDI = Fiber Distributed Data Interface

• ESF = Extended Superframe Format

• B8ZS = binary eight-zero substitution

• LBO = Line Build Out

Components Used

This document is not restricted to specific software and hardware versions.

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Troubleshooting Using the show interfaces serial Command

The output of the show interfaces serial EXEC command displays information specific to serial interfaces. Figure 15-1 shows the output of the show interfaces serial EXEC command for a High-Level Data Link Control (HDLC) serial interface.

This section describes how to use the show interfaces serial command to diagnose serial line connectivity problems in a wide area network (WAN) environment. The following sections describe some of the important fields of the command output.

Other fields shown in the display are described in detail in the section «Detailed Information on the show interfaces serial Command,» later in this chapter.

Serial Lines: show interfaces serial Status Line Conditions

You can identify five possible problem states in the interface status line of the show interfaces serial display (see Figure 15-1):

• Serial x is down, line protocol is down

• Serial x is up, line protocol is down

• Serial x is up, line protocol is up (looped)

• Serial x is up, line protocol is down (disabled)

• Serial x is administratively down, line protocol is down

Figure 15-1 Output of the HDLC show interface serial Command

Table 15-1: Serial Lines: show interfaces serial Status Line Conditions — This table shows the interface status conditions, possible problems associated with the conditions, and solutions to those problems.

Status Line Condition	Possible Problem	Solution
Serial x is up, line protocol is up		This is the proper status line condition. No action required.
Serial x is down, line protocol is down (DTE mode)	Typically indicates that the router is not sensing a CD signal (that is, CD is not active). Telephone company problem-Line is down or line is not connected to CSU/DSU Faulty or incorrect cabling Hardware failure (CSU/DSU)	Check the LEDs on the CSU/DSU to see if CD is active, or insert a breakout box on the line to check for the CD signal. Verify that you are using the proper cable and interface (see your hardware installation documentation). Insert a breakout box and check all control leads. Contact your leased-line or other carrier service to see if there is a problem. Swap faulty parts. If you suspect faulty router hardware, change the serial line to another port. If the connection comes up, the previously connected interface has a problem.
Serial x is up, line protocol is down (DTE mode)	Local or remote router is misconfigured Keepalives are not being sent by remote router Leased-line or other carrier service problem-Noisy line, or misconfigured or failed switch Timing problem on cable (SCTE not set on CSU/DSU) Failed local or remote CSU/DSU Failed local or remote CSU/DSU Router hardware failure (local or remote)	Put the modem, CSU, or DSU in local loopback mode and use the show interfaces serial command to see if the line protocol comes up. If the line protocol comes up, a telephone company problem or a failed remote router is the likely problem. If the problem appears to be on the remote end, repeat Step 1 on the remote modem, CSU, or DSU. Verify all cabling. Make sure that the cable is attached to the correct interface, the correct CSU/DSU, and the correct telephone company network termination point. Use the show controllers EXEC command to determine which cable is attached to which interface. Enable the debug serial interface EXEC command. Caution: Because debugging output is assigned a high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of low network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use. If the line protocol does not come up in local loopback mode and if the output of the debug serial interface EXEC command shows that the keepalive counter is not incrementing, a router hardware problem is likely. Swap router interface hardware. If the line protocol comes up and the keepalive counter increments, the problem is not in the local router. Troubleshoot the serial line as described in the sections «Troubleshooting Clocking Problems» and «CSU and DSU Loopback Tests,» later in this chapter. If you suspect faulty router hardware, change the serial line to an unused port. If the connection comes up, the previously connected interface has a problem.
Serial x is up, line protocol is down (DCE mode)	Missing clockrate interface configuration command DTE device does not support or is not set up for SCTE mode Failed remote CSU or DSU Failed or incorrect cable Router hardware failure	Add the clockrate interface configuration command on the serial interface. Syntax: clock rate bps Syntax Description: bps-Desired clock rate in bits per second: 1200, 2400, 4800, 9600, 19200, 38400, 56000, 64000, 72000, 125000, 148000, 250000, 500000, 800000, 1000000, 1300000, 2000000, 4000000, or 8000000. Set the DTE device to SCTE mode if possible. If your CSU/DSU does not support SCTE, you may have to disable SCTE on the Cisco router interface. See the section «Inverting the Transmit Clock,» later in this chapter. Verify that the correct cable is being used. If the line protocol is still down, there is a possible hardware failure or cabling problem. Insert a breakout box and observe leads. Replace faulty parts as necessary.
Serial x is up, line protocol is up (looped)	A loop exists in the circuit. The sequence number in the keepalive packet changes to a random number when a loop is initially detected. If the same random number is returned over the link, a loop exists.	Use the show running-config privileged EXEC command to look for any loopback interface configuration command entries. If you find a loopback interface configuration command entry, use the no loopback interface configuration command to remove the loop. If you do not find the loopback interface configuration command, examine the CSU/DSU to see if they are configured in manual loopback mode. If they are, disable manual loopback. Reset the CSU or DSU, and inspect the line status. If the line protocol comes up, no other action is needed. If the CSU or DSU is not configured in manual loopback mode, contact the leased-line or other carrier service for line troubleshooting assistance.
Serial x is up, line protocol is down (disabled)	High error rate due to telephone company service problem CSU or DSU hardware problem Bad router hardware (interface)	Troubleshoot the line with a serial analyzer and breakout box. Look for toggling CTS and DSR signals. Loop CSU/DSU (DTE loop). If the problem continues, it is likely that there is a hardware problem. If the problem does not continue, it is likely that there is a telephone company problem. Swap-out bad hardware as required (CSU, DSU, switch, local or remote router).
Serial x is administratively down, line protocol is down	Router configuration includes the shutdown interface configuration command Duplicate IP address	Check the router configuration for the shutdown command. Use the no shutdown interface configuration command to remove the shutdown command. Verify that there are no identical IP addresses using the show running-config privileged EXEC command or the show interfaces EXEC command. If there are duplicate addresses, resolve the conflict by changing one of the IP addresses.

Serial Lines: Increasing Output Drops on Serial Link

Output drops appear in the output of the show interfaces serial command (see Figure 15-1) when the system is attempting to hand off a packet to a transmit buffer but no buffers are available.

Symptom: An increasing number of output drops on serial link.

Table 15-2 Serial Lines: Increasing Output Drops on Serial Link — This table outlines the possible problem that may cause this symptom and suggests solutions.

Possible Problem

Solution

Input rate to serial interface exceeds bandwidth available on serial link

Minimize periodic broadcast traffic (such as routing and SAP updates) by using access lists or by other means. For example, to increase the delay between SAP updates, use the ipx sap-interval interface configuration command. Increase the output hold queue size in small increments (for instance, 25 percent), using the hold-queue out interface configuration command. On affected interfaces, turn off fast switching for heavily used protocols. For example, to turn off IP fast switching, enter the no ip route-cache interface configuration command. For the command syntax for other protocols, consult the Cisco IOS configuration guides and command references. Implement priority queuing on slower serial links by configuring priority lists. For information on configuring priority lists, see the Cisco IOS configuration guides and command references. Note: Output drops are acceptable under certain conditions. For instance, if a link is known to be overused (with no way to remedy the situation), it is often preferable to drop packets than to hold them. This is true for protocols that support flow control and can retransmit data (such as TCP/IP and Novell IPX). However, some protocols, such as DECnet and local-area transport are sensitive to dropped packets and accommodate retransmission poorly, if at all.

Serial Lines: Increasing Input Drops on Serial Link

Input drops appear in the output of the show interfaces serial EXEC command (see Figure 15-1) when too many packets from that interface are still being processed in the system.

Symptom: An increasing number of input drops on serial link.

Table 15-3: Serial Lines: Increasing Input Drops on Serial Link — This table outlines the possible problem that may cause this symptom and suggests solutions.

Possible Problem

Solution

Input rate exceeds the capacity of the router or input queues exceed the size of output queues

Note: Input drop problems are typically seen when traffic is being routed between faster interfaces (such as Ethernet, Token Ring, and FDDI) and serial interfaces. When traffic is light, there is no problem. As traffic rates increase, backups start occurring. Routers drop packets during these congested periods. Increase the output queue size on common destination interfaces for the interface that is dropping packets. Use the hold-queue out interface configuration command. Increase these queues by small increments (for instance, 25percent) until you no longer see drops in the show interfaces output. The default output hold queue limit is 100 packets. Reduce the input queue size, using the hold-queue in interface configuration command, to force input drops to become output drops. Output drops have less impact on the performance of the router than do input drops. The default input hold queue is 75 packets.

Serial Lines: Increasing Input Errors in Excess of One Percent of Total Interface Traffic

If input errors appear in the show interfaces serial output (see Figure 15-1), there are several possible sources of those errors. The most likely sources are summarized in Table 15-4.

Note: Any input error value for cyclic redundancy check (CRC) errors, framing errors, or aborts above one percent of the total interface traffic suggests some kind of link problem that should be isolated and repaired.

Symptom: An increasing number of input errors in excess of one percent of total interface traffic.

Table 15-4: Serial Lines: Increasing Input Errors in Excess of One Percent of Total Interface Traffic

Possible Problem

Solution

The following problems can result in this symptom: Faulty telephone company equipment Noisy serial line Incorrect clocking configuration (SCTE not set) Incorrect cable or cable too long Bad cable or connection Bad CSU or DSU Bad router hardware Data converter or other device being used between router and DSU

Note: Cisco strongly recommends not using data converters when you are connecting a router to a WAN or serial network. Use a serial analyzer to isolate the source of the input errors. If you detect errors, it is likely that there is a hardware problem or a clock mismatch in a device that is external to the router. Use the loopback and ping tests to isolate the specific problem source. For more information, see sections «Using the trace Command» and «CSU and DSU Loopback Tests,» later in this chapter. Look for patterns. For example, if errors occur at a consistent interval, they could be related to a periodic function such as the sending of routing updates.

Serial Lines: Troubleshooting Serial Line Input Errors

Table 15-5: This table describes the various types of input errors displayed by the show interfaces serial command (see Figure 15-1), possible problems that may be causing the errors and the solutions to those problems.

Input Error Type (Field Name)	Possible Problem	Solution
CRC errors (CRC)	CRC errors occur when the CRC calculation does not pass-indicating that data is corrupted-for one of the following reasons: Noisy serial line Serial cable is too long, or cable from the CSU/DSU to the router is not shielded SCTE mode is not enabled on DSU CSU line clock is incorrectly configured Ones density problem on T1 link (incorrect framing or coding specification)	Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary. Make sure the cable is within the recommended length-no more than 50 feet (15.24 meters) or 25 feet (7.62 meters) for T1 link. Ensure that all devices are properly configured for a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the section «Inverting the Transmit Clock,» later in this chapter. Make certain that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS). Contact your leased-line or other carrier service and have it perform integrity tests on the line.
Framing errors (frame)	A framing error occurs when a packet does not end on an 8-bit byte boundary for one of the following reasons: Noisy serial line Improperly designed cable; serial cable is too long; the cable from the CSU or DSU to the router is not shielded SCTE mode is not enabled on the DSU; the CSU line clock is incorrectly configured; one of the clocks is configured for local clocking Ones density problem on T1 link (incorrect framing or coding specification)	Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary. Make certain you are using the correct cable. Make sure the cable is within the recommended length-no more than 50 feet (15.24 meters) or 25 feet (7.62 meters) for T1 link. Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the section «Inverting the Transmit Clock» later in this chapter. Make certain that the local and remote CSU/DSU is configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS). Contact your leased-line or other carrier service and have it perform integrity tests on the line.
Aborted transmission (abort)	Aborts indicate an illegal sequence of one bits (more than seven in a row). The following are possible reasons for this occurrence: SCTE mode is not enabled on DSU CSU line clock is incorrectly configured Serial cable is too long or cable from the CSU or DSU to the router is not shielded Ones density problem on T1 link (incorrect framing or coding specification) Packet terminated in middle of transmission-typical cause being an interface reset or a framing error Hardware problem-bad circuit, bad CSU/DSU, or bad sending interface on remote router	Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the section «Inverting the Transmit Clock,» later in this chapter. Shield the cable if necessary. Make certain the cable is within the recommended length-no more than 50 feet (15.24 meters) or 25 feet (7.62 meters) for T1 link. Ensure that all connections are good. Check the hardware at both ends of the link. Swap faulty equipment as necessary. Lower data rates and see if aborts decrease. Use local and remote loopback tests to determine where aborts are occurring. See the section «Special Serial Line Tests,» later in this chapter. Contact your leased-line or other carrier service and have it perform integrity tests on the line.

Serial Lines: Increasing Interface Resets on Serial Link

Interface resets that appear in the output of the show interfaces serial EXEC command (see Figure 15-1) are the result of missed keep-alive packets.

Symptom: An increasing number of interface resets on serial link.

Table 15-6: This table outlines the possible problems that may cause this symptom and suggests solutions.

Possible Problem

Solution

The following problems can result in this symptom: Congestion on link (typically associated with output drops) Bad line causing CD transitions Possible hardware problem at the CSU, DSU, or switch

When interface resets are occurring, examine other fields of the show interfaces serial command output to determine the source of the problem. Assuming that an increase in interface resets is being recorded, examine the following fields: If there is a high number of output drops in the show interfaces serial output, see the section «Serial Lines: Increasing Output Drops on Serial Link,» earlier in this chapter. Check the carrier transitions field in the show interfaces serial display. If carrier transitions are high while interface resets are being registered, the problem is likely to be a bad link or bad CSU or DSU. Contact your leased-line or carrier service and swap faulty equipment as necessary. Examine the input errors field in the show interfaces serial display. If input errors are high while interface resets are increasing, the problem is probably a bad link or bad CSU/DSU. Contact your leased-line or other carrier service and swap faulty equipment as necessary.

Serial Lines: Increasing Carrier Transitions Count on Serial Link

Carrier transitions appear in the output of the show interfaces serial EXEC command whenever there is an interruption in the carrier signal (such as an interface reset at the remote end of a link).

Symptom: An increasing number of carrier transitions count on serial link.

Table 15-7 outlines the possible problems that may cause this symptom and suggests solutions.

Table 15-7: Serial Lines: Increasing Carrier Transitions Count on Serial Link

Possible Problem	Solution
The following problems can result in this symptom: Line interruptions due to an external source (such as physical separation of cabling, red or yellow T1 alarms, or lightning striking somewhere along the network) Faulty switch, DSU, or router hardware	Check hardware at both ends of the link. Attach a breakout box or a serial analyzer and test to determine source of problems. If an analyzer or breakout box is unable to identify any external problems, check the router hardware. Swap faulty equipment as necessary.

Using the show controllers Command

The show controllers EXEC command is another important diagnostic tool when troubleshooting serial lines. The command syntax varies depending on the platform:

• For serial interfaces on Cisco 7000 series routers, use the show controllers cbus EXEC command.

• For Cisco access products, use the show controllers EXEC command.

• For the AGS, CGS, and MGS, use the show controllers mci EXEC command.

Figure 15-2 shows the output from the show controllers cbus EXEC command. This command is used on Cisco 7000 series routers with the Fast Serial Interface Processor (FSIP) card. Check the command output to make certain that the cable to the channel service unit/digital service unit (CSU/DSU) is attached to the proper interface. You can also check the microcode version to see if it is current.

Figure 15-2: show controllers cbus Command Output

On access products such as the Cisco 2000, Cisco 2500, Cisco 3000, and Cisco 4000 series access servers and routers, use the show controllers EXEC command. Figure 15-3 shows the show controllers command output from the Basic Rate Interface (BRI) and serial interfaces on a Cisco 2503 access server. (Note that some output is not shown.)

The show controllers output indicates the state of the interface channels and whether a cable is attached to the interface. In Figure 15-3, serial interface 0 has an RS-232 DTE cable attached. Serial interface 1 has no cable attached.

Figure 15-4 shows the output of the show controllers mci command. This command is used on AGS, CGS, and MGS routers only. If the electrical interface is displayed as UNKNOWN (instead of V.35, EIA/TIA-449, or some other electrical interface type), an improperly connected cable is the likely problem. A bad applique or a problem with the internal wiring of the card is also possible. If the electrical interface is unknown, the corresponding display for the show interfaces serial EXEC command will show that the interface and line protocol are down.

Figure 15-3: show controllers Command Output

Figure 15-4: show controllers mci Command Output

Using debug Commands

The output of the various debug privileged EXEC commands provides diagnostic information relating to protocol status and network activity for many internetworking events.

Caution:  Because debugging output is assigned a high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of low network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use. When you finish using a debug command, remember to disable it with its specific no debug command or with the no debug all command.

The following debug commands are useful when troubleshooting serial and WAN problems. More information about the function and output of each of these commands is provided in the Debug Command Reference publication:

• debug serial interface— Verifies whether HDLC keepalive packets are incrementing. If they are not, a possible timing problem exists on the interface card or in the network.

• debug x25 events— Detects X.25 events, such as the opening and closing of switched virtual circuits (SVCs). The resulting «cause and diagnostic» information is included with the event report.

• debug lapb— Outputs Link Access Procedure, Balanced (LAPB) or Level 2 X.25 information.

• debug arp— Indicates whether the router is sending information about or learning about routers (with ARP packets) on the other side of the WAN cloud. Use this command when some nodes on a TCP/IP network are responding but others are not.

• debug frame-relay lmi— Obtains Local Management Interface (LMI) information useful for determining if a Frame Relay switch and a router are sending and receiving LMI packets.

• debug frame-relay events— Determines if exchanges are occurring between a router and a Frame Relay switch.

• debug ppp negotiation— Shows Point-to-Point Protocol (PPP) packets transmitted during PPP startup, where PPP options are negotiated.

• debug ppp packet— Shows PPP packets being sent and received. This command displays low-level packet dumps.

• debug ppp errors— Shows PPP errors (such as illegal or malformed frames) associated with PPP connection negotiation and operation.

• debug ppp chap— Shows PPP Challenge Handshake Authentication Protocol (CHAP) and Password Authentication Protocol (PAP) packet exchanges.

• debug serial packet— Shows Switched Multimegabit Data Service (SMDS) packets being sent and received. This display also prints error messages to indicate why a packet was not sent or was received erroneously. For SMDS, the command dumps the entire SMDS header and some payload data when an SMDS packet is transmitted or received.

Using Extended ping Tests

The ping command is a useful test available on Cisco internetworking devices as well as on many host systems. In TCP/IP, this diagnostic tool is also known as an Internet Control Message Protocol (ICMP) Echo Request.

Note: The ping command is particularly useful when high levels of input errors are being registered in the show interfaces serial display. See Figure 15-1.

Cisco internetworking devices provide a mechanism to automate the sending of many ping packets in sequence. Figure 15-5 illustrates the menu used to specify extended ping options. This example specifies 20 successive pings. However, when testing the components on your serial line, you should specify a much larger number, such as 1000 pings.

Figure 15-5: Extended ping Specification Menu

Performing Ping Tests

In general, perform serial line ping tests as follows:

1. Put the CSU or DSU into local loopback mode.

2. Configure the extended ping command to send different data patterns and packet sizes. Figure 15-6 and Figure 15-7 illustrate two useful ping tests, an all-zeros (1500-byte) ping and an all-ones (1500-byte) ping, respectively.

3. Examine the show interfaces serial command output (see Figure 15-1) and determine whether input errors have increased. If input errors have not increased, the local hardware (DSU, cable, router interface card) is probably in good condition.

   Assuming that this test sequence was prompted by the appearance of a large number of CRC and framing errors, a clocking problem is likely. Check the CSU or DSU for a timing problem. See the section «Troubleshooting Clocking Problems,» later in this chapter.

4. If you determine that the clocking configuration is correct and is operating properly, put the CSU or DSU into remote loopback mode.

5. Repeat the ping test and look for changes in the input error statistics.

6. If input errors increase, there is either a problem in the serial line or on the CSU/DSU. Contact the WAN service provider and swap the CSU or DSU. If problems persist, contact your technical support representative.

Figure 15-6: ALl-Zeros 1500-Byte ping Test

Figure 15-7 All-Ones 1500-Byte ping Test

Troubleshooting Clocking Problems

Clocking conflicts in serial connections can lead either to chronic loss of connection service or to degraded performance. This section discusses the important aspects of clocking problems: clocking problem causes, detecting clocking problems, isolating clocking problems, and clocking problem solutions.

Clocking Overview

The CSU/DSU derives the data clock from the data that passes through it. In order to recover the clock, the CSU/DSU hardware must receive at least one 1-bit value for every 8 bits of data that pass through it; this is known as ones density. Maintaining ones density allows the hardware to recover the data clock reliably.

Newer T1 implementations commonly use Extended Superframe Format (ESF) framing with binary eight-zero substitution (B8ZS) coding. B8ZS provides a scheme by which a special code is substituted whenever eight consecutive zeros are sent through the serial link. This code is then interpreted at the remote end of the connection. This technique guarantees ones density independent of the data stream.

Older T1 implementations use D4-also known as Superframe Format (SF) framing and Alternate Mark Inversion (AMI) coding. AMI does not utilize a coding scheme like B8ZS. This restricts the type of data that can be transmitted because ones density is not maintained independent of the data stream.

Another important element in serial communications is serial clock transmit external (SCTE) terminal timing. SCTE is the clock echoed back from the data terminal equipment (DTE) device (for example, a router) to the data communications equipment (DCE) device (for example, the CSU/DSU).

When the DCE device uses SCTE instead of its internal clock to sample data from the DTE, it is better able to sample the data without error even if there is a phase shift in the cable between the CSU/DSU and the router. Using SCTE is highly recommended for serial transmissions faster than 64 kbps. If your CSU/DSU does not support SCTE, see the section «Inverting the Transmit Clock,» later in this chapter.

Clocking Problem Causes

In general, clocking problems in serial WAN interconnections can be attributed to one of the following causes:

• Incorrect DSU configuration

• Incorrect CSU configuration

• Cables out of specification-that is, longer than 50 feet (15.24 meters) or unshielded

• Noisy or poor patch panel connections

• Several cables connected together in a row

Detecting Clocking Problems

To detect clocking conflicts on a serial interface, look for input errors as follows:

1. Use the show interfaces serial EXEC command on the routers at both ends of the link.

2. Examine the command output for CRC, framing errors, and aborts.

3. If either of these steps indicates errors exceeding an approximate range of 0.5 percent 2.0 percent of traffic on the interface, clocking problems are likely to exist somewhere in the WAN.

4. Isolate the source of the clocking conflicts as outlined in the following section, «Isolating Clocking Problems.»

5. Bypass or repair any faulty patch panels.

Isolating Clocking Problems

After you determine that clocking conflicts are the most likely cause of input errors, the following procedure will help you isolate the source of those errors:

1. Perform a series of ping tests and loopback tests (both local and remote), as described in the section «CSU and DSU Loopback Tests,» earlier in this chapter.

2. Determine the end of the connection that is the source of the problem, or if the problem is in the line. In local loopback mode, run different patterns and sizes in the ping tests (for example, use 1500-byte datagrams). Using a single pattern and packet size may not force errors to materialize, particularly when a serial cable to the router or CSU/DSU is the problem.

3. Use the show interfaces serial EXEC command and determine if input errors counts are increasing and where they are accumulating.

If input errors are accumulating on both ends of the connection, clocking of the CSU is the most likely problem.

If only one end is experiencing input errors, there is probably a DSU clocking or cabling problem.

Aborts on one end suggests that the other end is sending bad information or that there is a line problem.

Note: Always refer to the show interfaces serial command output (see Figure 15-1) and log any changes in error counts or note if the error count does not change.

Clocking Problem Solutions

Table 15-8 Serial Lines: Clocking Problems and Solutions: This table outlines suggested remedies for clocking problems, based on the source of the problem.

Possible Problem	Solution
Incorrect CSU configuration	Determine if the CSUs at both ends agree on the clock source (local or line). If the CSUs do not agree, configure them so that they do. Usually the line is the source. Check the LBO setting on the CSU to ensure that the impedance matches that of the physical line. For information on configuring your CSU, consult your CSU hardware documentation.
Incorrect DSU configuration	Determine if the DSUs at both ends have SCTE mode enabled. If SCTE is not enabled on both ends of the connection, enable it. Make sure that ones density is maintained. This requires that the DSU use the same framing and coding schemes (for example, ESF and B8ZS) used by the leased-line or other carrier service. Check with your leased-line provider for information on its framing and coding schemes. If your carrier service uses AMI coding, either invert the transmit clock on both sides of the link or run the DSU in bit-stuff mode. For information on configuring your DSU, consult your DSU hardware documentation.
Cable to router is out of specification	If the cable is longer than 50 feet (15.24 meters), use a shorter cable. If the cable is unshielded, replace it with shielded cable.

Inverting the Transmit Clock

If you are attempting serial connections at speeds greater than 64 kbps with a CSU/DSU that does not support SCTE, you may have to invert the transmit clock on the router. Inverting the transmit clock compensates for phase shifts between the data and clock signals.

The specific command used to invert the transmit clock varies between platforms. On a Cisco 7000 series router, enter the invert-transmit-clock interface configuration command. For Cisco 4000 series routers, use the dte-invert-txc interface configuration command.

To ensure that you are using the correct command syntax for your router, refer to the user guide for your router or access server and to the Cisco IOS configuration guides and command references.

Note: On older platforms, inverting the transmit clock may require that you move a physical jumper.

Adjusting Buffers

Excessively high bandwidth utilization (over 70percent) results in reduced overall performance and can cause intermittent failures. For example, DECnet file transmissions may be failing due to packets being dropped somewhere in the network.

If the situation is bad enough, you must increase the bandwidth of the link. However, increasing the bandwidth may not be necessary or immediately practical. One way to resolve marginal serial line overutilization problems is to control how the router uses data buffers.

Caution: In general, do not adjust system buffers unless you are working closely with a Cisco technical support representative. You can severely affect the performance of your hardware and your network if you incorrectly adjust the system buffers on your router.

Use one of the following three options to control how buffers are used:

• Adjust parameters associated with system buffers

• Specify the number of packets held in input or output queues (hold queues)

• Prioritize how traffic is queued for transmission (priority output queuing)

The configuration commands associated with these options are described in the Cisco IOS configuration guides and command references.

The following section focuses on identifying situations in which these options are likely to apply and defining how you can use these options to help resolve connectivity and performance problems in serial/WAN interconnections.

Tuning System Buffers

There are two general buffer types on Cisco routers: hardware buffers and system buffers. Only the system buffers are directly configurable by system administrators. The hardware buffers are specifically used as the receive and transmit buffers associated with each interface and (in the absence of any special configuration) are dynamically managed by the system software itself.

The system buffers are associated with the main system memory and are allocated to different-size memory blocks. A useful command for determining the status of your system buffers is the show buffers EXEC command. Figure 15-8 shows the output from the show buffers command.

Figure 15-8 show buffers Command Output

In the show buffers output:

• total— Identifies the total number of buffers in the pool, including used and unused buffers.

• permanent— Identifies the permanent number of allocated buffers in the pool. These buffers are always in the pool and cannot be trimmed away.

• in free list— Identifies the number of buffers currently in the pool that are available for use.

• min— Identifies the minimum number of buffers that the Route Processor (RP) should attempt to keep in the free list:

  • The min parameter is used to anticipate demand for buffers from the pool at any given time.

  • If the number of buffers in the free list falls below the min value, the RP attempts to create more buffers for that pool.

• max allowed— Identifies the maximum number of buffers allowed in the free list:

  • The max allowed parameter prevents a pool from monopolizing buffers that it doesn’t need anymore and frees this memory back to the system for further use.

  • If the number of buffers in the free list is greater than the max allowed value, the RP should attempt to trim buffers from the pool.

• hits— Identifies the number of buffers that have been requested from the pool. The hits counter provides a mechanism for determining which pool must meet the highest demand for buffers.

• misses— Identifies the number of times a buffer has been requested and the RP detected that additional buffers were required. (In other words, the number of buffers in the free list has dropped below min.) The misses counter represents the number of times the RP has been forced to create additional buffers.

• trims— Identifies the number of buffers that the RP has trimmed from the pool when the number of buffers in the free list exceeded the number of max allowed buffers.

• created— Identifies the number of buffers that have been created in the pool. The RP creates buffers when demand for buffers has increased until the number of buffers in the free list is less than min buffers and/or a miss occurs because of zero buffers in the free list.

• failures— Identifies the number of failures to grant a buffer to a requester even after attempting to create an additional buffer. The number of failures represents the number of packets that have been dropped due to buffer shortage.

• no memory— Identifies the number of failures caused by insufficient memory to create additional buffers.

The show buffers command output in Figure 15-8 indicates high numbers in the trims and created fields for large buffers. If you are receiving high numbers in these fields, you can increase your serial link performance by increasing the max free value configured for your system buffers. trims identifies the number of buffers that the RP has trimmed from the pool when the number of buffers in free list exceeded the number of max allowed buffers.

Use the buffers max free number global configuration command to increase the number of free system buffers. The value you configure should be approximately 150 percent of the figure indicated in the total field of the show buffers command output. Repeat this process until the show buffers output no longer indicates trims and created buffers.

If the show buffers command output shows a large number of failures in the (no memory) field (see the last line of output in Figure 15-8), you must reduce the usage of the system buffers or increase the amount of shared or main memory (physical RAM) on the router. Call your technical support representative for assistance.

Implementing Hold Queue Limits

Hold queues are buffers used by each router interface to store outgoing or incoming packets. Use the hold-queue interface configuration command to increase the number of data packets queued before the router will drop packets. Increase these queues by small increments (for instance, 25 percent) until you no longer see drops in the show interfaces output. The default output hold queue limit is 100 packets.

Note: The hold-queue command is used for process-switched packets and periodic updates generated by the router.

Use the hold-queue command to prevent packets from being dropped and to improve serial-link performance under the following conditions:

• You have an application that cannot tolerate drops and the protocol is able to tolerate longer delays. DECnet is an example of a protocol that meets both criteria. Local-area transport (LAT) does not because it does not tolerate delays.

• The interface is very slow. Bandwidth is low or anticipated utilization is likely to sporadically exceed available bandwidth.

Note: When you increase the number specified for an output hold queue, you may need to increase the number of system buffers. The value used depends on the size of the packets associated with the traffic anticipated for the network.

Using Priority Queuing to Reduce Bottlenecks

Priority queuing is a list-based control mechanism that allows traffic to be prioritized on an interface-by-interface basis. Priority queuing involves two steps:

1. Create a priority list by protocol type and level of priority.

2. Assign the priority list to a specific interface.

Both of these steps use versions of the priority-list global configuration command. In addition, further traffic control can be applied by referencing access-list global configuration commands from priority-list specifications. For examples of defining priority lists and for details about command syntax associated with priority queuing, refer to the Cisco IOS configuration guides and command references.

Note: Priority queuing automatically creates four hold queues of varying size. This overrides any hold queue specification included in your configuration.

Use priority queuing to prevent packets from being dropped and to improve serial link performance under the following conditions:

• When the interface is slow, there is a variety of traffic types being transmitted, and you want to improve terminal traffic performance.

• If you have a serial link that is intermittently experiencing very heavy loads (such as file transfers occurring at specific times) priority queuing will help select which types of traffic should be discarded at high traffic periods.

In general, start with the default number of queues when implementing priority queues. After enabling priority queuing, monitor output drops with the show interfaces serial EXEC command. If you notice that output drops are occurring in the traffic queue you have specified to be high priority, increase the number of packets that can be queued (using the queue-limit keyword option of the priority-list global configuration command). The default queue-limit arguments are 20 packets for the high-priority queue, 40 for medium, 60 for normal, and 80 for low.

Note: When bridging Digital Equipment Corporation (DEC) LAT traffic, the router must drop very few packets, or LAT sessions can terminate unexpectedly. A high-priority queue depth of about 100 (specified with the queue-limit keyword) is a typical working value when your router is dropping output packets and the serial lines are subjected to about 50 percent bandwidth utilization. If the router is dropping packets and is at 100 percent utilization, you need another line.

Another tool to relieve congestion when bridging DEC LAT is LAT compression. You can implement LAT compression with the interface configuration command bridge-group group lat-compression.

Special Serial Line Tests

In addition to the basic diagnostic capabilities available on routers, a variety of supplemental tools and techniques can be used to determine the conditions of cables, switching equipment, modems, hosts, and remote internetworking hardware. For more information, consult the documentation for your CSU, DSU, serial analyzer, or other equipment.

CSU and DSU Loopback Tests

If the output of the show interfaces serial EXEC command indicates that the serial line is up but the line protocol is down, use the CSU/DSU loopback tests to determine the source of the problem. Perform the local loop test first, and then the remote test. Figure 15-9 illustrates the basic topology of the CSU/DSU local and remote loopback tests.

Figure 15-9: CSU/DSU Local and Remote Loopback Tests

Note: These tests are generic in nature and assume attachment of the internetworking system to a CSU or DSU. However, the tests are essentially the same for attachment to a multiplexer with built-in CSU/DSU functionality. Because there is no concept of a loopback in X.25 or Frame Relay packet-switched network (PSN) environments, loopback tests do not apply to X.25 and Frame Relay networks.

CSU and DSU Local Loopback Tests for HDLC or PPP Links

Listed below is a general procedure for performing loopback tests in conjunction with built-in system diagnostic capabilities:

1. Place the CSU/DSU in local loop mode (refer to your vendor documentation). In local loop mode, the use of the line clock (from the T1 service) is terminated, and the DSU is forced to use the local clock.

2. Use the show interfaces serial EXEC command to determine if the line status changes from «line protocol is down» to «line protocol is up (looped),» or if it remains down.

3. If the line protocol comes up when the CSU or DSU is in local loopback mode, this suggests that the problem is occurring on the remote end of the serial connection. If the status line does not change state, there is a possible problem in the router, connecting cable, or CSU/DSU.

4. If the problem appears to be local, use the debug serial interface privileged EXEC command.

5. Take the CSU/DSU out of local loop mode. When the line protocol is down, the debug serial interface command output will indicate that keepalive counters are not incrementing.

6. Place the CSU/DSU in local loop mode again. This should cause the keepalive packets to begin to increment. Specifically, the values for mineseen and yourseen keepalives will increment every 10 seconds. This information will appear in the debug serial interface output.

   If the keepalives do not increment, there may be a timing problem on the interface card or on the network. For information on correcting timing problems, see the section «Troubleshooting Clocking Problems,» earlier in this chapter.

   If the keepalives do not increment, there may be a timing problem on the interface card or on the network. For information on correcting timing problems, see the section «Troubleshooting Clocking Problems,» earlier in this chapter.

7. Check the local router, CSU/DSU hardware, and any attached cables. Make certain that the cables are within the recommended lengths-no more than 50 feet (15.24 meters) or 25 feet (7.62 meters) for a T1 link. Make certain the cables are attached to the proper ports. Swap faulty equipment as necessary.

Figure 15-10 shows the output from the debug serial interface command for an HDLC serial connection, with missed keepalives causing the line to go down and the interface to reset.

Figure 15-10: debug serial interface Command Output

CSU and DSU Remote Loopback Tests for HDLC or PPP Links

If you determine that the local hardware is functioning properly but you still encounter problems when attempting to establish connections over the serial link, try using the remote loopback test to isolate the problem cause.

Note: This remote loopback test assumes that HDLC encapsulation is being used and that the preceding local loop test was performed immediately before this test.

The following steps are required to perform loopback testing:The following steps are required to perform loopback testing:

1. Put the remote CSU or DSU into remote loopback mode (refer to the vendor documentation).

2. Using the show interfaces serial EXEC command, determine if the line protocol remains up with the status line indicating «Serial x is up, line protocol is up (looped),» or if it goes down with the status line indicating «line protocol is down.»

3. If the line protocol remains up (looped), the problem is probably at the remote end of the serial connection (between the remote CSU/DSU and the remote router). Perform both local and remote tests at the remote end to isolate the problem source.

4. If the line status changes to «line protocol is down» when remote loopback mode is activated, make sure that ones density is being properly maintained. The CSU/DSU must be configured to use the same framing and coding schemes used by the leased-line or other carrier service (for example, ESF and B8ZS).

5. If problems persist, contact your WAN network manager or the WAN service organization.

Detailed Information on the show interfaces serial Command

The following sub-sections cover the show interfaces serial command’s parameters, syntax description, sample output display, and field descriptions.

show interfaces serial Parameters

To display information about a serial interface, use the show interfaces serial privileged EXEC command:

show interfaces serial [number] [accounting]
show interfaces serial [number [:channel-group] [accounting] (Cisco 4000 series)
show interfaces serial [slot | port [:channel-group]] [accounting] (Cisco 7500 series)
show interfaces serial [type slot | port-adapter | port] [serial] 
(ports on VIP cards in the Cisco 7500 series)
show interfaces serial [type slot | port-adapter | port] [:t1-channel] [accounting | crb]
(CT3IP in Cisco 7500 series)

Syntax Description

• number-Optional. Port number.

• accounting-Optional. Displays the number of packets of each protocol type that have been sent through the interface.

• :channel-group -Optional. On the Cisco 4000 series with an NPM or a Cisco 7500 series with a MIP, specifies the T1 channel-group number in the range of 0 to 23, defined with the channel-group controller configuration command.

• slot -Refers to the appropriate hardware manual for slot information.

• port -Refers to the appropriate hardware manual for port information.

• port-adapter -Refers to the appropriate hardware manual for information about port adapter compatibility.

• :t1-channel -Optional. For the CT3IP, the T1 channel is a number between 1 and 28.

• T1 channels on the CT3IP are numbered 1 to 28 rather than the more traditional zero-based scheme (0 to 27) used with other Cisco products. This is to ensure consistency with Telco numbering schemes for T1 channels within channelized T3 equipment.

• crb-Optional. Shows interface routing and bridging information.

Command Mode

Privileged EXEC

Usage Guidelines

This command first appeared in Cisco IOS Release 10.0 for the Cisco 4000 series. It first appeared in Cisco IOS Release 11.0 for the Cisco 7000 series, and it was modified in Cisco IOS Release 11.3 to include the CT3IP.

Sample Displays

The following is sample output from the show interfaces command for a synchronous serial interface:

Router# show interfaces serial
Serial 0 is up, line protocol is up
   Hardware is MCI Serial
   Internet address is 150.136.190.203, subnet mask is 255.255.255.0
   MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, rely 255/255, load 1/255
   Encapsulation HDLC, loopback not set, keepalive set (10 sec)
   Last input 0:00:07, output 0:00:00, output hang never
   Output queue 0/40, 0 drops; input queue 0/75, 0 drops
   Five minute input rate 0 bits/sec, 0 packets/sec
   Five minute output rate 0 bits/sec, 0 packets/sec
       16263 packets input, 1347238 bytes, 0 no buffer
       Received 13983 broadcasts, 0 runts, 0 giants
       2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 2 abort
1 carrier transitions 
     22146 packets output, 2383680 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets, 0 restarts

Field Description

Table 15-9: show interfaces serial Field Descriptions — this table describes significant fields shown in the output.

Field	Description
Serial…is {up | down}…is administratively down	Indicates whether the interface hardware is currently active (carrier detect is present) or whether it has been taken down by an administrator.
line protocol is {up | down}	Indicates whether the software processes that handle the line protocol consider the line usable (that is, keepalives are successful) or whether it has been taken down by an administrator.
line protocol is {up | down}	Indicates whether the software processes that handle the line protocol consider the line usable (that is, keepalives are successful) or whether it has been taken down by an administrator.
Hardware is	Specifies the hardware type.
Internet address is	Specifies the internet address and subnet mask.
MTU	Maximum transmission unit of the interface.
BW	Indicates the value of the bandwidth parameter that has been configured for the interface (in kilobits per second). The bandwidth parameter is used to compute IGRP metrics only. If the interface is attached to a serial line with a line speed that does not match the default (1536 or 1544 for T1 and 56 for a standard synchronous serial line), use the bandwidth command to specify the correct line speed for this serial line.
DLY	Delay of the interface in microseconds.
rely	Reliability of the interface as a fraction of 255 (255/255 is 100 percent reliability), calculated as an exponential average over five minutes.
load	Reliability of the interface as a fraction of 255 (255/255 is 100 percent reliability), calculated as an exponential average over five minutes.
Encapsulation	Encapsulation method assigned to the interface.
loopback	Indicates whether loopback is set.
keepalive	Indicates whether keepalives are set.
Last input	Number of hours, minutes, and seconds since the last packet was successfully received by an interface. Useful for knowing when a dead interface failed.
Last output	Number of hours, minutes, and seconds since the last packet was successfully transmitted by an interface.Number of hours, minutes, and seconds since the last packet was successfully transmitted by an interface.
output hang	Number of hours, minutes, and seconds (or never) since the interface was last reset because of a transmission that took too long. When the number of hours in any of the last fields exceeds 24, the number of days and hours is printed. If that field overflows, asterisks are printed.
Output queue, drops input queue, drops	Number of packets in output and input queues. Each number is followed by a slash, the maximum size of the queue, and the number of packets because the queue is full.
5 minute input rate 5 minute output rate	Average number of bits and packets transmitted per second in the past five minutes. The five-minute input and output rates should be used only as an approximation of traffic per second during a given five-minute period. These rates are exponentially weighted averages with a time constant of five minutes. A period of four time constants must pass before the average will be within 2 percent of the instantaneous rate of a uniform stream of traffic over that period.
packets input	Total number of error-free packets received by the system.
bytes	Total number of bytes, including data and MAC encapsulation, in the error-free packets received by the system.
no buffer	Number of received packets discarded because there was no buffer space in the main system. Compare with ignored count. Broadcast storms on Ethernet networks and bursts of noise on serial lines are often responsible for no input buffer events.
Received… broadcasts	Total number of broadcast or multicast packets received by the interface.
runts	Number of packets that are discarded because they are smaller than the medium’s minimum packet size.
giants	Number of packets that are discarded because they exceed the medium’s maximum packet size.
input errors	Total number of no buffer, runts, giants, CRCs, frame, overrun, ignored, and abort counts. Other input-related errors can also increment the count, so this sum may not balance with the other counts.
CRC	Cyclic redundancy check generated by the originating station or far-end device does not match the checksum calculated from the data received. On a serial link, CRCs usually indicate noise, gain hits, or other transmission problems on the data link.
frame	Number of packets received incorrectly having a CRC error and a noninteger number of octets. On a serial line, this is usually the result of noise or other transmission problems.
overrun	Number of times the serial receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver’s ability to handle the data.
ignored	Number of received packets ignored by the interface because the interface hardware ran low on internal buffers. Broadcast storms and bursts of noise can cause the ignored count to be increased.
abort	Illegal sequence of one bits on a serial interface. This usually indicates a clocking problem between the serial interface and the data link equipment.
carrier transitions	Number of times the carrier detect signal of a serial interface has changed state. For example, if data carrier detect (DCD) goes down and comes up, the carrier transition counter will increment two times. Indicates modem or line problems if the carrier detect line is changing state often.
packets output	Total number of messages transmitted by the system.
bytes output	Total number of bytes, including data and MAC encapsulation, transmitted by the system.
underruns	Number of times that the transmitter has been running faster than the router can handle. This may never be reported on some interfaces.
output errors	Sum of all errors that prevented the final transmission of datagrams out of the interface being examined. Note that this may not balance with the sum of the enumerated output errors because some datagrams can have more than one error, and others can have errors that do not fall into any of the specifically tabulated categories.
collisions	Number of messages retransmitted due to an Ethernet collision. This usually is the result of an overextended LAN (that is, Ethernet or transceiver cable too long, more than two repeaters between stations, or too many cascaded multiport transceivers). Some collisions are normal. However, if your collision rate climbs to around 4 percent or 5 percent, you should consider verifying that there is no faulty equipment on the segment and/or moving some existing stations to a new segment. A packet that collides is counted only once in output packets.
interface resets	Number of times an interface has been completely reset. This can happen if packets queued for transmission were not sent within several seconds’ time. On a serial line, this can be caused by a malfunctioning modem that is not supplying the transmit clock signal, or by a cable problem. If the system notices that the carrier detect line of a serial interface is up but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.
restarts	Number of times the controller was restarted because of errors.
alarm indications, remote alarms, rx LOF, rx LOS	Number of CSU/DSU alarms, and number of occurrences of receive loss of frame and receive loss of signal.
BER inactive, NELR inactive, FELR inactive	Status of G.703-E1 counters for bit error rate (BER) alarm, near-end loop remote (NELR), and far-end loop remote (FELR). Note that you cannot set the NELR or FELR.

Troubleshooting T1

This section describes the techniques and procedures for troubleshooting T1 circuits for dial-in customers.

Troubleshooting Using the show controller t1 Command

This command displays the controller status that is specific to the controller hardware. The information displayed is generally useful for diagnostic tasks performed by technical support personnel only.

The NMP (Network Management Processor) or MIP (MultiChannel Interface Processor) can query the port adapters to determine their current status. Issue a show controller t1 command to display statistics about the T1 link.

If you specify a slot and port number, statistics for each 15-minute period will be displayed. The show controller t1 EXEC command provides information to logically troubleshoot physical layer and data link layer problems. This section describes how to logically troubleshoot using the show controller t1 command.

Most T1 errors are caused by misconfigured lines. Ensure that linecoding, framing and clock source are configured according to what the service provider recommends.

show controller t1 Conditions

The T1 controller can be in one of the following three states.

• Administratively down

• Down

• Up

Is the T1 Controller Administratively Down?

The controller is administratively down when it has been manually shut down. You should restart the controller to correct this error.

1. Enter enable mode.

   maui-nas-03>en
   Password: 
   maui-nas-03#

2. Enter global configuration mode.

   maui-nas-03#configure terminal
   Enter configuration commands, one per line. End with CNTL/Z.
   maui-nas-03(config)#

3. Enter controller configuration mode.

   maui-nas-03(config)#controller t1 0
   maui-nas-03(config-controlle)#

4. Restart controller.

   maui-nas-03(config-controlle)#shutdown
   maui-nas-03(config-controlle)#no shutdown
   

Is the Line Up?

If the T1 controller and line are not up, check to see if one of the following messages appears in the show controller t1 EXEC output:

• Receiver has loss of frame

• Receiver has loss of signal

If T1 Receiver Has Loss of Frame:

Follow these steps if T1 Receiver Has Loss of Frame:

1. Check to see if the framing format configured on the port matches the framing format of the line. You can check the framing format of the controller from the running configuration or the show controller t1 command output.

   To change the framing format use the framing {SF | ESF} command in the controller configuration mode as shown below:

   maui-nas-03#configure terminal
   

   Enter configuration commands, one per line. End with CNTL/Z.

   maui-nas-03(config)#controller t1 0
   maui-nas-03(config-controlle)#framing esf
   

2. Try the other framing format to see if the alarm clears.

3. Change the line buildout setting using the cablelength {long | short} command.

Line build out (LBO) compensates for the loss in decibels based on the distance from the device to the first repeater in the circuit. A longer distance from the device to the repeater requires that the signal strength on the circuit be boosted to compensate for loss over that distance.

Consult your service provider and the Cisco IOSÒ Command Reference for details on buildout settings.

If this does not fix the problem, proceed to the «If T1 Receiver Has Loss of Signal» section below.

If T1 Receiver Has Loss of Signal:

Follow these steps if T1 Receiver Has Loss of Signal:

1. Make sure that the cable between the interface port and the T1 service provider’s equipment (or T1 terminal equipment) is connected correctly. Check to see if the cable is hooked up to the correct ports. Correct the cable connections if necessary.

2. Check cable integrity. Look for breaks or other physical abnormalities in the cable. Ensure that the pinouts are set correctly. If necessary, replace the cable.

3. Check the cable connectors. A reversal of the transmit and receive pairs or an open receive pair can cause errors. Set the receive pair to lines 1 & 2. Set the transmit pair to lines 4 & 5.

   The pins on a RJ-45 jack are numbered from 1 through 8. Pin 1 is the leftmost pin when looking at the jack with the metal pins facing you. Refer to the figure below.

   Figure 15-10: RJ-45 Cable

   

4. Try using a rollover cable.

Run the show controller t1 EXEC command after each step to check if the controller exhibits any errors.

Check to see if the line is in loopback mode from the show controller t1 output. A line should be in loopback mode only for testing purposes.

To turn off loopback, use the no loopback command in the controller configuration mode as shown below:

maui-nas-03(config-controlle)#no loopback

If the Controller Displays Any Alarms:

Check the show controller command output to see if there are alarms displayed by the controller.

We will now discuss various alarms and the procedure necessary to correct them.

Receive (RX) Alarm Indication Signal (AIS) (Blue):

A received Alarm Indication Signal (AIS) means there is an alarm occurring on the line upstream of the equipment connected to the port.

1. Check to see if the framing format configured on the port matches the framing format of the line. If not, change the framing format on the controller to match that of the line.

2. Contact your service provider to check for mis-configuration within the Telco.

Receive (Rx) Remote Alarm Indication (RAI) (Yellow):

A received RAI means that the far-end equipment has a problem with the signal it is receiving from its upstream equipment.

1. Insert an external loopback cable into the port. To create a loopback plug refer to the section «Creating a Loopback Plug,» later in the chapter.

2. Check to see if there are any alarms. If you do not see any alarms, then the local hardware is probably in good condition. In that case:

   1. Check the cabling. See the section «If T1 Receiver Has Loss of Signal» for more information.

   2. Check the settings at the remote end and verify that they match your port settings.

   3. If the problem persists, contact your service provider.

3. Remove the loopback plug and reconnect your T1 line.

4. Check the cabling. See the section «If T1 Receiver Has Loss of Signal» for more information.

5. Power-cycle the router.

6. Connect the T1 line to a different port. Configure the port with the same settings as that of the line. If the problem does not persist, then the fault lies with the one port:

   1. Reconnect the T1 line to the original port.

   2. Proceed to the «Troubleshooting T1 Error Events» section.

      If the problem persists, then:

7. Perform a hardware loop test as described in the section «Performing Hardware Loopback Plug Test.»

8. Replace the T1 controller card.

9. Proceed to the «Troubleshooting T1 Error Events» section.

Transmitter Sending Remote Alarm (Red):

A Red alarm is declared when the CSU cannot synchronize with the framing pattern on the T1 line.

1. Check to see if the framing format configured on the port matches the framing format of the line. If not change the framing format on the controller to match that of the line.

2. Check the settings at the remote end and verify that they match your port settings.

3. Contact your service provider.

Transmit(Tx) Remote Alarm Indication (RAI) (Yellow):

A transmitted RAI at the interface indicates that the interface has a problem with the signal it is receiving from the far-end equipment.

1. Check the settings at the remote end and verify that they match your port settings.

2. A transmit RAI should be accompanied by some other alarm that indicates the nature of the problem the T1 port/card is having with the signal from the far-end equipment.

Troubleshoot that condition to resolve the transmit RAI.

Transmit(Tx) AIS (Blue):

Follow the steps below to correct the Transmit (Tx) AIS (Blue).

1. Check to see if the framing format configured on the port matches the framing format of the line. If not, correct the mismatch.

2. Power-cycle the router.

3. Connect the T1 line to a different port. Configure the port with the same settings as that of the line.

4. Perform a hardware loop test as described in the section «Performing Hardware Loopback Plug Test.»

5. Replace the T1 controller card.

6. Proceed to the «Troubleshooting T1 Error Events» section.

Troubleshooting T1 Error Events

The show controller t1 EXEC command provides error messages that can be used to troubleshoot problems. We will now discuss several error messages and how to correct the errors.

To see if the error counters are increasing, execute the show controller t1 command repeatedly. Note the values of the counters for the current interval.

Consult your service provider for framing and linecoding settings. A good rule of thumb is to use B8ZS linecoding with ESF framing and AMI linecoding with SF framing.

Slip Secs Counter is increasing:

The presence of slips on a T1 line indicates a clocking problem. The T1 provider (Telco) will provide the clocking to which the Customer Premises Equipment (CPE) should be synchronized.

1. Verify that the clock source is derived from the network. This can be ascertained by looking for Clock Source is Line Primary.

   Note: If there are multiple T1s into an access server, only one can be the primary, while the other T1s derive the clock from the primary. In that case verify that the T1 line designated as the primary clock source is configured correctly.

2. Set the T1 clock source correctly from the controller configuration mode.

   maui-nas-03(config-controlle)#clock source line primary
   

Framing Loss Seconds Counter is Increasing:

Follow these steps when the Framing Loss Seconds Counter is Increasing.

1. Check to see if the framing format configured on the port matches the framing format of the line. You can check this by looking for the Framing is {ESF|SF} in the show controller t1 output.

2. To change the framing format use the framing {SF | ESF} command in the controller configuration mode as shown below:

   maui-nas-03(config-controlle)#framing esf
   

3. Change the line buildout using the cablelength {long | short} command.

Consult your service provider and the Cisco IOSÒ Command Reference for details on buildout settings.

Line Code Violations are increasing:

Follow these steps when Line Code Violations are increasing.

1. Check to see if the linecoding configured on the port matches the framing format of the line. You can check this by looking for the Line Code is {B8ZS|AMI} in the show controller t1 output.

2. To change the linecoding, use the linecode {ami | b8zs} command in the controller configuration mode as shown below:

   maui-nas-03(config-controlle)#linecode b8zs
   

3. Change the line buildout using the cablelength {long | short} command.

Consult your service provider and the Cisco IOS® Command Reference for details on buildout settings.

Verifying that ISDN Switch Type and PRI-Group are Configured Correctly

Use the show running-config command to see if ISDN switch type and the PRI-group timeslots are configured correctly. Contact your service provider for correct values.

To change the ISDN switch type and PRI-group:

maui-nas-03#configure terminal
maui-nas-03(config)#isdn switch-type primary-5ess
maui-nas-03(config)#controller t1 0
maui-nas-03(config-controlle)#pri-group timeslots 1-24

Verifying the Signaling Channel

If the error counters do not increase but the problem persists, verify that the signaling channel is up and configured correctly.

1. Run the show interface serial x:23 command, where x should be replaced by the interface number.

2. Check to see if the interface is up. If the interface is not up, use the no shutdown command to bring the interface up.

   maui-nas-03#config terminal
   Enter configuration commands, one per line.  End with CNTL/Z.
   maui-nas-03(config)#interface serial 0:23
   maui-nas-03(config-if)#no shutdown
   

3. Ensure that encapsulation is PPP. If the interface is not using PPP then use the encapsulation ppp command in the interface configuration mode to correct it.

   maui-nas-03(config-if)#encapsulation ppp
   

4. Check to see if loopback is set. Loopback should be set only for testing purposes. Use the no loopback command to remove loopbacks.

   maui-nas-03(config-if)#no loopback
   

5. Power-cycle the router.

6. If the problem persists, contact your service provider or Cisco TAC

Troubleshooting a PRI

Whenever troubleshooting a PRI, you need to check to see if the T1 is running cleanly on both ends. If Layer 1 problems have been resolved, as described above, consider Layer 2 and Layer 3 problems.

Troubleshooting Using the show isdn status Command

The show isdn status command is used to display a snapshot of all ISDN interfaces. It displays the status of Layers 1, 2 and 3.

1. Verify that Layer 1 is active.

   The Layer 1 status should always say ACTIVE unless the T1 is down. If show isdn status indicates that Layer 1 is DEACTIVATED, then there is a problem with the physical connectivity on the T1 line. See the section «Is the T1 Controller T1 Down?»

   Also verify that the T1 is not administratively down. Use the no shutdown command to bring the T1 controller up.

2. Check to see if the Layer 2 State is MULTIPLE_FRAME_ESTABLISHED

The desired Layer 2 state is Multiple_Frame_Established, which indicates that we are exchanging layer 2 frames and have finished Layer 2 initialization.

If Layer 2 is not Multiple_Frame_Established , use the show controller t1 EXEC command to diagnose the problem. Refer to the Troubleshooting using the show controller t1 Command section in this chapter.

Since show isdn status is a snapshot of the current status, it is possible that layer 2 is bouncing up and down despite indicating Mulitple_Frame_Established. Use debug isdn q921 to verify that layer 2 is stable.

The debug isdn q921 command displays data link layer (layer 2) access procedures that are taking place at the router on the D-channel.

Ensure that you are configured to view debug messages by using the logging console or terminal monitor command as necessary.

Note: In a production environment, verify that console logging is disabled. Enter the show logging command. If logging is enabled, the access server may intermittently freeze up as soon as the console port gets overloaded with log messages. Enter the no logging console command.

Note: If debug isdn q921 is turned on and you do not receive any debug outputs, place a call or reset the controller to get debug outputs.

1. Verify that Layer 2 is stable.

   You should observe the debug outputs for messages indicating that the service is not bouncing up and down. If you see the following types of debug outputs, the line is not stable.

   Mar 20 10:06:07.882: %ISDN-6-LAYER2DOWN: Layer 2 for Interface Se0:23, TEI 0 
   changed to down
   Mar 20 10:06:09.882: %LINK-3-UPDOWN: Interface Serial0:23, changed state to down
   Mar 20 10:06:21.274: %DSX1-6-CLOCK_CHANGE: Controller 0  clock is now selected 
   as clock source
   Mar 20 10:06:21.702: %ISDN-6-LAYER2UP: Layer 2 for Interface Se0:23, TEI 0 changed 
   to up
   Mar 20 10:06:22.494: %CONTROLLER-5-UPDOWN: Controller T1 0, changed state to up
   Mar 20 10:06:24.494: %LINK-3-UPDOWN: Interface Serial0:23, changed state to up
   
   

   If Layer 2 does not appear to be stable, see «Troubleshooting T1 Error Events,» earlier in this chapter.

2. Verify that you are seeing only SAPI messages in both transmit (TX) and Receive (RX) sides.

   Mar 20 10:06:52.505: ISDN Se0:23: TX ->  RRf sapi = 0  tei = 0  nr = 0
   Mar 20 10:06:52.505: ISDN Se0:23: RX <-  RRf sapi = 0  tei = 0  nr = 0
   Mar 20 10:07:22.505: ISDN Se0:23: TX ->  RRp sapi = 0  tei = 0 nr = 0 
   Mar 20 10:07:22.509: ISDN Se0:23: RX <-  RRp sapi = 0  tei = 0 nr = 0 
   Mar 20 10:07:22.509: ISDN Se0:23: TX ->  RRf sapi = 0  tei = 0  nr = 0
   Mar 20 10:07:22.509: ISDN Se0:23: RX <-  RRf sapi = 0  tei = 0  nr = 0

3. Verify that you are not seeing SABME messages, which indicates that Layer 2 is trying to reinitialize. This is usually seen when we are transmitting poll requests (RRp) and not getting a response from the switch (RRf) or vice-versa. Below are example of SABME messages.

   Mar 20 10:06:21.702: ISDN Se0:23: RX <-  SABMEp sapi = 0  tei = 0
   Mar 20 10:06:22.494: ISDN Se0:23: TX ->  SABMEp sapi = 0  tei = 0

   If you are seeing SABME messages, use the show running-config command to see if ISDN switch type and the PRI-group timeslots are configured correctly. Contact your service provider for correct values.

   To change the ISDN switch type and PRI-group:

   maui-nas-03#configure terminal
   maui-nas-03(config)#isdn switch-type primary-5ess
   maui-nas-03(config)#controller t1 0
   maui-nas-03(config-controlle)#pri-group timeslots 1-24
   

4. Verify that the D-channel is up using the show interfaces serial x:23 command.

   If the D-channel is not up, then use no shutdown command to bring it up:

   maui-nas-03(config)#interface serial 0:23
   maui-nas-03(config-if)#no shutdown
   

5. Check to see if encapsulation is PPP. If not, use the encapsulation ppp command to set encapsulation.

   maui-nas-03(config-if)#encapsulation ppp
   

6. Check to see if the interface is in loopback mode. For normal operation, the interface should not be in loopback mode.

   maui-nas-03(config-if)#no loopback
   

7. Power-cycle the router.

8. If the problem persists, contact your service provider or the Cisco TAC.

Performing Hardware Loopback Plug Test

The Hardware loopback plug test can be used to test whether the router has any faults. If a router passes a hardware loopback plug test, then the problem exists elsewhere on the line.

Create a Loopback Plug:

Follow these steps to create a loopback plug.

1. Use wire cutters to cut a working RJ-45 or RJ-48 cable so that there are five inches of cable and the connector is attached to it.

2. Strip the wires.

3. Twist together the wires from pins 1 and 4.

4. Twist together the wires from pins 2 and 5.

The pins on a RJ-45/48 jack are numbered from 1 through 8. Pin 1 is the left-most pin when looking at the jack with the metal pins facing you.

Performing the Loopback Plug Test

Follow these steps to perform the loopback plug test.

1. Insert the plug into the T1 port in question.

2. Save your router configuration using the write memory command.

   maui-nas-03#write memory
   Building configuration...
   [OK]

3. Set the encapsulation to HDLC

   maui-nas-03#config terminal
   Enter configuration commands, one per line.  End with CNTL/Z.
   maui-nas-03(config)#interface serial 0
   maui-nas-03(config-if)#enc
   maui-nas-03(config-if)#encapsulation HDLC 
   maui-nas-03(config-if)#^Z
   

4. Use the show running-config command to see if the interface has an IP address.

   If the interface does not have an IP address, obtain a unique address and assign it to the interface with a subnet mask of 255.255.255.0.

   maui-nas-03(config)#ip address 172.22.53.1 255.255.255.0
   

5. Clear the interface counters using the clear counters command.

   maui-nas-03#clear counters
   Clear "show interfaces" counters on all interfaces [confirm]
   maui-nas-03#

6. Perform the extended ping test as described in the «Using Extended ping Tests,» section earlier in this chapter.

Troubleshooting E1

This section describes the techniques and procedures for troubleshooting E1 circuits for dial-in customers.

Troubleshooting Using the show controller e1 Command

This command displays the controller status that is specific to the controller hardware. The information displayed is generally useful for diagnostic tasks performed by technical support personnel only.

The NMP or MIP can query the port adapters to determine their current status. Issue a show controller e1 command to display statistics about the E1 link. If you specify a slot and port number, statistics for each 15 minute period will be displayed.

The show controller e1 EXEC command provides information to logically troubleshoot physical layer and data link layer problems. This section describes how to logically troubleshoot using the show controller e1 command.

Most E1 errors are caused by misconfigured lines. Ensure that linecoding, framing, clock source and line termination (balanced or unbalanced) are configured according to what the service provider recommends.

show controller e1 Conditions

The E1 controller can be in one of the following three states.

• Administratively down

• Down

• Up

Is the E1 Controller Administratively Down?

The controller is administratively down when it has been manually shut down. You should restart the controller to correct this error.

1. Enter enable mode.

   maui-nas-03>enable
   Password: 
   maui-nas-03#

2. Enter global configuration mode.

   maui-nas-03#configure terminal
   Enter configuration commands, one per line.  End with CNTL/Z.
   maui-nas-03(config)#

3. Enter controller configuration mode.

   maui-nas-03(config)#controller e1 0
   maui-nas-03(config-controlle)#

4. Restart controller.

   maui-nas-03(config-controlle)#shutdown
   maui-nas-03(config-controlle)#no shutdown
   

Is the Line Up?

If the E1 line is not up, check to see that the line configuration is correct and matches the settings of the remote end.

1. Check the framing of the line and the remote end. For E1 lines, the framing is either CRC4 or noCRC4

2. Check the linecoding of the line and the remote end. The linecoding is either AMI or HDB3.

3. Check to see if the line termination is set for balanced or unbalanced (75-ohm or 120-ohm).

Consult your service provider for more information regarding the correct settings. Make any changes as necessary to both local or remote end-devices.

If the E1 controller and line are not up, check to see if one of the following messages appears in the show controller e1 EXEC output:

• Receiver has loss of frame

• Receiver has loss of signal

If E1 Receiver Has Loss of Frame:

Follow these steps if E1 receiver has loss of frame.

1. Check to see if the framing format configured on the port matches the framing format of the line. You can check the framing format of the controller from the running configuration or the show controller e1 command output.

   To change the framing format, use the framing {CRC4 | no CRC4} command in the controller configuration mode as shown below:

   maui-nas-03#configure terminal 
   Enter configuration commands, one per line.  End with CNTL/Z.
   maui-nas-03(config)#controller E1 0
   maui-nas-03(config-controlle)#framing CRC4
   

2. Try the other framing format to see if the alarm clears.

   If this does not fix the problem, proceed to the «If E1 Receiver Has Loss of Signal» section below.

3. Check the framing format on the remote end.

4. Check the linecoding on the remote end.

If E1 Receiver Has Loss of Signal:

Follow these steps if E1 receiver has loss of signal

1. Make sure that the cable between the interface port and the E1 service provider’s equipment (or E1 terminal equipment) is connected correctly. Check to see if the cable is hooked up to the correct ports. Correct the cable connections if necessary.

2. Check cable integrity. Look for breaks or other physical abnormalities in the cable. Ensure that the pinouts are set correctly. If necessary, replace the cable.

3. Check the cable connectors. A reversal of the transmit and receive pairs or an open receive pair can cause errors. Set the receive pair to lines 1 & 2. Set the transmit pair to lines 4 & 5.

   The pins on a RJ-48 jack are numbered from 1 through 8. Pin 1 is the leftmost pin when looking at the jack with the metal pins facing you. Refer to the following figure for more information.

   Figure 15-11: RJ-45 Cable

   

4. Try using a rollover cable.

5. Check to see if there are far-end block errors. If so, the problem exists with the receive lead on the local end. Contact the TAC for more assistance.

Run the show controller e1 EXEC command after each step to check if the controller exhibits any errors.

If the Line is in Loopback Mode:

Check to see if the line is in loopback mode from the show controller e1 output. A line should be in loopback mode only for testing purposes.

To turn off loopback, use the no loopback command in the controller configuration mode as shown below:

maui-nas-03(config-controlle)#no loopback

If the Controller Displays Any Alarms:

Check the show controller command output to see if there are alarms displayed by the controller.

We will now discuss various alarms and the procedure necessary to correct them.

Receiver (Rx) Has Remote Alarm:

A received remote alarm means there is an alarm occurring on the line upstream of the equipment connected to the port.

1. Check to see if the framing format configured on the port matches the framing format of the line. If not, change the framing format on the controller to match that of the line.

2. Check the linecoding setting on the remote-end equipment. Contact your service provider for the correct settings. Correct any misconfigurations as necessary.

3. Insert an external loopback cable into the port. To create a loopback plug, see the section «Performing Hardware Loopback Plug Test,» earlier in the chapter.

4. Check to see if there are any alarms. If you do not see any alarms, then the local hardware is probably in good condition. In that case:

   1. Check the cabling. Refer to the section «If E1 Receiver Has Loss of Signal» for more information.

   2. Check the settings at the remote end and verify that they match your port settings.

   3. If the problem persists, contact your service provider.

5. Remove the loopback plug and reconnect your E1 line.

6. Check the cabling. See the section «If E1 Receiver Has Loss of Signal» for more information.

7. Power-cycle the router.

8. Connect the E1 line to a different port. Configure the port with the same settings as that of the line. If the problem does not persist, then the fault lies with the one port:

   1. Reconnect the E1 line to the original port.

   2. Proceed to the «Troubleshooting E1 Error Events» section.

      If the problem persists, then:

9. Perform a hardware loop test as described in the section «Performing Hardware loopback Plug Test»

10. Replace the E1 controller card.

11. Proceed to the «Troubleshooting E1 Error Events» section.

Transmitter Sending Remote Alarm (Red):

A Red alarm is declared when the CSU cannot synchronize with the framing pattern on the E1 line.

1. Check to see if the framing format configured on the port matches the framing format of the line. If not change the framing format on the controller to match that of the line.

2. Check the settings at the remote end and verify that they match your port settings.

3. Insert an external loopback cable into the port. To create a loopback plug, see the section «Performing Hardware Loopback Plug Test,» earlier in the chapter.

4. Check to see if there are any alarms. If you do not see any alarms, then the local hardware is probably in good condition. In that case:

   1. Check the cabling. Refer to the section «If E1 Receiver Has Loss of Signal» for more information.

   2. If the problem persists, contact your service provider.

5. Connect the E1 line to a different port. Configure the port with the same settings as that of the line. If the problem does not persist, then the fault lies with the one port.

   1. Reconnect the E1 line to the original port.

   2. Proceed to the «Troubleshooting E1 Error Events» section.

      If the problem persists, then:

6. Perform a hardware loop test as described in the section «Performing Hardware Loopback Plug Test.»

7. Replace the E1 controller card.

8. Proceed to the «Troubleshooting E1 Error Events» section.

9. Contact your service provider.

Troubleshooting E1 Error Events

The show controller e1 EXEC command provides error messages that can be used to troubleshoot problems. We will now discuss several error messages and how to correct the errors.

To see if the error counters are increasing, execute the show controller e1 command repeatedly. Note the values of the counters for the current interval. Consult your service provider for framing and linecoding settings.

Slip Secs Counter is increasing:

The presence of slips on E1 lines indicates a clocking problem. The E1 provider (Telco) will provide the clocking to which the Customer Premises Equipment (CPE) should be synchronized.

1. Verify that the clock source is derived from the network. This can be ascertained by looking for Clock Source is Line Primary.

   Note: If there are multiple E1s in an access server, only one can be the primary, while the other E1s derive the clock from the primary. In that case, verify that the E1 line designated as the primary clock source is configured correctly.

2. Set the E1 clock source correctly from the controller configuration mode.

   maui-nas-03(config-controlle)#clock source line primary
   

Framing Loss Seconds Counter is Increasing:

Follow these steps when framing loss seconds counter is increasing:

1. Check to see if the framing format configured on the port matches the framing format of the line. You can check this by looking for the Framing is {CRC4|no CRC4} in the show controller e1 output.

2. To change the framing format use the framing {CRC4 | no CRC4} command in the controller configuration mode as shown below:

   maui-nas-03(config-controlle)#framing crc4
   

Line Code Violations are Increasing:

Follow these steps when line code violations are increasing.

1. Check to see if the linecoding configured on the port matches the framing format of the line. You can check this by looking for the Line Code is {AMI/HDB3} in the show controller e1 output.

2. To change the linecoding, use the linecode {ami | hdb3} command in the controller configuration mode as shown below:

   maui-nas-03(config-controlle)#linecode ami
   

Verifying that ISDN Switch Type and PRI-Group are Configured Correctly

Use the show running-config command to check if ISDN switch type and the PRI-group timeslots are configured correctly. Contact your service provider for correct values.

To change the ISDN switch type and PRI-group:

maui-nas-03#configure terminal
maui-nas-03(config)#isdn switch-type primary-net5
maui-nas-03(config)#controller e1 0
maui-nas-03(config-controlle)#pri-group timeslots 1-31

Verifying the Signaling Channel

If the error counters do not increase but the problem persists, verify that the signaling channel is up and configured correctly.

1. Run the show interface serial x:15 command, where x should be replaced by the interface number.

2. Check to see if the interface is up. If the interface is not up, use the no shutdown command to bring the interface up.

   maui-nas-03#config terminal
   Enter configuration commands, one per line.  End with CNTL/Z.
   maui-nas-03(config)#interface serial 0:15
   maui-nas-03(config-if)#no shutdown
   

3. Ensure that encapsulation is PPP. If the interface is not using PPP, then use the encapsulation ppp command in the interface configuration mode to correct it.

   maui-nas-03(config-if)#encapsulation ppp
   

4. Check to see if loopback is set. Loopback should be set only for testing purposes. Use the no loopback command to remove loopbacks.

   maui-nas-03(config-if)#no loopback
   

5. Power-cycle the router.

6. If the problem persists, contact your service provider or the Cisco TAC.

Troubleshooting a PRI

When troubleshooting a PRI, you need to determine if the E1 is running cleanly on both ends. If Layer 1 problems have been resolved as described above, consider Layer 2 and Layer 3 problems.

Troubleshooting Using the show isdn status Command

The show isdn status command is used to display a snapshot of all ISDN interfaces. It displays the status of Layers 1, 2 and 3.

1. Verify that Layer 1 is active.

   The Layer 1 status should always say ACTIVE unless the E1 is down.

   If show isdn status indicates that Layer 1 is DEACTIVATED, then there is a problem with the physical connectivity on the E1 line. See the section «Is the E1 Controller Administratively Down?»

   Also verify that the E1 is not administratively down. Use the no shutdown command to bring the E1 controller up.

2. Check to see if the Layer 2 State is MULTIPLE_FRAME_ESTABLISHED.

The desired Layer 2 state is Multiple_Frame_Established, which indicates the startup protocol between ISDN switch and end-device has been established and we are exchanging Layer 2 frames.

If Layer 2 is not Multiple_Frame_Established, use the show controller e1 EXEC command to diagnose the problem. See «Troubleshooting Using the show controller e1 Command» section in this chapter and the «Troubleshooting E1 Error Events» section.

Because show isdn status is a snapshot of the current status, it is possible that Layer 2 is bouncing up and down despite indicating Mulitple_Frame_Established. Use the debug isdn q921 command to verify that Layer 2 is stable.

Using debug q921

The debug isdn q921 command displays data link layer (Layer 2) access procedures that are taking place at the router on the D-channel.

Ensure that you are configured to view debug messages by using the logging console or terminal monitor command as necessary.

Note: In a production environment, verify that console logging is disabled. Enter the show logging command. If logging is enabled, the access server may intermittently freeze up as soon as the console port gets overloaded with log messages. Enter the no logging console command.

Note: If debug isdn q921 is turned on and you do not receive any debug outputs, place a call or reset the controller to get debug outputs.

1. Verify that Layer 2 is stable. You should observe the debug outputs for messages indicating that the service is not bouncing up and down. If you see the following types of debug outputs, the line is not stable.

   Mar 20 10:06:07.882: %ISDN-6-LAYER2DOWN: Layer 2 for Interface Se0:15, TEI 0 
   changed to down
   Mar 20 10:06:09.882: %LINK-3-UPDOWN: Interface Serial0:15, changed state to down
   Mar 20 10:06:21.274: %DSX1-6-CLOCK_CHANGE: Controller 0  clock is now selected 
   as clock source
   Mar 20 10:06:21.702: %ISDN-6-LAYER2UP: Layer 2 for Interface Se0:15, TEI 0 
   changed to up
   Mar 20 10:06:22.494: %CONTROLLER-5-UPDOWN: Controller E1 0, changed state to up
   Mar 20 10:06:24.494: %LINK-3-UPDOWN: Interface Serial0:15, changed state to up
   
   

   If Layer 2 does not appear to be stable, see «Troubleshooting E1 Error Events,» earlier in this chapter.

2. Verify that you are seeing only SAPI messages in both transmit (TX) and Receive (RX) sides.

   Mar 20 10:06:52.505: ISDN Se0:15: TX ->  RRf sapi = 0  tei = 0  nr = 0
   Mar 20 10:06:52.505: ISDN Se0:15: RX <-  RRf sapi = 0  tei = 0  nr = 0
   Mar 20 10:07:22.505: ISDN Se0:15: TX ->  RRp sapi = 0  tei = 0 nr = 0
   Mar 20 10:07:22.509: ISDN Se0:15: RX <-  RRp sapi = 0  tei = 0 nr = 0
   Mar 20 10:07:22.509: ISDN Se0:15: TX ->  RRf sapi = 0  tei = 0  nr = 0
   Mar 20 10:07:22.509: ISDN Se0:15: RX <-  RRf sapi = 0  tei = 0  nr = 0

3. Verify that you are not seeing SABME messages, which indicates that Layer 2 is trying to reinitialize. This is usually seen when we are transmitting poll requests (RRp) and not getting a response from the switch (RRf) or vice-versa. Below are example of SABME messages. We should get a response from ISDN switch for our SABME messages (UA frame received).

   Mar 20 10:06:21.702: ISDN Se0:15: RX <-  SABMEp sapi = 0  tei = 0
   Mar 20 10:06:22.494: ISDN Se0:15: TX ->  SABMEp sapi = 0  tei = 0

   If you are seeing SABME messages, use the show running-config command to check if ISDN switch type and the PRI-group timeslots are configured correctly. Contact your service provider for correct values.

   To change the ISDN switch type and PRI-group:

   maui-nas-03#configure terminal
   maui-nas-03(config)#isdn switch-type primary-net5
   maui-nas-03(config)#controller e1 0
   maui-nas-03(config-controlle)#pri-group timeslots 1-31
   

4. Verify that the D-channel is up using the show interfaces serial x:15 command.

   If the D-channel is not up, then use the no shutdown command to bring it up:

   maui-nas-03(config)#interface serial 0:15
   maui-nas-03(config-if)#no shutdown
   

5. Check to see if encapsulation is PPP. If not use the encapsulation ppp command to set encapsulation.

   maui-nas-03(config-if)#encapsulation ppp
   

6. Check to see if the interface is in loopback mode. For normal operation, the interface should not be in loopback mode.

   maui-nas-03(config-if)#no loopback
   

7. Power-cycle the router.

8. If the problem persists, contact your service provider or the Cisco TAC.

Related Information

• Technical Support — Cisco Systems

In my topology, I have connected spirent — traffic generator to MRV switch ( L1 switch) and then connected to CISCO switch. I am seeing some input errors on this interface. What does it mean? Does it mean that some packets are bad from spirent MRV switch. here is the show command of the interface gi 0/13.

In my topology, spirent is connected to MRV switch ( L1 switch).
MRV switch interface gi 0/13 is connected to the CISCO switch interface gi 0/14.
I don’t see any errors on the gi 0/14 of the cisco switch. However, I see the errors on the gi 0/13 ,which is the interface coming from MRV. Does it mean that CISCO switch is telling that there is some problem in the MRV switch?

  1557336 input errors, 1557204 CRC, 0 frame, 0 overrun, 0 ignored

My quesion is what the above error indicates to me.

 c3560g_1>sh interfaces gi 0/13    
    GigabitEthernet0/13 is up, line protocol is up (connected) 
      Hardware is Gigabit Ethernet, address is 0013.c4d0.570d (bia 0013.c4d0.570d)
      Description: -- MRV, 1.1.19 --
      MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
      input flow-control is off, output flow-control is unsupported 
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input never, output 00:00:01, output hang never
      Last clearing of "show interface" counters never
      Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 0 bits/sec, 0 packets/sec
      5 minute output rate 1000 bits/sec, 2 packets/sec
         80971440 packets input, 125278495288 bytes, 0 no buffer
         Received 0 broadcasts (0 multicasts)
         132 runts, 0 giants, 0 throttles
         1557336 input errors, 1557204 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog, 0 multicast, 0 pause input
         0 input packets with dribble condition detected
         24289474 packets output, 36669631614 bytes, 0 underruns
         0 output errors, 0 collisions, 5 interface resets
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier, 0 PAUSE output
         0 output buffer failures, 0 output buffers swapped out

По snmp снимать входящие ошибки. В счетчике ifInErrors все подряд. Статус дуплекса есть в EtherLike-MIB как и множество других счетчиков.

input errors overrun

На шасси с установленной свичевой картой. Ошибки overrun вызваны недостаточной производительностью маршрутизатора. Причем ситуацию в конкретном случае можно усугубить. Для этого достаточно в таких маршрутизаторах использовать свичевые карты, чтобы трафик передавался через внутренний интерфейс Ba0/3. Cудя по таблице iftable, счетчики ошибок будут расти быстрее на этом интерфейсе при использовании свичевой карты с гигабитными портами.
В качестве примера вывод счетчиков с маршрутизатора cisco 1921

cisco1921# sh int | in overr|^GigabitEthernet0/[01]
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
GigabitEthernet0/0 is up, line protocol is up
     48677539 input errors, 0 CRC, 0 frame, 48677539 overrun, 0 ignored
GigabitEthernet0/1 is up, line protocol is up
     4533642 input errors, 0 CRC, 0 frame, 4533642 overrun, 0 ignored

input errors ignored

Недостаточная производительность. От приоритезации толку нет, будут дропы в голосе и видео.

98559 input errors, 0 CRC, 0 frame, 0 overrun, 98559 ignored

input errors unknown protocol drops

Вероятно LLDP включен на коммутаторе и маршрутизатором не поддерживается.

2238654 unknown protocol drops

input errors CRC

Проверить физику, дуплекс. Возможно небольшое колво ошибок при включении интерфейса.

59 input errors, 59 CRC, 0 frame, 0 overrun, 0 ignored

output drops

При настроенных на ограничение скорости политиках счетчик малоинформативен.

Input queue: 0/75/1/0 (size/max/drops/flushes); Total output drops: 3354485

input errors runts

Порт настроен в access, а на удаленной в trunk если ошибки постоянно растут.

32 runts, 0 giants, 0 throttles
32 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

input errors FCS

Менять SFP, порт оборудование на удаленной стороне.
Но это не точно поможет, т.к. если коммутатор на удаленной стороне работает в Cut-Through режиме коммутации, то фрейм с ошибкой мог передаться с предыдущего сетевого оборудования.
Возможно не работает автосогласование дуплекса. На 100мбит оптических линках это возможно. /network/eltex/#negotiation

input errors CRC

Проверить физику, дуплекс. На оптических линках возникает при заломах пачкорда, поэтому не надо его наматывать на органайзеры.
Ошибки могут возникать, если на одномодовом участке волс есть многомодовый патчкорд или пигтейл.

output discards

• если есть qos, смотреть очереди, буферы. Вот тут и начиаешь ценить коммутаторы со счетчиками очередей.
• speed mismatch (аплинк 1Gbit/s, downlink 100Mbit/s). Потери из-за burst, который не смог буферизировать коммутатор. Большинство свичей с shared буфером на всех портах.

дропы из-за неудачной модели

При выключенном на всех портах flowcontrol на тестах работает все медленно.
Счетчика ошибок нет. Смотреть и сравнивать счетчики на аплинке и даунлике.
Если не одинаково, то ставить другую модель коммутатора.

дропы из-за переподписки

На модульных коммутаторах при использовании плат с переподпиской возможны ошибки. Обратить внимание, что порты сгруппированы и не испольовать часть портов.