This section introduces the Monitoring and the Recording components. These components are software processes that can run on the same physical server or separate servers from the Cisco CRS Engine and Database components. Details on supported deployment scenarios are covered in Chapter 3.

Unified CCX Enhanced and Premium provide the ability for a supervisor to silently monitor agents. Unified CCX Enhanced and Premium also provide the ability for agent calls to be recorded.

Agent call recording can be triggered in the following ways:

•Supervisor clicks record button on Cisco Supervisor Desktop (CSD) for a specified agent call

•Agent clicks record button on Cisco Agent Desktop (CAD) or IP Phone Agent (IPPA)

•Workflows defined in Cisco Desktop Administrator automatically triggers complete call recording on certain types of calls for agents using CAD.

In order to use the silent monitoring or recording features, access to the RTP (Real-Time Protocol) packet streams is required. Silent monitoring and recording will work with either G.711 or G.729 RTP streams and a mixture of agents using G.711 and G.729 phones is supported. However, silent monitoring and recording will not work with encrypted media streams. Unified CCX provides two mechanisms for access to the RTP packet stream— SPAN port monitoring and desktop monitoring.

SPAN port monitoring requires a physical server running a Monitoring software process to be connected to the SPAN port of a VLAN on a Catalyst switch where the agent’s phone is installed. The SPAN port is like a broadcast port for all data traffic (including voice RTP streams) traversing a VLAN segment. When a supervisor clicks the silent monitor button on the CSD, it signals to the appropriate Monitoring component to forward a copy of the captured RTP streams for the selected agent to the requesting CSD. The CSD then plays the packets through the sound card on the CSD workstation. No IP Phone (or any type of phone) is involved when the silent monitoring stream is being played using CSD. The CSD can reside anywhere on the Unified Communications network, but the agent’s phone must be on the same VLAN where the SPAN port monitoring component is installed. The Catalyst switch RSPAN feature allows a VLAN to extend across multiple Catalyst switches. Please refer to Appendix B for more detail on SPAN port monitoring design guidance.

Note

For any deployment in which an agent desktop is the IP Phone Agent or any deployment in which the desktop is a Cisco Agent Desktop and the associated phone does not support desktop (endpoint) monitoring, monitoring and recording have to based on SPAN port monitoring. For a list of phones that support desktop (endpoint) monitoring, refer to Cisco CRS Software and Hardware Compatibility

Desktop monitoring provides a mechanism for the CAD application to obtain a copy of the RTP packet streams directly from the phone and therefore removing the need for a Monitoring component connected to the SPAN port on the Catalyst switch. A Cisco phone supporting desktop monitoring is required and the agent workstation running CAD must be connected to the data port on the back of the agent phone. The IP Communicators (softphones) also support using desktop monitoring for silent monitoring and recording.

Note

For all deployments in which agents use Cisco Agent Desktop and agent phones support desktop monitoring, use desktop monitoring instead of SPAN port monitoring.

When a supervisor clicks the silent monitor button on the Cisco Supervisor Desktop for an agent using desktop monitoring, the RTP streams are sent directly from Cisco Agent Desktop to Cisco Supervisor Desktop, and no SPAN port monitoring component is required. However, in order for silent monitoring to occur with desktop monitoring, there must still be at least one Monitoring process running somewhere. The monitoring process is used by Cisco Agent Desktop to retrieve the agent phone's MAC address from Unified CM. For desktop monitoring, the agent workstation must have a NIC that supports 802.1Q. This allows the NIC to process packets from both the data and voice VLANs. Appendix C of the Cisco CAD Installation Guide provides a quick and simple test to determine if a workstation NIC will operate properly with the desktop monitoring feature of CAD.

A Unified CCX deployment and individual locations and sites can have a mixture of some agents using desktop monitoring and some agents using SPAN port monitoring.

If an agent call requires recording, then a copy of the RTP packet streams is sent to the Recording Server. If desktop monitoring is being used by the agent being recorded, then CAD sends the RTP streams to the Recording component. If SPAN port monitoring is being used by the agent being recorded, then the Monitoring component (on the VLAN where the agent phone is connected) sends the RTP streams to the Recording component. Agents can be silently monitored and recorded at the same time. When that occurs, CAD or the Monitoring component are sending two copies of the RTP packet streams.

A normal G.7xx VoIP RTP call has two RTP streams (one representing what the agent is hearing and one representing what the agent is saying). These two streams flow in opposite directions across the network. When an agent call is being silent monitored or recorded, both of those RTP streams must be sent. For example, if a supervisor is silent monitoring an agent, two G.7xx RTP streams will be sent from either CAD (desktop monitoring) or the Monitoring component to the CSD. If an agent call is being recorded, two G.7xx RTP streams are sent to the Recording component. If the agent is being silent monitored and recorded, four RTP streams are being sent. This is in addition to the two bi-directional RTP streams of the actual call.
The monitoring and recording packet streams are true G.7xx RTP streams and should be tagged like any other RTP stream to ensure these packets are delivered with appropriate priority and minimal latency. Chapter 6 further discusses bandwidth requirements.

An Unified CCX cluster can have up to five Monitoring components with one of them running on the logical Recording component. The Recording component requires a co-resident Monitoring component (but only one—regardless of whether the Recording component is simplex or redundant). Four additional Monitoring components can be deployed if SPAN port monitoring at remote agent sites is needed.

The agent call recordings are stored on the hard drive of the Recording component server with agent data store locator records pointing to the actual recording files. If a redundant Recording component server is deployed, they operate in a load balancing fashion, and recordings are only stored on the hard drive of the Recording component server that actually received the RTP stream. The call recordings in Unified CCX 6.0 are stored in a raw format that is only playable using the Cisco Supervisor Desktop (CSD) Record Viewer. The CSD Record Viewer shows 7 days worth of call recording as well as those tagged for 30-day extended archiving. The CSD Record Viewer also provides the supervisor the option to save selected individual recordings into a .wav format in a specified folder.

The recording capability of Unified CCX is not intended for usage as a permanent recording archival solution. However, an export utility is also available to bulk export all recordings into a .wav format. The export utility has no ability to specify selected recordings and will export all recordings on a Recording component. System administrators could build their own customized command macros or process that would perform regular (at least weekly) exporting of the recordings for permanent archival of agent call recordings.

With Unified CCX Enhanced, up to 32 simultaneous agent calls can be recorded. With Unified CCX Premium, up to 80 simultaneous agent calls can be recorded. When a supervisor is playing back or saving a recording using the CSD Record Viewer application, a recording resource is used and therefore counts against the maximum simultaneous call recording capacity for the duration of that recording playback. Maximum simultaneous call recording and playback capacity is dependent upon the deployment model and server sizing. If more than 32 simultaneous recordings and playbacks are required, the Recording component must be separated from the Cisco CRS Engine and Database components. A dedicated 7845-class server for the Recording process is required for 80 simultaneous recordings and playbacks. The Configuration & Ordering tool can assist you in determining an appropriately sized server for the amount of recording required. See Appendix A, “Server Capacities and Limits” for the full capacity matrix.

Because IP Phone Agent (IPPA) does not include an agent using CAD, IPPA requires a SPAN port Monitor component on the local VLAN segment for silent monitoring or recording. Also, IPPA cannot be configured to have calls automatically recorded.
If no agent call recording is required but silent monitoring is required, then a deployment may contain 5 Monitoring components that are for SPAN port monitoring only. Even if all agents will use desktop monitoring, at least one Monitoring component must still be installed. The monitoring component is used by CAD to retrieve the agent phone's MAC address from Unified CM.

Unified CCX Premium is required for remote supervisory monitoring. Remote supervisory monitoring provides a mechanism to silent monitor calls using an IP Phone or PSTN phone. This form of silent monitoring does not require a CSD or any data network connectivity and is ideally suited for management from outsourcer customers of a call center service provider.
Agents are unaware when they are being silent monitored using remote supervisory monitoring.

A remote supervisor is configured with a numeric user ID and password and also with the CSQs and agents that the remote supervisor is allowed to silent monitor in this fashion. The remote supervisor then dials a specific number that invokes a Unified CCX application. The application begins by prompting the supervisor for the user ID and password. After the remote supervisor is authenticated, the remote supervisor is prompted on whether they wish to silent monitor calls for a specific agent or for a specific CSQ. Then the Unified CCX application requests a copy of the RTP streams for the selected types of calls, and the Unified CCX application and CTI Port relays those packets to the remote supervisor's phone. Remote supervisory monitoring works with both SPAN port monitoring and desktop monitoring. However, remote supervisory monitoring only works with a Cisco CRS Engine and CTI Ports and agent phones using G.711 encoding. Remote supervisory monitoring also places an additional impact on the Cisco CRS Engine. This activity is reflected in the Unified CCX 6.0 Configuration & Ordering Tool.

==========================================================

Monitoring and Recording Redundancy

Unified CCX supports up to six servers running the Monitoring component. Five of these servers can provide SPAN port monitoring service. The first Monitoring component must be installed on a server that is running the Recording component. If there is a second server running a Recording Component, a Monitoring component has to be installed on that second server as well, but SPAN port monitoring is supported only on one of these two servers. These two servers running the Monitoring service are sometimes considered as one “monitoring domain.” The other four monitoring components are typically for SPAN port monitoring for agents at remote sites. When configuring a phone with SPAN port monitoring, only one SPAN port monitoring server can be assigned to this phone. There is no redundancy when SPAN port monitoring is used.

For desktop monitoring, CAD forwards the RTP stream to CSD. But a server running the Monitoring component is still required in order to retrieve agent phone's MAC address from Unified CM. Any one of the six monitoring servers could be chosen for this purpose. If one or multiple Monitoring components fail, desktop monitoring will still work, as long as one server running the Monitoring component is still available in the CRS cluster.
It is possible to configure and enable both SPAN port monitoring and desktop monitoring for a phone. However, only one method is used at any time for that phone. If both SPAN port monitoring and desktop monitoring are configured correctly, desktop monitoring is chosen. If desktop monitoring fails, SPAN port monitoring is used as a backup. Please refer to the Cisco Desktop Administrator User’s Guide for more information.
When deploying High Availability with Recording component installed in CRS Engine node, the second CRS Engine node with Recording component must be deployed. The two physical Recording servers work as a single logical Recording server (a "recording domain") and recording tasks are load balanced in a round robin fashion across the two physical Recording Servers. An Unified CCX deployment only supports one "recording domain." The actual call recordings are stored only on the disk of the physical Recording component server where the recording task took place. Therefore, if a Recording server fails, the Supervisor will be unable to playback those recordings on the failed Recording server until that Recording server is operational again.

The two servers where the Recording components are running also serve as a backup for each other. In order to function properly during a period when one of the servers fails, the two Recording servers must be sized to be capable of supporting all recording for the Unified CCX cluster. For example, under normal operations, a large call center may be setup to handle 16 recording sessions on each Recording component (for a total of 32 simultaneous call recordings). If either server with a Recording component fails, the other server would be capable of providing 32 simultaneous call recordings; thus there would be no loss in recording capacity. The Unified CCX Configuration & Ordering tool takes into account this type of failover in the sizing of hardware resources for agent call recording.

Recording requires a Monitoring component. When SPAN port monitoring is configured for silent monitoring, the SPAN port monitoring server forwards the RTP stream to the Recording component. If that SPAN port monitoring server fails, recording is not possible. When desktop monitoring is configured, the Monitoring component is still required in order to retrieve agent phone's MAC address from Unified CM. Any one of the six monitoring servers could be used for this purpose. If one or multiple Monitoring components fail, recording still works, as long as one server running the Monitoring component is still available in the CRS cluster.

===============================================

Silent Monitoring Bandwidth Usage
The silent monitoring feature of the CAD desktop software, which includes both listening to and recording agent calls, has the largest bandwidth requirements for the CAD product. Properly configuring this feature is especially important for remote agents who are connected to the main site by a WAN connection.
An agent's call can be listened to or recorded by the CAD software. To do this, a request is sent to a VoIP provider. The VoIP provider, captures the voice streams representing the call (two voice streams per call) and sends them back to the requestor. The bandwidth requirements detailed in this section are for the network links between the requestor and provider.
Silent Monitoring Requestors
There are two possible requestors in the CAD software:
•Cisco Supervisor Desktop
•Recording service
Cisco Supervisor Desktops will send requests when the supervisor wishes to listen to an agent's call in real-time. The VoIP provider will capture the voice streams and send them back to the supervisor's desktop where they can be listened to over the desktop's speakers.
A Recording service will send requests when either a supervisor or agent wishes to record the call. The VoIP provider will send the voice streams and the Recording service will save the streams to disk so they can be listened to at a later time.
An installation may have one or two Recording services. An off-board Recording service may be installed at a remote office location, if all Agents and all Supervisors are at that remote location.
Silent Monitoring Providers
There are also two possible VoIP providers in the CAD software:
•Cisco Agent Desktop
•VoIP Monitor service

The Cisco Agent Desktop application contains a service referred to as the Desktop Monitor service that runs on the agent's desktop. It is responsible for processing silent monitoring requests only for the agent logged into the CAD application on the desktop. It captures voice packets sent to the IP or soft phone associated with the logged in agent. The IP phone must be connected in series with the agent desktop on the network for this to work.
By default, this service is active on all agent desktops when the application is started. After initial installation of the CAD servers, all agents are already configured to use the Desktop Monitor service for the silent monitoring feature.
A VoIP Monitor service is able to handle multiple requests for silent monitoring simultaneously. It captures packets directly from the switch via the switch's Switched Port Analyzer (SPAN) configuration. An installation may have up to four VoIP Monitor services on different machines. Off-board VoIP services may be installed at remote office locations. In some instances, this may be required due to network complexity and capacity planning.

NoteIP Phone agents, who don't have a desktop, must be configured to use a VoIP Monitor service for the silent monitoring feature.

It is easy to see where the bandwidth will be required for the silent monitoring feature when you can locate the requestors and providers. There are some notes of interest regarding bandwidth that are shown in the figure:
•Although an administrator is able to assign a specific VoIP service to an agent device, the Recording service that is used is determined at the time the request is made. This applies if two Recording services were installed in order to load-balance the installation. This may result if the provider and requestor being separated by a WAN and requiring the bandwidth on the WAN.
•If the VoIP provider is a VoIP Monitor service, the requestor is a Recording service, and these services reside on the same machine, there is no additional bandwidth used on the network to record the call.
Regardless of who the requestor and VoIP provider are, the bandwidth requirement between these two points is the bandwidth of the IP call being monitored and/or recorded. You can think of each monitoring and/or recording session as being a new phone call (2 voice streams) for calculating bandwidth. Therefore, to calculate bandwidth to support the silent monitoring feature, you can use the same calculations used to provision the network to handle call traffic.

==========================================

IP Call Bandwidth Usage
An IP phone call consists of two streams of data. One stream is sent from phone A to phone B. The other stream is sent from phone B to phone A. The voice data is encapsulated into packets that are sent over the network. The amount of data required to store a voice stream is dependent upon the CODEC used to encode the data. The CAD software can support both the G.711 and G.729 CODEC.
The voice data itself is transmitted over the network using the Real-Time Transport Protocol (RTP). The RTP protocol supports the idea of silence suppression. When silence suppression is used, no voice packets are sent over the network if there is not sound. Otherwise, even packets that contain silence are sent. This lowers the average required bandwidth for a call. Although CAD supports silence suppression, the lower bandwidth requirements for silence suppression should not be used when provisioning the network because the worst case scenario would be where there is not silence in the call, requiring the full call bandwidth as if silence suppression was not enabled.
When calculating bandwidth for an IP call, you must use the size of the RTP packet plus the additional overhead of the networking protocols used to transport the RTP data through the network.
For example, G.711 packets carrying 20 ms of speech data require 64 kbps (kilobytes per second) of network bandwidth per stream. These packets are encapsulated by four layers of networking protocols (RTP, UDP, IP, and Ethernet). Each of these protocols adds its own header information to the G.711 data. As a result, the G.711 data, once packed into an Ethernet frame, requires 87.2 kbps of bandwidth per data stream as it travels over the network. Since an IP phone call consists of two voice streams, in this example, a call would require 174.4 kbps.
The amount of voice data in a single packet also influences the size of the packet and bandwidth. The example above used packets containing 20 milliseconds of speech for its calculations, but this value can be changed in the Cisco Unified Communications Manager (Unified CM) configuration for each supported CODEC. Configuring packets to contain more speech information reduces the number of packets sent over the network and reduces the bandwidth since there are fewer packets containing the additional networking headers, but the packet sizes increase. Table 6-1 shows the bandwidth required for a phone call for the different combinations of CODEC and amount of speech per packet.

Both monitoring and recording packets are real RTP streams and are not tagged with QoS marking by default. Silent monitoring sessions are time sensitive because of the real-time nature. Tag the packets in the network infrastructure to the same QoS marking as other real-time voice traffic. Properly provision the priority queue to include silent monitoring traffic. Recording packets can be tagged with QoS marking that is lower priority than the real-time voice traffic, because the packets do not need to be delivered in real time. The ports used for monitoring and recording are listed in Port Utilization for Product Revisions, page 6-13, and can be used to classify monitoring or recording traffic.
For more information about QoS traffic classification, see QoS and Call Admission Control, page 6-14. For provisioning guidelines for centralized call processing deployments, see the Cisco Unified Communications Solution Reference Network Design documentation, available online at:
http://www.cisco.com/warp/public/779/largeent/it/ese/unifiedCommunications.html

=================================================

Voice Over IP Monitoring
Monitoring and recording of agent calls can be supported by two different methods in this release of Unified CCX:
•The traditional VoIP monitor Service: captures packets directly from an IP network switch via the switch's Switched Port Analyzer (SPAN) configuration. Design considerations for the traditional SPAN-based VoIP monitor Service are provided at the end of this appendix (see Design Considerations for SPAN-Based Services, page B-1).
•The Cisco Agent Desktop, also known as Endpoint monitoring or the Desktop Monitoring Service: The agent's IP phone repeats RTP packets to the agent's PC. When a supervisor wants to monitor/record the agent, the supervisor application sends a message to the agent desktop to forward the RTP packets to the supervisor, who can then monitor the agent/caller conversation via the sound card on his or her PC. This method requires the agent to use the Cisco Agent Desktop (not the IP Phone Agent) and a phone that supports desktop monitoring. For the list of phones that support desktop monitoring, please refer to the Cisco CRS Software and Hardware Compatibility Guide, available at:
http://www.cisco.com/en/US/products/sw/custcosw/ps1846/products_device_support_tables_list.html
•Design considerations for the new Desktop (Endpoint) Monitoring Service are provided in Chapter 6, “Bandwidth, Security, and QoS Considerations.”
Design Considerations for SPAN-Based Services
Traditional SPAN-based VoIP services allows the IP traffic from one or more ports to be copied and sent to a single destination port.
Be aware if these factors when configuring traditional SPAN-based VoIP monitor services:
•If you are using a second network card in the VoIP monitor, make sure that the network card used by the CRS Engine has a higher binding order than the one used by VoIP monitor services. Refer the Cisco CAD Installation Guide for detailed information about setting network card binding order.
•The following switches do NOT support SPAN sessions: 1700, 2100, 2800, 2948G-L3, 4840G
•Local SPANs (LSPANs) are SPANs where all the source ports and the destination port are physically located on the same switch. Remote SPANs (RSPANs) can include source ports that are physically located on another switch. The following switches do NOT support RSPAN (although they may be an intermediate switch in an RSPAN configuration): 1200, 1900, 2550, 2820, 2900, 2900XL, 2926GS, 2926F, 2926GL, 2926T, 2948G, 2950, 2980G, 3000, 3100, 3200, 3500XL, 3524-PWR XL, 3508GL XL, 5000, 5002, 5500, 5505, 5509

•Some switches do not allow the destination port of a SPAN configuration to act as a normal network connection. The only traffic that can flow through this port is the traffic copied from the SPAN source ports; this requires the computer running the VoIP monitor service to have two network connections (NICs) to function properly. The following switches do NOT support normal network traffic on SPAN destination ports: 2950, 3000, 3100, 3200, 3550
•In some configurations, the VoIP Monitor service can receive duplicate voice packets, which causes poor speech quality. To avoid this, only Ingress packets to a port are sent to the VoIP monitor service. This is a setting for SPAN, which the following switches do NOT support: 1900, 2820, 2900, 2900XL, 3000, 3100, 3200, 3500XL
•In some switches, SPAN cannot use VLANs as sources, which is known as VSPAN. In that case, SPAN must designate individual ports to use for monitoring. The following switches do NOT support VSPAN: 1200, 1900, 2820, 2900XL, 2950, 3000, 3100, 3200, 3500XL, 3524-PWR XL
For more information, refer to the Voice Over IP Monitoring Best Practices Deployment Guide.

The metrics used in call centers are often analyzed and then acted or reacted upon. The data in the form of abandoned calls, dropped calls or lost calls pretty much means the caller hung up. Why the caller hung up isn’t always because they are impatient, though recent statistics from Dimension Data’s 10th edition Contact Center Benchmarking report reveal that Call abandon rates – due to long hold times – increased in the past year by almost 127%, going from 6% of calls to nearly 14% of calls.

This yearly report measures and analyzes performance levels of contact centers worldwide. Dimension Data states, “callers have grown increasingly impatient while waiting, and abandon a call after waiting on average 45 seconds in 2007, compared to 53 seconds in 1997.” Stuck in queue or Get Back in Queue are options that customers just don’t like. The managers of call center metrics determine caller tolerances and weights of calls along with costs and even revenue or worth. Today, how many rings will you let any phone ring before hanging up? Then, how many of you that hung-up, expect the person/company that you called to call you back? (This is a social – younger/youthful behavior)

 Tolerance seems, in today’s standard, pretty low, especially when those call center engineers have to offer callers a choice of music just to appease them. If Dimension Data’s report is a reflection on life, then reexamination of the art of call centers is in order, and I don’t think self-service, Web portals and other gadgetry is the cure--but they will help. Instead, my bet is on the business process and this begins with throwing products over the wall at customers.

Once customers start throwing products back- sometimes the factory guy gets the message and even still, it’s often mis-communicated. Clearly, while people have grown impatient and less willing to sacrifice their time just because companies didn’t spend adequate time prior to releasing their product or service, this behavior (call abandonment) does cost the call centers money, especially if the same caller does call back. When the callers don’t call back, it’s loss of revenue.

In Beam Me Out, you can read some of Dimension Data’s previous findings. I want to use an old and good example of AT&T and the former (no longer manufactured but- still in service) Dimension 2000 PBX product manual that detailed behaviors and conditions of all the features and options within the product family. It was a great example of system documentation that was worthy. Why? Before implementing a feature and reading the guide as a first step; the documentation would provide enough detail to the reader, explaining that specific changes could mean a loss of a specific functionality or change in behavior of call routing. The down side to the manual was that it was available exclusively through the Consultant Liaison Program, unless of course you had national account status and/or pull to get one. Technical writing is no easy task.

In college, it was painful and even tiring, so my guess is that it’s appreciated when a manual or product/software document is well prepared. When it’s not, it will clearly generate telephone calls. Will better documentation solve the abandoned calls? No, but it will contribute to reducing calls, as will making better products and improving services so customers don’t call in. The same is true with the website supporting the product or service. The website must be user-centric not tech-centric, and it must flow smoothly or allow the users to flow through and obtain what they need to solve their own problems or answer their specific questions. Some of us try to figure out the answer to our questions while we’re stuck in queue waiting for support.

Recently, while in queue, I paid due diligence by poring through a 768-page user manual again, determined to find the buried answer (how to change the telephone ringer cadence) before speaking with factory first level support. It just wasn’t in the obvious sections of content, and while I listened to music, it was at the expense of the manufacturer and it did take me 27 minutes to find the solution. Rack up another successful abandoned call. Lousy documentation generates unnecessary telephone calls, and it’s not always because of laziness or desire for convenience. I hope that this behavior of mine goes noticed, because I’m not alone and these types of calls will impact your bottom line--and how many callers did I adversely affect by self serving myself at the factory’s expense while listening to their music for 27 minutes?

(Simple TIP for Better Documentation: 3Com did this very well. 3Com published an email address in the manuals to report discrepancies- so as dealers and end user customers, we could contact 3Com directly and note the discrepancy or issue. Empower your customers whomever they are, to do the same. You are enlisting a FREE workforce that improves the process for you) Alternatives: Improve the documentation, make better products or improve services offered, and enhance free flowing self-service through the web. Anyone assembling toys, bicycles or any baby items for the first time will have an appreciation of what I’m talking about. Life’s too short, and companies that don’t communicate in the end user’s national language by way of their manuals, documentation, CDs or other media forms can expect more abandoned calls and less repeat business.

That’s how it really works and that’s how your bottom lines are unnecessarily impacted. You can have the best call center in the world and still miss the mark. Dimension Data’s report goes on to say, “the effective use of technology – balanced by people, process and policies – continues to be more important than ever to turn the tide on customer service metrics, as more companies than ever are recognizing that customer service has a direct impact on their bottom line.”

My final suggestion is to reverse-engineer the process, and then you should be able to track the source of the problem. Make the call center agent drill back down to the customer to discover and document the source. Of course I know that takes more time and creates longer queues. So build better mousetraps instead.