Tropo is part of CiscoLearn More

Machine Detection Accuracy

Posted on December 14, 2015 by Phil Bellanti

As a follow up post to the Tropo voicemail detection application demo, I wanted to expand on the topic of accuracy for voicemail detection on outbound calls.  Since detection accuracy is a rather ambiguous term and one that’s open to a lot of interpretation, some additional insight is probably called for.

In truth, the only way to know if an answered call was detected accurately or not is to have a human score the call.  In other words, someone has to actually make a determination if the call was human or machine and compare that against the automated detection. This can be accomplished in a couple different ways.

The best method that would yield the most honest results would be to record all outbound calls, or at least random sampling of calls.  From there, log the detection result (HUMAN, MACHINE, etc.) in the application and in parallel, have a human note the actual result. Accuracy is then quantified by the percentage of calls that have the same result logged by the manual review and the automated detection.

The second approach to this is more commonly used to score voicemail detection accuracy.  Outbound calls answered by humans often end up being routed to a human agent.  So when an agent receives a transferred call that was actually answered by a machine, they score it accordingly. This means they’re only ever scoring calls that are detected as human. As such, when you hear other providers boast 90%+ detection accuracy rates, this is the method they’re most likely using.  Using this statistic, out of every ten calls they think are human, nine really are. However, they will have no idea how many calls they classified as machine that were really answered by a human. So all this metric means is “when we think it was a human, we were right 90% of the time.” Using this same logic, one could claim 100% accuracy rate by simply classifying all calls as a machine and never picking human at all.

Additionally, testing voicemail detection strictly in a lab setting can also yield overly optimistic scores due to the limited amount of  “real” calls that can actually be run.  To get real-world results, score the calls during a live outbound campaign that reaches a variety of voicemail/answering machines and individuals who answer the phone in their own unique way.

Adding Voicemail Detection to an Outbound Tropo Application

Posted on November 30, 2015 by Phil Bellanti

One of the most useful features for outbound voice call applications is “Call Progress Analysis” (CPA) which can detect whether the call was answered by a human or an answering machine/voicemail system.  Incorporating CPA-based logic in a Tropo application script can allow it to perform different functionality based on if a human or voicemail system answered the call. For example, if the outbound call is detected as answered by a HUMAN, the app can continue with the intended prompts and collect input from the end-user as normal. However, if the outbound call is detected as being answered by a MACHINE (voicemail system or answering machine), it can have some different logic to play a custom audio file specifically for voicemail or to retry the outbound call after a period of time.

How It Works…

This is accomplished using the machineDetection parameter in the call() method. When this function is implemented, the returned object will have the userType property, which returns ‘HUMAN’, ‘MACHINE’, or ‘NIL’ and can be accessed through event.value.userType ($event->value->userType if using PHP). It’s worth noting machineDetection can also return ‘FAX’ as the userType, but since this has become an extreme edge-case in recent years, it wasn’t even incorporated in this particular demo. The userType ‘NIL’ can occasionally pop up on calls answered with a lot of background noise, complete dead silence, or by a non-standard answering machine.  Here in the demo, we’re treating ‘NIL’ calls as answered by MACHINE, however there’s an added step in the application to give people a chance to identify as HUMAN, by requesting them to press ‘1’ before a message geared towards a voicemail system is played. This step also helps eliminate any initial false-positive detections of MACHINE that may arise in an outbound campaign. This demo application is written in Ruby, using Tropo’s Scripting API and can be hosted right on our cloud platform for you.   In case you haven’t already, you’ll first want to register for a free developer account on Tropo.com. From there, follow the steps to create an application as detailed here.  The full demo application script can be found on GitHub.

 

Application Walkthrough

The first part of the script is run when the userType ‘MACHINE’ is returned. In this block, we are further enhancing the machineDetection feature by asking for confirmation in case the call was actually answered by a human (falsely detected as MACHINE), by prompting the end-user to press ‘1’ to hear the message now. Obviously, a machine or voicemail system would ignore that question and the timeout will be triggered. This will take us to lines 15-22, where the application waits for silence after a beep tone. This part is certainly not required for machineDetection, but can help the accuracy when reaching unconventional voicemail machines.


def ask_vm (user)
	ask "Press one at any time to hear your message, or stand by and it will begin momentarily", {
		:timeout => 2,
		:choices => "1",
		:mode => "dtmf",
		:voice => "Tom",
		:onChoice => lambda {|event|
			user = "HUMAN"
			human_answer(user) },
		:onTimeout => lambda {|event|
			beep_listen()
			vm_answer(user) }}
end

def beep_listen ()
	record ".", {
		:beep => false,
		:timeout => 1,
		:silenceTimeout => 1,
		:maxTime => 15,
		:terminator => "#" }
end

The functions in lines 24-32 are run when the userType is returned as ‘HUMAN’. Here, we are asking the end-user if they want to opt-out of future messages, but this can be modified to whatever logic you want to perform when a HUMAN answers the call. Lines 35-38, will be run on calls only when all of the following occur:

  1. The initial userType was detected as ‘MACHINE’.
  2. A timeout on Line 2 was triggered (when the prompt to press ‘1’ is ignored).
  3. The beep and silence was detected from a voicemail system.

Again, 2 and 3 were added in the application just as enhancements to the built-in detection, essentially to help identify irregular voicemail systems with more unique qualities.


def human_answer (user)
	log "value returned from the machine detection  ==  " + user
	say $message , {:voice => "Tom"}
	ask "to opt out of future reminders, please press 1", {
		:choices => "1",
		:mode => "dtmf",
		:voice => "Tom" }
	say "Thank you, good bye" , {:voice => "Tom"}
end

def vm_answer (user)
	log "value returned from the machine detection  ==  " + user
	say $message , {:voice => "Tom"}
end

The settings for the outbound call and logic for machineDetection is in the last portion of code, lines 41-59. The callerID configuration is done in line 42, which sets the phone number to be sent and displayed at the destination being called. This app is using $numberToDial as the variable name to pass in the destination phone number parameter. This is sent along with the token-string to Tropo’s REST API when launching the application. You can find more information on passing in custom parameters to a Tropo application here.


call $numberToDial , {
	:callerID => "14072420001",
	:timeout => 60,Tom
	:machineDetection => {"introduction" => "Hello...", "voice" => "Tom" },
	:onAnswer => lambda {|event|
		if event.value.userType.nil?
			log "This was NIL   "
			user = "MACHINE"
		else
			user = event.value.userType
		end
		if user == "HUMAN"
			human_answer(user)
		else
			ask_vm(user)
		end	}}

hangup	

 

Bringing an API attitude to Cisco

Posted on November 5, 2015 by Adam Kalsey

One reason Cisco acquired Tropo was to bring in our expertise in building developer experiences that people love. Developer experience is more than just beautiful APIs, but extends to all the ways a developer interacts with the APIs and the company or product behind them. From documentation to billing to customer support, a good developer experience requires a conscious effort and doesn’t happen by accident.

I sat down with Beth Schultz, managing editor at No Jitter to talk about what makes for a good developer experience, why it can be hard for large enterprise software companies to create great APIs, and how Tropo is shaping Cisco’s API strategy.

If you’re evaluating a traditional communications product, you can pour over data sheets and find all sorts of information on what the product does and how to use it. You can usually even get access to it and give it a trial run. “But the APIs are often hidden behind NDAs, and you have to talk to a salesperson or somebody in support who has the magic access to give you a key to use the API. Then you might get a PDF document that is hard to read, hard to follow, and often written, created, and managed by engineers, rather than by people who have a [focus on] user experience,” Kalsey said.

When you think about the API from the entire ecosystem, you then consider questions such as: How is it going to fit in with the rest of the products? How is it going to fit in with the rest of the APIs the company develops? How are you going to support it? What is it going to look like?

“So just like you have somebody focused on user experience, you need to have somebody focused on what the developer experience is — and that’s everything from how the APIs look and feel to making sure there’s consistency between them.”

Go read the whole article, and come chat developer experience with us at one of our upcoming events.