Incorporate mechanisms for capturing user feedback for system evaluation, including the ability to intercept incoming 511 calls for survey or focus group recruitment.

An Arizona Department of Transportation experience in 511 implementation.

Date Posted
05/17/2006
TwitterLinkedInFacebook
Identifier
2006-L00242

Model Deployment of a Regional, Multi-Modal 511 Traveler Information System: Final Report

Summary Information

On July 21, 2000, the Federal Communications Commission assigned 511 as the nationwide traveler information telephone number and granted responsibility for it to government transportation agencies. Since that time, approximately 27 statewide and regional 511 systems have been implemented throughout the United States. In 2002, the Federal Highway Administration (FHWA) and Federal Transit Administration (FTA) jointly issued a request for proposals for participation in a national 511 Model Deployment. The Model Deployment was intended to demonstrate the benefits of 511 systems and generate lessons learned of use to other 511 deployers and the U.S. Department of Transportation (USDOT). The 511 Model Deployment was awarded to an Arizona Department of Transportation (ADOT)-led coalition of transportation agencies in Arizona. This lesson is based on the FHWA evaluation of the Arizona 511 Model Deployment.

Lessons Learned

User feedback is critical to the operation of an effective 511 system and in documenting the benefits of the system, many of which are qualitative. Consequently, deployers of 511 should build into their systems methods for soliciting user input. In the Arizona 511 Model Deployment, two user input mechanisms were included. The first was a caller comment option on the 511 menu: callers could leave messages on a voice mailbox. The second was a call intercept capability: the capability to automatically reroute every nth call (a variable, user defined value) to a survey recruiter. In this evaluation, the intercepted calls were routed to a live survey recruiter who asked the caller if they would be willing to participate in a survey, and if so, scheduled the survey for a time convenient to the caller. The survey was then administered by a live interviewer in a separate, follow-up phone call initiated by the interviewer. Deployers of 511 should be aware of these suggestions from the experience of Arizona DOT:

  • Build user input tools into the system instead of adding them later. Incorporating tools to capture user input from the start is usually more cost-effective than adding them when the system is up and running. Moreover, having the input mechanisms built into the system early allows the operator to easily engage and disengage data collection, including the option of continuously collecting input.
  • Use caller comments to help debug and refine the system. The caller comment feature has proven to be a very useful source of user input. It was especially useful in the first few months following the roll-out of the enhanced system when intensive debugging and refinement of the user interface was occurring. Caller comments were very useful -- both in demonstrating callers sensitivity to what many of them considered to be an inadequately performing user interface and in identifying and correcting specific malfunctions. Utilization of the caller comment featured varied over the first six months of post-enhanced operation from about 0.2% to 0.5% of all 511 calls, with the number of comments ranging from 37 to 373 per month.
  • Use a live intercept survey method to get a higher response rate and a less biased sample of users. Initially ADOT objected to intercepting 511 callers for survey recruitment. They felt that callers would view a brief recorded message inviting them to leave their contact information, or an option to complete an automated survey on the phone, as less disruptive than a live operator intercept of their call. The evaluation team argued that respondents are more likely to agree when invited to participate by a human recruiter. A higher overall response rate in turn reduces sample bias, that is, it mitigates the tendency for only strongly positive and negative callers to participate. ADOT agreed and the live intercept approach was successfully implemented. Approximately 31% of intercepted callers agreed to be surveyed and approximately 71% followed through, yielding 411 surveys and an overall response rate of 22% (that is, 22% of those intercepted completed a survey—a very respectable participation rate.) No complaints were received from intercepted callers regarding the method of recruitment of survey completion.


User feedback is essential in developing a 511 system that is responsive to user needs and preferences. Effectively capturing user feedback is important both for debugging new or enhanced systems and for on-going monitoring of system effectiveness and marketing efforts. Building mechanisms to collect user input into 511 systems as they are being developed is more cost-effective than adding mechanisms later. Two key mechanisms that should be included are a caller comment feature allowing callers to leave voicemail comments and the ability for operators to intercept incoming 511 calls for live recruitment of user survey participants.