Provide for large sample sizes when conducting before/after data collection efforts, to avoid impacting the ability to reveal statistically significant differences during the evaluation's statistical analysis.

Orlando, Florida's experience with a Field Operational Test (FOT) on using a single smart card for transportation payments at facilities operated by multiple regional agencies.

Date Posted
02/23/2006
TwitterLinkedInFacebook
Identifier
2006-L00192

Orlando Regional Alliance for Next Generation Electronic Payment Systems (ORANGES) Evaluation Final Report: Electronic Payment Systems Field Operational Test

Summary Information

In 2000, the US DOT awarded a Field Operational Test (FOT) grant to a group of public sector agencies located in the Orlando region. The project was titled Orlando Regional Alliance for Next Generation Electronic Payment Systems, or ORANGES. The US DOT was interested in identifying and evaluating issues associated with establishing partnerships between public transportation service providers and other transportation agencies, in developing and using multiple-application electronic payment systems that included smart card technology. The FOT requirements were specifically designed to test a payment system that could support a variety of payment applications, at a minimum including transit fare collection, parking payment and electronic toll collection.

As part of the national ITS program, the USDOT requires that each FOT have an independent evaluator. The evaluation is separately funded and has independent goals, objectives, schedules and deliverables. The USDOT evaluations also provide useful feedback to the local FOT participants, as well as other interested transportation stakeholders.

The FOT demonstrated the technical feasibility of implementing a regional smart card with a centralized clearinghouse for multimodal regional transportation payments, where a card issued by any participating agency could be used for payments with (and revalued at) smart card accepting equipment operated by any of the partner agencies.

Lessons Learned

When designing a field study or experiment to investigate the effects of different conditions in a before and after design, there are a number of the issues that the study designers must be aware of to properly design the study. Proper study design will help to ensure that both the before and after data collection efforts yield sufficient data to detect significant differences in the conditions. Several key issues include using sufficient large sample sizes, and ensuring that data collection equipment is functioning correctly. The ORANGES experience revealed that the before and after data analysis found that the 95% confidence intervals for the before and after samples of the performance measures overlapped (i.e., not providing evidence to support a statistically significant change between the before and after periods). Larger sample sizes would likely have decreased the size of the confidence intervals, increasing the opportunities to reveal evidence of a statistically significant change in the performance measures. Therefore, the following guidance is provided.

  • Avoid delays in collecting data, so that data collection problems can be detected early. Collect any data being gathered with unattended equipment frequently, to avoid long periods of data loss in the event of equipment problems. LYNX attempted to gather accumulated Automatic Passenger Counter (APC) data from the equipped buses near the end of the after data collection period (APC data was used to measure before and after average boarding times). Unfortunately, only at that point did LYNX become aware that door sensor malfunctions had prevented APC data collection throughout much of the after data collection period. If the data had been retrieved more frequently (e.g., weekly) the door sensor problems could have been identified and resolved quickly, providing much more data.



The ORANGES evaluation suggests the following guidance for similar future initiatives:

 

  • Plan to collect a substantial amount of before and after data so that the statistical evaluation of the data is more likely to reveal statistically significant differences. The number of observations will depend on the experimental design, the predicted magnitude of the effect, and the expected variance of the observations.

 

  • Plan to include someone with expertise in experimental design and statistics to help design the experiment, create the data collection and analysis plan, and analyze and interpret the data.
  • Include contingency plans that allow for supplementary data collection, if deemed warranted based on the initial statistical assessment of the initially collected data.



Developing a regional smart card payment system is related to the ITS Goal of improving customer satisfaction, through making payments for multimodal travel easier by establishing a regional payment method. The limitations in the ability to draw conclusions on statistically significant differences did not affect customer satisfaction.



The overall project was completed behind schedule. If the project had not been well behind schedule for other reasons, there would have been more flexibility to consider taking time for supplementary data collection.



The participating cardholders generally expressed a positive opinion about the technology, with concerns focusing primarily on the limited scale of deployment. Data collection issues were not a factor that limited the scale of deployment. In fact, a more extensive deployment would have inherently created additional data collection opportunities.



The ORANGES experience has demonstrated that there are a number of important issues in experimental design and data collection to maximize the chances of detecting significant differences associated with the before and after conditions study conditions. These issues include avoiding delays in collecting data, collecting a substantial amount of before and after data, and planning to include someone with expertise in experimental design and statistics. It is also important to include contingency plans that allow for supplementary data collection, if there are problems identified with the primary data collection effort.