Results using archived data at five study locations with a variety of seasonal traffic patterns
Nationwide, United States
Using Incomplete Archived ITS Data to Calculate Annual Average Traffic Statistics: How much missing data is too much?
Summary Information
Many traffic management centers (TMCs) have begun to archive traffic monitoring data for a variety of uses, including:
- Calculation of traffic count statistics and adjustment factors.
- Development of congestion or performance monitoring programs.
- Decision support for operations and maintenance (e.g., incident detection algorithms, ramp meter timing, work zone planning, etc.).
Some transportation planning and traffic operations/ITS groups are cooperating on data sharing initiatives, including the use of archived ITS data for planning statistics. The conventional procedures for the calculation of annual average planning statistics as presented in the American Association of State Highway Transportation Officials (AASHTO) 1992 Guidelines for Traffic Data Programs require a certain level of data completeness. For example, the 1992 AASHTO Guidelines require that a full day of traffic data be available for each day of the week and month of the year in the calculation of annual averages. That is, 84 daily traffic counts (monthly average days of the week, MADW) must be 100 percent complete, with no imputation permitted for incomplete days.
This data completeness requirement is not often met by archived ITS data, because the nature of archived ITS data is that small periods of data are typically missing throughout nearly all days. For example, in an archived ITS data set in which five percent of the data is missing for each day of the year would not be acceptable because there are no days that are 100 percent complete and no imputation is permitted. The paper tested the effects of various missing data patterns on several existing and modified annual average statistic calculation procedures.
This paper addresses the issue of using incomplete archived ITS data in planning statistics. Archived ITS data can be incomplete for a variety of reasons, including communication interruptions, sensor malfunctions, equipment maintenance, road construction, and software or hardware failures.
Five locations were chosen for the study: two urban interstate highways with significant commuter traffic, one urban parkway with commuter traffic and recreational trips, and two rural roads with pronounced seasonal patterns. A complete year of data was obtained for each study location, then missing data patterns (as identified from an empirical study) were simulated by randomly or systematically removing data.
FINDINGS
- At urban locations in which commuter traffic dampens seasonality patterns, a significant amount of missing data (up to 8 months of consecutive missing data) can be tolerated with little to no effect on annual average traffic statistics. All of the calculation procedures provided similar results at the urban locations.
- For the two rural locations, it appears that a month of missing data still results in tolerable error for most procedures. Two months of missing data is close to or exceeds tolerable error levels for most procedures.
- The authors conclude that systematically missing data up to 50 to 75 percent of all days from operations based detectors in urban locations with weekday commuter traffic is not fatal flaw for the purposes of calculating annual average traffic statistics. That is, up to 75 percent of all days can be missing data at urban locations, and the errors in AADT estimates will still be less than 5 percent.
- The authors conclude that modifications of conventional calculation procedures (that account for small gaps in data on a daily basis) are better suited to archived ITS data.
