Data Accuracy and Modeling Work Plan
SANDAG has launched an initiative to improve data and modeling accuracy and transparency.
To correct and prevent errors, SANDAG is embarking on an initiative to strengthen the transparency, accountability, and accuracy of the agency’s modeling and forecasting processes. SANDAG technical staff has organized into several teams to systematically tackle various reforms and will soon bring in outside experts to review its processes and develop enhancements for consideration by the Board of Directors.
A seven-part work plan, unanimously approved by the Board of Directors on February 24, 2017, will address all of the agency’s data programs, including its Demographic and Economic Forecasting Model (DEFM). DEFM is currently under staff review after a data loading error was found to have affected some of the model’s outputs, including future taxable retail sales estimates that are used in forecasting the financial capacity of the TransNet Extension program. The data, which turned out later to have been flawed, also was used in revenue calculations for a half-cent sales tax measure that went before voters regionwide in November 2016. The measure known as Measure A did not reach the necessary two-thirds vote needed to pass.
In addition, in order to ensure public trust, the SANDAG Board of Directors at its February 24, 2017, meeting voted to undertake an independent examination of who knew what when leading up to the Measure A election. The Board referred the matter to the Executive Committee, which formed a sub-committee during its March 10, 2017, meeting tasked with evaluating potential parties to conduct the examination and making a recommendation to the Board. The final decision on who to engage for the independent examination will be made by the full SANDAG Board, comprising 21 voting members who are elected officials from all 18 cities and county government. Board members have indicated that they would like the examination to be completed expeditiously. For more information see Additional Resources at the bottom of the page.
SANDAG Efforts to Improve Data Accuracy and Integrity
As a taxpayer-funded agency, SANDAG takes the integrity and reliability of its data very seriously. The agency has already taken the following steps:
- Generated an independent forecast for the existing TransNet sales tax program
- Consolidated the chief economist position with the technical services director position to integrate oversight of forecasting processes
- Put in place a new code base, new algorithms, and new source data for the DEFM computer model
The seven-part work plan includes the following elements:
- Conduct Detailed Review: Review the nature of the error in DEFM and its root cause. Review and validate input data, transformations, and equations in the Series 13
forecasting model (the most recent series completed using DEFM) to ensure data accuracy and integrity of the model results. Present this information to the Board of Directors.
- Conduct Dependency Analysis: Identify key SANDAG reports and deliverables that used data from Series 13. Evaluate the significance of the impacts from any potential forecasting errors and the potential effects on findings and policy recommendations. Address issues identified in the analysis.
- Map Modeling Process Flow: In preparation for future forecasts, map all data flow from source through databases, models, and outputs to provide transparency and identify areas for improved quality assurance processes. Complete online documentation and visual mapping of interactions in the model, showing all data sources, processes, interactions, and flows.
- Improve Data Governance: As a first step toward formal data governance, conduct interviews and document the customer-supplier relationship between SANDAG staff and the SANDAG Technical Services Department. This effort will lead to a better understanding of the type of data that agency staff request from Technical Services and a better understanding of how the data are requested, stored, used, and versioned. This information will be used to develop a data warehouse, standardize data extraction routines, and ensure consistency of data.
- Review and Oversight: Validate the new SANDAG population, housing, and economic forecasting model using an independent expert review committee, including convening a panel
of experts in economics, demographics, and land use to review the methods, data sources, and assumptions of the new SANDAG forecasting model. The panel will evaluate the efficacy and sufficiency of the proposed Series 14 forecasting model to adequately forecast population, housing, and economic variables for SANDAG planning purposes.
- Enhance Transparency: Develop a set of agency methods and standards to ensure data and analytic transparency, including establishing check points where full disclosure and analysis are provided to ensure that others can see how models were developed, how data was processed, and what assumptions were made along the way.
- Develop and Formalize Processes: Understand how staff roles, work flows, and technology (e.g., models, databases, software) contribute to producing key agency deliverables. This information will be used to realign the Technical Services Department, as well as add professional quality assurance staff and a dedicated database administrator. This effort will reduce single points of failure, and increase accountability, visibility, and efficiency. Conduct research and prepare an assessment of the current state of software and database platforms to facilitate a plan to implement industry best practices as they relate to data structures, data quality, database design and development, and database governance.
Keeping the Board and Public Informed
SANDAG staff will present ongoing progress reports to the agency Board of Directors and the TransNet Independent Taxpayer Oversight Committee to keep them and the public informed about the implementation of the work plan to improve data accuracy.
This web page will be updated periodically to provide additional information.
What is DEFM?
SANDAG uses a variety of forecasting tools to make estimates for planning purposes. One of them is known as the Demographic and Economic Forecasting Model (DEFM). Originally created in the 1970s, the model has continually undergone minor updates since then. Today, DEFM incorporates data from nearly 700 variables to make forecasts as far out as 2050.
In addition to being used for predicting demographic trends, DEFM also provides data for economic forecasts. One of the outputs from the model – taxable retail sales – is used by SANDAG staff as an input to estimate future sales tax revenues.
Fundamentally, DEFM is a synthesis of two widely used demographic and economic techniques. On the demographic side, the cohort method considers such factors as birth and death rates, and the age, gender, and ethnic distribution of the resident population to arrive at forecasts of demographic variables. On the economic side, time-series/regression methods are used to estimate economic relationships. The resulting econometric equations provide forecasts of employment, income, and other economic variables based on assumptions about national, local/state growth patterns, and local inter-industry relationships.
- RFP for Independent Examination of Measure A Communications (March 16, 2017)
- SANDAG Press Release: Update on Independent Examination (March 10, 2017)
- March 10, 2017, SANDAG Executive Committee Report: Independent Examination of the Measure A Revenue Estimate
- March 10, 2017, SANDAG Executive Committee Meeting Audio
- SANDAG Press Release: SANDAG Board Approves Independent Examination of Forecast Issues (February 24, 2017)
- February 24, 2017, SANDAG Board of Directors Report & Presentation: Proposed SANDAG Data Accuracy and Modeling Work Plan
- February 24, 2017, SANDAG Board of Directors Meeting Audio
- San Diego Union-Tribune Letter to the Editor: In Response: SANDAG takes integrity seriously (February 10, 2017)
- Voice of San Diego Op-Ed: SANDAG Can Still Deliver on TransNet Promises (January 6, 2017)
- December 16, 2016, SANDAG Board of Directors Report & Presentation: TransNet Program and Revenue Forecast