\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage[round]{natbib}
\let\cite\citep




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}










\begin{document}

\title{Term Paper: Model Validation in Structural Econometrics}



\author[1]{Janos Gabler}%
\author[1]{Janos Gabler}%
\author[2]{eisenhauer}%
\affil[1]{University of Bonn}%
\affil[2]{Group_1}%


\vspace{-1em}



  
  \date{May 07, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup








\sloppy




\section*{Introduction}

While model validation is not yet a common practice in structural econometrics, it has a long tradition in other disciplines. Prime examples are engineering \citep{Oberkampf2004}, physics \citep{Roache1998} and climate science \citep{Houghton1992}. Often validation is done by comparing predictions of a model to experimental data that was not used to estimate the parameters of the model. Only recently, some structural econometricians have started to validate their models in a similar way. However, there is no theory and not even a list of best practices that guides economic researchers in doing so. 

This article has two goals: 1) look at other disciplines that have a longer experience with model validation and compile a list of lessons that economists can learn from them. 2) look at econometric papers that claim to do model validation and analyze to what extent the validation fulfills the lessons learned in the first part. A rigorous theory of model validation is far beyond the scope of this short article. However, most best practices are based on common sense and even extremely stylized examples can provide a strong intuition for why they should be followed.

The focus on best practices based on clear intuitions is not only a limitation! After all, validation has the purpose of making policy recommendations derived from an econometric model more credible for economists and non-economists. Credibility is a highly subjective concept and it is likely that the non-experts find a validation based on principles they understand more credible than one that is based on a complex theory \citep{Manski_2012,Manski_2010}.

Throughout the article it will be assumed that the aim of the model is to predict one or several quantities of interest. This holds for models used for ex-ante policy evaluation. The structure of the model is usually not important. Examples could be OLS regressions, structural equation models derived from causal graphs or fully specified life-cycle models. The validation methods presented are not necessarily applicable to models whose main goal is to provide interpretable parameter estimates or to quantify the importance of causal channels that drive a treatment effect.






\section*{Some Lessons from Other Disciplines}

\subsection*{Example}
To illustrate some basic concepts, it is helpful to look at a very stylized example. The example is based on \cite{2012}. 

Several balls are dropped from a tower and the time until they reach the ground is measured. All balls have the radius of a baseball, but their masses range from 3 to 5 kilograms in steps of one kilogram.\footnote{Such dense objects are chosen in order to make the approximation error from using the vacuum formula small.}. The tower has a height of 60 meters, but balls can also be dropped from 20, 30, 40 and 50 meters. From the laws of physics it is known that the drop time $t$ in vacuum is given by:

\begin{equation}
    t=\sqrt{2h/g}
\end{equation}

where $h$ is the height and $g$ is the acceleration number. The functional form of the vacuum formula is believed to be a reasonable approximation of the truth even if the ball is not dropped in vacuum, as long as $g$ is adjusted. In cases other than this toy model, the reason for such a simplification could be computational burden, avoidance of overfitting in a small sample, data limitations or lack of a better model.

Defining $\beta = \frac{2}{g}$, squaring the equation and including an error term that captures unobserved variations in air pressure between drops, leads to:

\begin{equation}
    t^2 = \beta h + \epsilon
\end{equation}

Assume there is data from multiple drops of each ball from each height. Moreover, assume that, as in observational studies, there is no way to obtain more data. Under these circumstances, it is possible to estimate $\beta$ by OLS. $\beta$ is not estimated to back out a value for the true $g$ (known to be 9.8) that would describe acceleration in a vacuum, but to find a number that produces reasonable drop times under real-world conditions.

To account for the fact that the model is wrong, the weight of the ball could be added as additional regressor. In the natural sciences this is called a discrepancy term.\footnote{In linear regressions, economists would call it a control variable, but the concept of discrepancy terms is more general than that.} Alternatively, if enough observations are available, the regression could be run separately for each type of ball. The discrepancy only allows for a linear effect of weight on drop times, but the cost is also low, since only one new parameter is introduced. The separate estimation puts no restriction on functional forms at a cost of more free parameters. This is the classical bias-variance trade-off known from non-parametric econometrics. Which of the two options is better, depends mainly on the amount of data available. 

\subsection*{Limits of Validation}

How can this model be validated? To answer this question, one needs a definition of \emph{model validation}. In \cite{2012}, model validation is defined as follows: ``validation is the process of assessing whether or not the quantity of interest (QOI) for a physical system is within some tolerance -- determined by the intended use of the model -- of the model prediction.'' This leaves open how exactly the assessment happens, but is precise enough to make one thing very clear: ``A simple declaration that a model is "validated" cannot be justified. Rather, a validation statement should specify the QOIs, accuracy, and domain of applicability for which it applies.'' \citep{2012}

This last statement is extremely important, so it is helpful to build a strong intuition for it. How does it come that the model cannot be validated generally? The reason is that the model is wrong. This was known from the beginning! It is a simplified version of reality and was built because it is good enough for some relevant purpose. The model might not be good enough to predict outcomes other than drop time or if accuracy is extremely important (think of ballistic calculations for the military). The limits of validation are summarized in the following lesson:

\textbf{Lesson 1}: A model can always only be validated with respect to a set of policy questions. A well defined set of policy questions specifies 1) a set of admissible input values, 2) the outcome variable of interest and 3)  the required predictive accuracy.

The term policy question sounds a bit unnatural for the ball drop example, but in usual economic applications the input variables are what is changed by a policy. To avoid later confusion, the term is adopted already in this section. Before continuing with the implementation details of validation exercises, it is now important to define two sets of policy questions that will be used during the rest of the discussion. 

\begin{enumerate}
    \item What is the drop time for balls between 3 and 5 kg ball from heights between 20 and 50 meters? $\Rightarrow$ interpolation
    \item What is the drop time for balls between 1.5 and 2.5 kg from heights between 65 and 80 meters? $\Rightarrow$ extrapolation
\end{enumerate}

In both cases, the outcome of interest is the time until the ball hits the ground (the estimated parameter would also allow to calculate speed at the moment of impact and other quantities but predictive quality might be worse for those outcomes).  However, the sets of questions differ with respect to the range of inputs. The policy questions do not state the required predictive accuracy of the model because this is not needed to understand the main insights of this section. In a real application the accuracy should always be stated.

\subsection*{Out of Sample vs Out of Support}

Both sets of questions concern out of sample predictions in the sense that one is not interested in a realized drop time from the estimation sample but in the time it takes a newly dropped ball to reach the ground. However, only the second one concerns out of support predictions. It is intuitive, that interpolation leads to more precise predictions than extrapolation and that at some degree of extrapolation the model will fail to produce reasonable predictions. The degree of extrapolation depends on the support of the data, which can be visualized nicely in a plane.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/fig-1/fig-1}
\caption{{Support from the Example
{\label{355354}}%
}}
\end{center}
\end{figure}

\textbf{Lesson 2}: All forms of validation should assess the goodness of fit out of sample, but for some policy questions it is also important to validate the model out of support.

The main problem that hinders a model from making good out of sample predictions is overfitting to the estimation sample. Overfitting comes in two flavors: 1) Overfitting to realized unobservables and 2) overfitting to the support of the data. 

In the model the only unobservable is $\epsilon$. $\epsilon$ summarizes the effects of wind, fluctuations in pressure, approximation errors. Overfitting to unobservables here means that the estimated value of $\beta$ is chosen such that it does a good job at predicting observed drop times which include the realized values of $\epsilon$. Overfitting to the unobservables happens even if the functional form is completely correct. This is illustrated in the following figure, where the blue line fits the data much better than the true relationship (orange).\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/fig-2/fig-2}
\caption{{Overfitting to Unobservables With a True Model
{\label{323920}}%
}}
\end{center}
\end{figure}

The second kind of overfitting happens if the functional form is only a reasonable approximation to the truth, such as in the simple example above and in many other linear models. Approximations are usually only good in some regions and perform bad for input values in other regions. The following figures illustrate this. Figure three shows an almost perfect in sample fit. Figure four shows a horrible fit when extrapolating beyond the support of the estimation sample.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/fig-3-a/fig-3-a}
\caption{{Excellent in sample fit.
{\label{598190}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/fig-3-b/fig-3-b}
\caption{{Poor out of sample fit.
{\label{415031}}%
}}
\end{center}
\end{figure}

Both kinds of overfitting tend to be stronger for more flexible functional forms and more free parameters and are especially pronounced if the sample size is small. In contrast to overfitting to unobservables, overfitting to the support is not amenable to statistical analysis and not reflected in standard errors. This is because statistical analysis is based on true models and overfitting to the support only happens if the functional forms are only approximating true relationships.


\subsection*{Translating Policy Questions to a Validation Strategy}

The most credible form of experimental validation would experimentally assess the predictive accuracy of the model for each combination of input values that is permitted in the set of policy questions. This is impractical for interpolation questions and impossible for extrapolation questions and would defeat the whole purpose of building a model. Therefore, in practice only a random or non-random subset of the policy questions is validated experimentally in the hope that this inspires confidence for the rest. Since the data used for validation could also be used to estimate the model, it will be called a holdout sample. The choice of the holdout sample depends on the set of policy questions and greatly determines how much credibility can be gained from the validation exercise. The rest of this section briefly describes how the holdout samples could be chosen for each set of policy questions:

The first set of policy questions is concerned with interpolation. This is the prime use case of classical machine learning. To assess the reliability of the model, the parameter is usually estimated on random subsets of the data and the predictive quality is evaluated on the remaining points. If one was very concerned that prediction might be worse at points that were never observed, for example at (3.5, 35), one could deliberately choose a randomization that leaves out all observations at one combination of inputs and use those data points for validation.

The situation is visualized in the following figure\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/fig-4/fig-4}
\caption{{Holdout sample for validation of interpolation
{\label{622284}}%
}}
\end{center}
\end{figure}

The second set of policy questions is extrapolation. The policy extends the support in two directions: heights are greater and weights are lower than any observed value in the estimation sample. To find out whether the model can handle such a situation, the estimation sample could be restricted in a way that generates a holdout sample that leaves the support in the same directions. It would for example not be enough to have a holdout sample in the upper right area of the support, since theory suggests that the predictions become harder for very low weights.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/fig-5/fig-5}
\caption{{Holdout sample for validation of extrapolation in one~ direction
{\label{726475}}%
}}
\end{center}
\end{figure}











\textbf{Lesson 3}: The holdout sample for validation should step outside the support in the same direction as the set of policy questions.

\subsection*{Finding Holdout Samples in Economics}

Most ex-ante policy evaluations step outside the support of the estimation data. For the few cases that don't, a random subset of the data used for estimation can be used as holdout sample. The remainder of this section deals with the more complicated case. 

An inherent limitation of experimental and quasi-experimental papers is that they can only (ex-post) evaluate the effect of exactly one policy in the particular environment in which the policy has been implemented. Many ex-ante evaluations are not concerned with the evaluation of completely new policies but with the simulation of changes to existing policies. A good example is \cite{Todd2006}. They analyze the effect of a conditional cash transfer program. The main design parameters of this policy are the transfer size and the eligibility criteria. The focus of that paper is to ex-ante evaluate the policy for different choices of those parameters in order to find the most efficient transfer schedule. 

It is not possible to answer this set of policy questions with one (quasi-)experiment. Instead, each choice of the design parameters would require a different (quasi-)experiment. In practice only one particular version of the program was tried out in an experiment. The idea of the Todd Wolpin paper is to exploit this one experiment to validate their structural model. The model is estimated on data from the control group of the experiment and, importantly, there is no reason to believe that the model will do particularly well on the implemented parameterization of the policy. The treatment group of the experiment is then used as a holdout sample. By design, this holdout sample is outside the support of the estimation data and it leaves the support in exactly the same direction as the policies of interest. 

Ideally, this procedure allows to overcome the limitations of (quasi-)experimental  methods without sacrificing much of their credibility. For this, it is of course important that the (quasi-)experimental methods don't rely on very strong assumptions themselves. This is already a challenge for policy experiments and even more so for many quasi-experimental methods.













\section*{Review of Papers with Validation Exercises}



\subsection*{Behavioral Model versus Experimentation: The Effects of Housing Subsidies on Rent}

source: \cite{Wise1985}

This is the often cited very first experimental validation of a complex economic model. However, I can't find this paper!



\subsection*{Lumsdaine Stock and Wise Papers}

Unlike the other summaries, this section is based on a series of three related papers by Lumsdaine, Stock and Wise. The sources are \cite{Stock1990}, \cite{Lumsdaine1990} and \cite{Lumsdaine1994}. All three papers estimate similar models of retirement behavior and perform out of sample validation. 

\textbf{Policy Questions}: The outcome variable of interest are retirement decisions. The main input variable is the share of retirement benefits that people receive when they retire at different ages. The model is then used to simulate the effects of so called window plans. A window plan is a temporary increase in retirement benefits. Firms often use such plans in order to increase incentives for early retirement in phases where they want to reduce their labor force. The model can also be used to design optimal window plans.

\textbf{Complexity}: Several models are estimated: 1) a standard discrete choice dynamic programming (DCDP) model that is computationally hard given the computers of that time. Distributional assumptions are made to make the model tractable. 2) An option value model that is very similar to the DCDP model but replaces the Emax function by the maxE function and thus simplifies the computational burden considerably. 3) A static probit model.

\textbf{Data and Identification}: The data for the first two papers come from a single firm. The last paper uses data from several firms. Identification is not discussed explicitly. Since the aim is to make counterfactual predictions under different pension systems, it would be good to observe exogenous variation in benefit height in the data. A window plan could be used for this. However, here the window plan is used for validation and the model is only estimated with pre-intervention data. The hope is that fitting the model such that it explains the current data well leads to parameter estimates that are policy invariant.

\textbf{Type of Holdout Sample}: The holdout consists of observations in the same firm as the estimation data, but after the window plan was implemented. The holdout sample is thus not random and in some cases even contains the same individuals as the estimation sample.

\textbf{Interpolation vs. Extrapolation}: Designing optimal window plans is clearly an extrapolation question. The validation exercise steps outside the support in exactly the same direction as the policy simulation.

\textbf{Evaluation of Distance}: The distance between predicted and actual retirement ages is evaluated informally.

\textbf{Purpose of Evaluation}: Mainly model selection

\textbf{Overall Assessment}: All papers analyse the same questions, using similar models but slightly different data sets. For modern standards, a discussion of identification is missing. However, the validation exercises are very convincing and much ahead of their time. One interesting result is that the computationally much simpler option value models have almost the same predictive validity as the more complex DCDP models. Surprisingly, this insight has not lead to a wider adoption of such models. 


\subsection*{Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility}

Probably the most famous application of experimental validation in structural econometrics is \cite{Todd2006}. The paper evaluates the first Latin American conditional cash transfer (CCT) program. In this program, eligible mothers receive a monthly cash transfer if their children attend school and participate in regular health checkups. 

%Policy Questions
\textbf{Policy Questions}: The outcome variables of interest are school attendance rates at various ages and the percentage of children who complete high school. The policy influences people's decision only via the budget constraint. For eligible households, the budget is enlarged by the transfer amount for each child that complies with the conditionalities. Economically, this is the same as a school subsidy. The policy evaluations broadly simulate transfer heights between 0 and 50 USD per month for children between grade 3 and 9.

%Complexity:
\textbf{Complexity}: The structural model is a discrete choice dynamic programming model (DCDP) that is similar to the classic human capital investment model \citep{Keane1997}. An additional complication is that it explicitly models fertility decisions. The model is estimated by maximum likelihood. This requires the repeated solution of a dynamic programming problem. Computational concerns motivate several assumptions. Examples are that the maximum number of children is 8 or that a child can only attend school if all younger siblings attend school. Both assumptions are known to be wrong but are only violated for a small number of observations. Even after those simplifying assumptions the model is only solved approximately. Given the complexity, it is very likely that approximations and numerical errors influence the result. 

%Data and Surrogate Variation
\textbf{Data and Identification}: The model is estimated on the pre-intervention data of a randomized experiment. The experiment implemented the program with a particular transfers scheme and target group. Since there is no variation in tuition costs, the authors use variation in child wages (opportunity cost of schooling) to identify the policy effect. However, child wages are only observed for children who work. Thus, a Heckman-two-step procedure is applied to estimate the wage distribution. For this to be identified, distance to the next city is used as an exclusion restriction that affects child wage offers but not parental preferences for schooling.

\textbf{Type of Holdout Sample}: The holdout sample used for evaluation is experimental. There was random assignment of 320 treatment villages and 186 control villages. In treatment villages only 50 \% of households were eligible. Eligibility was determined according to a poverty index. Data was collected before the intervention and then in several rounds until about two years after the intervention. The Progresa experiment is one of the first large scale policy experiments in developing countries and is usually described as well executed.

%How far out of support?
\textbf{Interpolation vs. Extrapolation}: The relevant support for the policy questions has two dimensions: 1) child age, 2) transfer size. In the first dimension, only interpolation is needed. For the second dimension, this is hard to assess, since descriptive statistics of child wages are not reported in the paper. It is, however, plausible to assume that some extrapolation is needed. By experimental design, the holdout sample leaves the support of the data in exactly the same direction as the later simulations. The simulated transfer amounts are at most twice as high as the experimentally validated ones, thus there is some extrapolation beyond the validated region. The outcome variables in the validation perfectly match the outcome variables in the admissible set of policy questions.

\textbf{Evaluation of Distance}: The authors conclude that the distance between the experimentally obtained treatment effects with their simulated counterparts from the structural model is small enough to have confidence in the model. They do not state on what the assessment is exactly based, but mention that the simulated treatment effects lie within two standard errors of the experimental ones. However, it has to be mentioned that the standard errors of the experimental treatment effects are quite large. The 95 \% confidence intervals usually contain zero. Unfortunately, no standard errors or confidence intervals are provided for the treatment effects from the structural model. 

\textbf{Purpose of Validation}: Increase credibility of results

\textbf{Overall Assessment}: The paper by Todd and Wolpin respects most good practices of validation. The main weakness is the informal evaluation of distance between the experimental and structural treatment effects and the fact that that the holdout sample contains some individuals that were also used for estimation. Moreover, most of the good practices are fulfilled without formally discussing them. This is not a problem in itself, but as a consequence some relevant information is omitted. An example for this is, that the variation of child wages is not discussed and it is thus hard to say how much extrapolation is needed to answer the main policy questions of the paper. 

\subsection*{Exploring the Usefulness of a Nonrandom Holdout Sample for Model Validation: Welfare Effects on Female Behavior}


\cite{Keane2007} analyses the usefulness of nonrandom holdout samples for model validation. It is mainly a methodological paper in which the empirical results are only of second order importance. It is the first (and only) econometric paper that highlights the importance of using holdout samples outside the support of the estimation data, for cases in which the policy question extrapolates beyond the support of the estimation sample. It does, however, not emphasize that the direction in which the extrapolation steps outside the support should be the same in validation and simulation.

\textbf{Policy Questions}: The outcome variable of interest is welfare participation. The welfare participation is modeled as a function of welfare eligibility, demographic and household characteristics, job offers and other variables. While the paper mentions that the aim is to validate a model that can predict the effects of a change to the welfare regime, it does not have a specific change in mind. It is thus not possible to determine the exact dimensions and directions in which the simulation steps outside the support of the estimation sample.

\textbf{Complexity}: The paper estimates a discrete choice dynamic programming model and four multinomial logit models in order to contrast them according to their predictive ability in a holdout sample. While the logit models do not pose any technical problems, the structural model is highly complex. Therefore, it cannot be estimated by maximum likelihood. Instead, a simulation based estimator is used. It is possible that the complexity of the estimation problem has non negligible effects on the results. 

\textbf{Data and Identification}: The models are estimated with NLSY79 data. The sources of identification differ between models. The structural model exploits variation within and across states. Some of the logit models have state fixed effects and thus can only exploit variation within states. This is problematic for two reasons: 1) this variation is much smaller than the variation between states. 2) Most of this variation is transitory, which should provoke a different reaction than a permanent change if agents behave optimally. 

\textbf{Type of Holdout Sample}: As the title suggests, the holdout sample is nonrandom. The estimation sample consists of California, Michigan, New York (high benefit states), North Carolina and Ohio (medium benefit states). The holdout sample is Texas (low benefit state). 

\textbf{Interpolation vs. Extrapolation}: The paper clearly mentions that the aim is to have a model that predicts well outside the support of the data. It does, however, neither mention in which direction the support is left, nor by how much. 

\textbf{Evaluation of Distance}: The distance is formally evaluated according to a root mean squared error (RMSE) criterion. One problem is that some of the logit models have state fixed effects. To evaluate their performance in Texas, those fixed effects have to be estimated from data in the holdout sample. This contaminates the validation and leads (not surprisingly) to an excellent performance of those models in Texas. This should of course not be taken as evidence for their superiority. To make that clear, the authors show that such models lead to very counterintuitive predictions when simulating reforms in a state from the estimation sample. 

\textbf{Purpose of validation}: Mainly model selection

\textbf{Overall Assessment}: The biggest contribution of the paper is to highlight the importance of out of support validation and to illustrate the usefulness of nonrandom holdout samples for this purpose. The major weakness is, that it does not start from a clear policy question. The validation approach would be perfectly suited for a P2 policy question, i.e. if the aim is to predict what happens if a policy that was implemented in one environment is implemented in a different environment. It is less clear, how such a validation approach helps if the aim is to predict the effect of a policy reform within one state. 

\subsection*{A structural model of sales-force compensation dynamics: Estimation and field implementation}

\cite{Misra2009} analyses optimal payment schemes for sales-persons of a large contact lens manufacturer. The paper first structurally estimates an agent theoretic model under a particular payment scheme to uncover the policy invariant preferences of sales agents. It then uses the estimates to find an optimal payment scheme and simulate changes in revenues. The proposed policy is then actually implemented and the changes in revenue are found to be exactly as predicted. 

\textbf{Policy Questions}: The outcome variable of interest are firm revenue and profit. The potential policy changes are limited to three key dimensions: 1) abolishing quarterly ceilings above which increased sales don't lead to increased payment. 2) going from quarterly to monthly payment horizons. 3) Introduce heterogeneous plans to account for heterogeneity between agents. The main motivation for this limitation is tractability.

\textbf{Complexity}: It would be computationally burdensome to solve the model repeatedly. Therefore, the authors choose a two step estimation procedure that does not require the repeated solution of dynamic programming problems. 

\textbf{Data and Identification}: The data comes from the contact lens manufacturer whose payment scheme is analysed. The sales force comprises 87 agents. Monthly sales data is observed over 38 periods. Since the firm under analysis had no reforms in the payment scheme in this time, identification relies heavily on nonlinearities in the existing payment scheme. Intuitively, a sales agent faces very different incentives at he beginning and end of each quarter, as well as when he is close or far away from his quarterly ceiling. Such differences in incentives help to identify the underlying preferences. 

\textbf{Type of Holdout Sample}: Usually, model validation is done to gain confidence into a model before that model is used for simulation purposes. In \cite{Misra2009} this is different: the model is first used to simulate the effect of policy changes, then the proposed policy is implemented and the effects are found to be close to the predictions. The accuracy of the predictions is especially surprising since the validation is not implemented in a randomized control trial. Thus, the effect of the actual implementation could have been contaminated by other events happening at the same time. 

\textbf{Interpolation vs. Extrapolation}: The policy question is clearly concerned with extrapolation beyond the existing set of policies. By construction the holdout sample leaves the support in exactly the same direction as the simulations.

\textbf{Evaluation of Distance}: The distance is evaluated informally. The key prediction was ans increase in revenues of 8.3 \%. The actual increase turned out to be 9 \%.

\textbf{Purpose of validation}: Analyse ex-post whether the recommendations based on the model worked.

\textbf{Overall Assessment}: The paper by \cite{Misra2009} is a very impressive demonstration of how structural models can predict the effects of policy reforms. Predicted and actual effects are incredibly close. One criticism is that, while a flexible structural model was estimated, it was not actually used to find an optimal policy. Instead, the search space for policy improvements was limited ex-ante to 3 discrete changes as well as combinations of these changes. The motivation for this selection came from intuition and reduced form analysis. The only use of the structural estimates was to quantify the impact of the proposed changes.




\subsection*{The Flat Rental Puzzle}

\cite{Cho2010} analyses price setting of a car rental company. The analysis is motivated by the puzzling observation that rental companies charge the same price for older and newer cars (flat rental puzzle) and replace the cars after only two years (short holding puzzle). Given the very strong depreciation of cars in the first two years of usage and the relatively flat depreciation afterwards some simple economic models would suggest to keep cars longer even if older cars achieved a lower rental price. 

\textbf{Policy Questions}: The paper is very frank about limitations of the model and acknowledges that the model is ``'wrong' in the sense that it is simplified in several key aspects''. Due to these considerations, the authors don't try to find an optimal price policy. Instead they pose the question whether relatively minor changes to the existing price schedule could increase profits. Thus, the outcome variable is firm profit. The key variables changed by the policy are the time (or mileage) after which the company sells used cars and a discount function that depends on the odometer. The discounts range from 10 \% for people that are also eligible for other discounts to 40 \% for other customers.

\textbf{Data and Identification}: This paper is very different from many other structural papers. A standard approach to the policy question would require the structural estimation of demand for cars, based on price and age (or mileage). The data is not sufficient to identify this demand function because there is no observed variation in the pricing schedule. Instead of estimating a full structural model that is capable of simulating a reform to the price schedule, the authors only estimate some model components and make very conservative assumptions on the model components that cannot be identified from the data. As mentioned before, the results are not used to search for an optimal policy, but to provide some empirical underpinning for policy changes that are motivated by economic theory.

\textbf{Complexity}: Due to the very peculiar approach of this paper, it is not very likely that estimation errors that result from the model complexity are of first order importance for the results.

\textbf{Type of Holdout Sample}: Similar to \cite{Misra2009}, the holdout sample comes from an experiment that was implemented according to the authors' recommendation. In contrast to the previous paper, the experimental design involves a control group and thus provides a clean identification of the treatment effect.

\textbf{Interpolation vs. Extrapolation}: The analysed policies clearly require a strong extrapolation, because some key variables do not vary in the data. By construction the holdout sample leaves the empirical support in the same direction as the analysed policies. 

\textbf{Evaluation of Distance}: The experiment is mainly implemented to confirm that the approach chosen in this paper can result in successful policy recommendations. The estimated model cannot simulate the effect of the reform quantitatively. Therefore, no formal evaluation of distance between simulated and experimental outcomes is possible.

\textbf{Purpose of validation}: Analyse ex-post whether the recommendations based on the model worked.

\textbf{Overall Assessment}: The paper is an impressive demonstration of how extremely simplifying assumptions can still lead to good policy recommendations. The approach probably works best if the existing policy is far from optimal. Experimental validation is even more important for this approach than for models that do not suffer from such extreme data limitations. The paper also makes it extremely obvious that the validity (or usefulness) of a model can only be judged with respect to clearly formalized policy questions. The analysis relies heavily on the fact that some assumptions are conservative. However, those assumptions are only conservative in the sense that they lead to an underestimation for the policy questions of the paper. They might very well lead to an overestimation of the effects of many other possible reforms to the existing price schedule. 



\subsection*{A Structural Evaluation of a Large-Scale Quasi-Experimental Microfinance Initiative}

source: \cite{Kaboski2011}

\textbf{Policy Questions}: 

\textbf{Complexity}: 

\textbf{Data and Identification}: 

\textbf{Type of Holdout Sample}: 

\textbf{Interpolation vs. Extrapolation}: 

\textbf{Evaluation of Distance}: 

\textbf{Overall Assessment}:


\subsection*{Incentives Work: Getting Teachers to Come to School}

source: \cite{Duflo2012}

\textbf{Policy Questions}: 

\textbf{Complexity}: 

\textbf{Data and Identification}: 

\textbf{Type of Holdout Sample}: 

\textbf{Interpolation vs. Extrapolation}: 

\textbf{Evaluation of Distance}: 

\textbf{Overall Assessment}:




\subsection*{Yours, Mine, and Ours: Do Divorce Laws Affect the Intertemporal Behavior of Married Couples?}

source: \cite{Voena2015}

\textbf{Policy Questions}: 

\textbf{Complexity}: 

\textbf{Data and Identification}: 

\textbf{Type of Holdout Sample}: 

\textbf{Interpolation vs. Extrapolation}: 

\textbf{Evaluation of Distance}: 

\textbf{Overall Assessment}:




\subsection*{Essays on Development Economics}

source: \cite{Garlick2016}

\ldots



\subsection*{Voting to Tell Others}

source: \cite{Dellavigna2017}

\ldots

\selectlanguage{english}
\FloatBarrier
\bibliographystyle{plainnat}
\bibliography{bibliography/converted_to_latex.bib%
}

\end{document}

