** Welcome
*** { @unit = “”, @title = “Overview of the Program Evaluation”, @lecture, @foldout }
Evidence-Based Practices
What does it mean to live in an evidence-based world? How do we become more data-driven?
It turns out that using data to improve decision-making and organizatoinal performance is not a trivial affair because of a little problem called omitted variable bias (correlation does not equal causation). As a result, we need to combine judicious analytical techniques with feasible approaches to research design in order to understand causal impact of social programs.
Here is a great introduction to a case study that uses evaluation to better understant the impact of a government program by getting past anecdotes to measure program impact.
Understanding Causal Impact Without Randomized Control Trials
In most cases we don’t have resources for large-scale Randomized Control Studies. They typically cost millions of dollars, are sometime unethical, and are often times not feasible. For example, does free trade prevent war? How do you randomized free trade across countries?
Statistics and econometricians have spent 75 years developing powerful regression tools that can be used with observational data and clever quasi-experimental research designs to tease out program impact when RCT’s are not possible. The courses in the Foundations of Program Evaluation sequence build the tools to deploy these methods.
- Foundations of Eval I covers the mechanics of regression.
- Foundations of Eval II covers counterfactual analysis and quasi-experimental approaches to research design.
- Foundations of Eval III covers seven regression models used in causal analysis (for example, interrupted time series).
Let’s start with a simple example. Is caffeine good for you?
What evidence is used to create these assertions? [ link ]
Can you conduct a Randomized Control Trial to study the effects of caffeine on mental health over a long period of time? This would require us to demand that some individuals that enjoy coffee not consume it over long periods of time (several months if studying depression, several years if studying things like heart health, diabetes, or cancer), and you force people that don’t like coffee to drink it on a daily basis for years.
As you might expected, an RCT would be challenging. As a result most of our evidence comes from long-term observational studies where participants self-report daily coffee consumption, and these are physical health is measured through regular physician check-ups and self-reported health measures. For example, one of the most important public health studies began in 1976 with a sample of 121,000 nurses and has followed the cohort over 50 years [see the Nurses’ Health Study]. Does evidence from this study represent correlation or causation?
How can we be sure we are measuring the causal impact of coffee on health?
Why is evidence-based management hard?
Just listen to this summary of current knowledge on the topic, then try to succinctly translate it to a rule of thumb physicians should use on whether to recommend coffee to patients.
*** { @unit = “”, @title = “Background Reading”, @reading, @foldout }
Program Impact
This course provides foundational skills in quantitative program evaluation:
Reichardt, C. S., & Bormann, C. A. (1994). Using regression models to estimate program effects. Handbook of practical program evaluation, 417-455. [__pdf__]
The Broader Field of Evaluation
Program evaluation is a large field that deploys a diversity of methodologies beyond quantitative modeling and impact analysis. We focus heavily on the quantitative skills in the Foundations of Eval I, II, and III in this program because data is hard to use, so you need several courses to build a skill set. Qualitative and case-study approaches build from the same foundations in research design, so you can more fully develop some of those skills drawing from knowledge of formal modeling and inference.
For some useful context on evaluation as a field, this short (6-page overview) is helpful:
McNamara, C. (2008). Basic guide to program evaluation. Free Management Library. [__pdf__]
And to get a flavor for debates around approaches to measuring program impact in evaluation:
White, H. (2010). A contribution to current debates in impact evaluation. Evaluation, 16(2), 153-164. [__pdf__]
*** { @unit = “”, @title = “Review of Basic Stats”, @reading, @foldout }
Sampling Distributions
Stats Review
OpenIntro Statistics is an excellent resource for a clear chapter on regression analysis pdf.
- Chapter 8 - Linear Regression
- Diez, D. M., Barr, C. D., & Cetinkaya-Rundel, M. (2012). OpenIntro Statistics.
The Teacups, Giraffes & Statistics site has nice explanations of variance, covariance, and the standard error [ link ].
If you need a refresher on stats, you can try one of the free online statistics courses on Khan Academy, EdEx, Coursera, Udacity or other sites, or visit the OpenStax free Introductory Statistics textbook and review:
- Chapter 1 – Sampling Data
- Chapter 2 – Descriptive Statistics
- Chapter 8 – Confidence Intervals
- Chapter 9 – Hypothesis Testing with One Sample
- Chapter 10 – Hypothesis Testing with Two Samples
- Chapter 12 – Linear Regression and Correlation
*** { @unit = “”, @title = “Build Your Library”, @reading, @foldout }
Reference Material
Regression is a tricky topic to learn because the math is the same across all fields, but statisticians, econemetricians, and other social sciences use very different notation to describe models. Finding textbooks that suits your background and your learning style will help you in the long run.
The field of quantitative program evaluation is like a river with tributaries. The main branch consists of a set of practices around using OLS regression to estimate program impact. There are many sub-fields such as panel models, hierarchical models, Bayesian approaches, generalized least squares… etc etc. These are the tributaries in the field.
Tributaries might also be specific practices like visualization or model diagnostics. You can build up expertise in these areas to get really good at a specific type of model that suits a specific type of research (in Psychology, for example, you would spend a lot of times learning to create and analyze interaction models so that you can deploy fancy factoral designs).
The lecture notes for this class will focus entirely on the principles of basic regression - the main branch on this river. We intentionally try to condense the topics covered so that you can develop an understanding of how a handful of core concepts all fit together (slopes, residuals, standard errors, hypothesis testing, control variables, and bias).
As a result, lecture notes are good for understanding these topics, but will not include all of the relevant background concepts and all of the formulas, not reference more advanced models. It is good to find a textbook that you are comfortable with so you can review concepts and look something up. I have a couple on my shelf that I return to often to look up formulas or read-up on a model I am not familiar with.
If you want to invest in a textbook, these are some texts that present this content using algebra instead of calculus or matrix algebra notations.
- OpenStax free Introductory Statistics textbook
- OpenIntro Statistics pdf, especially CH-08 Intro to Regression
- Discovering Statistics in R as a comprehensive guide to using R for a variety of statistical models
- Real Stats: Using Econometrics for Political Science and Public Policy for material that presents regression models from a causal and evaluation perspective
There is no way to learn and retain all of this material in one course!!! It is completely normal for this material to not click the first time you see it. We will try to repeat concepts and build on them through the semester, and the Evaluation sequence in this program (CPP 523, 524, and 525) provide a lot of reinforcement learning.
You are investing in a skill-set that you will build over time. I will reference material throughout the semester that you can add to your reference library for this purpose.
I will also add books, articles, tutorials and reference sheets to the Resources page, but I suggest you create your own catalog to do the same!
** R Reference Material
*** { @unit = “”, @title = “The R Toolkit”, @reading, @foldout }
Introducing R
The R Toolkit
In this course we cover the foundations of data programming with the R language. In order to create robust and dynamic analysis we need to use a couple of tools that were built to leverage the power of R and create compelling narratives. R Studio helps you manage projects by organizing files, scripts, packages and output. Markdown is a simple formatting convention that allows you to create publication-quality documents. And R Markdown is a specific version of Markdown that allows you to combine text and code to create data-driven documents.
Installing R
You will need to install R (the open-source analytics platform used in this course) and R Studio (the graphical user interface for R).
R Markdown
Getting Started with R Markdown
You will have plenty of practice with these tools this semester. You will submit your labs as knitted R Markdown (RMD) files.
*** { @unit = “”, @title = “Tour of R Studio”, @lecture, @foldout }
Getting to Know RStudio
RStudio is a graphical user interface (GUI) and integrated developer environment (IDE) that makes it much easier to use R for writing code, importing and exporting data, installing extensions, and many other features.
R Studio Walk-Through
Use the bookmarks in the video description on YouTube to skip ahead to different parts of the tutorial.
Content:
- Opening a New Script: 00:31
- Saving Scripts: 01:35
- Sessions & Setting Working Directories: 04:13
- The Source/Script Pane: 06:15
- The Console Pane: 09:12
- The Environment, History, & Connections Panes: 12:04
- Importing Data: 12:26
- Objects & the Environment Pane: 14:26
- The Files, Plots, Packages, Help Panes: 16:16
- The Help Pane: 18:04
- Keyboard Shortcuts: 23:48
- The RStudio Cheatsheet: 29:07
- Customizing Your Interface: 30:20
*** { @unit = “”, @title = “The Data Science Ecosystem”, @lecture, @foldout }
The Data Science Ecosystem
R is a foundational tool within a toolkit that I will refer to as the “data science ecosystem”.
If you were not able to make either Zoom session, I did a brief introduction to the “ecosystem” - the community of people that are creating cool analytical tools and building tutorials and case studies for how they might be applied, as well as a core set of tools that are all designed to work nicely together in order to implement projects.
You can think of R, R Studio, and Markdown kindof like Excel (analysis), Word (report-writing), and Power Point (presentations). R allows you to analyze your data, but these results are not useful unless you can share them with others. Here is where data-driven documents developed using R Studio and Markdown really shine. You can quickly package your R code as cool reports, websites, presentations, or dashboards to format the information in whatever way is most accessible and useful for your clients or stakeholders.
The R Language
R is a data programming language developed for statistics.
We will be using R for lectures and labs in this course. If you are only taking the program evaluation course, you only need a rudimentary understanding of R. The code you need to run regressions and create tables will be provided to you. All of the datasets we will use for lectures and labs are pre-packaged, so you don’t have to know how to build datasets in R.
You will see, however, that I provide a lot of code in lecture notes. This is primarily for the students also taking the Intro to Data Science class (CPP 526) or enrolled in the masters program in data analytics. The more code you see, the more familiar it will become. It also let’s you build off of existing code and incorporate it into your own projects. It is presented as a resource, not because you are expected to be able to make sense of it all of it or use it in labs, rather because the courses are designed to be immersive.
Instead of isolating lectures and labs in a university learning platform, this content is presented in the formats and environments you will might encounter in professional analyst role.
*** { @unit = “”, @title = “Getting Help”, @assignment, @foldout }
Getting Help in this Class
Nerds are stereotypically perceived as being anti-social, but in reality data analytics is a very social activity and real-world analytics projects are almost always collaborative.
Learning how to seek help and use discussion boards will accelerate learning and facilitate collaboration. Social coding tools like GitHub use these features extensively.
We are going to throw a lot at you, but also provide a lot of support. Over these first couple of weeks feel free to reach out for anything you might need.
If you find something confusing let us know (likely others will find it confusing as well).
- We can jump on a Zoom call to do a screen share if you want to walk through anything.
- You can post a question to the homework discussion board for data science I or program evaluation I.
- Or you can request a walk-through of a problem. The instructor will provide an example with the solution.
As a general rule of thumb, if you are stuck on the math or code for a problem, need clarification about what the question is asking, want to make sure you understand a formula, or are having similar issues then the help discussion page is the easiest and quickest way to get help. If you are confused about concepts or having a hard time even formulating your question then virtual office hours are your best option.
Note that the discussion board is hosted by the GitHub issues feature. It is a great forum because:
- You can format code and math using standard markdown syntax.
- You can cut and paste images directly into the message.
- You can direction responses using @username mentions.
Please preview your responses before posting to ensure proper formatting. Note that you format code by placing fences around the code:
```
# your code here
lm( y ~ x1 + x2 )
```
The fences are three back-ticks. These look like quotation marks, but are actually the character at the top left of your keyboard (if you have a US or European keyboard).
GitHub does not have a native math rendering language (RMD documents, on the other hand, support formulas). So you have two options, type formulas as regular text and use code formatting to make them clear (this option is usually sufficient). Or you can type your formula in a formula editor and copy and paste an image of the nicely-formatted example.
```
y = b0 + b1•X1 + b2•X2 + e
b1 = cov(x,y) / var(x)
```
These courses are designed to be interactive, and a lot of learning occurs by practicing the technical jargon from the field and learning how to talk about data and models.
Please reach out if you have questions or feel stuck!
PRACTICE:
Navigate to the Help! Board for the course. Click on the practice post, and reply by telling the class your name and your favorite mathematical formula.
** Course Cadence
*** { @unit = “”, @title = “Course Cadence”, @reading, @foldout }
Course Cadence
Please note the organizatin of assignments for class.
Labs are due Friday, mini assessments covering the previous lab are on Tuesday.
Get started early on labs so you have time to ask questions on the discussion board if you get stuck.
Labs are graded pass-fail. The criteria is that you demonstrate an understanding of the topics covered, which is typically operationalized as getting over half of the questions on the assignment correct.
The mini assessments typically consist of three multiple-choice or matching questions to guage your understanding of the previous week’s lectures and lab. If you understood everything on the labs you should do fine on the mini assessments.
Mini-assessments open on Saturday and close Tuesday at 11:59pm (AZ time). You have 30 minutes to complete an assessment once they start. You can take it at any time Sat-Tues.
** Week 1 - Nuts and Bolts of Regression Models
*** { @unit = “”, @title = “Unit Overview”, @reading, @foldout }
Description
This section provides a review of the basic building blocks of a bivariate regression model:
- sample variance and standard deviation
- slope
- intercept
- regression line
- the “error term” or “residual”
- standard errors
Learning Objectives
Once you have completed this section you will be able to conceptually understand what a regression slope represents (the conditional mean), the formulas for regression coefficients, and the tabulation of residuals.
Assigned and Recommended Articles or Chapters
Required:
Reichardt, C. S., & Bormann, C. A. (1994). Using regression models to estimate program effects. Handbook of practical program evaluation, 417-455. [ pdf ]
For reference:
Diez, D. M., Barr, C. D., & Cetinkaya-Rundel, M. (2012). OpenIntro statistics (pp. 174-175). OpenIntro. [CH-08 Introduction to Linear Regression]
Multiple Regression overview chapter [pdf]
Lab Preview
Lab-01 Regression Review covers the following topics:
- Regression slopes
- Intercept
- Residuals
- Explained variance
Reference material for the lab:
Lewis-Beck, C., & Lewis-Beck, M. (2015). Applied regression: An introduction (Vol. 22). Sage publications. [pdf]
*** { @unit = “”, @title = “Regression Review”, @lecture, @foldout }
Test your baseline knowledge of regression models model with the regression review handout.
This review will be useful to benchmark your knowledge about regression models, and offers a good reference for the concepts that you should focus on over the first three weeks of class. On the final exam you will be expected to demonstrate an understanding of:
- Variance and covariance
- The sampling distribution
- Standard deviation
- Standard error
- Confidence intervals
- Slope
- “Explained” variance
*** { @unit = “”, @title = “Building a Regression Model”, @lecture, @foldout }
Lecture Materials
Is caffeine good for you? [ link ]
Is caffeine a treatment in this study? How do we know caffeine is the cause of the outcomes we see in this study?
Before we can understand causal impact we must first create a regression model that tells us about the RELATIONSHIP between caffeine intake and heart rate. We will then add nuance to our understanding of when the relationship can be interpretted as casual, and when it is simply correlational.
Lecture: [ Building a regression model ]
Data Used in this Section
Caffeine and Heart Rates based off of this caffeine study:
# paste this code into R
url <- "https://raw.githubusercontent.com/DS4PS/cpp-523-spr-2020/master/lectures/data/caffeine.csv"
dat <- read.csv( url, stringsAsFactors=F )
summary( dat )
plot( dat$caffeine, dat$heart.rate )
model.01 <- lm( heart.rate ~ caffeine, data=dat )
summary( model.01 )
*** { @unit = “FRI Jan-17”, @title = “Lab 01”, @assignment, @foldout }
Lab 01 - Regression Review
This lab will review the following topics:
- Calculating a bivariate regression slope (
b1) - Basic interpretation of the slope
- Intercept (
b0) - What is a residual?
- residual (or error) sum of squares
- regression (or explained) sum of squares
- R-squared: the measure of variance explained
This first lab is meant as a review of some important regression formulas to either shake out the cobwebs if you have covered this material before, or get everyone on the same page if it is new.
The lecture notes needed for the lab are available on the course shell:
[ Building a regression model ]
And the first chapter from Lewis-Beck serves as a reference for specific formulas if needed:
Lewis-Beck, C., & Lewis-Beck, M. (2015). Applied regression: An introduction (Vol. 22). Sage publications. [ pdf ]
This first lab is designed as a review (or getting up to speed) assignment to direct your attention on a handful of important regression formulas we will use this semester. For the most part we will rely on the computer the do the math for us, but these formulas are important for our conceptual understanding of the regressions, so there is value in working through a simple example (five data points) by hand.
For THIS ASSIGNMENT ONLY all of the work is done by hand so you can type your answers right into the Lab 01 word document:
Please show the steps for each calculation. You can check your results in R or on a calculator when you are done.
Save it using the naming convention:
Lab-##-LastName.doc
And submit via Canvas.
Lab 01 is due Friday. You will, however, have a full week for subsequent labs.
For future assignments we will be using R Markdown documents so that you can run models and submit the results directly.
If you have questions, please post them to the Assignment Discussion Board.
** Week 2 - Effect Size and Confidence Intervals
*** { @unit = “”, @title = “Unit Overview”, @reading, @foldout }
Description
This week covers the topic of building confidence intervals around our estimates of program impact. We use the CIs to conduct hypothesis-testing to see if our program has the impact we expected.
- Confidence intervals are built using standard errors
- Standard errors are creaed from residuals
- Residuals are generated through regression models (as we learned last week).
Learning Objectives
Once you have completed this section you should be able to build a confidence interval around a slope estimate of program impact, and interpret a table with several regressions.
Assigned and Recommended Articles or Chapters
There are no assigned readings this week.
Key Take-Aways
The lecture notes in this section cover the mechanics of standard errors and confidence intervals. These two important topics can be summed up in these animations of the sampling distribution of the mean.
Our model estimate for the slope is our best guess of the real statistic. It will always be pretty good, but not exact.
If we would repeatedly draw samples from a population and calculate slope estimates over and over, they would look like the distribution on the right. The “standard error” describes the average amount all of these guesses (statistics) are off from the true slope.
If we create 95% confidence intervals around these guesses, we can see that approximately 95 out of 100 of the CIs will contain the true slope (set to 3 here).
*** { @unit = “”, @title = “Regression Mechanics”, @lecture, @foldout }
Lecture Materials
Please review the following lecture notes:
- Variance and Covariance
- Partitioning the Variance of Y
- Standard Errors
- Confidence Intervals
- Effect Size
You might find this summary of notation helpful pdf
*** { @unit = “TUE Jan-21”, @title = “Mini-Assessment 01”, @assignment, @foldout }
Mini-Assessment 01
This mini-assessment consists of three multiple-choice questions. You have thirty minutes to complete from the time you start the assessment.
The assessment covers the definition of a regression line, and the concept of sums of squares from the first chapter:
*** { @unit = “FRI Jan-24”, @title = “Lab 02”, @assignment, @foldout }
Lab 02
This lab introduces a case study on education policy that we will be using for the remainder of the semester to demonstrate the importance and impact of control variables, and consequences of their omissions.
You will need the formula for confidence intervals, and the concept of visual hypothesis-testing with coefficient plots.
When you are complete:
** Week 3 - Control Variables
*** { @unit = “”, @title = “Unit Overview”, @reading, @foldout }
Description
This lecture introduces you to two distinctive types of control variables, and how the different ways they change our models. We will also start using Ballentine Venn Diagrams to compare models.
Learning Objectives
Once you have completed this section you will be able to explain how adding specific control variables to a model (or leaving them out) will impact (1) the slopes and (2) the standard error of your model.
Assigned and Recommended Articles or Chapters
Required:
“Bivariate regression” means 2 variables:
Y = b0 + b1•X + e
“Multiple regression” or “Multivariate regression” means one DV and multiple independent variables:
Y = b0 + b1•X1 + b2•X2 + e
In the lecture notes X1 will always be the policy variable, or intervention of interest. The additional variables X2, X3, X4, etc. are all “control variables” that are meant to improve models.
Skim: Multiple Regression
Recommended:
Baily CH5: Section 5.1-Multivariate OLS
Bailey, M. A. (2016). Real Stats: Using Econometrics for Political Science and Public Policy. Oxford University Press.
*** { @unit = “”, @title = “Control Variables”, @lecture, @foldout }
Lecture Materials
Lecture 06 on Control Variables
Nice explanation of the importance of control variables as competing hypotheses:
Tufte, E. R. (1974). Data analysis for politics and policy. Prentice Hall. CH4 Multiple Regression
*** { @unit = “TUE Jan-28”, @title = “Mini-Assessment 02”, @assignment, @foldout }
This mini-assessment tests your understanding of the interpretation of statistical significance using confidence intervals.
*** { @unit = “FRI Jan-31”, @title = “Lab 03”, @assignment, @foldout }
Lab 03
This lab introduces the important role of control variables in our models. They can be used to reduce standard errors of the model and thus increase model efficiency, or they can be used to to adjust slopes in the model. The lab draws on material from the lecture 06 notes.
I have provided the code for the regression models and scatterplots. This lab focuses on interpretation of results. You do need to create an RMD document, and knit an HTML file to submit your results.
When you are complete:
** Week 4 - Omitted Variable Bias
*** { @unit = “”, @title = “Unit Overview”, @reading, @foldout }
Description
This week introduces the concept of omitted variable bias, and how it can impact our inferences in observational studies.
Learning Objectives
Once you have completed this section you should be able to:
- identify variables the have the potential to cause omitted variable bias
- calculate the size of bias that results from omitting a variable from a study
Assigned and Recommended Articles or Chapters
Required:
‘Crack baby’ study ends with unexpected but clear result [ link ]
- A 1989 study in Philadelphia found that nearly one in six newborns at city hospitals had mothers who tested positive for cocaine. Troubling stories were circulating about the so-called crack babies. They had small heads and were easily agitated and prone to tremors and bad muscle tone, according to reports, many of which were anecdotal. Worse, the babies seemed aloof and avoided eye contact. Some social workers predicted a lost generation - kids with a host of learning and emotional deficits who would overwhelm school systems and not be able to hold a job or form meaningful relationships. The “crack baby” image became symbolic of bad mothering, and some cocaine-using mothers had their babies taken from them or, in a few cases, were arrested.
The study in the article was commissioned because of anedcotal evidence of a strong relationship between exposure to crack in the womb and poor development of a child, cognitively and socially.
If crack was the policy variable in this study, what was the omitted variable?
How did the understanding of the DIRECT impact of crack on child development change once the omitted variable was added to the models?
When are omitted variables a problem? What makes a variable a competing hypothesis?
*** { @unit = “”, @title = “Omitted Variable Bias”, @lecture, @foldout }
Lecture Materials
Lecture Notes: Omitted Variable Bias [ pdf ] [ example ]
Lecure Notes: A Taxonomy of Control Variables [ pdf ]
*** { @unit = “TUE Feb-04”, @title = “Mini-Assessment 03”, @assignment, @foldout }
This mini-assessment tests your understanding of the differences between the two types of control variables (those correlated with the policy or intervention, i.e. classroom size and socio-economic status, and those that are uncorrelated with the policy but correlated with the outcome, i.e. teacher quality).
These differences are summarized on some of the review notes for the final exam:
*** { @unit = “FRI Feb-07”, @title = “Lab 04”, @assignment, @foldout }
Lab 04
This lab examines the impact of omitted variable bias on our inferences.
Example of How to Calculate Bias [ PDF ]
A lab solutions RMD template has been provided. Submit your knitted files via Canvas.
When you are complete:
Omitted Variable Bias Calculations:
What happens when we omitt SES from the Classroom Size model?
# full regression: TS = B0 + B1*CS + B2*SES
# naive regression: TS = b0 + b1*CS
Recall when *B1 is used for the slope it represents the correct slope and b1 represents the slope that comes from an incomplete model or from a small sample and thus will likely be biased.
Calculations for bias:
URL <- "https://raw.githubusercontent.com/DS4PS/cpp-523-fall-2019/master/labs/class-size-seed-1234.csv"
dat <- read.csv( URL )
# naive regression in the example: TS = b0 + b1*CS
m.naive <- lm( test ~ csize, data=dat )
summary( m.naive )
# Coefficients:
# ----------------------------------
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) 738.3366 4.8788 151.34 <2e-16 ***
# csize -4.2221 0.1761 -23.98 <2e-16 ***
# ----------------------------------
# full regression: TS = B0 + B1*CS + B2*SES
m.full <- lm( test ~ csize + ses, data=dat )
summary( m.full )
# Coefficients:
# ----------------------------------
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) 665.289 76.574 8.688 <2e-16 ***
# csize -2.671 1.632 -1.637 0.102
# ses 16.344 17.098 0.956 0.339
# ----------------------------------
# auxiliary regression to get a1: SES = a0 + a1*CS
m.auxiliary <- lm( ses ~ csize, data=dat )
summary( m.auxiliary )
# lm(formula = ses ~ csize, data = dat)
# Coefficients:
# ----------------------------------
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) 4.469458 0.009033 494.8 <2e-16 ***
# csize -0.094876 0.000326 -291.0 <2e-16 ***
# ----------------------------------
# b1 = B1 + bias
# b1 - B1 = bias
b1 <- -4.22
B1 <- -2.67
b1 - B1
# bias = a1*B2
a1 <- -0.0949
B2 <- 16.34
a1*B2
** Week 5 - Dummy Variables: Revisiting Hypothesis Tests
*** { @unit = “”, @title = “Unit Overview”, @reading, @foldout }
Description
This week introduces modeling group differences within the data using dummy variables and interaction terms.
Dummy variables are binary 0/1 variables where 1 means the observation belongs to a group, 0 means the observation does not. We need one dummy variable for each level of a categorical variable.
y = b0 + b1(X) + b2(D) + e
Interactions are created by multiplying a covariate by a dummy variable.
y = b0 + b1(X) + b2(D) + b3(X)(D) + e
Adding dummy variables to models allows us to test several hypotheses about differences between groups.
Learning Objectives
Once you have completed this section you will be able to run and interpret regression models with dummy variables and interaction effects.
*** { @unit = “”, @title = “Adding Groups to Regression Models”, @lecture, @foldout }
Assigned Lecture Materials
Hypothesis-Testing With Groups Part-01
Hypothesis-Testing With Groups Part-02
Lab Preview
*** { @unit = “TUE Feb-11”, @title = “Mini-Assessment 04”, @assignment, @foldout }
This mini-assessment tests your understanding of the interpretation of the impact of control variables on statistical significance.
Note that the lecture on control variables covers cases where we add variables to a model:
Y = b0 + b1 X1 + e # add controls
>>>
Y = b0 + b1(X1) + b2(X2) + b3(X3) + e
And the lecture on omitted variable bias is the exact same concepts, but now in reverse:
Y = b0 + b1(X1) + b2(X2) + b3(X3) + e # full model
>>>
Y = b0 + b1 X1 + e # omit variables
We are interested in how these operations impact (1) the slope of our policy variable and (2) the standard errors (i.e. confidence intervals) of our policy variable. Statistical significance is determined by the combination of slope and SE.
*** { @unit = “FRI Feb-14”, @title = “Lab 05”, @assignment, @foldout }
Lab 05
This lab examines tests your understanding of constructing groups and conducting tests using dummy variables.
When you are complete:
** Week 6 - Specification
*** { @unit = “”, @title = “Unit Overview”, @reading, @foldout }
Description
This week introduces the concept of specification bias - various idiosyncracies of regression models such as outliers and functional forms that can introduce bias into the slope estimates.
Learning Objectives
Once you have completed this section you will be able to:
- Diagnose specification bias when it occurs
- Be able to identify outliers and conduct sensitivity analysis to check their impact
- Specify a quadratic model for non-linear relationships
- Interpret a logged regression
*** { @unit = “”, @title = “Specification Bias”, @lecture, @foldout }
Lecture Materials
Specification Bias I [ html ]
Specification Bias II [ pdf ]
Lab Preview
For this lab you will run and interpret a quadratic regression, and interpret coefficients from a logged regression model.
*** { @unit = “FRI Feb-21”, @title = “Mini-Assessment 05”, @assignment, @foldout }
This mini-assessment tests your understanding of the interpretation of dummy variables in regression models.
# test of group means
y = b0 + b1(female) + e
# test of slope differences
y = b0 + b1(X) + b2(female) + b3(female•X) + e
*** { @unit = “FRI Feb-21”, @title = “Lab 06”, @assignment, @foldout }
Lab 06
This lab gives you a chance to practice non-linear regression models.
When you are complete:
** Week 7 - Review!
*** { @unit = “”, @title = “Review”, @lecture, @foldout }
Please post questions on the discussion board if you are not sure of the correct solutions for questions on lectures or practice exams.
*** { @unit = “”, @title = “The Seven Deadly Sins of Regression”, @lecture, @foldout }
Concise summary of the seven most common issues that will corrupt your regression inferences [ pdf ]
- Selection
- Omitted variable bias
- Measurement error
- Specification
- Group Heterogeneity
- Multi-collinearity
- Simultenaeity
Some longer notes: Seven Sins of Regression [ pdf ]
*** { @unit = “”, @title = “Measurement Error”, @lecture, @foldout }
The impact of measurement error
Measurement Error in the DV
Increase in standard errors. No slope bias.
Measurement Error in the IV
Attenuation of the slope (tilts toward zero), but also a decrease in standard errors that offset the shift of the slope toward zero.
*** { @unit = “TUE Feb-25”, @title = “Mini-Assessment 06”, @assignment, @foldout }
Mini Assessment-06
This mini assessment covers material from the Taxonomy of Controls.
And on Measurement Error. See the animated simulations below for examples of the effects of measurement error in the DV versus IVs. And see questions 3 and 4 on the review sheet for the same examples explained as Venn diagrams.
This mini assessment should give you an idea of the types of questions you might expect on the final exam.
The impact of measurement error
Measurement Error in the DV
Increase in standard errors. No slope bias.
Measurement Error in the IV
Attenuation of the slope (tilts toward zero), but also a decrease in standard errors that offset the shift of the slope toward zero.
** FINAL EXAM
*** { @unit = “FRI Feb-28”, @title = “Review Session”, @lecture }
There will be a review session for the final exam at TBD. It is a chance to bring questions from the practice exams and discuss material before taking the exam.
Please post questions on the discussion board, or email me for a Zoom call if you want help before the exam.
*** { @unit = “Open Feb 29 to March 02”, @title = “Final Exam”, @assignment }
You can start the exam any time between Saturday, February 29th and Monday, March 2nd. You have 4 hours to complete the exam once it is started.
You are allowed to look back over your notes and use a calculator for help with math.
You are NOT allowed to work together on the final exam. Do not discuss questions after you have take it.
[add button here ]