Functions Used in this Lab

You will use the following functions for this lab:

logical operators  ( ==, >, <, ! )
sum()        # count TRUEs in logical vectors
mean()       # proportion of TRUEs in logical vectors
compound statements with & (AND) and | (OR) operators 

Data

This lab uses city tax parcel data, and open dataset released by the municipal government of Syracuse, NY.

• Each house, business, park or empty lot in the city sits on a parcel of land, and the tax rate for that parcel is determined by the city.
• All of the characteristics of each parcel is stored in a central database, along with information about the land use classifications, owners, and whether the property is up-to-date on taxes and water bills.
  
• Each row represents one tax parcel, and all tax parcels in the city are recorded in the dataset.

We will use the downtown area for this lab:

install.packages( "geojsonio" )
install.packages( "sp" )
install.packages( "rgdal" )

library( geojsonio )
library( sp )
library( rgdal )

URL <- "https://raw.githubusercontent.com/DS4PS/Data-Science-Class/master/DATA/downtown-syr.geojson"
downtown <- geojson_read( URL, what="sp" )
plot( downtown,  border="gray50", col="gray80" )

Questions

Challenge Questions

Map all of the land valued at over $1 million per acre.

Map all of the land valued at less than $500,000 per acre.

What is the total value of all of the commercial parcels in the city?

What is the total value of all of the non-commercial parcels in the city?

Submission Instructions

When you have completed your assignment, knit your RMD file to generate your rendered HTML file.

Login to Canvas at http://canvas.asu.edu and navigate to the assignments tab in the course repository. Upload your HTML and RMD files to the appropriate lab submission link.

Platforms like BlackBoard and Canvas sometimes disallow you from submitting HTML files when there is embedded computer code. If this happens create a zipped folder with both the RMD and HTML files.

Remember to:

• name your files according to the convention: Lab-##-LastName.Rmd
• show your solution, include your code.
• do not print excessive output (like a full data set).
• follow appropriate style guidelines (spaces between arguments, etc.).

See Google’s R Style Guide for examples.

Markdown Trouble?

If you are having problems with your RMD file, visit the RMD File Styles and Knitting Tips manual.

Steps in Identifying Groups

For this lab you will construct a group by translating some management question from plain English into a logical statement, then show a map of the newly constructed group by adapting the code provided:

How many parcels are larger than one acre?

these <- downtown$acres > 1   # the group I am defining
sum( these )                  # count of parcels in our defined group

## [1] 59

mean( these )                 # proportion of total parcels in our defined group

## [1] 0.151671

To show the location of this group on a map:

these <- downtown$acres > 1                               # define your group
group.colors <- ifelse( these, "firebrick", "gray80" )    # don't change this
plot( downtown,  border="gray70", col=group.colors )      # don't change this

What proportion of downtown parcels are occupied by commercial buildings?

these <- downtown$landuse == "Commercial"
mean( these )  # 54%

## [1] 0.5372751

Plot commercial parcels on a map:

these <- downtown$landuse == "Commercial"
group.colors <- ifelse( these, "firebrick", "gray90" )     # don't change this
plot( downtown,  border="gray70", col=group.colors )       # don't change this

Identifying Unique Values

We will often define a group using criteria relevant to the management question. For example, we might want to look at all single family homes that are valued over $250,000 in a city. Or perhaps we want to identify all commercial properties that are delinquent on taxes. These are examples of compound logical statements that combine information from two or more variables (land use and value, or land use and tax status).

# single family homes worth more than $250,000
downtown$landuse == "Single Family"   &    downtown$assessedval > 250000

We can find the current category levels defined within factors or character vectors using a table() or unique() function.

table( downtown$landuse )

## 
##          Apartment         Commercial Community Services         Industrial 
##                  6                209                 17                  4 
##            Parking              Parks         Recreation          Religious 
##                 78                  8                  5                  6 
##            Schools      Single Family          Utilities        Vacant Land 
##                  4                  1                  6                 45

unique( downtown$landuse )

##  [1] Parking            Commercial         Parks              Community Services
##  [5] Vacant Land        Utilities          Apartment          Recreation        
##  [9] Schools            Religious          Industrial         Single Family     
## 12 Levels: Apartment Commercial Community Services Industrial Parking ... Vacant Land

Note that the spelling has to be precise for the statement to work correctly.

sum( downtown$landuse == "Apartment" )

## [1] 6

sum( downtown$landuse == "apartment" )

## [1] 0

Missing Values

When there are missing values (NAs) in a vector you need to tell mathematical functions like sum() and mean() to ignore them, otherwise they will return NA.

x <- c( 1, 2, NA, 4 )
sum( x )

## [1] NA

sum( x, na.rm=T )

## [1] 7

Any ideas why R would default to NA for mathematical functions?

Consider an example with a free museum that brings in a special exhibit. They don’t charge when you enter, but if you visit the special exhibit you have the option to make a donation to support the event. The computer logs all visitors, but it only enters values for those that visited the special exhibit.

x <- c( NA, NA, 20, NA, 0, 15, NA, 45, NA, NA, NA, NA, 10, 0, NA, NA )

# how many people visited the museum today?
length( x )

## [1] 16

# how much was donated to the special exhibit? 
sum( x, na.rm=T )

## [1] 90

# what was the average donation made to the special exhibit? 
mean( x, na.rm=T )

## [1] 15

# if we know that 200 people will visit the meseum this weekend,
# how much do we expect the special exhibit to raise? 
#
# people visiting x average donation
200 * mean( x, na.rm=T )

## [1] 3000

Is this a correct estimate?

What average does this represent (who is in the denominator)?

x <- c( NA, NA, 20, NA, 0, 15, NA, 45, NA, NA, NA, NA, 10, 0, NA, NA )
# what was the average donation made to the special exhibit? 
mean( x, na.rm=T )

## [1] 15

When you add na.rm=T to the function it will drop all cases with missing data.

Let’s look at the data again and think about the counts of people more precisely:

# number of people that visit the museum
length( x )

## [1] 16

# number of people that attended the exhibit
sum( x >= 0, na.rm=T )

## [1] 6

# proportion of museum visitors that attended the exhibit
sum( x >= 0, na.rm=T ) / length( x )

## [1] 0.375

# average donation for ALL museum visitors
sum( x, na.rm=T ) / length( x )

## [1] 5.625

# average donation for ONLY THOSE that went to the exhibit
sum( x, na.rm=T ) / sum( x >= 0, na.rm=T )

## [1] 15

Our initial estimate was incorrect. We multiplied all museum visitors by the average donation of those that attended the exhibit, a subset of all visitors. We will greatly over-estimate the expected event revenue as a result.

# if we know that 200 people will visit the meseum this weekend,
# how much do we expect the special exhibit to raise? 
#
# people visiting x average donation per person
200 * mean( x, na.rm=T )

## [1] 3000

To get an accurate estimate of the special event revenues we need to consider how many museum visitors will attend the exhibit:

# accurate estimate:
# people x those at exhibit x ave donation
200 * ( sum( x >= 0, na.rm=T ) / length( x ) ) * mean( x, na.rm=T )

## [1] 1125

# alternatively: 
# number of visitors x ave donation per museum visitor
200 * ( sum( x, na.rm=T ) / length(x) )

## [1] 1125

Missing values serve a very important function in statistics, because they force the analyst to decide whether we can ignore them completely, or whether the need to be interpretted if we want the model to be accurate. So when you get this:

mean( x )

## [1] NA

R is telling you to take a step back to determine how we should treat these values. We can interpret missing values as zero when we tabulate the total of donations:

sum( x, na.rm=T )

## [1] 90

But it is problematic when we are looking at proportions or averages in this context.

Missing values means something different in every context, so pay attention to them when they exist. I guarantee that you will make an error in an important calculation at least once in your career because you ignore missing values.

Compound Logical Statements

We often want to create a new group by combining existing groups:

public.goods  <-  downtown$landuse == "Parks" | 
                  downtown$landuse == "Parking" | 
                  downtown$landuse == "Vacant Land"

mean( public.goods ) # proportion of downtown

## [1] 0.3367609

The is a compound logical statement because there is more than one criteria used to define the group. Compound statements require the use of the OR operator | or the AND & operator. They can also combine criteria from multiple variables in the model.

For example, a brown-eyed girl would be written:

brown.eyed.girl <- dat$gender == "female" & dat$eye.color == "brown"

Logical statements are one of the most useful tools in data, but they are a little tricky at first. Consider the difference between the criteria FEMALE & BROWN EYES vs FEMALE | BROWN EYES.

In a world where eyes is either brown or green and gender is either male or female, these two criteria would return the following:

FEMALE & BROWN EYES

• female + brown eyes
  
• intersection of all females and all people with brown eyes

FEMALE | BROWN EYES

• female + brown eyes
• female + green eyes
• male + brown eyes
• all females in the dataset and all people with brown eyes

Note that in plain English we say “females AND people with brown eyes” when we are combining groups, using the OR operator to create a union of groups.

A good way to keep the logic straight is to add the term “ONLY” to the front of each plain English group descriptor. “ONLY females in the dataset AND ONLY people with brown eyes”.

Also pay attention to the order of operations when applying the NOT operator ! to logical statements. The operator flips each TRUE to a FALSE in a logical vector, and vice-versa each FALSE to TRUE. It creates a compliment set, which is anything not in the current group.

Consider these two cases:

! ( FEMALE & BROWN EYES )

• female + green eyes
• male + brown eyes
• male + green eyes
• the values of brown-eyed girls become FALSE, and everything else is TRUE or part of the new group

! FEMALE & BROWN EYES

• male + brown eyes
• first identify NOT females (all males in the data), then intersect that group with brown eyes

Note that the NOT operator is not equivalent to what some people might consider an OPPOSITE operation. The compliment group of brown-eyed girls is not green-eyed boys! That would be written:

! FEMALE & ! BROWN EYES

If you pay attention to the order of operations, just like mathematical operators, it will help you avoid making errors.