Zoning and its impact on the housing market

By Robert Bangiyev

image-2.png

Overview

Hypothesis:

As zoning laws are made more stringent, the amount of new buildings constructed will fall, and as the amount of new buildings constructed declines, and population rises, the cost of housing will climb.

Hypothesis Testing:

The hypothesis was tested using data on the population of NYC over time, data on all the buildings which were constructed in NYC, and data of property sales in NYC. Pandas was used to read, clean, and filter the data. Rows containing missing values in important columns were dropped, and rows were grouped by a certain column to retrieve the desired DataFrame. Matplotlib was used to create visualizations showing the change in property prices, buildings constructed, and population over time. Bar graphs were created using matplotlib to show the number of buildings constructed in each borough's zoning district type, and line graphs were used to visualize the change in population, sale price, and building construction over time. If within the same time frame, for each of the five boroughs, the population was shown to be increasing, the number of buildings being built was shown to be decreasing, and the average price of property in that borough was shown to be rising, then a relationship between population, building construction, and property sale price would be established. Furthermore, if in each borough, the number of new buildings constructed were to fall after large zoning regulations were introduced, then stringent zoning regulations would be revealed to have a negative impact on the housing supply.

Datasets

  1. Buildings Built in NYC History

This dataset contains a csv file for each of the five NYC boroughs, Queens, Manhattan, Bronx, Brooklyn, and Staten Island. Each entry in the dataset is a building that was constructed in NYC history. Each building has a set of features, including school district, zip code, and address, but the most relevant columns are ZoneDist1, which is the zoning district classification of the area the building was built on, and YearBuilt, the year the building was built. The dataset contains data going as far back as the 1800s for some of the boroughs. However, it contains a non-negligible number of rows where the value for YearBuilt is 0. Since NYC did not exist when the year was 0, one can assume that this is just a placeholder for missing values, which would have an impact on the results if not dealt with.

  1. Annual Sales

This is not one dataset, but many, spread out over many excel files. There is one excel file for each borough, for each year, from 2003 to 2020, for a total of 90 excel sheets. Each file contains tens of thousands of rows. Each entry in the dataset is a property sale for that specific year. Each property sale contains information about itself, like the neighborhood, and the address, but the most relevant column for this project is SALE PRICE, which, as the name suggests, is how much the property was sold for. Each dataset in this collection contains a sizeable number of rows where the SALE PRICE is 0. It is hard to tell whether that was just a placeholder for a missing value, or if property was being sold for free, but no matter the reason, the rows would an impact on the results, and must be handled.

  1. NYC Population

This dataset contains the population of NYC over the decades, for each of the five boroughs. Each entry in the dataset is either a borough or the city itself. Each entry has columns for each decade, going as far back as 1950, and the columns contain the population of the row at that time. The relevant columns are 2000, 2010, and 2020, as those columns contain the entire time frame for when sales in NYC were recorded. A problem with the dataset is that the year values being columns would create complications when trying to create visualizations, which should be dealt with.

Techniques

I will be using the Python Pandas library to read the datasets, filter them, and clean them. I will be using it to deal with the Buildings Built in NYC History datasets by dropping rows where the YearBuilt value is 0. It will also be used to deal with dropping rows in the Annual Sales datasets where SALE PRICE is 0. For NYC Population, Pandas will be used to alter the DataFrame to make it easier to create visualizations. Matplotlib will be used to create bar graphs showing the number of buildings built in each zone district category, and line graphs showing the change in buildings built, population, and average sale price over time. Specific uses of these tools are further explained below, right before the code it relates to. I plan on dealing with visualizing the bar graphs first, but all the cleaning will be done before any further visualizations. At the end, the results will be compared to the hypothesis, and depending on the results, the next steps will be outlined.

The csv files containing information on building construction were the first to be read

Here is what the data looks like

In the DataFrame for Queens, there are 12800 rows where the YearBuilt column has a value of 0. Knowing that a value of 0 cannot be valid, the only option remaining was to get rid of all rows where YearBuilt was 0

Now that the DataFrames for each borough were cleaned, it was time to create visualizations. To make things easier, a function was made. This function allows for the creation of a bar graph visualizing the number of buildings constructed in each residential district. It takes in the DataFrame for the specific borough, the borough name, and the color which will be used for the bars. Then, it filters the DataFrame to only include rows where the first letter of the ZoneDist1 value is R, meaning that the building was built in a residential district. Having completed the filtering, groups the rows by ZoneDist1 to find the number of buildings built in each type of zoning district. It then creates a bar graph, with the x-axis being the zoning district category, and the y-axis being the number of buildings built in each zoning district category.

Immediate Observations:

Most of the boroughs have a healthy range of zoning district categories in which buildings are being constructed, with R2 to R7 containing most of the buildings constructed. The two outliers are Manhattan and Staten Island. Manhattan does not appear to have a single building built in a district zoned below R5, with most buildings being built in R7-2 and R8B districts. Staten Island is the opposite, having most of its buildings built in districts zoned below R4, with a few being built in R4 to R6.

This function handles the cleaning and filtering which is needed for each of the excel files for each borough. It reads the excel file, skipping the number of rows that need to be skipped, filters to include rows where the property was not sold for free, creates a new column containing the borough, and then finds the mean of all sales for that dataset, before creating a new column specifying the year, and returning the DataFrame. Rows where the SALE PRICE was 0 were dropped because they would heavily skew the results, and because it is most likely a placeholder for a missing value. Even if it were not, a property being given away for free would have little relevance to the average property sale price at the time, as one would assume that the property giveaway would have been done no matter what the average sale price was. There is a default parameter, which only exists to handle special excel files where each column has a newline character in its name, which would make things confusing. If the parameter is made true, an exact copy of the SALE PRICE\n column is made, only with no newline character in the column name.

The process of applying the function to all the excel files for that specific borough had to be applied to the other four as well.

After all the files for a borough were read and cleaned, all DataFrames for the borough were concatenated, and only the columns necessary for the analysis were kept.

After the concatenation, the DataFrame containing information about buildings built in that borough was grouped by year, to see how many buildings were built each year. The DataFrame was then filtered to only include data from 2003 to 2020, which was the time frame of the sales dataset.

The dataset needed some cleaning, since the Borough column had white space at the start of every column entry, and each decade had its own column, which would have been inconvenient when making visualizations. To deal with the first issue, I used the lstrip function to remove the empty space. To amend the second issue, I used the pd.melt function to transform the decades into a single column, and the values at each decade into a column titled Population. The information for each borough was saved in a separate DataFrame.

This function simplifies the process of creating line graphs for each borough to show the change in the average sales price over the years

Immediate Observations:

For all five boroughs, the average sale prices appear to be growing. Some are growing faster than others, but they are all growing.

This function simplifies the process of creating line graphs showing the change in the amount of buildings being constructed over time

Immediate Observations:

All five boroughs have had the number of new buildings constructed every year drop significantly. Combined with the knowledge that all five boroughs had the average sale prices of property rise over the same time period, it is beginning to look like there is some relationship between the two.

Immediate Observations:

Just like in previous visualizations, the trend appears to be shared between all five boroughs. From 2000 to 2020, all five boroughs had a large increase in population.

The visualization above reveals something which was originally not so easily visible by having each borough in its own graph. It reveals that Manhattan, despite having the less restrictive residential zoning, as revealed in the earlier visualizations, started off with the highest average sale price. The rest of the boroughs cannot catch up to it, although Bronx does shoot up in strongly in the last year. What's more interesting is that Staten Island, despite having the more restrictive zoning, had the lowest average sale price of all boroughs.

This line plot further reveals that Manhattan started off with the lowest number of new buildings being constructed, despite having the higher residential zoning. Staten Island, despite being zoned for lower constructions, was up there with Queens and Brooklyn in the number of new buildings being built, before falling down with the rest of the boroughs.

Each borough experienced an increase in population, although Staten Island's population remains miniscule in comparison to the other boroughs

This function plots the change in buildings built for the borough specified in the parameters, and then puts a vertical red line for the year 1961, which was the year NYC passed a big zoning resolution, which added a load of regulations to the way buildings were being built.

Conclusion:

By plotting the change in population, average sale prices, and the rate of building construction for each of the five boroughs, I managed to successfully demonstrate the connection between population, rate of construction and sale prices, in that as the number of buildings being constructed goes down, and the population increases, the average cost of housing will go up. To put it another way, as the supply of new housing fails to keep up with the increasing demand of living in NYC, the average cost of housing increases. The relationship between zoning and housing supply, however, was less conclusive. On one hand, the line graphs demonstrating change in the number of new buildings being constructed over time appear to show a decline in the rate of construction after the 1961 NYC Zoning Resolution, which added lots of zoning regulations, and would therefore prove that the decrease was caused directly by the Zoning Resolution. However, a closer inspection reveals some complications. For one, Manhattan had its rate drop about 20 years before the Zoning Resolution came to pass, and the rate did not change after it was passed. Brooklyn saw a decline begin in the 1930s, so even though the rate went down even more after 1961, it is possible that it would've done so even without the NYC Zoning Resolution. For Staten Island, the rate at which buildings were being constructed increased after 1961, which would be a huge counter to the relationship I was trying to prove. Overall, despite successfully showing that as rate of building construction declines, and population increases, so would the average sale price of property, the direct link between increased zoning regulations and housing supply has not been proven, and therefore, the results are inconclusive.

What went wrong?:

One reason for why the connection between increased zoning regulation and housing supply was failed to be proven is that there are more variables at play that were not included in this project. One example could be the area of the borough. It only makes sense that Manhattan would eventually end up in a stand still when it comes to the construction of new buildings if there is very little space left. Manhattan, for example, is by far the smallest borough in terms of area, with every borough being almost twice as large as it or more. Staten Island occupies over twice as much area, so it only makes sense for it to have more buildings built. This could explain why the rate of new buildings being built in Manhattan slowed down even before the 1961 NYC Zoning Resolution. Another variable that could have had an impact on the results was prior zoning regulations that were passed, which could have been responsible for the decline in housing being built in Brooklyn. Another problem was not taking into account the type of building being built. A ten story building can house more people than a single family detached house can, and failing to do so made it appear as though Manhattan had fewer homes available.

What Next?

The next steps are to find the rest of the variables which might have had an impact on the results, create additional visualizations using geodata, and create a Linear Regression model from the sklearn library to predict the number of housing built given a zone district category, and to predict the average sale price given a number representing the supply of housing.

Citations

Datasets:

Annual Sales

Buildings built in NYC history

NYC Population

Documentation Resources:

Matplotlib

Pandas

Other

Image Source

Information on Each Borough's Area