the fivethirtyeight urban index

tl;dr

I have attempted to recreate the fivethirtyeight urban index. Here’s the repo.

fivethirtyeight have a new urban index

fivethirtyeight have come up with a method for quantifying urban or rural-ness. The data, quite helpfully, are posted on GitHub. Less helpfully, (and unlike one of the other sources they mention) they don’t show how their results are derived, so I thought I’d spend some time over the weekend reproducing the calculation.

the process

1. get data for every census tract in the US
2. figure out which tracts are within five miles of each other
3. add up the population for tracts within five miles of each other

Geocomputation with R by Robin Lovelace, Jakub Nowosad, and Jannes Muenchow is very helpful for getting up to speed on the fundamentals of analyzing geospatial data, of which I know precious little.

getting the data

{tidycensus} by Kyle Walker makes pulling census data into R incredibly easy.

Once you have the data, {sf} makes it easy to do spatial calculations. Judging from my many searches through stackoverflow, it hasn’t always been this easy.

getting it wrong

How many people live within 5 miles of you? If you calculate this number for each person (well, each Census Tract) in the state, take the natural logarithm, then average them together (weighed based on the Census Tract’s population), you can come up with a nifty “urbanization index”

–Nate Silver

Did you think that meant you should compute five-mile buffers around each tract centroid and then perform an areal weighted interpolation of the tract populations into those buffers? No? Good. Neither did I.

Here are the first ten rows of that attempt.

# A tibble: 52 x 2
   state          avg_pop_within_five
   <chr>                        <dbl>
 1 Idaho                         8.54
 2 Texas                         8.51
 3 Georgia                       8.51
 4 Utah                          8.50
 5 Washington                    8.48
 6 California                    8.45
 7 Oregon                        8.45
 8 Florida                       8.44
 9 Massachusetts                 8.41
10 North Carolina                8.40

🙅🏾‍♂️

getting it less wrong

Using spatial joins, I created a dataset of intersections between each tract and a five-mile buffer around its centroid. This is surprisingly fast, and gives results that are close to the reference solution:

inner_join(pop_within_5_mi, states) %>% 
  filter(pop_within_five != 0) %>% # this drops 18 or so tracts
  mutate(log_pop_within_five = log(pop_within_five)) %>% 
  group_by(state) %>% 
  summarise(avg_log_pop_within_five = mean(log_pop_within_five, na.rm = TRUE)) %>% 
  arrange(desc(avg_log_pop_within_five))

one more alternative approach

Rather than intersecting tract boundaries with the five-mile buffer from their centroids, I tried using st_is_within_distance to join the data frame of tract centroids to itself. Since no index is built, this is not as fast. It also gives results that match the reference less well. You can install the package from the centroid-self-join branch for a look at those.

how to get even closer?

fivethirtyeight do some tweaking of their index for tracts whose centroid is further than five miles away from other tract centroids:

For a census tract that is more than 5 miles away any other census tract (centroid to centroid), this number is decreased based on the minimum distance to the nearest census tract

I might be using the wrong projection for this kind of analysis.

There may be a better spatial join for expressing the relationship described in the article.