Data Science Process

Adapted from r4ds

We have covered…

Today

Getting Data

• So far, we have simply needed to import one nice rectangular data set in a typical file type
• Real life often gets a bit more complicated!!

Motivating (Real) Example¹

Relational Data

Example - IMDb Movie Data

Example - IMDb Movie Data

Data Joins

We can combine (join) data tables based on their relations.

Keys

Some combination of variables (should) uniquely identify an observation in a data set.

• To combine (join) two datasets, a key needs to be present in both.

General Structure of a Join

inner_join()

Keeps observations when their keys are present in both datasets.

inner_join(): IMDb Example

directors_genres

movies_directors

inner_join(directors_genres, movies_directors, 
           by = "director_id")

inner_join(): IMDb Example

What if our key does not have the same name?

directors_genres

directors

inner_join(directors_genres, 
           directors, 
           by = join_by(director_id == id))

Piping Joins

Remember: the dataset you pipe in becomes the first argument of the function you are piping into!

• If you are using a pipe,
  • the piped in data is the left dataset
  • specify the right dataset inside the join function.

inner_join(directors_genres, movies_directors)

directors_genres |> 
  inner_join(movies_directors, by = "director_id")

More Mutating Joins

• left_join() – keep only (and all) observations present in the left data set

• right_join() – keep only (and all) observations present in the right data set

• full_join() – keep only (and all) observations present in both data sets

Why Use a Certain Mutating Join?

Which Join?

directors |> 
  ??_join(movies_directors, 
          by = join_by("id" == "director_id"))

directors |> 
  slice_head(n = 5)

movies_directors |> 
  slice_head(n = 5)

Which Join?

roles |> 
  ??_join(actors, 
          by = join_by("actor_id" == "id"))

roles |> 
  slice_head(n = 5)

actors |> 
  slice_head(n = 5)

Filtering Joins: semi_join()

Keeps observations when their keys are present in both datasets, but only keeps variables from the left dataset.

IMDb Data Example

directors_genres |> 
  distinct(director_id)

movies_directors |> 
  distinct(director_id)

Filtering Joins: semi_join()

directors_genres |> 
  semi_join(movies_directors)

directors_genres |>
  filter(director_id %in% movies_directors$director_id)

Filtering Joins: anti_join()

Removes observations when their keys are present in both datasets, and only keeps variables from the left dataset.

Filtering Joins: anti_join()

directors_genres |> 
  anti_join(movies_directors)

directors_genres |>
  filter(!director_id %in% movies_directors$director_id)

Building an Analytical Dataset

Now we have tools to:

• Combine multiple data sets (xx_join())
• Subset to certain observations (filter() and xx_join())
• Create new variables (mutate())
• Select columns of interest (select())

Transform and Tidy

What’s next?

Transform and Tidy

What’s next?

We may need to transform our data to turn it into the version of tidy that is best for a task at hand.

Allison Horst

Creating Tidy Data

Let’s say we want to look at mean cereal nutrients based on shelf.

library(liver)
data(cereal)
head(cereal)

Creating Tidy Data

Creating Tidy Data

cereal_wide <- cereal |> 
  group_by(shelf) |> 
  summarise(across(calories:vitamins, mean))

my_colors <- c("calories_col" = "steelblue", "sugars_col" = "orange3")

cereal_wide |> 
  ggplot() +
  geom_point(aes(x = shelf, y = calories, color = "calories_col")) +
  geom_line(aes(x = shelf, y = calories, color = "calories_col")) + 
  geom_point(aes(x = shelf, y = sugars, color = "sugars_col")) +
  geom_line(aes(x = shelf, y = sugars, color = "sugars_col")) +
  scale_color_manual(values = my_colors, labels = names(my_colors)) +
  labs(x = "Shelf", y = "", subtitle = "Mean Amount", color = "Nutrient")

cereal_long<- cereal |> 
  pivot_longer(cols = calories:vitamins,
               names_to = "Nutrient",
               values_to = "Amount") |> 
  group_by(shelf, Nutrient) |> 
  summarise(mean_amount = mean(Amount))

cereal_long |> 
  ggplot(aes(x = shelf, 
             y = mean_amount, 
             color = Nutrient)) +
  geom_point() +
  geom_line() +
  labs(x = "Shelf", y = "", subtitle = "Mean Amount")

Data Layouts

Kelsey Gonzalez

Manual Method

Consider daily rainfall observed in SLO in January 2023.

• The data is in a human-friendly form (like a calendar).
• Each week has a row, and each day has a column.

Data source

Manual Method: Steps

0. Keep the column Week.
1. Create a new column Day_of_Week.
2. Create a new column Rainfall (hold daily rainfall values).
3. Now we have three columns – move Sunday values over.
4. Duplicate Week 1-5 and copy Monday values over.

5. Repeat …

Computational Approach

We can use pivot_longer() to turn a wide dataset into a long(er) dataset.

pivot_longer()

Take a wide dataset and turn it into a long dataset.

• cols – specify the columns that should be pivoted.
  • Do not include the names of ID columns (columns to not be pivoted).
• names_to – the name of the new column containing the old column names.
• values_to – the name of the new column containing the old column values.

pivot_longer()

slo_rainfall |> 
  pivot_longer(cols      = Sunday:Saturday,
               names_to  = "Day_of_Week",
               values_to = "Daily_Rainfall")

Long to Wide

What are the mean amount of protein for cereals on each shelf and for each manuf?

mean_protein <- cereal |> 
  group_by(manuf, shelf) |> 
  summarize(mean_protein = mean(protein))

pivot_wider()

Take a long dataset and turn it into a wide dataset.

• id_cols – specify the column(s) that contain the ID for unique rows in the wide dataset.
• names_from – the name of the column containing the new column names.
• values_from – the name of the column containing the new column values.

pivot_wider()

Much easier to read!

mean_protein |> 
  arrange(shelf) |> 
  pivot_wider(id_cols = manuf,
              names_from = shelf,
              values_from = mean_protein)

Better names in pivot_wider()

Even better!

mean_protein |> 
  arrange(shelf) |> 
  pivot_wider(id_cols = manuf,
              names_from = shelf,
              values_from = mean_protein,
              names_prefix = "Shelf_")

PA 4: Military Spending

Today you will be tidying messy data to explore the relationship between countries of the world and military spending.

Resources - use them all!

To do…

• PA 4: Military Spending
  • Due Thursday, 4/24 before class