Summary Statistics

Author

Uswa

Review of key concepts from the last two sections:

  • Distribution

  • The Normal Distribution

  • Mean and SD as parameters of normal distribution

  • Thinking of the Mean as a Model

  • Other Summary Statistics: Median and Range

  • Boxplots and Interquartile Range

Summary Statistics in R

The goals for today’s session:

  • to observe more on random sampling from uniform and normally distributed data

  • to understand the relation between distribution and summary statistics

  • to summarize chapter 3

Uniform Data vs Normally Distributed Data

Generating Uniform Data

We can use the runif() function to generate uniform data. Specifically, this function enables us to create a set of random samples from a certain range with the same probability. Let’s try to generate 50 samples What can you notice from the generated data?

#Let's try to run a simple code with runif() function.
data <- runif(50)
data
 [1] 0.41163680 0.88732686 0.88283117 0.24069279 0.85872036 0.74907601
 [7] 0.50869655 0.46852038 0.16134834 0.68049021 0.56049318 0.18038639
[13] 0.37279643 0.59891815 0.20314998 0.24591329 0.93630751 0.60645428
[19] 0.92234917 0.68749450 0.49774591 0.38390026 0.67714099 0.84937463
[25] 0.10142332 0.40365496 0.89798771 0.88319269 0.76846795 0.82092781
[31] 0.56071468 0.12015907 0.10962531 0.72193883 0.78166062 0.49510585
[37] 0.23194611 0.44771159 0.09883868 0.97015904 0.87372047 0.67271855
[43] 0.26678082 0.17439998 0.12908144 0.67275414 0.08503159 0.10884754
[49] 0.08508302 0.36431324

Using the code as above allows the data to be different every time we run it, but the data will still range from 0 to 1 as this is the default range. To see the distribution of the data, take a look at this ggplot. What can you say about the distribution of the data?

#load tidyverse to access ggplot() function
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.0     ✔ stringr   1.5.1
✔ ggplot2   3.5.1     ✔ tibble    3.2.1
✔ lubridate 1.9.3     ✔ tidyr     1.3.1
✔ purrr     1.0.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
data <- data.frame(value = data)
ggplot(data, aes(x = value)) +
  geom_histogram(binwidth = 0.1, color ='palevioletred', fill = 'darkseagreen')

Since the data are randomly uniform, the data are not sampling from the normal distribution. Instead, the data are sampling assuming equal probability of occurrence. Thus, you can see that the distribution captures approximately the same number of occurrences. The more data you generate, the more ideal the histogram will be.

Generating Normally Distributed Data

The rnorm() function is used to generate random samples from a normal distribution. Let’s also try to generate 50 samples What do you notice when you run the code?

#Let's try to run a simple code with rnorm () function.
data <- rnorm (50)
data
 [1] -0.21113544  0.37358359 -0.08814539  0.08066252  0.09068120  1.02417102
 [7]  1.92487022 -0.00360271  1.74992184  0.19985935  1.34776733 -0.02602803
[13]  1.23055126  0.95043014  0.27220247  1.33357780 -0.81148029 -0.09922823
[19] -1.82518227 -0.31608019 -0.35856566 -0.52218100  0.42714771 -1.58907034
[25] -1.69406583 -0.69751311  1.37390946  0.41933800  2.11625366 -0.54623020
[31] -0.12666013  1.80338274  0.24499134  1.38224144  0.75665301  0.97770721
[37] -0.56501738 -0.56188922 -1.02181384  0.27935564  0.34836600  0.08469372
[43]  0.32286465  2.72190456 -0.08898858  0.76037163  0.49014447  0.59986373
[49] -0.85806494  2.09152662

By default, rnorm(50) generates 50 samples from a normal distribution with a mean of 0 and standard deviation of 1. The default range is set to infinity to allow normal distribution. However, we can always set the range to our needs. Let’s generate data again using rnorm(), but this time we specify the mean and SD.

#let's generate data by specifying the mean and SD
data <- rnorm (50, mean = 5, sd = 2)
data
 [1]  6.414177  1.704102  5.665492  6.309860  7.462552  8.033738  5.122666
 [8]  6.762058  3.794497  3.373541  7.307991  6.680818  5.032165  2.272561
[15]  2.741455  1.971826  5.918430  4.558752  8.346869  6.700781  3.945453
[22]  6.005638  1.530552  5.886570  6.600007  3.756451 10.163644  3.833530
[29]  1.562139  7.582037  5.009709  4.156152  5.660287  4.020888  4.440845
[36]  5.255390  3.105990  7.214715  8.151652  7.431598  8.770337  8.076802
[43]  4.615697  5.030587  8.134393  3.329948  4.173221  6.924346  4.671532
[50]  3.067837

We now also create histogram from this data. What can you notice from the histogram?

#to create a histogram, we need to create a dataframe where value is a column that contains the data.
data <- data.frame(value = data)
#now create a ggplot
ggplot(data, aes(x = value)) +
  geom_histogram(binwidth = 1, color ='deeppink', fill = 'palegreen')

From the histogram below we can see that even though the data are sampled from the normal distribution, the distribution does not really show the ideal bell curve. Lets try to generate more samples.

#generate bigger data with the same mean and SD
data <- rnorm (500, mean = 5, sd = 2)

Now, let’s create histogram from this data. This time, what do you notice?

#Again, make sure the data are stored in the 'value' column 
data <- data.frame(value = data)
#Generate the ggplot
ggplot(data, aes(x = value)) +
  geom_histogram(binwidth = 1, color ='lavenderblush', fill = 'mediumspringgreen')

So why do you think it is important for us to look at the differences between random samples drawn from uniform and normally distributed data?

Relationship between Summary Statistics and Distributions

Do you still remember the relationship between mean, SD, and distribution? Let’s explore more to jog your memory.

We know the simple way to get mean and SD from our data.

#Now that we have our data in 'value' column, we need to call them with $ symbol.
mean(data$value)
[1] 5.034002
sd(data$value)
[1] 2.023144

Another function that is very helpful in relation to summary statistics is quantile(). The quantile() function can help us look at the data within specific percentiles. This is beneficial if we want to look at Q1 (the 25th percentile), Q2 (the 50th percentile), and Q3 (the 75th percentile).

#putting $ symbol is important since the data are now inside the 'value' column.
quantile (data$value)
       0%       25%       50%       75%      100% 
-1.224574  3.708002  5.046864  6.399088 14.210587

Do you still remember the 68% - 95% rule in the normal distribution? Right! 68% of normally distributed data data lie within 1 standard deviation of the mean and 95% of data lie within 2 standard deviations of the mean. Based on the same rule, it is found the 68% of data corresponds to the 16th and 84th percentiles.

From Proclus Academy, 2025 (proclusacademy.com)

Thus using the function quantile() we can find the range of 68% of these normally distributed data.

#Finding the range of our 68% data
quantile (data$value, 0.16)
     16% 
3.109522 
quantile (data$value, 0.84)
     84% 
6.950875

Since the data are sampling from the normal distribution, this mean that 68% percent of the data numbers should fall in the range between the 16% to 84% percentiles. With the same understanding, we can say that the 50th percentile is the mean. This mean that there is a strong relationship between summary statistics and distribution.

In other way around, we can check whether the data follows normal distribution by checking whether the 16th and 84th percentiles match the mean and SD. Here is what we can do:

#the results here match the 16th and 84th percentiles above
mean (data$value) - sd (data$value)
[1] 3.010858
mean (data$value) + sd (data$value)
[1] 7.057146

So what is the relation between summary statistics and distribution based on what we have discussed?

One thing is that in the ideal normal distribution, the 50th percentile is the mean and adding 1 unit of standard deviation to the mean equals to the 84th percentile and subtracting 1 unit of standard deviation from the mean equals to the 16th percentile. Would you have any other insights from our discussion?

In relation to our discussion, is it true that our data should follow the normal distribution? Why do we still talk about normal distribution from time to time?

Application on Data Emotional Valence Ratings

let’s load the data. You can find the data at this repository or download it with this notebook using the link at the end of the notebook.

war <- read_csv('warriner_2013_emotional_valence.csv')
Rows: 13915 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): Word
dbl (1): Val

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
war
# A tibble: 13,915 × 2
   Word          Val
   <chr>       <dbl>
 1 aardvark     6.26
 2 abalone      5.3 
 3 abandon      2.84
 4 abandonment  2.63
 5 abbey        5.85
 6 abdomen      5.43
 7 abdominal    4.48
 8 abduct       2.42
 9 abduction    2.05
10 abide        5.52
# ℹ 13,905 more rows

There are two columns in the data frame: Word and Val. What do we need to do if want to find the range of Val? We can choose one of these three ways:

#1st way
range (war$Val)
[1] 1.26 8.53
#2nd way
filter(war, Val == min(Val) | Val == max (Val))
# A tibble: 2 × 2
  Word        Val
  <chr>     <dbl>
1 pedophile  1.26
2 vacation   8.53

Another function which is a bit more complicated that we can use:

#3rd way
filter(war, Val %in% range(Val))
# A tibble: 2 × 2
  Word        Val
  <chr>     <dbl>
1 pedophile  1.26
2 vacation   8.53

What does %in% do?

# %in% is similar to x in y. Let's see the example:
y <- c(2,4,6)
2 %in% y
[1] TRUE

We can use our knowledge on the data distribution from the previous subsection and see the distribution of these data. Let’s look at the mean and SD

mean(war$Val)
[1] 5.063847
sd(war$Val)
[1] 1.274892

We expect that 68% of the data falls between the percentile of 16% and 86%. Then we can find the rough number using mean () function and sd () function.

mean(war$Val) - sd(war$Val)
[1] 3.788955
mean(war$Val) + sd(war$Val)
[1] 6.338738

We can verify this by using quantile() function.

quantile(war$Val, c(0.16, 0.84))
 16%  84% 
3.67 6.32

The value from (mean-SD) and (mean+SD) is very similar to the 16% and 84% percentiles respectively. Why do you think that happened?

The same way we can look at the median.

median(war$Val)
[1] 5.2
quantile(war$Val, 0.5)
50% 
5.2

Key points

  • Each one of these three aspects (sample, mean, and standard deviation) affects distribution.

  • Summary statistics can help us understand the distribution of our data.

Practice

Now that we have completed chapter 3, let’s review our understanding by doing a simple practice exercise.

“First, create a data frame with specific sample, mean, and SD. You can use rnorm() or runif() funtions. Using ggplot, make a histogram, density plot, and/or boxplot from your data. Second, change one aspect from your data. You can change the sample number, the mean, or the SD. Make the same visualisations using ggplot. Do you notice differences in distribution. Share your findings!”

Summary of Chapter 3

Remember the key points mentioned in the beginning of our session, what are the things you still can recall from each?

  • Distribution

  • The Normal Distribution

  • Mean and SD as parameters of normal distribution

  • Thinking of the Mean as a Model

  • Other Summary Statistics: Median and Range

  • Boxplots and in Interquartile Range

Downloads

Use this link to download this .qmd file and the data.

Notebook Download