Introduction to Data Science

Reading Data and Data Types

Author

Joanna Bieri
DATA101

Important Information

Announcements

Exam 1 is handed out on Review Day, Thursday 10/1. Keep working on the Practice Exam from Day 7!

Please come to office hours to get help!

Day 8 Assignment

  1. Download Day8-HW.ipynb from the class website and move it into your Day8 folder in your repo.
  2. Open the file Day8-HW.ipynb and start doing the problems.
    • You can do these problems as you follow along with the lecture notes and video.
  3. Get as far as you can before class.
  4. Submit what you have so far Commit and Push to Git.
  5. Take the daily check in quiz on Canvas.
  6. Come to class with lots of questions!
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import plotly.express as px
from plotly.subplots import make_subplots
import plotly.io as pio
pio.renderers.default = 'notebook_connected'

Reading in Your Own Data

A big part of data science is finding your own data and then analyzing it. To be able to use Python to analyze data you first have to learn how to load data into a JupyterLab Notebook. There are lots of different ways, but in this class we will use Pandas (pd) to read data.

  • pd.read_csv This command lets you read files that end in .csv. You can create a file like this using a spreadsheet program and saving the file as a .csv. Often you can download data in .csv format. You will need to make sure that your data is in the same folder as the code you are writing or that you can tell your code EXACTLY where the file is.
  • pd.read_excel This command lets you read files created by Excel. Typically with .xlsx file types. You will need to make sure that your data is in the same folder as the code you are writing or that you can tell your code EXACTLY where the file is.
  • pd.read_json This command lets you read .json files. JSON stands for JavaScript Object Notation. It is a very popular way to save data. You will need to make sure that your data is in the same folder as the code you are writing or that you can tell your code EXACTLY where the file is.
  • pd.read_table If other methods fail, sometimes you can force Pandas to read a file by using the table method. You will need to make sure that your data is in the same folder as the code you are writing or that you can tell your code EXACTLY where the file is. You will have to tell what your delimiter is - what character separates the columns (sometimes this is a comma like in .csv, but sometimes it is something else like a colon or space)

Try reading in some data - csv

Go to the Cal Fire Website and scroll to the bottom to see the Incident Data. We will download the file named ALL DATA AS CSV this should put the data file into your Downloads folder.

Next you need to move the file mapdataall.csv from your Downloads folder into your Day8 folder where you are doing your homework. You can open your Downloads folder and drag the file into JupyterLab side bar. Then I can run the command

DF_raw = pd.read_csv('mapdataall.csv')

to load the data and look at the data frame.

NOTE: If you have downloaded this file before, your computer might name the new one mapdataall (1).csv. Rename it to mapdataall.csv before you run the code.

DF = pd.read_csv('mapdataall.csv')
display(DF)
incident_name incident_is_final incident_date_last_update incident_date_created incident_administrative_unit incident_administrative_unit_url incident_county incident_location incident_acres_burned incident_containment ... incident_latitude incident_type incident_id incident_url incident_date_extinguished incident_dateonly_extinguished incident_dateonly_created is_active calfire_incident notification_desired
0 Bridge Fire Y 2018-01-09T21:46:00Z 2017-10-31T18:22:00Z Shasta-Trinity National Forest NaN Shasta I-5 and Turntable Bay, 7 miles NE of Shasta Lake 37.0 100.0 ... 40.774000 NaN 2ca11d45-8139-4c16-8af0-880d99b21e82 https://www.fire.ca.gov/incidents/2017/10/31/b... 2018-01-09T21:46:00Z 2018-01-09 2017-10-31 N False False
1 Pala Fire Y 2020-09-16T21:07:35Z 2009-05-24T21:56:00Z CAL FIRE San Diego Unit NaN San Diego Hwy 76 and Pala Temecula, northwest of Pala 122.0 100.0 ... 1.000000 Wildfire 8f61f461-552d-4538-b186-35ab030da416 https://www.fire.ca.gov/incidents/2009/5/24/pa... 2009-05-25T07:00:00Z 2009-05-25 2009-05-24 N True False
2 River Fire Y 2022-10-24T18:39:23Z 2013-02-24T16:16:00Z CAL FIRE San Bernardino Unit NaN Inyo south of Narrow Gauge Rd & north of Hwy 136, e... 407.0 100.0 ... 36.602575 NaN 094719ba-a47b-4abb-9ec5-a506b2b9fd23 https://www.fire.ca.gov/incidents/2013/2/24/ri... 2013-03-01T04:00:00Z 2013-03-01 2013-02-24 N True False
3 Fawnskin Fire Y 2013-04-22T16:00:00Z 2013-04-21T00:30:00Z San Bernardino National Forest NaN San Bernardino west of Delamar Mountain, north of the communi... 30.0 100.0 ... 34.288877 NaN 58f89ff8-bd3e-4355-b1c0-8fa05c747d3f https://www.fire.ca.gov/incidents/2013/4/20/fa... 2013-04-22T16:00:00Z 2013-04-22 2013-04-21 N False False
4 Gold Fire Y 2013-05-01T14:00:00Z 2013-04-30T19:59:00Z CAL FIRE Madera-Mariposa-Merced Unit NaN Madera Between Road 210 and Road 200 near Fine Gold C... 274.0 100.0 ... 37.116295 NaN 357ffc13-bef9-48eb-810f-c5de851972eb https://www.fire.ca.gov/incidents/2013/4/30/go... 2013-05-01T14:00:00Z 2013-05-01 2013-04-30 N True False
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
3857 Bunny Fire Y 2026-09-14T17:42:23Z 2026-09-13T23:46:33Z CAL FIRE Siskiyou Unit NaN Siskiyou Bunny Lane and Lott Drive, Big Springs 4.0 100.0 ... 41.634570 Wildfire 4de89474-b003-47e6-ae01-82a669161ecb https://www.fire.ca.gov/incidents/2026/9/13/bu... NaN NaN 2026-09-13 N True False
3858 Bruns Fire Y 2026-09-15T13:34:03Z 2026-09-12T23:28:23Z CAL FIRE Santa Clara Unit NaN Alameda Grant Line Road, Altamont 15.0 100.0 ... 37.794882 Wildfire 368cd43b-9647-4729-a211-66e023b575c0 https://www.fire.ca.gov/incidents/2026/9/12/br... 2026-09-14T18:20:15Z 2026-09-14 2026-09-12 N True False
3859 Palos Fire N 2026-09-15T18:34:20Z 2026-09-14T22:05:25Z Los Angeles County Fire Department NaN Los Angeles Lupin Hill Road and Las Palomas Drive, Whittier 17.0 70.0 ... 33.968100 Wildfire 2806bcea-7e12-4de5-883a-3439244a0012 https://www.fire.ca.gov/incidents/2026/9/14/pa... NaN NaN 2026-09-14 Y False False
3860 Dome Fire N 2026-09-16T20:52:13Z 2026-09-15T15:28:00Z National Park Service NaN Mariposa North of Wawona Dome, Yosemite National Park 1177.0 NaN ... 37.570000 Wildfire 4883092e-485f-429d-878d-819f01172840 https://www.fire.ca.gov/incidents/2026/9/15/do... NaN NaN 2026-09-15 Y False False
3861 Floriston Fire N 2026-09-16T22:18:15Z 2026-09-16T17:01:00Z Unified Command: CAL FIRE Nevada-Yuba-Placer U... NaN Nevada I-80 and Floriston way, 9 miles northeast of T... 300.0 5.0 ... 39.416068 Wildfire 653b14c1-b18e-4531-9290-9362f5a2fd1b https://www.fire.ca.gov/incidents/2026/9/16/fl... NaN NaN 2026-09-16 Y True False

3862 rows × 23 columns

Answer some questions about this data

Q How many variables and observations?

Q How many different incident types are there?

Q Make a histogram of the acres burned and color the bars by whether or not the incident was a calfire incident. You will probably need to make a mask to remove very small and very large fires. How many fires burned more than 100,000 acres? What is the largest fire in the data?

Q EXTRA - CHALLENGE - See if you can create a graph that answers the question: Are fires getting bigger or more frequent over time? You get complete creative control on how to answer this question!

Try reading in some data from a website

Here we will explore academy award winning films. Go to the Wiki for the List of Academy Award Winning Films. Look at what type of data is there. How many tables? Any weird looking data?

More and more websites block Python from reading their pages directly (you will see an error like HTTP Error 403: Forbidden). So I saved a copy of the main table from this page as a .csv file on the class website. Notice that you can give pd.read_csv() a web address instead of a file name!

file = 'https://joannabieri.com/introdatascience/data/academy_award_films.csv'
DF_raw = pd.read_csv(file)

Let’s try a little analysis of this data.

display(DF_raw)
Film Year Awards Nominations
0 One Battle After Another 2025 6 13
1 Sinners 2025 4 16
2 Frankenstein 2025 3 9
3 KPop Demon Hunters 2025 2 2
4 Hamnet 2025 1 8
... ... ... ... ...
1396 The Yankee Doodle Mouse 1943 1 1
1397 The Yearling 1946 2 7
1398 Yesterday, Today and Tomorrow (Ieri, oggi, dom... 1964 1 1
1399 You Can't Take It with You 1938 2 7
1400 Zorba the Greek (Alexis Zorbas) 1964 3 7

1401 rows × 4 columns

DF_raw.describe()
Film Year Awards Nominations
count 1401 1401 1401 1401
unique 1387 98 22 25
top A Star Is Born 1945 1 1
freq 3 21 998 560

Notice that we are getting very limited statistical information!! What is going wrong here?

Something is wrong with our data!

What should the data types be:

  • Film - should be categorical (a word or string)
  • Year - should be discrete numerical (a number or integer)
  • Awards - should be discrete numerical (a number or integer)
  • Nominations - should be discrete numerical (a number or integer)

So we should be able to get good statistics on ‘Year’, ‘Awards’, and ‘Nominations’. Let’s sub select just these columns.

my_columns = ['Year', 'Awards', 'Nominations']
DF_raw[my_columns].describe()
Year Awards Nominations
count 1401 1401 1401
unique 98 22 25
top 1945 1 1
freq 21 998 560

This still is not making sense. What if I wanted to know the average number of awards?

Something is really wrong with our data - I get an error!

DF_raw['Awards'].mean()
TypeError: Could not convert string '64321111111111532221111111117421111111111742111111111136211111111111132212221121111142111111223111113433111111111114322221111111110 (1)[1]36221111111111126311111111111114431111111111111373221111111343222111111111552111111111111443222111111163222111111111832211111111143221111111111111432221111111111133331111111111114522111111111111122111111111632211111111111442211111111111154411111111115422111111111175311111111111222111111192111111111111152222111111111110 (1)6222111111111111733211111111433211111111111542221111111173211111111110 (1)432221111111111433 (1)11111111111119311111111110 (1)43321111111111117221111111111118322111111110 (1)544211111110 (1)842111111111144 (1)3321111114322211 (1)1111154221111111110 (1)532111111111110 (1)46 (1)31 (1)111111113442111111110 (1)0 (1)5431111111110 (2)632111 (1)11111117221111111111381 (1)1111111115322211111111721111111111111343211111111115 (1)3211111111111110 (1)532221111111116532221111115532111111110 (1)321311111162131161112145115115211211111117 (1)10 (1)1120 (1)1112111111111121211711111111113113121111130 (1)111411111211111111111111112111113111111211111131111110 (2)211211111110 (1)0 (1)1180 (1)21 (1)13119111171118 (2)2111112121113211411114110 (1)2114111110 (1)1531311211111120 (1)4111510 (1)22 (1)111122111512111110 (1)1111117111231110 (2)11111224311111114511211113210 (1)11111211161111811221111111111111211181211111112221112610 (1)1111111111112110 (1)1121221323120 (1)141111 (1)1112121111110 (1)1111110 (1)1111241 (1)111140 (1)111121111 (1)13114111331110 (1)1110 (1)1111131111112111111111111134111111131111211111111111110 (1)1111101111111111521111211111111312123' to numeric

Help my data is broken!

What you see happening here is that the data is not in the format that you expected. Now as a data scientist you have to decide what to do!

Data Types

In python there are four data types that we will work with most:

  • String - this is a word, sentence, letter… surrounded by quotes.
  • Integer - this is a number with no decimal points - fractions not allowed (rounded)
  • Float - this is a number with decimal places - fractions allowed
  • Boolean - this is True or False

Let’s decipher the error message:

“Could not convert string — to numeric.”

This is telling us that when Python tried to take that average it ran into strings (words). It then tried to convert those words into numbers, but ran into problems. So it cannot take the average here.

Data Type Exploration

Below are some variables that contain different data - we will use the type() command to see what data type they are.

num1 = 1
num2 = 1.0
num3 = '1'
num4 = 1.6
num5 = '1.2'
print(type(num1))
print(type(num2))
print(type(num3))
print(type(num4))
print(type(num5))
<class 'int'>
<class 'float'>
<class 'str'>
<class 'float'>
<class 'str'>

Does this result match our definitions above?

Changing Data Types

Now what if we wanted to change these data types from one type to another? We can use the commands:

  • str() - to change to a string
  • int() - to change to an integer
  • float() - to change to a float

Intentional Converting Between Types:

num1_str = str(num1)
print(type(num1_str))
num1_str
<class 'str'>
'1'
num2_int = int(num2)
print(type(num2_int))
num2_int
<class 'int'>
1
num3_int = int(num3)
print(type(num3_int))
num3_int
<class 'int'>
1
# int() will always round down!
num4_int = int(num4)
print(type(num4_int))
num4_int
<class 'int'>
1
num5_float = float(num5)
print(type(num5_float))
num5_float
<class 'float'>
1.2

Unintentional Converting Between Types

Sometimes Python functions will automatically convert between types or make type decisions for you!

Here is an example of adding an integer and a float. Python returns the result as a float.

num1 = 1
num2 = 1.0

type(num1+num2)
num1+num2
2.0

Here we see that even though we send in integers, when we take the average it returns a float.

my_numbers = [2,2,2,2]
np.mean(my_numbers)
np.float64(2.0)

Fun with Strings!

Many times your data issues will arise when someone enters data in a wrong format. This often means that you have to deal with strings that need to be fixed. Here are some useful commands

  • .lstrip() - removes left white space
  • .rstrip() - removes right white space
  • .replace(string1, string2) - replaces string1 with string2 as long as string1 is part of the original string
  • .split(character) - splits the string into chunks based on the character provided

There are lots more, but this is enough to learn for now!

my_string = '   Year: 2024/25   '
len(my_string)
19
my_string = my_string.lstrip()
len(my_string)
16
my_string = my_string.rstrip()
len(my_string)
13
my_string = my_string.replace(':',',')
my_string
'Year, 2024/25'
my_broken_string = my_string.split('/')
my_broken_string
['Year, 2024', '25']
# Now we can select just one part of the original string
my_broken_string[1]
'25'

So How Can I Fix my Data?

Remember that this entire problem started when we were exploring the movie data stored in DF_raw. Let’s learn a new command to help us see what type of data is in the DataFrame. To report the type of data in each variable use the command

DF_raw.dtypes
DF_raw.dtypes
Film           object
Year           object
Awards         object
Nominations    object
dtype: object

object means that the data is a string or that there is at least one string in the column. Does it make sense for year to be a string?

year_data = DF_raw['Year'].value_counts()
DF_year = year_data.reset_index().rename(columns={"index": "value", 0: "count"})
DF_year
Year count
0 1945 21
1 1949 20
2 1942 20
3 1950 19
4 1948 19
... ... ...
93 1931/32 10
94 1932/33 9
95 1928/29 7
96 1929/30 6
97 1930/31 6

98 rows × 2 columns

There are some years that look like normal integer years and others that are formatted like 1931/32. How can we fix this so we can do our analysis?

This is more of an art than a science - as a data scientist you need to make these types of decisions and keep track of why and how you make the decision. For the data above, my plan is to remove the extra data /32 and just use the first part of the string for year.

.apply() and lambda

  • The .apply() command lets you apply a function to every row along your data frame.

  • lambda is a special function you first write the word lambda and choose a variable followed by a colon

    lambda x:

    then you give an operation to do to every x (in this case every entry in the “Year” column)

    x.split('/')[0]

    this will split up the string so that 1930/31 becomes [1930, 31] then the [0] just selects the first part of the list.

We can apply this operation and save the information over the “Year” column so the data will be in a better format!

DF_raw['Year'] = DF_raw['Year'].apply(lambda x: x.split('/')[0])
year_data = DF_raw['Year'].value_counts()
DF_year = year_data.reset_index().rename(columns={"index": "value", 0: "count"})
DF_year
Year count
0 1945 21
1 1949 20
2 1942 20
3 1950 19
4 1948 19
... ... ...
93 1931 10
94 1932 9
95 1928 7
96 1929 6
97 1930 6

98 rows × 2 columns

Now we will turn all the data in the year column into integers. There are lots of ways to do this, but lets practice the .apply() with a lambda method.

DF_raw['Year'] = DF_raw['Year'].apply(lambda x: int(x))
DF_raw.dtypes
Film           object
Year            int64
Awards         object
Nominations    object
dtype: object

Notice that we fixed the year column! Lets start working on the other columns.

Do the same type of thing to the Awards column

Look at what kind of data is in the “Awards” column.

award_data = DF_raw['Awards'].value_counts()
DF_award = award_data.reset_index().rename(columns={"index": "value", 0: "count"})
DF_award
Awards count
0 1 998
1 2 160
2 3 79
3 4 48
4 0 (1) 32
5 5 26
6 6 13
7 7 12
8 1 (1) 8
9 8 7
10 9 3
11 11 3
12 0 (2) 3
13 0 (1)[1] 1
14 4 (1) 1
15 3 (1) 1
16 6 (1) 1
17 5 (1) 1
18 7 (1) 1
19 8 (2) 1
20 2 (1) 1
21 10 1

What the heck are these extra () and [] numbers?

To understand this better I had to go back to the website. It seems that things in parenthesis are special awards and things in square brackets are citations.

So I will choose to remove this information. THIS IS JUST ONE CHOICE - alternatively we could choose to add another column just for special awards if these were important to our analysis.

Looking at the data, I only want the numbers to the left of a ( so I will use this character to split my strings. I will also turn all the data into integer types.

DF_raw['Awards'] = DF_raw['Awards'].apply(lambda x: int(x.split('(')[0]))
award_data = DF_raw['Awards'].value_counts()
DF_award = award_data.reset_index().rename(columns={"index": "value", 0: "count"})
DF_award
Awards count
0 1 1006
1 2 161
2 3 80
3 4 49
4 0 36
5 5 27
6 6 14
7 7 13
8 8 8
9 11 3
10 9 3
11 10 1 
DF_raw.dtypes
Film           object
Year            int64
Awards          int64
Nominations    object
dtype: object

You Try - Nominations column

Q Now it is your turn. Fix the Nominations column so that all the data is in integer format.

First look at the data and notice that the [ character seems to split the number of nominations from something that looks like a reference.

nom_data = DF_raw['Nominations'].value_counts()
DF_nom = nom_data.reset_index().rename(columns={"index": "value", 0: "count"})
display(DF_nom)
print('I can see that I want the data to the left of the [ character')

DF_raw['Nominations'] = DF_raw['Nominations'].apply(lambda x: int(x.split('[')[0]))
nom_data = DF_raw['Nominations'].value_counts()
DF_nom = nom_data.reset_index().rename(columns={"index": "value", 0: "count"})
display(DF_nom)

DF_raw.dtypes

If everything went to plan you should see:

DF_raw.dtypes
Film           object
Year            int64
Awards          int64
Nominations    object
dtype: object

Now we can do some statistics and continue our analysis!

DF_raw.describe()
Year Awards
count 1401.000000 1401.000000
mean 1976.836545 1.576017
std 27.982361 1.390826
min 1927.000000 0.000000
25% 1952.000000 1.000000
50% 1976.000000 1.000000
75% 2002.000000 2.000000
max 2025.000000 11.000000

Warning - Be Kind!

Fixing messy data can be a challenging and complicated process. Please be nice to yourself!

What is expected in this class:

  1. You can find some data online and load it into Python
  2. You can figure out what types of data are in each column
  3. You can say in words what format you would like the data to be in.
  4. You can patiently TRY to fix the data.
  5. You come get help from your Professor, TA, or friends before you get too frustrated!

Try to find some data of your own.

Your homework today will be to see if you can find some data of your own. This can be the first steps you take toward your final project.