module 5 introduction to pandas

WELCOME TO GAGENERAL ASSEMBLY

Introduction to PandasModule 5

Modules of this Course

Module 1: Foundations of Programming

Module 6:EDA with Pandas

Module 2: Introduction to

Python

Module 3: Python Data Structures

Module 5:Introduction to

Pandas

Module 4: Intermediate

Python

Module 5: Lesson Objectives

After learning this module, you will be able to:

● Understand what data science is and the roles needed in data science● Apply the data science workflow● Understand programming tools for data science and how Pandas fit in ● Use Pandas to read in a dataset● Get data information and summary statistics● Investigate a dataset's integrity● Filter, sort, and manipulate DataFrame series

3 | © 2021 True Digital Academy


Introduction to Pandas

Intro to Data Science

What is Data Science?

Data science encompasses:

● Framing the problem● Collecting the raw data needed for your problem● Processing the data for analysis● Exploring the data● Performing in-depth analysis● Communicating results of the analysis


Data Science Examples

● Fraud and Risk Detection● Healthcare● A bank approving a credit card● Recommendation systems in marketing & advertising


Data Science Examples

● Oil and Gas Industry can benefit from Data Science:○ Exploration and discovery – rock types can be used to predict oil pockets.○ Production accounting – production data can be linked with alarms.○ Drilling and completions – Predictive analytics can make use of geological

completion and drilling data to determine between preferred, best, drilling locations.

○ Equipment maintenance – Real-time streaming data from rigs can be compared with historical drilling to help predict and prevent problems and better understand operation risks.


Conway Venn Diagram

8 | © 2021 True Digital Academy http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram

http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram

Job Roles in Data Science

What does that break down to?

● Machine Learning Engineer● Data Engineer● Research Science● Advanced Analyst


Machine Learning Engineer

● Identify machine learning applications.● Work in production code.● Manage infrastructure and data pipelines● “Straddle the line between knowing the mathematics and coding the mathematics.”

○ eBay VP of engineering Japjit Tulsi


Production codeIt is hard to give a general definition of what production code is, but a key difference with non-production code, is that production code gets read and executed by many other people, instead of just the person that wrote it. We should therefore aim for our code to be

● Reproducible, because many people are going to run it.

● Modular and well-documented, because many people are going to read it.

Data Engineer

● Create the architecture that allows data acquisition and machine learning problems to run at scale.

● Focus on the algorithm and the analysis.● Don't work much on the software side.


Data Engineer is a person who design, build and install the data system.Data Engineer is responsible for data flow, data pipeline, data storing and data structure.

Data pipeline is a set of tools and activities for moving data from one system to another system. It combines 4 phases; ingest, store, process and consume.

Research Scientist

● PhD-heavy field.● Determines new algorithmic optimizations.● Focused on driving scientific discovery.● Less concerned with pursuing industrial applications.

Applied research scientists:

● Specialized research scientist.● Backgrounds in both data science and computer science.● Invaluable members of any AI team.● “They can both pitch in on data science and write code. Finding a good applied

research scientist is worth her weight in gold.○ Japjit Tulsi


Advanced Analysts

● Quantitative-minded.● Apply data descriptive and inferential exploratory data analysis and

modeling.


Quick Review: Intro to Data Science

● Data science is the practice of:○ Acquiring, organizing, and delivering complex data; discovering relationships

and anomalies among variables.○ Building and deploying machine learning models.○ Synthesizing data to influence decision-making.

● Specific Data Science Roles Include:○ Machine Learning Engineer○ Data Engineer○ Research Science○ Advanced Analyst

14 | © 2021 True Digital Academy Next: Data Science Workflow



Data Science Workflow

How Do We...

● Go through data science workflow?● Solve a data science problem?● Craft a data science problem statement?


focus on how to conduct exploratory data analysis (EDA)

The Data Science Workflow


Frame Prepare Analyze Interpret Communicate

Develop hypothesis-driven questions to your

analysis

Get, understand, explore, and

scrub your data

Structure, visualize to find

significant patterns and trend using statistical

methods, and complete your

analysis

Make recommendations and business decisions from

your data

Present insights from your data to audiences

Notes on the Steps

● Not hard-set rules.● Really, problem-solving guidelines.

Every problem is different!

● Some projects may not require every step.● It's normal to repeat certain steps a few times.● The process is cyclical with new findings!


Step 1 is Always "Frame the Problem"

Solving data science task starts with a clearly defined problem.

● Poor results stem from no defined goal.

“A problem well stated is half solved.” — Charles Kettering

From there, you can apply your steps.


The Data Science Workflow: Applied

You need to reduce the costs of staffing.

You have a table of DSW current retail sales associates across department stores.

The first three rows look like this:


Assume that clothing retail company, Data Science Wearables (DSW), is interested in improving their human resource operations.

Job Level Current Employee

Reason for Termination

Years of Service

Candidate Source

Previous Employer

School Time to Fill (Days)

Associate N New offer 1.5 Referral Jake's Hawaiian Shirts

University of Minnesota

40

Associate Y N/A 2.0 Internship N/A University of Iowa

15

Associate No Tardiness 0.5 Online Hats and Caps

University of Nebraska

25

Time to Fill (Days): How long did it take to fill this person's role? Typically minimizing time to fill is key to lower costs.

Step One: Frame

We know:

● We want to reduce costs associated with staffing.

We don't know:

● What drives up costs of staffing?● Is there an underlying reason for those costs?● What hypothesis can we test to reduce costs?


Step Two: Prepare

What questions do you have about the dataset?


Job Level Current Employee

Reason for Termination

Years of Service

Candidate Source

Previous Employer

School Time to Fill (Days)

Associate N New offer 1.5 Referral Jake's Hawaiian Shirts

University of Minnesota

40

Associate Y N/A 2.0 Internship N/A University of Iowa

15

Associate No Tardiness 0.5 Online Hats and Caps

University of Nebraska

25

N/A missing values

inconsistencies

Step Three: Analyze

We want to:

● Create meaning and conduct statistical description and inference.

For example, the average Years of Service is ~1.33 years.

● Could we build a machine learning model to predict this?● The data could center on their background (school, previous employers, and

application source).

For example, is the relationship between Time to Fill and Years of Service positive or negative?

● Positive: when one increases, the other increases.● Negative: when one increases, the other decreases.


Step Four: Interpret

How do our results compare to our initial hypothesis?

What concrete actions do we recommend?

Question: Even with an extremely limited dataset (n=3), can you identify hypothesis-validating or invalidating anecdotes?

At this stage, treat metrics and results like "check engine lights."

● Result summaries may point you in the right direction, but they do not necessarily explain the full context at hand.


Step Five: Communicate

Results are only as convincing as they are conveyed to key stakeholders!

Back up your statement with evidence, including statistical tests, visualizations, and model results.


Quick Review: Data Science Workflow

The data science workflow:


Frame Prepare Analyze Interpret CommunicateFrame Prepare Analyze Interpret Communicate

Develop hypothesis-driven questions to your

analysis

Get, import, understand, explore, and

scrub your data

Structure, visualize to find

significant patterns and trend using statistical

methods, and complete your

analysis

Make recommendations and business decisions from

your data

Present insights from your data to audiences

Next: Data Science Tools



Data Science Tools

Why Python for Data Science

Easy to write

● Data science is inherently a cross-functional discipline!● A language for all audiences is key.

Open source

● New techniques become available daily!● Developers from around the world race to implement new libraries.● This places Python in contrast to closed source, paid data analysis tools like SAS

and SPSS.

Often used for data analysis, scripting, and rapid software development.


Getting Data Science Tools

● We can analyze data to determine what Python is most used for:

Pandas?

● A Python package for exploratory analysis.

● Let's use it!


"panel data"

https://en.wikipedia.org/wiki/Panel_data

Your Data Science Development Tools

Python packages in DS are ubiquitous (universal):

● Reading CSVs, linear algebra, linear regressions, matrices...

Anaconda ("Conda"):

● Package manager.● Downloads everything for us!

Follow these steps:

1. Download Anaconda: https://www.anaconda.com/download/.Select Python 3.7+ for your machine (macOS or PC)

2. Open the file. Follow the on-screen prompts. Don't hesitate to ask questions!


https://www.anaconda.com/download/

What Are We Downloading?

Pandas:

● The default tool for data exploration and manipulation in Python.

Jupyter Notebooks and JupyterLab:

● The preferred integrated development environments (IDEs) of data science.

● We'll write our code in this!

NumPy, SciPy, and more:

● Other packages for statistical inference, visualization, and parallelizing operations.


https://docs.anaconda.com/anaconda/packages/py3.7_win-64/

Why Jupyter Notebook?

Data science is both code and methods

What if we're missing many values?

● Do you fill in missing values with the mean or the median?● Easy to create code cells next to text cells.

Easy to connect to remote computers (data centers).

● Thus, the Jupyter Notebook is in your browser!


Quick Review: Data Science Tools

● Pandas○ A Python package for exploratory analysis.

● Jupyter Notebooks and JupyterLab:○ The preferred integrated development environments (IDEs) of data

science.○ We'll write our code in this!

● Anaconda helps us download these. You only had to download it once!


Quick Review: Data Science Tools

● Data scientists:○ Use data of all kinds (numbers, text, images).○ Make explanations and predictive decisions.

● Data Science Workflow:○ Frame -> Prepare -> Analyze -> Interpret -> Communicate.

● Jupyter Notebooks:○ The industry tool!○ Interactive with Python.

35 | © 2021 True Digital Academy Next: Pandas


Introduction to Pandas:

Pandas

What is Pandas?

● A group of adorable bears 🐼🐼🐼● A Python library for data manipulation.


So, Pandas the Library

The Swiss Army Knife of data manipulation!

Pandas:

● Is the library for exploratory data analysis (EDA).● Formats, wrangles, cleans, and prepares our data.

Quick Backstory from 2009:

● A humble open source project for Panel Data (hence "Pandas") from Wes McKinney.● A 'panel' is the name of the object (in pandas) holding an n-dimensional numpy array● Don't let the term fool you, a panel is effectively the same thing as an excel workbook (a

collection of sheets)● A 2-dimensional panel is a Dataframe (rows and columns)● A 1-dimensional panel is a Series (column)


Quick Review: Pandas

● Exploratory Data Analysis (EDA) is the process of understanding our dataset, and producing our first level of insights.

● Pandas is a prominent Python library used for exploratory data analysis.

39 | © 2021 True Digital Academy Next: Series and DataFrame



Series and DataFrame

Import Pandas


import pandas as pd

Alias

Pandas’ Data structures

pandas provides two data structures that shape data into a readable form:

● Series● Data frame


Series

A pandas series is a one-dimensional data structure that comprises of a key-value pair.


pd.Series()

To initialize a series, use pd.Series():


import pandas as pd

##### INITIALIZATION #####

#STRING SERIES

fruits = pd.Series(["apples", "oranges", "bananas"])

print("MY FRUIT SERIES")

print(fruits, "\n")

MY FRUIT SERIES0 apples1 oranges2 bananasdtype: object

pd.Series()

change the key column to user-defined keys by passing a list to the index argument in the pd.Series() method.


import pandas as pd

##### INTIALIZATION #####

#FLOAT SERIES

temperature = pd.Series([32.6, 34.1, 28.0, 35.9], index = ["a","b","c","d"])

print("TEMPERATURE IN CELSIUS")

print(temperature, "\n")TEMPERATURE IN CELSIUSa 32.6b 34.1c 28.0d 35.9dtype: float64

Query a Series

To query a series, use .iloc[] or [] to query using the index/position of the value and .loc[] to query using the user-defined keys.


##### QUERY #####

#USING INDEX

print ("2nd fruit: ", fruits.iloc[1])

#OR

print ("2nd fruit: ", fruits[1], "\n")

#USING KEY

print ("temperature at key \"b\": ", temperature.loc["b"])

2nd fruit: oranges2nd fruit: oranges

temperature at key "b": 34.1

MY FRUIT SERIES0 apples1 oranges2 bananasdtype: object

TEMPERATURE IN CELSIUSa 32.6b 34.1c 28.0d 35.9dtype: float64

Dataframe

A pandas dataframe is a two-dimensional data-structure that can be thought of as a spreadsheet.

A dataframe can also be thought of as a combination of two or more series.


Dataframe

To initialize a dataframe, use pd.DataFrame:


import pandas as pd

##### INITIALIZATION #####

fruits_jack = ["apples", "oranges", "bananas"]

fruits_john = ["guavas", "kiwis", "strawberries"]

index = ["a", "b", "c"]

all_fruits = {"Jack's": fruits_jack, "John's": fruits_john}

fruits = pd.DataFrame(all_fruits, index = index)

print(fruits, "\n")

new_fruits = fruits.reset_index(drop = True)

print(new_fruits, "\n")

Jack's John'sa apples guavasb oranges kiwisc bananas strawberries

Jack's John's0 apples guavas1 oranges kiwis2 bananas strawberries

fruits

new_fruits

Query a Dataframe


##### QUERY #####

#USING INDEX

print("1st fruit:")

print(new_fruits.iloc[0], "\n")

#USING KEY

print("Fruits at key \"c\":")

print(fruits.loc["c"], "\n")

#USING COLUMN NAME

print("Jack's fruits: ")

print(fruits["Jack's"], "\n")

#CHAINED QUERY

print("Johns third fruit: ")

print(new_fruits["John's"][2], "\n")

Jack's John'sa apples guavasb oranges kiwisc bananas strawberries

Jack's John's0 apples guavas1 oranges kiwis2 bananas strawberries

fruits

new_fruits

1st fruit:Jack's applesJohn's guavasName: 0, dtype: object

Fruits at key "c":Jack's bananasJohn's strawberriesName: c, dtype: object

Jack's fruits:a applesb orangesc bananasName: Jack's, dtype: object

Johns third fruit:strawberries

Quick Review: Series and DataFrame

● Pandas provides two data structures that shape data into a readable form:○ Series is a one-dimensional data structure.○ Data frame is a two-dimensional data-structure.

● To query a series or dataframe, use iloc() or loc.

50 | © 2021 True Digital Academy Next: Data Input and Validation



Data Input and Validation

Data Input

Pandas provides the easiest way to get data into a Python program is to read it from a file.

Pandas can read lots of kinds of files: csv, xls, xlsx, and so on.


Reading a Dataset

import pandas library and then read the dataset


# importing pandas package

import pandas as pd

# making data frame from csv file

df = pd.read_csv("nba.csv", index_col ="Name")

dataframe.shape

The shape property is used to get a tuple representing the dimensionality of the DataFrame.


df.shape(458, 8)

Number of Rows

Number of Columns

Getting the First n Rows Using head()

The head() function is used to get the first n rows. By default, it returns the first 5 rows of the Dataframe.


df.head()

Getting Last n Rows Using tail()

The tail() function is used to get the last n rows.


df.tail()

Printing Info about the DataFrame Using info()

The information contains the number of columns, column labels, column data types, memory usage, range index, and the number of cells in each column (non-null values).


df.info()<class 'pandas.core.frame.DataFrame'>Index: 458 entries, Avery Bradley to nanData columns (total 8 columns):# Column Non-Null Count Dtype--- ------ -------------- -----0 Team 457 non-null object1 Number 457 non-null float642 Position 457 non-null object3 Age 457 non-null float644 Height 457 non-null object5 Weight 457 non-null float646 College 373 non-null object7 Salary 446 non-null float64dtypes: float64(4), object(4)

Changing Column Name

● Rename column / index name (label): rename()● Change multiple names (labels)● Update the original object: inplace


df_new = df.rename(columns={'Team': 'BasketTeam'},

index={'Jeff Withey':

'changed_index'})

print(df_new)




print(df.rename(columns={'Team': 'Col_1', 'Position': 'Col_3'}))




df_copy = df.copy()

df_copy.rename(columns={'Team': 'Col_1'}, index={'Avery Bradley': 'Row_1'}, inplace=True)

print(df_copy)

Quick Review: Data Input and Validation

● Pandas provides the easiest way to get data into a Python program is to read it from a file and get information about data:○ Read files using pd.read_csv()○ Get dimensionality of dataframe using df.shape○ Get first rows and last rows using df.head(), df.tail() respectively○ Get information about dataframe using df.info()○ Change column names and label using df.rename()

61 | © 2021 True Digital Academy Next: Basic Analysis



Basic Analysis

Summary Statistics using .describe()

The describe() function is used to generate descriptive statistics that summarize the central tendency, dispersion, and shape of a dataset’s distribution, excluding NaN values.


data.describe()count

mean

std

min

25%

50%

75%

max

3.0

3.0

1.0

2.0

2.5

3.0

3.5

4.0

s = pd.Series ( [ 2, 3, 4 ] )s.describe ( )

3 numbers

mean or average

Standard Deviation

minimum value

25th percentiles

50th percentiles

75th percentiles

maximum value

Describing a Categorical Series


count

unique

top

freq

4

3

p

2

s = pd.Series ( [ ‘p’, ‘p’, ‘q’, ‘r’ ] )s.describe ( )

4 letters

unique letters 3 (p,q,r)

top letter is p

p has highest freq (2)

Describing a Timestamp Series


import numpy as np

import pandas as pd

s = pd.Series([

np.datetime64("2018-02-01"),

np.datetime64("2019-02-01"),

np.datetime64("2019-02-01")])

s.describe()

count 3unique 2top 2019-02-01 00:00:00freq 2first 2018-02-01 00:00:00last 2019-02-01 00:00:00dtype: object

mean

The mean() function is used to return the mean of the values for the requested axis.

axis: {index (0), columns (1)}.


import pandas as pd

info = pd.DataFrame({"A":[8, 2, 7, 12, 6],

"B":[26, 19, 7, 5, 9],

"C":[10, 11, 15, 4, 3],

"D":[16, 24, 14, 22, 1]})

print(info)

print("*************")

info.mean(axis = 0)

A B C D0 8 26 10 161 2 19 11 242 7 7 15 143 12 5 4 224 6 9 3 1*************A 7.0B 13.2C 8.6D 15.4dtype: float64

sum

DataFrame.sum() function is used to return the sum of the values for the requested axis.

axis: {index (0), columns (1)}


import pandas as pd


"B":[26, 19, 7, 5, 9],

"C":[10, 11, 15, 4, 3],

"D":[16, 24, 14, 22, 1]})

print("The sum of A is ", info['A'].sum())

The sum of A is 35

max

DataFrame.max() function is used to find the maximum value along the axis.


import pandas as pd


"B":[26, 19, 7, 5, 9],

"C":[10, 11, 15, 4, 3],

"D":[16, 24, 14, 22, 1]})

print("The max of A is ", info['A'].max())

The max of A is 12

Get Maximum Values of Every Column


import pandas as pd


"B":[26, 19, 7, 5, 9],

"C":[10, 11, 15, 4, 3],

"D":[16, 24, 14, 22, 1]})

print("The max of A is", info.max())

The max of A is A 12B 26C 15D 24

Get Maximum Values of Every Row


import pandas as pd


"B":[26, 19, 7, 5, 9],

"C":[10, 11, 15, 4, 3],

"D":[16, 24, 14, 22, 1]})

print(info.max(axis=1))

0 261 242 153 224 9

Next: value_counts()

value_counts

The value_counts() function returns an object containing counts of unique values.

Let’s explore titanic datasets.


# Importing necessary libraries

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline

df = pd.read_csv('titanic.csv')

Titanic DatasetVARIABLE DESCRIPTIONS

1. Pclass Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd) 2. Survived Survival (0 = No; 1 = Yes) 3. Name Name 4. Sex Sex 5. Age Age 6. SibSp Number of Siblings/Spouses Aboard 7. Parch Number of Parents/Children Aboard 8. Ticket Ticket Number 9. Fare Passenger Fare (British pound) 10. Cabin Cabin 11. Embarked Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton) 12. Boat Lifeboat 13. Body Body Identification Number 14. Home.dest Home/Destination


df.head()

Calling the value_counts() on the Sex Column of the Dataset


df['Sex'].value_counts() male 577female 314

S 644C 168Q 77

df['Embarked'].value_counts()

We can quickly see that the maximum people embarked from Southampton, followed by Cherbourg and then Queenstown.

value_counts() with Relative Frequencies of the Unique Values

Sometimes, getting a percentage of the total is a better criteria than the count.

By setting normalize =True, the object returned will contain the relative frequencies of the unique values.


df['Embarked'].value_counts(normalize=True) S 0.724409C 0.188976Q 0.086614

value_counts() in Ascending Order

sort the results obtained in ascending order, simply set the ascending parameter to True


df['Embarked'].value_counts(ascending=True) Q 77C 168S 644

value_counts() with NaN Values

By default, count of null values are excluded. However, this can be reversed by setting dropna=False.


S 644C 168Q 77NaN 2

df['Embarked'].value_counts(dropna=False)

value_counts() with Bins

value_counts() can also be used to bin continuous data into discrete intervalswith the help of bin parameter.

This option works only with numerical data.


# applying value_counts on a numerical

column

df['Fare'].value_counts()

This doesn't convey much since the function above has given a count of every available Fare amount. Instead, let's group them into 7 bins

Grouping Fares into 7 Bins.


df['Fare'].value_counts(bins=7)

We can easily see that most of the people out of the total population paid less than 73.19 for their ticket.

Next: sort_values()

Sorting Data Frame Using sort_values()

Pandas sort_values() function sorts a data frame in Ascending or Descending order of passed Column.


Sorting by Name


# sorting data frame by name

df.sort_values("Name", axis = 0, ascending = True,

inplace = True, na_position ='last')

# display

df

Changing Position of Null Values

Null values are kept at the top


# sorting data frame by Cabin

df.sort_values("Cabin", axis = 0, ascending = True,

inplace = True, na_position ='first')

# display

df

Next: Boolean masking

Boolean Masking

Boolean masking is used to filter a data.

Let’s filter Titanic datasets again!



import pandas as pd

import numpy as np



%matplotlib inline


Masking Data Based on Column Value

Using a comparison operator for filtering of data: ==, >, <, <=, >=



import pandas as pd

import numpy as np



%matplotlib inline


print(df.shape)

#mask

print(df['Age'] >25)

(891, 12)0 False1 True2 True3 True4 True

...886 True887 False888 False889 True890 TrueName: Age, Length: 891, dtype: bool

Applying a Boolean Mask (age > 25)

When we apply a boolean mask it will print only that dataframe in which we pass a boolean value True.



import pandas as pd

import numpy as np



%matplotlib inline


print(df.shape)

mask = df['Age'] > 25

print(df[mask])

Applying a Boolean Mask (Embarked == ‘S’)


mask = df['Embarked'] == 'S'

print(df[mask]) # print(df[df['Embarked'] == 'S'])

Combine Conditions Using Boolean Operators

Python's bitwise logic operators, & (and), | (or), ^ (xor), and ~(not).


mask = (df['Embarked'] == 'S') | (df["Age"] > 25)

print(df[mask])

# print(df[(df['Embarked'] == 'S') | (df["Age"] > 25)])

or

Next: String Manipulation

String Manipulations in DataFrame

Series.str can be used to access the values of the series as strings and apply several methods to it.

Pandas Series.str.contains() function is used to test if pattern or regex is contained within a string of a Series or Index.


import pandas as pd

df = pd.DataFrame({

'name': ['alice smith','bob jones','charlie joneson','daisy white'],

'age': [25,20,30,35]

})

Select by Partial String


import pandas as pd

df = pd.DataFrame({


'age': [25,20,30,35]

})

df[df['name'].str.contains('jones',regex=False)]

.str.contains('jones')

Set regex=False for better performance

Select by Regular Expression


import pandas as pd

df = pd.DataFrame({


'age': [25,20,30,35]

})

# names starting with 'b' or 'd'

df[df['name'].str.contains('^b|d')]

Concatenate String Columns


import pandas as pd

df = pd.DataFrame({

'first_name': ['alice','bob','charlie','daisy'],

'last_name':['smith','jones','joneson','white'],

'age': [25,20,30,35]

})

# just add the two columns

df['full_name'] = df['first_name'] + df['last_name']

df

Next: apply()

apply()

Pandas.apply allow the users to pass a function and apply it on every single value of the Pandas series.


Split String Column


import pandas as pddf = pd.DataFrame({

'name': ['alice smith','bob jones','charlie joneson','daisy white'],'age': [25,20,30,35]

})

# a function that takes the value and returns# a series with as many columns as you wantdef split_name(name):

first_name, last_name = name.split(' ')

return pd.Series({'first_name': first_name,'last_name': last_name

})

# df_new has the new columnsdf_new = df['name'].apply(split_name)df_new

df_new

df

Split String Column


import pandas as pd

df = pd.DataFrame({'name': ['alice smith','bob jones','charlie joneson','daisy white'],'age': [25,20,30,35]

})

# a function that takes the value and returns# a series with as many columns as you wantdef split_name(name):

first_name, last_name = name.split(' ')

return pd.Series({'first_name': first_name,'last_name': last_name

})

# df_new has the new columnsdf_new = df['name'].apply(split_name)

# append the columns to the original dataframedf_final = pd.concat([df,df_new],axis=1)

df_new

df

df_final

Two new columns were createdby splitting full_name into two

Quick Review: Basic Analysis

● Summary statistics using .describe()● The mean() function is used to return the mean of the values for the requested axis.● DataFrame.sum() function is used to return the sum of the values for the requested

axis.● DataFrame.max() function is used to find the maximum value along the axis.● The value_counts() function returns an object containing counts of unique values.● Pandas sort_values() function sorts a data frame in Ascending or Descending order

of passed Column.● Boolean masking is used to filter a data.● Series.str can be used to access the values of the series as strings and apply

several methods to it.


Module Summary

We’ve learned to:

● Understand what data science is and the roles needed in data science● Apply the data science workflow● Understand programming tools for data science and how Pandas fit in ● Use Pandas to read in a dataset● Get data information and summary statistics● Investigate a dataset's integrity● Filter, sort, and manipulate DataFrame series



THANK YOU!See you next time!

module 5 introduction to pandas

Documents