lesson 3: data analysis · lesson 3: data analysis. today’s agenda csv data pandas review....
TRANSCRIPT
LESSON 3: DATA ANALYSIS
TODAY’S AGENDA
CSV Data
Pandas
Review
VARIABLES
LISTS
RANGE FUNCTION TO GENERATE LIST
OPERATORS
LOOPS
CONDITIONALS
STEP 1:OPEN ANACONDA
STEP 2:LAUNCH JUPYTER
EXERCISE 1:
Write code to update a list containing the values [1,2,3,4,5] to [0,0,0,1,1].
EXTRA:Turn your solution into a reusable function so that you can pass it any list and apply the same logic to it.
EXERCISE 1 SOLUTION:
EXERCISE 1 SOLUTION (USING FUNCTION):
• Package for data manipulation and analysis• Optimized for data tables (Excel and csv files)• Library features: DataFrame, grouping, filtering• Efficient and easy to use
HOW DO YOU ANALYZE THIS DATA?
Potential issues:• Missing values• Metadata• Categorical data• Variance• Range• Outliers• Amount of data
Review of important concepts from lesson 2
Variables
In [1]:
Lists
In [2]:
Using range() function to generate list
In [3]:
Operators
In [4]:
a + b = 8
a = 1 # integerb = 2.57 # floatc = 'neurodev' # stringd = True # boolean
a = [1,2,3] # list of integersb = [2.57, 56.3, 9.71] # list of floatsc = ['neurodev', 'bch', 'hms'] # list of stringsd = [True, False, True] # list of booleanse = [1, 2.57, 'neurodev', True] # list of elements of various data types
# range(start, stop, step)a = range(5) # [0,1,2,3,4]b = range(1,5) # [1,2,3,4]c = range(1,10,2) # [1,3,5,7,9]
# + operator on INTEGERSa = 3b = 5print('a + b =', a + b)
In [5]:
In [6]:
Loops
In [7]:
In [8]:
Conditionals
a + b = welcome to neurodev
a + b = [1, 2, 3, 4, 5, 6]
012
012
# + operator on STRINGSa = 'welcome to'b = ' neurodev'print('a + b =', a + b)
# + operator on LISTSa = [1,2,3]b = [4,5,6]print('a + b =', a + b)
# for loopfor i in range(3): print(i)
# while loopi = 0while i < 3: print(i) i = i + 1
In [9]:
Exercise 1: Write code to update a list containing the values[1,2,3,4,5] to [0,0,0,1,1].
Print the content of the list before and after the update.Not allowed to simply overwrite the original list with the new one.Extra: Turn your solution into a function.
Solution:
In [10]:
Turning solution into a reusable function:
x is greater than 3
mylist (ORIGINAL) = [1, 2, 3, 4, 5]mylist (UPDATED) = [0, 0, 0, 1, 1]
x = 5if x > 3: print('x is greater than 3')elif x > 1: print('x is greater than 1, but not greater than 3')else: print('x is 1 or less')
# assign initial values to a listmylist = [1,2,3,4,5]print('mylist (ORIGINAL) =', mylist) # update values in list using a for loop with conditional statementslength_of_mylist = len(mylist)for i in range(length_of_mylist): # range(length_of_mylist) = range(5) = [0,1,2,3,4] if mylist[i] <= 3: mylist[i] = 0 else: mylist[i] = 1print('mylist (UPDATED) =', mylist)
In [11]:
In [12]:
Lesson 3 Material: Analyzing Data Stored in a .csvFile
Reading data from csv file
In [13]:
In [14]:
Displaying data of interest
mylist (ORIGINAL) = [1, 2, 3, 4, 5]mylist (UPDATED) = [0, 0, 0, 1, 1]
# defining a function to solve this problemdef is_greater_than_3(mylist): length_of_mylist = len(mylist) for i in range(length_of_mylist): if mylist[i] <= 3: mylist[i] = 0 else: mylist[i] = 1 return mylist # to return variable 'mylist'
# invoking our function with list [1,2,3,4,5] as inputmylist = [1,2,3,4,5]print('mylist (ORIGINAL) =', mylist) mylist = is_greater_than_3(mylist)print('mylist (UPDATED) =', mylist)
# import required librariesimport pandas as pd
# read csv filedf = pd.read_csv('celldata.csv')
In [15]:
cell_id genotype gene1 gene2 gene3 gene4 gene5 gene6 \0 1 WILDTYPE -21.150000 26.19 15.820 37.1704 87.4939 78.1250 1 2 WILDTYPE -58.960000 NaN 8.000 46.6919 93.8721 101.0130 2 3 WILDTYPE -50.690000 10.64 8.100 40.7715 79.4067 73.8525 3 4 WILDTYPE -45.870000 14.59 8.500 30.1819 70.8008 46.0815 4 5 WILDTYPE -67.440000 20.84 6.200 29.8462 64.4226 43.9453 5 6 WILDTYPE -27.710000 16.00 8.600 20.7520 56.1829 33.8745 6 7 WILDTYPE -46.300000 14.41 5.900 35.0037 80.7800 49.7437 7 8 WILDTYPE -31.340000 17.40 6.000 34.6680 72.6929 57.3730 8 9 MUTANT -42.660000 96.51 6.050 48.3398 98.8159 146.4840
# print all dataprint(df)
In [16]:
cell_id genotype gene1 gene2 gene3 gene4 gene5 gene6 \0 1 WILDTYPE -21.15 26.19 15.82 37.1704 87.4939 78.1250 1 2 WILDTYPE -58.96 NaN 8.00 46.6919 93.8721 101.0130 2 3 WILDTYPE -50.69 10.64 8.10 40.7715 79.4067 73.8525 3 4 WILDTYPE -45.87 14.59 8.50 30.1819 70.8008 46.0815 4 5 WILDTYPE -67.44 20.84 6.20 29.8462 64.4226 43.9453
gene7 gene8 ... gene81 gene82 gene83 gene84 \0 -36.3159 6.31714 ... 8.896188 8.206619 10.872804 8.199322 1 -42.1143 18.24950 ... 7.928045 8.852913 10.948357 7.912894 2 -20.7520 11.74930 ... 0.000000 8.235151 11.030531 8.752535 3 -22.5830 10.37600 ... 7.791213 8.666661 9.781027 8.028453 4 -20.7520 11.16940 ... 7.489289 9.043653 10.757345 6.617701
gene85 gene86 gene87 gene88 gene89 gene90 0 9.466440 9.372212 9.637256 7.080700 11.706856 9.854501 1 9.334569 10.136471 8.620111 6.760025 11.693019 6.307249 2 9.502819 9.549367 9.716100 3.310809 11.467424 17.461985 3 8.046876 8.870417 9.024922 5.757125 10.592519 10.740293 4 8.632594 8.905817 9.194336 5.587619 10.696437 9.388160
[5 rows x 92 columns]
# print first 5 rowsprint(df.head())
In [17]:
In [18]:
Index(['cell_id', 'genotype', 'gene1', 'gene2', 'gene3', 'gene4', 'gene5', 'gene6', 'gene7', 'gene8', 'gene9', 'gene10', 'gene11', 'gene12', 'gene13', 'gene14', 'gene15', 'gene16', 'gene17', 'gene18', 'gene19', 'gene20', 'gene21', 'gene22', 'gene23', 'gene24', 'gene25', 'gene26', 'gene27', 'gene28', 'gene29', 'gene30', 'gene31', 'gene32', 'gene33', 'gene34', 'gene35', 'gene36', 'gene37', 'gene38', 'gene39', 'gene40', 'gene41', 'gene42', 'gene43', 'gene44', 'gene45', 'gene46', 'gene47', 'gene48', 'gene49', 'gene50', 'gene51', 'gene52', 'gene53', 'gene54', 'gene55', 'gene56', 'gene57', 'gene58', 'gene59', 'gene60', 'gene61', 'gene62', 'gene63', 'gene64', 'gene65', 'gene66', 'gene67', 'gene68', 'gene69', 'gene70', 'gene71', 'gene72', 'gene73', 'gene74', 'gene75', 'gene76', 'gene77', 'gene78', 'gene79', 'gene80', 'gene81', 'gene82', 'gene83', 'gene84', 'gene85', 'gene86', 'gene87', 'gene88', 'gene89', 'gene90'], dtype='object')
0 WILDTYPE1 WILDTYPE2 WILDTYPE3 WILDTYPE4 WILDTYPE5 WILDTYPE6 WILDTYPE7 WILDTYPE8 MUTANT9 MUTANT10 MUTANT11 MUTANT12 MUTANT13 MUTANT
# print list of column headersprint(df.columns)
# print content of specified column, here 'genotype' columnprint(df['genotype'])
In [19]:
14 MUTANT15 MUTANT16 WILDTYPE17 WILDTYPE18 WILDTYPE19 WILDTYPE20 MUTANT21 MUTANT22 MUTANT23 MUTANT24 MUTANT25 WILDTYPE26 WILDTYPE27 WILDTYPE28 WILDTYPE29 WILDTYPE ... 62 WILDTYPE63 WILDTYPE64 WILDTYPE65 WILDTYPE66 WILDTYPE67 WILDTYPE68 WILDTYPE69 MUTANT70 MUTANT71 MUTANT72 MUTANT73 WILDTYPE74 WILDTYPE75 WILDTYPE76 WILDTYPE77 WILDTYPE78 WILDTYPE79 WILDTYPE80 WILDTYPE81 WILDTYPE82 WILDTYPE83 WILDTYPE84 MUTANT85 MUTANT86 MUTANT87 MUTANT88 MUTANT89 MUTANT90 MUTANT91 MUTANTName: genotype, Length: 92, dtype: object
cell_id 1
# print content of a 1st row (recall that Python starts counting at 0)print(df.iloc[0])
cell_id 1genotype WILDTYPEgene1 -21.15gene2 26.19gene3 15.82gene4 37.1704gene5 87.4939gene6 78.125gene7 -36.3159gene8 6.31714gene9 18.7683gene10 -50.3235gene11 2.48824gene12 1.55gene13 -34.98gene14 134gene15 72.8gene16 -8.442gene17 -61.32gene18 3gene19 24.89gene20 30gene21 3gene22 35.61gene23 71.11gene24 -36.32gene25 2.582gene26 1.65gene27 -35.5gene28 76.2 ... gene61 6.0403gene62 5.91573gene63 6.39201gene64 5.00435gene65 5.23268gene66 4.98107gene67 0gene68 0gene69 0gene70 0gene71 7.08311gene72 7.39265gene73 6.49876gene74 11.9055gene75 5.3892gene76 5.78466gene77 0gene78 10.9963gene79 8.28808gene80 7.6703gene81 8.89619gene82 8.20662gene83 10.8728
Handling missing values
Common approaches for dealing with missing values include dropping the cell or the column with such entries,or replacing the undefined values with some other values (0, mean of feature, ...).
In [20]:
Computing statistics
In [21]:
In [22]:
gene84 8.19932gene85 9.46644gene86 9.37221gene87 9.63726gene88 7.0807gene89 11.7069gene90 9.8545Name: 0, Length: 92, dtype: object
gene1 mean = -51.335224823478214gene1 std = 10.074972534348717
cell_id gene1 gene2 gene3 gene4 gene5 \count 92.000000 92.000000 92.000000 92.000000 92.000000 92.000000 mean 50.413043 -51.335225 28.375543 8.279772 39.469518 82.697657 std 29.822977 10.074973 17.896217 2.355555 11.561944 15.926417 min 1.000000 -70.340000 0.000000 4.060000 -0.701904 39.001500 25% 23.750000 -58.682500 17.400000 6.586000 33.996575 73.471100 50% 52.500000 -51.265000 21.850000 7.900000 42.037950 86.517350 75% 76.250000 -44.587500 32.170000 9.685750 46.951275 93.09
# here we replace missing values with 0 across the entire data tabledf = df.fillna(0)
# statistics for single colprint('gene1 mean =', df['gene1'].mean())print('gene1 std =', df['gene1'].std())
# statistics for all colsprint(df.describe())
3900 max 100.000000 -21.150000 100.600000 15.820000 61.920200 117.462000
gene6 gene7 gene8 gene9 ... gene81 \count 92.000000 92.000000 92.000000 92.000000 ... 92.000000 mean 84.974858 -31.827846 11.630179 20.592067 ... 8.889962 std 38.083474 15.271497 9.679855 9.035605 ... 1.387891 min 28.076200 -80.261200 -22.155800 -8.605960 ... 0.000000 25% 56.838950 -41.809100 9.902973 16.654975 ... 8.538428 50% 81.176750 -29.144300 14.236450 21.881150 ... 9.013163 75% 106.048500 -20.294200 18.280000 27.000400 ... 9.630572 max 234.070000 -4.577640 26.031500 37.170400 ... 11.209544
gene82 gene83 gene84 gene85 gene86 gene87 \count 92.000000 92.000000 92.000000 92.000000 92.000000 92.000000 mean 8.297394 11.261041 7.683915 9.151266 9.734449 9.017124 std 1.681904 0.889013 1.374126 1.354221 0.938592 1.538914 min 0.000000 8.908334 0.000000 5.933096 7.836979 0.000000 25% 8.084441 10.818807 7.130225 8.714562 9.284171 8.618571 50% 8.642634 11.175816 7.919103 9.335387 9.729373 9.210607 75% 9.130326 11.597686 8.496908 9.642977 10.138891 9.745609 max 9.874828 16.598447 11.292849 18.037770 14.451070 14.514292
gene88 gene89 gene90 count 92.000000 92.000000 92.000000 mean 4.578790 11.268790 9.401092 std 2.308754 0.617138 3.532641 min 0.000000 7.412216 0.000000 25% 3.825207 11.025496 7.406899 50% 5.278505 11.326811 9.158637 75% 6.114384 11.592893 10.740873 max 10.405329 12.894671 29.278119
[8 rows x 91 columns]
In [23]:
Data filtering (selecting subset of data that meet some criteria)
In [24]:
gene1 gene2 gene3count 92.000000 92.000000 92.000000mean -51.335225 28.375543 8.279772std 10.074973 17.896217 2.355555min -70.340000 0.000000 4.06000025% -58.682500 17.400000 6.58600050% -51.265000 21.850000 7.90000075% -44.587500 32.170000 9.685750max -21.150000 100.600000 15.820000
cell_id genotype gene1 gene2 gene3 gene4 gene5 gene6 \17 18 WILDTYPE -68.24 27.52 5.899 53.7415 102.8440 129.3950 33 35 MUTANT -70.34 32.17 9.825 41.1682 88.4705 82.7026 53 60 MUTANT -69.85 79.16 8.020 37.6282 78.0029 52.1851
gene7 gene8 ... gene81 gene82 gene83 gene84 \17 -37.8418 17.6086 ... 9.431388 7.524728 10.588949 8.231606 33 -36.3159 14.0076 ... 8.947872 0.000000 11.651503 7.475997 53 -27.1606 18.7073 ... 9.392386 8.933557 11.672922 9.669825
gene85 gene86 gene87 gene88 gene89 gene90 17 9.368081 10.719106 10.582344 3.924102 11.506342 9.415911 33 7.497953 10.992884 9.472242 4.357453 10.022963 12.792767 53 9.129298 9.597833 9.187616 5.693993 11.090065 6.011587
[3 rows x 92 columns]
# statistics for subset of columns, here for the first 3 genescolumns = ['gene1', 'gene2','gene3']print(df[columns].describe())
# select rows where gene1 value is below -68myfilter = df['gene1'] < -68 # boolean array [0,1,0,0,0,1,...]rows = df.loc[myfilter] # subset of rows # display selected rowsprint(rows)
Exercise 2: Select rows where gene5 is less than 120 and gene6 isgreater than 80.
How many cells meet this criteria?Extra: Compute statistics (mean, std, min, max) for this subset of the data.
In [25]:
In [26]:
cell_id genotype gene1 gene2 gene3 gene4 gene5 gene6 \1 2 WILDTYPE -58.96 0.00 8.000 46.6919 93.8721 101.0130 8 9 MUTANT -42.66 96.51 6.050 48.3398 98.8159 146.4840 10 11 MUTANT -59.27 11.65 5.900 49.2554 92.2241 112.3050 12 13 MUTANT -39.64 100.60 6.970 43.5486 90.6372 88.5010 16 17 WILDTYPE -37.90 12.30 10.220 56.0913 101.3180 169.6780 17 18 WILDTYPE -68.24 27.52 5.899 53.7415 102.8440 129.3950 18 19 WILDTYPE -46.72 32.17 5.400 59.4482 114.4100 195.3130 20 21 MUTANT -53.28 16.70 5.070 47.7905 95.6116 114.7460 21 22 MUTANT -58.23 25.07 4.649 45.5933 99.0601 118.4080
cell_id gene1 gene2 gene3 gene4 gene5 \count 47.000000 47.000000 47.000000 47.000000 47.000000 47.000000
# APPROACH 1: Apply filters sequentially # Step 1: Select rows where gene5 value is less than 120 myfilter = df['gene5'] < 120rows = df.loc[myfilter] # Step 2: From selected rows above, select rows where gene6 value is greater than 80myfilter = rows['gene6'] > 80rows = rows[myfilter] # Step 3: Display selectionprint(rows)
# Extra: compute statistics of selected rowsprint(rows.describe())
mean 57.085106 -52.406596 29.993830 8.072489 47.822345 94.042172 std 29.650009 9.061780 19.372939 2.315340 5.308260 8.832517 min 2.000000 -70.340000 0.000000 4.649000 34.576400 75.592000 25% 32.000000 -59.095000 19.220000 6.275000 44.097900 88.455200 50% 63.000000 -53.280000 24.630000 7.250000 46.905500 92.834500 75% 82.000000 -46.630000 32.245000 9.912500 50.720200 98.907450 max 100.000000 -34.000000 100.600000 13.830000 61.920200 117.462000
gene6 gene7 gene8 gene9 ... gene81 \count 47.000000 47.000000 47.000000 47.000000 ... 47.000000 mean 112.765672 -40.887064 17.007379 26.097070 ... 9.080744 std 31.289177 14.029070 4.184558 5.224466 ... 1.075076 min 80.566400 -80.261200 4.577640 13.275100 ... 5.737819 25% 90.179450 -48.065200 14.480550 22.827150 ... 8.714389 50% 105.896000 -40.893600 17.883300 26.428200 ... 9.250803 75% 121.002000 -34.179700 19.439700 28.732300 ... 9.696051 max 234.070000 -14.343300 26.031500 37.170400 ... 11.209544
gene82 gene83 gene84 gene85 gene86 gene87 \count 47.000000 47.000000 47.000000 47.000000 47.000000 47.000000 mean 8.271925 11.322943 8.027217 9.272756 9.891588 9.249585 std 2.106299 0.948713 0.958657 1.616155 1.010744 1.270638 min 0.000000 10.259384 5.241047 6.125413 7.836979 6.408702 25% 8.124656 10.851668 7.654979 8.831899 9.563989 8.758548 50% 8.852913 11.155357 8.079568 9.334569 9.874963 9.361260 75% 9.479372 11.560035 8.394659 9.681676 10.283700 9.728760 max 9.874828 16.598447 11.292849 18.037770 14.451070 14.514292
Save selection to a file
In [27]:
In [ ]:
gene88 gene89 gene90 count 47.000000 47.000000 47.000000 mean 4.811167 11.214948 8.742306 std 2.276253 0.724308 4.000292 min 0.000000 7.412216 0.000000 25% 4.147882 11.048588 6.783953 50% 5.136001 11.286992 8.281450 75% 6.293679 11.607117 9.968925 max 10.405329 12.448834 29.278119
[8 rows x 91 columns]
# save data to CSV filerows.to_csv('output.csv')