Upload File

ds280 - spring 2018 - final project presentation - olga ... · ô ] } À Ç ry µ Ç î t z ] z ] l...

  • Home
  • Documents
  • DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v
20
DS 280 – SPRING 2018 Big Data Architecture Final Project Olga Poleleyeva 1

Upload: others

Post on 05-Aug-2020

0 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

DS 280 – SPRING 2018

Big Data ArchitectureFinal Project

Olga Poleleyeva

1

Page 2: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

2

Bicycle sharing and weather conditions

Goal: To analyze Chicago bicycle sharing data in relations to available

weather information, to find possible correlations between the number,

frequency or duration of bicycle trips and various weather conditions or other variables in order to improve

the program.

Page 3: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

3

Dataset Details

• File Type: CSV• File Size: 3.24 GB• Data Source: Kagglehttps://www.kaggle.com/yingwurenjian/chicago-divvy-bicycle-sharing-data

Dataset contains the bicycle sharing data in Chicago from 2013 to2017 from Divvy website, combined with weather informationfrom Wunderground website.There are total of 13,774,715 records, in 27 columns. Each trip isrecorded with start/end place and time, duration, and additionalbike station’s info. The detailed weather info includedtemperature, visibility, precipitation, and so on.

Page 4: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

4

DB & Table Details

• Database Name: user13_final_db

• Table Name: bikeshare_initial_tbl

• Table Type: external• Field Names were exactly

as in dataset• Initial table had

information that was not intended to be used for this data analysis, so the new table was created.

Page 5: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

5

Adjusted Table Details

• Table Name: bikeshare_final_tbl

• Table Type: internal• Fields were adjusted

accordingly:- In order to make results more user-

friendly, trip duration was re-calculated from seconds into minutes

- In order to get more meaningful analysis, all trips shorter than 5 minutes were excluded

- In order to get the table size easier to manage, some weather and stations info fields were dropped

Page 6: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

6

Page 7: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

7

Data Discovery - Query 1

How many times were the bikes returned on the same day, on the next

day, and so on?

SELECTDATEDIFF(stoptime, starttime) as trip_days, COUNT(trip_id) as number_of_tripsFROM bikeshare_final_tblGROUP BY DATEDIFF(stoptime, starttime);

Results:0 11,838,936

1 46,111

Page 8: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

8

Data Discovery - Query 2

What is the bike rental distribution depending on the temperature ranges?

SELECT temp_range, FORMAT_NUMBER(COUNT(trip_id),0) AS number_of_tripsFROM(SELECT trip_id, temperature,CASE WHEN temperature > 85 THEN '86F and above: VERY HOT'

WHEN temperature > 76 THEN '77F - 85F: HOT'WHEN temperature > 54 THEN '55F - 76F: WARM'WHEN temperature > 32 THEN '33F - 54F: COOL'WHEN temperature > 0 THEN '1F - 32F: COLD'

ELSE '0F and below: VERY COLD'END AS temp_rangeFROM bikeshare_final_tbl)AS trips_by_rangeGROUP BY temp_rangeORDER BY temp_range DESC;

Page 9: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

9

Data Discovery - Query 2

Results: bike rental distribution depending on the temperature ranges

01,000,0002,000,0003,000,0004,000,0005,000,0006,000,000

86F andabove:

VERY HOT

77F - 85F:HOT

55F - 76F:WARM

33F - 54F:COOL

1F - 32F:COLD

0F andbelow:VERYCOLD

Num

ber o

f trip

sTemperature Range

Number of trips by temperature range

Page 10: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

10

Data Discovery - Query 3

What is the bike rental distribution depending on time of day?

SELECT time_range, COUNT(trip_id) AS number_of_tripsFROM(SELECT trip_id, HOUR (starttime) AS start_hour,

CASE WHEN HOUR (starttime) > '05' AND HOUR (starttime) <= '08' THEN '1. 6:00 am - 8:59 am: morning

commute'WHEN HOUR (starttime) > '08' AND HOUR (starttime) <= '10' THEN '2. 9:00 am - 10:59 am: morning'WHEN HOUR (starttime) > '10' AND HOUR (starttime) <= '13' THEN '3. 11:00 pm - 1:59 pm: lunch

break'WHEN HOUR (starttime) > '13' AND HOUR (starttime) <= '15' THEN '4. 2:00 pm - 3:59 pm: afternoon'WHEN HOUR (starttime) > '15' AND HOUR (starttime) <= '18' THEN '5. 4:00 pm - 6:59 pm: evening

commute'WHEN HOUR (starttime) > '18' AND HOUR (starttime) <= '20' THEN '6. 7:00 pm - 8:59 pm: dinner time'WHEN HOUR (starttime) > '20' AND HOUR (starttime) <= '23' THEN '7. 9:00 pm - 11:59 pm: party time'

ELSE '8. 12:00 am - 5:59 am: night time'END AS time_range

FROM bikeshare_final_tbl)AS trips_by_rangeGROUP BY time_rangeORDER BY time_range;

Page 11: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

11

Data Discovery - Query 3

Results: bike rental distribution depending on time of day

Page 12: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

12

Data Discovery - Query 3

What if we add some weather conditions?

SELECT time_range, COUNT(trip_id) AS number_of_trips_in_stormFROM(SELECT trip_id, HOUR (starttime) AS start_hour,

CASEWHEN HOUR (starttime) > '05' AND HOUR (starttime) <= '08' THEN '1. 6:00 am - 8:59 am: morning

commute'WHEN HOUR (starttime) > '08' AND HOUR (starttime) <= '10' THEN '2. 9:00 am - 10:59 am: morning'WHEN HOUR (starttime) > '10' AND HOUR (starttime) <= '13' THEN '3. 11:00 pm - 1:59 pm: lunch

break'WHEN HOUR (starttime) > '13' AND HOUR (starttime) <= '15' THEN '4. 2:00 pm - 3:59 pm: afternoon'WHEN HOUR (starttime) > '15' AND HOUR (starttime) <= '18' THEN '5. 4:00 pm - 6:59 pm: evening

commute'WHEN HOUR (starttime) > '18' AND HOUR (starttime) <= '20' THEN '6. 7:00 pm - 8:59 pm: dinner

time'WHEN HOUR (starttime) > '20' AND HOUR (starttime) <= '23' THEN '7. 9:00 pm - 11:59 pm: party

time'ELSE '8. 12:00 am - 5:59 am: night time'

END AS time_rangeFROM bikeshare_final_tbl WHERE events='tstorms')

AS trips_by_rangeGROUP BY time_rangeORDER BY time_range;

Page 13: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

13

Data Discovery - Query 3

Results: bike rental distribution depending on time of day for some weather conditions

0

20

40

60

80

100

120

140

160

0

500

1000

1500

2000

2500

3000

3500

Thou

sand

s

Thou

sand

s

Number of trips by time range affected by weather conditions

number_of_trips number_of_trips_when_sunny

number_of_trips_in_rain number_of_trips_in_storm

Page 14: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

14

Data Discovery - Query 4

Display 10 longest trips (in hours and minutes) which happened with no visibility in the snow

SELECTfrom_station_name, to_station_name, TO_DATE (starttime) AS startdate, trip_in_minutes,CONCAT (CAST (FLOOR(trip_in_minutes/60) AS STRING),' hrs ',CAST (trip_in_minutes-60*FLOOR(trip_in_minutes/60) AS STRING),' min') AS trip_time,temperature, trip_idFROM bikeshare_final_tbl AS fWHERE LOWER (events)='snow' AND visibility<=0GROUP BY from_station_name, to_station_name, starttime, trip_in_minutes, temperature, trip_idORDER BY trip_in_minutes DESC LIMIT 10;

Write SQL Output – For Query 1

Page 15: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

15

Data Discovery - Query 5

What is the total trip duration for all trips taken on September 11 each year, in hours rounded to one decimal

place and how it depended on weather conditions?

SELECT x.year, FORMAT_NUMBER(SUM(x.tim)/60,1) as total_trips_in_hours, x.conditionsFROM(SELECT YEAR(starttime) AS year, trip_in_minutes AS tim, conditionsFROM bikeshare_final_tblWHERE MONTH(starttime)='09' AND DAY(starttime)='11'GROUP BY starttime, trip_in_minutes, conditions) AS xGROUP BY x.year, x.conditions;

9-11-2013: 1,397.9 hours9-11-2014: 1,466.0 hours9-11-2015: 1,788.4 hours9-11-2016: 5,116.6 hours9-11-2017: 3,835.4 hours

Page 16: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

16

Data Discovery - Query 5

9-11-2013: 1,397.9 hours9-11-2014: 1,466.0 hours9-11-2015: 1,788.4 hours9-11-2016: 5,116.6 hours9-11-2017: 3,835.4 hours

Page 17: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

17

Data Discovery - Query 6

What is the average trip duration in minutes, rounded to one decimal, for the trips starting and ending at the same station, taken by female subscribers when the wind speed

is above 20?

SELECTROUND(AVG(trip_in_minutes),1) ASaverage_trip_length, wind_speedFROM bikeshare_final_tbl AS fWHEREfrom_station_id=to_station_idAND UPPER(usertype)='SUBSCRIBER' AND UPPER(gender)='FEMALE’ AND wind_speed > 20GROUP BY wind_speedORDER BY wind_speed;

Page 18: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

18

Bicycle sharing and weather conditions

Results:1. Less than 0.3% of bicycles are not returned on the same day2. Most bicycles are rented when temperature is between 55F and 76F3. Most popular time is between 4 and 7 pm regardless of weather4. Some harsh weather conditions like snow and no visibility make trips shorter, but not stopping people from using the bicycle sharing system5. Comparing the data of the same calendar date by year, the correlation between nice weather conditions and higher number of trips can be concluded6. There doesn’t seem to be a correlation between the wind speed and the length of the tripsApplications:By finding some weaker spots in the bicycle sharing system, the incentives program could be implemented to increase the use of the system on slow days/times.

Page 19: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

19

Bicycle sharing and weather conditions

https://eportolapol.wordpress.com/

e-Portfolio:

Page 20: DS280 - Spring 2018 - Final Project Presentation - Olga ... · ô ] } À Ç rY µ Ç î t Z ] Z ] l v o ] ] µ ] } v õ ] } À Ç rY µ Ç î Z µ o W ] l v o ] ] µ ] } v

20

Bicycle sharing and weather conditions


*ROGPDQ ,V * : } µ v Ç } ( Z ^ } µ o X...i } ] v ( } Á ] Z ^ v X t Z Ç } , o o } Á o o Z ] o u ] ( } µ v } ( o o µ M v Ç U Á l v } Á Z Z o o Ç v µ o Ç } o } À µ X

BKM INDUSTRIES LIMITED€¦ · 13. dZ }u vÇ] }v v }µ Z vÀ] }vu v v µ o]Ì v µ o }µ ]v µ ]v o ÁÇXt µ Ç}µ } µ Ç}µ u]o Á] ZÇ}µ } ] } ÇW ] v } v o µ } v Ç}µ Z vvµoZ

Z ] < } o U ^ µ P ] } < Ç o : v U µ Ç ^ µ P ] } : µ o Ç î ... · ^ } µ W P íd } o ^ Z À v µ v Æ v ] µ r ' } À v u v Á ]

µ ] v E u } ] v P µ ] v } µ v Ç ] Ç ] } Á u } µ v

] v µ P Z Z ] o v [ , } ] o Z ] Ç d µ [ Z } v } µ v z ... · Z ] o v v Ç } µ v P } o [ o ] À o ] v µ Ç ] o o v X

zE u ^ ] } µ v Ç ^ ] d Ç d v } v } µ v Ç , ^€¦ · o l & } Æ o u v Ç o Ç } µ v Ç rZ ] l ( Z } } o v & ] À ] Á } Ç v ' ] o o µ ^ } } µ v Ç rZ ] l ( Z } } o v

d u À ] o i o Ç Ç v U o ] v v ] l v u ] v i o ] v v À Ç v ... · r À } v v µ t v µ u ] v µ o À µ µ v v v µ o ] v v ] l v u ] v v < Ç W

&DPSXV /RFDWLRQ %XLOGLQJmams.rmit.edu.au/cryxlju486l2.pdf · E } } ( Z ] µ o ] ] } v u Ç } µ ] v v Ç ( } u } Ç v Ç u v U o

Z ^ ( ( v o Ç ] } ( W } v ] o u ] ] } v Z µ ] } v ^ P ] Ç ... · r · } ( s ] ] Ç & µ v µ } Z } rZ } s o ] v î ì í ó A E µ u } ( À ] ] u Ç ( µ v µ } l } r } À o

Brochure€¦ · /(Ç}µ vv} Z µ i Ç}µÁ] Z } µ Ç]v Z] o] U o } }v µ XdZ ] v Æ oo v Zv Z Á Á]oo o } vP }µ (} Ç}µ]v Ç}µ Z} v. o } ( µ ÇX u] W }P uu t Z}À Á v Ç ]+

v µ o ] Ç U ^ } ] o / v o µ ] } v v ] À ] o ^ } ] Ç v P ...kiat.or.id/resx/upload/Update_FINAL_GESI_and_CSE_Strate.pdf · ' v µ o ] Ç U ^ } ] o / v o µ ] } v v ] À ] o

Ç v ^ v µ Z } ] Ç } ( / v ] ^ µ } Ç µ Z } ] Ç µ v D ] v ] Ç } ( , o Z & u ] o ... · 2020. 7. 31. · W P î } ( ï ï } v } ( } P } u v ] } v &^^ / t Z r ì í l î ì î

K À } & o ] Ç Z À ] Á v v µ o Z } µ Ç Z } P } µ v Ç U î ì î ì

u µ o Ì ] v ( } u i ] Ì Á ] v Ç Z Ì Ç µ ] P v ] µ ... · s Ì v ] l v í & } u µ o Ì ] v ( } u i ] Ì Á ] v Ç Z Ì Ç µ ] P v ] µ ] ' } } u } ] v v v ] Ï } u } Á

LVW 2FWREHU ^ ] E u } µ v Ç µ Ç ] v P ' } µ ~ } d h v ... · ^ ] E u } µ v Ç µ Ç ] v P ' } µ ~ } d h v ] } v D o & u o d } o r î ó r ' ] }/ o Ç s ] ] v U ð ð U ' ]

BTSG Green Bond at Towards the Sustaibable …...^ µ ] v ] o ] Ç d^ ' } µ K µ Z } Á µ ] v ] o ] Ç d^ ' } µ ' v } v W v ] } v ñ ^ } } ^ µ ] v ] o ] Ç W } o ] Ç _ ] v o (

, À Ç } µ ~ } Z Ç ] Ç } µ ~ } Æ ] v , Z Z v P } Z , Z v ......, o Z } Ç } µ ( ] o o ] v P Z ] µ À Ç ] v } µ , o Z } Ç } µ ( ] o o ] v P Z ] µ À Ç ] v } µ

v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > …...v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > rD^ lD^ µ ] v P < ] v Æ } r^

h^W µ u v Ç v K P v ] / u µ ] ] Ç > rhs µ ] v P Z < ] v Æ

µ µ u v î ì î ì K ] v P ^ Ç u } u } v v v ^ µ µ...W } P u l } } l } E } Z } P u ] } o o Ç ] À i µ Ç } v ] l Z v } ] v µ ] } v } µ v } ] v ] v v } ( } P u ] v P Æ µ

DQG IULHQGV Ç } ( < } l µ } } µ } u u µ v ] Ç ] v v€¦ · 6wv 3hwhu dqg 3dxo ,qylwhv

Green Building Certification and the Commercial Buildings Energy Consumption … · 2019-07-01 · v P Ç } µ v v µ U v Ç } ( v P Ç r o µ ] u v ( } µ v ] v Z µ ] o ] v P X

u o , ] P Z À ] o ] o ] Ç ( } v Ç o ] ] } v ] v o } µ } K ... · w z Ç ] o u s ] µ o u o } µ u } v Ç } u ] v ] } v o ] ] } v z } À Ç < ] ~ z< z } À Ç k z ] } v z µ v

µ Z d µ v v µ Ç - WHO...WKd^ A } µ o } Z } ] Z Ç ] Ç v } u V ZW^ A } u o Æ P ] } v o ] v Ç v } u ' s^ Æ } µ K µ } u > ] v l

SETAC Technical Workshop SOP^ d d Z v ] o t } l Z } Æ v ( } u í r } ñ r Ç v ] v o µ µ µ o o Ç ï ì r ñ ì ] ] v X } À Ç ( } u ( µ o o Ç } ] o o Ç ( µ v v } µ µ o

µ Ç } v Ç v u ] } v } ( u ] r µ u ] o ( o } ] v P Á ] v µ ...€¦ · µ Ç } v Ç v u ] } v } ( u ] r µ u ] o ( o } ] v P Á ] v µ ] v } v ] ] v P } u ] v Á À v µ v o

^ Z } ( · 2019-07-05 · Z v µ u } ( v ] µ ] v î ì í ð X K µ o u ] o o ] À Ç } } } µ v Z µ v l v } Á v ] ] Ì v ] v À Ç v } } l v } v } ( } µ } µ v Ç ( o Z

^ Ç v Z ] v ] } Ç v Z ] ^ µ ] } v Ç } u ] ] v - Thieme^ µ } Ç ^ Ç v Z ] v ] } Ç v Z ] ^ µ ] } v Ç } u ] ] v ^ µ ] ] Á X D Z U E ] Z } o Z X > U D Z Á : X Ç v U > ] r

µ ] } ' } } / v µ Ç W ( } >W ' Ç o ] v D v P u v

Ç v ^ v µ Z } ] Ç } ( / v ] ^ µ } Ç µ Z } ] Ç µ v D ] v ] Ç } ( , …...Microsoft Word - FSSAI Advt DR-012020 dt 31.07.2020 Author VinayKumarTarun Created Date 7/31/2020 6:30:43

P µ ] v P Z ] o v µ ] v P > } l } Á v€¦ · ^ ( P µ ] v P Z ] o v µ ] v P > } l } Á v d Z o } } ( ] + v Á Ç Ç } µ v P Z o ] ( Ç } µ Á } ] Title: C:UserscscfsgsmAppDataLocalTempmso9EAB.tmp

Whats New in 2019 Env Excellence Awards…9 µ v Ç Á ~ } o ] U Ç o ] v P U ZDt U Z Ì } µ 9 µ v Ç v P Ç ~ o ] ] Ç 9 µ v Ç Á µ

E µ v P D ] o ] Ç } u u µ v ] Ç ] } Ç o v P v

^ u ï r í ñ Z - YMCA · ^ u ï r í ñ Z D } v Ç d µ Ç t v Ç d Z µ Ç & ] Ç ^ µ Ç ^ µ v Ç

2008 Annual Report - s2.q4cdn.com · July 2008 } v À u ] v ] v P Ç } µ } o v U v Á o v P Ç Minimize environmental ] u Ç } µ } o o } v v recycling Reduce u ] ] } v Ç µ µ