portal: applications of new technology to transportation data...
TRANSCRIPT
![Page 1: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/1.jpg)
+
Portal: Applications of New Technology to Transportation Data Archiving
Kristin Tufte & the Portal Team NATMEC, July 1, 2014, Chicago, IL
![Page 2: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/2.jpg)
+Who is Kristin? ! 20 years Data Management System Design and Implementation
! Paradise (1990-1996) - sold to NCR/Teradata
! NiagaraST Data Stream System (1996-2014)
! S-Store – Streams + OLTP (2013-…)
! 10 years Transportation Data Management ! Portal Transportation Data Archive (2004-…)
! Portland Observatory (2013-…)
![Page 3: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/3.jpg)
+New Technologies Big Data, Cloud, Data Streams…
![Page 4: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/4.jpg)
+Big Data…Why all the interest?
! An increased number and variety of data sources that generate large quantities of data
! Sensors (e.g. location, acoustical, …); Web 2.0
! Realization that data was “too valuable” to delete ! ADUS - ITS Program Plan addendum 1998
! Dramatic decline in the cost of hardware, especially storage ! If storage was still $100/GB there would be no big data
revolution underway
Slide credit: David DeWitt, Microsoft/UW-Madison, SQL Server PASS Talk 2011
0 1 1 0 0
0 0 1 0 1 1
1 0 1 0 0 1
0 0 1 0 1 1
1 1 0 0 1 1
0 0 1 0 1 1
1 1 1 1 1 1
0 0 1 0
0 0 1 0
![Page 5: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/5.jpg)
+Is Transportation Data Big Data?
0
100
200
300
400
500
600
700
800
900
1000 966
848
715
619
434 364
269 227
Amount of Stored Data By Sector (in Petabytes, 2009)
1 zettabyte? = 1 million petabytes = 1 billion terabytes = 1 trillion gigabytes
Sources: "Big Data: The Next Frontier for Innovation, Competition and Productivity." US Bureau of Labor Statistics | McKinsley Global Institute Analysis
Peta
byte
s
Figures credit: David DeWitt, Microsoft/UW-Madison, SQL Server PASS Talk 2011
![Page 6: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/6.jpg)
+Volume, Velocity and Variety ! Variety is the most difficult and potentially most critical
axis of Big Data ! Transportation data has Variety in spades
! Big Data’s Implications for Transportation Operations: An Exploration ! White Paper – March 2014; Prepared by Volpe Center;
Prepared for US DOT ITS JPO
Ace Sensor – Radar – Bluetooth – Signal loops
Bus AVL, boardings SCATS AQ Sensor
Other: • Incident
reports • Crossing
demands • Bus priority
requests
Slide credit: Adam Moore, Miguel Figliozzi, PSU
![Page 7: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/7.jpg)
+The Cloud: NoSQL vs. NewSQL vs. RDBMS
! Cloud Computing: Many definitions, many promises…
! Reduced time to insight…
! Saving resources…
![Page 8: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/8.jpg)
SQL:
8
NoSQL:
" No cleansing! " No ETL! " No load! " Analyze data where it lands!
RDBMS
Data Arrives
Derive a schema
Cleanse the data
Transform the data
Load the data
SQL Queries
1
2
3 4 5
6 Sometimes termed
“Schema First”
Sometimes termed
“Schema Later”
Data Arrives
Application Program
1 2
NOSQL System
Slide credit: David DeWitt, Microsoft/UW-Madison, SQL Server PASS Talk 2011
![Page 9: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/9.jpg)
+
Unused resources
Save Resources…
Data center in the cloud
Demand
Capacity
Time
Res
ourc
es
Demand
Capacity
Time R
esou
rces
Slide Credit: RAD Lab, UC Berkeley
9
![Page 10: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/10.jpg)
+But, Don’t be Fooled… ! NoSQL systems make lots of promises…
! But they don’t work for everything…
10
![Page 11: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/11.jpg)
+Portal An Update…
![Page 12: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/12.jpg)
+Portal – Where is it at today?
! Happy 10th Birthday Portal! (April 1994 - …)
! Publically funded (Thanks to NSF, FHWA, OTREC, Metro & Transport, RTC)
! Focus on open-source software (PostgreSQL, PostGIS, OpenLayers, HighCharts)
! Focus on open data (Thanks to all our wonderful collaborators)
! New Transit Load and Performance Map (GTFS + PAX data)
! Lots of new (local) data feeds
![Page 13: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/13.jpg)
+Freeway Data Flow
Portal OR-WA Archive
ODOT - Loop Detectors - Wavetronix - Bluetooth Travel Time
Lane County - Wavetronix
ODOT DAQ
# XML Feed # 20 second granularity # automated station inventory file
WSDOT - Loop Detectors - Wavetronix
# XML Feed # 20 second granularity
# XML Feed # 20 second granularity # being phased out
![Page 14: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/14.jpg)
+Transit Data Flow
Portal OR-WA Archive
TriMet - AVL/APC (Init) - GTFS Data
# GTFS data published publically
TriMet Enterprise Database
# PAX data inserted in Enterprise Database # Data is cleaned and aggregated
# Quarterly PAX data exported
C-Tran - AVL/APC (Init) - GTFS
# No enterprise database (yet) # Process to be determined
# Portal Archive import processing combines PAX and GTFS data
![Page 15: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/15.jpg)
+Arterial Data Flow - Current
Portal OR-WA Archive
City of Portland - Bluetooth
# Bluetooth data gathered from devices by scripts on CoP servers # Data uploaded to Portal weekly # Processing scripts calculate travel times
Washington & Clackamas County - Signal System (TransSuite)
# Central Signal Server is Shared
City of Portland - Signal System, including MOE Logs (TransSuite) - Bike/Ped Counts
# Hourly data feed created by TransSuite # Data uploaded to PSU hourly (sftp)
Clark County - Wavetronix
# Data generated using Wavetronix report-generation system # Data uploaded to PSU nightly
Near Future: City of Vancouver - Bluetooth Clark County - Bluetooth City of Vancouver - Wavetronix - Signal System (ATMS.Now)
![Page 16: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/16.jpg)
+Transit Map!
! Combines GTFS (General Transit Feed Specification) with AVL/APC (Automatic Vehicle Location/Automatic Passenger Counter) data ! Lots of GIS processing involved
! GIS process is complex – freeway data too!!
! Metrics include: ! Stop activity
! Ons/offf; On-time performance
! Segment activity
! Load and Utilized capacity
Segment Load
![Page 17: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/17.jpg)
+Stop-Level performance
![Page 18: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/18.jpg)
+S-Store Big Data Bikeshare Demo
! Demo applying Data Stream technology to a Bikeshare scenario
! Collaboration between PSU, Intel, MIT, Brown Univ.
! To be shown at VLDB 2014 (Hangzhou, China)
![Page 19: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/19.jpg)
+Portland Observatory: Urban Informatics Variety Testbed
! Challenges the third “V” of Big Data: variety
! Observations and context
! For researchers, planners, managers, public
! Building on experience from Portal transportation archive
! Increasing data touches
![Page 20: Portal: Applications of New Technology to Transportation Data …web.cecs.pdx.edu/~kat/presentations/Tufte-NATMEC-2014.pdf · 2014. 7. 9. · Figures credit: David DeWitt, Microsoft/UW-Madison,](https://reader034.vdocument.in/reader034/viewer/2022052017/60304873732a7574162975e8/html5/thumbnails/20.jpg)
+THANK YOU!! http://portal.its.pdx.edu [email protected]