kristoffer dyrkorn – beating the traffic jam using nosql - nosql matters barcelona 2014
TRANSCRIPT
Norwegian consulting firm
Private and public sector enterprises
Strategy, technology, digital services
370 employees
Responsible for state and county roads
Planning, construction, operation
7500 employees
Spending: € 7 Billion (2013)
BEKK Public Roads Administration
BACKGROUND
Population (mill)
Area (km2)
Roads (1000 km)
Germany 81 357 644 Spain 46 500 683 Norway 5 385 95
CONTEXT
BUILDING AND MAINTAINING ROADS IS EXPENSIVE
PROPER PLANNING DEPENDS ON TRAFFIC ANALYSIS
VEHICLE COUNTS & WEIGHTS DECIDE
PRECISE REPORTS ARE NEEDED
ROADS ARE INFRASTRUCTURE
TIME IS ESSENTIAL
ROAD TRAFFIC IS DYNAMIC
UPDATED TRAFFIC INFORMATION HELPS:
• SAFETY
• FLOW CONTROL
• ROUTE PLANNING
REAL TIME DATA IS NEEDED
SYSTEM GOALS & REQUIREMENTS
EASE OF INSTALLATION AND VERIFICATION OF ROADSIDE EQUIPMENT
INCREASED DATA QUALITY
INCREASED DATA AVAILABILITY
ALL EVENTS MUST BE KEPT (NO PRE-AGGREGATION)
MINIMAL LATENCY
AD-HOC REPORTING
SCALABILITY
ROBUSTNESS
Java VM
OS
Application logic
Support libraries
Traffic events
Reports (CSV, SOAP)
Java VM
OS
Elasticsearch
Request/response
Data logger
N data loggers M application servers K storage servers
HTML5 GUI (HTTP, JSON)
SOLUTION ARCHITECTURE
"measure_point_number": 1601436, "county_id": 16, "region_id": 2, "server_local_timestamp": "2014-‐10-‐01T01:58:44.330+02:00", "server_utc_timestamp": "2014-‐09-‐30T23:58:44.330Z", "client_utc_timestamp": "2014-‐09-‐30T23:58:45.229Z", "event_number": 2319762, "vehicle_type": 3, "vehicle_type_raw": "9", "vehicle_type_quality": 22228, "vehicle_number": 2319762, "speed": 80.9, "length": 16.46, "lane": 1, "gap": 10
HOW WE USE ELASTICSEARCH
BULK INDEXING, JAVA API
DATA IS INDEXED, STORED, NOT ANALYZED
TEMPORAL SHARDING
SPATIAL SHARDING
DATA CENTER-AWARE REPLICATION
NO SPECIAL OPTIMIZATIONS!
RAM, CPU, DISK
REPORTING
FOR A GIVEN TIME INTERVAL, PROVIDE:
• TOTAL VEHICLE COUNT AND AVERAGE SPEED,
• THE 85 AND 95 PERCENTILE SPEEDS,
• IN EACH OF 5 LENGTH CATEGORIES: THE VEHICLE COUNT AND AVERAGE SPEED,
• IN EACH OF 12 SPEED CATEGORIES: THE VEHICLE COUNT,
...AND ALL OF THIS FOR
• EACH TRAFFIC LANE AT A MEASURE POINT,
• EACH MEASURE POINT IN A REGION
SYSTEM VALUE
REPORTING:
• MORE COST-EFFICIENT ROAD MAINTENANCE
REAL TIME:
• ROUTING OF EMERGENCY VEHICLES
• GENERAL TRAFFIC INFORMATION TO THE PUBLIC
• ROUTE PLANNING ON HOLIDAYS
• ROUTE PLANNING FOR PARCEL SERVICES
EXPERIENCES USING ELASTICSEARCH
ENSURE FITNESS-TO-PURPOSE
UPGRADE CONTINUOUSLY
REVISE THE RUN-TIME ENVIRONMENT CONTINUOUSLY
THE AGGREGATIONS MODULE IS FANTASTIC
USE TOOLING (WE LIKE KOPF)
MONITOR THE RESOURCE UTILIZATION
THE JAVA API IS SOMETIMES COMPLEX
WE ARE HAPPY!