Vic LevyPrinceton Consultants, Inc.590 5th Ave, Floor 16New York, NYvlevy@princeton.com

Python on RailsAdvancing Safety with Locomotive-Hosted “Mobile” App

NYC Python Meetup, April 17, 2014Two Sigma Investments, LLC, New York, NY

1

2Outline

Positive Train Control 101

COSMA: Agent on the locomotive

Programming COSMA

Testing

3

Positive Train Control 101

4

PTC is a federal law applying to Amtrak and Class-I railroads.

CSX is a Class-I railroad.

5

• Chatsworth, CA, Sept 2008. Commuter and freight trains collide, many casualties

• Federal law passed same month• Class-1 railroads due Dec 2015

6Warns crew when any of several classes of danger detected

7If crew does not react to warning in time, PTC applies brakes

8Impact of PTC on Class-I Railroads, Including CSX

• New back-office servers• New wayside devices• New communications systems

9Impact of PTC on Class-I Railroads, Including CSX, Cont’d• New equipment on board locomotives

10

New PTC Equipment Aboard CSX Locomotives: PTC Computer

11

New PTC Equipment Aboard CSX Locomotives: Computer Displays

12New PTC Equipment Aboard CSX Locomotives: 220-MHz Radio

13

New PTC Equipment Aboard CSX Locomotives: Onboard Routing ServerCOSMA host system

14New PTC Equipment Aboard CSX Locomotives

NEED: Checkout, monitor, upgrade these devices and more

COSMA: Agent on the Locomotive

15

16COSMA: CSX Onboard Systems Management Agent

Python

17COSMA Primary Functions

1. Check out PTC equipment before deployment in the field

18

COSMA Primary Functions, Cont’d2. Report on conditions, problems,

assets in the onboard PTC environment

3. Correct problems, either autonomously or on command

19

COSMA Primary Functions, Cont’d4. Report on cell, radio and Wifi

signal strength

20COSMA Primary Functions, Cont’d

5. Perform software updates to onboard systems, either in the shop or on the rail

21

Programming COSMA

22Realities of Programming in the Locomotive Environment

1. ?

23Realities of Programming in the Locomotive Environment

1. No interfering with proper functioning of safety-critical systems

2. In and out of connectivity3. No controlled shutdown (circuit breaker)4. Cell bandwidth is expensive5. Limited writing to persistent storage

(SSD)

24Realities of Programming in the Locomotive Environment, Cont’d

6. Stay aware of changes in the onboard environment

• Did we wake up on another locomotive?

• Are we in home territory?• What is the PTC state?

25Realities of Programming in the Locomotive Environment, Cont’d

7. Python 2.6.68. Red Hat Enterprise Linux Server 6.2

26COSMA Is Multi-Threaded

• Most thread asleep most of the time• Wake up, check something, go back to sleep

• Signal-mapping watches GPS for distance

27Two Configuration Files:

Default (installed) Override (runtime)

28

Two Configuration Files (ConfigParser Library)

29

Pattern: JIT Configuration QueriesApplication is remotely configurable, so no caching configs in __init__

30

Pattern: Subsystem QueryDon’t overtax onboard systems if recently sampled data is good enough

31

Pattern: Logical Device

• Adapter pattern presents a common interface to business code querying devices with differing access methods

32Pattern: No Self-Deprecating or Humorous Comments

33Hardware Access Method: UDP

Nonintrusive read, no impact on monitored device

Monitored device must be broadcasting

Sample rate determined by device

34

Hardware Access Method: UDP• How to create a UDP socket depends on whether sending or receiving, or receiving a broadcast

35Hardware Access Method: SNMP• Available library: netsnmp

• Faster• Malformed OID can crash Python

• Available library: pysnmp• Pure-Python implementation• Slower• Unresponsive OID returns “NO DATA”

36

Hardware Access Method: SNMPExample pt 1: Create netsnmp session

37

Hardware Access Method: SNMPExample pt 2: Create a list of bindings

38

Hardware Access Method: SNMPExample pt 3: Query and recombine

39Hardware Access Method: SmartCtl

[vlevy@CSX-PTC3EAP-111100016 ~]$ sudo smartctl -a /dev/sda[sudo] password for vlevy:smartctl 5.39.1 2010-01-28 r3054 [i386-redhat-linux-gnu] (local build)Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===Device Model: FiD 2.5 SATA10000Serial Number: 20111005AAB000000030Firmware Version: 090928User Capacity: 7,998,455,808 bytesDevice is: Not in smartctl database [for details use: -P showall]ATA Version is: 7ATA Standard is: Exact ATA specification draft version not indicatedLocal Time is: Mon Apr 14 15:11:12 2014 EDTSMART support is: Available - device has SMART capability.SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSED

General SMART Values:Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled.Total time to complete Offlinedata collection: ( 0) seconds.Offline data collectioncapabilities: (0x00) Offline data collection not supported.SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer.Error logging capability: (0x00) Error logging NOT supported. No General Purpose Logging support.

SMART Attributes Data Structure revision number: 1280Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 331 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 0194 Temperature_Celsius 0x0007 032 100 000 Pre-fail Always - 0229 Unknown_Attribute 0x0002 100 000 000 Old_age Always - 259865206445036232 Available_Reservd_Space 0x0002 100 048 000 Old_age Always - 9037453799481233 Media_Wearout_Indicator 0x0002 100 000 000 Old_age Always - 0234 Unknown_Attribute 0x0002 100 000 000 Old_age Always - 678655194114235 Unknown_Attribute 0x0002 100 000 000 Old_age Always - 4261534239

Warning: device does not support Error LoggingSMART Error Log Version: 16No Errors Logged

Warning: device does not support Self Test LoggingSMART Self-test log structure revision number 16Warning: ATA Specification requires self-test log structure revision number = 1No self-tests have been logged. [To run self-tests, use: smartctl -t]

Device does not support Selective Self Tests/Logging

SSD Statistics:• Power cycle

count• Erase cycle

percent• Average & max

erase count

Technique:• Parse to int• struct pack &

unpack

40Hardware Access Method: Drive Vendor and Model

41More Hardware Access Methods

• SSH• Telnet

42

Development & Testing Environments

43

Development: Eclipse & Pydev

44

Unit Testing: unittest LibraryEclipse & Pydev rock. Also nosetests.

45Testing Platforms: Lab

46Testing Platforms: High-Rail

47Testing Platforms: The Real Thing

Vic Levyvlevy@princeton.com

Princeton Consultants, Inc.590 5th Ave, Floor 16New York, NY

Happy Springtime! 1

Princeton Consultants, Inc.

• Blend of information-technology and management consulting

• Service offerings are business strategy, software development, process engineering and project management

• Clients include many of the best-known and most innovative companies

• 85 full-time, on-shore professionals work out of our offices in Princeton, N.J. and Midtown Manhattan

• Over 1,500 projects successfully delivered since our founding in 1980

• Corporate website: www.princeton.com

• Careers website: www.princeton.com/join

2 Research WayPrinceton, NJ 08540

590 Fifth AvenueNew York, NY 10036