android sensor progamming-compressed -...
TRANSCRIPT
AndroidSensorProgramming- ArindamGhosh
SensingandSensors
• acapabilitythatcancapturemeasurementsaboutthedeviceanditsexternalenvironment
• Candetectsandrespondstosometypeofinputfromthephysicalenvironment.
• Thespecificinputcouldbelight,heat,motion,moisture,pressure,etc.
• Convertthemeasurementintoasignalthatcanberead.
ExampleSensorData
1|MPU6500AccelerationSensor|[-1.6741456,9.370906,2.6886885]|1441670212915
1|MPU6500GyroscopeSensor|[-0.02263687,-0.016777916,-0.008788432]|1441670213508
1|AK09911CMagneticfieldSensor|[16.86,-64.26,-62.7]|1441670213400
1|GPS|{"mProvider":"fused","mResults":[0.0,0.0],"mAccuracy":29.0,"mAltitude":83.0,” ,"mLatitude":40.748431”,mLongitude":73.985741 …}|1441573552851
1|WiFi|{"BSSID":"00:21:6c:87:02:d1","SSID":"eduroam","capabilities":"[WPA2-EAP-CCMP]","frequency": 2462,"level":-82}|1392465248466
AndroidOSFragmentation
http://opensignal.com/reports/2015/08/android-fragmentation/
http://opensignal.com/reports/2015/08/android-fragmentation/
http://opensignal.com/reports/2015/08/android-fragmentation/
SensorAvailability
• Variesfromdevicetodevice
• MayvarybetweenAndroidversions
NewerSensorsOverTime
Sensors
• PositionsensorsGPS,orientationsensorsandmagnetometers.
• Motionsensorsaccelerometers,gravitysensors,gyroscopes,etc.
• Environmentalsensorsbarometers,photometers,andthermometers.
SensorFramework
Accesssensorsandandacquirerawsensordata.
• Determinewhichsensorsareavailableonadevice.• Determineanindividualsensor'scapabilities,suchasitsmaximumrange,manufacturer,powerrequirements,andresolution.• Acquirerawsensordataanddefinetheminimumrateatwhichyouacquiresensordata.• Registerandunregistersensoreventlistenersthatmonitorsensorchanges.
Sensors•Hardware-based orSoftware-based•Hardware-basedsensors- physicalcomponentsbuiltintoahandsetortabletdevice• directlymeasuringspecificenvironmentalproperties.• acceleration,geomagneticfieldstrength,orangularchange.
• Software-basedsensors- mimichardware-basedsensors• derivetheirdatafromoneormoreofthehardware-basedsensorsandaresometimescalledvirtualsensorsorsyntheticsensors.• Thelinearaccelerationsensorandthegravitysensor.
SensorFramework
Accesssensorsandandacquirerawsensordata.AndroidSensorFrameworkincludesthreeclassesandoneinterface.
• SensorManager• Sensor• SensorEvent• SensorEventListener
http://developer.android.com/guide/topics/sensors/sensors_overview.html
• Identifyingsensorsandsensorcapabilities• Monitorsensorevents
1
2
3
4
5
SensorManager
• SystemServicethatmanagessensors
• FirsttheapplicationneedstogetareferencetotheSensorManager• getSystemService(SENSOR_SERVICE);
• Accessaspecificsensorwith• SensorManager.getDefaultSensor(int type)
Sensor
• Accelerometer• Sensor.TYPE_ACCELEROMETER
• MagneticField• Sensor.TYPE_MAGNETIC_FIELD
• Pressure• Sensor.TYPE_PRESSURE
GetaListofAllSensors
SensorEventListener
• ForanapplicationtoreceiveinformationfromaSensor• ItneedstoimplementaSensorEventListener• BeforestartingtoreceivesensorEvents
• TypeofSensor• Delay
SensorEventListener
SensorDelay
SensorEventListener
SensorEventListener
SensorEventListener
• ForanapplicationtoreceiveinformationfromaSensor• ItneedstoimplementaSensorEventListener
• Invokedwhenaccuracyofasensorchanges• Whenthesensoracquiresanewreading
onAccuracyChanged
• Accuracyisrepresentedbyoneoffourstatusconstants:• SENSOR_STATUS_UNRELIABLE• ConstantValue:0(0x00000000)
• SENSOR_STATUS_ACCURACY_LOW,• ConstantValue:1(0x00000001)
• SENSOR_STATUS_ACCURACY_MEDIUM,• ConstantValue:2(0x00000002)
• SENSOR_STATUS_ACCURACY_HIGH,• ConstantValue:3(0x00000003)
https://developer.android.com/reference/android/hardware/SensorManager.html
onAccuracyChanged
onSensorChanged
onSensorChanged
SensorEventListener
• OnceyouaredoneUsingtheSensor
1
2
3
4
5
SensorCoordinateSystem
• Thesensorframeworkusesastandard3-axiscoordinatesystemtoexpressdatavalues.• Xaxisishorizontalandpointstotheright• Yaxisisverticalandpointsup• Zaxispointstowardtheoutsideofthescreenface• coordinatesbehindthescreenhavenegativeZvalues
SensorCoordinateSystem
Suchacoordinatesystemisusedby:
• Accelerationsensor
• Gravitysensor
• Gyroscope
• Linearaccelerationsensor
• Geomagneticfieldsensor
PointstoRemember
• Yourapplicationmustnotassumethatadevice'snatural(default)orientationisportrait.• Thesensorcoordinatesystemisalwaysbasedonthenaturalorientationofadevice.• Thenaturalorientationformanytabletdevicesislandscape.
• Verifysensorsbeforeyouusethem• Verifythatasensorexistsonadevicebeforeyouattempttoacquiredatafromit
Thesensor'scoordinatesystemneverchangesasthedevicemoves
• Youmusttestyoursensorcodeonaphysicaldevice.• Youcurrentlycan'ttestsensorcodeontheemulatorbecausetheemulatorcannotemulatesensors.
• Thereare,however,sensorsimulatorsthatyoucanusetosimulatesensoroutput.
Verifysensorsbeforeyouusethem
IfyouarepublishingyourapplicationonGooglePlayyoucanusethe <uses-feature> elementinyourmanifestfiletofilteryourapplicationfromdevicesthatdonothavetheappropriatesensorconfigurationforyourapplication.
PointstoRemember• Unregistersensorlisteners
• whenyouaredoneusingthesensororwhenthesensoractivitypauses.• Ifasensorlistenerisregisteredanditsactivityispaused,thesensorwillcontinuetoacquiredataandusebatteryresourcesunlessyouunregisterthesensor.
• Don'tblocktheonSensorChanged()method• Sensordatacanchangeatahighrate- systemmaycalltheonSensorChanged(SensorEvent)methodquiteoften
• DoaslittleaspossiblewithintheonSensorChanged(SensorEvent)methodsoyoudon'tblockit
• Choosesensordelayswisely• Sensorscanprovidedataatveryhighrates.• Sendingextradatathatyoudon'tneedwastessystemresourcesandusesbatterypower.
OrientationSensors
• TYPE_ACCELEROMETER usestheaccelerometerandonlytheaccelerometer.Itreturnsrawaccelerometerevents,withminimalornoprocessingatall.
• TYPE_LINEAR_ACCELERATION (ifpresent)usesthegyroscopeandonlythegyroscope.Likeabove,itreturnsrawevents(angularspeedunrad/s)withnoprocessingatall(nooffset/scalecompensation).
• TYPE_ORIENTATION isdeprecated.Itreturnstheorientationasyaw/pitch/rollindegrees.• Thissensorusesacombinationoftheaccelerometerandthemagnetometer.• MarginallybetterresultscanbeobtainedusingSensorManager's helpers.• Thissensorisheavily"processed".
Orientations
Accelerometer Gyroscope
OrientationSensors
OrientationSensors.• TYPE_LINEAR_ACCELERATION,TYPE_GRAVITY,TYPE_ROTATION_VECTOR are"fused"sensorswhichreturnrespectively• thelinearacceleration,• gravityand• rotationvector(aquaternion).• Onsomedevicestheyareimplementedinh/w,• Onsomedevicestheyusetheaccelerometer+themagnetometer• Onsomeotherdevicestheyusethegyro.
TYPE_ACCELEROMETER
• MEMSaccelerometersaretinymassesontinysprings.• Theycansense• Speedinguporslowingdowninastraightline• Shakingthedevice• Earth’sgravity,whichis1gdownward
TYPE_ACCELEROMETER
TYPE_LINEAR_ACCELERATION
TYPE_ACCELEROMETER
TYPE_GYROSCOPE
Measuresrateofrotation.Youcannotdirectlymeasureangleusingagyroscope.Youcanintegratetherateofrotationovertimetogetangle
ActivityRecognition
Walking
45
Bus
46
Driving
47
Train
48
TYPE_MAGNETIC_FIELD
• HardwareSensor• MostlyHalleffectSensors• Androidreportsmagneticfieldsinmicrotesla.• Earth’smagneticfield canvaryfrom30microtesla to60microtesla• Uses• Compass• Themagneticfieldsensorcanbeinfluencedbynearbymetal,somepeoplehaveusedthesensortomakeanAndroiddeviceintoacrudemetaldetector• Duetoaneffectcalledhysteresis
TYPE_PROXIMITY•HardwareSensor•Letsyoudeterminehowfarawayanobjectisfromadevice• Someproximitysensorsprovideaboolean value(near/far).• Typically,thefarvalueisavalue>5cm,butthiscanvaryfromsensortosensor.
•Usuallyusedtodeterminehowfarawayaperson'sheadisfromthefaceofahandsetdevice• Tolockthescreenwhenauserisonacall
TYPE_PROXIMITY
TYPE_PROXIMITY
TYPE_STEP_COUNTER
• Returnsthenumberofstepstakenbytheusersincethelastrebootwhileactivated.• Resettozeroonlyonasystemreboot.• Thetimestampoftheeventissettothetimewhenthelaststepforthateventwastaken.• Thissensorisimplementedinhardwareandisexpectedtobelowpower.• Applicationneedstostayregisteredforthissensorbecausestepcounterdoesnotcountstepsifitisnotactivated.
TYPE_STEP_DETECTOR
• Triggersaneventeachtimeastepistakenbytheuser.• Theonlyallowedvaluetoreturnis1.0andaneventisgeneratedforeachstep.• Thetimestampindicateswhentheevent(herethestep)occurred•Whenthefoothitstheground- generatingahighvariationinacceleration.
ENVIRONMENTSENSORS
• TheAndroidplatformprovidesfoursensorsthatletyoumonitorvariousenvironmentalproperties.• AmbientPressure
• MeasurestheambientairpressureinhPa ormbar.• AmbientHumidity
• Ambienthumiditynearthephone(expressedas%atmospherichumidity)• Illuminance
• Usedtocontrolscreenbrightness(measuredinlux)• Ambienttemperature
• Ambienthumiditynearthephone(measuredindegreecentigrade)
• Theyareallhardwaresensors.
TYPE_HEART_RATE
• FoundinAndroidWearables.• Thereportedvalueistheheartrateinbeatsperminute.• Thissensorrequirespermissionandroid.permission.BODY_SENSORS• Itwillnotbereturned by SensorManager.getSensorsList orSensorManager.getDefaultSensor ifthepermissionismissing.
HEARTRATESENSOR
• MeasuredusingaPhotoplethysmography (PPGsensor).• Measuresthedifferentialreflectionoflightby oxygenatedanddeoxygenatedblood
HEARTRATESENSOR• Similartotheprincipleofandroidappstomeasureheartrateusingthecamera.
• SensorMovementArtifact• NervousFidgetingArtifact
RemovalofLocalFidgeting
ChangesinWristbandAcc +Physiologicalsignal=YESChangesinSmartphoneAcc/Gyro =NO
Removenext30seconds61
ARTIFACTSINHEARTRATE
ArtifactRemoval– ActivityRecognitionUseCase.
RemovalArtifactsusingFiltering
• Twocommonfilteringtechniques• Low-passfilters• Passfrequencieslowerthancutofffrequency• Deemphasizetransientforcechange(vibrations)• Emphasizeconstantforcecomponents• e.g.,forabubblelevel
• High-passfilters• Passfrequencieshigherthancutofffrequency• Emphasizetransientforcechanges• Deemphasizeconstantforcecomponents(gravity)• e.g.,foragamecontroller
SignalPreprocessing
𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)
𝐴"#$% = 𝐴() +𝐴+) +𝐴,)
𝐴.
𝐴+
𝐴,
𝐺(
𝐺+
𝐺,
Accelerometer
Gyroscope
Audio
InputStreams Preprocessing64
SignalPreprocessing
𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)
𝐴"#$% = 𝐴() +𝐴+) +𝐴,)
𝐴.
𝐴+
𝐴,
𝐺(
𝐺+
𝐺,
Accelerometer
Gyroscope
Audio
InputStreams Preprocessing
Tseconds
Nsec Nsec
N=T/10ifN<300
N=30secsOtherwise
BoundaryRemoval
65
SignalPreprocessing
𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)
𝐴"#$% = 𝐴() +𝐴+) +𝐴,)
𝐴.
𝐴+
𝐴,
𝐺(
𝐺+
𝐺,
Accelerometer
Gyroscope
Audio
InputStreams Preprocessing
Screen-oncheckingandremoval
Forcontrolledscenario: Ifscreen-unlocked>10seconds
40Hz
40Hz
8KHz
66
SignalPreprocessing
𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)
𝐴"#$% = 𝐴() +𝐴+) +𝐴,)
𝐴.
𝐴+
𝐴,
𝐺(
𝐺+
𝐺,
Accelerometer
Gyroscope
Audio
Segm
entinto3se
condwindow
InputStreams Preprocessing Segmentation
BoundaryRemoval
AndFiltering
HumanAverageStride rateisbetween80-120stepsaminute
67
𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)
𝐴"#$% = 𝐴() +𝐴+) +𝐴,)
𝐴.
𝐴+
𝐴,
𝐺(
𝐺+
𝐺,
Accelerometer
Gyroscope
Audio
Segm
entinto3se
condwindow
InputStreams Preprocessing Segmentation FeatureExtraction
ZeroCrossing rate
RMSEnergy
MFCCs
Mean
Standarddeviation
Numberofpeaks
Interpeakdistances
minimum
maximum
BoundaryRemoval
AndFiltering
SignalPreprocessing
68
SignalPreprocessing
𝐺"#$% = 𝐺() + 𝐺+) + 𝐺,)
𝐴"#$% = 𝐴() +𝐴+) +𝐴,)
𝐴.
𝐴+
𝐴,
𝐺(
𝐺+
𝐺,
Accelerometer
Gyroscope
Audio
Segm
entinto3se
condwindow
InputStreams Preprocessing Segmentation FeatureExtraction
ZeroCrossing rate
RMSEnergy
MFCCs
Mean
Standarddeviation
Numberofpeaks
Interpeakdistances
minimum
maximum
Classification
RandomForestLOSO
ACCGYRO
ACC+GYROACC+GYRO+AUDIOBoundary
RemovalAndFiltering
69
Challenge
Highsamplingratedrainsbattery
70
Effectsofloweringsamplingrates
71
PublicationRelatedtothissection
RecognizingHumanActivitiesfromSmartphoneSensorSignals.ACMMultimedia2014,Ghosh,Arindam ,andRiccardi,Giuseppe
72