ols2008 luis claudio goncalves application testing

8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing

1/37

Application Testing under

Realtime Linux

Luis Claudio R. GonalvesRed Hat Realtime Team

Software Engineer


2/37

Agenda

* Realtime Basics

* Linu and t!e "REE#"T$RT %atc!

* &'out t!e Tests

* Loo(ing for 'ad %rogramming %ractices

* Bad "riorit) &ssignment

* Resource &llocation

* C!anging &%%lication Be!avior

* Com%aring &%%les to &%%les

Conclusion


3/37

Realtime Basics


4/37

Realtime Basics

* eterminism a'ilit) to sc!edule t!e !ig!est%riorit) tas(s in a consistent and %redicta'le wa).

* Latenc) time 'etween a given event and t!edesired effect.


5/37

'Almost' is not enough...

* +!en determinism matters...

* ,av)- inancial Services- ederal sector-Telco/,etwor(

* inancial Services01 first to answer !as 'etter c!ances on t!e 'ids1 legal re2uirements for consistenc) in o%erations

* Telco / ,etwor(1 networ( %ac(et management1 3oS


6/37

Latency...


7/37

Latency in detail...


8/37

Sources of latency

* &%%lication "riorit)1 one a%%lication 'loc(s 4or %reem%ts5 anot!er1 !olds a contended resource 4loc(5

* T!e 6ernel1 w!en t!e (ernel runs a%%s sto%1 t!e longer t!e (ernel runs t!e longer a%%lications wait1 determinism gets limited ') t!e longest code%at!

Hig! "riorit)&%%lication

7nterru%t

&%%lication continues

6ernel0 interru%t !andler 8

sc!eduler


9/37

Linux + PREEMPT_RT


10/37

PREEMPT_RT patch

* #et!odical im%lementation- focusing u%stream ado%tion

* Started ') t!e 'uilding 'loc(s t!at would ena'le t!eim%lementation

* Com%lete RT- not 9ust a :#/sc!eduler !ac(

* #ature features- t!at 'enefits 'ot! RT and u%stream-!ave 'een merged

* Communit) 8 sand'o 4linu1rt1users5


11/37

Building Blocks

* T!readed 7R3s

* T!readed Softir2s

* Re%laced sema%!ores and s%inloc(s ') 4rt5mutees

* Slee%ing s%inloc(s

* "riorit) 7n!eritance


12/37

Strategies

* #easure and 7dentif) t!e longest code %at! in t!e (ernel* En!ance algorit!ms to ena'le more concurrenc)* S!orten t!e time t!e (ernel runs as non1%reem%ti'le* efer %rocessing to lower %riorit) (ernel t!reads

Hig!"riorit)&%%lication

7nterru%t

&%%licationcontinuesand finis!es

7nterru%t Handlersc!edules lower%rio t!read to

!andle t!einterru%t

7nterru%t Handler0lower %rio t!readcom%letes data

transfer

,et !ig!%riorit)&%%lication


13/37

About the Tests


14/37

About the tests

* :alidation of t!e 6ernel and Customer &%%s

* ;7t runs slower on RTt s!ow )ou or c!ange t!e code...=

* +!ere t!e %ro'lem lies? 4(ernel- a%%- com%arison5


15/37

Bad Practices


16/37

Bad Coding Practices

* Bad coding %ractices 4focus on Realtime51 some %ractices are 'ad onl) in t!e contet of "REE#"T$RT1 a%%s %orted from ot!er/older @S1 tunings t!at no longer ma(e sense

* How to s%ot 'ad %rogramming %ractices?1 s)stem calls t!at s!ould 'e avoided1 'ad %arameter assignment

* Reading t!e source code

* S)stemta% scri%ts

* Strace

* Eam%le0 avoid sc!ed$)ield45


17/37

Example

* How to s%ot a given s)scall usage

A cat test$)ield.st%

%ro'e s)scall.sc!ed$)ield

%rintf4 Ds4Dd0Dd5 Called sc!ed$)ield45n- eecname45- %id45-tid455F

A sta% 1v test$)ield.st%rs)slogd4IJ0IK5 Called sc!ed$)ield45

rs)slogd4IJ0JMIN5 Called sc!ed$)ield45rs)slogd4IJ0JMIN5 Called sc!ed$)ield45


18/37

Priority Assignment


19/37

The Short Version...

* "rioritiOe w!at is im%ortant1 c!rt is )our friend1 sc!ed$setsc!eduler451 alwa)s c!ec( >man M% PfunctionQ>1 avoid SCHE$7@ %riorit)


20/37

Priorities

* 7s t!e &%% suffering non1voluntar) %reem%tion?1 Tuna1 c!ec( /%roc/P%idQ/status 1 4nonvoluntar)$ctt$switc!es51 use t!e function getrusage451 create a s)stemta% scri%t to verif) t!is 4and ot!er5 information1 c!ec( t!e %riorit) of t!e a%%lication and its t!reads

rootvoid UVA %s 1emo %id-tid-%olic)-%ri-rt%rio-cmd

...MWJ 1 1 1 1 /usr/li'N/.../firefo1'in 1 MWJ TS 1 1

1 MWNW TS 1 1 1 MWN TS 1 1 1 MWKK TS 1 1 1 MWK TS 1 1 1 MWK TS 1 1 1 MW TS 1 1 1 WWKN TS 1 1

...


21/37

Priorities

*Eam%le s)stemta% scri%t

A if )ou !ave %ro'lems !oo(ing on eit$mm-A use %rofile$tas($eit instead.

%ro'e (ernel.function4eit$mm5 %rintf4Ds4Dd0Dd5 stats0n- eecname45- %id45- tid455

%rintf4t"ea(RSS0 Dd6B t"ea(:#0 Dd6Bn- Xtas(1Qmm1Q!iwater$rss*N- Xtas(1Qmm1Q!iwater$vm*N5

%rintf4tS)stem Time0 Ddms tYser time0 Ddmsn- Xtas(1Qstime- Xtas(1Qutime5

%rintf4t:oluntar) Contet Switc!es0 Dd t7nvoluntar)0 Ddn- Xtas(1Qnvcsw- Xtas(1Qnivcsw5

%rintf4t#inor "age aults0 Dd t #a9or "age aults0 Ddn- Xtas(1Qmin$flt- Xtas(1Qma9$flt5

F


22/37

Priorities

* Results0

ss!d4MIW0MIW5 stats0 "ea(RSS0 KWN6B "ea(:#0 NN6B S)stem Time0 ms Yser time0 Mms

:oluntar) Contet Switc!es0 MJ 7nvoluntar)0 I #inor "age aults0 MNI #a9or "age aults0 I

id4MIJ0MIJ5 stats0 "ea(RSS0 K6B "ea(:#0 N6B S)stem Time0 Jms Yser time0 Ims :oluntar) Contet Switc!es0 M 7nvoluntar)0 I #inor "age aults0 JJ #a9or "age aults0 I


23/37

Priorities

* ,otes a'out t!e s)stemta% scri%t

1 gat!ers information from tas($struct

1 eit45 1Q do$eit45

1 eit$mm45

* #ore o%tions are coming


24/37

Priorities

* "rioriOe (ernel t!reads 47R3s- Softir2s- ...57.e.0 &%%lications de%ending on networ( and not using dis( 7/@

N %osi$c%u$timerV

K softir21!ig!/IV softir21timer/IV softir21net1t/V W softir21net1r/V softir21'loc(/IV I softir21tas(letV softir21sc!ed/IV J softir21!rtimerV

WW (ac%idV I 7R31V W (sus%end$us'dV JN aio/IV MKI 7R31WV MWN 7R31NV NI 7R31V

NI %ccarddV


25/37

Resource Allocation


26/37

Good Habits

* &llocate Resources in advance1 minor / ma9or %age faults1 memor) 4mloc(51 create t!read %ool in advance

* Bind %rocesses to C"Ys and 7solate C"Ys1 tas(set1 isolc%us in t!e (ernel command line1 Tuna


27/37

Resource Allocation

* #emor) &llocation1 minor %age faults1 ma9or %age faults1 mloc(45 and friends1 don>t forget touc!ing memor) 'efore mloc(>ing1 can 'e c!ec(ed via /%roc- getrusage45 or s)stemta% scri%ts

* T!read creation1 can ta(e more t!an IIus1 can suffer %age faults

* T!reads1 some s)stem calls can generate %age faults 4i.e. fo%en51 select45 !as a ms resolution


28/37

CPU Affinity

* &llocate %rocesses to C"Y1 i.e. avoid running sender and receiver on t!e same C"Y

1 sometimes- tr) running sender and receiver on t!e same C"Y1 tas(set- eidus1 some (ernel t!reads can>t 'e moved1 tr) Tuna

* C"Y 7solation1 'oot %arameter- isolc%us1 removes all 4%ossi'le5 %rocesses from t!e C"Y


29/37

Application Behavior


30/37

Changing App Behavior

* ,o access to source code1 +!en s)stem tuning is not enoug!...1 +!at if t!is or t!at configuration was in use?1 !ac(ing t!e (ernel

1 Ysing s)temta%1 Li'rar) %reload tric(s1 6ernel !ac(s


31/37

Changing App Behavior

* Li'autocor(1 "erformance is (ing1 ,agle &lgorit!m1 TC"$,@EL&Z1 TC"$C@R6

A L$"REL@&[li'autocor(.so ./customer$a%%

* Li'localiOe


32/37

Comparing Results


33/37

Apples to Apples

* &7T1 Have )ou attended &rnaldo>s %resentation?1 +!ic! (no's were set?

1 id t!e tests run on t!e same environment?1 +!at was different?1 How to re%roduce a given test on a different s)stem?1 How to com%are all t!e tests t!at !ave a given set of features?


34/37

Conclusion


35/37

Conclusion

* Some features of "REE#"T$RT ma(es it uni2ue wit! regardsto s)stem tuning

* Several test cases solved wit! minimal s)stem and a%%licationtuning

* T!ere are cases w!ere a%% 'e!avior (ills t!e 'enefits oftunings

* Clues for finger%ointing


36/37

Questions...?

* T!e answer is NJ


37/37

References

L7B&YT@C@R6!tt%0//git.(ernel.org/%[linu/(ernel/git/acme/li'autocor(.git\a['lo'$%lain\f[tc%$nodela).tt

Real1Time +i(i "age!tt%0//rt.wi(i.(ernel.org

Tec!ni2ues t!at can !ave its 'e!avior c!anged w!en t!e (ernel is re%laced!tt%0//oo%s.g!ost%rotocols.net0W/acme/un'e!aved.tt

,etta%s!tt%0//oo%s.g!ost%rotocols.net0W/acme/netta%s.tar.'OJ
http://rt.wiki.kernel.org/http://rt.wiki.kernel.org/

ols2008 luis claudio goncalves application testing

Documents