ASSISTEDTARGETRECOGNITIONFOR X-RAY CARGOSCANNING

www.acxis.eu

Automated Comparison of X-ray Images forCargo Scanning (ACXIS) is a research projectfunded by the European Union under the 7thFramework Program. The ACXIS consortium(including CASRA) conducts research anddevelopment for improving effectiveness andefficiency of cargo screening. As part of this,ACXIS partners are developing AssistedTarget Recognition (ATR) methods toautomatically detect potentially illicit goods1

by means of dedicated algorithms. For thispurpose, first the Dutch Customs Organization(DTCA) and the Swiss Federal CustomsAdministration (FCA) defined several illicitgoods detection scenarios. The consortiumthen prioritized and selected scenarios,resulting in four ATRs being developed withSmiths Detection in the lead. As an example,methods for cigarette detection based ontexture analysis have been developed andshow excellent first results. While the overallproject has been described in the CASRANewsletter issue 12 (March 2016), thepresent article is focusing on the validationstudy of the ACXIS project.

How effective ATRs are does not only dependon the performance of the automatic algorithms.It is also important, how the informationprovided by the ATRs is implemented into thescreening process. The idea within ACXISwas that ATRs support visual inspection ofcustoms officers by directing their attentionon suspicious areas in the X-ray image andhelp decision making (red frame in Figure 1).In this implementation scenario, the screeningofficer still makes the final call on whetherthe cargo needs further inspection. Therefore,whether and how much the detection

performance of the screener improves due toATRs, depends on a variety of factors.Research on human–machine interaction invarious professions revealed that the benefitof automated support systems is influencedby the officers’ experience with the system([1]), their expertise in the task ([2]), and theoccurrence of hits and false alarms2 generatedby the system (e.g. [1], [3]). Research hasalso shown that automated support systemsdo not always lead to better performance(e.g. [4]). A key component of the ACXISproject was therefore to evaluate the designedimplementation of the ATRs. Since the officers’experience with the system and their expertisein the task can influence the benefit of anautomated system, the successful use of theATRs might depend on whether screeners aretrained and had the chance to gain experiencewith the ATRs. Computer-based training hasrepeatedly been shown to increase detectionperformance in X-ray image inspection of airpassenger bags (e.g. [5], [6]) and one studyalso showed its benefit in cargo screening([7]). The ACXIS validation study thereforealso aimed to investigate the influence oftraining. The validation study was designedwith following goals:

• Replication of the previously found effects of computer-based training on detectionperformance in cargo screening

• Evaluate whether ATRs lead to an increasein detection performance

• Investigate whether the benefit of the ATRsis influenced by the computer-basedtraining

Validation of improved customs detection proceduresHow useful are Assisted Target Recognition algorithms and trainingfor improving the detection performance of customs officers?

Authors: Yanik Sterchi, Wicher Visser

2

METHOD: SIMULATORFor the validation study, CASRA has developeda new platform for cargo X-ray imageinterpretation training and testing, called theCustoms X-Ray Simulator (described in theCASRA Newsletter issue 13, July 2016).

ATRS In the validation study, four assist functionshave been evaluated. ATRs 1-3 aimed atdetecting specific potentially illicit goodswithin the freight: hidden cigarettes (ATR1),weapons (ATR2) and narcotics (ATR3). Afourth ATR detected anomalies in thecontainer structure (ATR4), and was thereforeable to detect any sort of potentially illicitgoods, but only in the container structure andnot within the freight (Figure 2). For a bettercontrol over the experimental conditions, theofficers were trained and tested separately forthe location-specific ATR4 and the target-specific ATRs 1-3. The detection algorithms of

the ATRs have been still under developmentwhen the validation study was conducted,and the alarms of the ATRs were thereforeset manually by screening experts. All fourATRs were designed to detect 75% of theirrespective targets, while 25% of the X-rayimages produced a false alarm.

CONTENTTo create the training and testing material, theDutch Customs Organization (DTCA) providedimages of cargo recorded with a HCV scanner(also appropriate waybills were included butno top views of the X-ray scans). Empa andDTCA recorded cigarettes, weapons, andmockup narcotics, which were merged intocargo images by CASRA, using a mergingtool from CEA. Officers from DTCA andSwiss Federal Customs Administration (FCA)reviewed the images for quality (these officerswere excluded from the study).

Figure 1: Simulator interface for training and testing with ATRs

IMAGE DISPLAYORIGINAL

ZOOM100%

USERUSER_NAME

WAYBILL

Waybill

HS Code Trade name Weight [gr] Quantity9201.10 Upright pianos 531000 1 pcs8421.11 cream separators - 12 pcs

ATR RESET

1x

1x 2x 3x 4x HIGHLOW

2x 4x NEGLOWLUM

HIGHLUM

PSEUDOCOLOR

SEN B/W

MODULECUSTOM

TIME SPENT01:02:34

TIME LEFT01:15

EXIT Exit

Assist frame «cocaine»

Confidence rating

Response buttons(NOT OK / OK)

Image enhancementsZoom functionThumbnail view

This indicatorilluminates when a frame is displayed

Waybillinformation

Please indicate how confidentyou are with your answer.

unsure sure

Continue

1 2 3 4 5

3

PARTICIPANTS & DESIGN46 employees of the DTCA and 21 employeesof the FCA completed the study. All participantsconducted the Ishihara test in order to confirmtheir color perception ability, and their objectrecognition skills were assessed with the X-Ray ORT ([8]). Participants that passed theIshihara test were divided into three groupswith comparable object recognition skills (withthe exception of the control group of DTCA,which consisted of administrative personnel).As illustrated in Figure 3, the first group didnot receive training or assistance by ATRsand served as control group. The secondgroup (training group) was tested and trainedwithout ATRs, while the third group was testedand trained with ATRs. Tests to measuredetection performance in cargo screeningtook place once before training and thenagain at the end of the study. Each testconsisted of two parts, part one with illicitgoods only hidden within the freight and parttwo with half the illicit goods hidden in the

container structure (while the other halfremained hidden within the freight). In the ATRgroup (AG), part one used ATRs 1-3 and parttwo ATR4. Each part consisted of 96 imageswith illicit goods in one fourth of the images,resulting in eight targets for each of the threecategories of illicit goods. For the defined hitand false alarm rate of the ATRs this meantthat an alarm indicated an illicit good in 50%of the cases (and missing 25% of the illicitgoods). Participants agreed to two trainingsessions of 15 minutes per week over thecourse of four months. The training consistedof three difficulty levels where participantsprogressed to the next level after three to fourhours of training.

Figure 2: X-ray image of a container with hidden illicit goods

4

RESULTS & DISCUSSIONSince the tests consisted of images fromDTCA only, which differed notably from theimages familiar to officers from FCA (resultingin higher false alarm rates), results wereanalyzed for DTCA and FCA separately andshould not be compared between the twocustoms organizations.

Figure 4 summarizes the results of test partone (illicit goods within the freight only). Ascan be seen, for the training group detection(hit rate) was higher in Test 2 compared toTest 1, while for the control group, detectionremained constant. Statistical analysesconfirmed that training significantly improveddetection for officers from both customsorganizations. Further analyses revealed thatthis improvement did not differ significantlybetween the three categories of illicit goodsand that the false alarm rate was not affectedby training. Figure 4 also shows that detectionwas higher for the ATR group than the training

group, meaning that ATRs for the detection ofcigarettes, drugs, and guns could improvedetection when looking at these categoriescombined. Analyses confirmed this improvementto be statistically significant and not to differsignificantly between Test 1 and Test 2. Thismeans that the benefit of the ATRs (morespecifically: the difference between thetraining group and ATR group) was about thesame before and after training. Furtheranalyses revealed that the false alarm ratewas not influenced by ATRs but that thebenefit of ATRs differed between the threecategories of illicit goods, being larger for thedetection of guns than for the detection ofcigarettes and narcotics. This might beexplained by the size of the guns, which wererelatively small and therefore easily missed ifnot highlighted by the ATR.

Training Group (TG)Colour

PerceptionTest

ORT Test 1 Training Test 2

Control Group (CG)

Test 1 No Training Test 2

ATR Group (AG)Test 1

(with ATR frames)Test 2

(with ATR frames)Training

(with ATR frames)

Figure 3: Experimental design of the study

5

In test part two, where half of the illicit goodswere hidden in the container structure (Figure2) and the other half within the freight, trainingalso had a positive effect on the hit ratewithout affecting the false alarm rate. ButATR4 (detection of anomalies in the containerstructure) showed both, benefits anddisadvantages: the participants who wereassisted by ATR4 detected more illicit goodsthat were hidden in the container structure,but missed more of the other illicit goodswithin the freight. A possible explanation forthis might be that the ATR directed theofficers’ attention more towards the containerstructure and away from the freight.

Overall, the study replicated that training can bean important contribution to improve detectionperformance in cargo screening. The ATRs ledto improved detection performance, but theirbenefit depended on the category of illicit goods,and ATR4 also showed some undesirable side

effects. These results highlight the importanceof research in human–machine interaction,before automated support systems areimplemented in practice.

Contact us at info@casra.ch to learn moreabout CASRA’s research and applications.

1 While some goods are generally forbidden (e.g.cocaine), many goods require permits and are illicit ifnot properly declared on the waybill.

2 A hit is defined as a correct alarm by the system, e.g. if a potentially illicit good is hidden in the cargoand the system generates an alarm. If an alarmincorrectly indicates the presence of a potentiallyillicit good while no such good is hidden in the cargo,this is considered a false alarm.

Test 1

DTCA FCA

CG

TG

AG

Condition

Test 2 Test 1 Test 2

HR

Figure 4: Mean hit rate of test part one, before and after training. Error bars indicate standard errorsof the mean

6

REFERENCES[1] Bliss, J., Dunn, M., Fuller, B.S., 1995.Reversal of the cry – wolf effect: Aninvestigation of two methods to increase alarmresponse rates. Perceptual and Motor Skills,80 ,1231-1242.

[2] Automation in airport security X-rayscreening of cabin baggage: Examiningbenefits and possible implementations ofautomated explosives detection. Manuscriptsubmitted for publication.

[3] Halbherr, T., Schwaninger, A., Budgell,G.R., Wales, A., 2013. Airport SecurityScreener Competency: A Cross-Sectional andLongitudinal Analysis. Int. J. Aviat. Psychol.,23(2), 113-129.

[4] Koller, S., Hardmeier, D., Michel, S.,Schwaninger, A., 2007. Investigating trainingand transfer effects resulting from recurrentCBT of x-ray image interpretation, in D.S.McNamara and J.G. Trafton (Eds.), Proceedingsof the 29th Annual Conference of the CognitiveScience Society (pp. 1181-1186). Austin, TX :Cognitive Science Society.

[5] Michel, S., Mendes, M., de Ruiter, J.,Koomen, G., & Schwaninger, A. (2014).Increasing X-ray image interpretationcompetency of cargo security screeners.International Journal of Industrial Ergonomics,44, 551-560.

[5] Parasuraman R., Sheridan T.B., WickensC.D., 2000. A model for types and levels ofhuman interaction with automation. IEEETransactions on Systems Man andCybernetics Part a-Systems and Humans.30(3):286-297

[6] Strauch, B., 2016. The Automation-by-Expertise-by-Training Interaction. WhyAutomation-Related Accidents Continue toOccur in Sociotechnical Systems. HumanFactors: The Journal of the Human Factorsand Ergonomics Society. doi:10.1177/0018720816665459

7

www.acxis.eu