smart and adaptive interfaces for inclusive work environment · 2.2.1 expert usability evaluation...

SmartandadaptiveinterfacesforINCLUSIVEworkenvironment

GrantAgreementN°723373

_______________________________________________________________________________________

Deliverable6.1–Reportonmethodsandtoolstomeasureworkersatisfaction

Contractualdeliverydate:2018-09-30

Actualdeliverydate:2018-09-30

ResponsiblePartner:CIOP-PIB

Author(s):DorotaŻołnierczyk-Zreda,ZofiaMockałło,KamilaNowak

WPn°:6

WPleader:CIOP-PIB

ProjectCoordinator:Prof.CesareFantuzzi

ProjectCoordinatorOrganisation:UNIMORE

Disseminationlevel:Public

TheresearchleadingtotheseresultshasreceivedfundingfromtheEuropeanUnion’sHorizon2020researchandinnovationprogrammeundergrantagreementN723377.

The author(s) is/are solely responsible for its content, it does not represent the opinion of theEuropeanCommissionandtheCommissionisnotresponsibleforanyusethatmightbemadeofdataappearingtherein.

ExecutiveSummary

Thisdeliverablereportsonmethodsandtoolstomeasureworkersatisfaction.Specifically,aliteraturereviewonmethodsandtoolstomeasureworkersatisfactionwithHMIinparticularandworkingeneralhasbeenconducted.OnthebasisofthisreviewaswellaspreviousDeliverables,amodelofworkersatisfactionhasbeen proposed and a questionnaire tomeasureworker satisfactionwith the adaptive HMI andworkingconditions has been developed. The questionnaire consists of two main parts: I. Working conditions(psychosocialandphysical);andII.SatisfactionwiththeadaptiveHumanMachineInterface(HMI),includingsafety issues. It has also been proposed to use physiological datameasurement inWP7 as an objectivemethodtomonitorworkersatisfaction.

TableofContent

1. Introduction _______________________________________________________________________ 1

2. Literaturereviewonmethodsandtoolstomeasureworkersatisfaction _______________________ 2

2.1. Usabilityconcepts_______________________________________________________________ 2

2.2. Methodsandtoolstomeasureusabilityandusersatisfaction____________________________ 3

2.2.1Expertusabilityevaluation _________________________________________________________ 3

2.2.2End-usersatisfactionevaluation_____________________________________________________ 6

2.2.3Objectiveusabilitymeasurement____________________________________________________ 8

2.3Satisfactionwithworkingeneral______________________________________________________ 9

2.4ConclusionsformeasuringworkersatisfactionintheINCLUSIVEproject _____________________ 10

3. TheusersatisfactionwiththeadaptiveHMIandworkingconditionsmodel____________________ 11

4. Thequestionnaire__________________________________________________________________ 12

4.1.Questionnaire’spreamble__________________________________________________________ 12

4.2.Occupationalanddemographicdata _________________________________________________ 12

4.3WorkingConditions _______________________________________________________________ 12

4.3.1PhysicalWorkingConditions_______________________________________________________ 12

4.3.2PsychosocialWorkingConditions___________________________________________________ 12

4.4SatisfactionwiththeadaptiveHMI ___________________________________________________ 13

4.4.1Safety_________________________________________________________________________ 13

4.4.2Satisfactionwiththedesign/visibilityoftheinterface___________________________________ 13

4.4.3Satisfactionwithease ____________________________________________________________ 14

4.4.4Satisfactionwithefficiency________________________________________________________ 14

4.4.5SatisfactionwiththeAdaptmodule _________________________________________________ 14

4.4.6SatisfactionwiththeMeasuremodule_______________________________________________ 14

4.4.7SatisfactionwiththeTeachmodule _________________________________________________ 14

4.4.8OverallSatisfactionwiththeadaptiveHMI____________________________________________ 14

5. Objectivemeasurementofworkersatisfaction___________________________________________ 16

6. Conclusion________________________________________________________________________ 17

7.References _________________________________________________________________________ 18

Appendix1–‘SatisfactionwiththeadaptiveHMIandworkingconditions’questionnaire

1

1. Introduction

TheINCLUSIVEprojectaimsatdevelopingsmarthuman-machineinterfaces(HMIs)thatadapttohumanskillsandcapabilities,andprovidetrainingandsupport.

TheWP6of theproject focusesondevelopingamethodologyofmeasurementandevaluationofworkersatisfaction in the industrial environment,with examples taken from the three industrial use cases. Themethodologytomeasureworkersatisfactionwillbebasedonsubjectiveandobjectivemeasures.TheresultofthisWPisamethodologytoassesstheadaptiveworkingenvironmentfromthehumanpointofview.

In this regard, this deliverable reports on outcomes of literature review conducted in order to analysemethodsand tools tomeasureworker satisfaction.Worker satisfaction isunderstood in thisprojectasaspecific case of user satisfaction, which refers to working environments, and work-related products orservices,includingHMIinterfacesandotherICT-basedsolutionsimplementedincontrolsystemsofindustrialmachinery andautomatedmanufacturing systems. Therefore, the literature reviewhasbeen focusedonmeasuresandtoolstoevaluatetheusabilityandsatisfactionwithHMIandwithworkingeneral.ResultsofthereviewarepresentedintheParagraph2.

Next,amodelofworkersatisfactionhasbeenproposed.Itincludesthreemainfactors:usersatisfactionwithHMI(includinghealth&safetyissues),physicalworkingconditions,andpsychosocialworkingconditions&ethical aspects, taking into consideration individual differences, such as age, experience, capabilities,perception,cognitionormotorskills.ThemodelofsatisfactionwiththeadaptiveHMIandworkingconditionsispresentedintheParagraph3.

On thebasisof the literature reviewand theproposedworksatisfactionmodel,aquestionnaireentitled‘Satisfaction with the adaptive HMI andworking conditions’ has been developed. It is presented in theParagraph4.Thequestionnairecontainstwomainparts:I.Workingconditions;andII.SatisfactionwiththeadaptiveHumanMachineInterface(HMI),surveyingalsoskillsandcapabilitiesoftheusers.Inordertoensureworkers’satisfactionandparticipation, thequestionnairecontainsquestionsonproposedchangestotheHMI.ThequestionnairewillbepilottestedinTask6.2andthenadjustedforimplementationintheWP7.

As the worker satisfaction evaluation should be conducted by means of both subjective and objectivemeasurement, the ideaofphysiologicaldatameasurement intheWP7aspartof theMeasureModule isbrieflydescribedintheParagraph5.

Finally,theParagraph6followswithsomeconcludingremarks.

2

2. Literaturereviewonmethodsandtoolstomeasureworkersatisfaction

Theusersatisfactionderivesfromtheusabilityconcept.Ahighsystemusabilityisagoalofusers,employers,developersandresearchers.Itfacilitatessafe,productiveandenjoyableworkintheworkingenvironment(Ovaska,1991).Hence,accountingfortheHMIusabilityisoneofthekeyprojectandWP6goals.Inordertomeasure theHMI user satisfaction,we need to first understandwhat the usability principles are. In theliteraturereviewwepresentusabilityconceptsandmeasurementmethodsbasedonanexpertevaluationandend-user (worker) satisfactionmeasurement,bothsubjectiveandobjective. Itwasagroundwork forchoosingappropriatemetricstobeincludedintheworkersatisfactionmeasurement.

2.1. UsabilityconceptsUsabilityisabroadtermwithvariousdefinitions,dependingontheconceptorcontext.

Other,morebroadusabilitydefinitionsare:

• Ensuringthat interactiveproductsareeasyto learn,effectivetouseandenjoyablefromtheuserperspective(Rogersetal.,2011)

• Multidimensionalcharacteristicinthecontextofusersperformingtaskswithaproductinaspecificenvironment(Bevan,Kirakowski&Maissel,1991)

• The extent to which a product can be used by specified user to achieve specified goal witheffectiveness,efficiencyandsatisfactioninacontextofuse(ISO9241)

AccordingtoISO9241,effectivenessmeanstheaccuracyandcompletenesswithwhichspecifieduserscanachievespecifiedgoalsinaparticularenvironment.Efficiencymeanstheresourcesexpendedinrelationstothe accuracy and completeness of goals achieved. Satisfaction is defined as user comfort and useracceptabilityoftheworksystemandinrelationtootherpeopleaffectedbyitsuse.

In attempt of creating most comprehensive approach,Wang & Huang (2015) combined three differentusabilityprinciples:Nielsen’s(1993),Norman’s(2003)andYeh’s(2010).

Nielsen (1993) defined the following usability principles: memorability, errors, learnability, efficiency,satisfaction.

Norman’s(2003)conceptionofusabilityincludedsuchprinciplesas:visibility,agoodconceptualmodel,goodmappings,feedback.

Yeh’s(2010)usabilitydefinitionincludedfollowingprinciples:ease,enjoymentandeffectiveness.

On thebasisof theseconceptsandcomponents,Wang&Huang (2015)proposed the followingusabilityprinciples(fig.1):

• Visibility:clearinstructionsandinformation• Ease:easytolearnandfamiliarise,timeoflearningisminimal• Efficiency:oncelearneditiseasytousethefunctionsofthesystematfullcapacity• Enjoyment:usersaresatisfieduponcompletingataskwhenusingthesystem

3

Fig.1:Usabilityprinciples.Source:Wang&Huang,2015

InthecontextofHuman-Computer Interaction(HCI),Shackel (1991)definedusabilityas ‘thecapability inhuman functional terms tobeusedeasilyandeffectivelyby the specifiedrangeofusers, given specifiedtrainingandusersupport,tofulfillthespecifiedrangeoftasks,withinthespecifiedrangeofenvironmentalscenarios’.

Preece(1993)stated:`ThegoalsofHCIaretodevelopandimprovesystemsthatincludecomputerssothatuserscancarryouttheir tasks:safely,effectively,efficientlyandenjoyably.Theseaspectsarecollectivelyknownasusability’.

2.2. MethodsandtoolstomeasureusabilityandusersatisfactionTheusabilitycanbeevaluatedbymeansofasubjectiveandobjectivemeasurement,wherebythesubjectiveevaluationofusabilityincludesscalesandquestionnaires,whilstobjectivemeasurescomprisebehaviouralorphysiologicalmeasures.Theusabilityisalsomeasuredbyorwithhelpofexpertevaluators,andbyend-usersintermsofusersatisfaction.

Themost commonusability evaluation techniques are inspectionmethods and testmethods. Inspectionmethods identify usability problems and possibilities for improvement by checking it against establishedstandards. Usability test methods include testing usability with end users, in the presence of expertevaluators or by means of subjective user satisfaction questionnaires (Holzinger, 2005). Although it iscommon to understand the expert usability evaluation as an objectivemeasurement,we describe threegroupsofusabilitymeasurementmethods:expertusabilityevaluation,end-usersatisfactionandobjectivemeasurement,understoodasmonitoringbehaviouralorphysiologicaldata.

2.2.1ExpertusabilityevaluationSeveralapproachescanbeusedtoevaluatetheinterfaceusability(Lin,Choong&Salvendy,1997),e.g.:

• Thinking-aloud approach – end-users think aloudwhile they are using the system that is beingevaluated.Althoughthismethodincludesend-users,thereisanexpertevaluatortointerpretandassessuserreactionstothesystem.Itisaformofatestmethod.

• Guidelines/checklists–interfaceguidelinestobefollowedbydesigners.Itisaformofausabilityinspectionmethod.

4

• Heuristicevaluation-istheapplication(byevaluators)ofasetofheuristicstojudgetheadequacyoftheprototypedesign.Itisaformofausabilityinspectionmethod.

Ovaska(1991,p.51)noticedthatusabilityinspectionmethodssuchasguidanceandchecklistaredesignedto be used as engineering tools. Usually, they involve testing the interface in laboratory settings usingexample tasks. Themaindisadvantageof suchmeasures is that theyevaluate inadequateaspectsof thesystem,becausesomeaspectsofthesystemparticularlyimportanttouserscouldbeoverlooked(Ovaska,1991,p.51).

Guidance/checklistsareanexampleofameasureusedbythedevelopmentteamtogetherwiththeusersorexternalevaluators.Theevaluationisconductedonthebasisofanexemplarytaskdesignedbythesystemdevelopers after the user completes the task. An example of such amethod is the Evaluation ChecklistdevelopedbyRavdenandJohnson(1989).Thischecklistincludesthefollowingusabilityaspects:

• Visualclarity• Consistency• Compatibility• Informativefeedback• Explicitness• Appropriatefunctionality• Flexibilityandcontrol• Errorpreventionandcorrection• Userguidanceandsupport

The tool proposed by Ravden and Johnson (1989) consists of 9 sections, each from 13 to 17 questions.Examplequestionsare:‘Doesthesystemvalidateuserinputsbeforeprocessing,whereverpossible?’,‘Whereinterfacemetaphorsareusedaretheyrelevanttothetaskscarriedoutwiththesystem?’.AsOvaskanoticed,suchachecklistcanbedifficultfortheuserstofillinbecauseofdevelopers’terminologyorthelengthofthetool.Italsodoesnotincludeuserfeedbackandsuggestionsoncorrectiveactions.

Mostpopularusabilityinspectionmethodsareheuristicevaluation,togetherwithcognitivewalkthroughandheuristicwalkthrough(Cocktonetal.,2008).

• Cognitivewalkthroughprovidesevaluatorswithstep-by-stepinstructionsontheactionsequencetobe performed by users, where evaluators should determine whether the user would follow thecorrectaction(Lucas&Babaian,2015,p.844).Questionstobeansweredare:

- Willtheuserbetryingtoachievetherighteffect?- Willtheusernoticethatthecorrectactionisavailable?- Willtheuserassociatethecorrectactionwiththedesiredeffect?- Ifthecorrectactionisperformed,willtheuserseethatprogressisbeingmade?

Althoughthismethodprovedtobeabletoidentifyuniqueproblems,notidentifiedusingothermethods,italso has a tendency to overlook problems related to missing functionalities or error recovery (Lucas &Babaian,2015).

• Heuristicevaluation(Nielsen,1994)isamethodwhereanevaluatorassessesapartoftheinterfaceaccordingtothefollowingheuristics:

- Visibilityofsystemstatus- Matchbetweensystemandtherealworld- Usercontrolandfreedom- Consistencyandstandards- Errorprevention

5

- Recognitionratherthanrecall- Flexibilityandefficiencyofuse- Aestheticandminimalistdesign- Helpsusersrecognise,diagnoseandrecoverfromerrors- Helpanddocumentation

AsLucasandBabaianstated(2015),evaluatorsusingthismethodaregivenlittleguidanceconcerningtheusersorthetaskstoperform.

• Heuristicwalkthrough(Sears,1997)isatwo-passprocess,whereevaluatorsareprovidedwithalistofheuristicsandquestions,butalsowithaprioritisedlistofusertasks.Thismethodincludesafree-formandtask-specificevaluation.Thismethod,comparingtothecognitivewalkthroughandheuristicevaluation,wasproventoidentifymoreintermediate,minorandsevereusabilityproblemsthanthecognitivewalkthrough.Theheuristicwalkthroughandcognitivewalkthroughmethodstendedtoyieldlessfalsepositivesthantheheuristic evaluation, but this is explainedby the fact that theheuristic evaluationmethoddoesnot testspecifictasks,unliketwoothermethods(Lucas&Babaian,2015).

Anothermethodforexpertevaluationoftheuserinterfaceusabilityisthecollaborativecritique,designedfortestingHuman–ComputerCollaboration(Lucas&Babaian,2015).Theaimofthismethodistolowermentalloadofusers.Accordingtothisconcept,thesystemshouldserveitsuserstomeettheirgoals.Hence,thefocusisonthesystemusabilityinthecontextofthecurrenttask,userandinteraction,andnotonusercapabilities touse thesystemefficiently.Collaborativecritiquemethodconducts theassessment stepbystep. On every stage, experts-evaluators answer questions assessing the quality and possibility of thecollaborationandthesystem’sreactionstoerrors.Evaluatorsareaskedtofamiliarisethemselveswithagiventaskandemployee’sprofile.Everytaskisdividedintosub-taskswhicharethestagestobeevaluated.Theevaluators should be supported by experts in the field in the process of usability assessment. Thecollaborativecritiquemethodisconductedusingthefollowingquestions:

• Willtheuserfindtheoptionsforwhatshewantstodointhecurrentscreen?• Fortheusertofigureoutwhattodonow,howmuchexplorationisinvolved?• Fortheusertofigureoutwhattodonow,howmuchconfusionisinvolved?• Isthesystemusingknowledgeofthetaskingeneral,thecurrentuser,andthecontextofthecurrent

actiontothefullestextentinordertoa)appropriatelyguidetheuser?;b)reducetheeffortinvolvedinuserinput?

• Afterexecutionofthecurrentaction,willtheuserunderstand:a)whatprogresshasbeenmadesofartowardcompletingtheoveralltask?;b)whatdoesremaintobedoneinordertocompletetheoveralltask?

• Doesthesystemdisplayinformationthatclearlyexplainstheproblemtotheuser?• Doesthesystempresentstepstheusercantakeforpossiblecorrectiveactions?• Doesthesystempresentaneasywaytotakecorrectiveactions?

Althoughthismethodseemstobecomprehensiveinthecontextofthesystem’sabilitytocollaboratewiththe user, it does not contain themain usability components and amore subjective dimension, e.g. usersatisfaction.Moreover,theevaluationhastobeconductedbyusabilityexpertsandexpertsinthefield,notbyemployees.

The usability testing conducted by external expert evaluators usually in laboratory settings is a popularmethodtodesignusablesystemsorproducts.However,itischaracterisedbysomedisadvantagessuchaslackoftheend-userinvolvementintheprocessorlackofarealindustrialenvironmentandafailuretotesttheusability in everydayworkingprocesses. Including theuser assessmentof usability seems tobeof aparticularlyhighimportance.

6

2.2.2End-usersatisfactionevaluationOther approach includes measuring end-user satisfaction with the Human-Machine Interaction/Human-Machine Interfacewhich isalsoa formofusabilitymeasurement (Ovaska,1991).Theuser satisfaction isconsideredasthemostprevalentmeasureofaninformationsystem’ssuccessthankstoitsapplicabilityandeaseofuse,anditisdirectlyrelatedtothesystem’ssuccess(Zviran&Erlich,2003).Althoughthesubjectiveevaluation of usability tends to be neglected in favour of objective performancemeasures, Lund (2001)stated that the subjective measurement is most closely related to user behaviour. Other authors alsosuggestedthatlesstangiblefactorsofusability(e.g.enjoyment)arebecomingofahigherimportanceandshouldbeincorporatedinthesystemusabilitytesting(Linetal.,1991).

Themostpopularandwidely-usedtoolsare:theComputerUserSatisfactionscale(CUS),theQuestionnaireforUserInteractionSatisfaction(QUIS),theSystemUsabilityScale(SUS),andtheUSEQuestionnaire,whicharecharacterisedbelow.

TheComputerUserSatisfaction(CUS)questionnaire(Bailey&Pearson,1983)isa39-factorscale,witheachfactormeasuredwithfiveratings.Amongthe39factors,thereare:confidenceinthesystems,securityofdata,outputformat,convenienceofaccess,personaljobeffectsresultingfromthecomputer-basedsupport,precisionofinformationoutputandsystemflexibility,andothers.

Thefirstfourresponseitemsareforqualityratingsandthefifthisanimportancerating.Authorsfoundthatmostimportantfactorswere:accuracy,reliability,timeliness,relevancyandconfidenceinthesystem.Thefactorsofleastimportancewere:feelingsofcontrol,volumeofoutput,vendorsupport,degreeoftraining,andorganisationalpositionofelectronicdataprocessing.

Eachitemhastobeansweredusinga7-pointscaleandthelengthofthequestionnairecouldresultinerrorsofattrition(Ives,OlsonandBaroudi(1983).

QuestionnaireforUserInterfaceSatisfaction(QUIS;Chinetal.,1988)containsfiveareasofusersatisfaction:

• Overallreactiontothesoftware• Screen-Readingcharactersonthescreen,organisationofinformationandsequenceofscreens• Terminologyandsysteminformation–useof termsthroughoutsystem,terminology,positionon

messagesonscreen,promptsforinput,errormessages• Learning–Learningtooperatethesystem,performingtasks• Systemcapabilities–systemspeed,systemreliability,noiselevel,designforalllevelsofusers

The answers are pair-wise/dichotomous and focusedmainly on interface’s consistency, helpfulness andclearness.

TheQUISquestionnaireconsistsoffourfactors:screen,terminologyandsysteminformation,learning,andsystemcapabilities.however,factoranalysisfailedtoconfirmedtheproper loadingofdefinedfactors.AsLewisnoticed(1995,p.61),“thelackofcorrespondencebetweenitemgroupsandunderlyingfactorsandthepaucityofinformationregardingsubscalereliabilityareproblemsthatlimittheusefulnessoftheQUIS”.

Computer System Usability Questionnaire (CSUQ; Lewis, 1995) contains 19 questions measuring suchcomputersystemcharacteristicsas:

• Easeofuse• Easeoflearning• Simplicity• Effectiveness• Information• Userinterface

7

Exampleitemsare:“Overall,Iamsatisfiedwithhoweasyistousethissystem”,“Ifeelcomfortableusingthissystem”,“Theinterfaceofthissystemispleasant”.Thescaleisreliable,Cronbach’salpha=0.95forthewholescaleandhighCronbach’salphacoefficient for thesubscales: systemusefulness, informationqualityandinterfacequality. thequestionnaireproved tobe reliableandusefulalso innon-laboratorysettingand isopentousebyresearchers.

System Usability Scale (SUS) is a ten-item scale developed by Brooke (1986). It is a validated tool formeasuring theusabilityof awide varietyofproducts and services, includinghardware, software,mobiledevices,websites,applications.Itwasdesignedtomeettheneedofashort,simpletoolthatcouldbeusedinindustrialsettings.SUShasbeenmadefreelyavailableforuseinusabilityassessment.Theexemplaryitemsare:“IthinkthatIwouldliketousethissystemfrequently”,“Ifoundthesystemunnecessarilycomplex”,“Ifeltveryconfidentusing thesystem”.However, there isno informationon theusability/usersatisfactionprinciplesthatwoulddeterminefactorsincludedinthetool.ResultsfromtheanalysisofBangor,KortumandMiller(2008)showthattheSUSisahighlyrobustandversatiletoolforusabilityprofessionals.

USE Questionnaire is a public-access tool developed by Lund (2001), dedicated to measure the mostimportant dimensions of usability for users, across domains. It can measure the usability of interface,software,hardware,services,usersupportmaterials.Thequestionnairecontains30items(17itemsintheshort-formversion)onfoursubscales,correspondingtomostimportantusabilitydimensions:

• Usefulness• EaseofUse• EaseofLearning• Satisfaction

Example itemsare:“ItmakesthethingsIwanttoaccomplisheasiertogetdone”,“ItrequiresthefeweststepspossibletoaccomplishwhatIwanttodowithit”,“Ilearnedtouseitquickly”,“Itiswonderful”.

Lesscommonquestionnairesare:ASQ,CUSI,UISandPUTQ.

TheAfter-ScenarioQuestionnaire(ASQ) isanothertooldevelopedasapartoftheIBMquestionnaireset(Lewis,1995).Itcontainsonlythreequestionscorrespondingtothefollowingsystemusabilitysatisfactionfactors:easeoftaskcompletion,timetocompleteatask,adequacyofsupportinformation.Theexemplaryitemis:“Overall,Iamsatisfiedwiththeamountoftimeittooktocompletethetaskinthisscenario”.Thelengthofthequestionnaire isanadvantagebut it isaquestionnairedesignedfortheusabilityexpertstoanswerusingpredefined scenarios and tasks.However, thequestionnairehasbeenalso validated in theoffice-applicationstudies.IthasbeenshownthatthreeASQitemsshouldbecondensedintoasinglescale.Thatiswhyitcannotbespecified,whichusabilitydimensionneedsanimprovement.

ComputerSatisfactionInventory(CUSI)isascaledevelopedbyKirakowskiandDillon(1988;in:Lewis,1995).Itisa22-itemquestionnairecontainingtwosubscales:affectandcompetence.competencefactoraddressesusers’feelingofmasteryoverthecomputersystem.affectfactoraddressesusers’feelingoffearorpleasure.theoverallinternalconsistencyis0.94,with0.91foraffectand0.89forcompetence.itisdesignedtobeusedinlaboratoryenvironmentwithtasksdevelopedbyresearchers.UnliketheQUIS,researcherswhowishtousetheCUSImustpurchaseitfromtheauthors,whichlimitsitsgeneralavailabilityandusefulness(Lewis,1995,p.61).Also,Ovaskastated(1991,p.56)thatalthoughthismeasureisvalidated,“usabilityismuchmorethanuserfriendlinessorusercompetenceandaffect”.

UserInformationSatisfaction(UIS)isashortformoftheCUSquestionnaire,developedbyIvesetal.(1983).Thisquestionnairecontains13 factorswith tworatingspereach factor (insteadof five)withmaintainingsatisfactorypsychometriccharacteristics.Unfortunately,authorshavenotpublishedthefullversionofthequestionnairebutitisavailableuponrequest.

8

PurdueUsabilityTestingQuestionnaire(PUTQ)isaquestionnairedevelopedbyLinetal.(1997).Themainadvantageofthistool is that itwasdedicatedfortheendusers.Thequestionnaire isbasedonfollowingusabilityprinciples:

• Compatibility• Consistency• Flexibility• Learnability• Minimalaction• Minimalmemoryload• Perceptuallimitation• Userguidance

Althoughtheseprincipleswerededicatedtoman-machinesystems’usabilityingeneral,thequestionnairedevelopedonthebasisoftheseprinciples(PUTQ)wasdesignedspecificallyforaconventionalgraphicaluserinterfacesoftwarewithavisualdisplay,keyboardandmouse,thatiswhyitcanbeusedinspecificusabilityevaluationenvironments.

Zviran and Erlich (2003) suggested that future studies on system usability should includemodernworldchallenges,e.g.securityofthesystem,aswellasotherdimensionsreceivinglessattention,e.g.organisationalsupport.Otherfactorsthatshouldbeconsideredintheindustrialenvironmentarehealth&safetyissuesorthe rapid changes in technology, allowing for adapting the interface to the users’ capabilities and skills.QuestionnairedevelopedintheTask6.1wasanattempttoincludethesefactors.

2.2.3ObjectiveusabilitymeasurementPhysiologicaldata

Themonitoringofphysiologicaldata,suchaseye-tracking(Poole&Ball,2006),stresslevel(Mullins&Treu,1991),humanmovementanalysis(Belda-Loisetal.,2010),HeartRateVariability-HRV,ElectrodermalActivity-EDA,Electroencephalograph–EEG,Electromyography–EMG,CriticalFlickerFrequency -CFF (Hercegfi,2011)isconsideredasausabilitytestingmethodintheHuman-ComputerInteractionenvironment,althoughthereisstilllackofresearchinthisarea.

Behaviouraldata

This group of objective measurement factors can include primary-task performance, secondary-taskperformance, interaction times, time to execute task, number of mistakes, usage time, number ofreports/queriesissuedoveraspecificperiod,numberoffileupdates(Zviran&Erlich,2003;Kampetal.,2001,Harvey, 2009). This sortofmeasurement is regarded tobemoreaccuratebut in somecases it couldbedifficulttoapplyortomeasure,dependingonthefield/environment.TherearefewreasonsconsideredinZviranandErlich(2003)review:suchmethodsareperceivedbybusinessownersasexpensivebecausetheyoftenrequirefinancialinvestment(e.g.installingdedicatedsoftware).Moreover,usersareawareofbeingmonitoredandtheymayaltertheirbehaviourforthemeasurementperiod.Finally,accordingtoZviranandErlich(2003)thesemethodsdonotalwaysreflectthesuccessorusabilityofthesystem.

The subjective andobjectivemethodsofmeasuring systemusability have got their ownadvantages anddisadvantages.However,asKissel(1995)hasdemonstrated,subjectiveratingsofusabilityoftendifferfromobjective performancemeasures. Recognising the value of both objective and subjectivemeasurement,Harvey(2009)proposedtocombinedifferentapproacheswhenmeasuringtheusability.

9

2.3SatisfactionwithworkingeneralThejobsatisfactionisoftendefinedas‘anemotionalreactiontothejob’(e.g.Spector,1997;in:Rafferty&Griffin,2009)oras‘anindividual’sevaluationofthejob,beliefsaboutthejob,andaffectiveexperiencesonthejob’(Weiss,2002;in:Rafferty&Griffin,2009).

Theworkersatisfactionisafactorthatconstituteshealthyandproductivecompanies.Poorjobsatisfactionis relatedwithmental-health problems and increases the risk of sickness absence anddisability pension(Andersenetal.,2017).

Themostcommonmethodofmeasuringworkersatisfactionisaquestionnairemeasurement,usuallybeingapartofworkingconditionssurveys.

TheEuropeanWorkingConditionSurvey (EWCS) includesasingle-itemscaleofsatisfactionwithworkingconditions,whichisconsideredasaprerequisiteforworkermotivation.Answersaregivenona5-pointLikertscale,rangingfrom‘notatallsatisfied’to‘verysatisfied’.Thesurveyhasshownthatfactorspositivelyrelatedto satisfactionwithworkingconditionsare:goodqualityofmanagement,goodwork-lifebalance,havingcareerprospects.Supervisoryroleisalsorelatedtohighersatisfactionwithworkingconditions,whichshouldremindaboutgivingspecialattentiontoworkersonthebottomoftheorganisationalladder.Factorsthatarelikely to leadto lowsatisfactionwithworkingconditionsare:adversesocialbehaviour, feelingthatone’shealthisatriskbecauseofwork,holdingatemporarycontractandhavingexperiencedrestructuringinthecompany.

Intermsofassociationwithjobqualityindices,satisfactionwithworkingconditionsismoststronglyrelatedto social environment, prospects, and skills and discretion. The last factor – skills and discretion – is afoundationofworker’sautonomy/control,whichisoneofmostcrucialaspectsofmaintainingemployees’wellbeing(e.g.Karasek,1979).

Anothersingle-itemscaleisalsowidelyusedinworkingconditionsurveys.Thisquestionis:‘Howsatisfiedareyouwithyourjobingeneral–allthingsconsidered?’(RaffertyandGriffin,2009)

TheCopenhagenPsychosocialQuestionnaire(COPSOQ;Peterjsenetal.,2011)measuresworksatisfactionwithfouritems,askingaboutsatisfactionwithworkprospects,physicalworkingconditions,thewayone’sabilitiesareused,andjobasawhole,everythingtakenintoconsideration.

However,accordingtoRafferty’sandGriffin’sreviewofstudiesmeasuringjobsatisfaction,themostpopularjobsatisfactionmeasureshavebeenMichiganOrganizationalAssessmentQuestionnaire(Cammannetal.,1983;in:Rafferty&Griffin,2009)andJobDescriptiveIndex(Smithetal.,1969;in:Rafferty&Griffin,2009).

TheMichiganOrganizationalAssessmentQuestionnaire(Cammanetal.,1983;in:Rafferty&Griffin,2009)isatoolmeasuringanoveralljobsatisfactionwiththreeitems,e.g.‘Allinall,Iamsatisfiedwithmyjob’.Thisscaledemonstratedasufficientreliabilityanditisapartofaninstrumentmeasuringworkingconditions.

TheJobDescriptiveIndex(JDI;Smithetal.,1969;in:Rafferty&Griffin,2009)measuresjobsatisfactionwith72 items, assessing five dimensions of job satisfaction: satisfaction with work itself, pay, promotionopportunities,supervision,andco-workers.

Theabove-mentionedexamplesofmethodsandtoolstomeasurejobsatisfactioningeneralvarybetween1-itemto72-itemscales.Alloftheseinstrumentsarewidelyusedandvalidated.Thechoiceoftheinstrumentshould be dictated by the research goals, but the length of the scale is an important factor when thequestionnairecontainsotherscalesandshouldberespondent-friendly.

Theworkersatisfactioncanbealsomeasuredbymeansofaphysiologicalparametersmonitoring.Theuseoftheobjectivemeasurementisbasedonscientificevidence,statingthatworkers’wellbeing,includingworksatisfaction,isrelatedtophysiologicalreactiononthelevelofcardiovascularsystem,endocrinesystemor

10

immunesystem(Kuykendall,2015)thatareinvolvedintheorganismstressresponse.Thestimulationofthenervoussystemisreflected,amongothers,inanincreasedhearrate,stimulationofsweatglands.Therefore,thechangesinheartrate,galvanicskinresponse,skintemperaturecanbeusedtoevaluatestressand,inturn,satisfactionlevel.

2.4ConclusionsformeasuringworkersatisfactionintheINCLUSIVEprojectIntheliteraturereview,wepresentedcurrentapproachestothemeasurementofusability/usersatisfactionandsatisfactionwithworkingeneral.Thisreviewhasshownthatusersatisfactionwiththesystem(e.g.HMI)isa formofusability.As themaingroupof interest in the INCLUSIVEprojectareend-users–employeesworkingwiththeadaptiveHMI–wethereforedecidedtoputanemphasisontheirassessmentoftheHMI,insteadofconductingexternalexpertusabilityevaluation.However,thequestionnaireneedstobetailoredto the realworkenvironmentsand thenew,adaptive interface, reflecting threeHMIModules:Measure,Adapt,andTeach(describedinDeliverableD1.1).

Moreover,traditionally,thesatisfactionwithworkiscommonlymeasuredasaffectivereactionstoone’sjob.However, in INCLUSIVEprojectworker satisfaction is understoodas a specific caseofusers’ satisfaction,whichreferstoworkingenvironments,andwork-relatedproductsorservices,includingHMIinterfacesandother ICT-based solutions implemented in industrial machinery control systems and automatedmanufacturing systems. It is a more broad understanding than the classic approach presented in theliteraturereview(althoughsomeofthereviewedtoolsconsideredthesatisfactionwithworkingconditions).Accordingly,we are proposing a user satisfactionwith the adaptiveHMI andworking conditionsmodel,understood as a worker satisfactionmodel. Consequently, we needed to develop a new questionnaire,measuring user satisfaction with HMI, including satisfaction with the three INCLUSIVE modules, andsatisfactionwithpsychosocialandphysicalworkingconditions,takingintoaccountindividualvariables,suchascapabilitiesandskills.

ItisalsoforeseentoanalysetheobjectivedataobtainedintheWP2andWP7testsassatisfactionindicators,inadditiontothesubjectivemeasurement.

11

3. The user satisfaction with the adaptive HMI and working conditionsmodel

WhendevelopingthemodelofHMIusersatisfactionwithwork,weassumedthatsomeotherfactorspresentat the workplace can influence its assessment or are particularly important in the context of inclusiveindustrialenvironment.Thesearephysicalfactors,suchasnoise,temperature,dustorposture,aswellaspsychosocialworkingconditions,suchasautonomy,participation,justiceorsocialsupport.Equally,asthesystemwillprocesssensitivepersonaldata,whichdisclosesbarriersofhumancapabilities,differentethicalandlegalrequirementstoprotecttheuseragainstharmanddisadvantageshavealsobetakenintoaccount.BasedontheELSImodelofethical,socialandlegalaspects(ELSI)whicharedescribedintheDeliverableD1.2–‘Summaryofallsafety,healthandethicsrecommendationsfortheworkingenvironments’wehavealsoincludedsomeethicalandsocialaspectsinthe‘SatisfactionwiththeadaptiveHMIandworkingconditions’model.

Itsmaincomponent,namelysatisfactionwhileworkingwiththeHMI,itselfaccountsforhealthandsafetyrequirements which were identified and specified in the WP1 and described in the abovementionedDeliverableD1.2.

Fig.2:ProposedmodelofworkersatisfactionintheINCLUSIVEprojectframework.

Basedon thismodelwe thendeveloped thequestionnaire thatwould cover allmentioned sections andaspects.

12

4. Thequestionnaire

Before starting to develop the questionnaire, we had reviewed some other existing questionnaires (seeParagraph2).WhenthefirstdraftofoursurveyhadbeenreadywesentittotheConsortiumPartners,sotheycouldcommenton itoradd somemissing issues.After receiving thesecomments,we included themajorityofthemintotheattachedversionofthequestionnaire(seeApendix1).

4.1.Questionnaire’spreambleThisquestionnaire startswithapreambleexplaining thequestionnaire’saim,which is theassessmentofsatisfaction with the implementation of the adaptive HMI, as well as its general structure. In thequestionnaire preamble its potential respondents are assured that survey is anonymous, and that theindividualdatawillnotbedisclosedtoanyoftheirsupervisors.

4.2.OccupationalanddemographicdataThefirstdatatobecollectedinthequestionnaireisthedatarelatedtothecompany,approximatenumberofemployeesandjobpositionstructure.

Astheinteractivesystemisbeingdesignedforageingworkersorworkerswithlanguagefluencydifficulties,thenextsetofitemsrelatestosuchdatafactorsas:age,levelofeducation,nationality, leveloflanguageofficiallyusedintheworkenvironment.

Wearealso interested inusers’workexperience,sothequestionsaboutthe lengthofprofessionalworkexperience,includinganytrainingtoperformcurrenttasks,arealsoposedinthissection.

Fromthepointofviewofsystemadaptability,someaspectsrelatedtousers’healthandcapabilitiesarealsoimportant.Hence,usersareaskedquestionsonpotentialproblemswithvision,hearing,handmovements,precisemovements,memorisinginformation,concentrationorunderstandingandfollowinginstructions.

Attheendofthissection,theuserisaskedtoassesshis/hergeneralhealth,andhowoftenhe/sheisbeingstressedduringworktime.Inordertogetusersfamiliarwithwhatitmeans,thenotionofstresshasbeenexplained.

4.3WorkingConditions

4.3.1PhysicalWorkingConditionsInthissection,weaskusershowfrequentlytheyareexposedtodifficultworkingenvironmentconditionssuchas:excessivenoise,extremetemperatures,dust,toobright/toodarklight.Wealsoaskwhetheruser’sworkrequiretomaintainawkwardbodyposition,lifting,bendingorhandsup.

Theanswersformatisa4-pointLikertscalewhere4=mostofthetime,and1=never.

4.3.2PsychosocialWorkingConditionsInordertocoverallpossiblepsychosocialconditionspresentattheworkplace,weincludedastandardisedandwellvalidatedscale-TheCopenhagenPsychosocialQuestionnaire(COPSOQ)developedbyKristensenetal.(2005)initsrevisedversion(Pejtersenetal.,2010).Conceptually,itincludesthemaindimensionsofthe most influential psychosocial theories at work, including the Job-Strain, Demand-Control-Support(Karasek&Theorell,1990)andEffort-Reward-Imbalance(Siegriest,1996)models,butalsoothertheoriesandaspectsignoredinprevioustools,forinstanceemotionaldemandsorroleclarity.ThismakesCOPSOQusefulinanyworkplaceeitherintheindustrialorintheservicesbranch.AmongpsychosocialriskassessmenttoolsCOPSOQisunique,becauseitincludespopulation-basedreferencevaluestoassesstheneedforactionand to help the decisionmaking process on preventivemeasures at the workplace level. A strength of

13

COPSOQisthatithasbeentestedinmanycountriesallovertheworld.Thesevalidationstudiesshowthequestionnaire‘scapacityandusabilityinthelocalcontext.

Tryingtokeepthequestionnaireasshortaspossible,wedidnotincludewholescalesfromCOPSOQ,onlychosenquestions,mostappropriatetotheworkingconditionspresentintheinclusiveHMIenvironment.

Weusedquestionsassessingpsychologicalworkdemands,suchas:‘Doyouhaveenoughtimeforyourworktasks?’or‘Doyouhavetokeepyoureyesonalotofthingswhileyouwork?’.Theaspectoflearningnewthingswasalsoincludedusingsuchquestionsas:‘Doesyourworkdemandthatyouaregoodatcomingupwithnewideas?’and‘Doyouhavethepossibilityoflearningnewthingsthroughyourwork?’.

Theworkers’influenceonhowhe/sheperformshis/hertasksisassessedusingthequestionslike:‘Canyouinfluencetheamountofworkassignedtoyou?’and‘Doyouhavealargedegreeofinfluenceconcerningyourwork?’.Socialsupportisanotherimportantaspectofpsychosocialworkingconditions,inthequestionnaireitisassessedusingquestionslike:‘Howoftendoyougethelpandsupportfromyourcolleaguesifyouneedit?’,‘Howoftencanyougethelpandsupportfromyournearestsuperiorifyouneedit?’.

Themeaningofworkwillbeevaluatedusingthequestion:‘Doyoufeelthattheworkyoudoisimportant?’,andmanagementstyleintheorganisationwiththequestions,suchas‘Doyoufeelmotivatedandinvolvedinyourwork?’or‘Isyourworkrecognisedandappreciatedbythemanagement?’.

An importantfactorofworker’ssatisfaction isalsoorganisational justice,whichwillbeassessedwiththequestions, such as: ‘Are you treated fairly at yourworkplace?’ or ‘Is thework distributed fairly?’.Workinsecurity,specificallyrelatedtothemoderntechnologiescouldalsobeasignificantpredictorofworker’sfearandworkdissatisfaction,inthequestionnaireitwouldbemeasuredwiththequestions:‘Areyouworriedaboutnewtechnologymakingyouredundant?’and‘Areyouworriedaboutbeingtransferredtoanotherjobagainstyourwill?’.

Apotentialdiscriminationcouldalsosignificantlyinfluencetheworker’ssatisfaction,wearegoingtocheckitusing the followingquestions: ‘Is therespace foremployeesofadifferent raceandreligion?’, ‘Is therespaceforbothmenandwomen?’,‘Istherespaceforelderlyemployees?’or‘Istherespaceforemployeeswithvariousillnessesordisabilities?’.

4.4SatisfactionwiththeadaptiveHMI

4.4.1SafetyAsmentionedaboveintheParagraph3andbasedonDeliverableD1.2–‘Summaryofallsafety,healthandethicsrecommendationsfortheworkingenvironments’,wearegoingtoaskwhetherthesafetyfunctions(Emergencystop,Guardlockingfunctions,Indicationsandalarms)andthecontrolbuttons(ManualReset,Mode selection/muting, Hold-To-Run, Enabling Device Function, Two-hand control function, Locking –unlockingofthepanel)are:clearlyidentifiable,clearlyvisible,readilyaccessible.Similarly,errormessagesand warningmessages are very important from the point of view of safety, so the question ‘Are errormessagesandwarningmessagesclear,informative/sufficientlydetailed,unambiguous?’isalsoincludedinthissection.

Theanswerformatisa5-pointLikertscale,where0=Never/HardlyEver,and4=Always.

4.4.2Satisfactionwiththedesign/visibilityoftheinterfaceIn this section, questions on characters, visibility and clarity of texts/messages, signs/symbols on theinterface,buttonswereincluded.Wealsoaskusersaboutthesequenceofscreens,positionofmessagesonthescreen,thecoloursusedintheHMI,theHMIlayout.Exampleitemsonthisscaleare: ‘Ingeneral,theorganisationofinformationisclear’,‘Thesoundsdistractand/orannoyme’.


14

4.4.3SatisfactionwitheaseInthissection,auserisaskedabouttheeaseofthesystem–oneoftheusabilityprinciplesdefinedintheParagraph2.1. Itmeans that thesystem iseasy to learnandbecomefamiliarisedwithaswellas timeoflearningisminimal.Theitemsrelatedtotheprocessofbecomingfamiliarwiththesystem’sfunctionsandoperations,memorisingthesystem’sfunctionsandoperations,findingtheinformationneeded,performingtasks, etc. Example items are: ‘I can easily withdraw an accidental command/action’, ‘Use of termsthroughoutthesystemisconsistentandunderstandable’.


4.4.4SatisfactionwithefficiencySatisfactionwithefficiencyisanothersectiondirectlyrelatedtousabilityprinciples.Inthissectiontheuseris asked about the system efficiency. The items cover the amount of information presented, number ofoperations to perform the task, having sense of control over the system, cooperation with themachine/robot.Exampleitemsare‘Ifeelfullyincontrolofthemachine’and‘Ingeneral,theHMIhelpsmetobemoreproductiveinmywork’.


4.4.5SatisfactionwiththeAdaptmoduleThis section covers satisfaction with one of the three core INCLUSIVE modules: Adapt Module (seeDeliverable D1.1). Users are here asked about their reactions to the newly developed system and itsadaptation to their capabilities and skills. The items verify whether the HMI has been adapted tocapabilities/mental statesanduserguidance.Example itemsare ‘I canget startedeasilyon the system’snewlyaddedfunctions’,‘IfeelImakelessmistakes/errorsusingtheadaptiveHMI’,‘IfeelIcancompletethetasksevenifIamtired’.


4.4.6SatisfactionwiththeMeasuremoduleAccordingly, this section covers satisfaction with the Measure Module (see Deliverable D1.1) – usersreactionstomonitoringtheirphysiologicalparameterswithaneye-tracker,wristbandorspeechdetector.Itemsareprecededbyashortexplanationoftheaimofthemeasurementmodule,i.e.enablingthesystemtodetecthigherstresslevelsandtoreact(adapt).Exampleitemsare:‘Ifeelthatmonitoringmystraincanbeadvantageousforme’,‘Ifeelitcanchallengemyphysicalcomfort’.

Theanswerformatwasa4-pointLikertscale,where0–Never/HardlyEver,1–Seldom,2–Sometimes,3–Often,4–Always.

4.4.7SatisfactionwiththeTeachmoduleThissectioncoversuserreactionstotheTeachModule(seeDeliverableD1.1).Asthismoduleincludesbothan on-line and off-line teaching system, items have also been developed to measure satisfaction withdifferent teaching techniques. Example items are ‘The chosen way of assistance (AR-based assistance,Speech-basedassistance,Supportassistance)intheon-lineteachingwasappropriate’,‘Theoff-lineteachingwashelpfultomastertheHMI’.


4.4.8OverallSatisfactionwiththeadaptiveHMIThefinalsectioncoverstheoverallsatisfactionwiththeadaptiveHMIanditismeasuredwiththequestion:‘Regarding the adaptive HMI in general. How pleased are youwith it as awhole, everything taken intoconsideration?’.

15

Theanswerformatisa5-pointLikertscale,where1=Verysatisfied,and5=Veryunsatisfied.

Inorder toensure theengagementofworkers in thedesignandadaptationof theirworkplaces to theirneeds, which is the prerequisite to ensure the attractiveness of these novel solutions for workers, thequestionnaireswillincludecollectedusers’opinionsand/orproposalsonpreferredmodificationsoftheHMIdesignanditsoperatingmode.Thereforeweasktwoopenquestions:‘Ifyouaresatisfied,whatdoyoulikethemostintheadaptiveHMI?’and‘WhatshouldbeimprovedintheadaptiveHMI?’.

16

5. ObjectivemeasurementofworkersatisfactionAsmentionedearlier, the aimof theWP6 is tomeasureworker satisfaction, both subjectively (with thequestionnairedeveloped)andobjectively.Inordertoconductanobjectiveassessment,thedataobtainedfromtheobjectiveindicatorsformeasuringworkers’capabilityinthedemonstrators(WP7)willbeanalysed.The results of real-time measurements of indicators selected in WP2 (e.g. eye-tracking, galvanic skinresponse)will be correlatedwith the results of questionnaire-based subjectivemeasurementsofworkersatisfaction.

17

6. ConclusionTheaimofthisDeliverablewastoproposemethodsandtoolstomeasureworkersatisfactioninframeworkoftheINCLUSIVEproject.Theliteraturereviewconductedaspartofthisworkpackagetaskhasdescribedthecommonlyusedmethodsforassessingsystem’susability,usersatisfactionandsatisfactionwithwork.BasedonthisreviewaswellasonpreviousDeliverables(D1.2),wehaveselectedfactorstobeconsideredintheworkersatisfactionmodelandthequestionnairetomeasureworkersatisfactionwiththeadaptiveHMIandworkingconditions.Thequestionnairewillbetranslated,pilottestedandadjustedintheTask6.2.

18

7.ReferencesAndersen,L.L.,Fishwick,D.,Robinson,E.,Wiezer,N.M.,Mockałło,Z.,Grosjean,V.(2017).Jobsatisfactionis

morethanafruitbasket,healthchecksandfreeexercise:Cross-sectionalstudyamong10,000wageearners.ScandJPublicHealth,45(5),476-484.

Bailey, J.E.& Pearson, S.W. (1983). Development of a Tool forMeasuring andAnalyzing ComputerUserSatisfaction.ManagementScience,29,530-545.

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale.InternationalJournalofHuman-ComputerInteraction,24(6),574-594.

Belda-Lois,J.-M.,deRosario,H.,Pons,R.,Poveda,R.,Morón,A.,Porcar,R.,...Gómez,A.(2010).Canhumanmovementanalysiscontributetousabilityunderstanding?HumanMovementScience,29(4),529-541.

Bevan,N.,Kirakowski,J.andMaissel,J.(1991).Whatisusability?In:H.J.Bullinger(Ed.):Proceedingsofthe4thInternationalConferenceonHumanComputerInteraction,Stuttgart,September1991.Elsevier.

Brooke,J.(1996).SUS:Aquickanddirtyusabilityscale.UsabilityEvaluationinIndustry,189(194),4–10.

Chin,J.P.,Diehl,V.A.,Norman,K.L.(1988).DevelopmentofanInstrumentMeasuringUserSatisfactionoftheHuman-ComputerInterface.ACMCHI'88Proceedings,213-218.

Cockton,G.,Woolrych,A.,&Lavery,D.(2008).Inspection-basedevaluations.InJ.A.Jacko&A.Sears(Eds.),The human–computer interaction handbook: Fundamentals, evolving technologies and emergingapplications(2nded.),pp.1171–1190.Hillsdale,NJ:Erlbaum

Eurofound(2016).SixthEuropeanWorkingConditionsSurvey–Overviewreport.PublicationsOfficeoftheEuropeanUnion,Luxembourg.

Harvey,C.(2009).DevelopmentofaMethodforEvaluatingtheUsabilityofIn-VehicleInformationSystems(IVISs).Human-ComputerInteraction-INTERACT2009,5727,856-859.

Hercegfi,K.(2011).HeartRateVariabilityMonitoringduringHuman-ComputerInteraction.ActaPolytechnicaHungarica,8,205-224.

Holzinger,A.(2005).Usabilityengineeringmethodsforsoftwaredevelopers.CommunicationsoftheACH,48,71-74.

Ives, B., Olson, M.H. & Baroudi, J.J. (1983). The Measurement of User Information Satisfaction.CommunicationoftheACM,26,785-793.

Kamp,J.F.,Marin-Lamellet,C.,Forzy,J.F.,Causeur,D.(2001).HMIaspectsoftheusabilityofinternetserviceswithanin-carterminalonadrivingsimulator.IATSSRESEARCH,25,29-39.

Karasek,R. (1979).JobDemands,JobDecisionLatitude,andMentalStrain: ImplicationsforJobRedesign.AdministrativeScienceQuarterly,24,285-308.

Karasek,R.A.,&Theorell,T.(1990).Healthywork:stress,productivityandthereconstructionofworkinglife.NewYork:BasicBooks.

Kissel, G.V. (1995). The effect of computer experience on subjective and objective software usabilitymeasures.CHI’95ConferenceCompaniononHumanFactorsinComputingSystemsProceeding,284-285.

19

Kristensen,T.S.,Hannerz,H.,Høgh,A.,andBorg,V.(2005).CopenhagenPsychosocialQuestionnaire–atoolfortheassessmentandimprovementofthepsychosocialworkenvironment.ScandJWorkEnvironHealth,31(6),438-449.

Lewis, J.R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation andInstructionsforUse.InternationalJournalofHumanComputerInteractions,7,57-78.

Lin,H.X.,Choong,Y.-Y.&Salvendy,G.(1997).Aproposedindexofusability:Amethodforcomparingtherelativeusabilityofdifferentsoftwaresystems.Behaviour&InformationTechnology,16,267-277.

Lucas,W.andBabaian,T.(2015).TheCollaborativeCritique:AnInspectionMethodforExpertEvaluationofUserInterfaces.InternationalJournalofHuman–ComputerInteraction,31,843-859.

Lund,A.M.(2001).MeasuringusabilitywiththeUSEquestionnaire.UsabilityInterface,8(2),3-6.

Mullins, P.M. & Treu, S. (1991). Measurement of stress to gauge user satisfaction with features of thecomputerinterface.Behaviour&InformationTechnology,10(4),325-343.

Nielsen,J.(1993).Usabilityengineering.Boston:AcademicPress.

Nielsen,J.(1994).Heuristicevaluation.InJ.Nielsen&R.L.Mack(Eds.),Usabilityinspectionmethodspp.25–62.NewYork,NY:Wiley&Sons.

Norman,D.A.(2002).ThedesignofeverydaythingsNewYork:BasicBooks.

Ovaska, S. (1991). Usability as a Goal for the Design of Computer Systems. Scandinavian Journal ofInformationSystems,3,47-62.

Pejtersen, J.H., Kristensen, T.S., Borg,V. andBjorner, J.B. (2010). The secondversionof theCopenhagenPsychosocialQuestionnaire.ScandinavianJournalofPublicHealth,38,8–24.

Poole,A.&Ball,L.J.(2006).EyeTrackinginHCIandUsabilityResearch.EncyclopediaofHumanComputerInteraction,211-219.

Preece,J.(1993).AGuidetoUsability:HumanFactorsinComputing,Addison-Wesley,Wokingham

Rafferty,A.&Griffin,M. (2009). Job satisfaction inorganizational research. In:D.Buchanan&A.Brynan(Eds.),TheSageHandbookofOrganizationalResearchMethods.196-212.

Ravden, S. & Johnson, G. (1989). Evaluating Usability of Human-Computer Interfaces, Chichester: Ellis-Horwood

Rogers,Y.,Sharp,H.&Preece,J.(2011).Interactiondesign:Beyondhuman-computerinteraction,3rded.USA:JohnWiley&Sons.

Sears,A. (1997).Heuristicwalkthroughs:Findingtheproblemswithoutthenoise. International JournalofHuman–ComputerInteraction,9,213–234.

Shackel,B.(1991).Usability-context,framework,designandevaluation.InB.Shackel&S.Richardson(Eds.),Humanfactorsforinformaticsusability,pp.21-38.Cambridge:CambridgeUniversityPress

Siegrist, J. (1996). Adverse health effects of high-effort/low-reward conditions. Journal of OccupationalHealthPsychology,1(1),27-41.

Wang,Ch.-M.&Huang,Ch.-H.(2015).AStudyofUsabilityPrinciplesandInterfaceDesignforMobilee-Books.Ergonomics,58,1-19.

20

Yeh,C.-J.(2010).Theprinciplesofinteractiondesigninthepost-digitalage.Taipei:ARTIST-MAGAZINE

Zviran,M.&Erlich,Z.(2003).MeasuringISUserSatisfaction:ReviewandImplications.CommunicationsoftheAssociationforInformationSystems,12,81-103.

21

Appendix1– ‘SatisfactionwiththeadaptiveHMIandworkingconditions’questionnaire

22

SatisfactionwiththeadaptiveHMIandworkingconditions

DearParticipant!

ThesurveyisconductedwithintheInclusiveProjectthatyourcompanyisinvolvedin.

Thisquestionnaireisaimedatassessmentofsatisfactionwiththeimplementationoftheadaptivehuman

machineinterface(HMI).Itconsistsoftwomainparts:satisfactionwithworkingconditionsandsatisfaction

withthenewadaptiveinterface.Thesurveyisanonymous.Theindividualdatawillnotbeavailabletoany

ofyoursupervisors.Wearereallyinterestedinwhatyouthinkabouttheadaptiveinterface.Weencourage

youtogiveyourhonestopinion!

DemographicDATA

1. Age__________________

2. Gender:qMan qWoman

3. Levelofeducation:

qPrimaryschool qSecondaryschool qUniversity(Bachelor)qUniversity(Master)

4. Forhowlongdoyouworkinthecompany?___________

5. Howlonghaveyoubeeninvolvedinthepresentworktasks?___________

6. Didyoureceiveanytrainingtoperformyourcurrenttask?qYESqNO

7. Nationality?………………………………………

8. Isyournationalitythesameasthemajorityofyourcolleagues?qYESqNO

9. Yourleveloflanguageofficiallyusedinyourworkenvironmentis:

qBasicqCommunicativeqAdvancedqNativespeaker

Company:

Code:

Approximatenumberofemployees:

JobPosition:

23

HEALTHANDCAPABILITIES

ERSONA

1. Ihaveproblemswithvision:qYES(checkbelow)qNO

IfYES:qNearqFarqIamcolorblindqOther(pleasedescribe)………………………………….

2. Ifananswertothequestion1isyes,myvisionwithcorrectionglasses/lensesis:

qExcellentqGoodqModerateqPoor

3. Ihaveproblemwithhearing:qYESqNO

4. Ifananswertothequestion3isyes,myhearingwithhearingaidis:

qExcellentqGoodqModerateqPoor

5. Ihaveproblemswithmovingmyhands:

qNotatallqToasmallextentqSomewhatqToalargeextent

6. Ihaveproblemswithprecisemovements(e.g.manipulatingsmallobjects):


7. Ihavedifficultiestorememberthings:


8. IngeneralIhaveproblemswithconcentration:


9. Ihaveproblemswithunderstandinginstructionsandfollowinginstructions:


10. Ingeneral,Iwouldsaymyhealthis:

qExcellentqVerygoodqGoodqFairqPoor

11. Howoftenhaveyoubeenstressed*duringthelast4weeks:

qEverydayqMostoftheweekqOnceaweekqNever/HardlyEver

*Stressmeansthesituationwhenapersonfeelstense,restless,nervous,oranxious,orisunabletosleepatnightbecausehismindistroubledallthetime.

I.WorkingConditionsPHYSICALWORKINGCONDITIONS

1. Duringworkareyoufrequentlyexposedtodifficultconditionssuchas:

a) ExcessiveNoise qMostofthetimeqSometimesqRarelyqNever

b) ExtremeTemperatures qMostofthetimeqSometimesqRarelyqNever

c) Dust qMostofthetimeqSometimesqRarelyqNever

d) Toobright/toodarklight qMostofthetimeqSometimesqRarelyqNever

24

2. Doesyourworkrequiretomaintain:

a) Awkwardbodyposition qMostofthetimeqSometimesqRarelyqNever

b) Lifting qMostofthetimeqSometimesqRarelyqNever

c) Bending qMostofthetimeqSometimesqRarelyqNever

d) Handsup qMostofthetimeqSometimesqRarelyqNever

e) Other

(Short description)………………………………………………..............………………………………………………………………………

………………………………………………………………………………………………………………………………………

PSYCHOSOCIALWORKINGCONDITIONS

0–Never/HardlyEver1–Seldom2-Sometimes3–Often4–Always

Never/H

ardlyEver

Seldom

Sometim

es

Often

Always

1. Doyouhaveenoughtimeforyourworktasks? 0 1 2 3 4

2. Doyouhavetokeepyoureyesonalotsofthingswhileyouwork? 0 1 2 3 4

3. Doesyourworkrequirethatyourememberalotofthings? 0 1 2 3 4

4. Doesyourworkdemandthatyouaregoodatcomingupwithnew

ideas?0 1 2 3 4

5. Doesyourworkrequireyoutomakedifficultdecisions? 0 1 2 3 4

6. Canyouinfluencetheamountofworkassignedtoyou? 0 1 2 3 4

7. Doyouhavealargedegreeofinfluenceconcerningyourwork? 0 1 2 3 4

8. Howoftendoyougethelpandsupportfromyourcolleaguesifyou

needit?0 1 2 3 4

9. How often can you get help and support from your nearest

superiorifyouneedit?0 1 2 3 4

10. Doyoufeelpartofacommunityatyourplaceofwork? 0 1 2 3 4

0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent4–Toaverylargeextent

Toaverysm

all

extent

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylarge

extent

25

11. Isitnecessarytokeepworkingatahighpace? 0 1 2 3 4

12. Doyouhavethepossibilityof learningnewthings throughyour

work?0 1 2 3 4

13. Doesyourworkrequireyoutotaketheinitiative? 0 1 2 3 4

14. Canyouuseyourskillsorexpertiseinyourwork? 0 1 2 3 4

15. Doesyourworkgiveyoutheopportunitytodevelopyourskills?

0 1 2 3 4

16. Doyoufeelthattheworkyoudoisimportant? 0 1 2 3 4

17. Doyoufeelmotivatedandinvolvedinyourwork? 0 1 2 3 4

18. Isyourworkrecognizedandappreciatedbythemanagement? 0 1 2 3 4

19. Areyoutreatedfairlyatyourworkplace? 0 1 2 3 4

20. Areyouworriedaboutnewtechnologymakingyouredundant? 0 1 2 3 4

21. Are youworriedaboutbeing transferred toanother jobagainst

yourwill?0 1 2 3 4

22. Areconflictsresolvedinafairway? 0 1 2 3 4

23. Areemployeesappreciatedwhentheyhavedoneagoodjob? 0 1 2 3 4

24. Istheworkdistributedfairly? 0 1 2 3 4

25. Istherespaceforemployeesofadifferentraceandreligion? 0 1 2 3 4

26. Istherespaceforbothmenandwomen? 0 1 2 3 4

27. Istherespaceforelderlyemployees? 0 1 2 3 4

28. Istherespaceforemployeeswithvariousillnessesordisabilities? 0 1 2 3 4

OverallSatisfactionwithworkingconditionsRegarding your work in general. How pleased are you with your job as a whole, everything taken intoconsideration?qVerysatisfiedqSatisfiedqNeithersatisfied,nordissatisfiedqUnsatisfiedqVeryunsatisfied

26

II.SatisfactionwiththeadaptiveHumanMachineInterface(HMI)SAFETY

0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent

4–Toaverylargeextent5-NotapplicableAccordingtoyouropinion:

Toaverysm

allexten

t

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylargeextent

Notapp

licab

le

1. Safety functions (Emergency stop, guard locking functions,

indicationsandalarms)are:

a) Clearlyidentifiable 0 1 2 3 4 5

b) Clearlyvisible 0 1 2 3 4 5

c) Readilyaccessible 0 1 2 3 4 5

2. The control buttons (Manual Reset, Mode selection/muting,Hold-To-Run, Enabling Device Function, Two-hand controlfunction,Locking–unlockingofthepanel)are:

d) clearlyidentifiable 0 1 2 3 4 5

e) Clearlyvisible 0 1 2 3 4 5

f) Readilyaccessible 0 1 2 3 4 5

3. Errormessagesandwarningmessagesare:

g) Clear 0 1 2 3 4 5

h) Informative/Sufficientlydetailed 0 1 2 3 4 5

i) Unambiguous 0 1 2 3 4 5

4. Overall,theadaptiveHMIfulfillsallthesafetyfunctions 0 1 2 3 4 5

27

SATISFACTIONWITHTHEDESIGN/VISIBILITYOFTHEINTERFACE

0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent4–Toaverylargeextent5-Notapplicable

Accordingtoyouropinion:

Toaverysm

all

extent

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylargeextent

Notapp

licab

le

1. Charactersareeasytoread. 0 1 2 3 4 52. Thetexts/messagesontheinterfacearevisible 0 1 2 3 4 53. Thesigns/symbolsontheinterfaceareunambiguous/clear 0 1 2 3 4 54. Theinterfacebuttons(options)arevisibleonthescreen 0 1 2 3 4 55. Thesizeofthebuttons(options)isoptimal(nottoolarge,nottoo

small)0 1 2 3 4 5

6. Sequenceofscreensisappropriate 0 1 2 3 4 57. Positionofmessagesonthescreenisconsistent 0 1 2 3 4 58. ThecoloursusedintheHMIhelptobetterperceivetheinformation

onthescreen0 1 2 3 4 5

9. ThecoloursusedintheHMIdonotdistractme 0 1 2 3 4 510. TheHMIlayoutisaesthetic 0 1 2 3 4 511. TheHMIlayoutisminimalistic 0 1 2 3 4 512. Ingeneral,theorganizationofinformationisclear 0 1 2 3 4 513. Ingeneral,thelayoutoftheadaptiveHMIisappropriate 0 1 2 3 4 514. ThesoundsoftheHMIhelpmetobetteroperatetheHMI. 0 1 2 3 4 515. Thesoundsdistractand/orannoyme. 0 1 2 3 4 5

16. Thechanginginterfacedistractsme 0 1 2 3 4 5

17. IcaneasilyoperatetheadaptiveHMIusingmyhands 0 1 2 3 4 5

28

SATISFACTIONWITHEASE

0–Toaverysmallextent1–Toasmallextent2-Somewhat3–Toalargeextent4–Toaverylargeextent

5-NotapplicableAccordingtoyouropinion:

Toaverysm

all

extent

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylargeextent

Notapp

licab

le

1. Thesystemiseasytolearn. 0 1 2 3 4 5

2. I quickly familiarized myself with the system's functions andoperations.

0 1 2 3 4 5

3. Itwaseasytomemorizethesystem'sfunctionsandoperations. 0 1 2 3 4 5

4. Useoftermsthroughoutsystemisconsistentandunderstandable. 0 1 2 3 4 5

5. Thesigns/symbolsontheinterfacehelpmetonavigatethroughtheHMI

0 1 2 3 4 5

6. IcaneasilyfindalltheinformationIneed. 0 1 2 3 4 5

7. Icaneasilywithdrawunaccidentalcommand/action. 0 1 2 3 4 5

8. Icaneasilyreturntotheearliersteps. 0 1 2 3 4 5

9. Performingtasksisstraightforward. 0 1 2 3 4 5

SATISFACTIONWITHEFFICCIENCY



Toaverysm

allexten

t

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylargeextent

Notapp

licab

le

1. TheHMIprovidesinsufficientamountofinformation. 0 1 2 3 4 5

2. TheHMIprovidesexcessiveamountofinformation. 0 1 2 3 4 5

3. Thenumberofoperationstoperformatask/toachieveagoal/tosetupaprocessisoptimal.

0 1 2 3 4 5

4. Ifeelfullyincontrolofthemachine. 0 1 2 3 4 5

5. The HMI helps me to more efficiently cooperate with themachine/robot.

0 1 2 3 4 5

6. Ingeneral,theHMIhelpsmetobemoreproductiveinmywork. 0 1 2 3 4 5

29

SATISFACTIONWITHADAPTMODULE



Toaverysm

allexten

t

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylargeextent

Notapp

licab

le

1. Icangetstartedeasilyonthesystem’snewlyaddedfunctions 0 1 2 3 4 5

2. IfeeltheadaptiveHMIadjuststomyactualcapabilities/mentalstates 0 1 2 3 4 5

3. IfeelIcancompletethetasksevenifIamtired 0 1 2 3 4 5

4. IfeelIcanbeeasilyguidedwhenIgetlost/commitanerror 0 1 2 3 4 5

5. IfeellessstressedusingtheadaptiveHMI 0 1 2 3 4 5

6. Itrustthatthesystemallowsmetoperformmytaskssmoothly 0 1 2 3 4 5

7. IfeelmoreconfidentusingtheadaptiveHMI 0 1 2 3 4 5

8. IfeelImakelessmistakes/errorsusingtheadaptiveHMI 0 1 2 3 4 5

9. IfeellessexhaustedaftermyshiftwhenusingtheadaptiveHMI 0 1 2 3 4 5

SATISFACTIONWITHMEASUREMODULE

Pleaseanswerthesubsequentquestionsconsideringthefollowingscenario:Theworkingmachines

areequippedwithsensorsthatareabletotrackstrainofaworkingpersonbyreal-timemeasurementof

his/herphysiologicalparameters,e.g.heartrate,bloodpressure,etc. Ifthemeasuredstrainindicatorsare

toohigh,thehuman-machine-interfaceadaptstothesituationresultinginalowerstresslevel.

Howdoyoufeelaboutmonitoringyourphysiologicalparameters(e.g.usingawristband,eyetracker,etc.)?


4–Toaverylargeextent5-Notapplicable

Toaverysm

all

extent

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylarge

extent

Notapp

licab

le

1. Ifeelitcanchallengemyphysicalcomfort 0 1 2 3 4 5

2. Itrustthesystemandthatmypersonaldatawillnotbeabused 0 1 2 3 4 5

3. I feel uncomfortable with not knowing what happens with mypersonaldata

0 1 2 3 4 5

4. Ifeelthatmonitoringmystraincanbebenfecialforme 0 1 2 3 4 5

30

SATISFACTIONWITHTEACHINGMODULE


4–Toaverylargeextent5-NotapplicableWhatdoyouthinkabouttheon-lineandoff-linetraining?

Toaverysm

allexten

t

Toasm

allexten

t

Somew

hat

Toalargeextent

Toaverylargeextent

Notapp

licab

le

1. Thechosenwayofassistance(AR-basedassistance,Speech-basedassistance,Supportassistance)intheon-linetrainingwasappropriate

0 1 2 3 4 5

2. The way the on-line training system presented theinformationwasadaptedtomycurrentworktask.

0 1 2 3 4 5

3. Theinformationintheon-linetrainingsystemwaseasytoreadandperceive.

0 1 2 3 4 5

4. Iwouldhaveneededmoredetailedinstructionstocompletemytasks.

0 1 2 3 4 5

5. Theon-linetrainingsystemwasadequateinrelationtomyskillsandcapabilities.

0 1 2 3 4 5

6. The interaction with the on-line training system wasintuitive.

0 1 2 3 4 5

7. Ienjoyedusingtheon-linetrainingsystem. 0 1 2 3 4 5

8. Theon-linetrainingsystemmakesworkeasier. 0 1 2 3 4 5

9. Theon-linetrainingsystemwashelpfultomastertheHMI. 0 1 2 3 4 5

10. Theoff-linetrainingsystemshouldhavebeenmorerealistictopreparemefortheinteractionswiththeHMI.

0 1 2 3 4 5

11. Iwould have neededmore detailed instructions from theoff-linetrainingsystemstolearnthetasksuccessfully.

0 1 2 3 4 5

12. Theoff-linetrainingsystemcouldreplaceteaching-inbyatrainerforthisprocedure.

0 1 2 3 4 5

13. The interaction with the off-line training system wasintuitive.

0 1 2 3 4 5

14. Ienjoyedusingtheoff-linetrainingsystem. 0 1 2 3 4 5

15. Theoff-linetrainingsystemwasadequateinrelationtomyskillsandcapabilities.

0 1 2 3 4 5

16. Theoff-linetrainingsystemwastoocomplex. 0 1 2 3 4 5

17. Theoff-linetrainingsystemwashelpfultomastertheHMI. 0 1 2 3 4 5

18. Thecontentoftheoff-linetrainingwasappropriate 0 1 2 3 4 5

31

OverallSatisfactionwiththeadaptiveHMIRegarding the adaptiveHMI in general. How pleased are youwith it as awhole, everything taken intoconsideration?qVerysatisfiedqSatisfiedqNeithersatisfied,norunsatisfiedqUnsatisfiedqVeryunsatisfiedIfyouaresatisfied,whatdoyoulikethemostintheadaptiveHMI?

________________________________________________________________________________

________________________________________________________________________________

WhatshouldbeimprovedintheadaptiveHMI?

________________________________________________________________________________

________________________________________________________________________________

Thankyouforyouranswers!!!

smart and adaptive interfaces for inclusive work environment · 2.2.1 expert usability evaluation...

Documents