impada_o3_a1_ research methodology successful practices for disadvantaged learners

49
IMPADA Research methodology to guide IMPADA partners on collecting successful adult education practices for the disadvantaged learner Output type: Intellectual Output PROMEA November 2016

Upload: andrea-ciantar

Post on 19-Mar-2017

57 views

Category:

Education


0 download

TRANSCRIPT

Page 1: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

IMPADAResearch methodology to guide IMPADA partners on

collecting successful adult education practices for

the disadvantaged learner

Output type: Intellectual Output

PROMEANovember 2016

Page 2: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Project acronym: IMPADAProject name: Improving the effectiveness of adult education for disadvantaged

groupsProject code: 2015-1-UK01-KA204-013666

Document InformationDocument ID name: IMPADA_03A1_good practices collection guide_2016-12-15Document title: Research methodology to guide IMPADA partners on collecting suc-

cessful adult education practices for the disadvantaged learner Output Type: Intellectual Output Date of Delivery: 30/11/2016Activity type: DocumentActivity leader: DACESDissemination level: Public

Document History

Versions Date Changes Type of change Delivered by

Version 1.0 15/12/2016Initial docu-ment

- PROMEA

DisclaimerThe European Commission support for the production of this publication does not constitute an endorsement of the contents which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

The project resources contained herein are publicly available under the Creative Commons license Attribution-NonCommercial-ShareAlike 4.0 International.

Table of Contents

2

Page 3: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

1. Introduction............................................................................................................................................................4

2. Basic definitions and questions to explore...............................................................................................5

3. Purposes and activities......................................................................................................................................6

3.1. Why collect the practices?......................................................................................................................6

3.2. Sharing successful adult education practices for the disadvantaged..................................7

4. Stakeholders...........................................................................................................................................................7

5. Sampling methodology......................................................................................................................................8

6. Identifying Good Practices............................................................................................................................10

6.1. Defining evaluation fields....................................................................................................................11

6.2. Defining evaluation criteria................................................................................................................11

6.3. Ranking and Classification...................................................................................................................12

7. Collection process, means and tools.........................................................................................................16

7.1. Processes.....................................................................................................................................................16

7.2. Means............................................................................................................................................................ 17

7.2.1. Questionnaire template...............................................................................................................18

7.2.2. Identity of the organisation where the successful adult education practice on the disadvantaged is adopted......................................................................................................................19

7.2.3. Description of the successful adult education practice on the disadvantaged learner 20

7.2.4. Impact................................................................................................................................................. 22

7.2.5. Evaluations.......................................................................................................................................23

7.2.6. Challenges.........................................................................................................................................24

7.2.7. Scale.....................................................................................................................................................24

7.2.8. Transferability.................................................................................................................................25

7.2.9. Other details of the good practice...........................................................................................26

7.3. Supporting tools.......................................................................................................................................26

7.3.1. Invitation email...............................................................................................................................26

7.3.2. Follow up email..............................................................................................................................27

7.3.3. Thank you email.............................................................................................................................28

8. Quality assurance..............................................................................................................................................28

9. Annexes..................................................................................................................................................................29

9.1. Invitation email template.....................................................................................................................29

9.2. Follow up email template.....................................................................................................................30

9.3. Thank you email template...................................................................................................................31

9.4. Link to the questionnaire in English...............................................................................................31

9.5. Template form for the registration of the invited providers of successful adult edu-cation practices for the disadvantaged learner.........................................................................................31

3

Page 4: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

1. Introduction This document describes the methodology used to identify and evaluate successful adult educa-tion practices for the disadvantaged learner that will be collected within the context of the IM -PADA project and are going to be used for the development of O3A2 “Recommendations for adult learning providers to improve effectiveness towards the disadvantaged learner”.

The purpose of the research methodology is: To guide the practices collection activities of Partners, in their efforts to gather good prac -

tices on successful measures adopted by European adult learning providers, to improve their effectiveness on the disadvantaged learner.

To specify the criteria for identifying the best among the collected ones.

Collection of practices will follow a survey format to be promoted by all IMPADA partners to relev-ant stakeholders of their network, such as managing and training staff of adult learning providers, associations of providers, field experts. The methodology will contribute towards:

Revealing the objectives for the reporting and the benefits of sharing the practices. Illustrating successful practices definitions. Pointing out the parameters, metrics and issues on successful adult education practices

for the disadvantaged learner in the European Adult Education Providers. Providing IMPADA Partners the necessary guidelines and practical material for the report-

ing, monitoring and coordination of the collection of the successful adult education prac-tices for the disadvantaged learner.

Identifying the actions that the IMPADA partners should undertake for the collection of successful adult education practices for the disadvantaged learner.

Under this premise, successful adult education practices on the disadvantaged learner should be reviewed within a specific evaluation scheme in terms of their effectiveness, applicability and transferability. The proposed methodology also:

a) defines certain evaluation fields and criteria (i.e. impact, previous evaluations, challenges en-countered in implementation, extent of integration throughout the organisation, and transferabil -ity), and

b) sets score thresholds and weights in order to ensure a minimum quality level and results, graded according to the weighted significance of each evaluation field. (N.B. This evaluation / ranking process should be used in order to filter case studies that are going to be included in O3A2 “Recommendations for adult learning providers to improve effectiveness towards disad-vantaged learner”.

2. Basic definitions and questions to exploreA practice is defined as a procedure or a technique that has been tried and shown to work (whether fully or in part but with at least some evidence of effectiveness), and that may have im -

4

Page 5: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

plications for practice at any level elsewhere. In these contexts, three possible levels of practice stem:

Promising practices have been used for a certain period of time by an organization in or-der to better perform its tasks and to increase its productivity. The promising practices have not proven their transferability to different geographic or thematic areas, nor to be successful according to any result indicators.

Demonstrated practices have proven to be successful for the organization where they have been applied. The success of the practices is confirmed through some result indicat-ors, but it is not demonstrated how and to which area these practices can be transferred.

Replicated practices have the potential under certain circumstances and conditions to be transferred and successfully adopted to a different geographic or thematic area.

The following set of criteria should indicate whether a practice is a “good practice”:

Effective and successful: A “good practice” has proven its strategic relevance as the most effective way in achieving a specific objective; it has been successfully adopted and has had a positive impact on individuals and/or communities.

Economically and socially sustainable: A “good practice” meets current needs, in particu-lar the essential needs of the disadvantaged, without compromising the ability to address future needs.

Gender sensitive: A description of the practice must show how actors (men and women), involved in the process, were able to improve their livelihoods.

Technically feasible: Technical feasibility is the basis of a “good practice”. It is easy to learn and to implement.

Inherently participatory: Participatory approaches are essential as they support a joint sense of ownership of decisions and actions.

Replicable and adaptable: A “good practice” should have the potential for replication and should therefore be adaptable to similar objectives in varying situations.

Reducing disaster/crisis risks, if applicable: A “good practice” contributes to disaster/crisis risks reduction for resilience.

Within the IMPADA project, a good practice refers to a successful adult education practice for the disadvantaged learner that has been tried by a European Adult Education Provider and has proved to solve – most effectively – a problem in the organization.

Considering the aforementioned definitions several questions are arising, such as:

Why should the practices be collected? Which are the benefits from sharing the practices? Who are the stakeholders of successful adult education practices for disadvantaged

learner? Which is the population to be investigated? How is the sample defined and how it will be

reached?

5

Page 6: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

How the sample will be motivated for the reporting of successful adult education prac-tices for the disadvantaged learner, and which are the benefits from this motivation?

Which are the factors and the elements that they should be explored and revealed from the reporting of practices?

How the requested data will be collected? How will they be analysed? What actions and tasks should each IMPADA partner undertake in order to achieve the

desired results?

This document aims to guide the practices’ collection activities of partners, and to specify the cri -teria for identifying the best among those collected, by providing the answers to the above ques -tions and thus, ensuring the collection of the successful adult education practices for the disad-vantaged learner with response to the needs and requirements of the IMPADA project.

3. Purposes and activities

3.1. Why collect the practices?

The aims of the collection of successful adult education practices for the disadvantaged learner by adult education providers in the Partnership countries are:

• To identify the success factors and the criteria under which successful adult education practices for the disadvantaged learner are assessed as good practices.

• Acquiring knowledge on specific relevant issues and assuring homogeneity and compar-ability of the gathered practices for the identification of the key success factors and the transferability potential of the collected evidence.

• To provide input for the development of IMPADA O3A2 “Recommendations for improve-ments on effectiveness of adult education for disadvantaged learner”.

• To contribute to the catalogue of successful adult education practices for the disadvant-aged learner.

• To disseminate established successful adult education practices for the disadvantaged learner.

In this context adult learning providers in need of greater improvements will gain insight on prac-tices of others with greater effectiveness on the disadvantaged learner, while highly effective adult learning providers will benefit from comparing their practices with practices from other European countries and will be motivated to devise measures to further improve their effective-ness towards the disadvantaged learner.

3.2. Sharing successful adult education practices for the disadvantaged

Identifying and sharing an organization's successful adult education practices for the disadvant-aged learner helps staff members and educators to learn from each other and reuse proven prac-tices. Transferring the practices from one adult education provider to another leads to the im-provement of its effectiveness towards the disadvantaged learner, and to better quality results and impact on learners.

6

Page 7: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

The immediate benefits from effective sharing are the following:

• Transferring know-how to other adult education providers, which are less experienced in working with disadvantaged learners

• Avoidance of replication of effort or “reinventing the wheel”

• Replication of successes throughout an organization, and also from one adult education pro-vider to another

• Self-improvement of adult education providers which have adopted successful practices on the disadvantaged learner, through the exchange of experience with other adult education providers and the education community

Benefits can also be identified on a macro level horizon:

• Increase of the overall quality of services of adult education providers

• Minimization of organizational knowledge loss (both tacit and explicit)

• Decrease of the learning curve of new educators/recruiters

• Promotion of dissemination of the results, stimulation of the communication and the co-ordination among the stakeholders, animation of the debate between the interested parties

• Contribution to the development of policy concerning inclusive education by the European and national policy-makers

4. StakeholdersThe identification of stakeholders is of crucial importance for the quality of the collected data. In the context of the IMPADA project, the practitioners of successful adult education practices for the disadvantaged learner are those persons who will transfer their factual knowledge related to a particular case of adoption of a successful practice, by reporting the practice of their organiza-tion and disseminating it to other interested EU adult education providers.

For the IMPADA project, stakeholders adopting successful adult education practices for the disad-vantaged learner are:

Adult Education centres Life-long learning centres Vocational training centres FE and skills providers Adult Education Associations Educational research institutes Educational policy making organizations Educational and training National/EU authorities Associations of disadvantaged groups

The above should be motivated to participate by becoming aware of the importance, goals and objectives of the IMPADA project. Furthermore, they should know the reasons why they are

7

Page 8: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

chosen to report their practice (e.g. because it is well known that their organization uses a suc-cessful inclusive practice, and thus, as experts, they are requested to share their factual know-ledge for that specific practice). Stakeholders should be also motivated by being well informed of the benefits from the collection of successful adult education practices on the disadvantaged (i.e. sharing practices with any other interested party, the foreseen identification of at least 20 good practices on successful measures to improve adult learning providers’ effectiveness on the disad-vantaged learner, and the fact that these good practices will provide input for the development of the IMPADA O3A2 “Recommendations for improvements on effectiveness of adult education for the disadvantaged learner”).

The motivation of adult education providers, research bodies and adult education associations will also benefit the IMPADA project, as relevant stakeholders adopting or developing successful adult education practices on the disadvantaged learner will be willing to share their practice, assuring the quality of the produced evidence. Besides, their interest and – hopefully- enthusiasm for the IMPADA project can also lead to fulfilling further project’s needs in a later time, when their contri-bution will be requested again. Moreover, possible new players in the field of adult education may be found who can play a crucial role in the exchange of experience among the EU-wide strong supporters of inclusive adult education.

Moreover, disseminating collected evidence among stakeholders can lead to the replication of the successful practices in the organizations of the recipients, by adopting already tested and proven ones. Furthermore, disseminating collected practices could attract stakeholders such as associ-ations for the disadvantaged learner that may have interest in investing resources for the im-provement of a specific practice. Finally, propagating to policy makers successful adult education practices for the disadvantaged learner could facilitate up-scaling relevant EU and/or national policies.

5. Sampling methodologyThe suggested sampling methodology for the collection of successful adult education practices for the disadvantaged learner is “quota sampling”, which is the non-probability equivalent of strati-fied sampling. Quota sampling emerges as an attractive choice when time is an issue, as primary data collection can be achieved in a cost and time saving manner. Moreover the lack of an ex-haustive and accurate sample-frame across all consortium countries further justifies this choice.

In quota sampling, a population is first segmented into mutually exclusive clusters, and then pur -posive1 (or judgment) sampling is used to select the subjects or units from each cluster, based on a specified proportion. For the purposes of this research, Partners have been foreseen to gather at least 10 practices each; therefore it is suggested that at least 40 units should be sampled by each Partner.

The population for the collection of successful adult education practices for the disadvantaged learner are the executives and managerial staff of adult education providers (including LLL and VET), educational research organisations, and policy relevant bodies of the consortium countries.

1Purposive sampling is a non-probability sampling method which, nonetheless, is superior to simple convenience sampling, and where elements selected for the sample are chosen by the judgment of the researcher. It is argued that researchers can obtain a representa-tive sample by using a sound judgment, which will result in saving time and money. Therefore, given the profiles and competences of Partners purposive sampling can safely be employed.

8

Page 9: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

The mutually exclusive clusters for the aforementioned are: adult education providers, adult edu-cation associations, adult education research bodies and adult education related policy making bodies. Partners will target specific clusters according to their profile, capacities and relevance, and will accordingly judge which units should be respectively sampled from each cluster. Table 1 below presents the type of organization for the IMPADA consortium Partners, and the suggested cluster to retrieve successful sample-level information. (N.B. project Partners are advised to invite at the reporting process those stakeholders who are most appropriate and easy to reach).

IM-PADA partner

Type of organisation Suggested target cluster

DACES (UK) Adult Education com-

munity centre

adult education Providers at national level educational research organisations policy relevant bodies at national level associations of disadvantaged groups

IBE (PL) National educational research institute

adult education Providers at national level educational research organisations policy relevant bodies at national level associations of disadvantaged groups

UPTER (IT) Private Adult Education

provider

adult education Providers at national level educational research organisations adult Education Associations at national level associations of disadvantaged groups

ENAEA (EE)

Adult Education Associ-ation

adult education Providers at national level adult education providers at EU level adult Education Associations at EU level associations of disadvantaged groups

PROMEA (GR) Private non-profit re-

search organization

adult education Providers at national level Educational research organisations Policy relevant bodies at national level associations of disadvantaged groups

Table 1 Suggested target cluster per IMPADA Partner

6. Identifying Good PracticesThis chapter describes the methods used for the identification, validation and evaluation of good practices on adult education provision for the DISADVANTAGED LEARNER. It describes the evalu-ation fields and criteria used to filter case studies and define the quality levels of a good practice. The methodology described is structured and applied in the following, indicative stages:

1. Define evaluation fields.

2. Define evaluation criteria.

3. Set score levels, weights and thresholds.

9

Page 10: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

4. Define ranking groups / classification categories

Figure 1 Structure of the research methodology

6.1. Defining evalu-ation fields

Implemented adult education practices for the disadvant-aged are going to be examined and evaluated within specific fields in order for the IMPADA project to assess their appro-priateness as good practices, and classify them accordingly. These fields are also taking into account the specificities

10

Define evaluation fields.

Define evaluation criteria.

Set score levels, weights and thresholds.

Define ranking groups /

classification categories

Fields Questions

Impact Which of the CAF key aspects has the good practice shown to be most effective at?

Evaluations Has an implemented practice ever been validated or evaluated, either by internal or external evaluators?

Challenges To what extent have problems or diffi-culties challenged implementation?

Scale The extent to which the practice been in-tegrated across the organisational struc-ture.

Transferability Is the practice customisable and adaptable for use across disadvantaged learners groups?

Page 11: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

and objectives of the IMPADA project and address a series of issues and aspects that differentiate various implemented practices and could also be phrased as questions (see table 2).

By defining certain fields of investigation, a more focused and clearly structured evaluation pro-cess is expected to emerge, targeting specific aspects and factors that differentiate adult educa -tion practices for the disadvantaged learner, in terms of effectiveness and transferability.

6.2. Defining evaluation criteria

The evaluation criteria to be used in order to differentiate and rank practices correspond to the aforementioned evaluation fields and are set and justified as shown below:

1. Impact.Assess the impact of an implemented practice according to the number of CAF criteria it ad-dresses.

2. Previous evaluations The type and level of any previous evaluation (e.g. internal, external, preliminary, systematic) is an important factor. Practices that have been already validated or evaluated to some extent are rather more promising than others.

3. Challenges in implementationPossible problems encountered, difficulties met and lessons learned in implementing a practice are issues that any stakeholder needs to know in order to assess a practice. If encountered prob -lems rose to a prohibiting level or too much time was consumed in trouble-shooting, then the practice should not be considered successful.

4. Scale of practice integration across organisation departmentsIdentifying the extent of adoption for a particular practice within an organisation is an important aspect. Practices involving higher numbers of individuals or adopted within entire sections or de-partments within an organisation are more likely to increase their success potential and transfer-ability value.

5. Transferability potentialA practice’s potential or proven record of being successfully transferred to different disadvant-aged groups and organisational settings is one of the most important criteria for identifying it as a good or a promising practice. Practices that demonstrate strong evidence of transferability should be evaluated higher than others.

Evaluation fields and criteria are linked and structured as shown in the table 3.

Good practices evaluation fields and criteria

Evaluation fields Evaluation Criteria Objectives

Impact Number of criteria Assess how the practice addresses CAF identi-fied criteria

Evaluations Type of previously ap-plied evaluation

Identify the type and extent of any previous val-idation and evaluation (e.g. internal, external,

11

Table 2Fields and Questions

Page 12: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

preliminary, systematic)Challenges Type and extent of

problems encountered in implementation

Assess the extent of encountered problems and difficulties that have hindered a practice’s im-plementation, and judge whether this chal-lenges are inherent in the good practice or not

Scale Involved individuals or organisational depart-ments

Identify the extent to which the practice been integrated across the organisational structure

Transferability Level of transferability Identify the extent of transferability of the prac-tice across various disadvantaged groups

Table 3 Good Practice evaluation fields and criteria

6.3. Ranking and Classification

Practices collected through IMPADA partners’ efforts in the context of O3A1 and also O4A1 will be ranked and classified in the following indicative stages:

1. Practices are examined in relation to the predefined evaluation fields and criteria.

2. Practices are given scores according to the “IMPADA Good Practice Evaluation Chart”.

3. Practices are grouped on an eligible / non eligible basis.

4. Practices are ranked in predefined ranking groups and classified accordingly.

Figure 2 the process of practices’ ranking and classification

The IMPADA Good Practice evaluation chart (table 4) is proposed as a tool to facilitate and stand-ardize practice evaluation and classification. As shown in the evaluation chart below, each collec-ted practice could get a score ranging from 1 to 5 for each evaluation field. A score weight and a score threshold have been assigned to each evaluation field and criterion in order to ensure a minimal quality level and a graded output of results.

12

Practices are examined in relation to the

predefined evaluation fields and criteria.

Practices are given scores according to the “IMPADA Good Practice

Evaluation Chart”.

Practices are grouped on an eligible / non eligible

basis.

Practices are ranked in predefined ranking

groups and classified accordingly.

Page 13: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Fields Criteria/Score

1 2 3 4 5

Impact2

WEIGHT: 1.5Matches no CAF criteria

THRESHOLD:Matches 1 CAF criterion

Matches 2-3 CAF criteria

Matches 4-6 CAF criteria

Matches 6 or more CAF cri-teria

Evaluations 3

WEIGHT: 1Practice has never been evaluated, either intern-ally or extern-ally.

THRESHOLD:Practice has been evalu-ated on a pre-liminary / non standardised stage.

Practice has been internally evaluated.

Practice has been extern-ally evalu-ated.

Practice has been system-atically evalu-ated by both internal and external evalu-ators.

Challenges4

WEIGHT: 1.5[note: sub-jective judge-ment will be used to de-termine whether or not identified challenges are inherent and therefore could affect implementa-tion]

Had serious challenges and difficulties that significantly impacted its results

THRESHOLD:had some challenges and diffi-culties that hindered its implementa-tion

Had only minor challenges and difficulties.

Had almost no problems and diffi-culties.

Had no prob-lems or diffi-culties whatso-ever.

Scale5

WEIGHT: 1THRESHOLD:Practice has been only im-plemented within a lim-ited group of people.

Practice has been only im-plemented within a small department.

Practice has been imple-mented in an organisational department involving the entire depart-ment’s staff.

Practice has been imple-mented in the entire organi-sation involv-ing most of its staff.

Practice has been imple-mented in the entire organi-sation involv-ing its entire staff.

2 Refer to questionnaire section 3

3 Refer to questionnaire section 4

4 Refer to questionnaire section 5

5 Refer to questionnaire section 613

Page 14: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Fields Criteria/Score

1 2 3 4 5

Transferabil-ity6

WEIGHT: 1.5

The good prac-tice cannot be used with other learners

THRESHOLD:The good practice can only be used with this type of learner in similar con-texts.

The good prac-tice can be cus-tomised and adapted so it can be used with a larger number of learners

The good practice can be customised and adapted so it can be used with learners with a wider range of disadvantaged/to address other issues of inclusivity

The good practice is universal andcustomisable and can be used with any learner

Table 4 IMPADA good practice evaluation chart

According to the IMPADA Good Practice Evaluation Chart possible score levels for both eligibility and non-eligibility scenarios are configured as shown in the table 5.

Score results are calculated using the following formulae:

Practice score for each field = score x field weight. Overall practice score = Sum of all field scores.

N.B. To achieve a “good practice status”, a practice should score no less than the threshold set for the “Impact” and “Challenges” fields.

According to this score grading, 3 ranking levels are defined as shown in the tables 5 & 6.

Ranking level

Score Evaluation Status Explanation

Level 1 >20.5 – 32.5 Very good Eligible High quality level. Practice demon-strates proven effectiveness, trans-ferability, and relevance to CAF.

Level 2 >6.5 – 20.5 Critical Eligible, if threshold met for “Impact” and “Chal-lenges”

If score ≥12, and practice has met “Impact” and “Challenges” thresholds then practice can be considered eligible.

Level 3 0-6.5 Not good Non Eligible Non-satisfactory quality level.

6 S Refer to questionnaire section 7

14

Page 15: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Critical area

Should not be considered a good practice.

Table 5 IMPADA good practice ranking levels

Practices are ranked according to a predefined set of three levels on a 1-3 scale. Practices classi-fied in levels 1 (high quality level) and 2 (good / eligible quality level) can be considered good practices. Practices classified in level 3 should be considered non-eligible. It is expected that this ranking scheme will ensure a certain degree of consistency in evaluating and classifying practices also allowing for updated inputs and comparative analysis of additional practices to be collected.

7. Collection process, means and tools

The collection of practices is of crucial importance for the IMPADA project, as the evidence collec -ted will be edited and analysed with the aim to contribute to the development of the catalogue with case studies, which will be available for interested parties and stakeholders, and will also provide input for the development of the recommendations for improvements on effectiveness of adult education for the disadvantaged learner (O3A2).

This chapter de-scribes the actions that IMPADA partners should undertake for the collection of the anticipated evidence, a time plan for the collection process, the means with which data can be collected and the tools towards that purpose.

7.1. Processes

Collection of practices will follow a survey format to be promoted by all IMPADA partners to relev-ant stakeholders of their network. The target will be to collect at least 10 practices from each of the partners' networks. In addition to partners own efforts, the survey will be promoted through the IMPADA network to be developed in O4-A1. Interested parties will fill-in the respective form and submit their responses. After the completion of the survey, PROMEA will gather and evaluate all responses collected, to select at least 20 good practices on successful measures to improve adult learning providers’ effectiveness on disadvantaged learner.

More specifically:

1. Each partner will translate in own language the survey developed in this document (see Tables 8- 15 for development guidelines, and the link to an online form of the question-naire)

2. PROMEA will create the template online & offline surveys3. All Partners should identify the potential practices’ implementers, according to the in-

formation of chapters 4 “Stakeholders”, and 5 “Target population and sample” of this document.

4. All Partners should register the selected invited stakeholders in the template form of an-nex 9.5 “Template form for the registration of the invited providers of successful adult education practices for the disadvantaged learner” and send it to PROMEA.

15

Table 6 Eligibility and non-eligibility ranges

Page 16: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

5. All Partners should send inviting emails according to the instructions given in chapter 7.3.1 “Invitation email” and to the template email of the annex 9.1.

6. All Partners should send follow up emails based on the instructions given in chapter 7.3.2 “Follow up email” and to the template email of the annex 9.2.

7. Stakeholders should report their practices based on the template form, developed in chapter 7.2.1 “Questionnaire”.

8. All IMPADA Partners should send a thank you email to those who have responded accord -ing to the instructions given in chapter 7.3.3 “Thank you email” and to the template email of the annex 9.3 ”Thank you email template”.

9. Each IMPADA Partner is responsible to collect at least 10 practices on effective adult edu-cation provision for the disadvantaged learner.

10. DACES assisted by PROMEA has the overall responsibility for the monitoring of the pro-cess of the collection of practices and it should perform the following actions:

a. Check if the desired number of practices is reported.b. Notify Partners which have not promptly provided practices.

11. All IMPADA Partners should have sent by March 3rd 2017 at least 10 practices each to PROMEA

The following table contains a summative description of the responsibilities of each IMPADA part -ner and the deadlines for the collection of the requested evidence.

IMPADA partner

Action Deadline

PROMEA Release of methodology and tools By January 30th 2017

PROMEA Development of the final online version of the ques-tionnaire

By January 30th 2017

All Partners Translations of the questionnaire By February 7th 2017

All IMPADA partners

Identification of relevant stakeholders and adult education providers and registration of identified invitees. (Send the form with the identified invitees to PROMEA)

By February 7th 2017

All IMADA partners

Sending invitation email with the questionnaire to the providers

8 – 13 February 2017

All IMPADA partners

Sending follow-up email to the providers, when it is appropriate

15– 20 February 2017

All IMPADA partners

Send to PROMEA collected practices 13 -24 February 2017

All IMPADA partners

Sending thank you email to the providers 27-28 February 2017

PROMEA Checking if questionnaires are being filled in 8 – 24 February 2017

16

Page 17: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

IMPADA partner

Action Deadline

PROMEA Administering and monitoring the collection process Throughout the whole collection process

Table 5 Time line of activities for collecting practices

7.2. Means

There are two means for collecting the practices from the providers: an online survey (Google form) and a questionnaire to be distributed through emails.

Reporting successful adult education practices for the disadvantaged learner in a Google form makes the practice available to everyone who is interested in this area and has the link to the sur -vey, and the content of the survey can be more easily integrated in the IMPADA platform for the exchange of good practices. Furthermore, a web interface guarantees the confidentiality and the safeguarding of the information and it does not invoke any intermediary; therefore the content cannot be altered or changed. Moreover, reporting practices in a Google form web is fast, simple and flexible.

Alternatively, adult education practices for the disadvantaged learner can be sent in a question -naire form, as an attachment to the inviting email which would have to be sent back filled in. This procedure is simple, but it involves intermediaries or additional persons, thus mitigating the con-fidentiality of information gathered and it could possibly increase the effort needed for the collec-tion of practices (e.g. due to tracking emails back and forth).

The Google form should be user friendly, robust and should guarantee the safety and the confid-entiality of the personal data. PROMEA will develop the questionnaire template in English and all Partners will then develop translated in own languages. The questionnaire needs to include an explanatory, introductive text so as to provide to the target groups of the questionnaire, guidelines and tips for the web site login, the uploading of the practice and the filling in of the re-porting form (this should be translated to all Partners’ languages). Furthermore, a counter will track the number of persons that have uploaded their practice. Finally, people filling in the ques-tionnaire should be prompted -through a message- in case they have not completed all requested fields.

7.2.1. Questionnaire template

It is proposed to use the following template for creating the questionnaire form for the reporting of adult education practices for the disadvantaged learner.

The questionnaire template is designed to consist of the following thematic categories:

1. Personal and organizational information Description of the practice 2. Description of the successful adult education practice on the disadvantaged learner3. Impact4. Evaluations5. Challenges6. Scale7. Transferability issues8. Details of good practice

17

Page 18: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

This division facilitates the filling in of the questionnaire from the provider and the analysis and editing of the collected data. Each thematic category contains a table, where the questions are listed in and which consists of the following fields:

1. The 1st field has the indexing code of the question. The questions are indexed so as to permit the quick reference and localization in case any clarifications are requested by the providers.

2. The 2nd field contains the question.3. In the 3rd field there are instructions for the questionnaire respondent. It is suggested

that these instructions could be included in the “description” field of the online question-naire

4. In the 4th field there are some suggestions for the developer of the online questionnaire, regarding the html format the questions should have (not to be included as a distinct column/field in the actual questionnaire form).

The 1st page of the questionnaire template is an introductory page, which describes shortly the IMPADA project, the aims and the target groups of the questionnaire, the benefits for the pro-viders of practices and the total time needed for the filling in.

The suggested questionnaire template is the following:

7.2.2. Identity of the organisation where the successful adult education practice on the disadvantaged is adopted

Index

Question Help, instructions for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

1.1 Name of the organisation in English

Short answer

1.2 Country:a) UKb) Polandc) Italyd) Estoniae) Greecef) Other

Please select the country or the countries where the practice is adopted.

Multiple choice

18

Page 19: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Index

Question Help, instructions for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

1.3 Type of organisation:a) Adult Education

centresb) Life-long learn-

ing centresc) Vocational

training centresd) FE and Skills

providerse) Adult Education

Associations f) Educational

research insti-tutes

g) Educational policies making organizations

h) Educational and training Na-tional/EU au-thorities

i) Associations of disadvantaged learner

j) Other

Please select the type of the organisation where the practice is adopted.

Multiple choice

1.4 Name and job title of person completing the questionnaire in English

Short answer

1.5 Estimated number of adult learners within the organisation:

a) 1 – 50b) 51 – 100 c) 101 – 500 d) 500 -1000e) 1000 – 5000f) 5000+

The answers should be in a drop down menu format.

1.6 Number of campuses – local/national branches:

a) 1 b) 2-5 c) 5-10d) 10+

The answers should be in a drop down menu format.

Table 6 organisational info

19

Page 20: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

7.2.3. Description of the successful adult education practice on the disadvantaged learner

Index Question Help, instruction for the ques-tionnaire respondent

Specification/ suggested HTML form for the de-veloper of the online ques-tionnaire

2.1 Brief description (and title if applicable) of the adopted practice

Required Short answer. 100 words max

2.2.a Does the good practice address teaching and learning?

Required Yes/no

2.2.b Which of the following leadership and manage-ment aspects does the good practice address?

a) Inclusive strategy development

b) Inclusive and targeted planning and recruitment

c) Quality improve-ment

d) Learner voicee) None

Please identify the area that the practice addresses. Tick as many as apply. In case of “other” please spe-cify.

Check boxes

2.2.c Which the following learner outcomes as-pects does the good practice address?

a) Skills and quali-fications

b) Socio-economic benefits of learning

c) Destinationsd) None

Please identify the area that the practice addresses. Tick as many as apply. In case of “other”, please spe-cify.

Multiple choice

2.3 Current state of the good practice is

a) Under consid-eration

b) Under develop-ment

c) In pilot phased) Operationale) Discontinued

partiallyDiscontinued totally

Multiple choice

20

Page 21: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Index Question Help, instruction for the ques-tionnaire respondent

Specification/ suggested HTML form for the de-veloper of the online ques-tionnaire

2.4 Who/what prompted your organisation to de-velop this good practice?

a) external inspec-tion

b) quality assur-ance

c) Internal quality assurance pro-cedures, such as Observation of Teaching and Learning

d) Learner feed-back

e) Partner feed-back

f) Stakeholder feedback

g) Employer feed-back

h) Funding policy and targets

i) Other

Multiple choice

2.5 When was the good practice implemented?

Please indicate the year when the practice was first adopted

Short text

Table 7 Description of the practice

21

Page 22: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

7.2.4. Impact

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

3.1 Which of the following leadership and management aspects has the good practice been shown to have positive impact upon? (tick as many as apply)

a) inclusive strategy development

b) strategic partnerships with stakeholders

c) strategic policy for disadvantaged learners

d) inclusive curriculum development

e) promotion to disadvantaged learners

f) inclusive recruitment strategy

g) continuous professional development (CPD)/staff training for working with disadvantaged learners

h) Monitoring and improving teaching, learning and assessment

i) Using feedback to effect change and inform planning

j) Nonek) Other

Required Check boxes

3.2 Which of the following teaching, learning and assessment aspects has the good practice been shown to have positive impact upon ? (tick as many as apply) a. Inclusive initial

assessmentb. Inclusive diagnostic

Required check boxes

22

Page 23: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

assessmentc. Inclusive learning

outcomesd. Evidence-based

pedagogy/andragogye. Differentiated

teaching, learning and assessment to meet needs

f. Inclusive summative assessment

g. Noneh. Other

3.3 Which of the following learner outcomes aspects has the good practice been shown to have positive impact upon? (tick as many as apply) a. Narrowing the

achievement gap for disadvantaged learners

b. Impact of ‘soft skills’c. Positive destinations

and progression for disadvantaged learners

d. Nonee. Other

Required Check boxes

Table 8 Impact

7.2.5. Evaluations

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

4.1 Has this good practice been evaluated?

a) Practice has never been evaluated, either internally or externally

b) Practice has been evaluated on a preliminary / non standardised stage

c) Practice has been

Required Multiple choice

23

Page 24: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

internally evaluatedd) Practice has been

externally evaluatede) Practice has been

systematically evaluated by both internal and external evaluators

Table 9 Evaluations

7.2.6. Challenges

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

5.1 Practical implementation of this good practice:

a) had serious challenges and difficulties that significantly impacted its results.

b) had some challenges and difficulties that hindered its implementation.

c) had only occasional challenges and difficulties that have not hindered its implementation.

d) had only minor challenges and difficulties.

e) had no notable challenges or difficulties

Tick as many as apply. Multiple choice

5.2 Please detail the challenges encountered in implementation (up to 150 words)

. Short answer

Table 10 Challenges

7.2.7. Scale

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

6.1 To what extent has the practice been integrated across the organisational structure?

a) Practice has been implemented within a

Select which best applies Multiple choice

24

Page 25: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

limited group of people.b) Practice has been

implemented within a small department.

c) Practice has been implemented in an organisational department involving the entire department’s staff.

d) Practice has been implemented in the entire organisation involving most of its staff.

e) Practice has been implemented in the entire organisation involving its entire staff.

Table 11 Scale

7.2.8. Transferability

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

7.1 Select which is the most accurate for this good practice

a) The good practice cannot be used with other learners

b) The good practice can only be used with this type of learner in similar contexts

c) The good practice can be customised and adapted so it can be used with a larger number of learners

d) The good practice can be customised and adapted so it can be used with learners with a wider range of disadvantage/to address other issues of inclusivity

e) The good practice is universal and customisable and can be used with any

Identify the extent to which this practice is customisable and adaptable for use with disadvantaged learner

Multiple choice

25

Page 26: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

learner

Table 12 Transferability

7.2.9. Other details of the good practice

Index Question Help, instruction for the questionnaire respondent

Specification/ suggested HTML form for the developer of the online questionnaire

8.1 Please provide a description of the good practice (max 250 words)

Paragraph

8.2 Specify some conclusions or main lessons learnt (up to 3) from the adoption of the practice (max 250 words)

Paragraph

8.3 URL for downloading any additional documents, resources or other helpful data

Please provide also a short text description to describe what is provided with URL

Short text

Table 13 Other details

7.3. Supporting tools

7.3.1. Invitation email

An explanatory text along with the inviting email helps increasing the credibility of the question-naire by granting legitimacy and strength to it, gaining at the same time the invitee’s interest and trust. Therefore, the inviting email should include:

1. A brief description of the purpose of the questionnaire.A brief but meaningful text is necessary to introduce the purpose the questionnaire serves. Say a few things about that purpose, e.g. the IMPADA Project is a European co-funded project that aims at improving the systematic evaluation of the effectiveness of adult education provision towards the disadvantaged learner with a view to reduce disparities in and enhance access to adult learn-ing.

2. The reason why the person receiving the email was chosen. In the context of gaining the recipients’ trust it is advised to explain them the reason they were chosen to participate. Use the available information about the potential provider of the practice, their relevance to adult education, type of organisation and so on to develop a convincing argu -

26

Page 27: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

ment about their selection. (e.g. Dear [salutation, surname], we would like to invite you to report an adult education practice for the disadvantaged adopted in. Your experience and involvement in adult education are very much aligned with the target group expected characteristics for the re-porting of successful adult education practices for the disadvantaged learner, and we would thus highly appreciate your valuable contribution in our research.)

2. The importance of the collection and sharing of practices/ the reason(s) why the recipi-ent should complete the questionnaire.

Motivate the recipients and make them understand how and why their participation will promote science, knowledge or any other noble purpose. Stay to the facts and do not try to make it heavier than it is. Not all recipients are experts but they do have some knowledge about this issue. (i.e. “the collection of successful adult education practices on the disadvantaged learner will facilitate the development of the recommendations for improvement on effectiveness of adult education for disadvantaged learner”, is an explanation that people will understand and accept. “The collec -tion of practices will help us bring a new era in the field of adult education” is somewhat far -fetched and could be judged negatively.)

4. Explanation of how the reporting of practices will be used.The participants will want to know how the information they reveal will be treated, regardless of privacy and consent statements. Spend a few lines explaining how the practices will help provid -ing with guidance and useful instructions to the interested stakeholders. If the practices will be-come publicly available let them know it and promise to keep them updated about the outcomes: the IMPADA network of adopters and repository of results.

5. The link to the questionnaire“Our questionnaire can be found in the projects’ official page” is not the right indication. Provide the recipient with the exact link where the questionnaire can be found, so that they will be imme-diately directed to the questionnaire’s first page. You are already asking for somebody’s time; do not make them spend extra time searching for the questionnaire; chances are they will not. Do not forget to include an attached file with the questionnaire.

6. A request for sending feedback on the practice within the specified deadlineThe participants will have to know until when they can report their practices. Ask them kindly to report the practices by 18th February 2017 so as to collect all the requested data within the time limits specified by the IMPADA project.

7. A contact person’s detailsApart from the obvious reason of feedback, the contact details of a person grant reliability to the email per se and the request in general. The recipients would feel more comfortable knowing that this is a responsibly signed email and that there is an actual person making this request.

8. An appreciation concluding line Within the context of appreciation to the recipient a thank you line should conclude the email. (i.e. We would like to thank you in advance for your participation in the reporting of successful adult education practices on the disadvantaged.”)

27

Page 28: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

7.3.2. Follow up email

Whatever the reason might be, some people will not report the practice of own organisation by completing the questionnaire. In these cases a follow up email should be sent after a rational period of time (in our case one week after the inviting email was sent). It is important that this email will not be a copy of the original so as to show that some effort has been put in it.

What is more, it should strictly be a reminder of the questionnaire. It should include the initial invite rephrased, the link to the questionnaire and a kind concluding phrase. This has to be altered depending on whether it is a targeted follow up or not. If it is a targeted one, a line like “Thank you for your time. Your participation is much appreciated/ important” is fine. Otherwise, the con-cluding line should distinct between the intended and not intended recipients of this reminder. A line like “if you have already answered the questionnaire, we apologise for this reminder/please ignore this reminder” is a good idea. Do not send a second reminder.

7.3.3. Thank you email

It is polite to send a thank you reply to people that spent some time to complete the question-naire. It shows appreciation and acknowledgment.

The thank you email should be brief and to the point. It could also contain some information about the outcome of the survey and the results to be produced (e.g. the date that the recom-mendations for improvement on effectiveness of adult education for the disadvantaged learner is assumed to be published and the available link).

8. Quality assuranceQuality assurance is the process of verifying or determining whether products or services meet or exceed expectations. Quality assurance is a process-driven approach with specific steps to help define and attain goals. This process considers design, development, production, and service.

PROMEA should check progress on the reported practices and on the on-line questionnaire and notify the Partners in case practices are not reported yet or contain incomplete questions or data. This will guarantee the quality of the expected data and the collection of the foreseen practices. Furthermore, it is recommended to test the questionnaire before it is uploaded so as to identify any problems or difficulties for the providers of adult education practices for the disadvantaged learner. It is also suggested, in case of availability, to circulate the questionnaire to external ex-perts and ask for their advice or any suggestion regarding the design or development of the ques -tionnaire.

Parameters for evaluating, categorizing, and prioritizing risks typically include risk likelihood (i.e., the probability of risk occurrence), risk consequence (i.e., the impact and severity of risk occur-rence), and thresholds to trigger management activities. Risk parameters are used to provide common and consistent criteria for comparing the various risks to be managed. For the upcoming collecting of practices the following risks have been identified:

Id Risk Likelihood(Low / Medium / High)

Impact(Low / Medium / High)

28

Page 29: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

R1 Insufficient practices gathered Medium High

R2 Heterogeneous and/or incomplete data gathered

Low High

R3 Delays in the reporting of practices Medium Medium

Table 14 Risk management: identifying risks

A critical component of a risk mitigation plan is to develop alternative courses of action, work-arounds, and fallback positions, with a recommended course of action for each critical risk. Risks are watched when there is an objectively defined, verifiable, and documented threshold of per-formance, time, or risk exposure (the combination of likelihood and consequence) that will trigger risk mitigation planning or invoke a contingency plan if it is needed. In many cases, risks will be accepted. Risk acceptance is usually done when the risk is judged too low for formal mitigation, or when there appears to be no viable way to reduce the risk. The identified risks form a baseline to initiate risk management activities as presented in table 17 below:

Id Risk Threshold Proposed Action

Preventive Contingency

R1 Insufficient practices gathered

Less than 50 prac-tices

Identify alternative providers of adult education practices for the disadvantaged learner

Contac the altern-ative providers and engage in a tele-phone interview to collect data on their practice

R2 Heterogeneous and/or incom-plete data gathered

More than 20% of the questions are not filled in.

Warning messages to the providers during filling in and before the publication of the practice

Obligatory re-sponses for those questions that cor-respond to the re-spective evaluation fields

R3 Delays in the reporting of practices

The providers have not de-livered the re-ports within the specified dead-lines

PROMEA has to monitor the total number of practices uploaded at the website

PROMEA should notify the respect-ive partners to fol-low up and pursue responses

Table 15 Risk management: corrective measures

9. Annexes

9.1. Invitation email template

Dear [salutation, surname]

29

Page 30: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

I represent the consortium of the European co-funded project IMPADA 2015-1-UK01-KA204-013666 Improving the effectiveness of adult education for disadvantaged learner) of the Erasmus+ programme, which aims at improving the systematic evaluation of the effectiveness of adult education provision towards the disadvantaged learner with a view to reduce disparities in and enhance access to adult learning

Knowing of your organization’s long and significant involvement in the field of Adult Education (you can mention specific events or previous projects or experiences) we would like to invite you to the reporting of a successful adult education practice for the disadvantaged learner that has been implemented by your organization. We trust that you are the appropriate person to offer your valuable insight by reporting the practice adopted or developed by your organisation.

To this end, we have prepared one questionnaire so that you can provide us with all the useful information and detailed description of the practice adopted. You are kindly requested to report your adult education practice for the disadvantaged learner following this link and according to the instructions attached in the respective file “instructions for log in at the website” with this email. Alternatively you can also fill in the attached file and kindly return it to us at your earliest possible convenience, and by 17th February 2017. We understand that in some of the topics tackled you may have questions on what to fill in, and in that case please consult the instructions found in the introductory section of the questionnaire. If the instructions are proven not to be helpful to you, please do not hesitate to contact us for further instructions.

In the context of the IMPADA project, we intend (given your permission of course) to refer both to the name of your organization and to your practice with the aim to include them in the recom-mendations on successful adult education practices which is going to be developed from the IM-PADA project. Furthermore, the reported practice could also be transferred and adopted by any other interested party, so would be grateful if you could provide us any additional information, documentation and files relevant to the implementation of the practice that your organisation has implemented. Once the aforementioned deliverables are finished and published, you can find them at the following link www.impada.eu at the tab “Outputs”, and I would also be happy to send you a copy of them.

The deadline for the reporting of practices is by 17th February 2017, so please fill in the question-naire before that date.

I would also like to kindly ask you to notify me whether you agree that we list your name as one of the experts that assisted us and also on whether you agree that we can cite your views in our re -port.

On behalf of the IMPADA consortium I would like to thank in advance for your time and effort.

Kind regards,

Signature, Contact details

30

Page 31: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

9.2. Follow up email template

Dear Sir/Madame

We would like to kindly remind you that you have been requested to report a successful adult education practice on the disadvantaged learner of your organization in the context of the European Project IMPADA.

We noticed that we have not yet received any response from your side, so we took the liberty to remind you that your knowledge and contribution are very important to the achievement of the goals of the IMPADA project.

Your personal data will be treated as confidential.

Here is the link to the questionnaire

We would like to thank you in advance for your attention and actions, and we trust that you will find the time to fill the questionnaire.

Please ignore this reminder if you have already answered the questionnaire. (optional)

SignatureContact details

9.3. Thank you email template

Dear Sir/Madame

We would like to thank you for your time reporting the successful adult education practice on the disadvantaged learner. Your contribution is much appreciated.

We would be pleased to keep you updated once the results are published.

Yours sincerely,

Signature, Contact details

9.4. Link to the questionnaire in Englishhttps://docs.google.com/forms/d/e/1FAIpQLSfTZVLyPPiV010mxJL9qrKMFcW4wDsmG-aDynXO-qVnXjAKjKg/viewform

31

Page 32: IMPADA_O3_A1_ Research methodology successful practices for disadvantaged learners

9.5. Template form for the registration of the invited providers of successful adult education practices for the disadvantaged learner

Index Name & surname

Profession Name of organisa-tion

Email address

Contact details7 Brief de-scription of practice

Invitation date

R/ NR8

7 Phone number, mobile phone, address

8 Report/ Not report yet

32