A tool to monitor and evaluate a HTA network: the case of REBRATSLeão,L.S.C.1; Vanni,T.1

1 Department of Science and Technology, Brazilian Ministry of Health

INTRODUCTION

After six years of existence, The Brazilian Network for HealthTechnology Assessment (REBRATS) now encompasses 78 participatinginstitutions. Since its creation, there is a growing need to certificate and tomonitor the activities of its members.

The creation of a methodology is required in order to monitor thenetwork’s production and collaboration process. An annual certification ofthe institutions through an activity report aims to fill this gap.

This process would also contribute to the promotion of effectiveparticipation of member institutions and to greater collaboration betweenthem.

The study of scientific collaborationnetworks seeks to correlate attributes of itsforming entities, patterns of relationshipsbetween then and network performance. In astudy done on SISREBRATSi a "small worldphenomenon" was observed, suggesting thatmost of these collaborations occurs in a smallcircle of researchers. Therefore there is space tobetter integrate the network in order to generatehigher economy of scale, expertise, knowledgetransfer, and to reduce the duplication of work.

However, SISREBRATSi data analysis islimited since it refers only to the includedstudies, which are a fraction of the totalproduction of the network institutions. Besidesthat it does not capture other types ofcollaborations like trainings and events. Thus it isnecessary to conduct primary data collectionrelated to these collaborations.

iREBRATS’ studies data base.

BACKGROUND

Figure 2. REBRATS’ sociogram based oncoauthorship evaluation of SISREBRATS’studies.

Figure 1. REBRATS’ website: rebrats.saude.gov.br

A tool to monitor and evaluate a HTA network: the case of REBRATS.

Leão,L.S.C.1; Vanni,T.1

1 Department of Science and Technology, Brazilian Ministry of Health

In order to define a model of primary data collection relating to HTAcollaborations, a systematic review was conducted on MEDLINE . The consultationyielded 260 records. By screening titles and abstracts, we identified 8 eligiblestudies. During the analysis of the methodologies used in these eight articles,important aspects related to the purpose of the study, the instrument used, thecollection strategy and the response rate were identified.

Search strategy: ("network analysis"[All Fields] OR "network structure"[All Fields]OR "network theory"[All Fields]) AND ("data collection"[All Fields] OR "primarydata"[All Fields] OR "data-collection"[All Fields] OR "data gathering"[All Fields] OR"data-gathering"[All Fields] OR "survey"[All Fields])

RESULTS

Katerndahl (2012) demonstrated a high response rate (100%) whenperforming data collection along with the institutional annual report that allmembers of the institution needed to fill. This strategy also guaranteed thelongitudinal data collection for 13 years. Kossinets (2006) states that a responserate between 50% to 70% is acceptable, since it is unlikely to affect the test results(Grosser, Lopez-Kidwell, & LaBianca, 2010). Although this is a common problem formany network analysis studies with primary data collection, there is no consensualsolution. The solutions ranging from substitution with symmetrical knots(Huisman, 2009; Steglich & Huisman, 2008; Stork & Richards, 1992) to more robustmethods for Bayesian inference (Butts, 2003).

There are different types of collaboration that must be evaluatedseparately. Like the study of Okamoto et al (2015) which considered six types ofcollaboration: study or research protocol, co-authored publication, co-authoredpresentation, mentoring or training, committee / working group and others. Thecategory "other" was included for quality control purposes in order to get a generalmeasure of global collaboration. This study also considered the current andprevious collaborations network formation. Mays et al (2013) brings another typeof collaboration that can be interesting to measure at REBRATS, the collaboration inthe implementation / translation of evidence into decision making.

METHODS RESULTS

RESULTS

Eight studies were reviewed in search of data for: purpose of thestudy, data collection instrument, collection strategy, collection period andresponse rate. It should be noted that there were no studies focusing on healthtechnology assessment network, which highlights the pioneering spirit of Brazil inthis area.

Most studies used online data collection, with an email call. It was also heldcollection by telephone and by letter, especially to encourage those who had notyet responded via email. The data collection period was short varying from 2 to 6months for most studies. During the data collection period, those who had not yetresponded also received an e-mail reminder. Another strategy that seems to havebeen effective was the request to the directors of different centers to stimulateother researchers of the centers to answer the questionnaires.

A tool to monitor and evaluate a HTA network: the case of REBRATS

Leão,L.S.C.1; Vanni,T.1

1 Department of Science and Technology, Brazilian Ministry of Health

Based on these data, an online survey strategy was developed. The call willbe done by e-mail to all members of the network. The form will be available to fillout for a month. After this time, members who have not sent their response will berecollected by email and phone and given a deadline of five working days. Data willbe collected annually.

As part of the implementation strategy, it is that the participation on thesurvey is mandatory to all institutions members of REBRATS, and that it wouldserve as a network member certification. It would be a sine qua non condition forthe permanence of the institution on the network on the following year.

For this strategy to be implemented, the inclusion of this new obligation on theinternal network regiment is required. This proposal must be submitted andapproved by the Executive Committee of REBRATS.

Data Analysis

Socioeconomic data will be analyzed using STATA / IC 12.1 for Windows(StataCorp LP, Texas, USA). Confidence intervals for discrete variables ratios will becalculated using the logit transformation, so that the end points are situatedbetween 0 and 1. Common confidence intervals will be used to mean thecontinuous variables. The geographical origin and distribution of participants willbe plotted using ArcGIS Desktop 10 (ESRI, Redlands, USA). The social network datawill be organized using a program developed in C ++ to reorganize the data in amixing matrix. This data is then analyzed in Gephi, which is an open-sourcenetwork analysis software for viewing and operation of networks and complexsystems.

CONCLUSIONS

Survey Development

Based on the information collected in the eight selected studies and also based onthe specific characteristics of REBRATS, the survey called "Annual Report of theactivities of the REBRATS member institutions" was elaborated. The survey has thefollowing sections:

1. General Information

Institution’s name, address, e-mail, phone, expertise, kind of studies developed (ex.systematic review). Team contacts and curriculum.

2. Studies Production and Dissemination

This session aims to quantify the production of studies and identify which wereconducted in collaboration with another network member institution. It also hasspecific questions about the use of the study, as “This study has been used toinform a decision making? What kind of decision?”

3. Participation at network activities

In this section the institution must give details of its participation at networkactivities during the year, such as working groups meetings, events and trainings. 4.4. Activities Promoted by the Institution

In this session the institution should report the HTA activities developed during theyear, like trainings, workshops, etc. It should also inform if the activity was donewith the collaboration of another network member.

5. Final Comments

This section is free for general comments.

RESULTS RESULTS

A tool to monitor and evaluate a HTA network: the case of REBRATS.

Leão,L.S.C.1; Vanni,T.1

1 Department of Science and Technology, Brazilian Ministry of Health

REFERENCES

1. Scientific collaboration and team science: a social network analysis of the centers for population health and health disparities. Okamoto J; Centers for Population Healthand Health Disparities Evaluation Working Group.Transl Behav Med. 2015 Mar;5(1):12-23. doi: 10.1007/s13142-014-0280-1.2.

2. Research collaboration in the discovery, development, and delivery networks of a statewide cancer coalition. Provan KG, Leischow SJ, Keagy J, Nodora J. Eval ProgramPlann. 2010 Nov;33(4):349-55. doi: 10.1016/j.evalprogplan.2009.12.005. Epub 2009 Dec 16.

3. Identifying emerging research collaborations and networks: method development. Dozier AM, Martina CA, O'Dell NL, Fogg TT, Lurie SJ, Rubinstein EP, Pearson TA. EvalHealth Prof. 2014 Mar;37(1):19-32. doi: 10.1177/0163278713501693. Epub 2013 Sep 8.

4. Implications of network structure on public health collaboratives. Retrum JH, Chapman CL, Varda DM. Health Educ Behav. 2013 Oct;40(1 Suppl):13S-23S. doi:10.1177/1090198113492759.

5. Co-evolution of departmental research collaboration and scholarly outcomes.6. Katerndahl D. J Eval Clin Pract. 2012 Dec;18(6):1241-7. doi: 10.1111/j.1365-2753.2012.01881.x. Epub 2012 Jul 9.7. Engaging Public Health Settings in Research Implementation and Translation Activities: Evidence from Practice-Based Research Networks. Mays GP, Hogg RA, Castellanos-

Cruz DM, Hoover AG, Fowler LC. Am J Prev Med. 2013 Dec;45(6):752-62. doi: 10.1016/j.amepre.2013.08.011.8. Leadership in complex networks: the importance of network position and strategic action in a translational cancer research network. Long JC, Cunningham FC, Wiley J,

Carswell P, Braithwaite J. Implement Sci. 2013 Oct 11;8:122. doi: 10.1186/1748-5908-8-122.9. A comparative study of 11 local health department organizational networks. Merrill J, Keeling JW, Carley KM. J Public Health Manag Pract. 2010 Nov-Dec;16(6):564-76.

doi: 10.1097/PHH.0b013e3181e31cee.10. Newman MEJ, Barabási A-L, Watts DJ (2006) The Structure and Dynamics of Networks: Princeton University Press.11. De Nooy W, Mrvar A, Batageli V (2011) Exploratory Social Network Analysis with Pajek: Cambridge University Press.12. Easley D, Kleinberg J (2010) Networks, Crowds, and Markets: Reasoning About a Highly Connected World: Cambridge University Press.13. Brandes U (2001) A faster algorithm for betweenness centrality. The Journal of Mathematical Sociology 25: 163-177.14. Page L, Brin S, Motwani R, Winograd T (1998) The PageRank Citation Ranking: Bringing Order to the Web.