universitat de barcelona facultat de biblioteconomia i documentació grau d’informació i...

UNIVERSITAT DE BARCELONAFacultat de Biblioteconomia i Documentació

Grau d’Informació i DocumentacióResearch Methods

Non-experimental designs

Professor: Ángel Borrego


Longitudinal studies

Content analysis

Delphi studies

Bibliometrics

Qualitative research


Research on a sample on more than one occasion over a period of time

Types of longitudinal studies:• Trend• Cohort• Panel

Longitudinal trend design

• Deals with changes in a population.

• The whole population (or a representative sample) is observed.

Time 1 Time 2 Time 3 Time n

Longitudinal trend design

Longitudinal cohort design

A cohort is a group of people who share a characteristic, usually age.


Cohort is observed

Cohort is observed

Cohort is observed

Cohort is observed

Longitudinal panel design

The same set of subjects is always observed.


Longitudinal panel design

http://www.kantarmedia1.es/sections/product/panel-audiencia




Delphi studies

Content analysis

Bibliometrics

Log analysis


Content analysis

Content analysis is the quantitative study of the features of recorded information.

It consists in classifying textual (or visual) material reducing it to more relevant and manageable categories.

That reduction leads to the calculation of descriptive statistics that summarize or characterize datasets.

Content analysis is essentially a coding operation. Coding is the process of transforming raw data into a standardized form.

Examples of content analysis in LIS

• Analysis of job advertisements to describe the library job market.

• Analysis of the use of Twitter by libraries to understand how to use social media successfully.

• Analysis of reference questions posed by library users in order to improve the reference service.

Steps in content analysis1) Formulate your research problem: define questions or research

aims.

2) Identify the contents to analyse (population or sample) and a chronological framework.

3) Define the variables precisely: use exhaustive and mutually exclusive categories. Create a book of codes and an analysis grid.

4) Codify the contents.

5) Apply a test of reliability: guaranties that conclusions are independent of the coder. At least 10% of the sample should be codified independently by two coders to reach an agreement of at least 80%.

Library job advertisements

• How many library positions are being offered in academic libraries?

• What qualifications are needed to entry the job market?

• What language and computer skills are required?

• What salaries are being offered?

• What new positions are emerging?


http://www.jobs.ac.uk

http://www.jobs.ac.uk/

http://www.jobs.ac.uk/

Use of Twitter by public libraries

• What do public libraries tweet about? What types of tweets are posted by academic libraries?

• What makes a tweet useful (measured by the number of retweets and favorites received)?

Use of Twitter by public libraries

Reference questions

http://www.pregunte.es

http://www.pregunte.es/

http://www.pregunte.es/

Reference questions

• What subjects do users ask about (computers, health, travel, etc.)?

• Do users ask for sources or for factual data?

• How long does it take to respond?

• How many questions are rejected? Why are they rejected? (unclear, not appropriate, erroneous email address, etc.)?

• How can we improve the service?

Test of reliability

Example of a reference question:

Hello, I am searching for information about the León Cathedral: architecture, history, opening hours...

How would you classify this question?

Art Travel

Intercoder reliability

Two independent analysts have classified ten reference questions. As you can see there are agreements and disagreements between them. Analyst A Analyst BQuestion 1 History HistoryQuestion 2 Travel TravelQuestion 3 Education LanguagesQuestion 4 History HistoryQuestion 5 Science ScienceQuestion 6 Languages LanguagesQuestion 7 Employment EmploymentQuestion 8 Law LawQuestion 9 Philosophy ReligionQuestion 10 Languages Languages

Intercoder reliability

We can calculate the degree of agreement (DA) between both analysists using the following formula:

where A is the number of agreements and n the number of questions.



Content analysis

Delphi studies

Bibliometrics


Delphi studies• It aims to obtain consensus of opinion of a group of experts by a

series of questionnaires.

• Supports planning, allocation of resources and decision making.

Development Delphi study (I)

• Selection of experts.

• First questionnaire: open-ended questions asking for a list of opinions, predictions, recommended activities, etc.

Example: evaluation of institutional repositories

• The aim of the research is to develop a set of evaluation indicators for institutional repositories in academic libraries.

• Experts: 25 directors of academic libraries.

• First questionnaire: open-ended question asking for a list of indicators to evaluate these repositories.

• Researchers identify and group together items from the experts’ replies.

Evaluation of institutional repositories: replies to the first questionnaire

Number of documents uploaded Researchers with uploaded documents

Documents published in the last 3 years Documents downloaded

Link from library homepage to repository Number of users

Metadata completeness System failures per month

Integrated search of OPAC and repository Number of users’ training sessions

Document types (articles, dissertations…) User satisfaction

Cost per document uploaded Thematic diversity of the collection

Development Delphi study (II)

• Second questionnaire: a copy of the collective list is sent to each expert. They are asked to rate or evaluate each item according to its importance.

Evaluation of institutional repositories: second questionnaire

Less Moreimportant important

Number of documents uploaded 1 2 3 4 5

Documents published in the last 3 years 1 2 3 4 5

Link from library homepage to repository 1 2 3 4 5

Metadata completeness 1 2 3 4 5

Integrated search of OPAC and repository 1 2 3 4 5

Document types (articles, dissertations…) 1 2 3 4 5

Cost per document uploaded 1 2 3 4 5

Researchers with uploaded documents 1 2 3 4 5

Documents downloaded 1 2 3 4 5

Number of users 1 2 3 4 5

System failures per month 1 2 3 4 5

Number of users’ training sessions 1 2 3 4 5

User satisfaction 1 2 3 4 5

Thematic diversity of the collection 1 2 3 4 5

Evaluation of institutional repositories: results to second questionnaire

AverageNumber of documents uploaded 4,8Thematic diversity of the collection 4,6Cost per document uploaded 4,5Researchers with uploaded documents 4,5User satisfaction 4,0Metadata completeness 3,8Integrated search of OPAC and repository 3,6Document types (articles, dissertations…) 3,5Link from library homepage to repository 3,5Documents downloaded 3,3Number of users 3,2System failures per month 3,2Number of users’ training sessions 3,0Documents published in the last 3 years 3,0

Development Delphi study (III)

• Third questionnaire: includes the list and the average ratings. The experts are asked to either revise their opinions or discuss their reasons for not coming to consensus with the group. If they disagree with the general opinion they provide explanations for their ratings.

Evaluation of institutional repositories: third questionnaire

“I don’t agree with the high rating for the “thematic diversity of the collection”. Our University specializes in engineering, so most of the documents in our repository fall in this discipline. We won’t have any documents in Social Sciences or Humanities. It doesn’t make much sense to measure the thematic diversity of the documents in our institution.”

Expert #7

“Just measuring the number of researchers with uploaded documents is not informative. This indicator benefits big institutions with a large number of researchers. We would have to measure the percentage of researchers with uploaded documents”.

Expert #12

“User satisfaction is interesting… but how are we going to measure it? We already do a lot of surveys among our users and the response rate is quite low since users are tired of replying to questionnaires. I think conducting another survey is unwise.”

Expert #17

Evaluation of institutional repositories: results

Category Item Performance indicator

ContentCurrency No. documents published in last 3 years

Metadata Metadata completeness

ManagementBudgeting Cost per document

Staffing IR librarians (% of total)

System and networkIntegration (OPAC) Integrated search of OPAC and IR

IR homepage Link from library homepage to IR

Use and UsersUse rate No. of downloaded documents

Support for users No. of IR training sessions

Advantages of Delphi studies

• Anonymity of participants.

• Heterogeneity of participants.

• Controlled feedback: it allows interaction but reducing discord among panel members.

• More individuals can participate than those that can effectively interact in a face-to-face exchange.

Limitations of Delphi studies

• Judgments are those of a select group of people and may not be representative.

• Tendency to eliminate extreme positions and force a middle-of-the-road consensus.

• Time consuming, requires participant commitment (about 30 to 45 days to complete the entire process).


Delphi studies

Content analysis

Bibliometrics

Log analysis


Bibliometrics

Bibliometrics is the application of mathematical methods to the study of information products (books, journals, dissertations, etc.).

Typical bibliometric variables include: authorship; country of publication; language; journal titles; age; etc.

BibliometricsA common approach in bibliometrics is citation analysis, which involves examining the reference lists of documents.

In academic libraries, it is usual to analyse faculty publications or students’ dissertations to determine types and number of documents cited, languages, age, changes in citation patters over time, etc.

Applications of bibliometric studies

• Identification of core literatures: enables a library or a database producer to identify the most important journals that need to be included in a collection.

•Estimation of completeness of a library collection.

•Decisions regarding retention of items in a collection: it is possible to use an obsolescence study to predict when an item will no longer be likely to attract a meaningful level of use.

•Research assessment: evaluate the performance of authors, institutions, journals, etc.


Delphi studies

Content analysis

Bibliometrics

Log analysis



Qualitative research emphasizes the importance of people’s views or behaviours, it seeks perceptions and opinions.

Qualitative research involves a small number of participants who are addressed open ended questions, without preconceived categories of responses.

Data analysis consists in repeatedly reviewing data to develop patterns and reconsider them. Data are organized into general themes.

Qualitative results often are not generalizable. This differs from quantitative research results, where statistical processing of data creates generalizable projections from the sample to the population it represents.

Bronisław Malinowski (1884–1942) was one of the most important 20th-century anthropologists and ethnographers.

Quantitative vs Qualitative research