nefis (wp5) evaluation meeting, 15-16 november 2004 evaluation metadata aljoscha requardt,...

Aljoscha Requardt, University of HamburgNEFIS (WP5) Evaluation Meeting, 15-16 November 2004

Evaluation Metadata

Response rate:

93% (14 of 15 partners send questionnaire back)

Response rate per question varies between 70-100%

Comments of “no response“ were:

This does not concern us as data provider

Question not applicable

Answers not significantly influenced by data types!

33%

17%22%

28%

NFI data

Geospatial data

Social and Economical Statistics

other forest and environmentalstatistics

Varity of data types (37 datasets):


Evaluation Metadata

21%

79%

yes

no

Are the NEFIS metadata the first contact you have had with metadata?


Evaluation Metadata

0

1

2

3

4

5

6

7

excellent higher thanaverage

average lower thanaverage

no experience

(nr)

0

1

2

3

4

5

6

7

excellent higher thanaverage

average lower thanaverage

no experience

(nr)

Partners knowledge and experience of elaborating metadata

Partners knowledge and experience using metadata for data source retrieval


Evaluation Metadata

8 of 14 partners are using tools to create metadata:

• Text editor

• Three Tab Metadata/ Arc Catalog

• Other…?

46%

54%

no

yes

Used standards or schema are:

• Dublin Core

• ISO/TC211, USGS

• guidelines compiled by Statistics Finland (in line with EUROSTAT instructions)

• FoxPro, Oracle

Do you use any metadata standard or schema to describe and catalogue your datasets?


Evaluation Metadata

Preparation phase I Studying documentation (metadata schema, metadata guidelines, metadata template)

Preparation phase II Preliminary version of metadata records

Completion phaseEntering final version of metadata records

How much time did you spend to enter the metadata records?


Evaluation Metadata

Preparation phase 1

Preparation phase 2

Completion phase 3 Total

P1 4 11 15

P2 4 2 1 7

P3 15 10 5 30

P4 3 2 0,5 5,5

P5 8 20 8 36

P6 4 3 3 10

P7 2 3 2 7

P8 8 8 4 20

P9 4 4 6 14

P10 1,5 1,5 0,5 3,5

P11 2 2 2 6

P12 3 1,5 0,5 5

P13 3 2 0,5 5,5

P14 25 10 5 40

Max 25 20 8 40

Min 1,5 1,5 0,5 3,5

Average 6,2 5,7 2,9 14,6


Evaluation Metadata

0

5

10

15

20

25

30

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Partner

Wor

kloa

d (h

ours

)

Preparationphase 1

Preparationphase 2

Preparationphase 3

Workload to enter metadata records


Evaluation Metadata

Workload to enter metadata records (Cluster)

0

2

4

6

8

10

12

0,5-5 6-10 11-15 16-20 21-30

(hours)

(nr)

phase 1

phase 2

phase 3


Evaluation Metadata

most partners between

5-10 hours

0

5

10

15

20

25

30

35

40

45

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Partner

Wor

kloa

d (h

ours

)

Workload to enter metadata records (Total)


Evaluation Metadata

0

2

4

6

8

10

12

very high high acceptable low very low

(nr)

Level of complexity to prepare and enter metadata: 29% = complex

71% = acceptable

Do you consider the workload to enter metadata records as:


Evaluation Metadata

• NEFIS Terms could include more structured levels (hierarchy)

• NEFIS Terms could be revised to improve the usability and standardize the terms between the different lists.

• as soon as this version of metadata preparation is implemented in a real Internet environment, drop down menus would reduce the time of data entry substantially.

• the different provided lists were not easy to use and data entry was neither easy, …some efficient tools…for preparing and entering metadata would be useful.

Suggestions for improving metadata elaboration:


Evaluation Metadata

0

1

2

3

4

5

6

very easy easy sufficient difficult very difficult

(nr)

Are the guidelines easy to understand?


Evaluation Metadata

0

1

2

3

4

5

6

7

8

9

10

very good good sufficient deficient inadequate

(nr)

0

1

2

3

4

5

6

7

very good good sufficient deficient inadequate

(nr)

• It is very difficult for us to judge this from the data user's point of view.

Is the current NEFIS metadata schema operational and applicable for your use as a data provider?

Is the current NEFIS metadata schema functional and applicable for a data user - for the purpose of data retrieval?


Evaluation Metadata

71%

29%

yes

no

• Yes, necessary even !!

• No, but there should be a contact person that I could ask if questions arise

• No, but in the case of individuals not familiar with metadata YES

• a separate meeting would bee too much, but a session in a meeting that takes place anyhow would be helpful

Would a metadata tutorial or training course be helpful and appropriate?


Evaluation Metadata

Metadata schema – evaluation questionnaire matrix


Evaluation Metadata

0

2

4

6

8

10

12

14

1. T

itle

2. C

reat

or

3. S

ubje

ct

4. D

escr

iptio

n

5. P

ublis

her

6. C

ontri

buto

r

7. D

ate

8. T

ype

9. F

orm

at

10. I

dent

ifier

11. S

ourc

e

12. L

angu

age

13. R

elat

ion

14. C

over

age

15. R

ight

s

16. A

udie

nce

(nr)

Metadata records without problems


Evaluation Metadata

0

1

2

3

4

5

6

(nr)

Problem of usability and functionality in general


Evaluation Metadata

0

1

2

3

4

5

6

7

8

(nr)

Problem of understanding


Evaluation Metadata

0%

20%

40%

60%

80%

100%

high

acceptable

low

Workload: time to prepare and enter required information


Evaluation Metadata

0

1

2

3

4

5

(nr)

Too much information is required


Evaluation Metadata

0

1

2

3

4

(nr)

Element should be deleted


Evaluation Metadata

0

1

2

3

4

5

6

7

8

9

(nr)

Should be mandatory


Evaluation Metadata

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

not achieved to some extent achieved

Standardisation for better interoperability achieved?


Evaluation Metadata

0%

20%

40%

60%

80%

100%

not relevant low relevance relevant high relevance

Relevance - data and resource documentation


Evaluation Metadata

0%

20%

40%

60%

80%

100%

not relevant low relevance relevant high relevance

Relevance - data retrieval


Evaluation Metadata

Metadata – specific issues (elements/ refinements)

• Type

• Format

• Coverage

• Quality Report

• Subject


Evaluation Metadata

0

1

2

3

4

5

6

7

8

9

yes to some extent- sufficient

to some extent-defcient

no

(nr)

• It might not be completely clear to everyone what georeferenced actually means. This could be more explicitly stated in the guidelines.

• The Dataset georeferenced description seems to be too much accurate for the purpose of this project

Type: Clear differentiation DatasetGeoreferenced/ Dataset?


Evaluation Metadata

• It gives the user valuable additional information, in particular if the user is interested in utilising the DatasetGeoreferenced for work.

• I am not sure that all georeferenced datasets can be described by the current version. …longitude and latitude (as decimal degrees or in degrees/minutes/seconds) are not provided as variables.

Format: Is the "reference system encoding scheme" an appropriate addition to the NEFIS metadata schema?

0

1

2

3

4

5

6

7

appropiate to some extent -sufficient

to some extent -deficient

not appropiate

(nr)


Evaluation Metadata

• It could be better to have the reference system within the element “coverage”

• It could actually be an own “Element”

83%

17%

yes

no

Is "format" the appropriate place for "reference system"?


Evaluation Metadata

Is Point encoding scheme an appropriate addition?

Is Box encoding scheme an appropriate addition?

0

1

2

3

4

5

6

7


to some extent - deficient

not appropiate

(nr)

0

1

2

3

4

5

6

7

8



not appropiate

(nr)

3 times no answer

4 times no answer


Evaluation Metadata

60%

40%yes

no

• Would be helpful for more harmonisation (user end), but may cause difficulties

• Yes, but we do not have to use this. It would mean less work for those trying to locate information from an EFIS…but it might mean that data providers have to convert their data sets.

• The purpose is mainly to provide information on where to find the data - not to harmonise the data itself, and a predefined coordinate system could require much work for some datasets.

• Metadata includes map-projection definitions (RT 90)

Should the coordinate system be predefined?

4 times no answer


Evaluation Metadata

0

1

2

3

4

5

6

7

8

fully addressed addressed to someextent - sufficient

addressed to someextent - deficient

not addressed

(nr)

Named elements which address “quality” within the DCMI schema:

Creator, Description, Publisher, Coverage, Source, Date, Audience

"Quality" addressed within DCMI elements, refinements and encoding schemes


Evaluation Metadata

0

1

2

3

4

5

6

7

8

9

10

very important important interesting,but not

necessary

not important

(nr)

0

2

4

6

8

10

12

14

yes no - qualityunder theelement...

no - extrametadataelement

no - no extraquality report

(nr)

Relative value of a “quality report“

“quality report“ under the element “descriptions“?


Evaluation Metadata

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Collection Mandate

Availability of datacollection and data

processing guidelines

Defintions

Sampling methods

Explanatory notes

very important important interesting, but not necessary not important

• The question here is: Which of these options is appropriate for which type of data?

Value of listed options to describe and structure "quality report"


Evaluation Metadata

• the question is: Which user can use this information correctly?

• maybe applicable to some types of datasets but such information could also be included in the “quality report” description

• it could become very complicated (and time consuming for data providers) to collect such information in a standardised way.

Value of quantitative measures of quality (e.g. standard error, sample size, sampling unit, resampling for measurement control)

0

1

2

3

4

5

6



not appropiate

(nr)


Evaluation Metadata

• they are different: some are detailed, others are of more general nature. Some harmonisation is needed.

• sometimes difficult to group the term under a theme

• I see some need for further elaboration and specification. In particular concerning the themes: ‘research’, ‘forestry institutions’ and ‘economics’.

•It will be important to review the list of NEFIS themes and their definitions.

Are the definitions for the NEFIS Themes appropriate?

0

1

2

3

4

5

6

7

8

9



not appropiate

(nr)


Evaluation Metadata

• sometimes difficult to group the term under a theme

• very time consuming to go through the lists … more structure would be helpful. Question of balance between accuracy of description and time investment becomes apparent.

• some overlap of NEFIS terms can be found. The level of detail and the accuracy of the terms relating to a resource will always be an issue which will be nearly impossible to solve to everyone’s satisfaction.

Can the dataset be appropriately described by using the NEFIS terms?

0

1

2

3

4

5

6

7



not appropiate

(nr)


Evaluation Metadata

0

1

2

3

4

5

6

7

8

very important important interesting, butnot necessary

not important

(nr)

• it could be the way to improve the term lists

• it’s very important during the development phase

• nominated terms are really needed; in some cases they are more relevant than NEFIS terms

•…it allows the provider to demonstrate which additional terms are seen necessary to describe the particular resource more accurately

• …it is important….but who will be the editorial board?

How important is the option to add "Nominated Terms"


Evaluation Metadata

0

1

2

3

4

5

6


not important

• This element is in particular important for the data provider. The provider should have a clear vision on the addressees of the information resource…

• The data provider can not evaluate the class of entity to whom the resource is indented or useful. It is too subjective.

Is the element "audience" an important addition?


Evaluation Metadata

Metadata Element

1. Title

2. Creator

3. Subject

4. Description

5. Publisher

6. Contributor

7. Date

8. Type

9. Format

10. Identifier

11. Source

12. Language

13. Relation

14. Coverage

15. Rights

16. Audience

Metadata Schema - Resume

= without problems

? = problems (hot spots)

? = with some problems?

?

?

?

?

?

?

?

?

?


Evaluation Metadata

0

1

2

3

4

5

6

7


not important

(nr)

Perspective: establish a metadata working group?


Evaluation Metadata

Discussion

nefis (wp5) evaluation meeting, 15-16 november 2004 evaluation metadata aljoscha requardt,...

Documents