tsgs sa #58, barcelona 1 © 3gpp 2012 annex to sp-120885 3gpp tsg sa meeting #58, barcelona, spain,...

61
TSGs SA #58, Barcelona 1 © 3GPP 2012 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis John M Meredith Director, ETSI Mobile Competence Centre 3GPP Specifications Manager

Upload: sonny-silk

Post on 01-Apr-2015

216 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 1

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

3GPP satisfaction survey, 2012

Analysis

John M MeredithDirector, ETSI Mobile Competence Centre

3GPP Specifications Manager

Page 2: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 2

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

IntroductionIn late 2012 MCC conducted a survey of the satisfaction of delegates and users of the 3GPP web site and of the services offered by MCC.

There were:

236 (79) responses from delegates (of which about 10% were chairmen or vice-chairmen) representing a reasonable spread amongst the TSGs and WGs indicated in the chart* …

and

173 (89) non-delegate responses

* Respondents could choose more than one TSG/WG.

Throughout this presentation, where similar questions were asked in the 2010 survey, the corresponding scores are shown in gray italic font.

“Instant” MCC reactions offered in blue font.

Page 3: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 3

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 9 (delegates):Which TSGs or WGs do you regularly attend?

Average score: (2010 survey)delegates: 3.4 (3.2)non-delegates: 3.3 (3.2)

TSG CT, 7.8%CT WG1, 6.9%

CT WG3, 4.7%

CT WG4, 4.7%

CT WG6, 3.0%TSG GERAN, 3.9%

GERAN WG1, 3.0%

GERAN WG2, 3.9%

GERAN WG3, 3.4%

TSG RAN, 13.8%

RAN WG1, 12.9%

RAN WG2, 12.9%RAN WG3, 8.6%

RAN WG4, 19.4%

RAN WG5, 8.6%

RAN WG AH1 ITU, 1.3%

TSG SA, 13.4%

SA WG1, 11.2%

SA WG2, 13.4%

SA WG3, 4.7%

SA WG4, 8.6%

SA WG5, 3.4%

(2010 survey)

Page 4: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 4

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 8 (delegates):What is your status?

Elected Official, 9.5%

Delegate, 90.5%

Responders were asked to identify themselves as either delegates (including chairmen and vice-chairmen) or as 3GPP non-participants.

Most questions were rated1 (very poor) to 5 (very good).The scores are shown graphically, and the mean values given.

Page 5: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 5

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Executive summary (1)

• What is good?– Up to date web site content– Web site overall, including ease of navigation– MCC service

What is bad?– Finding information on the web site (not involving the search engines)– The advanced search engine– TDoc allocation methods / tools

Overall score out of 5:

Average of averages: 3.5 (3.5)

No question scored less than the median of possible scores, 2.5.

Page 6: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 6

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Executive summary (2)

• What are recurrent comments ?– The search engine:

» too difficult to restrict the search to find, for example, a single Tdoc

– Too many clicks on the web site needed to reach the desired page.– TDoc number allocation method varies from group to group

» Manual allocation suffers from bottlenecks at busy times» Automatic allocation (ADN application) not really fit for purpose» Need local server based application during meetings, to relieve secretary of this burden

– Need improved method to identify (group of) Specs sought

Page 7: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 7

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Detailed results …

Page 8: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 8

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 1:How easy is it to navigate our web site?

0

5

10

15

20

25

30

35

40

45

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Average score: (2010 survey)delegates: 3.4 (3.2)non-delegates: 3.3 (3.2)

Page 9: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 9

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 1 – non-delegate remarks:

• Logical structure, easy to navigate.

• Not easy to find (U)SIM test cases.

• Too many clicks needed to reach documents.

• Button icons on banner are not in fact buttons!

• Links labelled “click here” not descriptive enough.

• Good in general, but with some inconsistencies.

• Search engine useless.

• Nice layout, easy to find what I was looking for.

• Inconsistent spec series per responsible WG.

• FTP platform is a nightmare.

• I like the style and design.

• Poor design and layout, poor colour scheme.

• Work plan pages not easy to navigate, very wide pages.

How easy is it to navigate our web site?

Page 10: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 10

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 2:How easy is it to find the information you are looking for on our web site?

Average score:delegates: 3.1 non-delegates: 3.0

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: Includes use of search engines?

Page 11: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 11

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 2 – non-delegate remarks:

• Search results not always very useful.

• Easy to find specs via numbering plan.

• Difficult to identify a spec if you don’t know the number.

• Search using Google better than 3GPP site search.

• Need summaries with real time aspects.

• Feature list per Release is vague and doesn’t help determine the improvements actually achieved from Release to Release.

• Needs more explanation of terminology (eg Stages 1, 2, 3).

• Keywords.

• Need to be able to search by date, and within databases (specs, TDocs).The advanced search engine provides some of this; too difficult to use?

• Where to start? Help!

• Layer 1 specs use Word’s Equation Editor, which makes searches difficult.

How easy is it to find the information you are looking for on our web site?

Page 12: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 12

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 3:How visually appealing is our web site?

Average score: (2010 survey)delegates: 3.2 (3.7)non-delegates: 3.2 (3.5)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: Unchanged since last survey. New look needed?

Page 13: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 13

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 3 – non-delegate remarks:

• Banner image of doctor using her tablet may confuse end users; not intuitive.

• Very nice, appealing.

• Unattractive, geeky.

• Needs some Flash video.

• Simple and explanatory.

• Not interested in visual appearance.

• Need more variety of images on page banners.

How visually appealing is our web site?

Page 14: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 14

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 4:How up-to-date is the content on our web site?

Average score: (2010 survey)delegates: 3.6 non-delegates: 4.6

05

101520253035404550556065707580859095

100

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 15: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 15

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 4 – non-delegate remarks:

• Seems to be up-to-date.

• Yes, I am only interested in accessing published standards.

• As a technical writer, I am mildly impressed by the quality of the MS Word templates.

How up-to-date is the content on our web site?

Page 16: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 16

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 5:How easy is it to access and to use the specification database?

Average score: (2010 survey)delegates: 3.4 (3.2)non-delegates: 3.3

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 17: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 17

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 5 – non-delegate remarks:How easy is it to access and to use the specification database?

• Finding particular versions of specs is easy, but not to find the rationale behind choice of technologies.

• Finding a particular spec is difficult, especially if you don’t know its number.

• Need graphical hierarchy of specs, with possibility for block downloads by technology area.

• No centralized WG document list per meeting, hence difficult to trace the WG discussions.These do exist, of course, but maybe not easy enough to find?

• Specs to have an index referring to contributing TDocs and meeting minutes.

• Web site not structured to allow a view of the whole picture.

Page 18: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 18

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 6 (delegates):How satisfied are you with the information about membership?

Average score: (2010 survey)delegates: 3.3 (3.0)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

54

Page 19: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 19

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 6 (non-delegates):If you visited the web site seeking membership information, did you find what you needed?

not applicable, 59%

no, 14%

yes, 27%

Page 20: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 20

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 7:How likely are you to recommend our web site to others?

Average score: delegates: 3.7 non-delegates: 3.9

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 21: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 21

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 7 – delegate remarks - 1: How likely are you to recommend our web site to others? Point out any missing features or weaknesses.

• Would like to be able to use site on a smartphone.

• Would like direct links to WGs (eg for incoming LSs).

• Handling of mailing lists is very confusing. Why two different pages for managing subscriptions?

• Can’t get to the ETSI meeting calendar home page directly from the main menu.

• Don’t hide TDocs inside zip files, prevents Google search.(But doesn’t prevent 3GPP search!)

• News items are too European. Consider other regions.

• Not clear who are the stakeholders in 3GPP. This is revealed on the Partners page; too cryptic?

• Too many clicks needed to reach your target page.

• Need instructions how to use the FTP site.

• Links on work items per WG pages are to full work plan page, which takes too long to load; would be better to a page for that one work item.

Page 22: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 22

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 7 – delegate remarks - 2: How likely are you to recommend our web site to others? Point out any missing features or weaknesses.

• Need electronic handling of TDocs: checking out/in; numbering, allocating to agenda items, co-signing, revising, etc. Present system is outdated.

• Not only are technical terms difficult to deduce from abbreviations, but also administrative terms like TSG, WG, CR.

• Should have a delegate feedback survey for every meeting.This is already planned, in response to the PCG ahg on IT Improvement

recommendations.

• Workplan still difficult to navigate.

• Some key pages difficult to find.

• Difficult to find 3GPP procedural documents.

• Very poor design, links hard to find.

• No phone numbers for MCC personnel.Of course, these are given; too hard to find?

Page 23: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 23

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 7 – non-delegate remarks:How likely are you to recommend our web site to others? Point out any missing features or weaknesses.

• Would recommend to engineers working in this field.

• I use it every day, it is critical to my work.

• Fast download.

• Good site for the latest technology updates for LTE.

• Beginners find the site difficult to navigate, keep it simple.

• I want to know the member organizations of 3GPP.

Page 24: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 24

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 8 (non-delegates) - 1:Can you point out any missing features or any weaknesses on the 3GPP web site?

• Can find what but not why.

• Poor search engine. Poor, or poorly understood, thus poorly explained?

• Search tool to go directly to a particular Spec page, or Tdoc.

• Web site needs massive index to find relevant spec easier.It has one, and it drives the advanced search engine!

• Powerful, self-learning, search tool like Google.

• Spec numbering scheme not a very good guide to Spec contents.

• Long specs should have an executive summary.

• No FAQ page.Actually, failed to find FAQ page; because not easy to find?

• Too structured, not sufficiently results-oriented.

Page 25: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 25

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 8 (non-delegates) - 2:Can you point out any missing features or any weaknesses on the 3GPP web site?

• Provide specs in PDF.

• Provide specs in HTML.

• Provide implementation examples.

• More technology tutorials, by RAT, by Release; with message flows.

• No link between specs and contributing TDocs.Perhaps they don’t understand the principle of CRs … ?

• Mapping of specs to network architecture.

• Need guide to spec series numbering.There is, of course, exactly this. Too difficult to find?

• Make white papers available in PDF.

• Smartphone rankings.Seek elsewhere!

Page 26: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 26

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 8 (non-delegates) - 3:Can you point out any missing features or any weaknesses on the 3GPP web site?

• Need overview of architecture, protocols, procedures, which points to corresponding specs; maybe graphical interface*.• Graphical interface to (all) CRs.• Grouping of specs by function (eg Charging).

More rigorous use of keywords?• Grouping of CRs by function.• Need link on home page to technology areas.

There is one, from main menu!• Need link to GSMA site, collaboration is important.• Selective download of TDocs by subject area, agenda item, duration of discussion.• CRs should indicate which RAT they apply to.• Pointers to external press articles relating to 3GPP technologies.• Explanation of why some specs sometimes not available.• Add TS and TR number to the Release Description documents.• Need a high level overview page, describing what 3GPP is.

Is the “About” page deficient?• Need different areas of web site targeted at different audiences.• Better grouping of work items into categories such as “most challenging”.

* Commercial offering proposed!

Page 27: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 27

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 10 (delegates):How satisfied are you with the following on-line services …

Page 28: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 28

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 10a:E-mail exploder list management?

Average score: (2010 survey)delegates: 3.7 (4.1)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: The e-mail exploder facilities are unchanged. Delegates expectations have evidently increased.

Page 29: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 29

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 10b:Specification Group pages?

Average score: (2010 survey)delegates: 3.5 (3.8)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: These pages are almost unchanged in content type and format. Delegates expectations have increased.

Page 30: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 30

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 10c:Meeting information?

Average score: (2010 survey)delegates: 3.6 (3.9)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: These pages are unchanged in content type and format. Delegates expectations have increased.

Page 31: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 31

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 10 – delegate remarks:How satisfied are you with … on-line services?

• Can’t manage exploder list subscriptions due to technical issue in my company.Possibly due to asymmetric addresses (different rx & tx).

• Too many clicks to reach meeting invitations.• Information for next meeting seems to be in several different places, difficult for newcomers to find. Propose one page for next meeting, and one for most recent meeting, each containing all relevant info.• Meetings page is confusing.• Meeting invitations and attendee lists difficult to find.• Too many clicks to reach frequently accessed pages.• TDoc handling needs improving.• Junk mail on exploder lists.• Pages for groups out of date.• Submitting TDocs by email is painful.• Lists should be used more for discussions.• Meeting registration: ICS file difficult to find.

Page 32: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 32

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 11 (delegates):How satisfied are you with the support you receive from your MCC support officer for TSGs and WGs?

Average score: (2010 survey)delegates: 4.0 (3.2)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 33: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 33

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 12 (delegates):How would you rate the IT infrastructure during meetings …?

Page 34: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 34

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 12a:Access and robustness of the local server?

Average score: (2010 survey)delegates: 3.6 (3.2)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: A pleasing, if modest, improvement.

Page 35: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 35

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 12b:Speed and reliability of the public internet access?

Average score: (2010 survey)delegates: 3.2 (3.2)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 36: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 36

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 12 – delegate remarks - 1:How would you rate the IT infrastructure during meetings …?

• WiFi bandwidth has improved considerably in recent years but hotel IT infrastructure not so good.• Sometimes excellent, sometimes poor. (Especially internet speed.)• In general, performs well. Fit for purpose.• Poor in China. (Internet and local server in Qingdao)• Takes me half a day to get IP address at beginning of (RAN WG) meeting.• Current solution does not allow me to run VPN.

This is generally a problem with companies’ VPN settings, not with MCC infrastructure.• Internet bandwidth needs to be considerably increased.• Very variable, depending on location and host.• Hotel (bedroom?) internet access frequently poor.• Occasional interference in meeting rooms from delegates’ ad hoc networks.• Need automatic TDoc number allocation tool on meeting server.• Better since the introduction of 5GHz band.• Occasional problems early in the week.• IT support could have been more responsive (at a recent meeting).

Page 37: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 37

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 12 – delegate remarks - 2:How would you rate the IT infrastructure during meetings …?

• Internet access often virtually unusable in meetings. Unacceptable due to need for email discussions.• WiFi infrastructure in meetings is very variable.• Collocated meetings can be poor (eg Jacksonville megameeting, November 2010).• Difficult to find the right meeting folder for CRs and meeting reports. Suggest a URL such as http://3GPP70.

On the meeting server? Really??• Difficult to get IP address at “rush-hour”.• Sometimes 3GPP wifi network suffers from interference from hotel network. MCC should know how to fix this.• Overall current experience is excellent.• Frequent reboots.

Page 38: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 38

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 13 (delegates):How satisfied are you with the accuracy and speed of implementation of approved change requests?

Average score: (2010 survey)delegates: 3.8 (4.0)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 39: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 39

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 13 – delegate remarks:How satisfied are you with the accuracy and speed of implementation of approved change requests?

• Had to resubmit 2 CRs in the last 12 months.Because not implemented first time?

• Some CR packs presented to one meeting were empty.• Formatting issues with one spec, crashed MS Word.• Occasionally CRs not implemented, or misimplemented.

Current reported error rate is 2 un/misimplemented CRs per thousand.• Problem when WG meeting follows soon (3 weeks) after TSG meeting.• I don’t understand the question.

Page 40: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 40

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14 (delegates):How would you rate the following tools …

Page 41: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 41

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14a:3GPP search tools?

Average score: (2010 survey)delegates: 3.1 (2.7)non-delegates: (2.8)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: In view of plain text remarks, surprised to find this score higher than last time.

Page 42: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 42

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14b:Membership web page?

Average score: (2010 survey)delegates: 3.3 (3.0)non-delegates: (3.4)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 43: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 43

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14c:Administrative information?

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Average score: (2010 survey)delegates: 3.4 (3.0)non-delegates: (3.4)

Page 44: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 44

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14d:On-line registration?

Average score: (2010 survey)delegates: 3.8 (4.4)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: New registration tool introduced shortly before start of survey.

Page 45: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 45

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14e:Document number allocation?

Average score: (2010 survey)delegates: 3.5 (3.8)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Remark: Modest decline in satisfaction corresponds with verbal comments.

Page 46: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 46

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 14 – delegate remarks:How would you rate the […] tools?

• ADN performance poor under overload conditions.• ADN should allocate CR numbers as well as TDoc numbers.• TDoc number allocation differs from group to group; needs harmonizing across all groups.• Manual TDoc number allocation, poor response during pre-meeting rush.• Manual TDoc numbering is acceptable.• ADN now asks for too much information.• ADN is far too slow if you need several documents.• ADN sucks!• “Hiring company” and “Represented company” as entered at registration time are not accurately reproduced in the participants list. The order of clicks is important at registration time.• Improve the TDoc upload process, need a progress bar for large documents.• Membership page could do with wildcard capability when searching for company name.• Revisions of TDocs should use the same number, with “rev” suffix.• During meetings, new TDoc numbers communicated by word of mouth, often misheard, so multiple appearances of same number.• TDoc numbering scheme is good, but finding them is very difficult.

Page 47: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 47

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 15 (delegates):Use of the 3GPP databases …

Page 48: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 48

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 15a:Do you use the CR database?

no, 60%

yes, 40%

Page 49: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 49

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 15b:Do you use the specifications status database?

no, 41%

yes, 59%

Page 50: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 50

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 15c:Are you satisfied with the information provided in the 3GPP workplan?

no, 20%

yes, 80% (67%)

(33%)

Page 51: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 51

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 15d:Are you satisfied with the distribution and tracking of Liaison Statements?

no, 21%

yes, 79% (67%)

(33%)

Page 52: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 52

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 15 – delegate remarks:Use of the 3GPP databases

• Outgoing LSs could be associated with related incoming LSs, and vice versa.• CR database is difficult to use; the tool provided by Netovate* is useful.• Send an automatic email to a (dedicated) exploder when a new incoming LS for the group is posted.• Need a tool for tracking i/c and o/g LSs from previous meetings.• I am unaware of / never use these tools; they need to be better publicized.• I prefer to use web pages and search engine than directly access databases.• Zip files are hard to access.• Work plan information not intuitive to find.• I can’t open the CR database.

* Independent company offers search engine for individual TDocs and CRs, link from 3GPP site.

Page 53: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 53

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 16:Do you download the specifications via the web site (HTTP) or via FTP client?

FTP, 45%

HTTP, 55%

Page 54: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 54

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 16 – delegate remarks:Do you download the specifications via the web site (HTTP) or via FTP client?

• FTP easier than HTTP (eg using Filezilla, FTP Voyager, etc).• FTP quicker for multiple documents.• HTTP only for single documents, FTP for multiple.• My company does not allow FTP on its PCs, therefore use HTTP.• FTP does not work well through corporate firewalls and filters, therefore use HTTP.• FTP is outdated, prefer HTTP.• HTTP: easy to use via website.• Prefer FTP, I am familiar with FTP client tool.• FTP is slower than HTTP.• I use FTP sync tool to keep fully up to date.• I use customized FTP scripts to download what I am interested in.

Page 55: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 55

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 17:How would you assess the overall service provided by MCC?

Average score: (2010 survey)delegates: 3.8 (4.2)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

score

cou

nt

% delegates

non-delegates

Page 56: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 56

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 17 – delegate remarks:How would you assess the overall service provided by MCC?

• Good bunch of people!• Excellent, professional, courteous.• My Group’s secretary is not very good, messes up CR implementations a lot.• Work plan seems artificial and unusable.• My Group’s secretary is excellent. Very responsive, very detail-oriented, goes above and beyond the call of duty to assist delegates.• TF160 does an excellent job.• I don’t know what MCC stands for.• Always very helpful.• Top notch personnel who care about the industry and the output of the specifications.

Page 57: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 57

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 18:Which area of 3GPP is most in need of improvement?

On-line tools67%

Secretary role9%

Meeting support24%

Remark: MCC plans to concentrate on improvements to on-line tools over the next two years.

Page 58: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 58

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 18 – delegate remarks - 1:Which area of 3GPP is most in need of improvement?

• ADN is useless for WGs.• Network support / dimensioning is sometimes very poor.• Needs upgrading to professional level.• Email exploder is unmanageable.• Need a new secretary.• WiFi connection problems.• Quality of meeting reports deteriorating over time. Sometimes report is unintelligible for those not having attended.• Databases are cumbersome, difficult to use.• LS page should show “none” rather than Error 404 page when no LSs are currently available for a given group.• Access to the correct spec is difficult for non-delegates.• Managing / storing / retrieving information on the web site could be improved.• More tech info (eg CSFB, SRVCC, rSRVCC, MTC, …)• ADN needs improving, to include CR number allocation, filename verification at upload time.• Automatic CR cover page completion by ADN.• Automatic WID completion by ADN.

Page 59: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 59

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 18 – delegate remarks - 2:Which area of 3GPP is most in need of improvement?

• Accuracy of CR implementation.• WiFi at meetings.• Minor improvements to web site.• Application on meeting server for TDoc number allocation.• Search tool.• Wiki is not very good.• Web site access for mobile devices.• Need ability to check meeting room assignments.• Document handling is archaic.• Difficult to find status of other groups’ work items.• ADN is poor for allocating / uploading multiple documents.• ADN crashes.• Difficult to find information.• Secretary has work and meeting preparation which distract him from his tasks.• Some hosts do not provide original visa letters, just electronic copies.• Meeting facilities and internet connection need harmonizing.• Difficulty in uploading TDocs to meeting server.• Agenda not always displayed during meetings.

Page 60: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 60

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

Question 18 – delegate remarks - 3:Which area of 3GPP is most in need of improvement?

• Need on-line meeting agenda and scheduling, Tdoc number and status tracking, etc.• Need near real-time mirroring of meeting server on public server.• On-line tools are scattered between 3GPP and ETSI servers, some are difficult to find (eg setting up new user account).• Meeting rooms sometimes too small.• Reliable internet access during meetings is essential.• Occasional wifi bottlenecks.• Tools work less well than the meeting support / secretary.• Specs on the web site not always up to date.• Meeting invitations should include links to public transport (not just taxi) services.• Need a tool to track progress in parallel subgroup sessions.• More microphones, preferably wireless, even in small groups.• Workload (implementation) at implementation freeze.• Web site is too green.

Page 61: TSGs SA #58, Barcelona 1 © 3GPP 2012 Annex to SP-120885 3GPP TSG SA Meeting #58, Barcelona, Spain, December 2012 3GPP satisfaction survey, 2012 Analysis

TSGs SA #58, Barcelona 61

© 3GPP 2012

© 3GPP 2012

Annex to SP-120885

3GPP TSG SAMeeting #58, Barcelona, Spain, December 2012

www.3gpp.org

More Information about 3GPP:

[email protected]