Success as a Student Researcher:A Discussion of Best Practices

Fall 2017

Dr. Scott Ferguson

Dr. Mark Pankow

Dr. Kate Saul

Why do you need to be here?

Doing good research is hard

• There are many new skills to learn

– Understanding the literature

– Setting up your experiments/simulations

– Data analysis

– Interpretation of your results

– Writing and presenting

• There is often a long time-scale on rewards

• At many levels, you are responsible for directingyour own education

You

• Who are you?

– Your expertise / interests

– Your career / life goals

• Graduate school should fit into your lifeplan

What is your life plan?Do you need to develop a strategic plan?

Your strategic plan

• A strategic plan has three parts:

– Where are you today?

– Where do you want to be in the future(5, 10, 20 years from now)

– How do you get from here to there?

A strategic plan is a roadmap for your life

What is research?

What IS research?

• The process of finding out something thatwe (everyone) do not already know

• Builds upon the extant knowledge base

• It is methodical, repeatable and verifiable

– You can specify, in advance, a procedure toaccomplish your stated objective

A research topic must …

• be research

• not have been done before

• be significant

• have a greater than zero probability that you cando it

• lend itself to a viable research plan

• be accomplished with the facilities you haveavailable

• fit into your strategic plan

Why is problem choice so important?

• You will be spending a lot of time on yourproblem

• Want to be personally interested

• Good perspective of the field

– See the big and small picture

• “When one can achieve self-expression inscience, work becomes revitalizing, and ladenwith personal meaning.” (Alon)

Factors to consider

• Feasibility: How hard or easy it is

– Problems are always more difficult than theylook

• Importance: Impact on the communityand beyond

– Who cares?

– What will they do when they see your work?

– How long will the answer be important?

Factors to consider

• Interest

– Both internal and external

– Do you have a passion for the topic?

– What will keep you working on it?

• Competence

– Why are you qualified?

– Do you have an advantage?

Choosing a research problem

Alon, Molecular Cell 35, 2009

Innovation in research

Finding an advisor

• Ideally, you have selected your school byidentifying faculty you would like to workwith

– If not, start looking NOW

• A good advisor will serve as a mentorAND a source of technical assistance

– They should help you set and achieveshort-term and long-term goals

Establishing a research objective / direction

Heilmeier questions

• What are the risks, and what can we do aboutthem?

• How long will it take?

• How much will it cost?

• What is the timeline and what are thedeliverables that we should expect throughoutthe project?

Heilmeier questions

• What are you trying to do?

• How is it done today? What are the limitationsof current practices?

• What is new in your approach and why do youthink it will succeed?

• Assuming success, what differences does thismake to us and society?

Establishing a research objective

• Concise statement of what you intend to find outthat we don’t already know

• Do not use words that mean “not research” todefine your objective

– Develop

– Design

– Optimize

– Control

– Manage

How to do it wrong!

• This project aims to advance the researchin predictive modeling for manufacturingprocess optimization

• The proposed study will significantlyadvance the theory of random fields

How to do it right!

• The research objective of this project is toaccount for uncertainty in engineeringdesign decision making through theapplication of utility theory

• The research objective of this project is tomeasure the cross-section of themuon-neutrino interaction at 5 GeVaccurate to 10%

Understanding the work presented in the literature

Understanding the literature

• Start with a 2 page review of the current state ofthe art

– Longer more detailed version for thesis

• What are the critical papers for determining howwe got here?

• What is the important information?

• What questions remain?

• What new directions should be explored?

How do I…

• …know what and where to search?

• …do the search?

• …know if what I’m reading is important?

• …keep track of what I read?

• …keep up to date on the latest “news”after my preliminary search?

• …maintain my own records?

Searching a new topic

• Identify keywords for your topic and browseALL the articles you find

• Identify key sources and browse them

• Identify key people (leaders! heroes!) and lookup their work

• Identify review papers

• Follow the references

• Revise the keywords

• Create alerts

What sources should you read?

• Journals– Inside your field

– Interdisciplinary (e.g. Nature, Science)

– Outside your field but encompassing complementary topics

• Conference proceedings (abstracts or papers)– Newest research

– Check out webpages for specific conferences

• Theses/Dissertations– More detail than papers

• Patents and industry reports– If your work has commercial applications, patents can give

design/process details.

– Can understand state of intellectual property

Conducting a literature review

Starting with a Clearly Defined

Research Problem

Trying to find a New Research

Problem

Start with 1-3 Research Areas

Google Scholar Results

Papers Behind Paywalls(1) Use University Credentials

(2) Try other versions on main Scholar page

…or, Google the author

The library

Down the rabbit hole

• Pull the cited papers

• READ the cited paper itself before citing it –don’t rely on another paper’s interpretation.Sometimes they aren’t right! Don’t propagateerror!

• Look at the citation numbers of papers to gaugeimportance (date of publication and subfieldwill influence this, so this is imperfect)

– Science Citation Index, Web of Science, Sci Search

Understand your community

• Identify the research groups that are most relevant– Follow their work

– Suggest as reviewers

– Make contact at meetings

• Flag important methods– Understand best practices

– Identify different information obtained from differentapproaches

– Incorporate into your work

• Keep track of next steps suggested in discussions orraised in your reading

• Identify ways in which your work builds on othergroups

Research social networks

• LinkedIn

• ResearchGate

• ResearcherID

• ORCID

• GoogleScholar

• Unique identifier to link you to research,distinguish you from other researchers

• Provides citation information

• Links to co-authors

The process is iterative

Find enough papers to create a

foundation for your

work, then do your work

Find papers as you’re working

Find papers to help

solidify and motivate the work you’ve already done

Check for papers early

and often (Google

scholar has an alert system)

Reading papers

Questions to ask while you are reading…

• Who are these authors? Do they have therelevant expertise?

• Is the source credible?

• Do I understand the terminology that is beingused in the paper?

Start with the abstract

Start with the abstract Purpose(why)

Start with the abstract Purpose(why)

Methodology(how)

Start with the abstract Purpose(why)

Methodology(how)

Results(what was found)

Start with the abstract Purpose(why)

Methodology(how)

Results(what was found)

Conclusions(what it means)

One strategy…

Why the introduction is important

• Tells you what is known in the field

• Explains the limitations of our currentunderstanding

• Leads to a focal point – what question is going tobe answered

PROVIDES PERSPECTIVE

Three pass approach

• Pass 1: General idea

• Pass 2: Content, but not details

• Pass 3: Understand in depth

Pass 1

• READ:– Title, abstract, introduction– Section and subsection headings (if they exist)– Glance at math– Conclusions– Skim references

• ANSWER:– Category: What kind of paper is this? Clinical? Technical

development?– Context: What is it related to?– Correctness: Assumptions valid?– Contributions: Main contributions.– Clarity: Well-written?

• Keep going? Peripheral to research area?

Pass 2

• READ:

– Careful look at figures/graphs

– Mark relevant references you haven’t read

• ANSWER:

– Summarize main thrust with evidence – coversheet/notes

• Keep going?

Pass 3

• CLOSE ATTENTION:

– Thought experiment

– Check all assumptions – any missing?

– Check all details – any errors?

– Limitations – any unidentified? How toaddress?

– Important citations – any missing?

– Think about future work

Questions to ask after reading the paper…

• Is the proposed approach a good one?

• Are the findings persuasive? Supported byenough evidence?

• Is there an alternative interpretation of the datanot addressed by the authors?

• How does this relate to my work?

Takeaways from reading EVERY paper

• What did they actually do?

• What knowledge did we gain because of this?

• What assumptions did they make?

• What papers did they reference?

• What opportunities for future work exist?

Now that you’ve read the paper

• How do I store this paper?

• Should I use a template for my notes?

• How do I save my notes?

Software for building a personal library

Sit down and write your notes

• As you’re reading, take notes (a few sentences)about each paper

– Method

– Major contribution

• Bin papers by common theme

• Write a paragraph (or more) about each theme,citing the papers you’ve found.

– Tie to your work

Experimental design

Experimental design

• What are we trying to learn?

• What variables are important?

• How are you going to test those variables?

• Is this even feasible?

• Is there a major flaw?

Essential characteristics

• Manipulation

– Change independent variables

• Control

– Something to compare it to

– Calibration

• Randomization

– Statistical Distribution

– Biology has more scatter

Process

• Determine Objectives

• Choose Response

– (variable to be measured)

• Choose factors and level

– (set independent variables)

• Choose experimental plan

• Conduct experiment

• Analyze Data

• Draw conclusions and Recommendations

Key concepts

• Change only one variable at a time

– Turning too many knobs doesn’t clarifyanything

• Isolate things as much as possible

– This is why labs are so controlled

• Can you calibrate the setup to show themeasurement is correct?

Before you run any experiments

• Check the gauges and instrumentation

• Identify sources of error

• Have you made test as simple as possible

• Watch for drift (measurements and system)

• Avoid changes (don’t go to lunch)

• Think about what can go wrong and how toprevent it

• Did you document everything

• Double check all your bolts

• What is your safety plan

Running the experiment

• Safety first

• Write down all the settings

• Save the raw data and make a copy

• When it is over make sure everything isshut off

Something will always go wrong

• Did you check the calibrations?

• Did you measure everything correctly?

• Did you check your units?

• Did you check your precision?

• Did you ground your equipment?

Be flexible

Computer simulations

• Can be used as surrogates for a physicalexperiment

• Many of the issues associated with aphysical experiment still apply

• Validation?

Simulations = Control

• With real data, you never perfectly know the real-world processes that caused the particularmeasured values to occur

• In a simulation, you control all of the factorsmaking up the data and can manipulate thesesystematically to see directly how specificproblems and assumptions affect the analysis

• Allow direct contact with the assumptions that aremade and allow exploration of their implications

Simulation management

• If you have generated a code or a model,you should:

– Save each iteration of it in a separate location(folder)

– Always have backups of your simulations

– Try to maintain good “coding” practice(comments, variable names, etc.)

Setting up the simulation

• Should you use pre-defined values?

– Are they correct for your problem?

– How do they influence the outcome?

Setting up the simulation

• User-defined values

– Have some justification for them

– Record your settings

– Run some experiments to determine the effect

Stochastic simulations

• Does your simulation give you the exactsame output each time you run it?

– Should it?

• What are the sources of randomness?

• If any randomness exists, a simulationmust be run multiple times

Data management plan

• Is now an essential element of any NSFproposal

• What this means for you

– All data must be saved

• Only delete data associated with a mistake in the code

– Data should be backed up in multiple locations

– Public accessibility

Recording your data

• Simulation data needs to have BOTHinput and output values

– What were the defaults?

– What did you change?

– What was the outcome?

Recording your data

• Come up with an effective strategy thatworks for you (example – subfolders)

• Example:

– Problem formulation

Settings

Run number

Reporting run-times

• Is this the right metric to use?

• Be sure to maintain consistency

– Are you using the same computer?

– What else do you have open?

– What is running in the background?

Validation

• Simulations are an imitation of the real-world

– Do your answers make sense?

– What are you measuring them against?

– Is there physical experimental data that youcan use to validate your results?

Data analysis

• Statistical Quantities

– Mean, Standard Deviation

• Analysis of Variance (ANOVA)

– If factor influences response

– Statistical Significance

• Regression Analysis

– Curve or equation to fit data

– R2 value

Writing

"I can't write""I never really

learned the right way to write"

"I don't like technical writing"

"I don't have to write well to be

successful"

"I'm an engineer and engineers

don't write"

”WRITING IS NOT SOMETHING A SCIENTIST DOES, BEING A WRITER IS

SOMETHING A SCIENTIST IS”

Writing is your currency

• Your success is measured largely by

– How many papers you write

– The reputation of the conferences and journalsin which your papers are published

– How many citations your papers get

Good ideas

• Often come when you least expect them

• Carry a note book to write ideas

• Often when you are thinking quietly andsomewhat distracted

You should write every day

• Summary of thoughts that you had

• Summary of papers you read

• Research ideas you want to explore

• Analysis of your data

To be a successful writer

• Write a lot of papers (good)

• Write a lot of papers that are good enoughto get into reputable journals (better)

• Write papers that people want to read(best – these are “sticky” papers)

My sticky paper

• Hazelrigg, G. A., 1998, “A Framework for Decision-Based Engineering Design,” Journal of Mechanical Design,120: 653-658.

My sticky paper

• Hazelrigg, G. A., 1998, “A Framework for Decision-Based Engineering Design,” Journal of Mechanical Design,120: 653-658.

Traditional technical writing process

Collect Data

ImposeImpending

Writing Deadline

“Write up” the data

This doesn’t work!

• Deadlines can be motivational, but

– Good writers need enough time to embracethe drafting process

– Polishing takes time

• Data isn’t very interesting

– Build a story from the data, but the data isn’tthe story

Traditional technical writing process

Collect Data

ImposeImpending

Writing Deadline

“Write up” the data

A better technical writing process

Data Information Knowledge Understanding

A better technical writing process

Data Information Knowledge Understanding

Locations, sizes, power

development, and fitness values for a

series of wind

turbines Layouts

Global optimality for wind

farm optimization

Best practices for wind

farm developers

and optimization

Objec&ve(Func&on(Evalua&

on(

Number(of(Turbines(

When you write:

• Work backwards!

– Start with the “understanding” – this is yourstory!

– Knowledge is generally related to your results

– Information is generally related to yourmethodology/approach and introduction

– Make your data publically available!

Writing takeaways

• You have to work at being a good writer

• Being a good writer is the most important part of yourjob

• Embrace the drafting process

• Give yourself time

• Don’t write about the data, write about theunderstanding– And don’t know the story before you start collecting data!

• Sketch your knowledge as a cartoon

• Write research objectives that speak to what you expectthe knowledge to be

Making figures from your data analysis

Raw data

• What it needs:

– Formatting

– Figures

– Description

– Evaluation

10 -20 20 -10 2636.81 4746.25 -80.00

-20 -90 30 -20 2428.04 1683.13 30.68

10 -10 30 0 3961.51 4103.52 -3.58

-20 90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

-30 -90 30 -30 1631.49 1062.10 34.90

10 -10 30 0 3961.51 4103.52 -3.58

0 -30 20 -10 2745.43 3847.66 -40.15

-20 90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

10 -10 30 0 3961.51 4103.52 -3.58

-30 -90 30 -30 1631.49 1062.10 34.90

-30 -90 30 -30 1631.49 1062.10 34.90

-20 -90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

0 -30 20 -10 2745.43 3847.66 -40.15

0 -20 10 -10 3962.40 4211.82 -6.29

-20 -90 30 -20 2428.04 1683.13 30.68

-30 -90 30 -30 1631.49 1062.10 34.90

-20 -90 30 -20 2428.04 1683.13 30.68

10 -10 30 0 3961.51 4103.52 -3.58

-20 -90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

-30 -80 30 -40 786.90 1995.60 -153.60

-30 -80 30 -40 786.90 1995.60 -153.60

-20 -90 30 -20 2428.04 1683.13 30.68

-20 90 30 -20 2428.04 1683.13 30.68

-30 -90 30 -30 1631.49 1062.10 34.90

-30 -90 20 -40 1090.39 1828.01 -67.65

-30 -90 30 -30 1631.49 1062.10 34.90

-20 -90 30 -20 2428.04 1683.13 30.68

-20 90 30 -20 2428.04 1683.13 30.68

-30 -90 30 -30 1631.49 1062.10 34.90

-20 90 30 -20 2428.04 1683.13 30.68

-30 -90 30 -30 1631.49 1062.10 34.90

-20 90 30 -20 2428.04 1683.13 30.68

-20 -90 30 -20 2428.04 1683.13 30.68

-40 -90 40 -40 718.55 642.42 10.59

What type of graph?

Are my data points connected?

Is there a relationship I am looking for?

What are the sources of error?

Is the data meaningful?

Are the results significant?

How sensitive is my output to the inputs?

Figure construction

A figure should

• Make only one point

• Be self-explanatory and easy to read

• Use the chosen visual elements efficiently

• Be suitable for use on a presentation slide

– How are you going to layer your information?

Other thoughts

• Can I read your axes labels?

• Did you include units?

• Is the title appropriate?

• Can I distinguish between your data points andyour trendlines?

Other thoughts

• Are you telling the correct story?

• Is the figure meaningful?

Meaningful data?

http://www.tylervigen.com/spurious-correlations

Tufte’s compositional principles

• Good figures should:

– Use words, numbers, and drawing together

– Reflect a balance, a proportion, a sense of relevantscale

– Have a story to tell about the data

– Be drawn in a professional manner

– Avoid content-free decoration

Doumont, 2009

Low SNR High SNR

Noise

Signal

A low SNR makes the message unclear

Elements of figure composition and style

• Color coding

• Text

• Lines, axes and arrows

• Alignment and grouping

• Choice of shape

• Drawings / images

• Negative space

Color transitions

Color transitions

Alignment

Color and alignment

Annotations and visual cues

Showing practical scales

Show scale and function

Figure-making software

• Adobe illustrator, photoshop

• Inkscape (free)

• IPE (Vector graphics)

• Microsoft Visio

• OmniGraffle

• SmartDraw

• Origin (for plotting)

• Others……

Giving a (good) presentation

Who are you presenting to?

• Who is your audience?

• What is the purpose of the presentation?

• What do you want them to take away?

• How long should it be?

What is wrong with these slides?

Content from http://www.assertion-evidence.com/teaching.html

The Chesapeake Bay, which is the country’s largest estuary, has only two places for traffic to cross

Content from http://www.assertion-evidence.com/teaching.html

The assertion-evidence approach is an effective technique for designing slides

• Using the assertion-evidence approach leads to bettercomprehension and recall by the audience– Joanna K. Garner, Michael Alley, Keri Wolfe, and Lauren

Sawarynski

– International Journal of Engineering Education (vol. 29, no. 6,2013)

• Using the assertion-evidence approach leads to deeperunderstanding of the content by the presenter– Joanna K. Garner. Michael Alley, and Shannon Aippersbach

– International Journal of Engineering Education (vol. 32, no. 1(A),2016)

• http://www.assertion-evidence.com/

There are three main tenets driving the assertion-evidence approach

• Build talks on messages, not topics

• Support message with visuals, not bullets

• Explain visuals by performing sentences on thespot

Your slide design MUST be clean

Content from http://www.assertion-evidence.com/teaching.html

A title slide should convey information about the talk

• Things to include:

– Catchy title

– Not too long (Engineers do this LIKE CRAZY)

– Your name

– Names of your co-authors or co-presenters

– Affiliation (School, Course, etc.)

– Date

There are a few good strategies to get started

• Things to say:

– Introduce yourself

– “Today, I’d like to speak to you about [title]”

– Look at audience

– Don’t put your hands in your pockets/cross yourarms

Use a picture to kick off the presentation

Motivation slide

• Motivation should be made clear:

– What?

– Why?

– Why is this important?

• If possible, have a precise problem statement

• If possible, don’t have a text-only motivationslide

Outlines should be used to only provide context

You cannot read and listen at the same time.

Fact: The human brain is fighting you

About text …..

• If you need text, use animated bullets

• Define acronyms the first time

• No two consecutive slides should be text-only

• Watch your spacing

Visualization opportunity

• Demand for electricity is expected to increase by 39% by2020 [1]

• United States Department of Energy (DOE) wants tomeet 20% of this total electricity need using wind power

• Right now, wind power is about 6% of the totalelectricity supply• Approximately 51 GW

• Must increase to 305 GW

• Must make sure new farms are performing optimally inorder to meet this goal

Visualization opportunity

U.S. ELECTRICITY

U.S. ELECTRICITY

Wind Power

305

GW

51 GW

39%

Final thoughts - style

• Keep layout simple (don’t clutter)

• Be consistent (color, font,

emphasis, etc.)

• Fonts:

• Serif Font (Good for large blocks of text)

• Sans Serif Font (Easy to read for short bursts of text)

• When in doubt, use sans serif font

• Calibri, Arial, use courier for code

• Do not use Comic Sans. Ever.

• Visualize, but be smart about it!

• Avoid using uninformative clipart or animations

Making a (good) poster

Why do we present posters?

• More personal interaction with peopleinterested in your topic

• Ability to reach people you might notnormally catch with a 15 minute talk

• People standing are more engaged thanpeople sitting in chairs

Where are posters presented?

How do we make a poster?

• You can try looking on the internet forexamples

– Many are awful

• Most people who have to make postershave never been trained

– Typography

– Color choice

– Page layout

We all know it’s a problem

Presenting your poster

• Do not chew gum

• Keep your hands out of your pockets

• Speak to your viewers

• Give a 2-sentence overview

References

Other references

http://www.ncsu.edu/project/posters/index.html

Time management

Why do I need a strategy?

• Demands on your time will only grow.

• Bad time management leads to stress andpoor productivity

• LIFE skill!

Why is academic time hard to manage?

• Lots of choices (“academic freedom”)

• Open ended tasks (research)

• Bureaucracy inherent in researchadministration

• Short and long term tasks, and “thinking”time

Time management

Keep a time log

• For 2 weeks monitor your time DURINGthe day

• Are you being efficient?

• Where are you wasting time?

• Are you wasting OTHER people’s time?

• Where could you delegate?

• Are you paying attention to thequadrants?

Learn your schedule

• What is your creative time and place? USE IT!

• When is your dead time?

• Work with your advisor to set up a semester-plan with deliverables that you both like

– Make sure you stay honest, don’t cheat

Things that waste time

• Interruptions

• Messy working area

• No calendar

• Being unprepared

• Lack of sleep

Interruptions

• Keep idea list, Google tasks, notebook topick up later

• Don’t check email constantly

• Ignore your phone, chat

• Close your door

Avoiding procrastination

• Set internal deadlines

• Make big projects into bite size pieces

• Identify why you aren’t enthusiastic aboutsomething

• Challenge embarrassment or failure

• Find strategies to force you to get started(schedule meetings for accountability,make promises to someone else…)

• Ask for guidance if you’re really stuck

Accountability

• Set stretch goals for yourself, and realisticdeadlines for items due to others.

• NEVER break a promise

– If you need more time, try to renegotiate thedeadline ahead of time NOT after

• Be clear about priorities

• Say no and don’t overextend

Work / life balance

• Finding a balance is not easy

• It is important for your mental andphysical health to have other interests

• Develop a schedule and be consistent

• Setbacks will happen. Have an action planto handle them

Managing your manager

What is the role of a research advisor?

• Boss AND a mentor• Big picture research• Research collaborator

– Technical advice– Writing and presentation training

• Professional development and career advice• Psychosocial

– Putting it all in context– Handling adversity

• Mutual expectations must be CLEAR

Getting feedback

• Accept course corrections graciously(revisions will be necessary)

• Anticipate requests for feedback(respond immediately when asked – if notsooner)

• Give presentations – within your own researchgroup, at seminars, attend conferences

How to hold a meeting

• Clear agenda

• Advance preparation

• Open discussion with constructivecriticism and mutual participation

• Leader manages discussion and keeps ontrack

• Action items are clearly communicated

Manage your advisor-advisee relationship

• Ask for time/help when you need it– Know EXACTLY what you want to accomplish– Express objectives clearly– Suggest next steps– Identify times and meeting styles most efficient for you

both

• Understand your advisor’s priorities and pressures• Help with research proposals• Go to conferences to identify what YOU want to know

• DON’T just wait to be told what to do next

This is YOUR graduate career

• Be thoughtful about your course selection• Be responsible for programmatic requirements

and deadlines• Know your career goals• Within appropriate bounds for your advisor’s

preferences and expectations:– Be prepared for meetings– Know what needs to be accomplished– Take initiative to seek information for things you

don’t know– Suggest solutions– Identify journals, funding opportunities relevant to

your work– Communicate