nu acm talk virtual scientific communities for driving innovation and learning karl lieberherr joint...

NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning

Karl Lieberherrjoint work with

Ahmed Abdelmeged and Bryan Chadwick

04/18/23 1SCG Innovation

Supported by Novartis and GMO

Introduction

• Scientific Community Game(X) [SCG(X)]– Goal: Foster innovation and learning in some

domain X

• A virtual scientific community consists of virtual scholars that propose and oppose hypotheses maximizing their reputations

• Applications: Learning and innovation through focused interaction, “Netflix in the small”

04/18/23 Innovation 2

How to model a scholar?

• Solve problems• Provide hard problems• Propose hypotheses about Solve and Provide

(Introspection)• Oppose hypotheses– Strengthen hypotheses– Refute hypotheses • Supported opposing failed• Refuted opposing succeeded

04/18/23 3Innovation

Where SCG comes from

• J ACM

04/18/23 SCG Innovation 4

Lieberherr/Specker 1981

Outline• Introduction (done)• Highest safe rung example • SCG Scholar / Agent• SCG Agent in Action• Highest safe rung example (opposition)• Who is the winner?• Competition and collaboration• Disadvantages of SCG• Further Examples• SCG-based Software Development Process• Conclusions

Example: Jar Stress Testing

• You have a ladder with r rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung problem (r,b).

• How many experiments do you need? Minimize.

• (r,infinity)• (r,1)

04/18/23 6SCG

Highest Safe Rung ProblemProblems and Solutions

• Problems: p=((r,b),secret hsr), secret hsr in [0,r], r,b natural numbers• r = number of rungs• b = number of jars that are allowed to break• (r,b) is called a niche

• Solutions: sequence of queries of the form n? to find hsr. Responses: yes/no.

• Quality of solution: q = length of sequence of queries

Highest Safe Rung ProblemHypotheses

• Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality q: abbreviated H = ((r,b),q)

• Problems to be delivered for H = ((r,b),q) are of the form ((r,b), s). Important: A hypothesis defines a family of problems.

• Propose: Hypotheses H1 = ((25,2),11), H2 = ((25,2),6)

(from Kleinberg/Tardos)

Scholars propose and oppose

egoisticAlice egoistic

Bob increases his reputation

opposes (1)

provides problem (2)

solves problemnot as well as she expected based on HA2 (3)

WINS!LOSES

proposed hypotheses

social welfare

Life of a scholar: (propose+ oppose+ provide* solve*)*

What is the purpose of SCG?

• The purpose of playing an SCG(X) competition is to assess the "skills" of the agents in: – solving problems in domain X, – making good predictions about niches in domain

X, – finding the hardest problems in a specific niche

What is SCG(X)

no automationhuman plays

full automationagent plays

degree of automation used by scholar

some automationhuman plays

more applications:test constructive knowledge

transfer to reliable, efficient software

agent Bobagent Alice

What is SCG(X)?

TeamsDesign Problem Solver

Develop SoftwareDeliver Agent

Agent Alice Agent Bob

Administrator SCG police

I am the best

Let’s play constructive

ly04/18/23 12Innovation

TeamAlice

TeamBob

For agents: Full Round Robin Tournaments or Swiss-Style

• Agents to play the SCG(X). Repeat a few times with feedback used to update agents.

• Within the group of participating agent, the winning agent has the– best solver for X-problems – best supported knowledge about X

SCG in Action: Competitions

• http://www.ccs.neu.edu/home/lieber/courses/cs4500/f09/files/competitions/past_competitions/11_23/tournament_1/final_results_tournament_2009_11_24_12_03_41.html

• http://www.ccs.neu.edu/home/lieber/courses/cs4500/f09/files/competitions/past_competitions/10_22/tournament_1/final_results_tournament_2009_10_23_04_35_18.html

Highest Safe Rung Problemopposing

• opposing(refuting, strengthening)• Alice claims: Hypothesis ((25,2),5)– Bob opposes it by refuting it: Bob invents problem

((25,2), secret 22). Alice: 5? no, 10? yes, 6? no, 7? no, 8? no. Already 5 questions asked and answer still unknown. Alice’ claim is refuted.

• Alice claims: Hypothesis ((25,2),12)– Bob opposes it by strengthening it to ((25,2),9);

and he can successfully support this hypothesis

Highest Safe Rung Problemsupporting

• Alice claims: Hypothesis ((25,2),12)– Bob tries to discount but Alice supports it: Alice:

5? no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes. Only 9 questions asked and problem ((25,2), secret 23) is solved. Alice has supported her hypothesis.

Who wins? Alice or Bob?

• Reputation of Alice = – the number of hypotheses that Alice proposed that

were never successfully opposed by Bob (neither refuted nor strengthened) +

– the number of hypotheses that Bob proposed that were successfully opposed by Alice

• RA = HAnotOpposedB + HBOpposedA• The scholar with the highest reputation wins• encourages: creating strong knowledge and

discounting knowledge created by others

Motivated by real scientific community

competitive / collaborative

Agent Alice: claims hypothesis H

Agent Bob: opposes H, refutes: providesevidence for !H

Alice wins knowledge

Bob wins reputationmakes public knowledge

Highest Safe Rung Problemcompetition / collaboration

• Alice claims: Hypothesis ((25,2),12)– Bob tries to discount but Alice supports it: Alice: 5?

no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes.

– From this exchange which is prompted by Alice defending her reputation, Bob gets an idea: For problem: p=((r,b),secret hsr), consider f(r,q) =(r/q + q) and find a q so that f(25,q) is minimized. f(25,5)=10; f(25,6)=11;f(25,4)=11.

– From this idea Bob knows that he can strengthen the hypothesis to ((25,2),10)

– General solution: Given r, find q to minimize (r/q + q).

Scholars and Agents:Same rules

• Are encouraged to

1. offer results that are not easily improved.

2. offer results that they can successfully support.

3. strengthen results, if possible.

4. expose results that are wrong.

5. stay active and publish new results.

6. be well-rounded: solve posed problems and pose difficult problems for others.

7. become famous!

Soundness Theorem

• SCG is sound: The agent with the best algorithms / knowledge wins (there is no way to cheat)– best: within the group of participating agents

Highest Safe Rung ProblemAsymptotic Hypotheses

• Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality f(r,b) : abbreviated H = ((r,b),f(r,b))

• Problems to be delivered for H = ((r,b),f(r,b)) are of the form ((r,b), secret hsr).

• Propose: Hypotheses H1 = ((r,b),(log(r))b), H2 = ((r,b),r1/b)

Highest Safe Rung Problemdiscounting asymptotic hypothesis

• discounting (refuting, strengthening)• Alice claims: Hypothesis ((r,b),(b*log(r)))– Bob discounts it by refuting it: Bob invents

problem ((1024,2), secret hsr). log(1024) = 10. 20 questions are not enough! Alice: 30? no, 60? yes, 31? no, 32? no, etc.. Already 20 questions asked and answer still unknown. Alice’ claim is refuted.

• Alice claims: Hypothesis ((r,2),r/2)– Bob discounts it by strengthening it to ((r,2),2*r½ );

and he can successfully support this hypothesis.

Disadvantages of SCG

• The game is addictive. After Bob having spent 4 hours to fix his agent and still losing against Alice, Bob really wants to know why!

• Overhead to learn to define and participate in competitions.

• The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems.– if admin does not, cheap play– watching over the admin

How to compensatefor those disadvantages

• Warn the scholars.• Use a gentleman’s security policy: report

administrator problems, don’t exploit them to win.

• Occasionally have a non-counting “attack the administrator” competition to find vulnerabilities in administrator.– both generic as well as X-specific vulnerabilities.

GIGO: Garbage in / Garbage out

• If all agents are weak, no useful solver created.

Physics Maximum Height ProblemProblems and Solutions

• Problems: p=(v, a), v, a: positive real numbers• The maximum height obtained by a projectile

launched with speed v at angle a to the horizontal is z.

• Solutions: real number z.• Quality of solution: Number of correct decimal

places.

Physics Maximum Height Problem Hypotheses

• Alice claims the hypothesis: I can solve any maximum height problem p=(v,a) with quality q in 1 minute: abbreviated H = (MHP,q)

• Problems to be delivered for H = (MHP,q) are of the form (v,a).

• Propose: Hypotheses H1 = (MHP,3), H2 = (MHP,6)

http://scienceworld.wolfram.com/physics/Height.html

Physics Maximum Height Problem discounting

• discounting (refuting, strengthening)• Alice claims: Hypothesis (MHP,3)– Bob discounts it by refuting it: Bob invents

problem (25,60 degrees). Alice fails to solve the problem in 1 minute with 3 correct digits. Alice’ claim is refuted. Checking is done by experiment or trusted third party.

• Alice claims: Hypothesis (MHP,1)– Bob discounts it by strengthening it to (MHP,2);

RegExpToAutomata ProblemProblems and Solutions

• Problems: p=(r,n); r a regular expression of size n.• r = regular expression; a + b* a + a a a b*• n defines a niche of regular expressions

• Solutions: DFA d equivalent to r.• Quality of solution: Number of states of d.

RegExpToAutomata Problem Hypotheses

• Alice claims the hypothesis: I can solve any problem p=(r,n) with quality q or less: abbreviated H = (n,q)

• Problems to be delivered for H = (n,q) are of the form p=(r,n). Important: A hypothesis defines a family of problems.

• Propose: Hypotheses H1 = (5,11), H2 = (5,10)

RegExpToAutomata Problem opposing

• opposing(refuting, strengthening)• Alice claims: Hypothesis (5,11)– Bob discounts it by refuting it: Bob invents a

regular expression r of size 5, gives it to Alice and she fails to deliver a DFA d with 11 or fewer states. Alice’ claim is refuted.

• Alice claims: Hypothesis (5,20)– Bob discounts it by strengthening it to (5,19); and

he can successfully support this hypothesis

RegExpToAutomata Problem supporting

• Alice claims: Hypothesis (4,12)– Bob tries to discount but Alice supports it: Bob

gives to Alice a regular expression r of size 4. Alice provides and equivalent DFA with 12 or fewer states. Alice has supported her hypothesis.

were never successfully opposed by Bob (neither refuted nor strengthened) +

– the number of hypotheses that Bob proposed that were successfully opposed by Alice.

• RA = HAnotOpposedB + HBOpposedA• The scholar with the highest reputation wins.• encourages: creating minimum automata for

regular expressions of a given size.

Software Development Process

• Increase targeted interaction between software developers.

Traditional Approach

Human Developers

Develop new softwarefor problem solving domain X

Static Evaluation.

No competition.

human1 human2

Testing unit testing integration testing

Benchmark is usedto evaluate software

human3 human4

Requirements for X

37SCG-SP201004/18/23

Why Software Development through a virtual scientific community?

Human Developers

Develop new softwarefor problem solving domain X

SCG(X)

Erika-Patrick-agent

winning-agent

Evaluates fairly, frequently,

constructively and

dynamically.

Drives innovation.

Challenges humans.

Agents point humans to

what needs attention in

the software.

human1 human2

Erika Patrick

Benchmark is usedto evaluate software

Requirements for X

38SCG-SP201004/18/23

Erika-Patrick Agent

• Surrogate of combined knowledge of Erika and Patrick successfully transferred to agent.

• Transfer knowledge by programming.

39SCG-SP201004/18/23

Conclusions

• How to make learning and problem solving fun: design a game and interact.

• Scientific Community Game = Specker Challenge Game = SCG

• How to create reliable problem solving software? Have it tested through SCG.

Final Slide

• More Questions?

SCG concepts

• Scholars working in a domain with niches. Define functions on niches.

• Hypotheses: claims about functions on niches:– Discounting protocol for HA: Alice selects niche

element ne and Bob applies fBob so that claim about function does not hold

– Strengthening protocol

• Reputation

SCG concepts

• Scholars working in a domain with niches. Function f: Niche -> S for Alice and Bob.

• Hypotheses: claims about niches: belief: f has property b(s, dn, fdn). (Niche,Belief)– Discounting protocol: Alice selects niche element

ne and Bob applies fBob creating s, so that !b(s,ne)

– Strengthening protocol• Reputation

Hypothesis Structure

• Algorithm Solver: Problems -> Solutions• For all p in Problems with feature f in Features

algorithm Solver solves p using resources p(f) with quality(p,Solver(p),f).

• Algorithm Provider: Features -> Problems• For feature f, Algorithm Provider provides a

problem p, for all solutions of p, !quality(p,Solver(p),f).

Two person SCG

• Alice, Bob• Domain: Source, Target; fA, fB-> Source-> Target;

Source defined by niche predicate.• Hypotheses HA (HB): claims about fA (fB)• Discounting protocol for HA:– Bob provides element ne in Source so that fA(ne)

contradicts HA.– Alice provides element ne in Source so that fB(ne)

contradicts HA. • Strengthening protocol– Bob proposes HB, HA => HB and Alice cannot discount HB.

Two person SCGSpecialize for problem solving

• Alice, Bob• Domain: Problems -> Solutions• Hypotheses• Discounting protocol for hypothesis HA by Alice:– Bob attacks in one of two ways (depends on HA)

• Bob provides a problem for which Alice constructs a solution that contradicts HA.

• Alice provides a problem for which Bob constructs a solution that contradicts HA.

• Strengthening protocol– Bob proposes HB, HA => HB and Alice cannot discount HB.

Hypotheses

• Solution algorithm A: Problems->Solutions• For all elements p in Problems that have

feature F and a secret solution ss(p), algorithm A(p) constructs with resource constraint Prediction(F) an element s(p) in set Solutions(p) with property Q(p,s(p),ss(p),F).

Hypotheses

• Problem creation algorithm A-1

Discounting protocol

SCG by Example

• Highest safe rung problem• Speed prediction problem• graph diameter / average pair-wise distance

Example: Jar Stress Testing

• You have a ladder with r rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung problem (r,b).

• How many experiments do you need? Minimize.

• (r,infinity)• (r,1)

04/18/23 52SCG

Highest Safe Rung ProblemHypotheses

• Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality q: abbreviated H = ((r,b),q)

• Problems to be delivered for H = ((r,b),q) are of the form ((r,b), s). Important: A hypothesis defines a family of problems.

• Propose: Hypotheses H1 = ((25,2),11), H2 = ((25,2),6)

(from Kleinberg/Tardos)

Highest Safe Rung Problemopposing

• opposing(refuting, strengthening)• Alice claims: Hypothesis ((25,2),5)– Bob opposes it by refuting it: Bob invents problem

((25,2), secret 22). Alice: 5? no, 10? yes, 6? no, 7? no, 8? no. Already 5 questions asked and answer still unknown. Alice’ claim is refuted.

• Alice claims: Hypothesis ((25,2),12)– Bob opposes it by strengthening it to ((25,2),9);

Highest Safe Rung Problemsupporting

• Alice claims: Hypothesis ((25,2),12)– Bob tries to discount but Alice supports it: Alice:

5? no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes. Only 9 questions asked and problem ((25,2), secret 23) is solved. Alice has supported her hypothesis.

were never successfully discounted by Bob (neither refuted nor strengthened) +

– the number of hypotheses that Bob proposed that were successfully discounted by Alice

• RA = HAnotDiscountedB + HBdiscountedA• The scholar with the highest reputation wins• encourages: creating strong knowledge and

discounting knowledge created by others

Motivated by real scientific community

Highest Safe Rung Problemcompetition / collaboration

• Alice claims: Hypothesis ((25,2),12)– Bob tries to discount but Alice supports it: Alice: 5?

no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes.

– From this exchange which is prompted by Alice defending her reputation, Bob gets an idea: For problem: p=((r,b),secret hsr), consider f(r,q) =(r/q + q) and find a q so that f(25,q) is minimized. f(25,5)=10; f(25,6)=11;f(25,4)=11.

– From this idea Bob knows that he can strengthen the hypothesis to ((25,2),10)

– General solution: Given r, find q to minimize (r/q + q).

Highest Safe Rung ProblemAsymptotic Hypotheses

• Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality f(r,b) : abbreviated H = ((r,b),f(r,b))

• Problems to be delivered for H = ((r,b),f(r,b)) are of the form ((r,b), secret hsr).

• Propose: Hypotheses H1 = ((r,b),(log(r))b), H2 = ((r,b),r1/b)

Highest Safe Rung Problemdiscounting asymptotic hypothesis

• discounting (refuting, strengthening)• Alice claims: Hypothesis ((r,b),(b*log(r)))– Bob discounts it by refuting it: Bob invents

problem ((1024,2), secret hsr). log(1024) = 10. 20 questions are not enough! Alice: 30? no, 60? yes, 31? no, 32? no, etc.. Already 20 questions asked and answer still unknown. Alice’ claim is refuted.

• Alice claims: Hypothesis ((r,2),r/2)– Bob discounts it by strengthening it to ((r,2),2*r½ );

and he can successfully support this hypothesis.

Physics Maximum Height ProblemProblems and Solutions

• Problems: p=(v, a), v, a: positive real numbers• The maximum height obtained by a projectile

launched with speed v at angle a to the horizontal is z.

• Solutions: real number z.• Quality of solution: Number of correct decimal

places.

Physics Maximum Height Problem Hypotheses

• Alice claims the hypothesis: I can solve any maximum height problem p=(v,a) with quality q in 1 minute: abbreviated H = (MHP,q)

• Problems to be delivered for H = (MHP,q) are of the form (v,a).

• Propose: Hypotheses H1 = (MHP,3), H2 = (MHP,6)

http://scienceworld.wolfram.com/physics/Height.html

Physics Maximum Height Problem discounting

• discounting (refuting, strengthening)• Alice claims: Hypothesis (MHP,3)– Bob discounts it by refuting it: Bob invents

problem (25,60 degrees). Alice fails to solve the problem in 1 minute with 3 correct digits. Alice’ claim is refuted. Checking is done by experiment or trusted third party.

• Alice claims: Hypothesis (MHP,1)– Bob discounts it by strengthening it to (MHP,2);

RegExpToAutomata Problem Hypotheses

• Alice claims the hypothesis: I can solve any problem p=(r,n) with quality q or less: abbreviated H = (n,q)

RegExpToAutomata Problem discounting

• discounting (refuting, strengthening)• Alice claims: Hypothesis (5,11)– Bob discounts it by refuting it: Bob invents a

regular expression r of size 5, gives it to Alice and she fails to deliver a DFA d with 11 or fewer states. Alice’ claim is refuted.

• Alice claims: Hypothesis (5,20)– Bob discounts it by strengthening it to (5,19); and

he can successfully support this hypothesis

RegExpToAutomata Problem supporting

• Alice claims: Hypothesis (4,12)– Bob tries to discount but Alice supports it: Bob

gives to Alice a regular expression r of size 4. Alice provides and equivalent DFA with 12 or fewer states. Alice has supported her hypothesis.

were never successfully discounted by Bob (neither refuted nor strengthened) +

– the number of hypotheses that Bob proposed that were successfully discounted by Alice.

• RA = HAnotDiscountedB + HBdiscountedA• The scholar with the highest reputation wins.• encourages: creating minimum automata for

regular expressions of a given size.

Calculus Maximization ProblemProblems and Solutions

• Problems: p=(f: function in one variable,J interval);

• Solutions: maximum of f in interval I.

Calculus Maximization Problem Hypotheses

• Alice claims the hypothesis(Polynomial, k): I can solve any problem p=(f,J) for f a polynomial in time (size of the polynomial)^k. H =(Polynomial, k).

• Problems to be delivered for H = (Polynomial, k) are of the form p=(f,J), f a polynomial. Important: A hypothesis defines a family of problems.

• Propose: Hypotheses H2 = (Polynomial, 2), H1 = (Polynomial,1).

Calculus Maximization Problem discounting

• discounting (refuting, strengthening)• Alice claims: Hypothesis (Polynomial,1)– Bob discounts it by refuting it: Bob invents a

polynomial (e.g., x^2 – x + 1) in one variable and an interval, gives them to Alice and she fails to deliver, in the given time, the maximum of the polynomial in the interval. Alice’ claim is refuted.

• Alice claims: Hypothesis (Polynomial,3)– Bob discounts it by strengthening it to (Polynomial,2);

• I claim I can solve this problem with one program that runs in time t on a single core machine and that runs in time 1.2 * t/c on a machine with c>1 cores.

• Many kinds of hypotheses. They are defined by– Problems, Solutions– Discounting protocol • Refuting protocol• Strengthening protocol• Problems and solutions to be exchanged in protocols

HypothesesAlice constructive claims

• I can solve problems of kind k – with quality q– close to your quality– better than you

• I claim statement S of the form – ForAllExists– ExistsForAll

Extra: too complex

• Problems: p=(function in two variables f(t,b); interval for t; interval for b).• n defines a niche of regular expressions

• Solutions: min max solution.• Quality of solution: Number of states of d.

Minimizing and Maximizing FunctionsProblems and Solutions

• Problems: minimizing and maximizing functions. • Solutions: correct values.

Minimizing and Maximizing Functions Hypotheses

• Alice claims the hypothesis: function in two variables f(t,b); interval for t; interval for b. min [t] max [b] < h. H = (f(t,b),It,Ib,h)

• I can solve any problem p=(r,n) with quality q or less: abbreviated H = (n,q)

Calculus Alice claims the hypothesis: min t max b f(t,b) < 0.8. t and b are vectors in a subset of some vector space. Bob opposes Alice' hypothesis by strengthening it: min t max b f(t,b) < 0.7. Alice opposes Bob's hypothesis by strengthening it further: min t max b f(t,b) < 0.6. Bob opposes Alice' hypothesis by challenging it. Alice provides t=t0 and Bob finds b=b0 and it turns out that f(t0,b0)=0.65. Therefore Bob wins reputation from Alice.

p=((r,b),floating)

nu acm talk virtual scientific communities for driving innovation and learning karl lieberherr joint...

scg slide

scg innovation slide

scg innovation5 slide

purpose of scg

r rungs

form r

highest safe rung problem

infinity r

Documents

metrics and monitoring on fermigrid keith chadwick fermilab...

karl lieberherr northeastern university college of computer...

ben chadwick idr

gloria chadwick - arhiva spirituala · pdf filegloria...

keith chadwick fermilab chadwick@fnal

goal setting the florence chadwick story. florence chadwick,...

the scientific community game for stem innovation and...

chadwick international viewbook

weaving generic programming and traversal performance ·...

the algorithms we use to learn about algorithms karl...

case bsc chadwick

chadwick 2011 h

sheila chadwick - introductions

mountain goat software, llc an introduction to scrum karl...

aspect security - ravishekhar gopalan - prof. lieberherr...

chadwick 2009 bb

organizing open online computational problem solving...

in the united states court of federal claims - first...

chadwick 2009 x

visit at mit: daniel jackson’s lectures on coupling karl...