Policy Studies Review, May, 1984, Vol. 3 , No. 3-4

MICHAEL GIBBONS

THE EVALUATION OF GOVERNMENT POLICIES FOR INNOVATION

Government policies f o r t h e promotion o f technological innovat ion a r e ubiquitous. Whether one i s deal ing w i t h developing o r developed coun- tr ies, w i t h governments o f t h e l e f t o r t h e right, w i t h i ndus t r i a l magnates o r ent repreneurs tha t run small f i rms, a l l acknowledge t h e importance o f technological innovat ion f o r t h e i r s u r v i v a l and t h e c ruc ia l ro le p layed by government in suppor t i ng it. The reason f o r t h i s i s simple enough: any nation, government o r f i r m depends upon a cont inual f low of new technolo- gy to maintain o r improve i t s si tuation. As a consequence, the re has been, since the e n d of World War I I , a v i r t u a l explosion o f government incentives, subsidies, and programs aimed a t st imulat ing one o r another aspect o f t h e innovat ion process. T h e number and range o f these pol icies have become so bewi lder ing tha t recent ly several attempts have been made t o classify them and bring some o r d e r t o an increas ing ly complex situation. One recent example has been p rov ided by Rothwell and Zegveld (see t h e i r ar t ic le in t h i s volume).

Sti l l , classif icat ion i s n o t evaluation and i f attempts a t classif icat ion have been recu r ren t , evaluation now occupies a pos i t ion o f p r i o r i t y f o r many governments and indust r ia l is ts . The reason for t h i s i n te res t i s n o t d i f f i cu l t t o discern. T h e r e i s widespread bel ie f many innovat ion policies do no t achieve the objectives f o r wh ich they were designed. Sometimes these bel iefs a re rooted more in pol i t ical ideology than in empir ical science. B u t a t times empir ical studies have been unable t o i d e n t i f y any pos i t ive effects from the pub l i c monies invested in technological innovation. Typi- cal o f t he outcome o f t h i s so r t o f empir ical w o r k i s t h e conclusion reached by Allen and h i s associates who in 1978 ca r r i ed ou t a major in ternat ional review aimed a t evaluat ing t h e in f luence o f government policies on the process o f innovat ion in Western Europe and Japan:

O f course, the most s i g n i f i c a n t aspect o f t he study l i e s i n i t s f a i l u r e t o de tec t any e f f e c t , on p r o j e c t performance, o f govern- ment attempts t o s t imu la te innovat ion. Governments have t r i e d , i n a v a r i e t y o f ways ... t o do j u s t t h i s . I f p r o j e c t success or f a i l u r e can be taken as a measure o f e f fec t i veness o f t h e i r act ions, then l i t t l e can be sa id t o have r e s u l t e d from a l l t h i s expenditure o f e f f o r t and money. (A l len, e t a l , 1978:148)

The reasons w h y innovat ion policies fa i l t o produce t h e des i red resu l t s are no doubt as va r ious as t h e countr ies, governments, and agencies i n i t i a t i ng them. Nonetheless, a usefu l l i s t o f some of t h e problems has been p repared by Rothwell and Zegveld a n d appears in t h e i r a r t i c l e in t h i s volume. In summary, t he l i s t includes lack o f market know-how among pub l i c policymakers; i ndus t r i a l subsidies tend ing t o favo r large, well- establ ished f irms; governments tending t o adopt a passive r a t h e r than an act ive stance towards information dissemination; t h e lack o f pract ica l knowledge, o r an imaginative conception, o f t h e innovat ion process by policymakers; over-concentrat ion o n t h e R t D e n d o f t h e innovat ion p ro - cess; t h e lack o f interdepartmental coordination between re levant agencies; a tendency f o r innovat ion pol icy t o b e subject t o changes in pol i t ica l phi losophy r a t h e r than t o changing national and in ternat ional economic

476

CibbonsfEvaluation of Government P o l i c i e s f o r Innovat ion 477

needs o r condi t ions. A t t h e e n d o f one o f t h e many papers d e r i v e d f rom t h e i r major book, Rothwel l a n d Zegveld conclude w i t h a n obse rva t i on (echoed in t h e i r c o n t r i b u t i o n to t h i s symposium) succ inc t l y s t a t i n g t h e p rob lemat ique of t h i s ar t ic le :

F i n a l l y , hav ing d iscussed i n n o v a t i o n p o l i c y i n some d e t a i l , i t must be admi t ted t h a t we know ve ry l i t t l e about i t s e f f e c t i v e - ness. It i s a f a c t t h a t few i n n o v a t i o n p o l i c y i n i t i a t i v e s have been sub jec ted t o o b j e c t i v e assessment rega rd ing t h e i r e f f i c a c y . There i s , thus, a p ress ing need f o r d e t a i l e d impact assessment across a wide range o f p o l i c i e s i n many c o u n t r i e s .... I n v iew o f t h i s i t migh t be s a i d t h a t perhaps t h e g r e a t e s t problem o f i n n o v a t i o n p o l i c y i s t h a t i t has been more an o b j e c t o f f a i t h than o f understanding. (Rothwel l and Zegveld, 1981: 153)

Evaluat ion, t hen , i s becoming a n issue main ly because gove rnmen t innova- t i o n pol ic ies a r e fa i l i ng t o a t t a i n t h e i r ob ject ives and, as t h e p reced ing p a r a g r a p h ind icates, t h e r e i s a p r e s s i n g need f o r impact assessments. T h e r e a r e severa l po in ts w h i c h shou ld b e made abou t t h i s i n junc t i on . F i r s t , po l i cy f o r i nnova t ion seldom has a s ingle, c lea r l y de f i ned , ob ject ive. Not o n l y t h a t , t h e pol ic ies emanat ing f rom d i f f e r e n t p a r t s o f t h e g o v e r n - ment bu reauc racy may ac tua l l y con f l i c t w i t h one ano the r so t h e i r e f fec t o n bus iness e n t e r p r i s e i s mere ly t o c rea te amb igu i t y a n d u n c e r t a i n t y (a v e r y b r i e f l i s t i s g i v e n in Table 1) .

Table 1

Some Con t rad i c to ry P o l i c i e s i n t h e U.S. A f f e c t i n g t h e Operat ions o f I n d u s t r y

The Environmental P r o t e c t i o n Agency The Energy Department i s pushing i s pushing ha rd f o r s t r i n g e n t a i r p o l l u t i o n c o n t r o l s .

The Na t iona l Highway T r a f f i c Sa fe ty The T ranspor ta t i on Department i s A d m i n i s t r a t i o n mandates weight - i n s i s t i n g on l i g h t e r veh ic les t o adding s a f e t y equipment f o r cars . conserve gasol ine.

The J u s t i c e Department o f f e r s gu id- The S e c u r i t i e s and Exchange Com- ance t o companies on complying w i t h m iss ion w i l l n o t promise immunity t h e Fore ign Corrupt P rac t i ces Act. f rom p rosecu t ion f o r p r a c t i c e s the

J u s t i c e Department might permi t .

The Energy Department t r i e s t o keep The T ranspor ta t i on Department t r i e s r a i l r a t e s f o r h a u l i n g coal t o en- t o keep c o a l - r a i l r a t e s h i g h t o courage p l a n t conversions. b o l s t e r t h e a i l i n g r a i l i n d u s t r y .

The Environmental P r o t e c t i o n Agency The A g r i c u l t u r a l Department pro- r e s t r i c t s use o f p e s t i c i d e s . motes p e s t i c i d e s f o r a g r i c u l t u r a l

and f o r e s t r y use.

The Occupat ional Safety and Hea l th The Environmental P r o t e c t i o n Agen- A d m i n i s t r a t i o n chooses t h e l owes t cy uses more f l e x i b l e standards l eve l o f exposure t o hazardous sub- f o r comparing r i s k l e v e l s w i t h

o f bank rup t ing an i ndus t r y .

companies t o s w i t c h f rom imported o i l t o d i r t i e r coa l .

stances t e c h n i c a l l y f e a s i b l e s h o r t costs .

Source: Business Week, June 3 0 , 1980.

47 8 Policy Studies Review, 3:3-4, May, 1984

Second, and because o f th is , it is no t clear t o what ex ten t it i s va l i d t o evaluate pol icy by comparing pol icy in tent ions w i t h outcomes.

It w i l l be use fu l t o d i s t i ngu ish between moni tor ing and evaluation. Monitoring answers the question, W h a t happened?" Evaluation, by con- t rast , i s addressed t o the question, "Has t he pol icy made any d i f ference?" Thus much o f what has been classed as evaluation i s monitoring. To i l lus t ra te t h i s d is t inct ion, consider government po l icy fo r t he UK a i r c ra f t i n d u s t r y between 1945 and 1968. In h is analysis Card iner 11975) estimates the flow o f receipts from government t o i n d u s t r y t o b e M 1.505.4 a t input pr ices while receipts b~ government in 1974 had total led on ly M 141.9 a t the sane prices. On the basis o f t he d is t inct ion between monitoring and evaluation, it i s clear Gard iner has monitored a po l icy and has g i ven a par t ia l account o f what happened. The quest ion o f whether the pol icy o f giving indus t r i a l subsidies t o the a i r c ra f t i n d u s t r y made any dif ference and to whom it may have made a d i f ference remains unanswered. Even though the ra te o f r e t u r n on capital was low, it could b e argued tha t during the per iod t h e i n d u s t r y was able t o completely modernize i t s plant, it was able t o ho ld together a f irst-class technological capabil i ty while senior management looked fo r new markets. If e i the r o f these were t rue, the pol icy could be said t o have made a d i f ference t o t h e funct ion ing o f the i n d u s t r y and the question then becomes whether t h i s was f o r t he be t te r o r not. It also should be clear i f t h e d i f ference i s judged to be a pos i t ive improvement, it does no t follow we w i l l b e able t o i den t i f y any congruence between th i s outcome and the pol icy in tent ions tha t in i t ia ted the change.

A Few New Developments in the Literature

Though the re has been more empir ical work aimed a t monitoring than at evaluation, t he re a r e a number o f in terest ing developments in the l i tera- t u r e pa r t i cu la r l y re levant t o the theme o f evaluat ing innovat ion pol icy. A t present, much o f t he w o r k seems fragmented and the re i s l i t t l e indication o f cross-fert i l izat ion between researchers who plow d i f f e ren t corners of t he publ ic pol icy f ield. Par t o f t he reason fo r this lies in the fact pol icy research i s organized around d i f f e ren t models o f t he pol icy process.

In an in terest ing paper, Nelson (1974) re f lected on t h e " c u r r e n t malaise o f rat ional analysis o f social problems.lI T h e paradox f o r Nelson was, while it had p roved possible--one might even b e tempted t o say easy--for the USA t o put a man on t h e moon, t h e persistence o f u r b a n ghet tos i s a s ta rk reminder, in o the r areas o f social l i fe, pub l i c po l icy seemed power- less to change the s i tuat ion in any palpable way. In trying t o comprehend why t h i s should b e so, Nelson ident i f ied th ree analyt ical t rad i t i ons in pol icy research h e fe l t were exerc is ing a contro l l ing in f luence on the solut ion o f pract ica l problems. T h e f i r s t o f these sees social and economic problems as stemming from the past allocation o f sc ient i f ic and technical resources and, appropr ia te ly , proposes solut ions in terms o f reallocation o f these resources. T h e second t rad i t i on looks a t problems o f organizational s t ruc tu res a n d searches f o r solut ions o f inst i tu t ional reforms, whi le t h e third t rad i t i on looks a t problems o f inadequate pol icy machinery and t h e i r resolut ion o f b e t t e r analyses feeding t h e pol icy process.

The t h r u s t o f Nelson's c r i t i ca l comment is tha t t he re is l i t t l e in t h e way o f exchange o f information o r ideas between these analyt ical t rad i t ions, and if l i t t l e advance has been made towards so lv ing complex pub l i c pol icy problems, i t Is in n o small p a r t because researchers a re hermetical ly sealed w i th in t h e i r o f ten overs impl i f ied research paradigms. T o some ex ten t t h i s

CibbonsIEvaluatfon of Government Policies for Innovation 479

continues but t h e boundar ies o f these t rad i t ions a r e becoming less rigid and, o f ten w i thou t real izing it, researchers in widely separated areas are con t r i bu t i ng t o a more sophist icated view o f pol icymaking in general.

Th ree examples o f c u r r e n t research may he lp t o i l l us t ra te some o f these points.

( 1 ) The w o r k o f Mar t i n and l r v i n e (1983) in t h e UK prov ides an excel- lent exemplar o f what can b e achieved, by dint o f ca re fu l analysis, in the evaluation o f t h e o u t p u t o f basic research. T h e authors, who could b e said t o b e operat ing w i t h i n t h e resource al location t rad i t ion, a re ca re fu l t o avoid the obvious dangers o f select ing a single var iab le t o evaluate scien- t i f ic performance. Rather, t hey make use o f t h ree ind icators o f ou tpu t : publ icat ions, impact (neasured by c i ta t ions) and in ternat ional peer rank - ing. What i s impor tant and novel about th is , methodologically speaking, i s it t r i es t o develop a set o f in ter re la ted measures which, if t h e y a l l po in t in the same d i rect ion, increases the re l i ab i l i t y w i t h which one can r a n k the performance o f scientists engaged in similar so r t s o f sc ient i f ic work.

Sti l l , t he w o r k o f Mar t i n and l r v i n e has n o t escaped cr i t ic ism f rom the scienti f ic community whose performance t h e ind icators aim t o evaluate. Th is cr i t ic ism fal ls i n t o two b road categories, one methodological and the other pol i t ical. With r e g a r d t o method, it i s n o t clear t o what extent publ icat ions, c i ta t ion indexes, and peer rank ings measure th ree independ- ent aspects o f performance. If they a re no t independent, t hen what in terpretat ion can b e attached t o t h e i r convergence? With regard t o the second, scientists in pa r t i cu la r have been concerned about the use that might be made o f performance ind icators in t h e allocation process. To put the matter bluntly, if somewhat super f ic ia l ly , w i l l a low rank ing, in terna- t ional ly, b e used more t o terminate programs and shu t laboratories than t o promote new development? T h i s i s a real d i f f i c u l t y f o r t he scienti f ic community and pol icy analysts but it does no t follow tha t such measures because t h e y a re imper fect a re useless. Far be t te r t o see what extent such measures can b e improved and how they can b e be t te r in tegrated in to the decisionmaking process.

( 2 ) In t h e evaluation o f innovation policies, knowledge o f t he relevant decisionmaking processes a re v i t a l , as the recent work o f Kogan and h is colleagues (1980) on t h e implementation o f t he Rothschi ld reforms in the Department o f Heal th and Social Secur i ty (HDSS) in the UK amply demon- strate. In t h e 1971 White Paper, t h e B r i t i s h Government backed Lo rd Rothschi ldls recommendations tha t pub l i c resources fo r scienti f ic and technical research b e allocated on t h e basis o f t he l lcustomer-contractor" pr inc ip le . T h e customer (a government department), knowing what i t s needs are, commissions t h e appropr ia te research f rom t h e appropr ia te laboratories in t h e pub l i c o r p r i v a t e sector. A coro l lary o f t h i s was tha t t h e Di rectors o f government research establishments had t o s t r u c t u r e and j u s t i f y t h e i r programs in terms o f departmental objectives. Kogan, e t al, s tud ied in deta i l what happened in t h e DHSS t o t h e re la t ionship between the customers and the contractors, i.e., between the scientists and the pol icymakers as t h e y met in the annual attempt t o a l i gn sc ient i f ic resources and departmental requirements. The analysis showed t h a t t h e process o f col laboration between scientist and pol icymaker o r i g ina l l y envisaged by Rothschi ld in fact q u i c k l y deter iorated i n t o a form of adversa ry relat ion- sh ip in which t h e norms and values o f t he two d i f f e r e n t constituencies become polarized.

( 3 ) We a r e now in a pos i t ion t o move t o t h e n e x t h i g h e r viewpoint: po l icy analysis. Whereas scholars such as Mar t i n and l r v i n e focus t h e i r

480 Policy Studies Review, 3:3-4, Hay, 1984

at tent ion on the percept ions o f ind iv idual scientists, engineers and admin- is t ra tors , and whereas Kogan, e t al, concentrate on t h e relat ionships between functional groups, Peter deLeon (1980) d i rec ts h i s a t tent ion t o t h e relat ionships ex i s t i ng between sets o f inst i tu t ions. In a s tudy o f t he national nuclear reactor programs o f s ix countr ies, deLeon iden t i f i ed fou r factors which may have much wider appl icabi l i ty t o t h e development o f heur is t ics f o r t he evaluation o f innovation pol icy. The main conclusions o f h is s tudy were:

1. In each national program several important inst i tu t ional actors each were vying fo r center stage and each a t d i f f e ren t times hav ing g rea te r o r lesser roles t o p lay during the development and d i f f us ion o f t h e tech- nology.

2 . Because the re a re mul t ip le actors and each has his own pa r t i cu la r organizational goals, t he re a re mult iple objectives (wh ich may or may not be harmonious w i t h one another) simultaneously be ing pursued.

3. Each technological development can b e characterized by a number o f a t t r i bu tes tha t were in f luent ia l (by being present o r absent) in de f in ing the course o f technical development. In t h i s framework technological development i s based upon a series o f in teract ions between a set o f inst i tu t ional objectives and a set o f technological a t t r i bu tes which def ine and characterize these sane developments.

4. A changing set o f pol i t ical, economic, and technological condit ions may b e expected during the course o f a major technological development. Th i s suggests d i f f e ren t stages o f development may p resen t d i f f e r e n t problems. In par t icu lar , deLeon points o u t " the t rans i t i on between periods ( o f development) are especially c r i t i ca l because they suggest how readily t he development can move t o new and more appropr ia te objectives or i s bound t o outmoded ones.Il (deLeon, 1980:291) .

5. Final ly, t he re i s a close paral lel between t h e set o f relat ionships and the sor t o f continuous in teract ion between technological oppor tun i t ies and market need character iz ing innovat ion w i t h i n a business enterpr ise. As Rothwell and Zegveld (1981) pointed out , t he re i s a clear need fo r pol icy t o take on board what is known about technological innovation. Clearly, t he re i s p len ty o f room fo r f u r t h e r s t u d y o f t he ex ten t suc- cessful development and d i f f us ion o f technology i s determined by " the cooperative and continual involvement o f t h e developer, t h e vendor and the consumer."

CONCLUSIONS

The f i r s t conclusion wh ich should emerge f rom t h e discussion i s t ha t evaluation has two in ter re la ted aspects: t he use o f va r ious methods t o monitor t h e outcomes o f pub l i c policies and programs and t h e application o f some scale o f va lue t o determine whether t h e outcome i s o f bene f i t t o a g r o u p o r t o society as a whole. The evaluation question, "Has t h e pol icy made any d i f ference?" carr ies w i t h it t h e impl ic i t assumption that , i f it has had any ef fect a t al l , it has made mat ters b e t t e r o r worse. I f one accepts th i s d is t inct ion between monitoring and evaluat ing pol icy it i s clear that , as f a r as government policies fo r innovat ion a re concerned, the re has been more o f t h e former than o f t h e latter. T h e w o r k s o f Mar t i n and l r v i n e in

C i b b o d E v a l u a t i o n of Government P o l i c i e s f o r Innovat ion 481

t he resource allocation t rad i t ion, Kogan e t al in the organizational t rad i - tion, and deLeon in the pol icy analysis t rad i t i on a r e concerned mainly w i t h moni tor ing t h e outcomes o f pol icy, t hough the resu l t s have implications fo r evaluation.

One o f t he more promising approaches, as f a r as evaluation i s con- cerned, emerges from the work of Kogan e t al. They have analyzed the process o f implementinq a pol icy. The re has been much talk, pa r t i cu la r l y in the USA and t h e UK, of o r i en t i ng research t o national needs. But , as far as I am aware, Kogan i s among t h e few who have t r i e d to observe the problems o f implementing programs t o at ta in such an objective. Kogan and h is colleagues have discovered t h a t t h e re levant var iab les a re n o t j us t numerical measures of o u t p u t but the mode o f decisionmaking. I f Roths- ch i l d has failed t o make a dif ference in B r i t i s h RED policy, it is because of inst i tu t ional factors--because o f t he sha rp funct ional d i v i s ion w i th in the of f ice o f t he Chief Scientists whereby sc ient i f ic a n d pol icy mat ters a re kep t isolated. Koganls research o f fe rs some ins igh ts i n t o the question o f what d i f ference Rothschi ld policies have made in t h e commissioning o f research but, more impor tant ly , he g i ves some idea o f what steps might be taken t o improve i t s implementation.

Koganls research i s a test o f de leon ' s model. To evaluate innovation policies, one assumes tha t mul t ip le actors and mul t ip le objectives are involved: cent ra l and local government, f i rms, banks, etc. B u t de leon 's w o r k g i ves u s t h e f u r t h e r i ns igh t t h a t actors and objectives s h i f t over time in r o u g h re la t ionship w i t h t h e stage o f development o f t he project. Seen in t h i s context, t he weaknesses and possible dangers o f a "snapshot analysis" emerge: recommendations based upon the in teract ion o f objectives and actors a t a g i v e n po in t in time--say ea r l y in a project--may no t b e re levant a t a la ter stage. D u r i n g t h e implementation process d i f f e ren t combinations o f actors and objectives may become dominant.

We a re now in a posit ion t o suggest t h e problems o f monitoring and evaluating pol icy a re questions about implementation. Much o f t he l l tera- t u r e and pract ice o f pol icymaking takes a Ittop-downat view o f t he process and t rea ts admin is t ra tors as agents or implementors f o r pol icymakers while tending t o p lay down power relat ions, conf l ic ts o f in terest , and value dif ferences between ind iv iduals and agencies responsible fo r policymaking, In o the r words, implementation i s conceptualized as "putting pol icy in to effect." The problems o f implementation a re f rom t h i s "managerial per- spective" def ined in terms o f coordination, cont ro l , and obta in ing compli- ance w i t h pol icy.

Bu t , as deLeon and Kogan show, t h i s i s a fau l t y view o f implemen- tat ion: pol icy does n o t implement i tse l f , and at tent ion needs t o b e d i rected a t t h e processes whereby pol icy i s t rans lated i n t o action and a t i n f l uenc ing these processes. Implementation may b e viewed as a process o f in teract ion between t h e se t t i ng o f goals and act ion geared t o achiev ing them. B u t one can go f u r t h e r a n d r e g a r d implementation as the ab i l i t y t o forge l i nks in t h e causal chain leading f rom objectives t o action. T h i s i s n o t a one-time, single e f fo r t , but a cont inual and col lect ive one. As Dunsi re (1978) argues, implementation i s a complicated job of:

Creat ing and e s t a b l i s h i n g l i n k s between separate bodies-making a chain, n o t j u s t us ing one; a chain which, i n p r i n c i p l e , might be made up o f sets o f ( i n s t i t u t i o n s o r groups) f o r each implementa- t i o n exercise, though the more o f t e n a chain i s ' f o rged ' , the more e a s i l y i t i s ' f o rged ' the next t ime, u n t i l i t may be v i r - t u a l l y permanent. (Dunsire, 1978: 131)

482 Policy Studies Review, 3:3-4, Hay. 1984

I am suggesting the t h r u s t o f evaluation studies should b e d i rected n o t only a t monitoring outcomes but a t a f fect ing the process o f irnplementa- tion--the fo rg ing o f l inks between d i f fe ren t inst i tu t ions and groups. Th is i s par t icu lar ly cruc ia l f o r innovation pol icy, where t h e pr inc ipa l inst i - tu t ions which are supposed t o be beneficiaries o f the pol icy are corpo- rations. Here, the implementation process is crucial. One plausible explanation of the fa l te r ing performance o f government innovation policies outl ined above may be traceable t o the fa i lure t o " forge the appropr ia te l inks" w i t h the indus t r ies and indust r ia l is ts concerned. Viewed from the outside, it sometimes appears as i f these policies were in tended t o imple- ment themselves.

REFERENCES

Allen, T.J., e t al. (1978) . Government inf luence o n the process o f innovation in Europe and Japan. Research Policy, I , pp. 124-149.

deLeon, P. (1980) . comparative technology a n d pub l ic pol icy: The devel- opment o f the nuclear power reactor in s ix countr ies. Policy Sciences,

Dunsire, A. (1978) . Implementation in bureaucracy: The execution process, Volume 1. London: Mar t in Robertson.

Cardiner, N. ( 1 975) . Economics of industrial subsidies. HMSO. Depart- ment o f Indus t ry .

I rv ine, J., Mart in, B., & Schwarz, M. (1981, Ju ly ) . Government sup- port for industrial research in Norway: A SPRU report . A repor t prepared f o r t h e lndustriforskningsutvalget by the Science Policy Research Unit, Un ivers i ty o f Sussex.

Kogan, M., e t al. ( 1 9 8 1 ) . Government's commissioning of research: A case study. Department o f Government, Brunet Univers i ty .

Kogan, M., & Henkel, M. (1983) Government and research: The Roths- child experiment in a government deportment. London: Iieinemann.

Martin, 6.. t I rv ine , J. (1983) . Assessing basic research: Some par t ia l indicators of scienti f ic progress in radio astronomy. Research Policy, 7 2 ( 2 ) , pp. 61-90.

Nelson, R. R. (1974) . Intel lectual izing about the moon-ghetto metaphor: A s tudy of the c u r r e n t malaise o f rat ional analysis. Policy Sciences,

Rothwell, R., & Zegveld, W. (1981) . Industrial innovation and public

11, pp. 285-307.

5 ( 4 ) , pp. 375-414.

policy. London: Frances Pinter.