watson and open source tools
DESCRIPTION
Presented at BJUG, 5/8/2012 by Ivan Portilla IBM Watson is a reasoning system with a question and answer front end that processes natural language coming from both structured and unstructured data. Watson additionally incorporates analytics from which the system learns to derive answer confidence and scoring. We will discuss the Watson System and some of its key foundations that came from the Open Source Apache Software Foundation. We will share the lessons learned of using Open source technologies including UIMA, Derby, Hadoop and Tomcat in Watson. We will explain how the primary (shallow) search was built with Apache Lucene and how the team followed Agile best practices for its Software development efforts.TRANSCRIPT
![Page 2: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/2.jpg)
If I have seen further it is by standing on the shoulders of giants.
Isaac Newton, Letter to Robert Hooke, February 5, 1675
2Sunday, May 20, 12
![Page 3: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/3.jpg)
3
ObjectivesBy the end of this session, you
should be able to:üDescribe the main characteristics
of Watson QA system.ü Identify the key open source SW
used in Watson.üRecognize examples of Agile
development best practices.
3Sunday, May 20, 12
![Page 4: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/4.jpg)
Disclaimers
4Sunday, May 20, 12
![Page 5: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/5.jpg)
Disclaimer 1ü This presentation represents the view of the
author and does not represent the view of IBM. ü All opinions expressed in this presentation are
strictly of the speaker, and do NOT represent those of IBM, IBM management, or anyone else.
ü IBM and IBM (logo) are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries.
5Sunday, May 20, 12
![Page 6: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/6.jpg)
Disclaimer 2I (We) do not work for the Watson team.
6Sunday, May 20, 12
![Page 7: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/7.jpg)
Let’s Play Jeopardy
7Sunday, May 20, 12
![Page 8: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/8.jpg)
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
7Sunday, May 20, 12
![Page 9: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/9.jpg)
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
7Sunday, May 20, 12
![Page 10: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/10.jpg)
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
7Sunday, May 20, 12
![Page 11: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/11.jpg)
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north.
7Sunday, May 20, 12
![Page 12: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/12.jpg)
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north.
7Sunday, May 20, 12
![Page 13: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/13.jpg)
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north.
Geography: Chile shares its longest land border with this country
7Sunday, May 20, 12
![Page 14: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/14.jpg)
Natural Language Processing
Understanding natural language is hard!
8Sunday, May 20, 12
![Page 15: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/15.jpg)
Watson educa@on
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
9Sunday, May 20, 12
![Page 16: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/16.jpg)
A Brief History of Watson
10Sunday, May 20, 12
![Page 17: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/17.jpg)
A Brief History of Watson§ Deep Blue Ended in 1997§ Looking for a new research challenge§ 2004, IBM Research manager Charles Lickel,
§ Ken Jennings§ Started in 2005
• David Ferrucci• DeepQA in 2007• Won Jeopardy Match, Feb 2011
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
11Sunday, May 20, 12
![Page 18: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/18.jpg)
A Brief History of Watson§ Deep Blue Ended in 1997§ Looking for a new research challenge§ 2004, IBM Research manager Charles Lickel,
§ Ken Jennings§ Started in 2005
• David Ferrucci• DeepQA in 2007• Won Jeopardy Match, Feb 2011
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
11Sunday, May 20, 12
![Page 19: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/19.jpg)
Baseline
12/2007
8/2008
5/2009
10/2009
11/2010
12/2008
5/2008
4/2010
Prec
isio
n
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771712Sunday, May 20, 12
![Page 20: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/20.jpg)
What is Watson?
üUnderstands natural language.
üGenerates & evaluates hypothesis for beAer outcomes.
üAdapts & learns from user selec@ons and responses.
hAp://www.ibm.com/innova@on/us/watson/
13Sunday, May 20, 12
![Page 21: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/21.jpg)
Watson metricsDevelopment Team: 25 peopleProject Duration: 4 years
Hardware: 90 IBM Power-750 servers 2880 Power7 cores @ 80+ TFLOPS20 TB Disk, 16 TB RAM (memory)10 Gbps network
hAp://na11.apachecon.com/talks/19932
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771714Sunday, May 20, 12
![Page 22: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/22.jpg)
Open Source Software
15Sunday, May 20, 12
![Page 23: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/23.jpg)
OSS -‐ Linux
hAp://video.linux.com/videos/linuxcon-‐vancouver-‐day-‐2-‐1/
16Sunday, May 20, 12
![Page 24: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/24.jpg)
Open Source too
hAp://@.arc.nasa.gov/opensource/hAp://ocw.mit.edu/index.htm
17Sunday, May 20, 12
![Page 25: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/25.jpg)
How does it work?
18Sunday, May 20, 12
![Page 26: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/26.jpg)
Learning To Rank (Basic Architecture)
HypothesisEvidenceScoring
Keywords
Watson by R.Yates
19Sunday, May 20, 12
![Page 27: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/27.jpg)
Watson Architecture
hAp://en.wikipedia.org/wiki/Watson_(computer)
20Sunday, May 20, 12
![Page 28: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/28.jpg)
Who is the 44th President of the United States?
21Sunday, May 20, 12
![Page 29: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/29.jpg)
Who is the 44th President of the United States?
Who is the 44th President of the United States?
Watson by R.Yates
22Sunday, May 20, 12
![Page 30: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/30.jpg)
Who is the 44th President of the United States?
Who is the 44th President of the United States?
'Who' is the '44th' 'President' of the 'United States'?
Lexical Answer Type
→ Person
Focus
Can be replaced by the correct answer to make a true statement
Keywords
23Sunday, May 20, 12
![Page 31: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/31.jpg)
Who is the 44th President of the United States?
Question
HypothesisGeneration
QuestionAnalysis
Keywords:44th President United States
Watson by R.Yates
24Sunday, May 20, 12
![Page 32: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/32.jpg)
Who is the 44th President of the United States?
Question
HypothesisGeneration
QuestionAnalysis
Keywords:44th President United States
25Sunday, May 20, 12
![Page 33: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/33.jpg)
Question
PrimarySearch
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
26Sunday, May 20, 12
![Page 34: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/34.jpg)
Question
PrimarySearch
HypothesisGeneration
QuestionAnalysis
Barack ObamaGeorge W. BushHarvard Law SchoolIllinois
Who is the 44th President of the United States?
Watson by R.Yates
27Sunday, May 20, 12
![Page 35: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/35.jpg)
ScoringScoring
Question
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Who is the 44th President of the United States?Barack Obama
Who is the 44th President of the United States?George W. Bush
Who is the 44th President of the United States?Harvard Law School
Who is the 44th President of the United States?Illinois
AnswerScoring
28Sunday, May 20, 12
![Page 36: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/36.jpg)
AnswerScoringAnswer
Scoring
ScoringScoring
Question
ContextualAnswer Scoring
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Who is the 44th President of the United States?Barack Obama
Who is the 44th President of the United States?George W. Bush
Who is the 44th President of the United States?Harvard Law School
Who is the 44th President of the United States?Illinois
AnswerScoringAnswer
Scoring
Who is the 44th President of the United States?
→ Person
Is Barack Obama a Person? .90Is George W. Bush a Person? .90Is Harvard Law School a Person? .10Is Illinois a Person? .15
Watson by R.Yates
29Sunday, May 20, 12
![Page 37: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/37.jpg)
ScoringScoring
Question
Unstructured Informa@on Management Applica@ons -‐ UIMA
Scoring
HypothesisGeneration
QuestionAnalysis
ContextualAnswer Scoring
Who is the 44th President of the United States?
Barack Obama is the 44th President of the United StatesGeorge W. Bush is the 44th President of the United StatesHarvard Law School is the 44th President of the United StatesIllinois is the 44th President of the United States
30Sunday, May 20, 12
![Page 38: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/38.jpg)
ScoringScoring
Question
Unstructured Informa@on Management Applica@ons -‐ UIMA
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Barack Obama is the 44th President of the United StatesGeorge W. Bush is the 44th President of the United StatesHarvard Law School is the 44th President of the United StatesIllinois is the 44th President of the United States
Barack Hussein Obama II (i/bəәˈrɑːk huːˈseɪn oʊˈbɑːməә/; born August 4, 1961) is the 44th and current President of the United States.
George Walker Bush (born July 6, 1946) is an American politician who served as the 43rd President of the United States from 2001 to 2009 and the 46th Governor of Texas from 1995 to 2000.
Barack Obama .95George W. Bush .80Harvard Law School .05Illinois.10
Watson by R.Yates
31Sunday, May 20, 12
![Page 39: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/39.jpg)
ScoringScoring
QuestionTrained Models
EvidenceRetrieval
Unstructured Informa@on Management Applica@ons -‐ UIMA
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Candidate Answer Answer Scoring
Evidence retrieval & scoring
Confidence
Barack Obama 0.90 0.90 .95
George W. Bush 0.90 0.80 .65
Harvard Law School 0.10 0.05 .05
Illinois 0.15 0.10 .10
Watson by R.Yates
32Sunday, May 20, 12
![Page 40: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/40.jpg)
. . .
Models
Answer & Confidence
Question
Evidence Sources
Models
Models
Models
Models
ModelsPrimarySearch
CandidateAnswerGeneration
HypothesisGenera@on
Hypothesis and Evidence Scoring
Final Confidence Merging & Ranking
Answer Sources
Ques@on & Topic Analysis
Answer Scoring
Evidence Retrieval & Scoring
Learned Modelshelp combine and weigh the Evidence
HypothesisGenera@on
Hypothesis and Evidence Scoring
Ques@onDecomposi@on
Balance& Combine
Merging &Ranking
Synthesis
DeepQAMassively Parallel Probabilistic Evidence-Based Architecture
ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker
33Sunday, May 20, 12
![Page 41: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/41.jpg)
OSS in Watson 1.0
34Sunday, May 20, 12
![Page 42: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/42.jpg)
OSS in Watson
üUIMA, UIMA-‐ASüHadoop, Map ReduceüLucene, Indri
ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker
35Sunday, May 20, 12
![Page 44: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/44.jpg)
UIMA-‐Asynchronous Scaleout
UIMA AS provides more flexible and powerful scale out capability.
Innovate 2011, How Does It Work? The Architecture of Watson. Grady Booch
37Sunday, May 20, 12
![Page 45: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/45.jpg)
Think Hadoop
A framework for storing & processing big data.
üUp to 4,000 machinesüUp to 20 PB
High reliability done in soiware:üAutomated failover for data & computa@onüImplemented in Java
hAp://hadoop.apache.org/mapreduce/
38Sunday, May 20, 12
![Page 46: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/46.jpg)
Map reduce
hAp://hadoop.apache.org/mapreduce/
39Sunday, May 20, 12
![Page 47: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/47.jpg)
UIMA pipelines in Hadoop
hAp://blogs.apache.org/founda@on/entry/apache_innova@on_bolsters_ibm_s41
HadoopDistributedFileSystem
Input
.
.
.
~5000 “splits”
Mapper
Mapper
Mapper
Mapper
.
.
.
~50 -‐ 100 “mappers”
Mul@ple threads, each runing a UIMA pipeline
ThreadThread
ThreadThread
. . .
ThreadThread
ThreadThread
. . .
ThreadThread
ThreadThread
. . .
ThreadThread
ThreadThread
. . .
Reducer
Reducer
.
.
.Shuffl
e/Sort
HadoopDistributedFileSystem
Output
~400-‐800 threads
40Sunday, May 20, 12
![Page 48: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/48.jpg)
Architecture
hAp://lucene.apache.org
41Sunday, May 20, 12
![Page 49: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/49.jpg)
Indri &Lemur
Indri is a text search engine developed at Umass & CMU. Indri is part of the Lemur project.
hAp://lemurproject.org/indri/
42Sunday, May 20, 12
![Page 50: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/50.jpg)
© 2011 IBM Corporation
Watson answers in 2-‐6 seconds
Ques@on100s Possible Answers
1000’s of Pieces of Evidence
Mul@ple Interpreta@ons
100,000’s scores from many simultaneous Text Analysis Algorithms100s sources
. . .
HypothesisGenera@on
Hypothesis and Evidence Scoring
Final Confidence Merging & RankingSynthesisQues@on &
Topic AnalysisQues@onDecomposi@on
HypothesisGenera@on
Hypothesis and Evidence Scoring Answer &
Confidence
ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker
43Sunday, May 20, 12
![Page 51: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/51.jpg)
Other OSS in Watson
hAps://www.ibm.com/developerworks/mydeveloperworks/blogs/InsideSystemStorage/entry/ibm_watson_how_to_build_your_own_watson_jr_in_your_basement7?lang=en
44Sunday, May 20, 12
![Page 52: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/52.jpg)
J-‐Archive data
hAp://www.j-‐archive.com/showgame.php?game_id=3577
45Sunday, May 20, 12
![Page 53: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/53.jpg)
Development Process
üWar room setting with continuous collaboration.
üWeekly integration.üResults driven with E2E
regression testing.
ü About 8,000 experimentsü10 GBs of test data/wk.üAgile development
Innovate 2011, How Does It Work? The Architecture of Watson. Grady Booch
46Sunday, May 20, 12
![Page 54: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/54.jpg)
Other OSS
hAp://www.apache.orghAp://manning.com/
47Sunday, May 20, 12
![Page 55: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/55.jpg)
Take AwayOSS is powerful and scalable enough for the Watson team, what about your project?
48Sunday, May 20, 12
![Page 56: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/56.jpg)
ResourcesIBM Journal of Research and DevelopmenthAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717IBM WatsonhAp://www.ibm.com/innova@on/us/watson/hAp://www.research.ibm.com/deepqa/index.shtml
NovahAp://www.pbs.org/wgbh/nova/tech/smartest-‐machine-‐on-‐earth.html
49Sunday, May 20, 12
![Page 57: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/57.jpg)
51
Review of Objectives
Now that you have completed this session, you are able to:
ü Describe the main characteristics of Watson QA system.ü Identify the key open source tools used in Watson.ü Recognize examples of Agile development best practices.
50Sunday, May 20, 12
![Page 58: Watson and Open Source Tools](https://reader033.vdocument.in/reader033/viewer/2022051322/545c96e1b0af9fb32c8b49c3/html5/thumbnails/58.jpg)
52
…any final questions ?
51Sunday, May 20, 12