digital concept builder

7/30/2019 Digital Concept Builder

1/74

DigitalConceptBuilder

True Natural Language Understanding

through a Conceptual LanguageUnderstanding Engine

ByRoy, Philippe| 11 Jan 2010

Win2KWinXPVS2008C++JavascriptWin32STLArchitectDevIntermediate

An article introducing an Artificial Intelligence technique that understands and manipulates

concepts from text.

See Also

More like this More by this author

Print Article

Twitter

Digg

Facebook

Del.icio.us

Reddit

Stumbleupon

Newsvine

Technorati

Mr. Wong

Yahoo!

Google

Windows Live

Send as Email

Add to your CodeProject bookmarks

Discuss this article

13

Article Browse Code StatsRevisions (5)

4.78 (43 votes)

1 23 4 5

Sponsored Links

Is your email address OK? You are signed up for our newsletters but your emailaddress is either unconfirmed, or has not been reconfirmed in a long time. Please clickhereto
http://www.codeproject.com/script/Membership/View.aspx?mid=1078443http://www.codeproject.com/script/Membership/View.aspx?mid=1078443http://www.codeproject.com/script/Membership/View.aspx?mid=1078443http://www.codeproject.com/info/search.aspx?aidlst=23http://www.codeproject.com/info/search.aspx?aidlst=77http://www.codeproject.com/info/search.aspx?aidlst=87http://www.codeproject.com/info/search.aspx?aidlst=110http://www.codeproject.com/info/search.aspx?aidlst=118http://www.codeproject.com/info/search.aspx?aidlst=118http://www.codeproject.com/info/search.aspx?artkw=True+Natural+Language+Understanding+through+a+Conceptual+Language+Understanding+Enginehttp://www.codeproject.com/info/search.aspx?artkw=True+Natural+Language+Understanding+through+a+Conceptual+Language+Understanding+Enginehttp://www.codeproject.com/script/Articles/MemberArticles.aspx?amid=1078443http://www.codeproject.com/script/Articles/MemberArticles.aspx?amid=1078443http://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#_commentshttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#_commentshttp://www.codeproject.com/script/Articles/ViewDownloads.aspx?aid=51026http://www.codeproject.com/script/Articles/ViewDownloads.aspx?aid=51026http://www.codeproject.com/script/Articles/Statistics.aspx?aid=51026http://www.codeproject.com/script/Articles/ListVersions.aspx?aid=51026http://www.codeproject.com/script/Membership/SendConfirmRequest.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Membership/SendConfirmRequest.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Membership/SendConfirmRequest.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Bookmarks/Add.aspx?obid=51026&obtid=2&action=AddBookmark&bio=truehttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/script/Membership/SendConfirmRequest.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Articles/ListVersions.aspx?aid=51026http://www.codeproject.com/script/Articles/Statistics.aspx?aid=51026http://www.codeproject.com/script/Articles/ViewDownloads.aspx?aid=51026http://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#_commentshttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#_commentshttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx?display=Printhttp://www.codeproject.com/script/Articles/MemberArticles.aspx?amid=1078443http://www.codeproject.com/info/search.aspx?artkw=True+Natural+Language+Understanding+through+a+Conceptual+Language+Understanding+Enginehttp://www.codeproject.com/info/search.aspx?aidlst=118http://www.codeproject.com/info/search.aspx?aidlst=118http://www.codeproject.com/info/search.aspx?aidlst=110http://www.codeproject.com/info/search.aspx?aidlst=110http://www.codeproject.com/info/search.aspx?aidlst=87http://www.codeproject.com/info/search.aspx?aidlst=87http://www.codeproject.com/info/search.aspx?aidlst=77http://www.codeproject.com/info/search.aspx?aidlst=77http://www.codeproject.com/info/search.aspx?aidlst=23http://www.codeproject.com/info/search.aspx?aidlst=23http://www.codeproject.com/script/Membership/View.aspx?mid=1078443


2/74

have a confirmation email sent so we can confirm your email address and start sending you

newsletters again. Alternatively, you canupdate your subscriptions.

/w EPDwUKMTAy

/w EWCALr89jXA

Download source - 831 KB

Download executable (Win32) - 894 KB

Download non-abridged dictionary - 2.2 MB

Introduction
http://www.codeproject.com/script/Membership/Subscribe.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Membership/Subscribe.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/script/Membership/Subscribe.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspxhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderSources.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderSources.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderSources.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderExecuteable.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderExecuteable.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderExecuteable.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/DigitalConceptBuilderSources.ziphttp://www.codeproject.com/script/Membership/Subscribe.aspx?rp=%2fKB%2frecipes%2fDigitalConceptBuilder.aspx


3/74

People spend a significant amount of time and energy at putting their thoughts into

writing. So, it is time for a technology to concentrate on automating the retrieval and

manipulation of these thoughts in such a way that the original meaning is not lost in theprocess. To that end, a means by which concepts are reconstructed and then manipulated

from natural language is required. That goal cannot be achieved by a diminutive form of

natural language understanding that limits itself to word-spotting or a superficial associationof semantic blocks, but rather by a process that mimics the functions of the human brain in its

processing and outcome. This article exposes a novel means by which such processing is

achieved through a Conceptual Language Understanding Engine (CLUE).

The dominant part of the word "recognition" is "cognition", which is defined in the Oxford

dictionary as "the mental acquisition of knowledge through thought, experience, and the

senses". The present approach uses techniques that encapsulate most aspects associated with

a cognitive approach to recognition.

Communication is a procedural sequence that involves the following processes:

a "de-cognition" process - producing a syntactic stream representing the cognitive

aspect to communicate.

a transporting process - putting such syntactic stream on a medium.

a perceptive process - senses acquiring the syntactic stream.

a "re-cognition" process - rebuilds the original cognitive substance that hopefully has

not been lost throughout the syntactic, transporting, and perceptive processing.

Because a full cycle language analysis involving natural language understanding requires

putting the concept back in its original form, without incurring any loss of the conceptual

manipulations that can be achieved following a "de-cognition" and "re-cognition" processes,

the conceptual aspect of language cannot and must not be overlooked. Only when a

conceptual dimension to speech analysis becomes available will the syntactic aspect of

processing language becomes unconstrained. That is, syntactic analysis will limit to a

required transient step for communications to fit on a medium. Only then the limitations that

we have endured until today related to natural language understanding and processing will

start to fade.

The reward with a CLUE is the ability to abstract the written syntactic forms from conveyed

concepts - the words used to communicate become irrelevant as soon as the concept is

calculated - while maintaining the ability to intelligently react to recognized content. It

further shifts the final responsibility of disambiguation to the conceptual analysis layer,instead of the phonetic and syntactic layers as it has typically been done to this day. That

frees the person communicating from the obligation of using predetermined sequences of

words or commands in order to be successfully understood by an automated device. In an era

where automated devices have become the norm, the bottleneck has shifted to the inability of

technology to deal effectively with the human element. People are not comfortable with a set

of syntactic related rules that appear to them as counter-natural in relation to their conceptual

natural abilities.

Background


4/74

This article is the third of a sequence. In the first article, "Implementing a std::map

Replacement that Never Runs Out of Memory and Instructions on Producing an ARPA

Compliant Language Model to Test the Implementation", the memory management technique

used within the code-base provided in the present article is exposed. As a result of that, you

shall observe there is only one delete call throughout the entire code-base, and you are

certain there is no memory leak within the said C++ code. In the second article, "TheBuilding of a Knowledge Base Using C++ and an Introduction to the Power of Predicate

Calculus," predicate calculus is introduced, but falls short of processing natural language

input. Although it also exposes techniques in order to infer from a knowledge base - not used

in this article, but could easily be adapted to do so - the article covers basic predicate calculus

techniques that are widely used in this article. A reading of these two articles will help you

gain a better understanding of the basis upon which this code is built. Although Conceptual

Speech Recognition is used to interpret speech (sound), for the sake of simplicity and

demonstration, this article limits itself to textual content input. A software implementation of

this technology can be referred to as a Conceptual Language Understanding Engine, or

"CLUE" for short. The approach to process speech recognition is comparable to what is

exposed here, but with a couple of software engineering twists in order to include furtherbiases from speech through the integration of a Hidden-Markov-Model that is not discussed

in this article.

Using the Code

The current project builds under Visual Studio 2008. It is composed of about 10,000 lines of

C++ source code, and about 500 lines of JavaScript code that is processed byGoogle V8.

The main components of the code are divided as follows:

The dictionary: A 195,443 words, 222,048 parts-of-speech dictionary, held in a 2.1

MB file, which can return information on spelling almost instantly. [IndexStructure.h,

StoredPOSNode.h, StoredPOSNode.cpp, DigitalConceptBuilder::BuildDictionary]

Tokenization: Transforming a stream of natural language input into tokens based on

content included into a dictionary. [IndexStructure.h, POSList.h, POSList.cpp,

POSNode.h, POSNode.cpp, IndexInStream.h, DigitalConceptBuilder::Tokenize]

Syntactic Analysis: The algorithms to extrapolate complex nodes, such as SENTENCE,

from atomic nodes, such as NOUN, ADJECTIVE, and VERB obtained from the dictionary

following the tokenization phase. [SyntaxTransform.txt, POSTransformScript.h,

POSTransformScript.cpp, Permutation.h, Permutation.cpp, POSList.h, POSList.cpp,

POSNode.h, POSNode.cpp, Parser.h, Parser.cpp, BottomUpParser.h,BottomUpParser.cpp, MultiDimEarleyParser.h, MultiDimEarleyParser.cpp]

Conceptual Analysis: The building of concepts based on syntactic organizations, and

howGoogle V8, the JavaScript engine, is integrated into the project. [Conceptual

Analysis/*.js, Conceptual Analysis/Permutation scripts/*.js, Conceptual Analysis/POS

scripts/*.js, POSList.h, POSList.cpp, POSNode.h, POSNode.cpp, Predicate.h,

Predicate.cpp, JSObjectSupport.h, JSObjectSupport.cpp, POSNode::BuildPredicates]

Execution of the Code
http://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://code.google.com/p/v8/http://code.google.com/p/v8/http://code.google.com/p/v8/http://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#dictionaryhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#dictionaryhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#tokenizationhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#tokenizationhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#syntactic_analysishttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#syntactic_analysishttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#conceptual_analysishttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#conceptual_analysishttp://code.google.com/p/v8/http://code.google.com/p/v8/http://code.google.com/p/v8/http://code.google.com/p/v8/http://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#conceptual_analysishttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#syntactic_analysishttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#tokenizationhttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder.aspx#dictionaryhttp://code.google.com/p/v8/http://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://www.codeproject.com/KB/cpp/conceptual_dependency.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspx


5/74

Execution of the code as provided with this article runs the test cases specified in

SimpleTestCases.txt. Each block within curly brackets defines a scope to run, where CONTENTis the natural language stream to analyze.

Each variable definition prior to the test case sentences can also be inserted within curly

bracket scopes to override its value. For example, to enable only the first test case, thefollowing change is possible:

Collapse...#-------------------------------------------------------------------

# ENABLED# _______## Possible values: TRUE, FALSE#

# TRUE: enable the test cases within that scope.## FALSE: disable the test cases within that scope.

ENABLED = FALSE

#-------------------------------------------------------------------

{ENABLED = TRUECONTENT = Is a red car a car?ID = CAR1

}

...

Executing the test cases, as available from within the zip files attached, results in the

following output. The remainder of this article exposes the approach, philosophies, and

techniques used in order to transition from the Natural Language input from these test-cases

to concepts and responses as exposed here.

CollapseEvaluating: "Is a red car a car?" (ID:CAR1)

YES:

DO[ACTION:RECEIVEINPUTMOOD:INTEROGATIVEOBJECT:DO[OPERATION:IS

VALUE1:PP[CLASS:CARCOLOR:REDQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]

VALUE2:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]

Total time: 250 msSyntactic: 63 ms

Conceptual: 187 ms.
http://www.codeproject.com/KB/recipes/DigitalConceptBuilder/SimpleTestCases.txthttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/SimpleTestCases.txthttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/SimpleTestCases.txt


6/74

Evaluating: "Is a red car the car?" (ID:CAR2)

MAYBE:

DO[ACTION:RECEIVEINPUT

MOOD:INTEROGATIVEOBJECT:DO[OPERATION:ISVALUE1:PP[CLASS:CAR

COLOR:REDQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]

VALUE2:PP[CLASS:CARDETERMINED:TRUEQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]

Total time: 2 sec (2235 ms)Syntactic: 31 msConceptual: 2 sec (2204 ms).

Evaluating: "Is the red car a car?" (ID:CAR3)

YES:


VALUE1:PP[CLASS:CARCOLOR:REDDETERMINED:TRUEQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]



Conceptual: 172 ms.

Evaluating: "Is the red car red?" (ID:CAR4)

YES:


VALUE1:PP[CLASS:CARCOLOR:REDDETERMINED:TRUEQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]

VALUE2:PP[COLOR:RED]]]

Total time: 219 ms


7/74

Syntactic: 31 msConceptual: 188 ms.

Evaluating: "The car is red" (ID:CAR5)

No inquiry to analyze here:

DO[ACTION:RECEIVEINPUTMOOD:AFFIRMATIONOBJECT:PP[CLASS:CAR

COLOR:REDDETERMINED:TRUEQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]


Conceptual: 532 ms.

Evaluating: "Is a red car blue?" (ID:CAR6)

NO:



VALUE2:PP[COLOR:BLUE]]]

Total time: 2 sec (2453 ms)Syntactic: 31 ms

Conceptual: 2 sec (2422 ms).

Evaluating: "Is a red car red?" (ID:CAR7)

YES:



VALUE2:PP[COLOR:RED]]]


Conceptual: 203 ms.

Evaluating: "Is a car or a boat a car?" (ID:CAR8)

YES:



8/74

MOOD:INTEROGATIVEOBJECT:DO[OPERATION:IS

VALUE1:PP[CLASS:OR[VALUE1:BOATVALUE2:CAR]

QUANTITY:1TYPE:VEHICLE

WHEELCOUNT:4]VALUE2:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]


Conceptual: 203 ms.

Evaluating: "Is a car an object that is not a car?" (ID:CAR9)

NO:


VALUE1:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]

VALUE2:AND[VALUE1:PP[QUANTITY:1TYPE:{DEFINED}]

VALUE2:NOT[VALUE:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]]]



Evaluating: "Is a boat an object that is not a car?" (ID:CAR10)

YES:


VALUE1:PP[CLASS:BOATQUANTITY:1TYPE:VEHICLE]


VALUE2:NOT[VALUE:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]]]


Conceptual: 250 ms.

Evaluating: "Is an object that is not a car a boat?" (ID:CAR11)


9/74

MAYBE:


VALUE1:AND[VALUE1:PP[QUANTITY:1

TYPE:{DEFINED}]VALUE2:NOT[VALUE:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]

VALUE2:PP[CLASS:BOATQUANTITY:1TYPE:VEHICLE]]]



Evaluating: "Is a car that is not red a car?" (ID:CAR12)

YES:


VALUE1:PP[CLASS:CARCOLOR:!REDQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]



Conceptual: 250 ms.

Evaluating: "Is a car an object that is a car or a boat?" (ID:CAR13)

MAYBE:



VALUE2:OR[VALUE1:PP[CLASS:BOATQUANTITY:1TYPE:VEHICLE]

VALUE2:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]]



10/74


Evaluating: "Is a red car an object that is a car or a boat?" (ID:CAR14)

MAYBE:



VALUE2:OR[VALUE1:PP[CLASS:BOATQUANTITY:1TYPE:VEHICLE]

VALUE2:PP[CLASS:CARQUANTITY:1

TYPE:VEHICLEWHEELCOUNT:4]]]]



Evaluating: "Is a car that is not red a car?" (ID:CAR15)

YES:





Conceptual: 297 ms.

Evaluating: "Is a red car not red?" (ID:CAR16)

NO:



VALUE2:PP[COLOR:!RED]]]


11/74



Evaluating: "Is a car a car that is not red?" (ID:CAR17)

MAYBE:



VALUE2:PP[CLASS:CARCOLOR:!REDQUANTITY:1TYPE:VEHICLE

WHEELCOUNT:4]]]



Evaluating: "Is a car that is not red a blue car?" (ID:CAR18)

MAYBE:



VALUE2:PP[CLASS:CARCOLOR:BLUEQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]



Evaluating: "Is a red car a car that is not red?" (ID:CAR19)

NO:



VALUE2:PP[CLASS:CARCOLOR:!RED


12/74

QUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]



Evaluating: "Is an object that is not a car a car?" (ID:CAR20)

NO:



VALUE2:NOT[VALUE:PP[CLASS:CARQUANTITY:1

TYPE:VEHICLEWHEELCOUNT:4]]]VALUE2:PP[CLASS:CAR




Evaluating: "Is an object that is a car or a boat a car?" (ID:CAR21)

YES:


VALUE1:PP[CLASS:OR[VALUE1:BOATVALUE2:CAR]

QUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]



Conceptual: 640 ms.

Evaluating: "Is an object that is a car or a boat a red car?" (ID:CAR22)

MAYBE:


VALUE1:OR[VALUE1:PP[CLASS:BOAT

QUANTITY:1TYPE:VEHICLE]


13/74

VALUE2:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]

VALUE2:PP[CLASS:CARCOLOR:RED




Evaluating: "Is an object that is a car and a boat a red car?" (ID:CAR23)

MAYBE:


MOOD:INTEROGATIVEOBJECT:DO[OPERATION:ISVALUE1:PP[CLASS:AND[VALUE1:BOAT

VALUE2:CAR]QUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]

VALUE2:PP[CLASS:CARCOLOR:REDQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]



Evaluating: "Is a car an object that is a car and a boat?" (ID:CAR24)

NO:



VALUE2:AND[VALUE1:PP[CLASS:BOATQUANTITY:1TYPE:VEHICLE]

VALUE2:PP[CLASS:CARQUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]]]]



Evaluating: "Is an object that is a car and a boat a car?" (ID:CAR25)


14/74

YES:


VALUE1:PP[CLASS:AND[VALUE1:BOAT

VALUE2:CAR]QUANTITY:1TYPE:VEHICLEWHEELCOUNT:4]



Conceptual: 641 ms.

Done.

The Dictionary

The dictionary that is made available through the zip files is a file,

article_testcases_streams.txt, with partial content cumulated for the sole purpose to provide

the words and parts-of-speech necessary to run the test-cases successfully. That special step

was done in order to save readers of this article from a significant download in size since the

non-abridged dictionary is more than 8 MB in size.

The non-abridged dictionary is available for download byclicking here(2.2 MB). Unzip

the downloaded file and place streams.txtinto theDigitalConceptBuilderdirectory, at thesame level as article_testcases_streams.txt, to have it loaded the next time the program will

be launched.
http://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.ziphttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/FullDictionary.zip


15/74

The text files streams.txtand article_testcases_streams.txtare editable. The content in

streams.txtwas obtained over the years from a variety of sources, some of which are under

copyright as expressed in the "Licensing information.txt" file from the non-abridged

dictionary zip file.

The format used is as follows:

Collapse::

()

Example: December:dusembar:DATE(m_12).

The pronunciation is mandatory, yet is not used for the purpose of the current article. When

the topic of speech recognition is covered in a later article, its use will be necessary.

The text file holding the spellings and parts-of-speech is loaded into a three-way decisiontree, whose code can be found in the IndexStructure template, as described in the article:"Implementing a std::map Replacement that Never Runs Out of Memory and Instructions on

Producing an ARPA Compliant Language Model to Test the Implementation". A

representation of the three-way decision tree can be seen here holding the primary keys

"node", "do", "did", "nesting", "null", and "void", as illustrated below:

Holding the dictionary in such a structure provides the best possible performance whentokenizing, while ensuring that the tokens generated from the process have a corresponding
http://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspxhttp://www.codeproject.com/KB/audio-video/ARPALM.aspx


16/74

entry in the dictionary. Furthermore, the IndexStructure template used can hold data

partially on disk and partially in memory, making it a suitable medium for that purpose.

Tokenization

The goal of tokenization is to create a POSList object that holds spelling and correspondingparts-of-speech for the provided syntactic stream.

Following is the partial output from the first test-case executed with DEFINITIONNEEDED =

FALSE, OUTPUTTOKENS = TRUE with the non-abridged dictionary loaded.

CollapseEvaluating: "Is a red car a car?" (ID:CAR1)

Tokens before syntactic analysis:

[NOUN & "IS"], from 0, to: 1, bridge: 3, index: 0[VERB & "IS"], from 0, to: 1, bridge: 3, index: 0[AUX & "IS"], from 0, to: 1, bridge: 3, index: 0[NOUN & "A"], from 3, to: 3, bridge: 5, index: 1[PREPOSITION & "A"], from 3, to: 3, bridge: 5, index: 1[VERB & "A"], from 3, to: 3, bridge: 5, index: 1[DEFINITE_ARTICLE & "A"], from 3, to: 3, bridge: 5, index: 1[ADJECTIVE & "RED"], from 5, to: 7, bridge: 9, index: 2[NOUN & "RED"], from 5, to: 7, bridge: 9, index: 2[VERB & "RED"], from 5, to: 7, bridge: 9, index: 2[PROPER_NOUN & "RED"], from 5, to: 7, bridge: 9, index: 2[NOUN & "CAR"], from 9, to: 11, bridge: 13, index: 3[PROPER_NOUN & "CAR"], from 9, to: 11, bridge: 13, index: 3[NOUN & "A"], from 13, to: 13, bridge: 15, index: 4[PREPOSITION & "A"], from 13, to: 13, bridge: 15, index: 4[VERB & "A"], from 13, to: 13, bridge: 15, index: 4[DEFINITE_ARTICLE & "A"], from 13, to: 13, bridge: 15, index: 4[NOUN & "CAR"], from 15, to: 17, bridge: 18, index: 5[PROPER_NOUN & "CAR"], from 15, to: 17, bridge: 18, index: 5[PUNCTUATION & "?"], from 18, to: 18, bridge: 0, index: 6

Tokenization can appear as an easy task, but it has hidden difficulties of its own.

Tokenization must allow words to be included in other words (for example, 'all' and

'all-in-one').

While tokenizing, special provisions for numbers handling, which are not part of thedictionary, must be covered.

Allowed punctuation must be taken into account.

Punctuation that is not allowed must be ignored, unless it is part of a dictionary entry.

Any word from the resulting list of words in POSList must be readily accessible

based on its position in the stream and its part of speech to ensure that later processing

will not be negatively affected in regards to performance.

Collapse#ifndef __POSLIST_HH__#define __POSLIST_HH__

#include


17/74

#include

#include

using std::vector;

using std::map;using std::string;

#include "POSNode.h"

#include "shared_auto_ptr.h"

#include "IndexInStream.h"

class POSList;

class POSList

{ public:

// Constructor, pass duplicateDefense// to true if the POSList should filter out// apparent duplicates.POSList(bool duplicateDefense = false);

// AddToPOSList, call to add a POSNode to the list.virtual POSNode *AddToPOSList(shared_auto_ptr dNode);

// BrigdgeableNodes, call to acquire all POSNode// in the POSList object that respect the conditions// stipulated in dPOSNode at stream index positionvector

BrigdgeableNodes(int position, shared_auto_ptrdPOSNode);

// GetLowestStartPosition, returns the lowest// start position from all POSNode in the POSList.int GetLowestStartPosition();

// GetHighestEndPosition, returns the highest// end position from all POSNode in the POSList.int GetHighestEndPosition();

// AccumulateAll, accumulates all POSNode respecting// the condition stipulated in dPOSNode and sorting// the resulting vector based on sort,// all accumulated nodes need to respect the start position fromPos// (or -1 to ignore) and end position toPos (or -1 to ignore).vector AccumulateAll(

shared_auto_ptr dPOSNode,POSNode::ESortType sort = POSNode::eNoSort,int fromPos = -1, int toPos = -1);

// Output, outputs the POSNode accumulated into// the POSList at position pos (or -1 if the position is// irrelevant).void Output(int pos = -1);


18/74

// Count, returns how many POSNode are accumulated into thePOSList.

unsigned long Count();

// Clear, removes all POSNode from the POSList.virtual void Clear();

// ResetPOSNodeIteration, resets the tracer// to the first POSNode in the POSList.void ResetPOSNodeIteration();

// GetPositionInList, returns a copy// of the tracer for the current position in stream.virtual IndexInStream *GetPositionInList();

// GetNext, gets the next POSNode from the// tracer's position, returns true if one is available,// false otherwise.virtual bool GetNext(shared_auto_ptr &dPOSNode);

protected:virtual string GetLineOutput(shared_auto_ptr dNode);IndexInStream m_position;int m_lowestStart;int m_highestEnd;bool m_duplicateDefense;map m_uniqueEntries;int m_count;

};

#endif

In order to have a quick retrieval mechanism, the POSList class uses the m_position

member that is an IndexInStream instance. The IndexInStream template is usedin some cases when quick retrieval of an object, in our case, a POSNode, is needed based on a

position in the stream and a POSEntry type. The implementation of the IndexInStream

template follows:

Collapse#ifndef __INDEXSINSTREAM_H__#define __INDEXSINSTREAM_H__

#include

#include

using std::map;using std::vector;

#include "POSEntry.h"

template class IndexInStreamPosition{

public:IndexInStreamPosition();void Add(shared_auto_ptr dContent, POSEntry dPosEntry);

typedef typename mapcontainer_map_vector_type;


19/74

typedef typenamecontainer_map_vector_type::iterator iterator_map_vector_type;

container_map_vector_type m_content;};

template IndexInStreamPosition::IndexInStreamPosition() {}

template void IndexInStreamPosition::Add(shared_auto_ptr dContent, POSEntry dPosEntry)

{m_content[dPosEntry.GetValue()].push_back(dContent);

}

template class IndexInStream{

public:IndexInStream();void Reset();void Clear();

void Add(shared_auto_ptr dObject, int position, POSEntry dPos);bool GetNext(shared_auto_ptr &dPOSNode);vector ObjectsAtPosition(int position,

POSEntry dPOSEntry, int *wildcardPosition = NULL);protected:

typedef typename mapcontainer_map_index_type;

typedef typenamecontainer_map_index_type::iterator iterator_map_index_type;

typedef typename map container_map_vector_type;

typedef typename container_map_vector_type::iteratoriterator_map_vector_type;

container_map_index_type m_allSameEventPOSList;iterator_map_index_type m_iterator1;iterator_map_vector_type m_iterator2;int m_posVectorEntryIndex;

};

template IndexInStream::IndexInStream():m_posVectorEntryIndex(-1){

Reset();m_allSameEventPOSList.clear();

}

template void IndexInStream::Clear(){

Reset();}

template vectorIndexInStream::ObjectsAtPosition(int position,POSEntry dPOSEntry, int *wildcardPosition)

{vector dReturn;if (m_allSameEventPOSList.find(position) !=

m_allSameEventPOSList.end())

{


20/74

if (m_allSameEventPOSList[position]->m_content.find(dPOSEntry.GetValue()) !=

m_allSameEventPOSList[position]->m_content.end()){

dReturn =m_allSameEventPOSList[position]-

>m_content[dPOSEntry.GetValue()];}}if (wildcardPosition != NULL){

vector temp =ObjectsAtPosition(*wildcardPosition, dPOSEntry);

for (unsigned int i = 0; i < temp.size(); i++){

dReturn.push_back(temp[i]);}

}return dReturn;

}

template bool IndexInStream::GetNext(shared_auto_ptr&dObject){

while (m_iterator1 != m_allSameEventPOSList.end()){

while ((m_posVectorEntryIndex != -1) &&(m_iterator2 != m_iterator1->second.get()->m_content.end()))

{if (m_posVectorEntryIndex < (int)m_iterator2->second.size()){

dObject = m_iterator2->second[m_posVectorEntryIndex];m_posVectorEntryIndex++;return true;

}else{

m_iterator2++;m_posVectorEntryIndex = 0;

}}m_iterator1++;m_posVectorEntryIndex = 0;if (m_iterator1 != m_allSameEventPOSList.end()){

m_iterator2 = m_iterator1->second.get()->m_content.begin();}else{

m_posVectorEntryIndex = -1;}

}Reset();return false;

}

template void IndexInStream::Add(shared_auto_ptr dObject, int position, POSEntry dPos)

{

if (m_allSameEventPOSList.find(position) ==m_allSameEventPOSList.end())


21/74

{m_allSameEventPOSList[position] =

shared_auto_ptr(new IndexInStreamPosition());

}m_allSameEventPOSList[position]->Add(dObject, dPos);

}

template void IndexInStream::Reset(){

m_iterator1 = m_allSameEventPOSList.begin();m_posVectorEntryIndex = 0;if (m_iterator1 != m_allSameEventPOSList.end()){

m_iterator2 = m_iterator1->second.get()->m_content.begin();}else{

m_posVectorEntryIndex = -1;

}}

#endif

With the POSList class implemented, the tokenization is implemented as follows:

Collapseshared_auto_ptr DigitalConceptBuilder::Tokenize(string dContent,

string posNumbers,string posPunctuation,string punctuationAllowed,bool definitionNeeded)

{struct TokenizationPath{

TokenizationPath(shared_auto_ptr

dPosition,unsigned int dStartIndex): m_position(dPosition),

m_startIndex(dStartIndex) {}shared_auto_ptr

m_position;unsigned int m_startIndex;

};unsigned int dWordIndex = 0;shared_auto_ptr dReturn = shared_auto_ptr(new

POSList(true));string dNumberBuffer;vector activePaths;vector floatingBridges;int latestBridge = -1;for (unsigned int i = 0; i


22/74

{isAllowedPunctuation = true;latestBridge = i;

}}activePaths.push_back(TokenizationPath(

shared_auto_ptr(new IndexStructureNodePosition(m_POS_Dictionary.GetTopNode())), i));

for (unsigned int j = 0; j < activePaths.size(); j++){

if ((activePaths[j].m_position.get() != NULL) &&(activePaths[j].m_position->get() != NULL) &&(activePaths[j].m_position->get()->m_data != NULL))

{if ((i == dContent.length()) ||

(IsDelimitor(dContent.c_str()[i]))){

string dKey = activePaths[j].m_position->GetNode()-

>GetKey(); if ((i < dKey.length()) ||(IsDelimitor(dContent.c_str()[i - dKey.length() -

1]))){

StoredPOSNode *dPOS = activePaths[j].m_position->get();

for (int k = 0; k < kMAXPOSALLOWED; k++){

if (dPOS->m_pos[k] != -1){

shared_auto_ptr candidate =

POSNode::Construct("["+POSEntry::StatGetDescriptor(dPOS->m_pos[k]) + " & \"" + dKey + "\"]",

NULL,activePaths[j].m_startIndex, i-1, 0,(dPOS->m_data[k] != -1)?m_data[dPOS-

>m_data[k]]:"");if (PassedDefinitionRequirement(candidate,

definitionNeeded)){

POSNode *dNewNode = dReturn->AddToPOSList(candidate);

if (dNewNode != NULL){

floatingBridges.push_back(dNewNode);}latestBridge =

activePaths[j].m_startIndex;}

}}

}}

}if ((latestBridge != -1) && (((i < dContent.length()) &&

(!IsDelimitor(dContent.c_str()[i]))) ||(i == dContent.length()) || (isAllowedPunctuation)))

{bool atLeastOneAdded = false;


23/74

for (int l = (floatingBridges.size() - 1); l >= 0; l--){

if ((floatingBridges[l]->GetBridgePosition() == 0) &&(floatingBridges[l]->GetStartPosition() !=

latestBridge)){

atLeastOneAdded = true;floatingBridges[l]->SetWordIndex(dWordIndex);floatingBridges[l]-

>SetBridgePosition(latestBridge);floatingBridges.erase(floatingBridges.begin() + l);

}}if (atLeastOneAdded){

dWordIndex++;}if (isAllowedPunctuation){

shared_auto_ptr candidate =POSNode::Construct("[" + posPunctuation + " & \"" +dCharStr + "\"]", NULL, i, i, 0);

if (PassedDefinitionRequirement(candidate,definitionNeeded))

{POSNode *dNewNode = dReturn-

>AddToPOSList(candidate);if (dNewNode != NULL){

floatingBridges.push_back(dNewNode);}latestBridge = i;

}}else{

latestBridge = -1;}

}if (i == dContent.length()){

break;}shared_auto_ptrdNewPosition = m_POS_Dictionary.ForwardNodeOneChar(activePaths[j].m_position, toupper(dContent.c_str()[i]));

if (dNewPosition.get() != NULL){

activePaths[j].m_position = dNewPosition;}else{

activePaths[j].m_position->Clear();}

}if ((posNumbers.length() > 0) && ((dContent.c_str()[i] >= '0') &&

(dContent.c_str()[i] 0))))

{

if ((i == 0) || (dNumberBuffer.length() > 0) ||(IsDelimitor(dContent.c_str()[i-1])))


24/74

{dNumberBuffer += dContent.c_str()[i];

}}else if (dNumberBuffer.length() > 0){

shared_auto_ptr candidate =POSNode::Construct("["+posNumbers + " & \"" +dNumberBuffer + "\"]", NULL, i - dNumberBuffer.length(), i -

1, 0);if (PassedDefinitionRequirement(candidate, definitionNeeded)){

POSNode *dNewNode = dReturn->AddToPOSList(candidate);if (dNewNode != NULL){

floatingBridges.push_back(dNewNode);}latestBridge = i - dNumberBuffer.length();dNumberBuffer = "";

}}for (int j = (activePaths.size() - 1); j >= 0; j--){

if ((activePaths[j].m_position.get() == NULL) ||(activePaths[j].m_position->GetNode().get() == NULL))

{activePaths.erase(activePaths.begin() + j);

}}

}for (int l = (floatingBridges.size() - 1); l >= 0; l--){

floatingBridges[l]->SetWordIndex(dWordIndex);}return dReturn;

}

Syntactic Analysis

The purpose of Syntactic Analysis is to produce complex nodes from the atomic nodes passed

in the POSList and to identify the targeted complex nodes to provide to the Conceptual

Analysis process. In a CLUE, Syntactic Analysis is not the final disambiguator; rather, theConceptual Analysis process working in conjunction with the Syntactic Analysis process

shall determine which concept prevails over the others based on meaning and syntactic

integrity. Consequently, there is no requirement to fully disambiguate during the Syntactic

Analysis process, meaning that Syntactic Analysis produces a multitude of syntactic

organizations that later need to be disambiguated by Conceptual Analysis. Prior to Syntactic

Analysis, there is a lot of ambiguity because the process only holds a list ofPOSNodes that

each have associated parts-of-speech; following Syntactic Analysis, there is less ambiguity

because targeted parts-of-speech have been identified and associated with their corresponding

sequences of words and parts-of-speech required to build them. Syntactic Analysis is also

useful in providing Conceptual Analysis with syntactic information to rely upon in predicate

calculation. As is later exposed in this article, a Predicate Builder Script is composed of code

that mostly relates to syntax, and making the transition from a syntactic stream to concepts

relies heavily on syntactic information produced during Syntactic Analysis.


25/74

The Syntactic Transform Script stored in SyntaxTransform.txtis central to Syntactic

Analysis. The content of that file holds sequencing decisions used to build complex nodes

from a configuration of complex nodes and atomic nodes found in the dictionary. The

Syntactic Transform Script is composed of about 50 lines of code of a language that is

created for the sole purpose of permuting nodes.

A closer look into the first three lines of code from SyntaxTransform.txthelps in

understanding that language.

CollapseADJECTIVE PHRASE CONSTRUCTION 1: ([ADVERB])[ADJECTIVE] -> ADJECTIVE_PHRASEMAXRECURSIVITY:2ADJECTIVE PHRASE ENUMERATION: [ADJECTIVE_PHRASE]([CONJUNCTION])

[ADJECTIVE_PHRASE] -> ADJECTIVE_PHRASE

# Verbs

MAXRECURSIVITY:1COMPLEX VERB CONSTRUCTION: [VERB & "is" | "was" | "will" | "have" |

"has" | "to" | "will be" | "have been" |"has been" | "to be" | "will have been" |"be" | "would" | "could" |

"should"]([ADVERB])[VERB] -> VERB

Lines have been wrapped in the above snippet to avoid scrolling.

The first line permutes all possibilities from ([ADVERB])[ADJECTIVE] and creates a resulting

node that is a part-of-speech, ADJECTIVE_PHRASE. For example, tokens such as "more red"

result in an ADJECTIVE_PHRASE since "more" is an ADVERB and "red" is an ADJECTIVE. But

since the ADVERB node is between parentheses, it is identified as being optional.Consequently, the ADJECTIVE token "blue" also results in an ADJECTIVE_PHRASE node.

The following line, [ADJECTIVE_PHRASE]([CONJUNCTION])[ADJECTIVE_PHRASE] , takes

sequences ofADJECTIVE_PHRASE nodes, optionally separated by a CONJUNCTION node, and

creates a new ADJECTIVE_PHRASE node with them. Such a script line is recursive since it

transforms into a part-of-speech that is part of its sequencing. To that effect, in order to limit

computing to a reasonable level of parsing, we may want to limit recursion as it is done on

the preceding line: MAXRECURSIVITY:2 . That basically states that only two successful passesat this transformation line are allowed. That means that tokens such as "more blue and less

green" are transformed successfully, while tokens such as "red, some green and grey" are nottransformed successfully since a recursion level of at least 3 is required for that transform to

happen. Note that recursion limitations are only relevant while performing bottom-up parsing

and not multi-dimensional Earley parsing. More on that later...

The next line, [VERB & "is" | "was" | "will" | "have" | "has" | "to" | "willbe" | "have been" | "has been" | "to be" | "will have been" | "be" |

"would" | "could" | "should"]([ADVERB])[VERB], has comparable rules, but also states

some conditions in regards to spellings for the first node. The components between double-

quotes are spelling conditions, where at least one of which must succeed for the node match

to occur. From that line of code, a successful transform happens for the token sequence:

"could never see", but fails for the token sequence: "see always ear".


26/74

Here is the complete script. It encapsulates most of the English language, although slight

adaptations may be required if more complex test cases are not transformed as expected.

Collapse# NOTES ON THE SYNTAX OF THE SCRIPTING LANGUAGE

# - BEFORE THE ':' CHARACTER ON A LINE IS THE LINE NAME# - A NODE BETWEEN PARENTHESIS IS INTERPREATED AS BEING OPTIONAL# - CONTENT BETWEEN QUOTES RELATES TO SPELLING# - SPELLINGS THAT BEGIN WITH A '*' CHARACTER ARE INTERPREATED#- AS A 'START WITH' STRING MATCH# - ON THE RIGHT SIDE OF THE CHARACTERS '->'# - IS THE DEFINITION OF THE NEW ENTITY (AFFECTATION)

# SCRIPT

# Adjective phrase construction

ADJECTIVE PHRASE CONSTRUCTION 1: ([ADVERB])[ADJECTIVE] -> ADJECTIVE_PHRASEMAXRECURSIVITY:2ADJECTIVE PHRASE ENUMERATION: [ADJECTIVE_PHRASE]([CONJUNCTION])

[ADJECTIVE_PHRASE] -> ADJECTIVE_PHRASE

# Verbs

MAXRECURSIVITY:1COMPLEX VERB CONSTRUCTION: [VERB & "is" | "was" | "will" | "have" |

"has" | "to" | "will be" | "have been" |"has been" | "to be" | "will have been"

|"be" | "would" | "could" |"should"]([ADVERB])[VERB] -> VERB

# Noun phrase construction

GERUNDIVE ING: [VERB & "*ing"] -> GERUNDIVE_VERBGERUNDIVE ED: [VERB & "*ed"] -> GERUNDIVE_VERBPLAIN NOUN PHRASE CONSTRUCTION: ([DEFINITE_ARTICLE |INDEFINITE_ARTICLE])

([ORDINAL_NUMBER])([CARDINAL_NUMBER])([ADJECTIVE_PHRASE])[NOUN | PLURAL |

PROPER_NOUN | TIME |DATE | PRONOUN] -> NOUN_PHRASE

MAXRECURSIVITY:2

NOUN PHRASE ENUMERATION: [NOUN_PHRASE]([CONJUNCTION])[NOUN_PHRASE] ->NOUN_PHRASEMAXRECURSIVITY:1

# Preposition phrase construction

PREPOSITION PHRASE CONSTRUCTION: [PREPOSITION][NOUN_PHRASE] ->PREPOSITION_PHRASEMAXRECURSIVITY:2PREPOSITION PHRASE ENUMERATION: [PREPOSITION_PHRASE]([CONJUNCTION])

[PREPOSITION_PHRASE] ->PREPOSITION_PHRASE

# Verb phrase construction


27/74

VERB PHRASE CONSTRUCTION 1: [VERB]([ADVERB])[NOUN_PHRASE]([PREPOSITION_PHRASE]) -> VERB_PHRASE

VERB PHRASE CONSTRUCTION 2: [VERB][PREPOSITION_PHRASE] ->VERB_PHRASEVERB PHRASE CONSTRUCTION 3: [ADJECTIVE_PHRASE][PREPOSITION][VERB] ->VERB_PHRASE

# Noun phrase construction while considering gerundive

GERUNDIVE PHRASE CONSTRUCTION: [GERUNDIVE_VERB]([NOUN_PHRASE])([VERB_PHRASE])([ADVERB]) ->

GERUNDIVE_PHRASEMAXRECURSIVITY:2NOUN PHRASE CONST WITH GERUNDIVE: [NOUN_PHRASE][GERUNDIVE_PHRASE]

([GERUNDIVE_PHRASE])([GERUNDIVE_PHRASE]) -> NOUN_PHRASEPREPOSITION PHRASE CONSTRUCTION 3: [PREPOSITION][GERUNDIVE_PHRASE] ->PREPOSITION_PHRASE

# Noun phrase construction while considering restrictive relative clauses

RESTRICTIVE RELATIVE CLAUSE: [WH_PRONOUN & "who" | "where" | "when"|

"which"][VERB_PHRASE] -> REL_CLAUSERESTRICTIVE RELATIVE CLAUSE 2: [PRONOUN & "that"][VERB_PHRASE] ->REL_CLAUSEMAXRECURSIVITY:2NOUN PHRASE WITH REL_CLAUSE: [NOUN_PHRASE][REL_CLAUSE] ->NOUN_PHRASE

# Make sure the restrictive relative clauses built are part of the verbphrases

VERB PHRASE WITH REL_CLAUSE: [VERB_PHRASE][REL_CLAUSE] -> VERB_PHRASEVERB PHRASE CONSTRUCTION 4: [VERB][NOUN_PHRASE][REL_CLAUSE]

([PREPOSITION_PHRASE]) -> VERB_PHRASEMAXRECURSIVITY:2WH_PRONOUN CONSTRUCTION ENUMERATION: [WH_PRONOUN][CONJUNCTION][WH_PRONOUN]-> WH_PRONOUN

# Make sure the gerundive built are part of the verb phrases

VERB PHRASE CONSTRUCTION 5: [VERB][NOUN_PHRASE][GERUNDIVE_PHRASE]([GERUNDIVE_PHRASE])([GERUNDIVE_PHRASE])([PREPOSITION_PHRASE]) -> VERB_PHRASE

VERB PHRASE CONSTRUCTION 6: [VERB][NOUN_PHRASE][ADJECTIVE_PHRASE] ->VERB_PHRASEVERB PHRASE CONSTRUCTION 7: ([VERB])[NOUN_PHRASE][VERB] ->VERB_PHRASEMAXRECURSIVITY:2VERB PHRASE CONSTRUCTION 8:[VERB_PHRASE][NOUN_PHRASE][GERUNDIVE_PHRASE]

([GERUNDIVE_PHRASE])([GERUNDIVE_PHRASE])([PREPOSITION_PHRASE]) -> VERB_PHRASE

MAXRECURSIVITY:2VERB PHRASE CONSTRUCTION 9: [VERB_PHRASE]([NOUN_PHRASE])

[ADJECTIVE_PHRASE] -> VERB_PHRASEMAXRECURSIVITY:2VERB PHRASE CONSTRUCTION 10: [WH_PRONOUN][VERB_PHRASE] ->

VERB_PHRASEMAXRECURSIVITY:2


28/74

VERB PHRASE CONSTRUCTION 11: [VERB_PHRASE][NOUN_PHRASE]([PREPOSITION_PHRASE |GERUNDIVE_PHRASE]) -> VERB_PHRASE

MAXRECURSIVITY:2VERB PHRASE CONSTRUCTION 12: [VERB_PHRASE][PREPOSITION_PHRASE] ->VERB_PHRASE

# WH-Phrases construction

WH_NP CONSTRUCTION 1: [WH_PRONOUN][NOUN_PHRASE] -> WH_NPWH_NP CONSTRUCTION 2: [WH_PRONOUN][ADJECTIVE]([ADVERB]) -> WH_NPWH_NP CONSTRUCTION 3: [WH_PRONOUN][ADVERB][ADJECTIVE] -> WH_NPMAXRECURSIVITY:2WH_NP CONSTRUCTION 4: [WH_NP][CONJUNCTION][WH_NP | WH_PRONOUN] -> WH_NP

# Sentence construction

SENTENCE CONSTRUCTION QUESTION 1: [VERB & "is" | "was" | "were"]

[NOUN_PHRASE][NOUN_PHRASE]([PUNCTUATION & "?"]) -> SENTENCESENTENCE CONSTRUCTION QUESTION 2: [VERB & "is" | "was" | "were"]

[VERB_PHRASE][VERB_PHRASE]([PUNCTUATION & "?"]) -> SENTENCE

SENTENCE CONSTRUCTION 1: [VERB_PHRASE]([PREPOSITION & "at" | "in" |"of" |

"on" | "for" | "into" | "from"])([PUNCTUATION & "?"]) -> SENTENCE

SENTENCE CONSTRUCTION 2: ([AUX])[NOUN_PHRASE][VERB_PHRASE | VERB]([PREPOSITION & "at" | "in" | "of" |"on" | "for"])([ADVERB])([PUNCTUATION &

"?"]) -> SENTENCEWH_NP SENTENCE CONSTRUCTION 1: [WH_NP][VERB_PHRASE]([PREPOSITION &"at" |

"in" | "of" | "on" | "for" | "into"|

"from"])([PUNCTUATION & "?"]) ->SENTENCEWH_NP SENTENCE CONSTRUCTION 2:[WH_NP]([AUX])[NOUN_PHRASE][VERB_PHRASE | VERB](

[PREPOSITION & "at" | "in" | "of" |"on" | "for"])

([ADVERB])([PUNCTUATION & "?"]) ->SENTENCEWH_NP SENTENCE CONSTRUCTION 3: [NOUN_PHRASE |VERB_PHRASE]([PREPOSITION &

"at" | "in" | "of" | "on" | "for" |"into" |

"from"])[WH_NP]([PUNCTUATION & "?"])-> SENTENCEMAXRECURSIVITY:2SENTENCE CONSTRUCTION 4: [SENTENCE]([CONJUNCTION])[SENTENCE] ->SENTENCE

Lines have been wrapped in the above snippet to avoid scrolling.

One goal for the Syntactic Transform Script is to create [SENTENCE] complex nodes.

Sentences are special as they are expected to encapsulate a complete thread of thoughts that

can later be represented by a predicate. Although predicates can also be calculated for other


29/74

complex nodes, only [SENTENCE] parts-of-speech can reliably fully be encapsulated into a

predicate. That does not mean the [SENTENCE] part-of-speech is necessarily self-contained,

though. For example, think about the two following [SENTENCE]s: "I saw Edith yesterday.

She is feeling great." The object of knowledge "she" in the second [SENTENCE] refers to the

object of knowledge "Edith" from the first [SENTENCE]. For the purpose of this sample

project, although it is possible to do so, we will not keep a context between different[SENTENCE] nodes.

Permuting the Syntactic Transform Script

One of the first things done by the syntactic analyzer is to create a decision-free version of

the Syntactic Transform Script. That is, it creates a version of the Syntactic Transform Script

into POSDictionary_flat.txtthat is functionally equivalent to SyntaxTransform.txtthat does

not hold any decision nodes (nodes between parenthesis in SyntaxTransform.txt, or different

spellings conditions that are delimited by the pipe character within a node). The content of

POSDictionary_flat.txt, as generated by the downloadable executable attached to this article,

can be referred to byclicking here. In order to do that, it performs two distinct steps intoPOSTransformScript::ManageSyntacticDecisions , where the parameter decisionFile

is the string holding the name of the Syntax Transform Script.

Collapsevoid POSTransformScript::ManageSyntacticDecisions(string decisionFile,

string dFlatFileName){

FlattenDecisions(decisionFile, "temp.txt");FlattenDecisionNodes("temp.txt", dFlatFileName);

}

During the first step (into POSTransformScript::FlattenDecisions ), each line of the fileis read, and each element between parenthesis is removed from parenthesis in one line and

ignored into another. Since an element between parenthesis is optional, there are indeed only

two possibilities - it is included or not.

Collapsevoid POSTransformScript::FlattenDecisions(string dDecisionFileName,

string dFlattenDecisionFileName){

fstream dOutputStream;dOutputStream.open(dFlattenDecisionFileName.c_str(),

ios::out | ios::binary | ios::trunc);ifstream dInputStream(dDecisionFileName.c_str(), ios::in |ios::binary);

dInputStream.unsetf(ios::skipws);string dBuffer;char dChar;while (dInputStream >> dChar){

switch (dChar){

case '\n':case '\r':

if (dBuffer.find("#") != string::npos)

{ dBuffer = SubstringUpTo(dBuffer, "#");}
http://www.codeproject.com/KB/recipes/DigitalConceptBuilder/POSDictionary_flat.txthttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/POSDictionary_flat.txthttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/POSDictionary_flat.txthttp://www.codeproject.com/KB/recipes/DigitalConceptBuilder/POSDictionary_flat.txt


30/74

if (dBuffer.length() > 0){

unsigned int i;vector *matches;Permutation perms(dBuffer);matches = perms.GetResult();

for (i = 0; i < matches->size(); i++){dOutputStream.write(

matches->at(i).c_str(), matches->at(i).length());dOutputStream.write("\r\n", strlen("\r\n"));

}}dBuffer = "";break;

default:dBuffer += dChar;

}}

dInputStream.close();dOutputStream.close();}

To help the calculation of all possible permutations of a single line, the Permutation class isused. Since there may be more than one permutation on a line, the use of the recursive

Permutation::FillResult method is the best approach to perform these calculations.

Collapsevoid Permutation::FillResult(){

m_result.clear();

vector matches;if (PatternMatch(m_value, "(*)", &matches, true)){

string tmpString = m_value;size_t pos = tmpString.find("(" + matches[0] + ")");if (SearchAndReplace(tmpString, "(" + matches[0] +

")", "", 1, true) == 1){

Permutation otherPerms(tmpString);vector *otherMatches = otherPerms.GetResult();for (unsigned int i = 0; i < otherMatches->size(); i++){

// We push the value with

// the potential value out and then in...m_result.push_back(otherMatches->at(i));m_result.push_back(m_value.substr(0, pos) +

matches[0] + otherMatches->at(i).substr(pos));}

}}else{

m_result.push_back(m_value);}

}

Once this first step of taking care of content between parenthesis is done, decisions withinatomic nodes need to be handled. An atomic node with a decision is like


31/74

[DEFINITE_ARTICLE | INDEFINITE_ARTICLE], where the part-of-speech is a choice, or

[WH_PRONOUN & "who" | "where" | "when" | "which"], where the spelling constraint is

also a choice. Most of the logic is in POSTransformScript::FlattenOneDecisionNodes ,

which itself relies heavily on POSNode::Construct for parsing.

Collapsevector POSTransformScript::FlattenOneDecisionNodes(string dLine){

vector dReturn;vector matches;if (PatternMatch(dLine, "[*]", &matches, true) > 0){

for (unsigned int i = 0; i < matches.size(); i++){

vector possibilities;POSNode::Construct("[" + matches[i] + "]", &possibilities);if (possibilities.size() > 1){

for (unsigned int j = 0; j < possibilities.size(); j++){

string dLineCopy = dLine;SearchAndReplace(dLineCopy, "[" + matches[i] + "]",

possibilities[j].get()->GetNodeDesc(),1);

vector dNewLineCopies =FlattenOneDecisionNodes(dLineCopy);

for (unsigned int k = 0; k < dNewLineCopies.size();k++)

{dReturn.push_back(dNewLineCopies[k]);

}

}break;

}}

}if (dReturn.size() == 0){

dReturn.push_back(dLine);}return dReturn;

}

shared_auto_ptr POSNode::Construct(string dNodeDesc,

vector *allPOSNodes,unsigned int dStartPosition,unsigned int dEndPosition,unsigned int dBridgePosition,string data)

{vector tmpVector;if (allPOSNodes == NULL){

allPOSNodes = &tmpVector;}vector matches;vector spellings;

vector dPOS;string dLeftPart = "";if (PatternMatch(dNodeDesc, "[* & *]", &matches, false) == 2)


32/74

{dLeftPart = matches[0];string dRightPart = matches[1];if (PatternMatch(dRightPart, "\"*\"", &matches, true, "|") > 0){

for (unsigned int i = 0; i < matches.size(); i++)

{ spellings.push_back(matches[i]);}

digital concept builder

Documents