11/50/50
IPSI Belgrade Ltd.IPSI Belgrade Ltd.
ARTSHOP GALLERYARTSHOP GALLERY
[email protected]@ipsi.co.yuhttp://www.ipsi.co.yu
22/50/50
AuthorsAuthors
Milutinovic Veljko
Toskov IvanVujovic Ivana
Skundric Nikola Milutinovic Darko
Stojanovski Aleksandar Nikezic Gavro
Anucojic Goran
Radakovic MiroslavMarinkovic Ivan
33/50/50
Introduction – IPSI BgdIntroduction – IPSI Bgd
IPSI BelgradeIPSI Belgrade is a company jointly founded by German and Serbian capital
Partners: - IPSI Fraunhofer, Darmstadt, Germany- Telecom Italia Learning Services, Italy- NYU, School of Continuous Professional Studies, USA- Instituto Tecnologico de Durango, Mexico
44/50/50
Introduction – IPSI BgdIntroduction – IPSI Bgd
- Workspaces of the Future- Environments for Cooperative Working and Learning- Virtual Information and Knowledge Environments- Mobile Interactive Media- Open Adaptive Information Management Systems- Publication Engineering and Technology- Hardware Design and Operating Systems- Networks and WWW
55/50/50
Introduction – IPSI BgdIntroduction – IPSI Bgd
Products:
• Advanced Virtual Gallery• Semantic Web tutorial and book development• The injection cache, the STS cache, VLSI Detection for Internet/Telephony Interfaces, Genetic Search with Spatial/Temporal Mutations, Browser Acceleration, Technology transfer, Testing the Infrastructure for EBI, Socratenon Distant Web Educating Machine, e-Tourism…
66/50/50
Problem StatementProblem Statement
- Creating Web based art gallery with “look and feel” of the real world exhibitions
- Visitor moves through the gallery by “walking with options”
77/50/50
Existing SolutionsExisting Solutions
- Musee national des Arts asiatiques http://www.museeguimet.fr/tour-guimet/index.html
- Web Server of the Galleria degli Uffizi in Florence http://www.uffizi.firenze.it
- The Distributed Interactive Virtual Environment (DIVE) http://www.sics.se/dive/
- The Web3D Repository http://www.web3d.org/vrml/artgal.htm
88/50/50
Proposed SolutionProposed Solution
- Virtual reality gallery
- Advanced search capabilities
- Visitor’s criteria based room generation
99/50/50
Why is it better?Why is it better?
- Dynamically generated gallery- Dynamically generated gallery
- Content based search engine- Content based search engine
- User satisfaction - User satisfaction
1010/50/50
Conditions and AssumptionsConditions and Assumptions
- PC- Internet connection- Internet Explorer 5.0 or higher - Netscape 7.0 - Cortona VRML plug-in for IE
1111/50/50
Analysis and ImplementationAnalysis and Implementation
- Application is written in ASP.NET using C# as code-behind, and ADO.NET for database access.- Database server is SQL Server 2000.- Communication with the database is entirely made through XML (using SQLXML3.0 framework).
- Queries are made in XPath, while adding, changing and deleting of the records is done through UpdateGrams.
- Application is optimized for Internet Explorer 5.0 or higher, at the 1024x768 screen resolution. Netscape 7.0 or higher is also supported.
- 3D gallery is completely generated on the server side (dynamically) using VRML.
1212/50/50
Track 1Track 1
Track Requirements:Track Requirements:• To stand as the integrative part for the other two tracks• To provide:
– User interaction– Database connectivity (database independent)– Search functions
(simple and advanced using Track3 output)– Information brokering between artists and buyers– Administration tools– Artworks management tools, etc.– Thin client (3D scene generation on server side)
1313/50/50
Track 1Track 1
Development Tools:Development Tools:• Application server platform:
– Windows XP Professional– IIS 5.1– MS SQL Server 2000
• Development platform:– ASP.NET– C# as code-behind.
• Communication with the underlying database:– XML & XSD using XPath queries (DB independent)– Currently using SQLXML3.0 add-on for ADO.NET
1414/50/50
Track 1Track 1
Database Design:Database Design:
1515/50/50
Track 1Track 1
Administrator Tools:Administrator Tools:
• Separate entry point:
http://<server_address>/artshop/admin
1616/50/50
Track 1Track 1
Users & Exhibitors:Users & Exhibitors:• Entry point:
http://<server_address>/artshop/index.htm
1717/50/50
Track 1Track 1Interesting Details:Interesting Details:
• Native XML DBMS under development at IPSI Fraunhofer• Practical testing of the XML/XPath database access• Dynamic addition (to the system) of new multimedia types• 3D view of search results
1818/50/50
Track 1Track 1Interesting Details:Interesting Details:• Application that can connect on the fly to any DBMS which
supports XML/XPath is an interesting and possibly useful idea (user just has to set one XML file containing local field mapping, and one XSD to map the database fields to the pre-defined scheme)
• Cons:– XPath queries are lot less powerful then standard SQL queries– Inherently, loss of speed
(one complex SQL query had to be simulated with couple of XPath queries and additional processing in the code).
– For now, SQLXML3.0 does not support complete XPath standard.
1919/50/50
Track 1Track 1Errors Made:Errors Made:
• Initially, content analysis, picture processing, and adding data to database were completely separated (as specified in the contract), with the idea of later (partial) integration.
• Turned out to be a bad idea(required a lot of interventionfrom the ArtShop system administrator when adding artworks).
2020/50/50
Track 1Track 1Lessons Learned:Lessons Learned:• Problem solved by complete integration of forementioned tasks
into the one system process which monitors input directory, automatically schedules picture processing and content analysis, and takes care of updating of all necessary fields in all required databases.
• With that, we achieved maximum automation, reduced time needed for artwork addition, and reduced amount of data transferred through the Internet (between the administrator’s machine and the application host).
2121/50/50
Track 2Track 2
Image-Content-Oriented SearchImage-Content-Oriented SearchTrack requirements:Track requirements:
• Images used for extracting objects are artistic paintings• Image analyses• Extraction of the features• Create XML file for each image• Fetch the database with the features
2222/50/50
Track 2Track 2
Algorithm:Algorithm:
Open the picture
Load parameters, input and output directory
Determine the filter value
Put the picture into the reduced matrix
Determine histogram
Create objects
Merge objects into bigger objects
Create sorted array of objects
Create database objects, prepare them, and put them in XML file and tables in database
2323/50/50
Track 2Track 2
Determine histogram:Determine histogram: Create general histogram
Remove zero values
Sort histogram
Remove redundancies
Sort histogram
Refresh matrix
2424/50/50
Track 2Track 2
Regions forRegions for processing matrix: processing matrix:
P8
P4P5
P5P6P3
P3
P4P5
P3
P8
P8P8
P8
the first colon the last row
the last colon
the rest of the matrix
the last element in the last row
2525/50/50
Creating Creating objects:objects:
Any pixel left in the current region?
p8 = the current pixel
call matrix.unite_pixels method
p8.oi == 0 (it doesn’t belong to any object)
create new object
Any neighbour pixel left?
p8.oi != px.oi (don’t belong to the same object)
take the next pixel
take the next neighbour pixel
px = the neighbour pixel
true
true
true
false
true
false
false
false
2626/50/50
Merge Merge objects objects into into bigger bigger objects:objects:
Is Picture_Objects list empty?
q==true (are there any objects inside Border which colors after translation are the same as the color of Core object)
q = true; Edge[0] = Picture_Objects[0]
i < number of objects inside Edge list
Find all neighbour objects, which colors after translation are the same as the color of Core object, and put them into Border
Move all pixels from objects inside Edge to Core object
Remove objects that are inside Edge from Picture_Objects list
Is Border empty
q=false; add Core object to big list; Remove pixels belonging to Core object from Picture_Objects
false
true
true
Take ith object
i = i+1
Move all objects inside Border to Edge
false
true
false
true
false
2727/50/50
Track 2Track 2
Tools used in development:Tools used in development:• C# programming language – the chosen tool
– Advantage: Includes the best properties from other programming languages (C++, Java, Visual Basic)
– Disadvantage: slower processing speed than C++, which is not necessary in this application
• C++ - the best alternative tool
– Advantage: faster processing speed (unnecessary)
– Disadvantage: more complicated code, 50% of all bugs due to use of pointers
2828/50/50
Track 2Track 2
Original picture:Original picture:
2929/50/50
Track 2Track 2Picture after applyingPicture after applyinghistogram values:histogram values:
3030/50/50
Track 2Track 2Picture representedPicture representedthrough extracted objects:through extracted objects:
3131/50/50
Track 2Track 23D HSL space => 1D histogram3D HSL space => 1D histogram
Index of histogram array Hue Saturation Luminance Description
0 any any <=30 black
1 any any >=lummax white
2 any <20 >30<30+lum
the darkest grey
... … … … …
ilmax-1 any <20 >=30+(ilmax-3)*lum<30+(ilmax-2)*lum
the lightest grey
ilmax <=hue/2>=240-hue/2
>20<20+1*sat
>30<30+lum
the darkest red with the smallest saturation
ilmax+1 <=hue/2>=240-hue/2
>=20+1*sat< 20+2*sat
>30<30+lum
the darkest red with smaller saturation
… … … … …
ilmax+isat-2 <=hue/2>=240-hue/2
>=20+(isat-3)*sat< 20 +(isat-2)*sat
>30<30+lum
the darkest red with the biggest saturation
ilmax+1*(isat-1) <=hue/2>=240-hue/2
>20<20+1*sat
>=30+lum< 30+2*lum
darker red with the smallest saturation
ilmax+1*(isat-1)+1 <=hue/2>=240-hue/2
>=20+1*sat< 20+2*sat
>=30+lum< 30+2*lum
darker red with smaller saturation
… … … … …
ilmax+1*(isat-1)+isat-2 <=hue/2>=240-hue/2
>=20+(isat-3)*sat< 20 +(isat-2)*sat
>=30+lum< 30+2*lum
darker red with the biggest saturation
ilmax+2*(isat-1) <=hue/2>=240-hue/2
>20<20+1*sat
>=30+2*lum< 30+3*lum
dark red with the smallest saturation
3232/50/50
Track 2Track 23D HSL space => 1D histogram3D HSL space => 1D histogram
ilmax+2*(isat-1)+1 <=hue/2>=240-hue/2
>=20+1*sat< 20+2*sat
>=30+2*lum< 30+3*lum
dark red with smaller saturation
… … … … …
ilmax+(ilmax-3)*(isat-1)+1 <=hue/2>=240-hue/2
>=20+1*sat< 20+2*sat
>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum
the brightest red with smaller saturation
… … … … …
ilmax+(ilmax-3)*(isat-1)+(isat-2) <=hue/2>=240-hue/2
>=20+(isat-3)*sat< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum
the brightest red with the biggest saturation
ilmax+1*(ilmax-2)*(isat-1) > hue/2< 3*hue/2
>20<20+1*sat
>30<30+lum
the darkest orange-red with the smallest saturation
ilmax+1*(ilmax-2)*(isat-1)+1 > hue/2< 3*hue/2
>=20+1*sat< 20+2*sat
>30<30+lum
the darkest orange-red with smaller saturation
… … … … …
ilmax+1*(ilmax-2)*(isat-1)+(ilmax-3)*(isat-1)+(isat-2)
> hue/2< 3*hue/2
>=20+(isat-3)*sat< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum
the brightest orange-red with the biggest saturation
ilmax+2*(ilmax-2)*(isat-1) >=3*hue/2< 5*hue/2
>20<20+1*sat
>30<30+lum
the darkest red-orange with the smallest saturation
ilmax+2*(ilmax-2)*(isat-1)+1 >=3*hue/2< 5*hue/2
>=20+1*sat< 20+2*sat
>30<30+lum
the darkest red-orange with smaller saturation
… … … … …
ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+1 >=(2*ihmax-1)*hue/2<(2*ihmax+1)*hue/2
>=20+1*sat< 20+2*sat
>30<30+lum
the darkest magenta-red with smaller saturation
… … … … …
ilmax+(ihmax-1)*(ilmax-2)*(isat-1)+(ilmax-3)*(isat-1)+(isat-2)
>=(2*ihmax-1)*hue/2<(2*ihmax+1)*hue/2
>=20+(isat-3)*sat< 20 +(isat-2)*sat
>=30+(ilmax-3)*lum< 30+(ilmax-2)*lum
the brightest magenta-red with the biggest saturation
3333/50/50
Track 2Track 2Lessons learned:Lessons learned:
• It is impossible to extract objects using only colors as a criterion• It is impossible to extract objects,
even using textures, edges, different transformations as criteria• Semantics should be used in segmentation• Colors are the most important features in artistic paintings
3434/50/50
Track 3Track 3
Track requirements:Track requirements:• Possibility of moving through 3D galleries
• Automatic generation of 3D galleries based on user’s query
• Manual generation of 3D galleries
• User interface for image zooming
• Application for image processing
3535/50/50
Track 3Track 3Underlying algorithms:Underlying algorithms:
• Dynamic creation of gallery
• Creation of static galleries
• Algorithm for picture zooming
• Algorithm for picture processing
3636/50/50
Track 3Track 3
Creation of galleries:Creation of galleries:
• Validation of the created gallery
• Forming VRML files depending on users query
• Determining the number of pictures in the gallery
• Drawing a 2D floorplan based on the 3D gallery
• “Forest fire” algorithm for filling the floorplan with color
3737/50/50
Track 3Track 3
Picture processing:Picture processing:
• Loading image into memory
• Clone image into different-size copies
• Filtering of copies
• Parting of copies
3838/50/50
Track 3Track 3
Development tools:Development tools:
• C# in .NET Framework for programming image processing
• Macromedia Dreamweaver for programming zoom tool
• VRML Pad v2.0
3939/50/50
Track 3Track 3
Flowchart:Flowchart:
4040/50/50
Track 3Track 3
Creation of galleries:Creation of galleries:
• Making files based on users data
• Putting data on server so it can be available for artists
• Artist chooses which gallery he/she will be using for exhibition
• User can move through 3D world
• Selecting the textures for gallery
• Selecting the starting position of the user
4141/50/50
Track 3Track 3
NF filter details:NF filter details:
• If the new picture is smaller, every pixel is one pixel of the old picture.
• If the new picture is bigger, pixels are calculated based on the pixels surrounding the current.
4242/50/50
Track 3Track 3
Errors made:Errors made:
• Requests were not precise, so there was a gap at the end of the project between wanted and done
• Better results could be done with better using of ASP and XML
4343/50/50
Track 3Track 3
Lessons learned:Lessons learned:
• Every member of the team gets a part where his experience is dominating
• More planning at the start reduces a lot of work later
• Good communication between programmers can save a lot of time
4444/50/50
DemoDemo
4545/50/50
Future PlansFuture Plans
• Improving the existing 3D dynamic gallery
• Improving search engine capabilities
• Improving feature extraction algorithms and objects recognition
4646/50/50
Future Track 1Future Track 1
3D Multimedia Showroom Environment:3D Multimedia Showroom Environment:
• Implementing a general Web based 3D Multimedia Showroom Environment
• Exhibiting various MM data types: images, 3D objects, videos, audio, etc.
• Set of MM data types should be extendable
4747/50/50
Future Track 2Future Track 2
MM Object Feature Extraction:MM Object Feature Extraction:
• Implementing algorithms and software components for extracting features from MM data types (images, videos, 3D objects), in order to enable content based search
• System should be extendable (“plug-in”)
4848/50/50
Future Track 3Future Track 3
Semantic Abstraction of MM Feature Spaces:Semantic Abstraction of MM Feature Spaces:
• Developing methods and SW components which derive mapping from extracted features of MM objects to semantic concepts
• Using intelligent classification algorithms (Neural Networks, Fuzzy Classifier)
• Developing semantic query engine (answering questions, which could previously only be answered by humans)
4949/50/50
UsabilityUsability
• Art GalleriesArt Galleries
• MuseumsMuseums
• Exhibition FairsExhibition Fairs
5050/50/50
Instead of ConclusionInstead of Conclusion
IPSI Belgrade, [email protected]://www.ipsi.co.yu