approachtotheextractionofdesignfeaturesofpapers.cumincad.org/data/works/att/caadria2018_314.pdf ·...

APPROACH TO THE EXTRACTION OF DESIGN FEATURES OFINTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITIONTECHNIQUE

JIN SUNG KIM1, JAE YEOL SONG2 and JIN KOOK LEE31,2,3Department of Interior Architecture & Built Environment, YonseiUniversity, Republic of Korea1,2{wlstjd1320|songjy92}@gmail.com [email protected]

Abstract. This paper aims to propose deep learning-based approach tothe auto-recognition of their design features of interior design elementsusing given digital images. The recently image recognition techniqueusing convolutional neural networks has shown great success in thevarious field of research and industry. The open-source frameworks andpre-trained image recognition models supporting image recognition taskenable us to easily retrain the models to apply them on any domain. Thispaper describes how to apply such techniques on interior design processand depicts some demonstration results in that approaches. Furniturethat is one of the most common interior design elements has sub-featureincluding implicit design features, such as style, shape, function as wellas explicit properties, such as component, materials, and size. This papershows to retrain the model to extract some of the features for efficientlymanaging and utilizing such design information. The target element ischair and the target design features are limited to functional features,materials, seating capacity and design style. Total 3933 chair imagesdataset and 6 retrained image recognition models were utilized forretraining. Through the combination of thosemultiplemodels, inferencedemonstration also has been described.

Keywords. Deep learning; Image recognition; Interior designelements; Design feature; Chair.

1. Introduction1.1. RESEARCH OBJECTIVE

The artificial intelligence (AI) and the machine learning have been applied in manyfields of research and industry. Especially, deep artificial neural networks haveachieved great success on many contests about pattern recognition (Schmidhuber2015). The advent of deep convolutional neural networks(CNNs) and the powerfulgraphics processing unit(GPU) are an important role of successful large-scaleimage recognition. General-purpose image recognition model like Alexnet(Krizhevsky et al. 2012) and GoogLeNet(a.k.a Inception) (Szegedy et al. 2015)were proposed and opened to people to utilize them for the various domains.

T. Fukuda, W. Huang, P. Janssen, K. Crolla, S. Alhadidi (eds.), Learning, Adapting and Prototyping,Proceedings of the 23rd International Conference of the Association for Computer-Aided ArchitecturalDesign Research in Asia (CAADRIA) 2018, Volume 2, 287-296. © 2018 and published by the Associationfor Computer-Aided Architectural Design Research in Asia (CAADRIA) in Hong Kong.

288 J.S. KIM, J.Y. SONG AND J.K. LEE

The framework for deep learning such as Tensorflow (Abadi et al. 2016) andits related library also was provided as open source and have helped people invarious domains to retrain image recognition model. Furthermore, deep learningmodel could catch some object on image or video. Through the object detectionapproach like Faster R-CNN (Ren et al. 2015) and YOLO (Redmon et al. 2016), itis possible to analyze a scene and extract object and its information like the label.This could be very useful for the various domain where image processing is animportant task.

To go beyond the general purpose of image recognition and object detection,domain knowledge-based approach is needed. In terms of interior design, furnitureis one of the most common interior design elements in the interior design processor for designer and client. It is usually carried out to analyze each design elementand its feature from the indoor scene or product image for various activities relatedto interior design. In this regard, it is more meaningful to recognize design featuresof each element such as function, materials, style and etc. than to classify whetherit is chair or table. Sub-feature of furniture includes implicit design features,such as design type, style, shape and usage, and also explicit properties, suchas component, color, materials, and size. Interior design experts who trainedwith sufficient and good resource could successfully recognize such things. Forsuccessful automated extraction of such things, good training resources are neededto learn what features are and train to recognize and classify. The ultimate purposeof this research is to develop a system or applications to efficiently manage andutilize design information about the interior design and its elements on the digitalimage. The applications may include database system managing interior imageswith interior design information, tools for analysis on interior design and designrecommendation for interior designer and client. The objective of this paper is tolook for possibilities of deep learning-based approach to automated recognitiondesign features of the interior design element. We developed image dataset fortraining some design features of the chair, retrained some models using thatdataset and pre-trainedmodel and showed the results of testing automate extractionthrough application demonstration.

1.2. RESEARCH SCOPE AND PROCESS

This research ultimately aims to develop automated extraction of whole interiordesign element and its design feature from the image and utilize it for interiordesign. Among the needed researches for that, detecting interior element is alreadypossible and has shown high accuracy. As the basic research for next step, thispaper describes to simply survey necessary technique and approach and utilizethem to test to retrain deep learning model to recognize interior design elementand its design features (e.g. figure 1). We surveyed on available techniques suchas CNNs and image recognition library and used them for our purpose. For thedemonstration, target interior design element was limited to seating things likechair, sofa, and stool. Design features were limited to some features that could bevisually identified on image. Image data for training was collected from Googleimage search and relevant professional website like the Houzz, one of the mostpopular interior e-commerce website. The process of this paper is summarized as

APPROACH TO THE EXTRACTION OF DESIGN FEATURES OFINTERIOR DESIGN ELEMENTS USING IMAGE RECOGNITION

TECHNIQUE

289

follows:1. Training Automated extraction of design features of chair: Furniture and its

feature are identified and interesting criteria features which can be checked onvisual. Next, image dataset has been built according to features from variousweb-based source. Multiple image recognition models are retrained for eachobjective.

2. Demonstration of automate chair and its some design features using re-trainingmodels: to collaborate multiple automized extraction models, the various designfeatures are inferred from given image.

Figure 1. Scope of work for automated extraction of interior design elements and their designfeatures.

2. BackgroundFully reviewing deep learning and convolutional neural networks is out of thescope of this paper. In this paper, the survey on important background and theway to use such approach and tools for interior design domain are covered.

2.1. DEEP LEARNING-BASED IMAGE RECOGNITION

Deep convolutional neural networks (CNNs) is the method to process imagewith pixel data. It has lots of layer such as convolution filter, pooling filterand fully connected layer (e.g. figure 2). Passing through these layers madefrom given image, the weighs are trained for optimizing performance of imagefeature recognition task (LeCun et al. 2015) After success of “Supervision” ofKreizhevsky et al. (2012) in ILSVRC2012 image classification challenge, interestin CNNs were dramatically increased and new model and application approachhas been proposed. Recently, the outstanding result of general purpose imageclassification and object detection has been shown. CNNs have been applied to


various problems with high performance such as classifying image photographystyle (Karayev et al. 2013) and recognizing faces (Taigman et al. 2014). TheInception-v3 (Szegedy et al. 2016) that was utilized in this paper is recentsuccessful image recognition model. This model and retrain package are openedto people by Google, so it greatly helps for us to use it for the purpose. TheInception-v3 has more convolution and pooling layer than other models, but itsaccuracy was proved and it is easy to retrain with its library and transfer learning.

Figure 2. The structure of the typical convolutional neural networks for image recognition(LeCun and Bengio 1995).

2.2. APPLICATIONS OF IMAGE RECOGNITION FOR INTERIOR DESIGN

2.2.1. Learning visual similarity of interior design element and recommendationsystem.Bell and Bala (2015) proposed an approach to learning visual similarity of interiordesign product and image retrieval. They developed a visual search mechanism tomatch product image in an indoor scene with the clear well-light image on whitebackground. They showed CNNs can successfully extract visual features, trainingvisual similarity, such as shape, color, etc. and retrieve some image that visuallyvery similar to inputting image. Instead of classifying and extracting the features ofinterior products, it was carried out to recommend similar products. This is usefulto search some furniture of interest, but it is manually processed to recognize ofinteresting information of interior design product.

2.2.2. Automate classification of furniture design styleHu et al. (2017) proposed an approach to automate classification of furniture imageaccording to some design style. They utilize a series of machine learning methodsfor exploring the meaningful features of furniture styles, in terms of the aestheticrole of furniture in indoor design. Going beyond the general perspective, variousdetails of furniture such as the color, line shape, materials, size and so on areconsidered. They developed Training image database limited to 16 Categories offurniture style which are most popular. For this, handcrafted classification andlearning-based classification was used. To increase accuracy, combination of toptwo model showed final accuracy of 70.2%.

These two researches showed potentials of utilization of deep learning-basedapproach for interior design elements, such as learning similarity of product


TECHNIQUE

291

images or some furniture design styles. As a little different application of suchimage recognition technique, this paper proposes to approach to extraction of morefine-grained design feature.

3. Automized extraction of design features of interior design elementThis chapter describes definition features of furniture that can be visually identifiedand approach to training automated extraction model using deep learning-basedCNN. As a basic research, target indoor element is limited to the chair and itsfeature. It is introduced to prepare training image data according to the featurecategories and train a series of automated extraction model. Figure 3 shows theframework of automated extraction of furniture and its feature.

Figure 3. The framework of combinations of multiple models for the design features extraction.

3.1. FURNITURE AND ITS DESIGN FEATURES

Furniture is the main one of the interior design element. As the product, it hassome features that can be identified by designers, customers, and users. Furnitureoften has similar structural, technological, functional, operational and aestheticfeatures, so dividing distinctively and clearly, them is difficult (Smardzewski2015). Despite its difficulty, Classification of furniture could help to understandit and even design, purchase and use it. Accordingly, the various researcheswere carried out to identify and classify features of them. McDonagh et al.(2002) categorized the features of product design into practical applications of theproduct, human factors engineering, and aesthetic style. Lee (2010) categorizedthe of the furniture product designs into a functional design feature, human factordesign feature and external design feature. In summary, features of furniture arecategorized into three features: 1) related to function such as usage, supportingsome action 2) related to feeling like safety, comfort and etc. 3) related toappearances such as style, color, and materials.

The first feature is related to supporting some human action or position, so it isthe core feature. This feature is determined by whether the furniture has specificfunctional components. These kinds of components can be easily identifiedvisually. As an example of a chair, legs, sheet, backrest, armrest, headrest, and


frame could be included in the components. This feature is very explicitly shown.On the other hand, the other features were defined in various perspective. This isthe because those non-functional features are subjective and implicit. The featurerelated to human‘s feeling, for example, is affected by one‘s characteristic andpreference. It is also difficult to distinguish clearly style that is the main one ofthe most important feature of appearance. However, these features obviously helppeople to understand the furniture.

Hence, this research aims to check whether a computer could learn thesefeatures from furniture images and then recognize them. Automated extractionof them helps many people like a designer, customer, and user to make a decisionon interior design development. As the basic study for that, the target is limitedto chair and its features. The categories of target features of the chair that can beidentified on the image are as follows (e.g. figure 4):1. Functional features that can be check by additional components of chair such as

armrest, backrest, headrest2. Materials that are limited to five materials commonly used, such as leather, fabric,

wood, plastic, and velvet3. Seating capacity that can be checked by the number of seats4. Design style that are limited most popular, such as classic, eclectic, modern,

Scandinavian style

Figure 4. The categories and their detailed classes of target design feature for trainingautomized extraction.

3.2. PREPARATION TRAINING DATA

Training the automated extraction model requires specific dataset. Throughcategory of features proposed in the previous section, images were collectedfrom Google image search and www.Houzz.com that one of the biggest interiorand furniture e-commerce website. To build each dataset, we utilized automateimage scraper API and easily download on the website. As the unsupervisedlearning method, after collecting data with the keywords of features suggestedin the previous session, they were manually classified again by interior designexperts. It is easy to classify them according to functional features, because arm,back, head component is very visually explicit. we selected 5 detailed classes ofmaterials that common and well-identified. Classification of seating capacity was


TECHNIQUE

293

processed according to count of sheet division and length up to four in this paper.In case of design style, we selected only four different classes. Classification ofdesign style is even controversial because its definition is implicit and may varyfrom person to person. Hence, we collected and classified not some chairs ofmodern style but chairs that can seem modern style according to criteria of Miller(2010). Total 3933 images were prepared for training and the number of trainingimages per detailed classes of each category is shown in figure 5.

Figure 5. The number of training images per detailed classes of each category .

3.3. RE-TRAINING MULTIPLE IMAGE RECOGNITION MODELS

This section suggests an approach to automated extraction of chair features using acombination of deep learning-based image recognition models. Using the subsetsof dataset developed in the previous chapter, multiple recognition models weredeveloped by training a series of other CNN models. General purpose imagerecognition model that has been already developed by other researchers is a singlemodel and have multiple classes more than 10 classes. That approach is effectiveto classify a category of an object but it is impossible to recognize various categoryfrom features. For recognition of various features of an object, it is important todivide features set that each CNNmodel train. For example, training single modelfor the presence of armrest is better rather than one for the presence of armrestand backrest. Combination of Multiple models is effective to the recognition ofvarious types category. We retrained total 6 models, three for functional features,one for each of materials, seating capacity and style. Table 1 shows accuracy andtrain steps of training and validation of each model. The Accuracies of the modelsfor functional features are averagely higher than the others. This is because thosefunctional features are shown as more explicit by additional components like anarmrest.


Table 1. The results of validation and training of each model.

4. INTERFACE OF AUTOMIZED EXTRACTION OF DESIGNFEATURES OF INTERIOR DESIGN ELEMENTSWe developed GUI tool for automated extraction of design features of interiordesign elements for visually well-showing extraction results and making it easyfor anyone to use this. This was This was developed in Python and PyQt5 forcross-platform. This tool yet supports only automated extraction phase. Thisapplication looks like Figure 6. The user can open lists of image files and click therun button for automated extraction of design features of each file. The results ofeach image are shown in the table of inference result. Now, the results of extractionare only six that are derived from retrained models in the previous section.

Figure 6. Interface of GUI tool for automized extraction of design features.


TECHNIQUE

295

5. ConclusionThis paper proposed deep learning-based approach for automized extraction ofdesign feature of from image. This research aims ultimately to develop automizedextraction of such features and make application for in interior design, suchas supporting case study of interior design or interior element database andrecommend system. As the basic research for that, we utilized deep learning-basedCNNs model for extracting design features of interior design elements on digitalimage. Using well-trained model, we made multiple model learn four differentcategories of design features. We categorized design feature of interest, such asfunctional features, materials, seating capacity and design style, and collectedimage dataset and subset according to the category. Each category includesdetailed classes. functional features include armrest, backrest and headrest.materials do fabric, leather, plastic, velvet and wood. Seating capacity do seatof division and length. Finally, design style does classic, modern eclectics,Scandinavian. Total 3933 images were used and 6 models were re-trained. Meantraining accuracies of functional features is 94.3% and higher than other models.This is because that such features are more explicit by existence and nonexistenceof components. However, detailed texture of material, seating capacity and designstyle were not well recognized. We developed GUI tool for extracting designfeatures of given images. Through demonstration of the tool, it is checked foranyone to easily use them for extracting some series of images. For future works,the effective training methods will be needed as each feature with expansionof target design element and features. With these, approach to applications forinterior design will be able to be proposed, such as digital image-based interiordesign element and information database or recommendation system of interiordesign element according to requirements.

AcknowledgementsThis work was supported by the National Research Foundation of KoreaGrantfunded by the Korean Government (NRF-2015R1C1A1A01053497).

ReferencesAbadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A.,

Dean, J., Devin, M. and Ghemawat, S.: 2016, Tensorflow: Large-scale machine learningon heterogeneous distributed systems, OSDI, 16, 265-283.

Bell, S. and Bala, K.: 2015, Learning visual similarity for product design with convolutionalneural networks, ACM Transactions on Graphics (TOG), 34(4), 98.

Dazkir, S.S. and Read, M.A.: 2012, Furniture forms and their influence on our emotionalresponses toward interior environments, Environment and Behavior, 44(5), 722-732.

Hu, Z., Wen, Y., Liu, L., Jiang, J., Hong, R., Wang, M. and Yan, S.: 2017, Visual Classificationof Furniture Styles, ACM Transactions on Intelligent Systems and Technology (TIST), 8(5),67.

Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T., Hertzmann, A. andWinnemoeller, H.: 2013, Recognizing image style, arXiv preprint arXiv:1311.3715.

Krizhevsky, A., Sutskever, I. and Hinton, G.E.: 2012, Imagenet classification with deepconvolutional neural networks, Advances in neural information processing systems,1097-1105.


LeCun, Y. and Bengio, Y. 1995, Convolutional networks for images, speech, and time series,inM.A. Arbib (ed.), The handbook of brain theory and neural networks, MIT press.

LeCun, Y., Bengio, Y. and Hinton, G.: 2015, Deep learning, Nature, 521(7553), 436-444.Lee, A.S.: 2010, A study on predicting consumers‘satisfaction based on the features of furniture

product designs, International Journal of Organizational Innovation (Online), 2(3), 138.McDonagh, D., Bruseberg, A. and Haslam, C.: 2002, Visual product evaluation: exploring

users’ emotional relationships with products, Applied Ergonomics, 33, 231-240.Miller, J.: 2010, Furniture: World Styles from Classical to Contemporary, Dorling Kindersley.Redmon, J., Divvala, S., Girshick, R. and Farhadi, A.: 2016, You only look once: Unified,

real-time object detection, Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, 779-788.

Ren, S., He, K., Girshick, R. and Sun, J.: 2015, Faster R-CNN: Towards real-time objectdetection with region proposal networks, Advances in neural information processingsystems, 91-99.

Schmidhuber, J.: 2015, Deep learning in neural networks: An overview, Neural networks, 61,85-117.

Smardzewski, J.: 2015, Furniture design, Springer.Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke,

V. and Rabinovich, A.: 2015, Going deeper with convolutions, Proceedings of the IEEEconference on computer vision and pattern recognition, 1-9.

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z.: 2016, Rethinking the inceptionarchitecture for computer vision, Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, 2818-2826.

Taigman, Y., Yang, M., Ranzato, M. and Wolf, L.: 2014, Deepface: Closing the gap tohuman-level performance in face verification, Proceedings of the IEEE conference oncomputer vision and pattern recognition, 1701-1708.

Zehtaban, L., Elazhary, O. and Roller, D.: 2016, A framework for similarity recognition of CADmodels, Journal of Computational Design and Engineering, 3(3), 274-285.

approachtotheextractionofdesignfeaturesofpapers.cumincad.org/data/works/att/caadria2018_314.pdf ·...

Documents