A MULTIMODAL DESIGN ENVIRONMENT1

Andreas Zabel+, Joachim Deisinger*, Frank Weller~, Thorsten Hamisch#+Institute for Machining Technology (ISF), University of Dortmund, 44227 Dortmund,

email: zabel@isf.mb.uni-dortmund.de*Institute for Human Factors and Technology Management (IAT) University Stuttgart, Competence Center VR, 70569 Stuttgart,

email: joachim.deisinger@iao.fhg.de~LS VII Computer Graphics, University of Dortmund, 44227 Dortmund,

email: weller@ls7.cs.uni-dortmund.de#State Academy of Art and Design Stuttgart, 70191 Stuttgart

email: design.design@abk-stuttgart.de

1 This project is funded by the Volkswagen Foundation, AZ: I/73526-73529.

KEYWORDS

Computer Aided Design, Manufacturing, Graphicspackages, Virtual Reality, Man-machine interfaces

ABSTRACT

This paper describes a multimodal environment tosupport all stages of the industrial design process. Thesystem is considered to support a designer starting at thevery early stages of the design process up to theproduction and reengineering of physical models. Thisswitching between the virtual and the physical world iscalled multimodal. To deal with all those stages, thesystem's architecture comprises commercially availabletools as well as completely new developed components.This architecture is presented and some parts of it willbe described in detail. Special emphasis is placed on thekernel of the system, a VR-modeling tool and on itsuser interface. An overview of the provided interfacesfrom and to the physical world, and an outlook onfurther work complete this paper.

THE "designDesign"-WORKSHOP

Nowadays, product design is a highly complex andcreative task, carried out under a number of constraintssuch as time and costs. The design process not onlyfocuses on creating the shape for a new product, butcomprises the whole range of product development,starting with the recognition of a "real-world problem"which is to be solved, and ending with the production ofa first prototype. Several phases of the design processcan be supported by computer modeling tools, whileothers cannot.

The very early stages, also called "conceptual phases",only depend on the experience, view, and creativity ofthe designer. There is no process description for theseearly stages and a designer can hardly explain how hedeals with them. In these stages, design is an entirelymental process dealing only with ideas and thoughts(Hofmann 1998).

During the subsequent stages of design - namedvisualization -, there is a strong need for supporting thedesigner with tools. In this case, the definition of toolscomprises digital tools as well as physical ones. The

main idea described in this paper is to combine thesetwo worlds, physical and digital, and to allow a designerto switch easily between them (Weinert et al. 1998b).

To achieve this goal there are many ideas andconclusions which were developed during a workshopheld in Stuttgart in June 1998. 22 designers fromseveral fields of design (architecture, industrial design,WWW-design, etc.) took part in the workshop anddiscussed their needs for additional digital and physicaltools to improve their personal design processes. Atechnique called brainwriting was used to extractfundamental ideas and gather opinions. Thosefundamental ideas are now serving as a basis for theproposed Multimodal Design Environment (MDE)(Hofmann 1998). These ideas are summarized in thefollowing fig. 1:

Fig. 1: Multimodal Design Process

Digital vs. Physical Modeling

It turned out that designers wish to use digital modelsfor various reasons:

A digital model allows fast and easy changesof material, shape, lighting, color, etc.

Duplication and transportation of digitalmodels are fast, low-cost operations. Variantscan be generated with little extra effort, andclose co-operation with colleagues and/orcustomers around the world becomes feasible.

A digital model is directly available forcomputer processing, e.g., stress or flowsimulations and CAD/CAM-production.

On the other hand, there are several reasons whydesigners are not likely to abandon physical models:

A physical model can be touched, and it has aweight.

There are properties of a model which cannotbe simulated nowadays, like flavor andsmelling, and even some geometric aspectsmust be felt by hand.

Many designers are used to physical modelsand have acquired high levels of skill withphysical tools. Some see no need for digitaltools and cannot imagine using them in areasonable way.

MDE is an approach to integrate both worlds in onedesign process by providing fast and easy transitionsbetween them. This is necessary in order to combine theadvantages of both worlds by avoiding their respectivedisadvantages.

THE MULTIMODAL DESIGN ENVIRONMENT

The MDE comprises a variety of different tools whichare designed to work together closely. There arestandard CAD/CAM-tools as well as a newly developedVR-modeling tool. Not all of them are running in a VRenvironment because for such tasks as surfacereconstruction and NC-data generation there is no needto carry them out in an highly immersive environment.

On the other hand, generation of new shapes orchanging an existing virtual model are tasks whichrecommend an immersive VR environment. Thefollowing fig. 2 shows the overall structure of (thecomputer-based part of) the system.

Fig. 2: Overall System Structure

The figure shown above presents basically three layersand the degree of immersions grows from the outer tothe inner circle. Between the layers there have to beinterfaces which are described later in more detail. It isimportant to notice that the overall system structurecomprises more than the VR-modeler and that only theco-operation of different tools on different levels willhelp designers in different stages of design.

The next paragraph first describes in more detail theVR-modeler, the different tools it provides and theirimpact on the chosen data representation. Theinteraction concept is central to a further paragraph. Atlast, the tools and interfaces which are used to changethe virtual models into physical ones and vice versa arepresented.

The VR-modeler

During the workshop mentioned above it became clearthat a great disadvantage of existing tools for virtualdesign is the lack of immersion. A potential user has toreach a high level of mathematical abstraction to workeffectively with those tools. The VR-modeler will closethis gap by allowing modeling in highly immersiveenvironments such as a CAVETM. On the other hand themodeler should scale down to desktop workstationseven though it's main operational field is anenvironment like a CAVETM.

Out of the results of the workshop and during a numberof further sessions in co-operation with designers a setof tools and corresponding operations was defined. Thefollowing table shows these results.

Intention Operation Toolspraying aerosolsketching/lines,splines

3D-Pen/Hand

sketching/surfaces,volumes

3D-Pen/Hand

generation ofgeometry

generation of 3D-primitives

selector(s)

addition: (spraying) aerosolsubtraction: etching rubber,

stencilsubtraction: cutting knife, cuttersubtraction: drilling driller

manipulationof geometry

subtraction: carryingof

file

deformation: deform hand, fingerdeformation: bend handmathematical: scale -mathematical: smooth -mathematical: morph -

combininggeometry

morph -

loft -union glue

copying copy -mirror mirror

manipulatecolor/texturekinematics -geneticoperators

nature

Table 1: Tools and Operations

Of course the modeler provides the basic functionalityof each CAD/CAM-system, like loading/storing,importing/exporting, changing the point of view andadding and manipulating of light sources, which is notdescribed in detail here.

It was a great effort to identify this list of basicoperations for modeling and it is certain that there aremany designers who will miss a special operation.There were great difficulties to classify designprocesses and it becomes clear that extracting basicoperations and tools for design processes is verydifficult too.

Notice that many tools like the sketching tools willwork in a fuzzy way. That means they are notimplemented as a common CAD/CAM-system where auser has to define control points of curves and so on.Sketching is allowed by directly painting in the 3D-environment and the computer cares about the technicaldetails like putting lines together and generating data.This is very similar to other approaches like (Igarashi1999) (Lüddemann 1996).

At the beginning of the work it was tried to find outdigital metaphors for existing tools. This emerged to beinsufficient because there are digital tools which haveno equivalent in the real world and vice versa. Ofcourse some metaphors have to be provided but to usethe whole potential of modeling in virtual space it isnecessary to even provide those tools without real worldequivalent, e. g. the morpher. The designer has to acceptthese tools if he wants to profit from the advantages ofvirtual modeling. They may be able to improve thedesigner's creativity just when he starts playing withthem and when he uses them in a way the developers ofthe modeler never thought of2.

A Hybrid Data Model

Due to the various tools and operations which should beprovided by the VR-modeler it turned out that onesingle data model is insufficient. For some operations,like spraying, discrete models are appropiate whileothers like deformation, scaling or morphing can beperformed best by using a continuous representation.Especially the fuzzy tools and operations fit much betterinto a discrete model structure. To union different datamodels in one system it is necessary to provide fastalgorithms to transform the data from onerepresentation to another. This should happen withoutthe user recognizing it. It has to be completelytransparent for the designer. To reach a goodperformance it is possible to maintain some data modelsin parallel and just updating them according to theactual operation.

2 Often designers can draw the biggest advantages outof new tools by "playing" with them and experiencetheir functions.

The VR-Modeler's Interaction Concept

The interaction concept can be split into basic conceptand application concept. The basic concept representsthe structural frame and defines menu structures. Hereinput and output devices as well as interactionmetaphors and data interfaces are defined. On top of thebasic concept the application concept defines thefunctional range of a specific application - in this casevirtual modeling - and the semantics to implement itinto the basic concept.

A designed basic concept interface to serve thevisualization and evaluation of CAD geometry in asmall group is shown below (Häfner 1999). Here thepresentation of the geometry has top priority.

The virtual cursor (fig. 3) is designed to support theuser in estimating linear and circular sizes in the virtualenvironment. It must be well-visible in the environment,but should not cover the visualized objects.

Fig. 3: Virtual Cursor

The interaction ball (fig. 4) has four separatedsegments with short text labels on them. Each segmentrepresents a possible selection. The circle segment infront of the ball represents the basic position. With theselection button pressed, the ball can be moved androtated. The rotation is measured, but the ball rotatesonly in defined angles. This means, the ball snaps to amenu point automatically.

Fig. 4: Interaction Ball

The action is selected by releasing the selection button.

The marker (fig. 5) is designed as an arched ring and ahalf cone pointing to the middle of the ring. This shapeis visible from different viewing angles and positions.Colors and brightness support the shape and thevisibility. Each marker is uniquely numbered andoutlined in the documentation with its unique position.

Fig. 5: Marker and Cutting Plane

Software System Used For The VR-modeler

The major goal of VR-systems is to let the user interactwith an application. In order to do so the user interfaceprovides the necessary software layers and devices. Thefollowing fig. 6 shows the complexity of such an VR-system.

Barriers for the industrial application of VR forvisualization and simulation as well as for the industrialdesign process are currently due to limitations ofusability (the non-intuitive nature of the interfaces, bothhardware and software, and concerns about possibleside and after effects particularly with HMDs).

Fig. 6: User Interaction with the Application

The software environment which is used, is thecommercially available VR-environment "Lightning"which was developed by IAT, one of the projectpartners. This environment provides interfaces toTcl/Tk and C++ and these interfaces will be used tointegrate existing volume models in Lightning. Inaddition Lightning allows its users to configure it in awide range and it is able to deal with a variety of inputand output devices.

MDE'S INTERFACES TO THE REAL WORLD

Of course the VR-modeler stands not alone. Asmentioned above there have to be interfaces into thephysical world. These interfaces consist of hardwarelike digitizers (laser and tactile), cameras and otherdigitizing devices on the input side and mainly a High-Speed-Cutting-Machine on the output side. Because oflower costs, the cutting process will often be replacedby simulating it and generating images of the desiredoutput (Weinert et al. 1998a) (Weinert et al. 1998c).The software which will be used to build the "real world

interface" is a bundle of existing CAD/CAM-packagesin addition to professional software for reverseengineering and some special purpose tools which weredeveloped at ISF. Main interface to the HSC-machineand the simulation is NC-Data according to DIN 66025which is generated by a standard CAM-system.

To involve a CAM-system in the process chain, the VR-modeler must be able to export the objects modeled bythe user. To simplify the process chain, it is sufficient ifthe modeler is able to export the triangulated surfaces ofthe actual model. The used CAM-system can thengenerate NC-paths based on such a triangulation. It maybe necessary to optimize the initial triangulation andthis can be done by using some special tools developedat ISF (Weinert et al. 1998d). Because of the objectswhich have to be "produced" there are no strongconstraints concerning accuracy and machiningtechnology. The used materials will often be plasticsand other model materials like aureole. Therefore thedesigner need not dealing with things like optimalcutting parameters and strategies. Also the accuracy isnot as important as it is normally in machiningtechnology. If there are fuzzy 3D-sketches which haveto be produced there is no reason to produce them withan accuracy of 1/100 mm. All in all there are just a fewstandard procedures the designer (or maybe an expertwho supports him) has to carry out to get his physicalmodel out of the virtual space.

The more challenging task is to reengineer an existingphysical model to transform it into a virtual one, whichcan be handled by the VR-modeler. Even here theaccuracy and technology are not first order problemsbut there are some constraints the designer at least hasto be aware of.

The institutes involved in this project dispose of variousscanner systems for three axis (tactile) and five axis(laser) scanning. The point clouds produced by thosesystems have major distinctions and it is important evenfor the end user to be aware of what kind of points hecan produce and how his physical model is shaped.Even if this knowledge is given it may be very difficultto generate the desired data. Often the models have tobe scanned in different positions and orientations andthe resulting point clouds need to be matched. Thismatching is a complex and time consuming work andthere are no tools known which can deal with this taskin a fully automatic way.

Many designers use materials like wire meshes filledwith gypsum or paper models. Those materials do notadmit tactile scanning due to their fragility and laserscanning them often produces poor results. Here itwould be helpful to use photographic systems. Also ifthe model can be scanned by a tactile scanner there is agreat effort in adjusting and preparing the scanningsystem. The user has to choose strategies and segmentthe model and both tasks strongly depend on themodel's shape. For this reason a kind of automatism has

to be found at least for 3-axis scanning of models. Thissubjects of current investigations at ISF.

Another problem is the so called "surfacereconstruction" that means the transformation of pointclouds into CAD logic. There are a few knownautomatic approaches (Keller et al. 1998) which can behelpful for this task. At this point it is an advantage thatthe VR-modeler is based on a hybrid data model. Thisallows to choose a suitable representation for the inputdata so that it is not necessary to perform the wholereconstruction process. For example a point cloud canbe transformed into a voxel model relatively easily. Thefollowing figure (fig. 7) shows the data paths andtransformations to switch as easily as possible betweenvirtual and physical world.

Fig. 7: Tools and Data Pathes for Switching betweenVirtual and Physical Models

Of course not all possible paths of data and all availabletool are of shown in fig. 7. There are jut those whichwill be implemented in the demonstrator MDE. Alsosteps like triangulation optimization and othersmentioned above are not shown in detail, because thetools are easy to use input/calculation/output-tools. It isimportant to notice that it may be necessary to converteven the virtual objects from one representation (dataformat) to the other. This is shown by the broken line inthe middle of the "virtual objects" ellipse. It is alsoimportant that the user does not have to "really"produce his workpiece. There is the possibility tosimulate the cutting process what is done normally toadopt feed rates. The provided simulation is alsocapable of creating rendered images of the workpiecewhich are very similar to the physically produced model(Weinert et. al 1998c). This feature can be used for thereducing of time and costs.

CONCLUSION

The MDE described in this paper will support designersduring different stages of their personal design tasks. Itfocuses on the shape giving process in industrial designbut of course there a many other design tasks.Nevertheless it can be very intuitive to sketch in avirtual environment even if the main goal of a design

process is not the production of shapes. There are otherapproaches like sketching user interfaces (Landay andMyers 1995) which show that tools that allow adesigner to focus on intuitive and fast "visualization" ofideas can be important instruments, e.g. incommunicating with customers.

During the next few months our main goal is toimplement a demonstrator-VR-modeler which providesonly a subset of tools mentioned above but whichincludes already the data-structures needed for theunderlying models. In addition to this task the processchain from digitizing up to producing of physicalmodels will be traversed for some example workpieces.During a session at State Academy of Art and Design inStuttgart, three so called reference models weredeveloped and they will be processed by thedemonstrator and the surrounding tools to measure theability of the system to give real support to theaddressed designers.

REFERENCES

Häfner U.; Doulis M. and Simon A. 1999. "Evaluation of ComplexCAD Data in a Virtual Environment". In Proceedings of the 3.International Immersive Projection Technology Workshop. (Stuttgart,Germany May 10.-11.). Forschung und Praxis T 52. Springer VerlagBerlin. 157-166.

Hofmann D. 1998. Documentation of the designDesign-Workshop.State Academy of Art and Design, June 22.-23. Stuttgart, Germany.

Igarashi, T.; Matsuoka, S. and Tanaka, H. 1999. "Teddy: A SketchingInterface for 3D Freform Design". In SIGGRAPH 99 ConferenceProceedings. (Los Angeles, CA , USA, Aug. 8.-13.).

Keller, R.; Banzhaf, W.; Weinert, K. and Mehnen, J. 1998. "CADsurface reconstruction from digitized 3D point data with geneticprogramming." In (Koza J. R. ed.) Proc. Genetic Programming (GP'98) - Late Breaking Papers (Madison, WI, July 22.-25). 106-112.

Landay, J. A. and Myers, B. A. 1995. "Interactive skteching for theearly stages of user interface design". CHI'95 ConferenceProceedings. 43-50.

Lüddemann, J. 1996. "Virtuelle Tonmodellierung zur skizzierendenFormgestaltung im Industriedesign". Berlin Techn. Univ. Diss.1996/5986. In: Berichte aus dem Produktionstechnischen ZentrumBerlin. UNZE Verlagsgesellschaft, Potsdam.

Weinert, K.; Albersmann, F.; Guntermann, G. 1998a. "MillingSimulation with an adaptive Strategy." In Proceedings of SculpturedSurface Machining Conference, Chrysler Technology Center AuburnHills. Michigan, USA, Nov. 9-11.

Weinert, K.; Albersmann, F.; Damm, P and Zabel, A. 1998b. "Virtual-Reality in Design und Anwendung – Einsatz von virtueller Realitätvon frühen Designphasen bis hin zur Fertigung". VDI-Z IntegrierteProduktion 140. no. 9 (Sept.): 48-51.

Weinert, K.; Albersmann, F. and Zabel, A. 1998c. "Verbesserung derProzeßsicherheit beim HSC-Hartfräsen durch eine adaptiveSimulation und effiziente Online-Visualisierung". VDI-Z IntegrierteProduktion 140. No. 6 (June): 61-63.

Weinert, K.; Mehnen J.; Albersmann, F. and Drerup, P. 1998d. "Newsolutions for surface reconstructions from discrete point data bymeans of computational intelligence." In Int'l Seminar on IntelligentComputation in Manufacturing Engineering (ICME '98) Capri, 1.-3.Juli, CIRP, Paris, 431-438.