Privashoe

Abstract For this project, a shoe rack with three levels was built that was used together with the Peek-A-Boo camera module system. To this camera module a distance sensor, sound sensor, and a switch were attached in order to make pictures of the shoe rack within a household of a period of 1,5 week. The goal of this research was to find out if the shoe rack photos would show the presence and daily activities of the household in a highly privacy retaining manner. The privacy was retained even more by coding the distance sensor and sound sensor in such a way that it would be prevented that photos of household members walking would be taken. In the end, it seemed hard to actually see what activities participants were doing. However, the

doorbell data and the placements and displacements of shoes showed a variety of interesting situations regarding presence and activities. Especially the way in which the shoe rack sometimes changed the behavior of participants showed an interesting opportunity for design. And although the shoe rack provided a minimal view, it did provide a lot of information about the presence of household members over time in a highly privacy retaining manner. Author Keywords Shoe rack; Data annotation; OOCSI; Privacy; Sensors; Module system; Peek-a-Boo Topic of Interest The goal of this project was to find a scene in the household that would be interesting to capture. The scene should allow enough data to be captured and give impressions or results of activity in the household. A value that was seen as highly important during this course was privacy. Especially the spectrum in which privacy and publicity can occur was regarded as an interesting topic for design (Nippert-Eng, 2007). Through means of brainstorming and making a development plan, the topic of a shoe rack was initiated. Here the plan was created to design a shoe rack with different levels for each member of the house. The shoe rack would then be installed in the house and photographed by the Peek-A-Boo camera. The goal of this research was, therefore, to capture the presence and activities of household members in an informative but privacy retaining way. In the set-up, the Peek-A-Boo camera would be placed on the ground directly at the shoe rack. Through this, the Peek-A-Boo cam would very likely, only be able to take pictures of

M.P. Verheijden 1234306 Technical University of Eindhoven Eindhoven m.p.verheijden@student.tue.nl K.P.P. Barten 1221456 Technical University of Eindhoven Eindhoven k.p.p.barten@student.tue.nl

D.R. Tjokroatmo 1234410 Technical University of Eindhoven Eindhoven d.r.tjokroatmo@student.tue.nl N.J.A.J. van den Berk 1234269 Technical University of Eindhoven Eindhoven n.j.a.j.v.d.berk@student.tue.nl

household members feet and legs. Only if a household member would kneel in front of the rack the face could be captured. This is one of the reasons why this view was so interesting to choose from. It shows the activity in a household, however, is unlikely to capture a residents’ face, even when there are no modules installed. Secondly, it was expected that the addition of the shoe rack would influence the behavior of household members. The shoe rack in combination with the Peek-A-Boo cam would not only capture the behavior of people living in the household but also influence it by introducing a new (and structuring) object in the household. Some privacy concerns were there, however. Especially regarding the capturing of presence in general. Being away from home might still be regarded as personal and not always something desirable to share with family members. Additionally, the shoe rack view would not be capturing faces of people but it could still capture private parts of the body which is not something desired. The modules would try to redeem these privacy concerns.

System Design Introduction In order to reach the research goal modules were added to the system. A total of three modules were added to the Peek-A-Boo camera system. It was decided by means of the development plan to add a distance sensor, a sound sensor (not to be confused with a microphone) and a switch. The distance sensor was added in order to prevent the Peek-A-Boo camera from making pictures while a person would walk past the camera module. This would redeem the privacy concern of making a photo of any part of a person whatsoever. Next to this a sound sensor (which would not record data, only detect sound) would be placed next to the doorbell. This sensor would detect if the doorbell rang and would, as a result, disable the Peek-A-Boo cam for a period of time. This would again redeem the privacy concern of taking pictures of people coming in and household members opening the door. This sensor would additionally be a good indication of whether a new member entered the house. This would not necessarily redeem privacy concerns but would ensure the system would know better if new people are present in the house or for what reason people would leave (a friend who is coming to pick someone up). The switch was placed next to the Peek-A-Boo to turn the camera of the module off for a period of time. This would increase the sense of control (Nippert-Eng, 2007) household members would have over the whole module system. Being able to turn the system off for a period of time would prevent privacy infringing situations if, for instance, the corridor had to be cleaned.

System Deployment As described before, it was needed to deploy 4 modules. Our main module, the shoe rack had to be placed underneath the staircase. This was decided because the inhabitants placed their shoes before our deployment and to not interrupt them with our research. The second module, the sound sensor, had to be placed next to the doorbell. The closer to the doorbell, the less background noise and the more reliable our sensor measures the activity of the doorbell. Our third sensor, the button, was placed in plain sight as it had to be had to be easily accessible. And our final sensor obviously had to be the Peek-a-Boo camera. We focused on the shoe rack, so the camera had to be in front of it. Again, to not hinder the inhabitants, the camera was placed on the other side of the hall against the wall to make sure there was still room to walk. In short, we made sure all the modules and sensors were placed to guarantee the privacy of the inhabitants and not disturb any of their daily activities.

Interview Questions Together with the sensors which gathered data, there also had to be an interview with certain explanations and insights in the daily life of the participants. To gain as much information as possible, questions were clustered in two groups: Privacy and about the module and Activities. The first cluster was mostly about how the inhabitants felt when the modules were deployed, for example, “Did you feel your privacy was being violated? Why?” or “Did you use the switch often? When?”. Just to see if the inhabitants privacy was as much maintained as we thought it would be. The second cluster was more about explanations of the pictures which were taken. So questions such as “Do you have a certain pair of shoes for a certain activity?” or “At what time do you go to work and do you come home?”. With the answers of those questions, we could elaborate more when the pictures were analyzed. The whole questionnaire and the answers of it are put in the appendix.

Data Annotation Strategy Since we chose to monitor our self-made shoe rack with the peekaboo camera, we have given ourselves a head start with regard to the data analysis. The camera generates very close up photos of the shoe rack, where the focus lies on the shoes themselves. This results in the photos having very little variables to take into account when analyzing. Since we only have to focus on shoes appearing and disappearing, we can already filter out a majority of the pictures (where no changes appear to be visible). This resulted in a rather small set of photos, where we could quickly see what happened over time. To analyze and annotate the data, we therefore chose not to use the built-in annotation tool of the peekaboo camera, since it would cost more time to use the data.csv. As mentioned, one of the key frames was the coming and going of shoes, but in more detail, we focused on:

1. Looking at routine of the shoes being gone and coming back. Is there a pattern per day? Ask about this in the interview.

2. Is there a relationship between the doorbell ringing and shoes being added in the shoe rack? If so, who has come in? Is there a routine within this as well?

3. Did the house members recognize the camera making a photo or not? Did it obscure their privacy in any way? Ask about this in the interview if they recognize one of the photos where they had to stop an activity in the hallway to not get in the picture.

4. Are there any Photos linked to an activity? Has there been a party where lot of shoes are added? Or has the family went exercising all together where at one point all sport shoes are gone? See if there is any pattern in certain shoes leaving or not that can be linked to an activity or event within the household.

After the data gathering, all unclear pictures (e.g. because of darkness) and pictures with no changes were discarded. This resulted in a picture set of about 40 photos. Since the quality of the peekaboo camera is quite bad, these pictures were then corrected, applying lighting corrections and cropping to the right format. This helped with analyzing what happened in the pictures. The selecting and filtering was done by two group members, while the editing and correcting was performed by a third member. KeyFrame Findings

Keyframe 1 - Going out with a friend was a very interesting finding which was not expected to be found initially. To find this key frame finding a real good look through the data had to be done. When looking at the

photos itself it would seem like the shoes just disappeared because a household member had left the house. However, when looking through the data a very interesting finding was done. Namely that the doorbell had between the two photos that were taken. It was not known why this person left the house afterward so there was a good opportunity to ask about this in the interview questions. When asking this it appeared that this household indeed left the house because a friend of him came to pick him up to go to the movies. This was kind of expected but the interview confirmed this finding. Such a finding could be especially interesting for a smart home design which would provide the household members information about the specific trip. Or a smart home design which would prepare for (e.g. in terms of changing the thermostat) for the comeback of the household member after the movie has finished.

Keyframe 2 - This keyframe finding became very interesting through means of the interview answer. At first, these two photos were chosen because it was recognized that the shoes that were first placed in the shoe rack were moved to a location behind the shoe rack. This indeed seemed to be true when interviewing. This keyframe, however, seemed to regard a deeper theme found within the module system proposed. As said earlier, the shoe rack system would bring a new system in the household. Interviewees indicated that they felt annoyed putting their shoes in the rack they used last. Most participants in this household had more than three shoes. However, the household members tried to apply to the rules of the shoe rack system which thus changed their structured behavior. This could bring a whole new opportunity for design. Finding out whether this shoe system would serve as a good system to track the presence or not intervene in normal life should be regarded as an important aspect of this project. The third keyframe would show even more that the shoe rack system together with the Peek-A-Boo camera would evoke a response for participants regarding their behavior in daily life.

Keyframe 3 - “What we did was just picking a random pair of shoes from the shoe rack and putting them on the other side of the hall. This way we could wear whatever we wanted.” As told in keyframe 2, the inhabitants were sometimes annoyed by the fact they could only wear a certain amount of pairs. They solved this by switching out a couple or pairs throughout the week to give them the possibility to wear different shoes whenever they liked.

Keyframe 4 - When the photos were edited, they had to be cropped a bit. This resulted that the aspect which is discussed, is not clearly visible. To make it clearer, in the bottom image there was a box placed at the right side of the photo which was not there in the op image. During the interviews, there were some notions of irregular working hours and such. After some more in-depth questions it turned out that one of the inhabitants was packing their stuff to get to work and put the bag and boxes in the hallway before he left the house.

Keyframe 5 - Next to the irregular working days from some of the inhabitants, there was also someone who had a very strict schedule when leaving for work and coming home again. We found out that a certain pair of shoes always disappeared and showed up at certain periods of time, namely 8:15am and 5:30pm. We came to the conclusion after the interviews in which one of the inhabitants said that only one person who had a more precise schedule. Fun to see from this keyframe is that not everything from our data is irregular.

Reflection When looking at the deployment and keyframe findings, there are a few things that could get improved. When a second round for observation would be performed, there are things that are liked to be done differently. Also, better observation could be made according to the research goal. In this section, it will all be discussed.

In the previous section, some keyframe findings were provided, as well as the interview results. One of the findings that came out of this data was that the participants did not like the fact that they could only choose out of three different pair of shoes every day. This problem could easily be solved. The only change that is needed is a fixed place in the shoe rack for every person, not for every pair of shoes. This way, the system still knows who is at home and who is not. However, now, it does not know which exact pair the inhabitant is wearing, which is no problem at all because that data is was also not used in this first round of the observation project.

The fact that the camera used a fixed exposure resulted in totally black pictures in the evening, morning and night. If a camera should be used that changes its expose according to the amount of light there is in the room, there could be gathered much more data. Then, the system would also know who is at home at the hours the sun is down. This leads to an overall better observation according to the research goal.

Another key finding that came out of the picture data was that there was actually a lot more recorded than just the shoe rack itself, because of the wide angle lens of the camera. During the second round, it would be better to use a normal lens, or place the camera nearer to the shoe rack. This way, no extra information is gathered, which is an improvement when taking into account the privacy of the participants.

Value Proposition In this section, it will be explained what value the design delivers to the customer, what customer problem it will help to solve and what customer needs are satisfied. Next to that, this section will provide a future perspective on how this design could be a real product service.

The design of team Privashoe provides customers information about the shoes that are placed in the shoe rack. This gives the customers insight into whether, and which shoes the other inhabitants are wearing. This information is on its own not very useful for the users. However, it can become valuable when this information is sent to other connected products and services Privashoe that could collaborate with in the future. These products could me smart thermostats, smart blinds or smart lights. Every inhabitant has its own preferences of temperature and light. The only problem is that these connected products do not know who is at home. When this information could be provided, the level of automation reaches a whole new level. No superfluous input by the users is needed anymore.

The system does not only know who of the inhabitants is at home, but it can also know when there are visitors because it makes use of a sound sensor that knows when the doorbell rings. When the sound sensor is triggered, the system knows there is someone at the door that does not live there. When the motion sensor is triggered shortly after this moment it means that the people have entered the house. This information can also be useful for the smart home system, in order to adjust to the users’ needs and preferences.

References 1. Nippert-Eng, C. (2007). Privacy in the United

States: Some implications for design. International Journal of Design, 1(2), 1-10.

Appendix 01: Interview Note Privacy and about the module

1. Did you feel your privacy was being violated? Why? o No, the camera was not pointed at us, but

at the shoe rack. Next to that, it never took a picture when we were in front of the camera

2. Did our module hinder your daily routine? Why? o It was quite frustrating to always choose

the shoes from the shoe rack. At a certain point, we started putting on other shoes. What we did was just picking a random pair of shoes from the shoe rack and putting them on the other side of the hall. This way we could wear whatever we wanted.

o The other frustrating thing was that we had to watch out for the electronica. We were not allowed to touch or move it. This made us scared to touch it by accident.

3. How did you solve these problems that hindered your daily routine? o I swapped one of my shoes standing

behind the rack with one standing in the rack. I wore the shoes standing behind the rack.

4. Did you use the switch often? When? o No, we never used it.

5. Have you got any feedback/improvement on our module? o Think of a way to make it possible to wear

every random shoe we want. Also, make sure that the camera module cannot be touched by accident.

Activities 6. Do you have a certain spot for a certain pair of

shoes? o Yes, every pair of shoes had their own

place in the rack. Every person in the house had its own row, containing 3 pairs.

7. Do you have a certain pair of shoes for a certain activity? o Yes, we had three pairs of shoes. One for

work, one for in the weekends and an extra pair for sunny days.

8. At what time do you go to work and do you come home? o Someone goes to work at 08:15 and

comes back home at 17:30h. Corina has changing times every day, just like Koen.

9. Have you got any hobbies/activities which requires you to leave the house? (groceries, sports, visits etc.) o Doing groceries, go walking, go to the

gym, go to friends, go shopping 10. What did you do during this transition of

photos? (Show change in photos) i. -

11. Do you get a lot of visitors? How did this influence the interaction with the module? o We got some visitors in the weekend.

These people held on their shoes. It will have triggered the doorbell sensor and the distance sensor because they walked through the hall.

12. Do you get a lot of post/packages? How did this influence the interaction with the module? o There were a few packages delivered. It

will have triggered the doorbell sensor and of course the distance sensor because we walked to the door via the hall. When post got delivered, no sensor got triggered because they did not have to ring the doorbell.

13. Is there anything in this area which the camera did not capture, which is important/surprising to know? o No.

14. Did you behave differently because the module was watching? Why? How? o We had to choose the shoes from the shoe

rack. That made our daily routine different. Next to that, it was very fun to watch the module taking pictures.

15. At date X we saw that the doorbell rang at around time X. The picture taken after this showed that a pair of shoes disappeared from the rack. What happened there? o A friend of mine picked me up to go see a

movie. It was very fun. I didn’t notice the Peek-A-Boo taking a picture. I only now notice the doorbell data was stored.

Appendix 03: Documentation of API Our system consists out of the peekaboo camera and three additional sensors; a distance sensor, a sound sensor and a on/off switch (the heart of the system). The system works in such a way that only the on/off switch controls the peekaboo camera. The distance sensor is able to send data to the on/off switch, which can then delay the picture being taken. The sound sensor only uploads data to the peekaboo cam, indicating when someone rang the door. The data flow can be visualized as:

Code On/off switch, MAIN SENSOR

// ON/OFF SWITCH (MAIN SENSOR) #include "OOCSI.h" const char* ssid = "Peekaboo"; const char* password = "boxeslittleboxes"; const char* OOCSIName = "mainsensor1"; const char* hostserver = "192.168.4.1"; OOCSI oocsi = OOCSI(); const int powerPin = 32; const int measurePin = 33; bool privacyMode = false; int privacyValue = 0; bool flagUp = false; bool picTaken = false; long startPoint = 0; long multiplier = 0; long offset = 0; long interval = 1800000; //30 minutes long flagInterval = 15000; // 30 seconds long bellSensor = 100; long distanceSensor = 100; void setup() { Serial.begin(9600); // Starts the serial communication oocsi.connect(OOCSIName, hostserver, ssid, password, processOOCSI); oocsi.subscribe("modulechannel"); pinMode(powerPin, OUTPUT); pinMode(measurePin, INPUT); digitalWrite(powerPin, HIGH); startPoint = millis(); }

void loop() { oocsi.check(); // Checking if privacy mode is turned on by the user privacyValue = digitalRead(measurePin); if (privacyValue == 1) { privacyMode = true; } else { privacyMode = false; } // Seeing if all conditions to take a pictures are met, and if it is time to raise the flag if (millis() > (interval * multiplier) - flagInterval + offset && privacyMode == false && flagUp == false && picTaken == true) { oocsi.newMessage("peekaboo_control"); oocsi.addString("flagUp", ""); oocsi.sendMessage(); flagUp = true; oocsi.printSendMessage(); oocsi.check(); varUpdate(); } // Seeing if all conditions to take a pictures are met, and taking the picture if (millis() > (interval * multiplier) + offset && privacyMode == false) { oocsi.newMessage("peekaboo_control"); oocsi.addString("triggerPhoto", ""); oocsi.sendMessage(); multiplier++; picTaken = true; offset = 0;

oocsi.printSendMessage(); oocsi.check(); // Giving the peekaboo cam some time to process the picture delay(10000); oocsi.newMessage("peekaboo_control"); oocsi.addString("flagDown", ""); oocsi.sendMessage(); oocsi.printSendMessage(); oocsi.check(); flagUp = false; varUpdate(); } delay(50); } void processOOCSI() { int moduleData = oocsi.getInt("moduledata", 0); // If moduledata is 1, the doorbell was triggered, here we save the data to the peekaboo camera if (moduleData == 1) { Serial.println("Doorbell triggered"); oocsi.newMessage("sensordata"); oocsi.addFloat("sensorvalue2", bellSensor); oocsi.sendMessage(); oocsi.printSendMessage(); oocsi.check(); bellSensor += 100; } // Here the distance sensor is triggered if (moduleData == 2) { oocsi.newMessage("sensordata"); oocsi.addFloat("sensorvalue1",

distanceSensor); oocsi.sendMessage(); oocsi.printSendMessage(); oocsi.check(); distanceSensor += 100; // We see if the photo should be delayed or not Serial.println("Shoerack triggered"); if ((interval * (multiplier + 1)) - millis() > 600000) { if (((interval * multiplier) + offset) - millis() < 10000) { offset += 30000; } else { // do nothing } } else { offset = 0; multiplier++; varUpdate(); } } } // Printing all variables for debugging void varUpdate() { Serial.println("Variable update:"); Serial.println(""); Serial.print("Multiplier: "); Serial.println(multiplier); Serial.print("Current time: "); Serial.println(millis()); Serial.print("Next picture at: "); Serial.println((interval * multiplier) + offset); Serial.print("Original time: ");

Serial.println(interval * multiplier); Serial.print("Offset: "); Serial.println(offset); } Distance Sensor, (Sensor value 1)

// DISTANCE SENSOR, (SENSORVALUE 1) #include "OOCSI.h" const char* ssid = "Peekaboo"; const char* password = "boxeslittleboxes"; const char* OOCSIName = "sensormodule3"; const char* hostserver = "192.168.4.1"; OOCSI oocsi = OOCSI(); const int trigPin = 32; const int echoPin = 33; long duration = 0; int distance = 0; int threshold = 65; bool measured = false; long measureTime = 0; long interval = 25000; void setup() { Serial.begin(9600); // Starts the serial communication oocsi.connect(OOCSIName, hostserver, ssid, password, processOOCSI); pinMode(trigPin, OUTPUT); pinMode(echoPin, INPUT); } void loop() {

// Measuring the distance digitalWrite(trigPin, LOW); delayMicroseconds(2); digitalWrite(trigPin, HIGH); delayMicroseconds(10); digitalWrite(trigPin, LOW); duration = pulseIn(echoPin, HIGH); distance = duration * 0.0343 / 2; // If the user is close enough to the shoe rack, the sensor is triggered, sending a signal to the main sensor if (distance < threshold && millis() > measureTime + interval) { measureTime = millis(); Serial.println(distance); oocsi.newMessage("modulechannel"); oocsi.addInt("moduledata", 2); oocsi.sendMessage(); oocsi.printSendMessage(); oocsi.check(); measured = true; } } void processOOCSI() { // don't do anything; we are sending only }

Sound Sensor, (Sensor value 2)

#include "OOCSI.h" const char* ssid = "Peekaboo"; const char* password = "boxeslittleboxes"; const char* OOCSIName = "soundsensor1"; const char* hostserver = "192.168.4.1"; OOCSI oocsi = OOCSI(); const int micPin = 33; int micLevel = 0; void setup() { Serial.begin(9600); // Starts the serial communication oocsi.connect(OOCSIName, hostserver, ssid, password, processOOCSI); pinMode(micPin, INPUT); } void loop() { // Measuring the level of sound, if triggered, sensing a signal to the main sensor micLevel = digitalRead(micPin); if (micLevel == 1) { oocsi.newMessage("modulechannel"); oocsi.addInt("moduledata", 1); oocsi.sendMessage(); oocsi.printSendMessage(); oocsi.check(); delay(10000); } delay(50); } void processOOCSI() { // don't do anything; we are sending only }