cs 607 final report

7/31/2019 CS 607 Final Report

1/18

SurveyAnywhere: A Field data collection and Synchronization system for

portable devices

Amiangshu Shekhar Bosu Md. Ashfakul Islam

[email protected] [email protected]

Introduction

Field data is important for any kind of research. Marketing researchers collect data from people to

analyze market and make their decisions about next marketing move. These decisions may run the

company to entirely different direction. Software engineering researchers are interviewing designers,

developers, quality tester and other stake-holders to gather their experiences, views and other data to

analyze them from different software engineering perspectives. Biological researchers go to various

natural inhabitations to collect their required data, pictures of various wild lives. Researchers in each

fields depends on field survey data to continue their research. Even different organizations like an

university runs survey among students to know about their opinions regarding different issues ( e.g.

course curriculum, course structure, instructor evaluation, transportation system, dining facilities, and

library services ) and make better decisions for students. At present surveys are very common.

Lot of organizations and projects do not have required manpower to conduct a survey. Outsourcing a

survey is very popular in almost each sector. Lot of organizations out-source their surveys on behalf of

their clients. Clients design their survey, target their population and make all decision related to convey

those surveys. Outsourced company make plan for execution, choose appropriate manpower and

distribute among workload among those surveyors. Surveyors go to field and communicate with or

examine their surveying subjects, fill all data collection forms and gather those data for the client.

Portable devices are getting smarter with time. Laptops, PDAs, Net-books, tablets, and smart phones are

becoming richer in sense of computational power, storage and network resources. Especially smart

phones are becoming very popular in these days. Smart phones can serve a variety of service to their

users besides regular phone services. Different kinds of apps are developed to serve different needs of

users. So, smart-phones can be used very conveniently to collect data for a survey. One thing a surveyor

needs to use their smart phones for data collection is an appropriate application, which is designed and

built for smart phones for that particular survey. But design and characteristic of data item in twodifferent surveys are not same. So surveyors need to install different applications for different surveys,

which is very non-user friendly especially for the outsourced surveyor companies. They need a

customized application for surveying. Moreover if they want to change some form on their survey, they

has to rebuild the application and redistribute. They also need a nice and structured framework for

these data collection process because uploading collected data directly from the portable devices is time

consuming and infeasible sometimes.
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


2/18

We have designed and developed a flexible framework and an android application to manage survey

design, survey distribution, data collection, and data synchronize between database server and smart

phones.

System Architecture:The system can be modeled in two parts: Server system and Client system. Researcher or owner of a

survey can create their data collection forms through the web interface easily. They can manage access

to their forms and collected data in database server. On the other hand surveyor can login to system

through android application and access to desired survey forms and upload data to server after

completion of a survey.

Figure 1: System Architecture

Database

ServerWeb Server

Rest

Service WebFrontend


3/18

Server system

Server system will have the data storage. It will have a web-based front-end that will enable

administrator to define the data collection form. He can specify the field names, data types, attributesetc. Based on those definitions the system will customize its REST based web service. The server system

will run on a cloud platform (SAAS- Software as A Service). The system will use database to store the text

data and storage (e.g. S3) for storing binary data (e.g. pictures, videos, sounds). Cloud platform will

enable the application to be scalable. And the users will not need to worry about data backup. The

system will also allow the users to download the files easily and export the data in different formats (e.g.

csv, xml)

Client systems

The client systems will run on different portable devices. Our initial target is to develop a client

application for Android devices. The application will connect to the server and retrieve the formattributes from the server. Using that definition the application will render the form to the user. The

users will fill the form using collected data and submit the form to the server. The users can attach

different types of files (e.g. videos, pictures, sound) when they upload the data.

As we are separating the two sub-components of our system using a defined interface (REST based web

service), the sub-components can be updated independently. And we can extend the scope of the

system by creating clients for other types of devices.

System Implementation

We have implemented the system according to our design. A web based system was implemented to

serve as Server system. And an android application was built to serve client system.

PHP was used to implement web front end and MySQL was used as back end database server. Kohana

framework (http://kohanaframework.org ) was used for system. Android SDK was used to develop

android application. JSON was used for exchanging data between android application and the server.

An administrator controls the web front end system. Password based user login was implemented. An

administrator can create new users for survey or delete users. He can also design the data collection

form. A nice UI was developed for data collection form building. He can add five types of field for data

collection form: Text box, Checkbox, Dropdown and Password box, Location data field. He can also

mention must require data fields in the forms. He is able see the android preview of the form and

reorganizes fields according his choice. After saving a form into database, a new table is created in

MySQL database automatically corresponding to that form. When surveyor submits survey data, it will

be stored to that table of the database. Administrator can also manage user access to forms: grant

access or revoke access.
http://kohanaframework.org/http://kohanaframework.org/http://kohanaframework.org/http://kohanaframework.org/


4/18

A password based login system was implemented in client side. After login, surveyors as user of client

system can see available form list and description of forms. Available form list will be downloaded from

server based on user. When user selects one form from the list, fields of the form will be downloaded

from database and it will be built dynamically by android application. User can also able to change their

password.

Figure 2: Database diagram of the system


5/18

Using the system

The system is divided into three components.

1. Web based frontend for administrative users2. Android application to access survey forms for mobile users3. REST web service to manage interaction between Database and Android application

1. Web based frontend for administrative usersThe web based frontend allows administrative users to:

1. Create add/edit/delete new mobile users (users that log in from portable devices and accessforms)

2. Create add/edit/delete survey forms3. Provide / Revoke access to a survey form to a mobile forms

Users login to the web-frontend using a public url. Our web frontend was hosted on Amazon Web

Service (AWS). AWS is cloud based and allows to dynamically scale the application. The URL of our

application was:http://ec2-107-22-45-207.compute-1.amazonaws.com/index.php(Figure. 3). The home

page contains brief motivation for the application and a login form for administrative users.

Figure 3: Home page and login form for administrative users

a) Administrative User homepageAfter the administrative user has successfully logged in, the system shows a welcome message (Figure.

4) and shows list of available actions to the user in top menu. Available actions are:

1. Form management: Edit / Delete/ Share existing survey forms2. Create new form: Create a new survey form3. User management: Add / Edit /Delete mobile users
http://ec2-107-22-45-207.compute-1.amazonaws.com/index.phphttp://ec2-107-22-45-207.compute-1.amazonaws.com/index.phphttp://ec2-107-22-45-207.compute-1.amazonaws.com/index.phphttp://ec2-107-22-45-207.compute-1.amazonaws.com/index.php


6/18

4. Change password: Change password for the administrative user

Figure 4: Homepage screen for administrative users after log in

b) User ManagementThe user management section allows to create Add / Edit /Delete new mobile users. On the landing page of User

management section there is a link to access the form to Create a new user (Figure. 5).

Figure 5: Landing page of User management section

The create new user form has 5 fields (Figure. 6). To create a new user the First name, Last name,

Email address and Password need to be filled. The confirm password field prevents accidental mistyping

of password.


7/18

Figure 6: Form to Create new user

As all the fields are required, there is Javascript-based validation implemented in the form. When anytype of validation is failed the system provides user proper feedback. It does not show any annoying

popup alert. Rather it highlights the field with red color. It also shows the user why the validation was

failed on the right side of the field (Figure. 7).

Figure 7: Dynamic validation and feedback to the user in new user creation form

After the user is created successfully, the user is redirected to the user management page. The new user is l isted

in the user list ofUser management page (Figure. 8 ). There is link to Edit / Delete user on the right side of each

user name on user list. Clicking Edit / Delete link will allow the admin user to Edit/ Delete that user.


8/18

Figure 8: New user is listed in the user list of user management page

For example, if the admin user clicks on the delete link the system will ask the admin user to confirm,

whether he really wants to delete that user (Figure. 9). This confirmation prevents accidental deletion of

a mobile user. If admin clicks Ok on the confirmation screen the user will be deleted.

Figure 9: Confirmation dialog to delete a user


9/18

After the user is created the system is redirected to the landing page of User management section. The

deleted user is no longer listed in the userlist. Admin can now Add/ Edit/ Delete more users. For

example, he might want to edit a user (Figure 10).

Figure 10: Editing a user

The edit user form is similar to the Create new user form. Similar validation is also implemented in this form

(Figure. 11).

Figure 11: Form to edit an existing user

After the user information is saved to the database. The form redirects to user listing page. It now shows that user

with modified information (Figure .12).


10/18

Figure 12: User list with edited user information

c) Create new formUser can access the section clicking on the Create New Form link from the top menu (Figure 10). The

form has two panels. On the left side user enters the name, description of the survey form. He can

dynamically add new fields to the form using the dropdown menu (Figure 13). On the right side there is

preview panel. Clicking on the Preview on Android button the user can see an approximate preview of

what the form might look like on an android device.

Figure 13: Section to create a new survey form


11/18

The create new form section is the main part of the application. This form is very dynamic and we

provided special attention to usability in while designing this form. In any type of survey basically there

are three types of questions (Figure. 13).

1. Free form response: Presented using text input2. Multiple choice: Presented using drop down list3. Multiple response: Presented using checkbox

We can have also included a special field named Location data. If that field is included, android will

send the location of the user when submitting the form. The fields can be added to the form by selecting

a new field type from the Add New Field from the drop down menu. Different fields have different

types of attributes. User can rearrange the fields, remove a field, or hide a field. User can mark a field as

required or not (Figure 14). User can see the Preview on Android in the preview panel. When user is

satisfied with the form he can save the form and the form will be created.

Figure 14: Using the Create new form section to create new survey form

The system show popup dialog to indicate that a form has been created successfully (Figure. 15).


12/18

Figure 15: Confirmation of new form creation

d)Form managementThe form management section allows the admin user to Edit / Delete and share form among mobile

users. This section can be accessed using Form management link from the top menu (Figure 16). If the

user clicks Edit the edit form is shown. The edit form is similar to the Create new form. If the user

clicks Delete link, the system asks confirmation from the user.

Figure 16: The form management section


13/18

When the user clicks the Share form link, the system shows the form to share with existing mobile

users. The right side shows the list of current users that has access to this form. There is Revoke access

link on the right side of each user. Clicking this link will revoke access of this form from that user. In the

User name textbox admin can type the name of a user to provide access. The system will show an

autocomplete list to allow easy selection of a user that do not have access to the form (Figure 17).

Admin selects a user from that autocomplete list and clicks on the Allow acccess button. The mobile

user will be added to the access list of that form. When that user now log in from an android device he

will see this form in the list of his forms.

Figure 17: Sharing / Revoking the survey form from mobile users

2.Android applicationA login page will be shown to the user after lunching the application on an android phone (Figure. 18 -

left). An email id is required as login id. This login was created in the user management section of the

web frontend. First default password is set by administrator and informed via email to user. If the user

provides incorrect login id or password a pop up will be shown to the user informing an invalid login try

(Figure. 18- right).


14/18

Figure 18: User login and validation on android

Application will communicate will the server for user authentication. After successful authentication

menu screen will be shown (Figure. 19).

Figure 19: Main menu of the android application

After clicking on Show forms icon a user will be directed to show forms page. Application will send a

REST message to server with user authentication token and server runs a query to database for form list

and send the list to application. A list view of forms will be showed up where first bold line shows name

of the forms and italic lines shows description of the forms. Those forms were created and access

granted to current user in form management section of the web-frontend.


15/18

Figure 20: List of forms accessible to the user

After selecting a form from the list, application will again send a REST message to server and server will

reply with list consists of field name, type. Application will build a dynamic form based on that field list

(Figure 21).

Figure 21: A sample dynamic form

After touching submit button, application will collect data from all fields, perform required validation

and if all the validation is passed then submit those data to database server. A confirmation message

will be shown to the user (Figure 22). The system will return to the main menu.


16/18

Figure 22: Confirmation of data submission to server

The android application allows user to change the password. Change password icon will navigate to

change password window (Figure 23).

Figure 23: Change password form on android


17/18

3. REST serviceThe rest service acts as bridge between the database and android application. Our implementation of

the service is stateless. On first authentication, the service sends an authentication token to the client.

For using other services a client needs to send that authentication token with their request. The

methods created by our service are described below.

Authentication

URL: http://SYSTEM_URL/index.php/webservice/login

Parameter: username, password

Return: Authentication token on success, on failure returns 401 HTTP status

Getting form list

URL: http://SYSTEM_URL/index.php/webservice/getformlist

Parameter: authentication token

Return: List of forms that user is allowed to access

Getting form field

URL: http://SYSTEM_URL/index.php/webservice/getformfields

Parameter: authentication token, form_id

Return: Details of the form fields

Saving a form data

URL: http://SYSTEM_URL/index.php/webservice/saveform

Parameter: authentication token, form_id, data of the form

Return: HTTP 200 on success, HTTP 500 on error

Changing password

URL: http://SYSTEM_URL/index.php/webservice/changepassword

Parameter: authentication token, current password, new password

Return: HTTP 200 on success, HTTP 500 on error

Possible Future EnhancementThe system functionalities can be enhanced further with some possible enhancements. Firstly, we can

add one more field type in form creation to handle uploading file from android device. This will allow

the users to upload images, audios and videos. Moreover, we can add video, audio or image data type

fields directly to form. When user activates one of this type of form the application will access

corresponding device to collect that type of data. For example, for image fields the application will

access camera of the device to take capture an image.
http://system_url/index.php/webservice/loginhttp://system_url/index.php/webservice/loginhttp://system_url/index.php/webservice/getformlisthttp://system_url/index.php/webservice/getformlisthttp://system_url/index.php/webservice/getformfieldshttp://system_url/index.php/webservice/getformfieldshttp://system_url/index.php/webservice/saveformhttp://system_url/index.php/webservice/saveformhttp://system_url/index.php/webservice/changepasswordhttp://system_url/index.php/webservice/changepasswordhttp://system_url/index.php/webservice/changepasswordhttp://system_url/index.php/webservice/saveformhttp://system_url/index.php/webservice/getformfieldshttp://system_url/index.php/webservice/getformlisthttp://system_url/index.php/webservice/login


18/18

On the web frontend, we can provide a data management section. From this section, user will be able to

browse and download the data in various formats. The data management section can also show some

simple descriptive statistics of uploaded data.

Conclusion

Portable devices are very popular now. Using these portable devices in data collection in survey is a very

convenient option. But problem of using portable devices is presentation of the data collection form of

different forms in different devices. Our developed system is a flexible and convenient solution for these

scenarios. Designing the for is very flexible and user friendly. User can view preview of the form on

android device from the web-frontend. This enhances the usability of the system greatly. The attributes

of the forms are downloaded from server and build dynamically in portable devices. User interfaces are

enhanced than consisting approach.

cs 607 final report

Documents