codescore - penn engineeringcse400/cse400_2013_2014/reports/03_report.pdf · codescore dept. of cis...

CodeScore

Dept. of CIS - Senior Design 2013-2014∗

Allison [email protected]

Univ. of PennPhiladelphia, PA

Spencer [email protected]


Tanvir [email protected]


Will [email protected]


ABSTRACTThe technology industry lacks automated tools for evaluatingsoftware quality. These tools would be helpful for individu-als desiring to improve their abilities, recruiters searchingfor top programmers, and educators needing to quickly as-sess student performance. This work describes CodeScore,an application that assesses internal software quality by de-tecting code smells. Code smells are easily recognized de-sign weaknesses that may indicate more significant problemswithin the system. The program recognizes code smells usinga set of specific rules that make use of the abstract syntaxof the source code. It then uses the results to compute aCodeScore: a single value that reflects the maintainabilityand understandability of the piece of software based on thepresence of code smells.

1. INTRODUCTIONSoftware failures result in annoyed users at best, and they

can cause catastrophic system failures at worst. Six peo-ple received massive radiation overdoses from a radiationtherapy device in one of the canonical examples of a fatalsoftware error [14]. Failures are consequences of poor soft-ware quality. Software quality is defined as “conformance toexplicitly stated functional and performance requirements,explicitly documented development standards, and implicitcharacteristics that are expected of all professionally devel-oped software”[20]. In addition to causing failures, disregardfor software quality is expensive and inefficient, requiringmore dollars and man-hours to maintain, test, and add fea-tures to a project than should be necessary. An increasingappreciation of well-designed software has been manifestedin the international standard for software quality, ISO/IEC25010 [12]. The standard affirms the importance of specify-ing and measuring characteristics of quality, and it definesquality models intended to aid in accomplishing these goals.

Software quality is commonly divided into internal qualityand external quality. External quality describes how well thesoftware meets performance expectations, and encompassescharacteristics such as correctness, accuracy, integrity, ro-bustness, reliability, usability, adaptability, and efficiency.Internal quality describes how well the software was designedand implemented, and is concerned with maintainability,flexibility, portability, reusability, readability, understand-ability, and testability [16]. Internal and external qualityare closely related, and deficiencies in internal quality leadto deficiencies in external quality. For example, code that

∗Advisor: Chris Murphy ([email protected]).

is difficult to understand will also be difficult to adapt tonew requirements, and code that cannot be easily testedwill likely be incorrect or unreliable.

One way to diagnose internal quality issues is by detect-ing code smells. Code smells are easily recognized designweaknesses on the surface of the code that may indicatedeeper, more significant problems within the system. In-dividual code smells are directly related to problems withspecific aspects of internal quality. Examples of code smellsinclude:

• Long parameter lists. Large numbers of inputs to amethod make the code difficult to read, and they makeit easy for the programmer to introduce a bug by pass-ing in parameters in the wrong order.

• Deeply nested conditional logic. A sequence of nestedconditionals is difficult to understand because the nor-mal path of execution is unclear.

• Message chaining (e.g. A.getB().getC().getD()). Mes-sage chains force class A to depend unnecessarily onclasses B and C in order to get information from classD. This could introduce complications when any of thefour classes need to be modified.

• Absence of Javadocs. Lack of documentation detractsfrom understandability and maintainability.

• Line length. Lines longer than the standard 80 charac-ters make it difficult to read code on smaller monitors.

• Method length. Complex,“do everything”methods arearduous to read and to modify.

• Hard coding. Values that are hard coded in multipleplaces must be changed individually by the program-mer, which often introduces bugs.

• Comment to code ratio. This is another form of docu-mentation that should be present in quality code.

CodeScore tests for the presence of all of the code smellson the preceding list in order to evaluate internal quality.The software focuses on code smells that relate to two ele-ments of internal software quality: maintainability and un-derstandability. These elements are underemphasized inmost classroom settings. Students work quickly and hap-hazardly to meet deadlines and never revisit their programsafter turning them in, so there is no need to write main-tainable or understandable code. However, understandabil-ity and maintainability are critical outside of the classroom,

1

where teams of developers need to understand code theydid not write, and to maintain it for years as the productevolves. CodeScore helps students assess and improve thesenecessary software engineering skills by providing objectivefeedback to make them aware of their weaknesses. Computerscience teachers can use the the application when grading as-signments to assess quality without spending hours readingthrough each student’s source code. It also allows recruitersto identify potential employees who will contribute to theircompany’s products with a high regard for quality at a muchlower cost than most current recruiting practices. Overall,this tool has the potential to help the technology industrytrain and recruit a strong developer workforce who will writeprograms that are internally sound.

At the core of the program are specific rules for definingcode smells. The rules are used by custom code smell de-tection algorithms. When a user uploads their code to beanalyzed via the web interface, each detector runs on eachfile in the sample. The results of the detections are fed toa scoring module to compute an internal quality score, theCodeScore, for the piece of software. The CodeScore and de-tailed reports with suggestions for improvement are returnedto the user. The detectors used to calculate the CodeScoreare accurate, and the CodeScore itself correlates well withhuman assessment.

2. RELATED WORKBefore describing our methods, we provide background

information on the different methods of analyzing code, thestandards for measuring code quality, and various servicesand applications that already strive to provide quantitativeanalyses of code quality.

2.1 Code Analysis MethodologiesThe majority of existing applications for measuring code

quality rely on well-established code analysis techniques.These techniques involve breaking code down into specificunits for processing. Such techniques include parsing thesource code into control structures [15], tokens [11], assem-bly instructions [19], or objects [5]. Once the source codehas been parsed into some unit (this process is called staticanalysis), the attributes of the code, such as reliability, secu-rity, efficiency, size, and maintainability can be measured bycalculating various metrics using the parsed results. The ac-tual approach to measuring these attributes originated in [3]and later became part of the ISO/IEC 25000 series of stan-dards relating to the quality and evaluation of software [12].These techniques and standards form the foundation for to-day’s code quality measurement applications, and they arethe basis for our approach as well.

2.2 OhlohLaunched in January of 2006, Ohloh [1] is web service and

online community owned by Black Duck Software which pro-vides basic metrics for over 650,000 open source projects con-taining over 24 billion lines of code located in over 620,000source code repositories and managed by more than 3 mil-lion contributors. These projects are retrieved from sourcecontrol repositories and analyzed. Metrics such as lines ofcode, amount of source code comments, and estimated soft-ware cost (based on the COCOMO model [4]) are provided,along with commit statistics, project activity, and program-ming language detection. This data is graphically displayed

when one views a project’s information on the site. Ohlohalso provides global statistics across all projects for differentprogramming languages and contributor statistics for differ-ent authors of open source code.

Ohloh primarily focuses on tracking project/contributoractivity for large open-source projects. CodeScore focusesmore on providing code quality-oriented metrics.

2.3 Common Weakness EnumerationThe Common Weakness Enumeration (CWE) [17] is a

community-developed list of software weaknesses hosted bythe MITRE Corporation. The CWE was developed with theintention of providing:

• a common standard of identifying, mitigating, and pre-venting software weaknesses.

• a common source of measuring weaknesses for softwaresecurity tools.

• a common language for describing the various types ofsoftware weaknesses that exist in computer architec-ture, software design, and source code.

CWE supports the discovery of common types of softwaresecurity flaws such as buffer overflows, handler errors, path-name traversals, and resource management errors (amongstothers) within code.

The Preliminary List of Vulnerability Examples for Re-searchers, or PLOVER [6], was an early form of CWE.PLOVER was the first attempt to take real-world examplesof software vulnerabilities and abstract them into commonclasses of more general vulnerabilities that can arise duringthe software development process. The goal of PLOVER wasto make this information available to developers, researchers,and analysts so that they may use it with the goal of im-proving code security. CWE expands upon PLOVER byestablishing community-developed definitions and descrip-tions of these common weaknesses.

CWE primarily strives to provide standards relating tothe weaknesses of code in terms of security. This may be avaluable resource as we develop additional features for Code-Score, as many security weaknesses are indicative of, or arethemselves, code smells, but our project has a wider scope.CWE also incorporates community feedback in developingdefinitions of weaknesses, which is a strategy we also intendto use in verifying and fine-tuning our models.

2.4 HISTHistorical Information for Smell deTection, or HIST [18],

is an approach developed to detect five specific code smellsusing change history from version control systems. The de-velopers of HIST point out that not all code smells are pos-sible to detect using just source code because some are bydefinition characterized by how the code changes during theproject’s development. One example is parallel inheritancehierarchies, in which “every tme you make a subclass of oneclass, you have to make a subclass of another” [9]. Thoughrevision histories often display changes at a file-level granu-larity, they use a tool called the Change History Extractorto parse changes at a method- and class-level granularity,and then they identify code smells from the parsed logs us-ing specific rules. CodeScore will soon make use of a sim-ilar strategy for incorporating version control information

2

into several metrics, but CodeScore’s analysis includes manymore metrics than just those that can be detected using re-vision logs.

2.5 JOCSJudgment of Code Style (JOCS) was a CIS 400/401 project

in the 2012-2013 academic year [7]. Their goal was to createa tool for an automated analysis of code style. Such a toolwould be useful in assigning the style grade often associatedwith assignments in introductory programming classes, suchas CIS 110 or 120 at the University of Pennsylvania. Fea-tures of interest included line length, modularity, and con-sistency. They used machine learning to compute a singlescore from the features. CodeScore will make use of similartechniques, but the two projects will be using mostly non-overlapping feature sets and different strategies for identify-ing them. Our focus is not on style but on code smells asthey relate to internal quality, which ultimately affect thecorrectness, efficiency, and overall external quality of theprogram.

2.6 Other Static Code Analysis ToolsA wide variety of other static code analysis tools also ex-

ist, including CodeSonar [10], KlocWork Insight [13], Find-Bugs [21], and Yasca [22]. These applications focus mostlyon external quality metrics such as security, reliability, cor-rect functionality, and efficiency. Most can be used with avariety of programming languages and offer reports or visu-alizations of the results.

CodeScore offers a novel solution amongst all of these ex-isting tools. While most of the current applications are prin-cipally concerned with external quality, our tool focuses oninternal quality. Internal quality has a strong correlationwith external quality, so writing code with an emphasis oninternal quality allows developers to spend less time test-ing and fixing bugs and more time on new products andfeatures. Additionally, many existing tools target industrydevelopers, but one of our primary target audiences is stu-dents. Students need to develop an appreciation for internalquality before entering the workforce or academia, but theseskills are often underemphasized at the undergraduate level.Finally, our solution attempts to provide more functional-ity than the majority of similar open-source applications.CodeScore aims to provide quality of functionality similarto commercial-grade applications in a form that is accessi-ble to a wider audience.

3. SYSTEM MODELCodeScore focuses on evaluating internal quality, and as

such primarily uses static code analysis techniques. Thisallows for the assessment of source code without bias fromdifferent system architectures or environments. Code smellsprovide a powerful but simple indicator of internal qualitybecause they are specific and detectable, and they have aclear relationship with internal quality. The program cur-rently focuses on understandability and maintainability.

Understandability is a measure of how easy a code sam-ple is for a human to interpret. It can be estimated in partby the length of message chains (a code smell in which onemethod invokes another, which invokes another, and so on ina long one-line sequence of function calls), length of parame-ter lists, by determining what fraction of variable names aredictionary words versus strings of letters and numbers, and

by analyzing the class structure of a program.Maintainability is a measure of how easy a code sample

will be to update and change. It can be estimated in partby detecting and recognizing coupling between classes, du-plicated code, classes and methods that are too large, andhard coding.

CodeScore implements the workflow illustrated in Fig-ure 1. The key components of the system are the maindriver and thread pool, the detectors, and the scoring mod-ule. CodeScore includes detectors for the following codesmells:

• Long parameter lists. This smell is detected by identi-fying method declarations and counting the number ofparameters in the signature. The maximum allowableparameter list length can be configured by the user.

• Deeply nested conditional logic. This smell is detectedby identifying conditional statements and determin-ing if they contain additional conditional statementswithin the blocks of code that are entered if a partic-ular condition is met. The level of nesting consideredacceptable can be configured by the user.

• Message chaining. This is detected by identifying allmethod invocations and determining if the calling ob-ject of the method is itself a method invocation. Notethat it is not sufficient simply to detect methods thatappear in the same line, because then an expressionsuch as:

System.out.println("x is: " + x.get() + " and y

is: " + y.get());

would be incorrectly labeled a message chain.

• Absence of Javadocs. This is detected by identifyingJavadocs and comparing the ratio of the number ofmethods and classes declared in the software to thenumber of Javadocs.

• Line length. This is detected by counting the numberof characters in each line.

• Method length. This is detected by counting the num-ber of lines of code (not including white space or com-ments) in a method declaration.

• Hard coding. This is detected by identifying literals(for example, string literals or integer literals). Thedetector then determines if the literals are being usedin an acceptable context, such as a variable assign-ment, annotation, or in a statement such as return

true;, or if they are hard-coded values.

• Comment to code ratio. This is detected by differen-tiating between lines of comments, lines of code, andblank lines, then counting the instances of each.

The detectors are the most complicated component of thesystem and present the greatest technical challenges. Eachcode smell requires a unique detection algorithm. The detec-tors search for all possible patterns indicative of a particularcode smell using the syntactical structure of the program.They track the number and location of all violations, com-piling them into comprehensive reports in JavaScript Ob-ject Notation (JSON) for interpretation by other modules

3

Figure 1: System architecture

in the system. In detectors which require thresholds, suchas the maximum depth allowed for nested conditionals, theprogram’s default parameters can be overriden by inputtingcustom parameters.

Users access CodeScore using a web application that pro-vides a platform for developers to assess and showcase theirabilities and for recruiters to identify promising candidates.Developers can create an account and upload their projectsfor analysis. Processing occurs with the detectors runningin parallel, and once completed, a report provides the over-all CodeScore for the project, which is a grade out of 100points that is analogous to a grade on an essay. It alsoshows the subscores of each individual file in the project,graphs displaying the frequency of violations for each met-ric, and specific suggestions for improvement that includeline numbers and snippets from the user’s code. This reportcan be shared as a quick and objective evaluation of one’sprogramming ability, used as a tool for self-improvement,or incorporated into teachers’ grading rubrics. The appli-cation also charts CodeScores for revisions of projects thatare uploaded for re-assessment so that the user can trackimprovement over time.

The interface for recruiters focuses on searching for andfinding information about talented programmers. Whenviewing a candidate, recruiters can see the candidate’s av-erage CodeScore and ranking as compared to other users.They can also view the candidate’s education history, previ-ous employers, answers to sample behavioral questions, anda project showcase.

4. SYSTEM IMPLEMENTATIONAll processing for CodeScore happens on an Amazon EC2

cloud processing machine. EC2 is a service that providesresizable computing capacity in the cloud. Users uploadJava code in the form of a single .java file or a zip file toan Amazon S3 bucket through our website. S3 stands forSimple Storage Service and is a component of the Ama-zon AWS suite of web services. The software ensures thatthe uploaded project is not malicious using basic validationtechniques that search for traits that exist in malware (e.g.known checksums or invalid filetypes). After passing valida-tion, the upload is indexed on Amazon RDS. Then the serverlaunches a process that unzips the archive if necessary and

scans for all appropriate files (Java files or preferences files).It saves some of the metadata to include in the report. Thepreferences file is encoded in JSON and includes informationthat is specific to each detector, such as thresholds for cer-tain code smells. It also includes optional weights if the userwould like to change how much a given detector influencesthe overall CodeScore. A sample preferences file can be seenin Figure 2. If no preferences file is included, the programonly runs using the default values. If a preferences file isincluded, the program runs once with the default settingsto compute a standard CodeScore used to compare the userwith other registered candidates and once with the customsettings.

Figure 2: User preferences file encoded in JSON

To perform the code smell detection, the controller classgenerates n worker threads, where n is the number of de-tection algorithms to be performed on the software sample.Currently, n = 8. Each of the workers performs a specificdetection task on each uploaded file and then sends resultsback to the scoring module. The type of result reported tothe scoring module depends on the detector, but in mostcases the detector reports a count of the number of viola-tions for that metric in each file normalized to a percentagebetween zero and one hundred. The percentage representshow acceptable the file is with regard to that particular codesmell, taking into account the length of the file and the num-ber of violations.

The scoring module combines all of the normalized detec-

4

tor outputs into a weighted average representing the sub-score for a particular file. In the default configuration, somedetectors are weighted more heavily than others. Theseweights are based on experiments and user surveys. Thesubscores for each file are combined into the final CodeScore.Each file is weighted by importance, considering factors suchas the number of lines it contributed to the project andthe number of references to methods or objects from thatfile throughout the project. The end result of this processis a single score that gives a user an at-a-glance measureof the internal quality of a project. The web applicationalso provides more detailed feedback using HighCharts [2], aJavaScript framework used for making graphs online. Userscan navigate to view reports for a specific file or a generaloverview. Example reports are shown in Appendix A.

The program controller, scoring module, and all detec-tors are implemented in Java. Multithreading and concur-rency of the detectors is accomplished with Java’s thread-ing libraries and a custom worker thread pool. The detec-tion algorithms use the Eclipse Java Abstract Syntax Tree(AST) API to parse the Java code as a first step in find-ing code smells. Traversing the AST allows the system toaccess and manipulate Java code at the syntactic level, effi-ciently searching for specific elements such as if statementsor method invocations. The AST API also provides infor-mation about the location of a particular expression withinthe tree, both in terms of line numbers and in terms of thesurrounding code. Though CodeScore only supports Javaat this time, detecting code smells at the AST level makesthe program easy to adapt to additional programming lan-guages in the future. Most object-oriented code smells havethe same properties from the perspective of the AST, whichabstracts away the specifics of individual programming lan-guages. A sample abstract syntax tree and the Java code itrepresents can be seen in Figure 3.

The web application (online at http://techruit.me) is im-plementented in PHP and makes use of search engine op-timization techniques. Custom PHP scripts generate pagecontent, HTML comments, and optimize formatting for eachview on the website. The website is also integrated with so-cial media using Facebook’s Open Graph tags. This allowsthe page to be liked and shared on Facebook and providesinformation to other users when the page receives attention.Several screenshots of the web application can be seen inAppendix A.

Efficiency is a challenge that is critical to the usefulnessof the CodeScore platform. CodeScore is designed to runon large and small software projects, which means that allparsing, processing, and reporting must be carefully engi-neered to produce results in a timely manner. CodeScoreemploys parallel processing techniques so that the runtimescales well with the number of metrics and the size of theinput.

One drawback of the overall CodeScore system is that itis largely centralized. If the server containing the controllercode is compromised at any time during the computation,all progress in the analysis will be lost. A possible futuredevelopment is to decentralize the process so that loss of aserver does not result in complete loss of progress. Ideally,this could be implemented using MapReduce [8].

5. RESULTSCodeScore has been evaluated on four main criteria:

• How accurately code smells are detected.

• How the CodeScore compares to human grading.

• How quickly the analysis is completed.

• The success of the website.

For the first criterion, we designed user surveys in whichusers identified the same code smells that are detected byCodeScore. We used code samples from existing projectsand some code samples that were written to intentionallycontain“bad code.”We had humans detect code smells in thesamples and then compared their results to those of Code-Score. Figure 4 displays the percentages of code smells cor-rectly identified by the CodeScore detectors as compared tohuman identification in the user studies.

CodeScore detected 100% of violations using the commentto code ratio, line length, method length, and number ofparameters detectors. The absence of Javadocs and messagechaining detectors were both over 80%. The detectors fornested conditionals and hard coding are the most technicallychallenging, which is reflected in their accuracy performance(72% and 68%, respectively). False positives were not anissue with any detectors except for hard coding.

To compare CodeScore’s final grade to the grade a humanwould assign, we also asked study participants to assign ascore from 0 to 100 to each piece of code they analyzed. Weused feedback from the first session to fine-tune the weightsof the detectors used in calculating the CodeScore. We foundthat the CodeScore assigned by the software was within 14%of the average score assigned by the study participants forfive of the six code samples assessed in the second and thirdsessions.

To analyze runtime, we pulled additional open source projectswith up to 10,000 lines of code to use as inputs to the sys-tem. Projects of this size were not feasible to ask user studypartipants to analyze in the detector and scoring accuracystudies, but they were informative for collecting runtime in-formation. The speed of execution varied roughly linearlywith the number of lines of code in preliminary tests. Thebottleneck in runtime seems to be the Java Abstract SyntaxTree API. However, the benefits of using the AST API interms of simplicity and modularity outweigh the slight slow-ness in processing. See Figure 5 for a graph of the runtimeanalysis results.

The success of the web application can be quantified usingsearch results and various analytics provided by Google. Atthe time of publishing of this report, the website (http://techruit.me)ranks in the top two results when searching Google for“techruit”.The website is in the top 8 results on Google and top 4 onBing when the search term is “codescore recruiting.” Thelanding page received over 2,000 pageviews during the timeof development, which is highly promising considering thatthere was no formal marketing. The bounce rate, or thenumber of users who are interested enough to view otherpages after the landing page, is about 50%. About 10.5%of users continue to look at the developers portal and 8.3%look at the recruiters portal.

6. FUTURE WORKMoving forward, there are a number of interesting ques-

tions to be investigated with CodeScore. An obvious im-provement is to add support for other programming lan-guages in addition to Java. Because CodeScore uses the Java

5

Figure 3: Sample Java abstract syntax tree

Figure 4: Detector Accuracy

Abstract Syntax Tree API to parse the source code, mostof the language-specfic considerations have been abstractedaway. The detectors look for patterns that are considereddetrimental to internal quality in any object-oriented lan-guage. Therefore, adding new programming languages thathave existing APIs for their syntax trees would simply be amatter of plugging in the new API. A more difficult prob-lem is to include other programming paradigms, such asfunctional programming.

Even within the realm of Java and object-oriented pro-gramming, there are enhancements that could help Code-Score perform a more thorough assessment of internal qual-ity. Certain code smells, such as shotgun surgery, cannotbe detected using just the source code. Shotgun surgerydescribes the situation in which making one change in be-havior requires modifying the code in several places. Forexample, if logging statements are implemented separatelyin each function in a class, then adding line numbers to thelogs will require considerable time and effort. A better solu-tion would be to write a log wrapper for all of the functions,so that any changes only need to be made once. Using re-vision history, detecting code smells like shotgun surgery is

possible. Analyzing revision history requires a custom parserthat can extract changes at method-level granularity in orderto determine which methods are changing together. Usingrevision history might also make it possible to analyze col-laborative projects and score each contributor based only onthe code that he or she wrote. After creating such a parserand adding the corresponding analyses to the program, itwould be simple to link the CodeScore web application toGithub and pull entire repositories, including git logs, forprocessing.

7. ETHICSCodeScore raises few ethical questions. The most sig-

nificant ethical issue with CodeScore is the possibility ofmisleading potential employers. If the software were to beadopted by companies to aid in their hiring procedures, andif flaws in the system made it appear that a particular pro-grammer was less qualified than he or she really was, thatperson might not be hired. CodeScore would be at leastpartially responsible for keeping that individual from find-ing work. It is also possible that the opposite could occur

6

Figure 5: Preliminary Runtime Analysis of Code-Score Module

- CodeScore might lead to someone being hired despite be-ing underqualified. If the person broke something critical,such as the Apple developer who introduced the “goto fail”bug, CodeScore might also be partially responsible. Otherthan these scenarios or a similar situations, however, thesoftware presents no major ethical issues. There is almostno physical, financial, or emotional risk to users.

8. CONCLUSIONSCodeScore is a program that performs an automated as-

sessment of the internal quality of a piece of software. Thecode is graded based on the presence of specific code smellsthat detract from the maintainability and understandabil-ity of software. Code smells are detected using custom al-gorithms, and the results of these algorithms are used tocompute a single numerical score (the “CodeScore”) for thesoftware. The score provides a quick, easy-to-understandmeasure of the internal quality of the project. The pro-gram also provides detailed reports and suggestions for im-provement. Of the eight detectors, six achieved over 80%accuracy, and the CodeScore itself was shown to be highlycorrelated with grades assigned by participants of user stud-ies. CodeScore is useful for individuals who want to improvetheir abilities, recruiters seeking an inexpensive way to as-sess potential hires, and instructors who want to reinforcethe importance of writing quality code. CodeScore is partof an online platform that allows developers to assess andoptionally showcase their code and helps connect recruiterswith talented programmers.

9. REFERENCES[1] J Allen, S Collison, and R Luckey. Ohloh web site.

https://www.ohloh.net/, 2009.

[2] HighSoft AS. Highcharts.http://www.highcharts.com, 2013.

[3] Barry W Boehm, John R Brown, and Myron Lipow.Quantitative evaluation of software quality. InProceedings of the 2nd international conference onSoftware engineering, pages 592–605. IEEE ComputerSociety Press, 1976.

[4] Barry W Boehm, Ray Madachy, Bert Steece, et al.Software Cost Estimation with Cocomo II with Cdrom.Prentice Hall PTR, 2000.

[5] Shyam R Chidamber and Chris F Kemerer. A metricssuite for object oriented design. Software Engineering,IEEE Transactions on, 20(6):476–493, 1994.

[6] Steve Christey. Plover: Preliminary list ofvulnerability examples for researchers. In NISTWorkshop Defining the State of the Art of SoftwareSecurity Tools, 2005.

[7] Nathan Close, Amalia Hawkins, SworupiniSureshkumar, et al. Judgement of code style, 2013.

[8] Jeffrey Dean and Sanjay Ghemawat. Mapreduce:simplified data processing on large clusters.Communications of the ACM, 51(1):107–113, 2008.

[9] M Fowler. Refactoring: Improving the design ofexisting code. Addison-Wesley, 1999.

[10] GrammaTech. Codesonar.http://www.grammatech.com/codesonar, 2013.

[11] Maurice H Halstead. Elements of Software Science.Elsevier Science Inc., 1977.

[12] ISO ISO. Iec 25010: 2011: Systems and softwareengineering–systems and software qualityrequirements and evaluation (square)–system andsoftware quality models. International Organizationfor Standardization, 2011.

[13] Klocwork. Klocwork insight.http://www.klocwork.com/products/insight, 2013.

[14] Nancy Leveson. Safeware: System Safety andComputers. Addison-Wesley, 1995.

[15] Thomas J McCabe. A complexity measure. SoftwareEngineering, IEEE Transactions on, (4):308–320,1976.

[16] Steve McConnell. Code Complete. Microsoft Press,1993.

[17] CWE MITRE. Common weakness enumeration.https://cwe.mitre.org/, 2006.

[18] Fabio Palomba, Gabriele Bavota, Massimiliano DiPenta, Rocco Oliveto, Andrea De Lucia, and DenysPoshyvanyk. Detecting bad smells in source code usingchange history information. http://www.cs.wm.edu/~denys/pubs/ASE%2713-HIST-Camera.pdf.

[19] Robert E Park. Software size measurement: Aframework for counting source statements. Technicalreport, DTIC Document, 1992.

[20] R Pressman. Software Engineering: A Practicioner’sApproach. McGraw-Hill, 4 edition, 1997.

[21] Bill Pugh and Andrey Loskutov. Findbugs.http://findbugs.sourceforge.net, 2013.

[22] Michael Scovetta. Yasca.http://www.scovetta.com/yasca.html, 2007.

7

codescore - penn engineeringcse400/cse400_2013_2014/reports/03_report.pdf · codescore dept. of cis...

Documents