IEEE Fellow
prof (full), cs, ncstate, usa
timm@ieee.org
SE, AI, data mining, optimization, explanation, privacy, fairness and prog languages.
Computer Science, NC State
box 8206, 890 Oval Dr
Raleigh, NC 27606
304-376-2859
I'm genuinely alarmed at SE for AI practices-- a headlong rush to "ship it" without thinking about what "it" is and who it might hurt. So I seek a new kind of Ph.D. student-- someone who is more reflective, more radical, and wants to use their Ph.D. to explore issues related to fairness, trust, and respect for anyone using software. Sure, that person will explore state-of-the-art data mining methods (and other algorithms). But that Ph.D. will position them to become a leader in the ethical future of software.
Interested? Then read a little more about that work. And maybe drop me a line? Or apply for grad study at NC State?
About me
Tim Menzies (IEEE Fellow, Ph.D., UNSW, 1995) is a full Professor in CS at North Carolina State University where he teaches software engineering, automated software engineering, and foundations of software science.
He is the directory of the RAISE lab (real world AI for SE) and
the author of over 280 publications (refereed). In his career, he has supervised 20 Ph.D. students,
and has been a lead researcher on projects for NSF, NIJ, DoD, NASA, USDA (total funding of $13+ million) as well as joint research work with private companies.
Prof. Menzies is the editor-in-chief of the Automated Software Engineering journal and
associate editor of TSE (IEEE Transactions on Software Engineering) and other leading SE journals.
For more, see his web site https://timm.github.io or the text of his successful IEEE Fellow nomination
Interested in Graduate Study?
It is no longer enough to do "it" faster. Now we have to do "it" fairer.
So I seek talented Ph.D. graduate students for AI + SE, specifically in the area of explanation and fairness.
Does that interest you? If yes review these notes before applying.
Information:
Interested in Industrial Research Opportunities?
I work with industry. A lot. Ask me how to innovate. On time. On budget.
Read my industrial case studies:
Current Projects
Simpler solutions to seemingly hard problems.
William of Occam: It is vain to do with more what can be done with less.
Edsger W. Dijkstra: Simplicity is prerequisite for reliability.
Dieter Rams: Less is more.
timm: less, plz
The physicist John Archibald Wheeler advised that "in any field, find the strangest thing and explore it." Accordingly, I explore the strangest thing about software—that it ever works at all.
Modern software is so complex that it should never work. So why aren't our incomplete test methods missing all too many critical errors? My research keep showing is that most software systems have "keys"; i.e. a few variables that control the rest. Outside SE, I have seen keys in hospital nutrition analysis and avionics control systems. Within SE, I’ve found keys while doing analytics for:
Outside SE, there is a long history of researchers reporting effects that are similar to the keys effect, dating all the way back to 1901 (see the first paper on principal components analysis). Since then, what I would call "key” have been reported many times in the AI literature. For example, data mining researchers report that most rows and columns in a data set can be discarded without losing the signals in those data. Also, when a randomized search is repeated many times, there is often a small number of variable settings shared by all solutions (which I would call "keys"). Furthermore, when researchers set those variables initially, then formerly exponential runtimes collapsed to low-order polynomial time.
Regardless of whether we are talking about AI or SE, when systems consist of just a few keys, then it is very simple to audit, understand, and modify them. For example, effective defect prediction or effort estimation is possible using less than half a dozen variables; Additionally, configuring complex systems is often remarkably simple and effective and can be achieved using very small decision trees that require only a few variables. Furthermore, by generating tests only for the main branches in the code, even applications that process large cloud databases can be tested via just a few dozen inputs. Hence, one reason to explore keys is to create ethical systems that enable community reflection and modification for AI software.
In any case, for decades, I've been applying the following heuristic:
IEEE Fellow Nomination (2018)
(With apologies for excess hyperbole...)
Internationally known for revolutionary advances exploring the synergy between artificial intelligence (AI) and software engineering (SE), Dr. Menzies has authored four books and over 260 refereed publications. His publications, with over 9000 citations, have appeared in leading journals and proceedings of prestigious conferences. He has supervised seven students earning PhDs and 23 MS thesis students. Dr. Menzies' distinctive contributions have had enormous impact for SE researchers and practitioners in software quality prediction and software optimization.
Because software plays a critical role in industry, government, and society itself, improving software quality is critical. In landmark papers in 2006 and 2007, Dr. Menzies was an early pioneer in applying data mining and AI to software quality predictors, introducing a method which identified software modules likely to contain defects. This method had a 71 percent mean probability of defect detection, significantly higher than the code inspections commonly used in software practice.
In his software quality prediction research, Dr. Menzies identified a serious problem: often, the analysis in SE papers is not reproducible because data underlying the analysis is unavailable. To address this problem, Dr. Menzies developed PROMISE, a public data repository of software data, in 2005 publishing a paper introducing PROMISE and co-founding the PROMISE workshop, so successful it became a conference in 2008. Today, the PROMISE repository contains hundreds of data sets used in thousands of papers by researchers around the world.
Dr. Menzies is a pioneer in applying data miners to optimize software-intensive systems. In 2002, he discovered that analyzing such systems with data miners augmented with genetic algorithms led to faster analysis and better optimizations. Even for systems with millions of configuration options, Dr. Menzies' optimizers quickly learn how to make code run quicker, make web servers handle more traffic, and compile programs faster. Dr. Menzies' optimizers have been applied at NASA for reasoning about safety-critical aerospace software.
Dr. Menzies has also applied his optimization techniques to understand the unstructured textual components of software artifacts and software research papers. His was one of the earliest successful efforts applying text mining and AI to the notes of software test engineers. By identifying anomalous reports that required a second opinion, he could increase assurance of NASA systems while reducing the overall effort required to achieve that assurance. Recently, he has designed tools that can review 10,000s of papers to learn the structure of the SE scientific community. These tools can guide researchers and practitioners to find relevant work that might otherwise be overlooked.
Dr. Menzies' contributions to SE and AI are widely recognized. For his research, in 2017, Dr. Menzies received the MSR (Mining Software Repositories) Foundational Contribution Award as "Recognition of fundamental contributions in the field of data mining software repositories which helped others advance the state of the art." International databases of scholarly achievement rank Dr. Menzies number three world- wide both in software analytics and in SE and data mining. Recently, Dr. Menzies clustered 35,000 papers from the last 25 years of top-SE journals and conferences. In the "software metrics" cluster, Dr. Menzies is the top-ranked author. In the papers from top-ranked venues, Dr. Menzies' h-index of 48 places him number 11 overall.
Dr. Menzies' contributions have had world-wide impact in software practice. In 2005, Turkish researchers found that when commercial teams restricted code inspections to 25 percent of the files identified by Dr. Menzies' methods, they detected 88 percent of the existing code defects. In 2005, his students commercialized his defect detection methods in the Predictive tool suite, subsequently purchased by companies such as Chevron, Northrop Grumman, LogLogic, Inc., and Compagnie Financière Alcatel, to find code defects. In 2017, the US Software Engineering Institute used Dr. Menzies' optimizers to guide discussions about costly updates to Department of Defense software.
NASA has benefited enormously from Dr. Menzies' research. In 2005, as science chair at a NASA facility, he received a commendation award from NASA's Chief of Mission Assurance saying: "...A great researcher in his own right, ...Tim has raised the bar on quality and level of work [expected] from our researchers." NASA used his algorithms in 2008 to find violations in Space Shuttle launch requirements; in 2010, to quickly explore the design of next-generation new Air Traffic Management concepts; and in 2017 to find better monitoring strategies for pilots flying planes in safety-critical situations. In 2016, based on Dr. Menzies' research, NASA's Jet Propulsion Laboratory created the NASA Analogy Software Cost Model as its official tool for predicting software development costs.
Tim Menzies, Jeremy Greenwald, Art Frank, "Data mining static code attributes to learn defect predictors," IEEE Transactions on Software Engineering, Vol. 33 (1), 2-13, 2007. Dr. Menzies is a pioneer in the development of predictors of software quality learned from data miners. A notable finding of this paper is that Dr. Menzies' methods (including decision trees and Bayesian learning) have a 71 percent mean probability of defect detection--a rate significantly higher than human manual inspections. The paper, with more than 920 Google Scholar citations, is one of the 100 most cited papers in software engineering. Moreover, nine of the 50 most cited papers in the IEEE Transactions on SE (2012-2017) use methods and/or data from the databases used by this paper. Methods for software defect prediction introduced in the paper have been applied commercially around the world. In this paper, Dr. Menzies was the lead researcher -he defined the problem, the technical approach, and designed and coded all of the experiments.
Martin S. Feather and Tim Menzies, "Converging on the optimal attainment of requirements," Proceedings, IEEE Joint International Conference on Requirements Engineering, 2002. Although optimization methods for numerical systems have been used widely, applying these methods is often ineffective in complex software systems where each "if" statement divides the software into regions with different properties. For software, Dr. Menzies found that applying non-numeric optimizers, e.g., simulated annealing or genetic algorithms, is effective. This paper is the first of its kind to reason about solutions to software requirement problems on the Pareto frontier. As witnessed by many papers in the last two years, this method is now widely used by researchers in the software requirements community. For this paper, Dr. Menzies led the AI-part of the research, and designed and implemented the AI algorithm used in the analysis.
Tim Menzies, Andrian Marcus, "Automated severity assessment of software defect reports," IEEE International Conference on Software Maintenance," 2008. This paper, with over 185 citations, describes one of the earliest successful efforts applying text mining methods to the notes of software test engineers. The method introduced in the paper identifies anomalous reports requiring a second opinion, thus increasing software quality assurance while reducing the overall effort required to achieve that assurance. For this work, Dr. Menzies was the lead researcher, defining the overall vision of the paper, as well as building the tools and running all of the experiments.
How to apply for PhD.
(Prospective students: please be aware that I have a new research direction. Please review those notes before applying.)
Please forgive this form letter as a reply but, as you might imagine, I get many such enquiries.
To start the application process, please go to this site.
To help you out, I list below some answers to frequently asked questions.
If the following does not answers your questions then please feel free to email me and I will do my best to reply to your queries.
Best wishes,
Tim Menzies
American universities accept new graduate students for start-of-study in mid-August and mid-January.
The application process for those dates starts months in advance. You should be planning your application at least one year before your desired start date. I have no authority to accept students but, once they are accepted, I can supervise graduate students. So please review our admissions procedures at our departmental Web site.
That said, during the admissions review process, all faculty are sent a sheet listing the candidate students and I am allowed to "nudge" a handful of names. So if you elect to enroll here then please ping me so I can watch for your application in the system.
When writing to a CS prof in the USA, here is some advice on how to attract their attention.
Do not send a form letter-- we get enough of those.
To prove that you are not sending a form letter, best to make some reference to their current research, perhaps even to one of their recent papers (and how your prior work or interests match up).
Demonstrate that you have done a little homework before sending and email about the department. E.g.
As to me supervising you, I do not take on students until they have completed one of my grad subjects. This lets us check each other out before we commit too much to each other.
As to funding, I do not guarantee that my supervised students get funding from me. Funding comes and goes depending on the whims of the funding agencies and my policy is to fund my long-term Ph.D. students before anyone else.
Thanks to the MSR'17 Award Committee
Mar 15, 2017
This evening I learned that I was the winner of the inaugural Mining Software Repositories Foundational Contribution Award.
According to the award web site, the award is a recognition of fundamental contributions in the field of mining software repositories, which helped others to advance the state of the art.
I was nominated for my work on the PROMISE repository .
I want to thank the committee for the award and I'd like to dedicate the award to the many people whose hard work made the PROMISE repo possible:
In this era of Github, GHtorrent, et al. it is hard to recall that only a decade ago, it was difficult to access project data. Nevertheless, that was the case.
Back in 2005 many people in the MSR field were analyzing large amount of (public) open source data but kept the tools and processed datasets to themselves as it was often considered a competitive advantage. In fact, within the MSR community, it was not until 2013 that they started their Data Showcase track to encourage sharing of data.
Meanwhile, back in 2005, I started the PROMISE workshop with Jelber Sayyad that "encouraged" data sharing. I put "encouraged" in quotes, because it actually was a very explicit requirement. Here are part of the 2005 call for papers for PROMISE, which put the following text in all caps:
A COPY OF THE PUBLIC DATASETS USED IN THE ACCEPTED PAPERS WILL BE POSTED ON THE PROMISE SOFTWARE ENGINEERING REPOSITORY. THEREFORE, IF APPLICABLE, THE AUTHORS SHOULD OBTAIN THE NECESSARY PERMISSION TO DONATE THE DATA BEFORE SUBMITTING THEIR PAPER."
This emphasis in shared and repeatable results was unthinkable at that time and many people predicted that PROMISE would not last long. Tee hee. We proved them wrong. The PROMISE workshop soon grew into its own stand-alone conference. Due to some cosmic quirk of scheduling, the PROMISE and MSR conferences often meet at the same time, in the same corridor, sometimes even in the next room. But both events had full schedules so we rarely made it to each other sessions. Hence, the conferences evolved differently. The following is Prem Devanbu's attempt to capture the differences (and to misquote George Box, he hopes his model is more useful than it is wrong):
Now it is true that most MSR people analyzed their data with statistics and ML, and many PROMISE people did spend time in data collection. But where the PROMISE conferences was different and unique was its analysis of the analysis of data. According to Robles et al. at MSR'10 paper, most MSR papers were not concerned with a repeated analysis of data explored by a prior paper. On the other hand, the PROMISE people routinely posted all their data on a public repository and their new papers would re-analyze old data, in an attempt to improve that analysis.
Since 2011, PROMISE stopped scheduling itself at the same time as MSR. This has lead to richer interactions between MSR and PROMISE people. Hence, as time passes, the directions of these two conferences grow less distinct. Today, MSR meets at ICSE and PROMISE meets at ESEM and both events draw international leaders in the field of software data science.
So just to be clear, the "PROMISE project" has two parts:
This award was given to me for my work on the repo. As to the conference, initially, that conference was tightly connected to the repo (to store the data from papers from the conference). Since then, the scope of the repo has extended to include data from many sources.
As to the PROMISE conference, I was its steering committee chair till 2012 when Stefan Wagner was kind enough to take on those duties. These days, the PROMISE conference is guided by its dedicated and talented steering committee Leandro Minku, Andriy Miranskyy, Massimiliano Di Penta, Burak Turhan, and Hongyu Zhang.
Here's a sample of what was achieved with PROMISE (and if anyone wants to add to this list, just email me at timm@ieee.org):
At the time of this writing, the PROMISE repository includes sharable data on defect prediction, effort estimation, model-based SE, requirements models, performance prediction, over 240,000 real-world spreadsheets (one of the largest collections ever assembled), and many other kinds of data as well.
The early focus of PROMISE was on effort estimation and defect prediction. One of the landmark results here was documenting the business case for the value of such predictors. For example, Misirli et al. (http://www.aaai.org/ojs/index.php/aimagazine/article/viewFile/2348/2216/) report studies where the guidance offered by defect predictors (a)reduced the effort required for software inspections in some Turkish software companies by 72%; (b)while, at the same time, still being able to find the 25% of the files that contain 88% of the defects.
The repo not only greatly influenced effort estimation and defect prediction, but also the field of vulnerability prediction. Several recent state of the art papers in that field use methods first widely described in the process of PROMISE data.
The first examples of successful cross-project learning in defect prediction and effort estimation came from work on PROMISE data. Working with Burak Turhan and Ayse Bener, we used PROMISE data to write a top-ten- most-cited EMSE paper (2009 to 2014) on how to share data between projects (see [On the relative value of cross-company and within-company data for defect prediction" (http://cs.gmu.edu/~offutt/classes/see/papers/turham2009.pdf)", EMSE 2009.
More recent uses of PROMISE include work in software sharing and privacy. Many researchers have reported that as data is anonymized (for privacy purposes), it becomes harder to make conclusions from that data. Using PROMISE data, my Ph.D. student, Fayola Peters, showed that it is possible to share privatized versions of data that hide details about individual projects, while at the same time allowed researchers to build effect predictors for SE quality attributes (see "[Better privacy-preserving data sharing for cross project defect prediction](https://lucas.ezzoterik.com/wp-content/uploads/2016/03/15lace2.pdf)", ICSE’15).
I need to also thank my nominees for their kind words about PROMISE. The following are quotes from those letters.
Now that PROMISE
repository
has achieved international
recognition, it is strange to report that the repo is being
decommissioned.
The ZENODO repo at the CERN Large Hadron Collider offers many services that significantly extend what PROMISE can offer:
Accordingly, we have nearly finished moving all the PROMISE data over to the ZENODO repo called SEACRAFT ("Software Engineering Artifacts Can Really Assist Future Task", http://tiny.cc/seacraft). In future, if anyone wants a long-term storage facility for data, or scripts in Github, please submit to https://zenodo.org/deposit/new?c=seacraft.
Videos
"(Re)use is Rampant"
My YouTube playlist: https://www.youtube.com/playlist?list=PLOgGLL2KRJ7x1qbFQjzDn1tCjCHAw6pGo
Other News
Papers:
