Academic Careers for Experimental Computer Scientists and Engineers (1994) / Chapter Skim
Currently Skimming:
4 Evaluating Research in ECSE
4 Evaluating Research in ECSE
Pages 60-81
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 60... ...
field itself of what constitutes scholarly work. The issue can perhaps be constructively introduced by reporting the results of a small, informal survey in which about 20 computer scientists from around the country were asked by the chair of the committee whether they thought a faculty member should "get tenure for inventing the mouse." The mouse is an example of an artifact that has realized the goals of ECSE and exemplifies an ECSE research success.
|
From page 61... ...
It also emphasizes that the research of junior faculty members either theoreticians or experimentalists-whose senior faculty are predominantly in the other area might not be fully appreciated at promotion time. In this chapter two general questions of evaluation are considered.
|
From page 62... ...
Quality is important because new ideas must be well explained, as well as convincing in their technical arguments, with comparative discussion of other approaches and often extensive quantitative evidence that substantiates the merit of an approach. A strong refereeing process plays a valuable role in identifying important and innovative ideas and in promoting those that are well justified.
|
From page 63... ...
Put differently, only a small number of respondents to the CRACSTB survey agreed that the best publication vehicle to gain university recognition was also the best vehicle for intellectual impact on the field; the remainder felt that there was a conflict between these two vehicles. Journals The leading journals in ECSE include the Association of Computing Machinery (ACM)
|
From page 64... ...
Journal reviewers typically spend considerable time understanding the argument presented in the paper and finding ways to strengthen it. Even highly favorable reviews of a paper usually have extensive comments about how to improve it.
|
From page 65... ...
The paper selection process for these conferences relies heavily on the program committee as the primary referees. Although external referees are also used, typically at least one-half of the refereeing process is handled by the program committee.
|
From page 66... ...
Papers that require substantial additional background may not easily fit within these constraints, or within the final length limitations imposed when the paper is accepted. A minor disadvantage of conference publication is the somewhat limited distribution, compared with that of journals.
|
From page 67... ...
As noted earlier, conference referees tend to prefer outright rejection rather than extensive revision because of their tight time constraints, although they often make comments intended to strengthen the paper. These two factors importance/timeliness and tight reviewing deadlines mean that papers rejected by important conferences would often have passed the quality threshold for journal publication, although perhaps with revisions required.
|
From page 68... ...
Useful artifacts are by definition useful to many people. Other potential measures of impact include how long a given artifact has been used, how many people spend substantial time modifying and enhancing it, and how many other pieces of experimental research build on it, although none of these measures are easy to obtain or even to define precisely.7 One immediate consequence of the focus on impact is that the importance and significance of a given research contribution may not 7 For example, the number of people "using" an artifact is hard to define.
|
From page 69... ...
Access to experimental computers is usually provided by researchers to other researchers via "remote log-in," which allows them to run programs on the machine over the Internet without being physically present. In addition to providing access, the researcher must provide documentation on the machine and its specific software, some amount of local disk storage, and some amount of "hand holding." · Chip designs.
|
From page 70... ...
Examples include trace data of programs, graphic data sets such as the Utah Tea Pot, benchmark programs such as the Perfect Club (from the University of Illinois) test suites, chip designs for evaluating CAD tools, and test data sets.
|
From page 71... ...
Moreover, if the artifact comes into wide use, even acknowledgments become less frequent, especially when it is not the actual program text that is used but rather its underlying algorithm or idea. THEORETICIANS' AND EXPERIMENTALISTS' VIEWS ON EXPERIMENTAL RESEARCH The committee believes that accomplishments in ECSE research should be evaluated in the context of the field's tradition as outlined above.
|
From page 72... ...
The goal is to reduce the complexity to a point where it is analytically tractable. As anyone who has tried it knows, theoretical analysis can be extremely difficult, even for apparently straightforward questions.
|
From page 73... ...
Assistant professors wrote, It is clear to me that experimental computer science is not considered to be of broad intellectual interest by the vast majority of the senior faculty in my department. [assistant professor at a well-known private university]
|
From page 74... ...
One insightful assistant professor at a private university observed: The department senior faculty and university-level tenure committee do seem to understand that experimental systems work is timeconsuming, important and needs to be evaluated differently. On the other hand, there is still a strict demand for demonstration of intellectual ability which is more easily and readily met by focusing on more theoretical journal publications.
|
From page 75... ...
It would be impossible, without being present at all of the deliberations and being party to the participants' thoughts, to second-guess an individual tenure decision and assert that someone was denied tenure simply because of prejudice against experimentalists. THE EFFECT OF EVALUATION ON PROBLEM CHOICES AND RESEARCH AREA The practicalities of evaluating ECSE research have substantial impact on how faculty members moving up the career ladder see their own careers.
|
From page 76... ...
In some ways, it is understandable that junior faculty are often frustrated by the need to subordinate their desire to pursue the most promising intellectual paths in order to respond to the immediate demands of producing documentary proof of achievement for the time-limited tenure processes. (The promotion from associate to full professor seems to be less of an issue in this regard.)
|
From page 77... ...
and a researcher maximizing his or her publication count by adopting a "least publishable unit" strategy in which the smallest possible increments of progress are published at frequent intervals (a highly undesirable mode of research that many faculty believe characterizes the reality of the tenure process at their institutions)
|
From page 78... ...
For example, a substantial number indicated that the realities of the T&P process at their institutions drove them to do work that was more theoretical in nature than they would have preferred, simply because experimental work in their environment was not as highly valued. In most cases, this issue arose because the T&P process placed the greatest weight on journal publications in its evaluation of research, journal publications that are themselves biased away from experimental work.
|
From page 79... ...
An aeronautical engineer may design a system to control the flight of an airplane under particular circumstances (e.g., strong wind shear) , and the flight control system will eventually be evaluated on the basis of its utility it A document that describes the evaluation of intellectual contributions of faculty in the arts is The Work of Arts Faculties in Higher Education, a report assembled by the Landscape Architectural Accreditation Board, National Architectural Accrediting Board, and the National Associations of Schools of Art and Design, Dance, Music, and Theater.
|
From page 80... ...
Thus, the aeronautical engineer is also likely to leave a "paper trail" on the way to implementing the flight control system. SUMMARY In conducting ECSE research, faculty members will imagine new computing ideas, create artifacts to implement them, and- measure properties of the artifacts.
|
From page 81... ...
EVALUATING RESEARCH IN ECSE 8 so often in ECSE that it may be the norm. There are examples of spectacular successes and unfortunate mistakes.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.