isern 2012

ISERN Program

During ISERN 2012, we will have two sessions that will contain a number of concurrent workshops. In order to ensure that we schedule workshops that are of interest to the ISERN attendees and to help guide the scheduling of the workshops, we are requesting that you provide us with feedback. Below are descriptions of each of the proposed workshops. Note that W5 and W6 are Restricted, meaning the workshop attendees are expected to be familiar with the background of the workshop topic. Workshop chairs will not cover any background material during the workshop. URL for Survey:

Sunday 16th September
18.00-20.00 Registration and Get-together
DAY 1: Monday 17 September
8.30-9.00 Registration

Welcome and new introductions
Chair: Dieter Rombach

ISERN is open to academic and industrial groups world-wide that are active in empirical software engineering research and willing to adopt the empirical research framework. ISERN members are pairs of organization and contact person. If the contact person leaves the organization, the organization must reapply for membership. Interested organizations may apply by sending an electronic proposal to "isern at" describing their past experience in empirical software engineering research as well as their expectations from a future ISERN membership. Candidates will be invited to observe the ISERN Meeting following their application.
The goal of the session is to facilitate the membership application process by giving an opportunity for candidates to present their research and for observers to introduce themselves. Membership is granted according to a 3-step procedure:

1) Attending as invited observer at an annual ISERN meeting.
2) Attending as invited candidate at the following ISERN meeting giving a presentation. Membership is granted if a two-thirds majority of current members approve the application in an email voting after the meeting.
3) Attending as a full ISERN member at following meetings.

Current members
present contact/affiliation changes (give a 2 min presentation each):
  • Lund University: Per Runeson.
  • University of Helsinki: Jürgen Münch.
Candidates (give a 5 min presentation each):
  • Ryerson University: Ayse Bener
  • Nanjing University: Guoping Rong
  • Peking University: Minghui Zhou
  • University of Twente: Maya Daneva
Observers (give brief introduction without a presentation):
  • North Dakota State University: Gursimran S. Walia.
  • Technische Universität München : Daniel Méndez
  • Univ. of Gronningen: Paris Avgeriou
Further information regarding changes (newly accepted members, leaving members)
10.30-11.00 Coffee break
11.00-12.00 Top Ten Unsolved Problems in ESE: Introduction
Chair: Natalia Juristo
  Introduction to the session and its goal: elaborate a list of the 10 most important problems the ESE community should focus on for the next years. The list will be elaborated following the Delphi approach. In this first session, participants are asked to propose a list of at least 5 problems they think ESE should try to solve in the near future.
12.00-13.30 Lunch
13.30-15.00 Ongoing collaborations
Chair: Claes Wohlin

Short presentations describing collaboration among ISERN members.

  • GQM+Strategies. Fraunhofer IESE (Germany), University of Maryland/Fraunhofer Center-Maryland (USA), University of Oulu (Finland). Vladimir Mandic, Jens Heidrich.
  • European Master on Software Engineering. Blekinge Institute of Technology (Sweden), University of Bolzano-Bozen (Italy), University of Kaiserslautern (Germany), Universidad Politécnica de Madrid (Spain), Universidade de Sao Paulo -San Carlos (Brazil). Barbara Russo.
  • A Framework for Improving Technology Adoption Decision Making. Fraunhofer IESE (Germany), Universidad Politécnica de Madrid (Spain). Andreas Jedlitschka.
  • Empirical studies on the benefits of using UML in software development. Leiden University (Netherlands), Ser&Practices (Italy), University of Bari (Italy), University of Castilla-La Mancha (Spain). Marcela Genero.
  • Experimental Studies on Technical Debt in the Software Industry. COPPE/Federal University of Rio de Janeiro (Brazil), University of Maryland - Baltimore College (USA), Federal University of Pernambuco (Brazil). Guilherme Travassos.
  • Evolution of Features and their Dependencies - An Explorative Study in OSS. University of Bolzano-Bozen (Italy), University of Calgary (Canada). Barbara Russo.
  • Decision-Making in Software and System Development: Models and Strategies. University of Rome "Tor Vergata" (Italy), Fraunhofer Center-Maryland (USA). Carolyn Seaman..
15.00-15.30 Coffee break
15.30-16.30 Searching for collaboration
Chair: Laurie Williams

Poster session. Participants are given 2 minutes each to present their poster (where they look for collaboration in a specific topic). The rest of the time people are left free to talk to the presenters.

  • A proposal of the collaborative experiment about the integrative project management using the Cloud environment with an in-process measurement facility. Information-Technology Promotion Agency (IPA), Japan. Yoshiki Mitani.
  • Proposal for replication of an experiment about real-time machine translation. University of Bari, Italy. Filippo Lanubile.
  • GQM+Strategies. University of Helskinki, Finland. Jürgen Münch.
  • Empirical investigation of safety engineering methods. University of Stuttgart, Germany. Stefan Wagner.
  • Social awareness as a trust building mechanism in global software teams. University of Bari, Italy. Filippo Lanubile.
  • Towards a Lean Assessment and Transformation Model for Software Domain. University of Oulu, Finland. Pilar Rodríguez.

Additionally, each candidate and observer will present a poster with future collaborations they expect to have.

16.30-17.15 ISERN recommended core reading list
Chair: Per Runeson

During the Banff meeting, Per Runeson and Sira Vegas were assigned the task to work on a core reading list for ISERN members. Although this list is mainly for new members, it is of interest for the whole community. The initial list will be presented and people will be asked to provide feedback.

17.15-17.30 Wrap-up and plan for Tuesday
17.30-18.30 ISERN SC Meeting (by invitation)
19.00-23.00 ISERN Dinner
DAY 2: Tuesday 18 September
9:00-10:00 Top Ten Unsolved Problems in ESE: The Voting
Chair: Natalia Juristo

Presentation of list of suggested interesting open problems in ESE. Each member should vote
10.00-10.30 Coffee break
10.30-12.00 Coding Contests – A Foundation for Large Scale Experiments in Empirical Software Engineering? (Open)
Chairs: Stefan Biffl, Dietmar Winkler, Christoph Steindl and Martin Kitzler
Professional Ethics in Writing Software Engineering Papers (Open)
Chairs: Reidar Conradi and M. Ali Babar

GQM+Strategies (Restricted)
Chairs: Vic Basili, Marku Oivo
  Background: It is often difficult to recruit a large number of participants in empirical studies to enable sufficient empirical evidence on the research area. Many empirical studies involve students. Professionals in industry typically do not participate in experiments because of resource limitations and cost. Coding contests are a promising option for carrying out empirical studies. In 2007, Catalysts started to organize coding contests ( in order to investigate the research question whether test-driven development is faster and leads to less defects. The contests have grown over the years from initially 60 participants to nearly 300 participants in 2012. Since 2011, Catalysts has been gathering Contest Data (about the progress per contestant, including source code per level, etc.) from the contests and has made that data available via public data servers (

Session goals: 1) Identification of candidate objectives for experiments/evaluation in context of coding contests. What are the most valuable objectives where coding contests can contribute? 2) Identification of most valuable (alternative) application areas? 3) Identification of collaboration opportunities between academia and industry coming from various organizations with respect to (a) research objectives, (b) data evaluation approaches, and (c) options to support coding contests within their organization.

Development of the session:
  • Presentation of Coding Contests: Concept and Strategy based on the history of past contests. Initial results from a series of Coding Contests.
  • Break-Out Groups: Identification of relevant research questions/issues based on previously conducted Coding Contests. Identification of relevant and valuable experiments for future Coding Contests. Brainstorming of alternative application areas which could be addressed by Contests.
  • Wrap-Up: Summary of Break-Out Groups (Plenary Discussion). Identification of the most relevant (three) items per Task.

Background: There has recently been much academic concern – particularly in medicine -- about poor ethics in writing scientific papers, i.e. avoiding data fraud, plagiarism, self-plagiarism, and so on. Now, we want to perform a prestudy of the prevalence of possible ethical misconduct in our own field, Software Engineering. Later, we will collect and compare cases of such misconduct also in other areas of Informatics (e.g. AI, GUI, DB ...), in non-Informatics areas (e.g. Chemistry, Sociology), and to study misuse trends.

Session goal: Based on personal experience in recent paper reviewing, we propose the following themes for discussion at the 2012 ISERN meeting: 1) Data fraud, by falsification or fabrication of any kind of data. 2) Plagiarism (of other people's work) is likewise regarded a very serious offence among researchers in all science disciplines. 3) Self-plagiarism. 4) Other forms of ethical misconduct upon writing and submitting papers. 5) The Vancouver rules -- originally developed for medicine -- express the minimum conditions for being listed as a co-author of a joint paper. 6) Finally, how to document cases of ethical misconduct?

Development of the session:

  • Welcome.
  • Panel 1: Prevalence of plagiarism among SE researchers. Using an emailed mini-questionnaire to assess the current state. Summing-up responses from some Editors, Program Chairs, and Event Organizers (see APPENDIX 2).
  • Panel 2: What do we do about it? Panelists: current/previous chief editors of key journals/magazines and of similar PC chairs. Discussing and sharing thoughts and experience, answering questions.
  • Conclusion. Are we all semi-crooks, and why: career building by Publish & Perish? And, is it better or worse than in other disciplines?
  • Final smalltalk.

Background: GQM+Strategies is an evolved version of GQM that aims at aligning and communicating an organizations goals and strategies at all levels, generating goals from strategies, making all goals measurable and interpreting high level measures in terms of the measurable goals that support them. Several organizations have been working on developing, evolving, and applying the approach in different environments.

Session goal: To coordinate what is being done at the various organizations and lay out a research and application plan for future work based upon what has been learned so far.

Recommended reading: V. Basili, M. Lindvall, M. Regardie, C. Seaman, J. Heidrich, J. Münch, D. Rombach, et al. Linking Software Development and Business Strategy through measurement. IEEE Computer, 43(4):57-65, 2010.

12.00-13.30 Lunch
13.30-15.00 What are (currently) the best practices in SE? (Open)
Chair: Andreas Jedlitschka
Empirically founded Requirements Engineering Improvement (Open)
Chairs: Stefan Wagner, Daniel Mendez
Qualitative syntheses (Restricted)
Chairs: Carolyn Seaman, Liliana Guzmán
  Background: At the Dagstuhl Seminar in 2006 and subsequent ISERN meetings, the level of maturity and several roadmaps towards achieving a higher level of maturity were discussed. Several of them dealt with the progress made in collecting different kinds of evidence and synthesizing them. Although the (E)SE community has made some significant progress in that regard, the question is: What have we learned or discovered about SE technologies? In consequence, would we be able to give (evidence-based) advice to practitioners, telling them about what the SE best practices are?

Session goals: If we compare ourselves to the evidence-based approach in medicine, we need to accept that the provision of evidence-based best practices cannot be placed on the shoulders of just a few organizations (people). In addition, it might be necessary to go beyond the borders of ESE and involve domain experts and practitioners. The session therefore aims at discussing and identifying the general interest in a community endeavor. Criteria for deciding about what a best practice is or when a technology gets the label "best practice" need to be discussed. Finally, we would be interested in ideas on potential SE topics that could be used as a pilot as well as in terms of cooperation and means for dissemination.

Development of the session: There will be a very brief introduction on the idea, followed by group work and discussions.
Background: Requirements Engineering (RE) constitutes an important success factor for software development projects. First steps have been done to investigate possibilities to analyze and improve RE processes in an inductive (qualitative) manner rather than doing so in a prescriptive way. At the latest EASE, we discussed first guidelines to empirically analyze software development processes by transferring empirical methods and measurements, or checklists to conduct empirical studies in general. The conclusion was, however, that we need to further extend the research co-operations between researchers and practitioners in order to support for further knowledge transfer, the harmonization and extension of available frameworks, and their empirically sound evaluation.

Session goals: 1) Discuss the state of the art in the transfer and experiences on the application of empirical methods and measurements to the continuous RE improvement to jointly create a brief thematic map, and 2) elaborate fields of further (potential) investigation, and identify potentials for collaborative research co-operations to extend and evaluate the created map.

Development of the session:
  • Short presentation: Introduction & Overview
  • Short joint discussion and definition of working groups that each focuses on one particular phase in the life cycl
  • Each group working on one of the life cycle phases to determine which empirical methods could be applied to which extent
  • Joint discussion on the integration and creation of a thematic map
  • Summary and joint discussion on potential research co-operations
Background: Research synthesis aims at analyzing, combining and summarizing the results of individual empirical studies on a research question. It attempts to confirm or build hypotheses as well as to support the prioritization and planning of future research. Research synthesis may also support practitioners in understanding the effects of a software technology, in which context those effects may be expected and how to better adopt it.

Session goal: To explore how to assess qualitative syntheses in SE. For this purpose, we plan to introduce and evaluate the reliability and validity of a checklist designed for assessing qualitative synthesis in SE

Development of the session: First, an introduction to the session is given. Then, participants will be randomly assigned to two groups, each assessing a different qualitative synthesis. In each group participants will individually assess the quality of the pre-assigned qualitative synthesis by using the predefined checklist. After completing the qualitative synthesis’ assessment, participants will answer a questionnaire regarding their perception on the understandability, consistency, completeness and usability of the checklist as well as their background in SE research

Recommended reading list:
  • Barnett-Page E. and Thomas J. (2009) Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology 9:59.
  • Dixon-Woods M., Agarwal S., Jones D., Young B. and Sutton A. (2005). Synthesizing Qualitative and Quantitative Evidence: A Review of Possible Methods. In: Journal of Health Services Research and Policy, 10(1): 45–53.
  • Dixon-Woods M., Bonas S., Booth A., Jones D., Miller T., Sutton A., Shaw R., Smith , J.A. and Young B. (2006). How can systematic reviews incorporate qualitative research? A critical perspective. In: Qualitative Research, 6: 27.
  • Steinke I. (2004) Quality Criteria in Qualitative Research. Book chapter in: Flick U., von Kardoff E. and Steinke I. (2004) A Companion of qualitative research. SAGE Publications.
  • Dixon-Woods M., Shaw R., Agarwal S. and Smith J.A. (2004) The problem of appraising qualitative research. Quality and Safety in Healthcare, 13: 223-225.
  • Mays N. and Pope C. (2000) Qualitative research in health care Assessing quality in qualitative research. BMJ, 320:50-52.
15.00-15.30 Coffee
15.30-16.30 Workshops summary session
Chairs: Jeff Carver and Sira Vegas

Each group describes conclusions, outcomes and future actions in 10 minutes
16.30-17.00 Top Ten Unsolved Problems in ESE: Final results
Chair: Natalia Juristo

Final list of 10 most voted problems that the ESE community should solve within the next few years
17.00-17.30 ISERN business
Chair: Dieter Rombach

This session includes the results of the “ISERN recommended core reading list” session