This article is an updated, extended, and translated version of a conference paper presented at DeLFI’14 [St14].

Introductory programming courses are part of computer science and many engineering, mathematics as well as science curricula. These courses usually employ programming tasks as practical exercises where students are frequently asked to write (small) programs from scratch, to implement interfaces, or to expand existing programs for training purposes. Programming tasks are often somewhat open-ended because it is important for students to develop their own solutions [Ro07]. The construction of high quality (i.e., meaningful, engaging, and challenging) programming tasks is complex, requires creativity and takes time [SW06]. There are many aspects to consider such as learning goals, knowledge of the students and possible solution strategies. Also assessment and timely feedback play an important role as the students’ submitted solutions have to be evaluated or graded. Due to the usually high number of enrolments in introductory programming classes, this results in a high workload for the teachers if done manually.

Therefore, the idea of (semi-)automatic assessment and feedback generation to programming tasks was born with early systems dating back to at least 1989 [Re89]. This reduced the manual grading effort at the expense of the development of sophisticated test procedures for a specific programming task: an automatic assessment of programming tasks is considerably complex than evaluating multiple choice questions [AR08]. In addition to the written description of each programming task, instructors usually need to create automated review, feedback, test data, or sample solutions depending on the automated grading system. Examples for automatically generated feedback are black box tests, Java JUnit tests [JUNIT] with check for various normal and “corner” cases, or comparisons with sample solutions e.g. for Prolog [Hü05].

The very high costs for creating sophisticated and creative programming tasks could be significantly reduced if tasks as well as tests were re-used, ideally independently of specific systems, and exchanged between teachers, e.g. by having a shared pool of programming tasks for a variety of learning scenarios. Although the development of support systems for computer education and automated graders in various programming languages has progressed for well over ten years (cf. [Ih10] for an overview and [SMB11, SOP11, PJR12, RRH12, Stö13] for examples of recent progress), there is no common exchange format yet. Also, only a few systems, such as DUESIE, eAIXESSOR, JACK, Mooshak and UVa Online Judge have or plan to have an easily accessible pool of re-usable tasks.

In particular, in systems with multiple installations, an import/export function could at least reduce the workload by re-using tasks within these systems. Nevertheless, import or export functionality is currently rare. It is only mentioned in publications about JACK, Praktomat, Mooshak and ELP. In other disciplines, e.g. math or engineering, there are established learning management systems (LMS) such as LON-CAPA [LONC] in which the worldwide exchange of tasks and learning material is a central component. In this paper we propose an interoperable XML-based exchange format (called ProFormA; based on a project name “Programming tasks for Formative Assessment”) for programming tasks to fill this gap.

The remainder of this paper is organized as follows: In section 2 three different systems for programming education used by the authors are described. Section 3 presents requirements for a generic exchange format based on a literature review on published systems, existing exchange specifications and their support for the derived requirements. Based on these requirements the ProFormA exchange format is introduced in section 4 and discussed in the subsequent section. The last two sections suggest some advanced usage scenarios, future work, and draw a conclusion.

This section describes three different support systems for programming education which have been in use by the authors for several years. Each system has an isolated pool of programming assignments. The purpose of this section is to point out the different usage scenarios and goals of these systems which motivated us to work on a common exchange format for programming tasks.

In this section we first present requirements for a generic exchange format for programming tasks which were derived from functionalities of existing systems. Three systems and their features were already presented in the previous section in more detail. These three systems already provide some insight into the spectrum of programming assessment and learning systems. For a generic exchange format, however, we need to systematically analyze a much broader spectrum of systems in order to derive their common requirements. Many systems have been published in a scientific context and there are even more systems in practical use which have not yet been extensively documented. Needless to say that we can only systematically review the published systems and, thus, we conducted a review on published programming assessment and learning systems which deal with source code written by students. The main criterion was that a system had to have been published since 2000 at scientific conferences or in journals related to learning science and technology. We also considered technical reports that have passed some kind of review process and have been published by their universities.

In the second part of this section, we then compare the derived requirements to existing import/export and exchange formats.

For each task the ProFormA format, described in this section of the paper, specifies a description (including its language for the purpose of internationalization), its programming language, supplied files, technical details of the submission, sample/model solutions and optional hints for marking and meta-data. A task can contain several tests where each has a test type (e.g. “unittest”) and a test configuration. Tests are often executed by standard software engineering tools such as compilers, unit tests, style checkers and debugging tools, and supplied in the format required by such tools. Therefore, the core ProFormA format does not need to specify the details of the content of such tests but only the parameters of their execution, for example tool names and version numbers. A system which imports files in the ProFormA format can determine whether or not the files are compatible with the system’s requirements by checking the test types.

The ProFormA format is XML-based and specified using an XML schema definition (cf. [PFMA]). The main reasons for using XML are that it is a widely accepted and established format for interoperability, that there are parsers available for various if not all programming languages, and that it is human as well as machine-readable which is helpful for debugging and adaptability. Furthermore, an XML schema definition allows to precisely specify all aspects of possible XML documents (e.g. fine grained data types, unique key constraints and logical structures), to validate XML documents if they comply to a specification and supports the usage of different XML namespaces.

Figure 1 depicts the structure of the ProFormA format and the nesting of elements and attributes. For referencing elements, XPath expressions are used. Elements which have three lines in the upper left corner correspond to XML simple elements, usually containing plain text, e.g. /task/description. Optional elements are surrounded by dashed lines (e.g. /task/grading-hints). Octagonal boxes symbolize XML sequences. Their cardinalities are provided unless they equal 1. The boxes with “any ##other” refer to possible extensions of the format. The ProFormA format only formalizes the common denominator of programming tasks. However, it can be extended to include tests from any other tool or programming language by providing “hot spots” in the sense of software frameworks which can be filled using format extensions defined in another XML namespace. The ProFormA format is therefore sufficiently flexible to incorporate local, system-specific configurations and novel testing tools. All specifications are versioned via their XML namespace URI in order to support future changes and extensions. For example “urn:proforma:task:v0.9.4” shows that this is the official ProFormA “task” specification in version “0.9.4”.

A task specification using the ProFormA format can either be a standalone XML document or a special layouted ZIP-archive whereby the XML document describing a programming task has to be named “task.xml” and placed in the root directory of the ZIP-archive (similar to the manifest file in IMS CP).

Some selected elements are discussed below in more detail, in particular with respect to the requirements from section 3.1. Listing 1 shows a minimal example of a task described in the ProFormA format. The task in this example consists of calculating the n-th Fibonacci number via loops or recursion, as often used in introductory Java classes.

Being required by most systems, file attachments (R2, R3, R5 and R9) and their handling in the ProFormA format are discussed first: For every file attached to a task there is a file element in /task/files. Thus, all attached files are defined at a central place within the XML document. A file element either holds the plain text contents of a file (cf. Listing 1 line 11ff) or, if the task is packed in a ZIP-archive as described above, a filename reference to a file in the ZIP-archive (cf. Figure 2). Both cases are distinguished by the type attribute which either contains the value “embedded” as a default or “file”. Especially for the embedded file case there is an optional filename attribute in order to specify how the file should be named when used. This is for example required for Java where the class name “defines” the filename and vice versa. Additionally there is a class attribute which specifies the intented access rights and visibility for the file (R2, R3, R5). The ProFormA format provides the following file classes: (Code-)templates/skeletons (template), libraries (library), input data (inputdata), additional information or instructions to process a task (instruction), internal libraries (e.g. libraries required for tests, internal-library) and internal files (e.g. test drivers or model solutions, internal). Especially the latter two are classes of files which should not be accessible to learners by default.

Every file has a unique ID for referencing purposes (id attribute). In this manner a file can be re-used in multiple contexts without the need to define multiple times (R7). Referencing is done using a filerefs element containing one or more fileref elements – one for every file reference. Eventually, a fileref element has a refid attribute which references the file ID of a specific file (/task/files/file).

Being semantically different from “normal” single files, model solutions (R9) are defined within the /task/model-solutions element. For each model solution there is one model-solution element which must have an unique ID (id attribute). It can also contain a human-readable comment describing the model-solution (comment attribute). This is useful if several model solutions are provided in order to point out their differences (e.g. an iterative and a recursive implementation). A model solution refers to at least one file. This is achieved using the already described filerefs/fileref elements where the filerefs element is nested directly under the model-solution element (see lines 34ff in Listing 1). Furthermore, there must be at least one model solution defined for each task. There are several reasons for this decision. First, on the technical level, there are systems which must have a model solution in order to compile JUnit tests provided as source code (e.g. GATE). In addition, there are practical reasons, because a sample solution together with the task description allows teachers to rate the difficulty and appropriateness of a task for the usage in their own classes.

The next important aspect of the ProFormA format is the definition and configuration of tests and evaluation methods (R5, R6 and R7): Tests can be defined in /task/tests/test elements (R6, see Listing 1 lines 39ff). In addition to specifying which kinds of tests are available for a task (for example compile/syntax or JUnit tests, test/test-type) it is possible to specify the configuration of a test (test/test-configuration). The definition of test types is also part of the ProFormA format in order to provide a shared set of possible types which are consistently interpreted. However, it is still possible to extend the format by defining personal test types. Extra files needed for a test such as test drivers are defined within /task/files/file elements. They are referenced using their unique IDs within filerefs/fileref elements (R5). In this manner it is known in advance which files are required for a test and which are not (R7). In order to be able to exchange test type specific parameters such as the name of the main class for a JUnit test, separate XML namespaces for every test type within the test-configuration element can be used. The test-configuration/test-meta-data element allows to use special namespaces to define system-specific meta-data which belongs to a test (e.g. restrictions for tests).

As already stated above, test types use their personal XML namespace for configuration. The test types “java-compilation” and “unittest” have been specified so far. The first official type “java-compilation” (Listing 1 lines 42f) does not need any special configuration for a minimal instance because the libraries to be used can be detected by referenced files and class of the files. JUnit tests, however, need a special configuration at least for the canonical name of the main class which should be executed and should ideally describe the version of the JUnit framework. Therefore, the unittest/main-class element in the XML namespace “urn:proforma:tests:unittest:v1” was defined (Listing 1 lines 47ff). There is nothing else needed for a minimal configuration as the system knows how to run the JUnit tests using the main class file and the referenced files containing the JUnit test cases.

Tasks should usually be self-contained as only this condition guarantees that a task is usable in the long-term without losing or missing components. In rare cases the use of external resources is unavoidable. For example, an automatically graded task may need a huge database dump or a large library that should not be bundled reasonably with the task itself (R2, R3, R5). For these cases the /task/external-resources element was added so that a task may reference the required external resources by a unique name (attribute reference). The semantics and format of external references are currently not standardized in the ProFormA format. An example for an external resource is an URL of a widely known database dump (e.g. ftp://ftp.fu-berlin.de/pub/misc/movies/database/ ). More complex references can be included in a personal XML namespace. As an example consider a Maven coordinate tuple identifying the name and version range of a library (a so called “GAV”) consisting of a small XML snippet, e.g. “<groupId>org.jmock </groupId><artifactId>jmock-junit4</artifactId><version>[2.4.0, 2.5.0]</version>”. The main reason for the inclusion of external references in the current generic form is the usage scenario described in section 6.

The /task/submission-restrictions element provides the option to define restrictions for submissions of task solutions. For example, it is possible to specify the maximally allowed file size (in byte) for a solution file which should be uploaded into the system (attribute max-size). Of course, while this limit represents an expected maximum by the author of a specific task, a system limitation can still be stricter (e.g. for security purposes). Constraints for filenames can be specified via regular expressions using the attribute allowed-upload-filename-regexp (line 8 in Listing 1). This is important for Java tasks where a class name determines the filename when using the default class loader. If learners want to upload a solution file for a task into a system, the system first verifies if all constraints of the specified regular expression are satisfied by the filename. If this is not the case the system can prevent learners from uploading their solution files and/or provide a warning. This is intended to ensure that tests will not fail because of an invalid filename (R4).

An instance of the /task/description element includes the task’s description (R1, Listing 1 lines 2ff). Furthermore, it is possible to provide a grading scheme (/task/grading-hints) and other meta-data (/task/meta-data). As existing systems use very different ways for defining grading schemes (cf. JACK, ViPS, Web-CAT) and grading schemes are quite personal, there is no common specification for this and the element was called “grading-hints”. In analogy to the test-meta-data element, a personal XML namespace in the /task/grading-hints element can be used to define system-specific details about grading schemes. The ProFormA format also offers the opportunity to specify attributes belonging to tasks or tests which are not relevant for all but for certain systems, to make sure that system specific information does not get lost on a task export. In this manner, a complete task export into the ProFormA format with subsequent lossless re-import of the task into the same system can be guaranteed. Furthermore, this ensures that only data is imported into a system which this system can handle (R8).

When discussing an interchange standard such as ProFormA, interesting questions are which aspects to include and, finally, how to model them. The first question has two extreme cases: On the one side there are specific aspects such as title, description, and test configurations which are required by all or most systems and on the other side there are specific aspects which are only used by one or very few systems such as sample solutions or reference files (cf. Table 1). The first case is quite clear; those aspects are important and, thus, have to be included. Elements of the latter extreme case are unlikely to be included as required features in the core format specification. All aspects between the two extremes are subject to discussion. The second question is always subject to discussion in order to achieve justified design decisions.

Our proposed specification is supported by the three systems, GATE, JACK, and an enhanced Praktomat version, described in section 2, for exporting and importing programming tasks. The initial focus on Java tasks is due to the fact that the three systems used by the authors mainly deal with Java tasks and also most systems of our review support Java (cf. Table 2 in the appendix for the concrete systems). Furthermore, half of the systems explicitly state that JUnit is used (cf. Table 2). Only 11 out of the 33 reviewed systems state explicitly or indirectly that JUnit is not used. Oftentimes these systems do not deal with Java or use other testing methods (cf. Table 2). The ProFormA exchange format has been implemented and tested by the tree systems for more than one year. However, the design of the format is not solely based on these three systems but also on requirements of other systems, especially ViPS using Prolog and LON-CAPA external response [LONCM] as well as others mentioned in section 3.1. This ensures that the specification does not only implement requirements of a single system or only supports some programming languages as it is the case for most existing specifications (cf. section 3.2). In addition, the ProFormA format is not designed as a fixed “one-size-fits-all” specification. It does not ignore system-specific aspects, but those can be embedded with freely definable XML namespaces as extensions. Extensions for individual systems, new programming languages as well as novel test types are supported. Extensibility using XML namespaces, however, harbors the risk that fundamental aspects are defined multiple times or in different ways. In order to mitigate this risk, on the one hand, we believe that all fundamental aspects have already been defined in the ProFormA format itself and, on the other hand, the proposed format is versioned. Therefore, changes, additions and moving of “wide-spread” aspects into the core format are possible without losing compatibility for existing tasks.

Metadata for categorization was deliberately not included in the first version of the specification. For metadata, there are, on the one hand, established standards such as IEEE Learning Objects Metadata (LOM) and Dublin Core and, on the other hand, these are often not fully used by authors of learning material [Go04], possibly due to motivational aspects because the author who creates the meta-data is usually not the person who benefits from it. Instead, at least one sample solution is required in the ProFormA format. This should allow potential users of tasks to rate them according to difficulty and appropriateness, thereby judging whether or not a task fits into their course.

Restrictions for filenames of requested files (R4) are a requirement of just some systems. However, we decided to include this into the format specification (/task/submission-restrictions element) with the optional allowed-upload-filename-regexp attribute. The rationale is that this does not cause harm if ignored or unsupported by systems, but can help to prevent failures of tests (see section 4). An alternative way which explicitly specifies all required filenames ensuring that no file is missing (as used in e.g. JACK) is under discussion. The explicit statement of filenames can easily be converted using the existing allowed-upload-filename-regexp attribute, however, the opposite direction is not possible.

A special editor is, in principle, not needed for the ProFormA format, since each system already provides its own authoring functionality. However, the development of a system-independent editor is planned which can be extended by plugins to support system-specific aspects – especially with respect to the usage scenario described in section 6. Also, a dedicated editor would allow for creating tasks system-independently.

In addition to the export functionality, support for importing tasks has been implemented for the three systems described in section 2. In this context, cross-system imports, which export from one system and import into another system, and same-system imports/exports were carried out for testing purposes for these tree systems. In principle an export is possible from all three systems (GATE, JACK and the enhanced Praktomat). However, in the JACK system there are no model solutions available and, thus, the necessary model solutions are not yet exported up to now. GATE and Praktomat generate fully valid XML documents complying with version 0.9.1 of the specification. Support for the new version 0.9.4 is currently being implemented for these systems and is already available in Praktomat.

A same-system and inter-system import is possible for each of the three systems in principle. However, it is not always guaranteed that all features of an imported task which was previously exported from another system are fully re-usable. On the one hand, this is due to the supported assessment procedures of the importing vs. exporting system and, on the other hand, because of the special system features which separate different systems. For example, all three systems are used for Java programming tasks. GATE currently only supports JUnit 3 tests and Praktomat supports several JUnit versions (hence the use of JUnit3 in Listing 1). Therefore, syntax and JUnit3 test procedures can be shared between these two systems without losing functionality. The JACK system, however, does not support JUnit tests. It relies on other methods which are currently exclusive to the JACK system. Nevertheless, system-specific elements can be exported in the form of metadata using the elements /task/meta-data and /task/tests/test/test-configuration/test-meta-data. As already mentioned above, the essential elements of programming tasks are shareable across systems, and also same-system imports are possible without losing information.

Learning management systems such as LON-CAPA [LONC], Stud.IP [STDIP], OLAT [OLAT], or Moodle [MOOD] are widely used in educational contexts and have their own user management, tools for communication, such as e-mail or forums, and sophisticated course management features. These tools can be seen as general purpose support tools in this context. Hence, they usually cannot assess computer science exercises or programming tasks. Many of the systems mentioned in section 3.1 are well-suited for this purpose (depending on their particular goals). Nevertheless, nearly all of the systems seem to be stand-alone (cf. Table 2 in the appendix), having their own user management but usually without sophisticated course management features. Thus, these programming assessment systems are called “grading systems” (GS) in this section as the grading functionality is the focus. Additionally, GSs tend to be specialized for the assessment of programming tasks in specific programming languages using a fixed set of test types. As a consequence LMSs and programming assessment systems are often either used side by side in various LMS and GS combinations or only a single system with a limited feature set.

An obvious idea is to combine the advantages of both system types, LMSs and GSs, in order to get the “best of both worlds” using the course management and community features from an LMS and adding the special features of grading systems. A first simple connection would be to bridge the gap between these distinct tools by employing some kind of secure link to the other system combined with an auto-login feature. This approach is used by the JACK system in combination with Moodle, or in eClaus with OLAT/OPAL. The IMS Learning Tool Interoperability (LTI, [IMSLT]) standard provides such a feature and also allows an external tool to pass learning or assessment results back to the calling LMS. However, when a student clicks on the secure link, full control is handed to the external tool which means that the student is directly interacting with the external tool. This often has a layout and handling that is completely different from the LMS. Furthermore, just providing a login is not sufficient for exchanging complex assessment data between the systems. A slightly more complex connection would be to integrate an assessment plugin directly into an LMS or to write a GS specific wrapper (ViPS, VPL, WeBWorK-JAG, or planned by SAC [Au08]). This, however, might have security implications or limits the usage of different GS at the same time.

A more flexible solution could be an architecture incorporating a special middleware which supports a detailed and secure data exchange between the systems based on the ProFormA format and a yet to be determined exchange protocol (cf. Figure 3): Lecturers as well as students utilize their preferred web clients for connecting to their usual LMS. Within this familiar learning environment, students can work on programming tasks which are stored within the LMS. Assessment and feedback generation is dealt with transparently by one or multiple external GSs which can even be of different types. The GSs are connected to the LMS using a middleware which only needs to send the task specification utilizing the ProFormA format and the student solution to the grader.

Such a middleware architecture has multiple advantages: It allows an LMS to use different GSs, it can implement load balancing for multiple GSs of the same type, and at the same time it hides the complexity of the connection details and of the data exchange. Thus it simplifies the interfaces at both ends – for the LMS and for the GS. A properly designed middleware can easily be re-used by different LMS and, therefore, lower the requirements for changes/extensions necessary to a specific LMS.

Special attention needs to be given to the results and feedback generated by the GS. In the proposed architecture, graders are concerned with generating feedback about a submission received from the middleware. Feedback usually consists of a grade, or a score, a percentage or just a boolean value and some documentation about the GS’s results. Documentation can range from a simple text (e.g., in form of a console dump) over XML documents with markup for message categories and message content to complete documents that are ready for immediate presentation to the user such as HTML, PDF, images or other media types. Some GSs can generate separate feedback for students and teachers at different levels of detail. Just as the ProFormA format of this paper aims at covering any programming exercise configuration and description, a unifying feedback format is needed that covers grading results from any grader. Ideally, an LMS should allow students and teachers to inspect the results working with the user interface of the familiar LMS. Many LMSs already support the concept of manually or automatically graded assignments. Extending an LMS so that it will integrate grader results into its LMS-specific assignment concept is less costly under the assumption of a unifying grading feedback format. Currently we are eliciting the respective requirements from several GSs and learning management systems for such a unifying format.

Two prototypes have been developed to serve as a connector component between LMS and GS. In [PJR12a] an LMS (LON-CAPA) is connected to a grader (Praktomat) with the help of a dedicated middleware server. Another implementation describes the “Grappa” server, which connects the LMS Moodle to the grader aSQLg [Stö14]. Both prototypes are either capable of or currently being developed towards supporting the ProFormA format described in this paper. But none of the systems already uses a shared feedback format.

In this article, specific requirements for a special exchange format for programming tasks were presented. Based on these requirements, an XML-based exchange format called ProFormA was derived. This format can be used for the exchange of programming tasks across systems. This schema is not a "one-size-fits-all" specification for fixed languages and test types, but a format which allows the use of different XML namespaces for extensibility. Consequently, this specification provides a good basis for increasing the interoperability between different systems. Export and import functionality using the ProFormA format is already available in several systems: Essential elements of programming tasks that were exported in the ProFormA format can be imported into these systems. Thus, a cross-system sharing of tasks using the ProFormA format has proven to be feasible. In principle, all 33 systems reviewed in section 3.1 could make use of it.

The next steps in our work include the refinement of the elements of the described specification and the integration with other systems: in stand-alone systems for import and export as well as usage for LMS-GS communication. The current version of the ProFormA format builds the basis for sharing the integral elements of programming tasks. Future refinements might include parametric exercises, exercise bundling and further aspects regarding security and runtime limitations of tests. Also, a formal way for suggesting new test type specifications for inclusion in the “official” ProFormA format will need to be developed.

For ease of sharing, a repository of programming tasks described in the ProFormA specification shall be set up in the future, so that tasks can be searched and imported directly into a specific system. A repository provides an opportunity to increase the quality of teaching by repeated usage of good tasks, especially across different universities. This does not mean that a separate repository needs to be built, the utilization of existing repositories is explicitly not ruled out. However, there are important requirements on a shared repository of programming tasks: Especially, it is of particular significance that access to the complete task specifications is restricted in a way so that learners cannot “cheat” by using the model-solutions given in the tasks (cf. LON-CAPA, [LONC]).

[Al05] Ala-Mutka, K.: A Survey of Automated Assessment Approaches for Programming Assignments. In: Computer science education, 15(2), 2005, pp. 93-102.

[APR06] Amelung, M.; Piotrowski, M.; Rösner, D.: eduComponents: experiences in e-assessment in computer science education. In: Proc. ITiCSE’06, 2006, pp. 88-92.

[AR08] Amelung, M.; Rösner, D.: Experiences in Hybrid Learning with eduComponents. In: Hybrid Learning and Education. Springer, Berlin Heidelberg, 2008, pp. 259-270.

[ASR06] Ahtiainen, A.; Surakka, S.; Rahikainen, M.: Plaggie: GNU-licensed source code plagiarism detection engine for Java exercises. In: Proc. Baltic Sea conference on Computing education research. ACM, New York, NY, USA, 2006, pp. 141–142.

[ASS08] Altenbernd-Giani, E.; Schroeder, U.; Stalljohann, P.: eAixessor - A Modular Framework for Automatic Assessment of Weekly Assignments in Higher Education. In: Proc. IASTED’08, 2008, p. 99.

[Au08] Auffarth, B.; López-Sánchez, M.; Miralles, J. C.; Puig, A.: System for Automated Assistance in Correction of Programming Exercises (SAC). In Proc. CIDUI’08, 2008, pp. 104-113

[CMM04] Cesarini, M.; Mazzoni, P.; Monga, M.: Learning Objects and Tests. In: The IASTED International Conference on Web-Based Education, 2004, pp. 520-524.

[DGNU] DejaGNU, GNU Project. https://www.gnu.org/software/dejagnu/ (last check 2015-02-26)

[DH04] Daly, C.; Horgan, J. M.: An automated learning system for Java programming. In: Education, IEEE Transactions on Education, 47(1), 2004, pp. 10-17.

[Ed04] Edwards, S. H.: Using software testing to move students from trial-and-error to reflection-in-action. In ACM SIGCSE Bulletin 36(1), 2004, pp. 26-30.

[EFK04] Ellsworth, C. C.; Fenwick Jr, J. B.; Kurtz, B. L.: The quiver system. In: ACM SIGCSE Bulletin, 36(1), 2004, pp. 205-209.

[Ei03] Eichelberger, H.; Fischer, G.; Grupp, F.; Von Gudenberg, J. W.: Programmierausbildung Online. In: Proc. DeLFI’03, 2003, pp. 134-143.

[Fu08] Fu, X.; Peltsverger, B.; Qian, K.; Tao, L.; Liu, J.: APOGEE: automated project grading and instant feedback system for web based computing. In: ACM SIGCSE Bulletin, 40(1), 2008, pp. 77-81.

[GATE] https://cses.informatik.hu-berlin.de/research/details/gate/ , https://github.com/csware/si/ , (last check 2015-02-26)

[Go04] Godby, C. J.: What Do Application Profiles Reveal about the Learning Object Metadata Standard? In: Adriane Article in eLearning Standards, 2004.

[GSW07] Gotel, O.; Scharff, C.; Wildenberg, A.: Extending and contributing to an open source web-based system for the assessment of programming problems. In: Proc. international symposium on Principles and practice of programming in Java, 2007, pp. 3-12.

[HQW08] Hoffmann, A.; Quast, A.; Wismüller, R.: Online-Übungssystem für die Programmierausbildung zur Einführung in die Informatik. In: Proc. DeLFI‘08, 2008, pp. 173-184.

[Hü05] Hügelmeyer, P.; Mertens, R.; Schröder, M.; Gust, M.: Integration des Virtuellen Prüfungssystems ViPS in die Lehr-/Lernplattform Stud.IP. In: Proc. Workshop on e-Learning 2005, HTWK Leipzig, pp. 187-196.

[HW08] Hwang, W. Y.; Wang, C. Y.; Hwang, G. J.; Huang, Y. M.; Huang, S.: A web-based programming learning environment to support cognitive development. In: Interacting with Computers, 20(6), 2008, pp. 524-534.

[Ih10] Ihantola, P.; Ahoniemi, T.; Karavirta, V.; Seppälä, O.: Review of recent systems for automatic assessment of programming assignments. In: Proc. Koli Calling‘10, pp. 86-93.

[IMSCP] IMS Global Learning Consortium. Content Packaging Specification. http://www.imsglobal.org/content/packaging/ (last check 2015-02-26)

[IMSLT] IMS Global Learning Consortium. IMS Learning Tools Interoperability (LTI) Implementation Guide. http://www.imsglobal.org/lti/ltiv2p0/ltiIMGv2p0.html (last check 2015-02-26)

[IMSQT] IMS Global Learning Consortium. IMS Question & Test Interoperability™ Specification. http://www.imsglobal.org/question/ (last check 2015-02-26)

[JACK] http://www.s3.uni-duisburg-essen.de/research/jack.html (last check 2015-02-26)

[JGB05] Joy, M.; Griffiths, N.; Boyatt, R.: The boss online submission and assessment system. In: JERIC, 5(3), 2005, p. 2.

[JUNIT] https://junit.org (last check 2015-02-26)

[KJ13] Kruse, M.; Jensen, N.: Automatische Bewertung von Datenbankaufgaben unter Verwendung von LON-CAPA und Praktomat. Proc. Workshop Automatische Bewertung von Programmieraufgaben, ABP’13, 2013.

[KSZ02] Krinke, J.; Störzer, M.; Zeller, A.: Web-basierte Programmierpraktika mit Praktomat. In: Softwaretechnik-Trends, 22(3), 2002, pp. 51-53.

[LLP13] Le, N. T.; Loll, F.; Pinkwart, N.: Operationalizing the Continuum between Well-defined and Ill-defined Problems for Educational Technology. In: IEEE Journal Transactions on Learning Technologies, 2013, 6(3), pp. 258-270.

[LONC] The LearningOnline Network with CAPA. http://loncapa.org (last check 2015-02-26)

[LONCM] Learning Online Network with CAPA. Author’s Tutorial And Manual. https://loncapa.msu.edu/adm/help/author.manual.pdf (last check 2015-02-26)

[LQ09] Leal, J. P.; Queirós, R.: Defining Programming Problems as Learning Objects. In: Proc. ICCEIT’09, 2009.

[LS03] Leal, J. P.; Silva, F.: Mooshak: A Web‐based multi‐site programming contest system. In: Software: Practice and Experience, 33(6), 2003, pp. 567-581.

[Ma09] Mareš, M.: MOE – Design of a modular grading system. In: Olympiads in Informatics, 3, 2009, pp. 60-66.

[Mo07] Morth, T.; Oechsle, R.; Schloß, H.; Schwinn, M.: Automatische Bewertung studentischer Software. In: Proc. Pre-Conference Workshops der DeLFI’07, 2007.

[MOOD] Moodle - Open-source learning platform. https://moodle.org (last check 2015-02-26)

[No07] Nordquist, P.: Providing accurate and timely feedback by automatically grading student programming labs. In: Journal of Computing Sciences in Colleges, 23(2), 2007, pp. 16-23.

[OLAT] OLAT Lernmanagement. https://www.bps-system.de/cms/produkte/olat-lernmanagement/ (last check 2015-02-26)

[PFMA] ProFormA specification and whitepaper. https://github.com/ProFormA/taskxml (last check 2015-02-26)

[PJR12] Priss, U.; Jensen, N.; Rod, O.: Software for E-Assessment of Programming Exercises. In: Informatik 2012, GI LNI, P-208, pp. 1786-1791.

[PJR12a] Priss, U.; Jensen, N.; Rod, O.: Software for Formative Assessment of Programming Exercises. In: elearning Baltics 2012, Proc. International eLBa Science Conference, 2012, pp. 63-72.

[PRAK] https://github.com/KITPraktomatTeam/Praktomat (last check 2015-02-26)

[QL11] Queirós, R.; Leal, J. P.: Pexil: Programming exercises interoperability language. In: Proc. Conferência - XML: Aplicações e Tecnologias Associadas (XATA), 2011.

[QL12] Queirós, R.; Leal, J. P.: PETCHA: a programming exercises teaching assistant. In: Proc. ACM ITiCSE’12, 2012, pp. 192-197.

[QL13] Queirós, R.; Leal, J. P.: BabeLO – An Extensible Converter of Programming Exercises Formats. In: IEEE Transactions on Learning Technologies, 6(1), 2013, pp. 38-45.

[Re89] Reek, K. A.: The TRY system-or-how to avoid testing student programs. In: ACM SIGCSE Bulletin, 21(1), 1989, pp. 112-116.

[RH08] Rößling, G.; Hartte, S.: WebTasks: online programming exercises made easy. In: ACM SIGCSE Bulletin, 40(3), 2008, pp. 363-363.

[RML08] Revilla, M. A.; Manzoor, S.; Liu, R.: Competitive learning in informatics: The UVa Online Judge Experience. In: Olympiads in Informatics, 2, 2008, pp. 131-148.

[Ro07] Romeike, R.: Three Drivers for Creativity in Computer Science Education. In: Proc. of Informatics, Mathematics and ICT: a 'golden triangle'. Boston, US, 2007.

[RRH12] Rodríguez-del-Pino, J.; Rubio-Royo, E.; Hernández-Figueroa, Z.: A Virtual Programming Lab for Moodle with automatic assessment and anti-plagiarism features. In: Proc. CSREA EEE’12, 2012.

[SBG09] Striewe, M.; Balz, M.; Goedicke, M.: A Flexible and Modular Software Architecture for Computer Aided Assessments and Automated Marking. In: Proc. CSEDU’09, 2009, pp. 54-61.

[SCORM] Rustici Software. Technical SCORM. http://scorm.com/scorm-explained/technical-scorm/ (last check 2015-02-26)

[SG10] Striewe, M.; Goedicke, M.: Visualizing Data Structures in an E-Learning System. In: Proc. CSEDU’10, 2010, pp. 172-179.

[SG11] Striewe, M.; Goedicke, M.: Using Run Time Traces in Automated Programming Tutoring. In: Proc. ACM SIGCSE ITiCSE’11, 2011, pp. 303-307.

[SG13] Striewe, M.; Goedicke, M.: JACK revisited: Scaling up in multiple dimensions. In: Proc. EC-TEL 2013, 2013, pp. 635-636.

[Sh05] Shaffer, S. C.: Ludwig: an online programming tutoring and assessment system. In: ACM SIGCSE Bulletin, 37(2), 2005, pp. 56-60.

[SMB11] de Souza, D. M.; Maldonado, J. C.; Barbosa, E. F.: ProgTest: An environment for the submission and evaluation of programming assignments based on testing activities. In: IEEE Software Engineering Education and Training (CSEE&T), 2011, pp. 1-10.

[SOP11] Strickroth, S.; Olivier, H.; Pinkwart, N.: Das GATE-System: Qualitätssteigerung durch Selbsttests für Studenten bei der Onlineabgabe von Übungsaufgaben? In: Proc. DeLFI’11, 2011, pp. 115-126.

[St14] Strickroth, S.; Striewe, M.; Müller, O.; Priss, U.; Becker, S.; Bott, O. J.; Pinkwart, N.: Wiederverwendbarkeit von Programmieraufgaben durch Interoperabilität von Programmierlernsystemen. In: Proc. DeLFI‘14, 2014, pp. 97-108.

[STDIP] Stud.IP-Portal: Stud.IP 3: Das Portal. Projekt, Informationen und Ressourcen. http://studip.de (last check 2015-02-26)

[Stö13] Stöcker, A.; Becker, S.; Garmann, R.; Heine, F.; Kleiner, C.; Bott, O. J.: Evaluation automatisierter Programmbewertung bei der Vermittlung des Sprachen Java und SQL mit den Gradern aSQLg und Graja aus studentischer Perspektive. In: Proc. DeLFI‘13, 2013, pp. 233-238.

[Stö14] Stöcker, A.; Becker, S.; Garmann, R.; Heine, F.; Kleiner, C.; Werner, P.; Grzanna, S.; Bott, O. J.: Die Evaluation generischer Einbettung automatisierter Programmbewertung in Moodle. In: Proc. DeLFI‘14, 2014, pp. 301-304.

[Sp06] Spacco, J.; Hovemeyer, D.; Pugh, W.; Emad, F.; Hollingsworth, J. K.; Padua-Perez, N.: Experiences with Marmoset: Designing and using an advanced submission and testing system for programming courses. In: ACM SIGCSE Bulletin 38(3), 2006, pp. 13-17.

[SVW06] Schwieren, J.; Vossen, G.; Westerkamp, P.: Using software testing techniques for efficient handling of programming exercises in an e-learning platform. In: The Electronic Journal of e-Learning, 4(1), 2006, pp. 87-94.

[SW06] Stevenson, D. E.; Wagner, P. J.: Developing real-world programming assignments for CS1. In: ACM SIGCSE Bulletin, 38(3), 2006, pp. 158-162.

[Tr07] Truong, N.: A Web-Based Programming Environment for Novice Programmers. PhD Thesis, 2007: http://eprints.qut.edu.au/16471/1/Nghi_Truong_Thesis.pdf (last check 2015-02-26)

[Tr08] Tremblay, G.; Guérin, F., Pons, A.; Salah, A.: Oto, a generic and extensible tool for marking programming assignments. In: Software: Practice and Experience, 38(3), 2008, pp. 307-333.

[Ve08] Verhoeff, T.: Programming Task Packages: Peach Exchange Format. In: Olympiads in Informatics, 2, 2008, pp. 192–207.

[Ve11] Verdú, E.; Regueras, L. M.; Verdú, M. J.; Leal, J. P.; de Castro, J. P.; Queirós, R.: A distributed system for learning programming on-line. In: Computers & Education 58(1), 2011, pp. 1-10.

[WW07] Weicker, N.; Weicker, K.: Automatische Korrektur von Programmieraufgaben – Ein Erfahrungsbericht. In: Flexibel integrierbares e-Learning - Nahe Zukunft oder Utopie, Proc. Workshop on e-Learning 2007, pp. 159-173.

The work of the authors Becker, Müller, Priss and Rod has been partially funded by the German Federal Ministry of Education and Research (BMBF) under grant numbers 01PL11066H and 01PL11066L.