bild
School of
Computer Science
and Communication

The TIGRIS testing system

Submissions to TIGRIS are made via mail, results are mailed back as two separate messages (one with compilation results and, if compilation of the compiler and tests went well, another with results from running the execute/nonexecute tests).

You need an account in order to use TIGRIS. This is not your KTH account, but a separate account system. Students of DD2488 will be given such an account at project start.

The TIGRIS test system for your compilers has been implemented. This system checks that compilers correctly accept just correct Minijava code, and that they generate correct code for well-defined testcases. It also checks that some run-time errors are handled when such feature are claimed.

After you have mailed your submission, it is queued for compilation of the compiler and compilation of the tests. If TIGRIS cannot even unpack your archive, you will get an error message as a mail bounce, or in some cases at the address registered and deduced from the ID in the DESC file (see below for information about the DESC file). There are some cases of malformed submissions that are silently ignored, to prevent TIGRIS from sending spam.

You will get mail when compilation of your compiler and compilation of the tests using your compiler are completed. Expect at least 3 minutes, and up to several hours if the queue is long. (There is a limit for how slow compilers are allowed to be; currently 8 minutes for compiling and running all tests. This is plenty, please don't worry about your compiler's performance.)

If you make several submissions in a short period of time, only the last will actually be handled by TIGRIS; older submissions will be archived but otherwise ignored.

Iff compilation went well, you will get a 2nd mail, with execution results. This will be quick for targets JVM and X86; expect it to arrive within 2 minutes after a compilation success mail. Other targets are currently much slower, they require 10-20 minutes.

Here is how to submit your compiler to the test system:

cd [DIRECTORY-JUST-ABOVE-COMPILER-SOURCE-DIR]
tar cf - [COMPILER-SOURCE-DIR] | gzip | uuencode x | mail -s "" submit@tigris.csc.kth.se

There is a simple shell script that submits compilers to TIGRIS. (This is also available from Nada's machines at /info/komp14/bin/tigrissubmit.)

Replace [DIRECTORY-JUST-ABOVE-COMPILER-SOURCE-DIR] and [COMPILER-SOURCE-DIR] as appropriate.

In [COMPILER-SOURCE-DIR], you need these files:

file / directory short description
build.xml Ant buildfile with specified targets (see below)
DESC Project DESCription file, informing TIGRIS about what you implemented
src Your source code, including lexer and parser source, and your own override runtime.c
lib E.g., JavaCup, JavaCC, Jasmin, or other tools you need
report.pdf Your project report

Note that you should not include any .class files (or .jar files) for any of the code you've written for this project. TIGRIS wants sources only, since examination is based on sources. However, in the lib directory, there may be ready-to-run .class or .jar files, which are referenced from your ant buildfile, either for use by the build process, or needed as part of the execution environment (e.g., parser generator runtimes, lexer runtimes).

TIGRIS has its builtin runtime.c, matching the one on the course pages. If you don't like that version, write your own and put it into your src directory. TIGRIS will find it there.

Files build.xml or Makefile

You may choose to let TIGRIS build your compiler with either make or ant. The presence of a file Makefile in the top-level directory instructs TIGRIS to use GNU make.

The ant buildfile should contain a default target that builds the compiler, runs the parser generator and lexer generator (from your lib directory), and then generates one compiler (or several compilers) as a .jar file.

The optional Makefile should work similarly. Note that tigrissubmit as well as the TIGRIS mail-in-scripts currently enforces the existence of a file build.xml. Please provide an empty one for now.

For compilers written in Java, The generated compiler .jar file should be called mjc.jar ("MiniJava Compiler") and be placed at the top-level directory. For compiler written in other languages, the generated compiler executable should be called simply mjc.

See "How your compiler should behave" below for additional instructions.

File DESC

This plain text file describes (1) who you are and (2) which features (see Levels on the project page) your compiler claims to implement. The file will contain one claim per line. You may combine back-end claims and other claims for features you have not implemented for every backend. Example:

MAIL=kalle@kula.com
ID=...
LANG=java
ARM
X86
ARM:SPILL
IWE
X86:ABC
ISC

This means that you have ARM and X86 backends. For the ARM backend, you have spilling. You support the IWE extension (which is in the syntax part and cannot be back-end specific, so ARM:IWE is not allowed). For the X86 backend, you support ABC, array bounds checks. You have inheritance syntax checks (which also must work for any backend, since it really isn't about codegen).

The LANG declaration specifies the compiler implementation language. TIGRIS understands java, c, c++, and haskell. Note that you need to write an ant build file that also invokes the correct compiler (for C: cc or gcc; for C++: c++ or g++; for java: javac; for haskell: ghc). TIGRIS does not know about the make build tool, nor does it know about cabal.

You must put the email addresses in the MAIL field to which your ID was mailed. The ID is used as a simple anti-abuse mechanism, and is used to identify you. (Please read-protect any DESC files to keep the ID private; actually, you should read-protect the entire directory just like with any school assignment.)

File report.pdf

The file report.pdf needs to exist for TIGRIS to accept a submission, but it may be a preliminary report or even just a placeholder. The final accepted submission needs to contain a completed report, since this is where we will find your report before your oral presentation.

How your compiler should behave

TIGRIS will invoke your compiler with a single file argument. The commands used for each allowed target can be seen in this table:

invocation back-end
java -cp mjc.jar:lib/X.jar:lib/Y.jar:... mjc.JVMMain /some/path/foo.java -S JVM compiler
java -cp mjc.jar:lib/X.jar:lib/Y.jar:... mjc.X86Main /some/path/foo.java -S x86-64 compiler
java -cp mjc.jar:lib/X.jar:lib/Y.jar:... mjc.PPCMain /some/path/foo.java -S ppc32 compiler
java -cp mjc.jar:lib/X.jar:lib/Y.jar:... mjc.SPARCMain /some/path/foo.java -S sparc32 compiler
java -cp mjc.jar:lib/X.jar:lib/Y.jar:... mjc.ARMMain /some/path/foo.java -S arm compiler
java -cp mjc.jar:lib/X.jar:lib/Y.jar:... mjc.MIPSMain /some/path/foo.java -S mips32 compiler

X.jar, Y.jar are files from your lib subdirectory. The test system will assume all .jar files from the lib subdirectory are needed.

(For compilers not written in Java, the generated compiler should be called mjc. The invocation changes accordingly, TIGRIS will pass the target (JVM, ARM, etc) as the first argument. A typical invocation will look like this: mjc JVM /some/path/foo.java -S

The -S option tells your compiler to generate assembly code (for a JVM backend jasmin code) where the filename should be foo.s if the source code was in /some/path/foo.java, i.e., just keep the basename for the output, i.e., place the file in ".".

For the target JVM, you actually need to generate one file per class, to adhere to the JVM execution environment. Use either .s or .j as extension, TIGRIS handles both. The files need to be placed in ".".

Generating anything but the assembly code (in one file for non-JVM targets, or one file per class for the JVM target) will result in errors. No directories, no class files! (But without -S, your compiler should perhaps generate class files, object files, or whatnot; TIGRIS will not explore the possibilities.)

If the test case is valid, your compiler should exit with System.exit(0). If the test case is invalid, your compiler should exit with System.exit(1). The validity criteria should be syntax, typing, etc.

A note on security

Your compiler is run on a Unix-type system, just like any process. It is not a limited environment. The system is however run using a throw-away disk image. Misbehaved processes should just lead to judging failure, but risk to slow down TIGRIS for other students.

Processes programmed for intentional misbehaviour will not be tolerated.

Acknowledgements

TIGRIS was developed by Torbjörn Granlund. The Xen based re-installation of TIGRIS for komp14 was done in February 2014 in close cooperation with Ragnar Sundblad of the CSC System Group.

TIGRIS innards and current program versions

This section is mainly for the curious. But if you experience a problem with TIGRIS which you cannot reproduce, the versions listed here might provide some guidance.

Incoming submissions are received by the master system and queued, and the same system then asynchronously takes judging tasks. An incoming compiler is first sent to a compilation virtual system where the compiler is compiled and the newly compiled compiler is run over the testsuite. The result is assembly code for the execute and nonexecute test categories, one set for each supported target. Each of the assembly code sets is then sent to an appropriate execute virtual system.

Each judging phase uses a pristine system disk image. This is part of the Tigris security model.

The virtual systems use a combination of Xen and Qemu. Xen is used whenever X86 is usable, Qemu is used for ARM, PPC, MIPS. Previous versions of TIGRIS used Qemu for everything, and the switch to Xen as well as other improvements made TIGRIS 20 times faster.

The master system runs NetBSD 6.1 with Xen 4.2.3. It employs Qemu 1.7.0. The hardware is a low-end rackmount system with an Intel Xeon E31220 CPU running at 3.10GHz, and two mirrored S3500 SSD drives.

Compilers written in Java, C, and C++ are handled by comp1 running NetBSD 6.1.3. This system's gcc invokes gcc 4.5.3. This system's gcc48 invokes gcc 4.8.2. This system's ant command invokes apache ant 1.9.2.

Compilers written in Haskell are handled by comp2 running FreeBSD 9.2. This system's ghc command invokes Glasgow Haskell compiler version 7.6.3.

X86 assembling/linking and target execution is handled by exec running NetBSD 6.1.3. This system's as and ld commands come from binutils 2.21.1.

ARM assembling/linking and target execution is handled by execarm running GNU/Linux Debian 7.3 under Qemu.

MIPS assembling/linking and target execution is handled by execmips running GNU/Linux Debian 7.3 under Qemu.

PPC assembling/linking and target execution is handled by execppc running GNU/Linux Debian 7.4 under Qemu.

SPARC assembling/linking and target execution is handled by execsparc running NetBSD 6.1.3 under Qemu.







Kattis meows, TIGRIS roars
Copyright © Published by: Torbjörn Granlund <www-kurs@csc.kth.se>
Updated 2014-05-30