Content-Type: text/x-zim-wiki Wiki-Format: zim 0.26 Creation-Date: Wed, 27 May 2009 06:57:44 +0200 Modification-Date: Thu, 09 Jul 2009 15:05:50 +0200 ====== Documentation of the New Configuration Scheme ====== Created Wednesday 27 May 2009 Configuration files tell ''make'' and the ''start''/''run'' commands how to compile and run the code. Here is a sample file: ''' # Sample configuration # Global section %include os/Linux %include mpi/open-mpi # Makefile section %section Makefile %include compilers/g95 %include compilers/gcc FFLAGS = -O0 \ -g CFLAGS += -g %endsection Makefile # Runtime section %section runtime %include runtime-common local_disc = 1 %endsection runtime # End of file ''' ==== Sections ==== A configuration file consists of up to three sections. The //Makefile section// and the //runtime section// are delineated by ''%section ...'' and ''%endsection ...'' , while any statements outside a ''%section''/''%endsection'' block make up the //global section//. Statements inside a section are either //assignements//, like ''FFLAGS = -O4'', ''CFLAGS += -g'' or //include statements//, like ''%include compilers/g95''. ==== Assignments ==== A simple assignment has the form ''key = value value value...''. Whitespace around the ''='' sign is insignificant, and repeated whitespace in the value part is compressed to one space character unless it is quoted [not implemented yet]. It associates the (space-separated) values with the given key, erasing any previous association from earlier assignments to the given key. A backslash at the end of the line continues the assignment statement to the following line: ''' CFLAGS = -O3 \ -ffast-math \ -fprefetch-loop-arrays \ -I../include \ -Wall ''' An incremental assignment has the form ''key += value value value...''. It behaves like a simple assignment, except that it appends the values (with one space character as separator) to any values already associated with the key. ==== Include statements ==== Statements of the form ''%include '' are recursively replaced by the content of the file ''.conf''. If the '''' starts with “''./''”, the search path for the included file contains just the directory where the current file (containing the ''%include'' statement) is located. Otherwise, the search path used for finding that file consists of * the directory ''${HOME}/.pencil-config'' if it exists; * the directory ''${PENCIL_HOME}/config'' . Thus, a file ''${PENCIL_HOME}/config/os/Linux'' can include ''${PENCIL_HOME}/config/os/Unix'' with either of ''' %include os/Unix # absolute file name, rooted at ${PENCIL_HOME}/config %include ./Unix # relative file name ''' ==== Finding the config file for your computer ==== === Determining the host ID === When you run ''make'' or one of ''start.sh'', ''run.csh'', ''start_run.sh'', //configure// [???] first assigns a host id to the computer it is running on. For computers with a permanent (global) IP address, this will normally just be the fully-qualified domain name, like ''workhorse.pencil.org''. However, when submitting MPI jobs to dedicated compute clusters, this will no longer work, since the IP addresses of the cluster nodes are in the private subnet range and are thus no longer globally unique. Nor are host names, which are often of the form //node0//, //node1// or //node000//, //node001//, etc. For those cases, //configure// tries to construct a unique ID using other pieces of information. The algorithm for determining the host ID is as follows. 1. If the file ''./.pencil/host-ID'' exists, its first line is used as host ID. 2. If the file ''~/.pencil/host-ID'' exists, its first line is used as host ID. 3. If the IP number of the current computer is not in the private range and it is possible to determine its fully-qualified host name (i.e. the host and domain name), then this is used as host ID. 4. [Describe some desperate measures, generating a host ID from the ''uname'', ''/etc/issue'', etc. ''uname'' will be needed anyway for the `default Linux' setup and the like.] If no configuration file for that host ID is found, the output from “''uname -s''” is tried as host ID. If still no configuration file for that host ID is found, the host ID “''default''” is tried. If still no configuration is found, //configure// aborts with an error message. Along the way, //configure// indicates which host IDs were tried (by default, or only with ''-v''?). \footnote{ Note that the files in 1. and 2. can be soft links, which could be used to produce different host IDs on different NFS clients if ''~'' or ''.' is shared via NFS.} === Locating the config file === The host ID is then used to find the appropriate config file for the given computer (which will include other config files as needed). E.g. if the host ID is ''workhorse.pencil.org'', //configure// will look for a file ''workhorse.pencil.org'' in all directories below 1. ''~/.pencil/config/hosts'' 2. ''${PENCIL_HOME}/config/hosts'' ===== Design decisions for discussion ===== [] Run pencil consistency check by default (''pc_run'' requires extra flag to suppress consistency check). //Rationale:// Otherwise, we will hardly ever see the pencil-check output, because ''auto-test'' suppresses output from successful tests and in case of failing tests, we are just interested in the error message. [] Command-line interface of ''pc_run'': ''' pc_run pc_run start pc_run run pc_run start run run run pc_run start run^3 pc_run ^3 pc_run stop pc_run reload ''' [In particular: get rid of non-intuitive ''start_run.csh'' that has long ago stopped to start and run] [] The ''./runtime/'' directory is strictly ephemeral: [] is cleaned each time ''pc_run'' starts; [] is ignored by SVN; [] is the place for ''STOP'', ''RELOAD'', ''RELOAD_ALWAYS'', ''RESUBMIT'', ''REINIT'', ''KEEP_RELOADING'', ''LOCK'' [] is the place for ''NO_PENCIL_CONSISTENCY_CHECK'' [] is //not// the place for ''NEVERLOCK'' [] What happens to ''COLOR'', ''NOERASE'', ''SGIFIX''? [] Possible alternative: All control files are moved to ''./runtime/''. ⊕ All control files in one place ⊖ Mix of permanent (SVN-controlled) and ephemeral control files in one directory: More complicated SVN setup; only insiders know whether control file is permanent or not [] Sources of information for the host ID * Try permanent IP? [Problem: DNS resolution takes a long time to fail when offline]