Start running GEMCLIM

To run GEM one needs appropriate executables as well as a set of scripts and configuration files.
You will find some examples at the end of this page.

Content

1)-3)	Set up the environment
4)	The executables
5)	The scripts
6)	The configuration files
7)	Launching the model
8)	What happens while the model is running
10)	What to do with the model output

Examples:
1) Create your own executables
2) Set up and start a model run
3) Modify some decks

1) Create the directories 'MODEL_EXEC_RUN' and 'listings'

You will have to create a few directories before you can start running the model.
You only need to do this step once!

You need to create the directories 'gemclim' and 'listings' directly in your ${HOME}. In each of these two directories you need to create soft links with the names of the machines you will use to run the model. These links should point to directories on a disk on which you have enough disc space.

First create the directories 'MODEL_EXEC_RUN' and 'listings' in your ${HOME}:

mkdir
~/gemclim

mkdir ~/listings

mkdir ~/MODEL_EXEC_RUN

Now create a directory under which the model will be running. You only need to create a directory for the machine, 'mach', on which you will run the model.

Running on marvin:

On marvin you will be running on the machine 'mach=headnode':mkdir -p /local/fiber1/${USER}/gemclim/EXECDIR ln -s /local/fiber1/${USER}/gemclim/EXECDIR ~/MODEL_EXEC_RUN/headnode

Running on st1/st2/st3 (development):

On st1/st2/st3 you will be running on the machine 'mach=st1':mkdir -p /st1_fs9/${USER}/gemclim/EXECDIR ln -s /st1_fs9/${USER}/gemclim/EXECDIR ~/MODEL_EXEC_RUN/st1

Running in Dorval:

In Dorval you could run on 'maia', 'naos', a Linux machine or cluster, alef... For now let's use 'maia':mach=maiaGo on 'maia'! The disk on which you have a lot of temporary disk space will probably be either be '/fs/dev/mrb/02' or '/fs/dev/mrb/03'. Find the disk you have a directory on, i.e.: /fs/dev/mrb/03/armn/${USER}
mkdir -p /fs/dev/mrb/03/armn/${USER}/gemclim/EXECDIR ln -s /fs/dev/mrb/03/armn/${USER}/gemclim/EXECDIR ~/gemclim/${mach}

Now create directories for the model, script and transfer listings during run time.
You need to create subdirectories for all the machines you will be using to run the entry, the model and the post processing and for the machine on which you will finally store the model output.

Running on marvin:

On marvin you will run everything on 'headnode', although the model output might be stored on another machine, i.e. st1/st2/st3mkdir -p /local/sata2/${USER}/gemclim/listings ln -s /local/sata2/${USER}/gemclim/listings ~/listings/headnode

Running on st1/st2/st3:

On st1/st2/st3 you will run everything on 'st1' mkdir -p /st1_fs9/${USER}/gemclim/listings ln -s /st1_fs9/${USER}/gemclim/listings ~/listings/st1

Running in Dorval:

In Dorval you could, in LAM mode, run the entry on a Linux machine, i.e. 'copernic', run the model and post processing on an AIX cluster, i.e. 'maia', and then store the output on another Linux machine, i.e. 'kepler'.
'alef' is the machine handling the transfer of the model in and output to/from maia from/to the machine on which you process or store your data. Therefore you also need to create a listings directory for 'alef'.

This means you have to create directories for:
   copernic, maia, kepler and alef!
Go on the, in this case, four machines into the directory '~/listings' and create links like i.e.this one:
   cd ~/listings
   ln -s /fs/mrb/03/armn/armnkwi/listings maia

At the end of each job all listings get zipped and transfered to arch_mach in your archdir as specified in your 'configexp.dot.cfg'.

2) Export RDIAG and EDITFST in your '.profile_usr'

Running in Dorval:

When running in Dorval export these two variables in you '.profile_usr':
export RDIAG=/usr/local/env/armnlib/scripts/latest.r.diag
export EDITFST=editfst_6.02

3) Set the model environment

Every time you start the model or any of the scripts you have to set the model environment:

. r.sm.dot gemclim 3.3.2

Note that this adds two (resp. three or four if you have scripts or bin directories under ~/modeles/GEMCLIM/v_3.3.2) directories to your environment variable ${PATH} and also sets up a new environment variable 'gemclim' which points to the directory under which you find the official model environment (source decks, scripts, etc.).

4) The executables

Usually GEM(CLIM) needs two absolutes: one for the entry ('...ntr...') and one for the model ('...dm...').

Their names are:
'maingemclimntr_${ARCH}_modelversion.Abs' for the entry,
'maingemclimdm_${ARCH}_modelversion.Abs' for the model.

To find out about how to create the absolutes you can either have a look at the general introduction to GEMDM web page or follow the little example below.

In climate mode the absolutes always need to be on the machine from which the model gets launched.

Running in Dorval:

If the model is run in LAM mode the entry is usually ran on another machine with another architecture than the model. In this case there need to be an absolute for the entry with the architecture of the entry machine and an absolute for entry and model with the architecture of the machine were the model will be running. All three absolutes have to be on the machine from which the model will be launched.

5) The scripts

Usually one can use the scripts from the environment.

But you might also want to ask either Bernard Dugas (Bernard.Dugas@ec.gc.ca) or Katja Winger (Katja.Winger@ec.gc.ca) if there is a newer updated version of the scripts.

There are two types of scripts:

version-dependent, that change with model versions: ${gemclim}/scripts/* and
version-invariant, that do not change: ${ARMNLIB}/scripts/Climat_*

In case you want to modify and use some of these scripts you must put your (modified) copies of any version-dependent scripts into the '~/modeles/GEMCLIM/v_3.3.2/scripts' directory and any version-invariant scripts into the '~/ovbin' directory.

After having created the directory '~/modeles/GEMCLIM/v_3.3.2/scripts' you need to set the model environment again so that this directory gets added to your 'PATH'!

You find more detailed explanations about the scripts here.

6) The configuration files

GEMCLIM needs three configuration files: 'configexp.dot.cfg', 'gemclim_settings.nml' and 'outcfg.out'

configexp.dot.cfg: file to control where to send the batch run, how much cpu time allowed, which machine to run the job on, which model version to use, which machine to do the post processing on, where to place the stdout (listings) and the model output, etc.

Click on the following links to get more information on the general variables and on the special climate variables.

This files is neither needed nor used when GEMCLIM is run interactively.

gemclim_settings.nml: This file contains all the namelists to control the model grid, the different schemes and parameters for the entry program (gemclimntr) and the main program (gemclimdm).

Click on the following links to get more information on the general namelist variables and on the physics namelist variables.

outcfg.out: file to control the RPN standard file output from the run such as frequency of output, which fields, at what levels, etc.. For output intervals ('steps=#,hour,<start,end,inc>';) the start and end hour (or step) will get updated automatically. Only for the first job in a run the start hour (or step) has to be set by the user!

Click on the following link to get more information on the outcfg.out file variables.

7) Launching the model

If you run the model in LAM mode you need to start it from the machine on which you want to do the entry otherwise go on the machine on which you want to run the model.
Go in the directory where you have your config files.
Set the model environment:
. r.sm.dot gemclim 3.3.2
Launch the model:
Um_lance

Or have a look at the example below.

8) What happens while the model is running

You can also have a look at the flowchart showing all major jobs, which listings they produce and where their output goes.

During its execution, 'Um_lance' messages will appear on your screen telling you how the config files got updated, that the executables got copied in your execution directory on $mach and finally that the entry was submitted.
At that point you should see the entry running (or being queued). The entry runs on only 1 cpu and takes only a few minutes if the model is not run in LAM mode; but it can take up to several minutes per month in LAM mode.
After the entry, the model will be started on $mach usually running on multiple cores.
The model output will be written to the directory '${outrep}/current_last_step'. Each tile will write to its own sub-directory, 00-00, 00-01, ... The domain geometry (actual tile number and configuration) is specified in the in the namelist 'ptopo' of 'gemclim_settings.nml'.

        There are four types of output files:

        dp... : dynamics on pressure level
        dm... : dynamics on model levels
        pm... : physics on model levels
        pp... : physics on pressure levels

The post processing always gets started at the end of a job/month. This means that the model output will get moved from the output-directory 'current_last_step' into 'last_step_xxx' (actually the directory just gets renamed).
Running in Dorval:

The post processing can also get started while the model is still running. This can be useful when there is not enough disk space since this early post processing reduces the output when ${strip_phy}=1.
The post processing is then started every 'cleanup' days (this number is specified by the variable 'climat_cleanup' in your 'configexp.dot.cfg').

Running in Dorval:

When the post processing is done on another machine (${mach} not equal ${xfer}) a job (..._FT_...) will be send to alef which moves the files from ${mach} to ${xfer}.

The post processing job (..._PP_...) is a little 1 cpu job which determines what kind of post processing is to be done, i.e. saving of time series, the usual post processing, diagnostics. If no file transfer or saving of time series needs to be done, this job will be run as part of the model job.
Then the diagnostic and reassemble job will start. It can be run on multiple cpu's, but not on more than tiles got written by the model. The number of cpu's used needs to be specified in ${climat_pp_cpus}.

First some diagnostics will be done if required ('diagnos=1;').

The diagnostic job creates the monthly means, time series, and variances for each individual month in this interval.

Then the different tiles will get reassembled (using 'd2z').
After this all model output plus the listings and jobs/scripts (created during the run) from this job are collected and archived on 'arch_mach' in 'archdir'.
In a seperate job, at the end of each model job, the restart files are also archived and transfered to 'arch_mach' into 'archdir'. The name of the archive will be '${exp}step${laststep}.ca.gz'. If the run continues, the "old" restart files will get erased from ${mach}. If the run is finished, they will stay on ${mach}.

Note the model does not depend on post processing and diagnostics. Even if one of these jobs aborts, the model will continue running and at the end of a job (in case the run continues) submit the next job automatically. You will then get another directory in your ${HOME}/gem/${mach} for the new job.

10) What to do with the model output

There are several ways to look at the model output, which is in the RPN standard file format, and to modify it.
Have a look at this web page for some examples.

Examples

We assume you did already point 1) and 2) of the section above.

1) Create your own executables:

Go on the machine for which you want to create the absolutes into the directory in which you would have your own decks (preferably under your ${HOME}). mkdir -p ~/gemclim/v_3.3.2/Abs/bcmk cd ~/gemclim/v_3.3.2/Abs/bcmk
Set up an active experiment:

Create the file '.exper_cour' which will contain the paths of the RCS's (archived model decks):

cat > .exper_cour << EOF RCSPATH="\${gemclim}/RCS_DYN \${gemclim}/RCS_PHY \${gemclim}/RCS_CLASS" RCSBASE="base" EOF
You can also copy the file '.exper_cour' from another directory.

To avoid limited disc space problems you may want to create some soft links.
malib${ARCH} is the directory to hold future object files created from recompiling new or modified routines (by you) from the GEMCLIM library. It is also recommended to create soft links for the GEM* binaries (main*.Abs) as they are also written in the working directory.

Running on marvin:

absdir=/local/sata2/${USER}/gemclim/Abs/v_3.3.2/bcmk

Running on st1/st2/st3:

absdir=/st1_fs9/${USER}/gemclim/Abs/v_3.3.2/bcmk

Running in Dorval:

It is up to you where you want to put your absolutes and object files. On 'maia' you could put them i.e. here:
absdir=/fs/dev/mrb/03/armn/${USER}/ABS/v_3.3.2/bcmk

mkdir
-p ${absdir}/malib${ARCH} 

ln -s ${absdir}/malib${ARCH}

ln -s
${absdir}/maingemclimdm_${ARCH}_3.3.2.Abs

ln
-s
${absdir}/maingemclimntr_${ARCH}_3.3.2.Abs

Set the model environment: . r.sm.dot gemclim 3.3.2
Build a makefile: r.make_exp This command creates the make file 'Makefile', the file 'arbre_de_dependance' containing a list of all the functions/decks with their included comdecks as well as the file 'make_cdk' containing a list of all the comdecks with the names of the functions/decks they are included in.
Build GEMCLIM: make gemclim
If you wish to make modifications to the GEMCLIM source code, see the expanded description below.

2) Set up and start a model run

Build and enter a configuration directory: mkdir -p ~/gemclim/v_3.3.2/Configs/bcmk cd ~/gemclim/v_3.3.2/Configs/bcmk

Copy the benchmark configuration scripts into the current directory:

Running on marvin:

LAM 60x60: cp ~winger/gemclim/BCMK/v_3.3.2_clim/Configs/LAM_60x60/* .Global uniform 24x12: cp ~winger/gemclim/BCMK/v_3.3.2_clim/Configs/Global_24x12/* .

Running on st1/st2/st3:

LAM 60x60: cp ~winger/gemclim/BCMK/v_3.3.2_clim/Configs/LAM_60x60/* .Global uniform 24x12: cp ~winger/gemclim/BCMK/v_3.3.2_clim/Configs/Global_24x12/* .

Dorval:

At the moment there is no example - sorry!

Replace the model configuration user name with your own user name: perl -pi -e 's/FOO/$ENV{USER}/g' configexp.dot.cfgDorval:

Make sure all paths are set correctly!
Launch the model run: Um_lance

3) Modify some decks:

You will find all the RCS model decks of the dynamics in ${gemclim}/RCS_DYN and of the physics in ${gemclim}/RCS_PHY.

Return to your experiment directory: cd ~/gemclim/v_3.3.2/Abs/bcmk
In case you did not do that already you need to set the model environment with: . r.sm.dot gemclim 3.3.2This will set the environment variable $gemclim to the model path.
Get an RCS repository listing for the dynamics segment of the model: ls ${gemclim}/RCS_DYN
Get an RCS repository listing for the physics segment of the model: ls ${gemclim}/RCS_PHY
Extract the function or comdeck you want, i.e. the CCCma radiation driver source from the physics RCS repository: omd_exp cccmarad.ftn

Use your favorite text editor to have a look at (and modify if you'd like) the source for cccmarad.ftn.
In case you include a new comdeck or change function calls you need to recreate the make file: r.make_exp Compile the modified (or unmodified) subprogram (create *.f and *.o): make cccmarad.o If you modify a comdeck or want to compile all the routines you have in the directory you should instead use: make objloc
(Optional) Directory cleanup:
After having compiled a function or comdeck you will also have all their included decks and '*.f' in your directory. They all will have the permission '-r--r--r--'. To erase them you can use: 'make clean'
This will erase all decks with the permission '-r--r--r--'. So be careful!!! If you write-protect a deck to not accidentally erase it 'make clean' WILL ERASE it!
make clean
Recompile the whole model to use your newly-compiled subprogram: make gemclim To only recompile the model you can also use: make gemclimdm To only recompile the entry you can also use: make gemclimntr
Return to your benchmark configuration directory: cd ../../Configs/bcmk
And launch you model:Um_lance

Author: Katja Winger and Ron McTaggart-Cowan
Last update: January 2010