Start running GEMCLIM version 3.3.3.1 on guillimin

To run GEM one needs appropriate executables as well as a set of scripts and configuration files.
You will find some examples at the end of this page.

Content

1)-3)	Set up the environment
4)	The executables
5)	The scripts
6)	The configuration files
7)	Launching the model
8)	What happens while the model is running
10)	What to do with the model output

Examples:
1) Create your own executables
2) Set up and start a model run
3) Modify some decks

1) Set up the "SSM environment"

Click here to learn about the SSM environment and follow the setup.
You only need to do this step once!

2) Set up the "CRCM5 environment"

Youhave to create a few directories and links for CRCM5
Click here and follow the setup.
You only need to do this step once!

3) Set the model environment

To have acccess to the model you need to set certain environment variables and make additions to the PATH.
You do this by executing the following command:

   . s.ssmuse.dot gemclim_3.3.3.1-dev

Since I find this rather long and difficult to keep in mind, I created an alias for everybody to use:

    alias 3331='. s.ssmuse.dot gemclim_3.3.3.1-dev'

So all you have to type now is

    3331

and you model environment is set.
You need to set the model environment once in every window in which you want to have access (run/compile/...) the model.
This command also sets the environment variable $gemclim which contains the directory name under which you can find all model routines and scripts.

4) The executables

Usually GEM(CLIM) needs two absolutes: one for the entry ('...ntr...') and one for the model ('...dm...').

Their names are:
'maingemclimntr_${ARCH}_modelversion.Abs' for the entry,
'maingemclimdm_${ARCH}_modelversion.Abs' for the model.

To find out about how to create the absolutes you can either have a look at the general introduction to GEMDM web page or follow the little example below.

In climate mode the absolutes always need to be on the machine from which the model gets launched.

5) The scripts

Usually one can use the scripts from the environment.
Once the model environment is set, you can find them in ${gemclim}/bin

In case you want to modify and use some of these scripts you must put your (modified) copies of the scripts into the '~/modeles/GEMCLIM/v_3.3.3.1/bin' directory.

You find more detailed explanations about the scripts here. But you really do not have to look at them for now!!!

6) The configuration files

GEMCLIM needs three configuration files: 'configexp.dot.cfg', 'gemclim_settings.nml' and 'outcfg.out'

configexp.dot.cfg: file to control where to send the batch run, how much cpu time allowed, which machine to run the job on, which model version to use, which machine to do the post processing on, where to place the stdout (listings) and the model output, etc.

Click on the following links to get more information on the general variables and on the special climate variables.

This files is neither needed nor used when GEMCLIM is run interactively.

gemclim_settings.nml: This file contains all the namelists to control the model grid, the different schemes and parameters for the entry program (gemclimntr) and the main program (gemclimdm).

Click on the following links to get more information on the general namelist variables and on the physics namelist variables.

outcfg.out: file to control the RPN standard file output from the run such as frequency of output, which fields, at what levels, etc.. For output intervals ('steps=#,hour,<start,end,inc>';) the start and end hour (or step) will get updated automatically. Only for the first job in a run the start hour (or step) has to be set by the user!

Click on the following link to get more information on the outcfg.out file variables.

7) Launching the model

If you run the model in LAM mode you need to start it from the machine on which you want to do the entry otherwise go on the machine on which you want to run the model.
Go in the directory where you have your config files.

Set the model environment:
3331
Launch the model:
Um_lance

Or have a look at the example below.

8) What happens while the model is running

You can also have a look at the flowchart showing all major jobs, which listings they produce and where their output goes.

During its execution, 'Um_lance' messages will appear on your screen telling you how the config files got updated, that the executables got copied in your execution directory on $BACKEND_mach and finally that the entry was submitted.
At that point you should see the entry running (or being queued). The entry runs on only 1 cpu and takes only a few minutes if the model is not run in LAM mode; but it can take up to several minutes per month in LAM mode.
After the entry, the model will be started on $BACKEND_mach usually running on multiple cores.
While the model is running, the model output will be written to the directory '${CLIMAT_outrep}/experiment_name/current_last_step'.

        There are four types of output files:

        dp... : dynamics on pressure level
        dm... : dynamics on model levels
        pm... : physics on model levels
        pp... : physics on pressure levels

The post processing always gets started at the end of a job/month. This means that the model output will get moved from the output-directory 'current_last_step' into 'last_step_xxx' (actually the directory just gets renamed).
The post processing job (..._PP_...) is a little 1 cpu job which determines what kind of post processing is to be done, i.e. saving of time series, the usual post processing, diagnostics. If no file transfer or saving of time series needs to be done, this job will be run as part of the model job.
Then the diagnostic and reassemble job will start. It can be run on multiple cpu's, but not on more than tiles got written by the model. The number of cpu's used needs to be specified in ${CLIMAT_pp_cpus}.

First some diagnostics will be done if required ('CLIMAT_diagnos=1;').

The diagnostic job creates the monthly means, (time series,) and variances for each individual month in this interval.

Then the different tiles will get reassembled (using 'd2z').
After this all model output plus the listings and jobs/scripts (created during the run) from this job are collected and archived on 'CLIMAT_arch_mach' in 'aCLIMAT_rchdir'.
In a seperate job, at the end of each model job, the restart files are also archived and transfered to 'CLIMAT_arch_mach' into 'CLIMAT_archdir'. The name of the archive will be '${UM_EXEC_exp}step${laststep}.ca.gz'. If the run continues, the "old" restart files will get erased from ${BACKEND_mach}. If the run is finished, they will stay on ${BACKEND_mach}.

Note: The model does not depend on post processing and diagnostics. Even if one of these jobs aborts, the model will continue running and at the end of a job (in case the run continues) submit the next job automatically.

10) What to do with the model output

There are several ways to look at the model output, which is in the RPN standard file format, and to modify it.
Have a look at this web page for some examples.

Examples

We assume you did already points 1) and 2) of the section above.

1) Create your own executables:

Go on the machine for which you want to create the absolutes into the directory in which you would have your own decks (preferably under your ${HOME}!!!).
For now let's call the directory 'Example'. Once you create executables for a certain project, please choose a better name.
mkdir -p ~/gemclim/v_3.3.3.1/Abs/Example cd ~/gemclim/v_3.3.3.1/Abs/Example
Set the model environment: 3331
Set up an active experiment:

Create the file '.exper_cour' which will contain the paths of the RCS's (archived model decks):

cat > .exper_cour << EOF RCSPATH="\${gemclim}/RCS_DYN \${gemclim}/RCS_PHY \${gemclim}/RCS_CLASS" RCSBASE="base" EOF
You can also copy the file '.exper_cour' from another directory.
To avoid limited disc space problems you may want to create some soft links.
malib${ARCH} is the directory to hold future object files created from recompiling new or modified routines (by you) from the GEMCLIM library. It is also recommended to create soft links for the GEM* binaries (main*.Abs) as they are also written in the working directory.
So create a directory under your project space:
    For people working for or having their account via Pierre Gauthier your project space is:
PROJECT_SPACE=/sf1/escer/gauthier/${USER}     If you are with Laxmi Sushama your project space is:
PROJECT_SPACE=/gs/project/ugh-612-aa/${USER}     If you are with René Laprise or Julie Theriault your project space is:
PROJECT_SPACE=/gs/project/bgx-133-aa/${USER}      If you are with another PI, ask your PI for your project space.

    Create the directory:
mkdir -p ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/malib${ARCH}
    And create the 3 links:
ln -s ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/malib${ARCH} ln -s ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/maingemclimntr_${BASE_ARCH}_${MODEL_VERSION}.Abs ln -s ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/maingemclimdm_${BASE_ARCH}_${MODEL_VERSION}.Abs
    Touch executables and make them executable:touch    ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/maingemclimntr_${BASE_ARCH}_${MODEL_VERSION}.Abs chmod +x ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/maingemclimntr_${BASE_ARCH}_${MODEL_VERSION}.Abs touch    ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/maingemclimdm_${BASE_ARCH}_${MODEL_VERSION}.Abs chmod +x ${PROJECT_SPACE}/Abs/v_3.3.3.1/Example/maingemclimdm_${BASE_ARCH}_${MODEL_VERSION}.Abs
Create the makefile: r.make_exp This command creates the make file 'Makefile', the file 'arbre_de_dependance' containing a list of all the functions/decks with their included comdecks as well as the file 'make_cdk' containing a list of all the comdecks with the names of the functions/decks they are included in.
Build the executables: make gemclim
If you wish to make modifications to the GEMCLIM source code, see the expanded description below.

2) Set up and start a model run

Build and enter a configuration directory.
For now let's call the directory 'Example'. Once you create executables for a certain project, please choose a better name. mkdir -p ~/gemclim/v_3.3.3.1/Configs/Example cd ~/gemclim/v_3.3.3.1/Configs/Example
Copy the example configuration scripts into the current directory: rsync -rvtp ${gemclim}/Configs/LAM_60x60 ./.
Go into your new config file directory:cd LAM_60x60
Launch the model run: Um_lance

While the model is running, all listing go into ~/listings/${TRUE_HOST}.
But at the end of each job all listings get zipped and transfered to arch_mach in your archdir as specified in your 'configexp.dot.cfg'. Unless, of course, something went wrong...

3) Check your jobs

While you job is running or queued, you can check it with
qsIf you want to kill a job, you have to use 'qdel' followed by the JobID you see with 'qs'.

4) Modify some decks:

You will find all the RCS model decks of the dynamics in ${gemclim}/RCS_DYN and of the physics in ${gemclim}/RCS_PHY.

Return to your experiment directory:cd ~/gemclim/v_3.3.3.1/Abs/Example/LAM_60x60
In case you did not do that already you need to set the model environment with: 3331This will set the environment variable $gemclim to the model path.
Get an RCS repository listing for the dynamics segment of the model: ls ${gemclim}/RCS_DYN
Get an RCS repository listing for the physics segment of the model: ls ${gemclim}/RCS_PHY
Get an RCS repository listing for the CLASS segment of the model: ls ${gemclim}/RCS_CLASS
Extract the function or comdeck you want, for example the CCCma radiation driver source from the physics RCS repository.
Never put here the ',v' at the end of the file name! omd_exp cccmarad.ftn

Use your favorite text editor to have a look at (and modify if you'd like) the source for cccmarad.ftn.
In case you include a new comdeck or change function calls you need to recreate the make file: r.make_exp
Compile the modified (or unmodified) subprogram (create *.f and *.o): make cccmarad.o If you modify a comdeck or want to compile all the routines you have in the directory you should instead use: make objloc
(Optional) Directory cleanup:
After having compiled a function or comdeck you will also have all their included decks and '*.f' in your directory. They all will have the permission '-r--r--r--'. To erase them you can use: 'make clean'
This will erase all decks with the permission '-r--r--r--'. So be careful!!! If you write-protect a deck to not accidentally erase it 'make clean' WILL ERASE it!
make clean
Recompile the whole model to use your newly-compiled subprogram: make gemclim To only recompile the model you can also use: make gemclimdm To only recompile the entry you can also use: make gemclimntr
Return to your benchmark configuration directory: cd ../../Configs/Example
And launch you model:Um_lance

Author: Katja Winger
Last update: February 2014