Start running GEM 4 and up - UQAM style

1) Set up SSM environment

Do this part only when running on clusters of "The Alliance" (Compute Canada ). On our UQAM servers this is already done at account creation.
You need to do part 1) only once!

Please, follow the instructions on the following wiki page:

     Set up the SSM environment

2) Set up the "GEM model environment"

Please, follow the instructions on the following wiki page:

     Set up the GEM model environment

3) Extra model scripts

You need to do part 3) only once as well - unless there is an update for the scripts.

The name of the personal scripts directory is not optional and depends on the GEM version you want to run.

For GEM 4.8.lts12 set:
    version='v_4.8.lts12.u'
For GEM 5.0 set:
    version='v_5.0-u1.rc1'
For GEM 5.1.1 set:
    version='v_5.1.1-u1'

Create a directory for all your extra or modified model scripts with:

    mkdir -p ~/modeles/GEMDM/${version}/bin

For now copy all scripts from my directory into yours, for example with:

    cp -p ~winger/modeles/GEMDM/${version}/bin_latest_version/* ~/modeles/GEMDM/${version}/bin

You are done now with the general model setup.

4) The configuration files

GEM needs four (or five) configuration files (short: "config" files) to run:

• configexp.dot.cfg
• gemclim_settings.nml
• outcfg.out
• physics_input_table

When running GEM5 and up with CLASS/CLASSIC one extra file is needed, called:

• CLASS_input_table

Click on the link "configuration files" to learn what they are use for.

You need to create one "config" directory per simulation.
Since these config files are small and very important they should always be kept under the home directory. To make it easier to help each other I suggest you put them somewhere under:

For GEM 4.8.lts12 set:
    version='v_4.8.12'
For GEM 5.0 set:
    version='v_5.0.0'
For GEM 5.1.1 set:
    version='v_5.1.1'

    ~/gem/${version}/Configs/...

Also, to make it easier to find the config files of a simulation that is already running or even finished running, I suggest to set the basename of the "config" directory to the same name as the experiment name of the simulation. So something like:

    ~/gem/${version}/Configs/experiment1
or
    ~/gem/${version}/Configs/project1/experiment1

But please, chose better names than 'project1' or 'experiment1'!!!

The config directory needs to contain the four config files and will also be the directory from which the simulation will get launched.

5) Set the model environment

You need to set the model environment in every window from which you want to run/launch the model or in which you want to create the model executable. This is not to be confused with the initial setup. Here you just need to set which model version you want to use to get access certain environment variables and get additions to the PATH.

• at UQAM:              . s.ssmuse.dot GEM/x/4.8.lts12.u   # for GEM 4.8.lts12
                                 . s.ssmuse.dot GEM/x/5.0-u1-rc1    # for GEM 5.0
                                 . s.ssmuse.dot GEM/x/5.1.1-u1        # for GEM 5.1.1

• on CC systems:    . s.ssmuse.dot GEM/x/4.8.lts12       # for GEM 4.8.lts12
                                 . r.load.dot GEM/x/5.0                      # for GEM 5.0
                                 . r.load.dot GEM/x/5.1.1-u1             # for GEM 5.1.1

To make our life easier, I created aliases for everybody to use on all servers. Instead of the commands above all you have to type is
    4812,  500,  511  respectively.
and your model environment is set.

Among others, these commands also sets the environment variables $gemdyn and $rpnphy which contain the directory name under which you can find all models dynamic recp. physics routines

6) Launching the model

1. Go into your config directory, the directory in which you have your config files.
2. Depending on the version you want to run set the model environment with (needs to be done only once per window):
       4812 / 500 / 511
   
3. Launch the model with one of the following two commands:
   

   a)  Um_lance

   This is the basic way to submit the model. When running GEM in climate mode (UQAM style) and your simulation consists of more than one month the model will run one job per month. At the end of each month/job the next month will get submitted automatically.

   b)   Chunk_lance

   While Um_lance works well on our own UQAM servers it can cause big delays on Compute Canada clusters, because each job will be queued again. To avoid these extra queued times there is a 'Chunk_lance' will execute as many months as it can in the given wall time of the job (BACKEND_time_mod). At the end of each job the next job will get submitted automatically. While 'Chunk_lance' can be used for all types of simulations it is especially useful for longer simulations.

   When the model crashes and you want to resubmit it to continue the simulation, all you have to do is go back into you config file directory, set the model environment, and execute 'Chunk_lance' again.

   If ever you want to restart a simulation from the very beginning, you can do so by executing:

         Chunk_lance  -start

   You can find more information about Chunk_lance on our wiki by clicking 'here'.

To check if your simulation is running use the command:
            qs

 You can also have a look at the example below.

7) What happens while the model is running

The model itself will run under the "execution directory" you created above, called:
    ~/MODEL_EXEC_RUN/machine_name/experiment_name

Each simulation consists of several parts which will all get started/executed/submitted automatically after the initial launch with Um_lance or Chunk_lance.

First some "preparation scripts" will get executed, preparing the execution directory as well as updating your gem_settings.nml and outcfg.out. Then the main "model" will get executed. At the end of each month, and upon request (Out3_close_interval_S) even during a month, the model will submit the "post processing". The post processing will collect and combine and archive all the direct model output, calculate monthly means and variances (if CLIMAT_diagnos=1) and archive the restart files. At the end, the "save listings" job will collect all the listings and put them in your archive (CLIMAT_archdir).

Note: The model does not depend on post processing job. Even if one of these jobs aborts, the model will continue running.

Listings

While the model is running the listings will appear under the directory:
            ~/listings/${TRUE_HOST}
They will all have ${GEM_exp} in the name.

Once a job/month finished properly they will all get archived under:
            ${CLIMAT_archdir}/Listings/listings_${GEM_exp}.zip

Model output

While the model is running the output will be under:
            ~/MODEL_EXEC_RUN/${TRUE_HOST}/${GEM_exp}/RUNMOD/output/cfg_0000/laststep_...

Whenever the model or the post processing crashes you should find the output produced so far under the above directories.

The post processing collects the model output and archives it under:
            ${CLIMAT_archdir}

Once the post processing is finished you should have the following directories in the above directory":

And then you should also have this file:
    maingemdm_Linux_x86-64.Abs  
which is a copy of the executable.

There are four types of output files:

8) What to do with the model output

9) Create your own executable

For version  5.1.1 click here!

Example 1: Run a LAM simulation with GEM 5.1.1

1) Config files

First you need to create a configuration directory.

For this example create and enter the following directory:

mkdir -p ~/gem/v_5.1.1/Configs/Examples/LAM_80x60_ISBA
cd ~/gem/v_5.1.1/Configs/Examples/LAM_80x60_ISBA

Then copy the config files from the example directory into you config directory:

cp ~winger/gem/v_5.1.1/Configs/bcmk/LAM_80x60_ISBA/* .

2) Submit simulation

Make sure you model environment is set (with command: 500).

Then submit the simulation with:

Um_lance

3) Check simulation

You can check if you are running with:

qs

If ever you need to kill a simulation you can do that with:

    qdel JobID

Click here to see what happens while the model is running.