Start running GEM 4 and up - UQAM style

1) Set up SSM environment

Do this part only when running on Compute Canada clusters. On our UQAM servers this is already done at account creation.
You need to do part 1) only once!

Please, follow the instructions on the following wiki page:

     Set up the SSM environment

2) Set up the "GEM model environment"

a) Basic setup

On the UQAM servers this part has already been done during the creation of your account. But on Compute Canada clusters you have to execute point 1) to 4) on the following wiki page:

      https://wikiscta.scta.uqam.ca/wikiscta/index.php/Set_up_model_environment

b) Create model working directories

GEM needs a place to run and another to put its listings (log files) while the model is running. Both these places are hardcoded in the model to places under your home. But since your home only has a limited amount of space and the model needs lots of it, these two places need to be links to a place where you have more space and not directories.

First pick a place under which you want to put these model working directories.

• At UQAM: On the server on which you want to run GEM, create a directory under your assigned data space, for example:

      mkdir -p /file_system/${USER}/GEM

  You need to replace 'file_system' by the name of the file system on which you have your data space.
  

• On Compute Canada systems:
  Because of the small quota for the number of files under the RRG project spaces and because these files are rather small, these two places need to be links to your scratch space, for example:

      mkdir -p /scratch/${USER}/GEM

  Since this is "scratch" space, all files in the above directories will get deleted. But once a simulation finished there should not be any files left there. Read more about Compute Canada’s scratch policy on the web: https://docs.computecanada.ca/wiki/Scratch_purging_policy

    model_space=...

Now create the following directories/links:
(Whenever possible use copy -> paste to avoid typos!!!)

After this you should have:

    ~/MODEL_EXEC_RUN/${TRUE_HOST} -> ${model_space}/EXECDIR
    ~/listings/${TRUE_HOST} -> ${model_space}/Listings

c) Create “storage_model"

When creating your executables the object files, extracted decks and executables will be put in a directory called ${storage_model}. A link will automatically be added to this place from the directory in which you create the executables.

You have to create this directory in a place in which you have space (therefore not under your home) and export the variable “storage_model“ set to that directory.

• At UQAM:
  Create this directory under the same directory under which you created your working dircetories above:

      mkdir -p /file_system/${USER}/GEM/Storage_Model

• On Compute Canada systems:
  Because of the small quota for the number of files under the RRG project spaces and because these files are rather small, create this directory under the default project (def-professor) space. Change our variable model_space not to:

      mkdir -p ~/projects/def-professor/${USER}/GEM/Storage_Model

  You need to replace ‘professor’ by the name of you professor.
  Note: Whereas the model working directories are under your scratch space because they are not needed anymore once a simulation finished, your "storage" space must be in a permanent place.

• At UQAM:
      export storage_model=/file_system/${USER}/GEM/Storage_Model

• On Compute Canada systems:
      export storage_model=~/projects/def-professor/${USER}/GEM /Storage_Model

Add this export to these two profiles (if you do not have them, create them):

    ~/.profile.d/.interactive_profile
    ~/.profile.d/.batch_profile

and create the following symbolic link:

    ln -s ~/.profile.d/.batch_profile ~/.profile.d/.ssh_profile

d) Set allocation account name under which to run simulations

When submitting any job on Compute Canada clusters you always need to specify which allocation account should get "billed" for the resources (cores) your job is running on.
Every PI (Principle Investigator - in general your professor) has a certain amount of cores assigned to her/his group which can get used on average(!) throughout the year. They are called "core years". Every PI has a default allocation (def-...) of 50 core years and some PIs also have a research project allocation (rrg-...), which varies in size from one PI to the next and from one year to the next. To get an rrg allocation the PI has to fill out an application once a year, describing the projects for which CC resources will get used.
You can find all the allocation names under which you can run under your directory:

    ~/projects

Pick the one you want to run your simulations under and write its name into a file (which you have to create) called:

    ~/.Account

The above file must only contain the name of the allocation you want to run under and nothing else!

For example:

    $ cat ~/.Account
    def-laprise

3) Extra model scripts

You need to do part 3) only once as well - unless there is an update for the scripts.

The name of the personal scripts directory is not optional and depends on the GEM version you want to run.

For GEM 4.8.lts12 set:
    version='v_4.8.lts12.u'
For GEM 5.0 set:
    version='v_5.0-u1.rc1'
For GEM 5.1.1 set:
    version='v_5.1.1-u1'

Create a directory for all your extra or modified model scripts with:

    mkdir -p ~/modeles/GEMDM/${version}/bin

For now copy all scripts from my directory into yours, for example with:

    cp -p ~winger/modeles/GEMDM/${version}/bin_latest_version/* ~/modeles/GEMDM/${version}/bin

You are done now with the general model setup.

4) The configuration files

GEM needs four configuration files (short "config" files) to run:

• configexp.dot.cfg
• gemclim_settings.nml
• outcfg.out
• physics_input_table

When running GEM5 with CLASS/CLASSIC one extra file is needed, called:

• CLASS_input_table

Click on the link "configuration files" to learn what they are use for.

You need to create one "config" directory per simulation.
Since these config files are small and very important they should always be kept under the home directory. To make it easier to help each other I suggest you put them somewhere under:

For GEM 4.8.lts12 set:
    version='v_4.8.12'
For GEM 5.0 set:
    version='v_5.0.0'
For GEM 5.1.1 set:
    version='v_5.1.1'

    ~/gem/${version}/Configs/...

Also, to make it easier to find the config files of a simulation that is already running or even finished running, I suggest to set the basename of the "config" directory to the same name as the experiment name of the simulation. So something like:

    ~/gem/${version}/Configs/experiment1
or
    ~/gem/${version}/Configs/project1/experiment1

But please, chose better names than 'project1' or 'experiment1'!!!

The config directory needs to contain the four config files and will also be the directory from which the simulation will get launched.

5) Set the model environment

You need to set the model environment in every window from which you want to run/launch the model or in which you want to create the model executable.

To have access to the model you need to set certain environment variables and make additions to the PATH.

• at UQAM:              . s.ssmuse.dot GEM/x/4.8.lts12.u   # for GEM 4.8.lts12
                                 . s.ssmuse.dot GEM/x/5.0-u1-rc1    # for GEM 5.0
                                 . s.ssmuse.dot GEM/x/5.1.1-u1        # for GEM 5.1.1

• on CC systems:    . s.ssmuse.dot GEM/x/4.8.lts12       # for GEM 4.8.lts12
                                 . r.load.dot GEM/x/5.0                      # for GEM 5.0
                                 . r.load.dot GEM/x/5.1.1-u1             # for GEM 5.1.1

To make our life easier, I created alias for everybody to use on all servers. So instead of the commands above all you have to type now is
    4812,  500,  511  respectively.
and your model environment is set.

Among others, these commands also sets the environment variables $gemdyn and $rpnphy which contain the directory name under which you can find all models dynamic recp. physics routines

6) Launching the model

1. Go into your config directory, the directory in which you have your config files.
2. Depending on the version you want to run set the model environment with (needs to be done only once per window):
       4812 / 500 / 511
   
3. Launch the model with one of the following two commands:
   

   a)  Um_lance

   This is the basic way to submit the model. When running GEM in climate mode (UQAM style) and your simulation consists of more than one month the model will run one job per month. At the end of each month/job the next month will get submitted automatically.

   b)   Chunk_lance

   While Um_lance works well on our own UQAM servers it can cause big delays on Compute Canada clusters, because each job will be queued again. To avoid these extra queued times there is a 'Chunk_lance' will execute as many months as it can in the given wall time of the job (BACKEND_time_mod). At the end of each job the next job will get submitted automatically. While 'Chunk_lance' can be used for all types of simulations it is especially useful for longer simulations.

   When the model crashes and you want to resubmit it to continue the simulation, all you have to do is go back into you config file directory, set the model environment, and execute 'Chunk_lance' again.

   If ever you want to restart a simulation from the very beginning, you can do so by executing:

         Chunk_lance  -start

To check if your simulation is running use the command:
            qs

 You can also have a look at the example below.

7) What happens while the model is running

The model itself will run under the "execution directory" you created above, called:
    ~/MODEL_EXEC_RUN/machine_name/experiment_name

Each simulation consists of several parts which will all get started/executed/submitted automatically after the initial launch with Um_lance or Chunk_lance.

First some "preparation scripts" will get executed, preparing the execution directory as well as updating your gem_settings.nml and outcfg.out. Then the main "model" will get executed. At the end of each month, and upon request (Out3_close_interval_S) even during a month, the model will submit the "post processing". The post processing will collect and combine and archive all the direct model output, calculate monthly means and variances (if CLIMAT_diagnos=1) and archive the restart files. At the end, the "save listings" job will collect all the listings and put them in your archive (CLIMAT_archdir).

Note: The model does not depend on post processing job. Even if one of these jobs aborts, the model will continue running.

Listings

While the model is running the listings will appear under the directory:
            ~/listings/${TRUE_HOST}
They will all have ${GEM_exp} in the name.

Once a job/month finished properly they will all get archived under:
            ${CLIMAT_archdir}/Listings/listings_${GEM_exp}.zip

Model output

While the model is running the output will be under:
            ~/MODEL_EXEC_RUN/${TRUE_HOST}/${GEM_exp}/RUNMOD/output/cfg_0000/laststep_...

Whenever the model or the post processing crashes you should find the output produced so far under the above directories.

The post processing collects the model output and archives it under:
            ${CLIMAT_archdir}

Once the post processing is finished you should have the following directories in the above directory":

And then you should also have this file:
    maingemdm_Linux_x86-64.Abs  
which is a copy of the executable.

There are four types of output files:

8) What to do with the model output

9) Create your own executable

For version  5.1.1 click here!

Example 1: Run a LAM simulation with GEM5.0.0

1) Config files

First you need to create a configuration directory.

For this example create and enter the following directory:

mkdir -p ~/gem/v_5.0.0/Configs/Examples/LAM_60x60
cd ~/gem/v_5.0.0/Configs/Examples/LAM_60x60

Then copy the config files from the example directory into you config directory:

cp ~winger/gem/v_5.0.0/Configs/bcmk/LAM_60x60/* .

2) Submit simulation

Make sure you model environment is set (with command: 500).

Then submit the simulation with:

Um_lance

3) Check simulation

You can check if you are running with:

qs

If ever you need to kill a simulation you can do that with:

    qdel JobID

Click here to see what happens while the model is running.