Vgrid/dev corner

The Developer's Corner section of the Vertical Grid Descriptor module documentation is intended for developers working within the module itself. Users of the package should not generally need to consult this document.

1 SVN Access

To check out a working copy of the Vertical Grid Descriptor module, use the command:

svn co svn://mrbsvn/others/vcoord

This will create a vcoord subdirectory in your current directory and will fill it with a working copy of the latest version of the Vertical Grid Descriptor package.

If you do not want to get all the tag versions but only the trunk, use the command:

svn co svn://mrbsvn/others/vcoord/trunk

If you want to get a specific tag versions, use the command:

svn co svn://mrbsvn/others/vcoord/tags/release-4.3.0

For more information about subversion (svn) commands, see the official subversion documentation.

2 Building the Package

To build the Vertical Grid Descriptors module, simply type change directory into the package and type

 gmake vgrid_descriptors.o

This will generate both the library (lib/libdescrip.a) and the Fortran module file (lib/vgrid_descriptors.mod). In order to make use of the new builds, make sure that the library is available in the library or object linking path, and that vgrid_descriptors.mod is available in the include path.

3 Using the Examples

A set of examples are provided with the package to help users become familiar with the Vertical Grid Descriptors module. These examples are available under the examples/ subdirectory, and can be built either individually or using the default make target. The examples use data provided with the release, and can be run in-place after compiling and linking. Note that the Vertical Grid Descriptors module itself should be built prior to attempting to compile the examples (see above for information about building the Vertical Grid Descriptors module).

4 Running the Unit Tests

The Vertical Grid Descriptors module is developed using Test Driven Development (TDD). This means that a unit test is built before any modifications to the module are made. Presumably, this test will fail initially since the desired feature has not yet been implemented. The developer codes until the unit tests passes, then refactors and stops. However, before committing changes, all unit tests should be run to ensure that no problems have been created by the modifications (regression testing). All tests are contained in the subdirectory tests/, with the source for each test residing in tests/src. Simply adding new source code in this directory is enough to register a new test. To compile and run all tests, enter the tests/ subdirectory and run the command

 make tests

To compile and run only a subset of the tests, identify them individually using the syntax

 make tests ONLY='get_ip1set_value get_nonalpha'

Here, the list provided to the ONLY= argument can contain any number of the unit tests contained in the tests/src subdirectory.

Data from all available architectures should be tested as part of each set of unit tests, since all data files should be cross-compatible (i.e. vertical descriptors generated on one architecture should behave identically to those produced on another architecture). To run the tests on data generated by another architecture, use the DATA_ARCH= argument to the make command using the syntax

 make tests DATA_ARCH=AIX

This example uses AIX-generated data for the tests regardless of the architecture of the current platform. The DATA_ARCH= argument can be combined with the ONLY= argument. To run the tests on data files generated by all architectures, use the command

 make alltests

5 Creating an SSM Release

Before creating an SSM Release make sure that You have a clean package by doing the following:

cd in the head directory of the package where the directory bin, lib, ssm ... are.
svn update
if there are unexpected updates, review them
make sure to run the unit tests on all architectures and compilers
make sure to run the tests on the binary (bin/run_test_applications.ksh) on all architectures and compilers

5.1 Style ". ssmuse-sh" with Build Helper (prefered method)

• Usage of Build Helpe

• • setup

To run the Build Helper make sure you have the following line in your .profile. Visit the official Build Helper site to get the latest recommended version.

. ssmuse-sh -p hpcs/tools/master/bh_0.17_all

• • Command line

Then adjust the following lines in the bh_vgrid.py python script:

environ["BH_PACKAGE_VERSION"] = "5.0.4"
environ["BH_PULL_SOURCE"] = "/users/dor/afsg/apm/data/vcoord/tags/release-${BH_PACKAGE_VERSION}"
b.shell("""export RMN=rmn_015""", environ)
environ["DISTINATION_MACH"] = "pollux"
environ["DISTINATION_DIR"] = "/users/dor/afsg/apm/ords/vgrid/${BH_PACKAGE_VERSION}"

Then in the home directory of the vgrid directory, execute the following command:

./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m intel -w /users/dor/afsg/apm/ords/vgrid/tempo_intel_${$}

#To redo one step of the building process, e.g. test do:
./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m intel -s test-test -w /users/dor/afsg/apm/ords/vgrid/tempo_intel_${$}
#
#To redo from start to test
./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m intel -s -test -w /users/dor/afsg/apm/ords/vgrid/tempo_intel_${$}
#
#To redo from make to test
./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m intel -s make-test -w /users/dor/afsg/apm/ords/vgrid/tempo_intel_${$}
#
#To redo the ssm package
./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m intel -s package -w /users/dor/afsg/apm/ords/vgrid/tempo_intel_${$}

Run a similar command to all platform and compilers

./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m intel
./bh_vgrid.py -p ubuntu-12.04-amd64-64 -m intel
./bh_vgrid.py -p ubuntu-10.04-amd64-64 -m pgi14
./bh_vgrid.py -p ubuntu-12.04-amd64-64 -m pgi14
./bh_vgrid.py -p aix-7.1-ppc7-64       -m xlf13

If there is a problem finding automatically a machine for a given platform (which is the case for ubuntu-12.04-amd64-64), log-on to the machine having the desired platform (arxt02 for ubuntu-12.04-amd64-64) and run the command with the --local flag, e.g. Note : for ubuntu-12.04-amd64-64 environment . /ssm/net/hpcs/shortcuts/get_ordenv.sh 20140730 is need.

./bh_vgrid.py -p ubuntu-12.04-amd64-64 -m intel --local -w /path/to/work/directory

• • Pitfall

• • • If you use the --local flag calling bh_vgrid.py make sure to use the -w argument with a path to data space. Otherwise, the work directory will be created where you launch the command and the contend of the current directory will be copied there. Redoing this will generate multiple copy of the directory and the task wil take more time.

• • • Make sure that all commands/scripts called Makefiles produce a signal that the error trapping will catch, otherwise Build Helper will produce a succesfull report. Always inspect the Build Helper log to find out untapped error especially when new features in building/testing are developed.

• • Exemple work flow

# cd to working directory on data
cd /data/ords/afsg/afsgapm/vgrid/5.3.0-a1
/users/dor/afsg/apm/data/vcoord/tags/release-5.3.0-a1/bh_vgrid.py -m xlf13 -p aix-7.1-ppc7-64
/users/dor/afsg/apm/data/vcoord/tags/release-5.3.0-a1/bh_vgrid.py -m intel -p ubuntu-10.04-amd64-64
/users/dor/afsg/apm/data/vcoord/tags/release-5.3.0-a1/bh_vgrid.py -m pgi14 -p ubuntu-10.04-amd64-64
# At the time of writing these lines, there was no support to ubuntu-12.04-amd64-64, so we run it on local machine (that must be a ubuntu-12.04-amd64-64 platform)
/users/dor/afsg/apm/data/vcoord/tags/release-5.3.0-a1/bh_vgrid.py -m intel -p ubuntu-12.04-amd64-64 --local
/users/dor/afsg/apm/data/vcoord/tags/release-5.3.0-a1/bh_vgrid.py -m pgi14 -p ubuntu-12.04-amd64-64 --local
# Check email or log file in launching directory, fix error ad rerun until succes.

5.2 Style ". ssmuse-sh"

From version 2.7.0 and up, the Vertical Grid Descriptor package is release as an SSM package of style ". ssmuse-sh", for older version see below. The top level of the Vertical Grid Descriptor package contains an ssm/ subdirectory, in which a Makefile is designed to simplify SSM packaging. The package configuration file ssm/config/config.mk should first be modified to update the release version number. Since the resulting SSM package is valid for the selected compiler and plaform, the package creation will have to be done for all desired platform and compiler.

cd ssm
make

This produces a .ssm file of the form:

vgriddescriptors_X.Y.Z-comipler_platform

where vgriddescriptors is the package name, X.Y.Z-comipler its version, and platform the platform it is intended for. To install and publish a packege, login under afsgsrc, make sure to use ord_env, do the following command and make sure to see the following (note that the actual version of ordenv may be different than 20140207).

more .profile
. /ssm/net/hpcs/shortcuts/get_ordenv.sh 20140207

ssh afsgsrc@joule

If you cannot log on to this account, please contact Andre Plante (phone 4789).

Copy the previously prepared .ssm file to /ssm/.repo/cmdn/utils/vgrid:

mkdir -p /ssm/.repo/cmdn/utils/vgrid/my_version
cd /ssm/.repo/cmdn/utils/vgrid/my_version
cp /users/dor/afsg/apm/data/vcoord/tags/release-5.0.0/ssm/vgriddescriptors_5.0.0-pgi1401_ubuntu-10.04-amd64-64.ssm .

Note: adjust the path and name to the .ssm file to match yours.

Run the following script after adjusting the variables in CAPITAL (see notes below) to match your .ssm file.

NAME=vgriddescriptors
VERSION=5.0.0
SOURCE=/ssm/.repo/cmdn/utils/vgrid/my_version
COMPILER=pgi1401
PLATFORM=ubuntu-10.04-amd64-64
DIR=/ssm/net/cmdn/vgrid
#
#========================================================
# Nothing to change below this line
#
PACKAGE=${NAME}_${VERSION}-${COMPILER}_${PLATFORM}
#
ssm created -d ${DIR}/${VERSION}/${COMPILER}
ssm install -d ${DIR}/${VERSION}/${COMPILER} -p ${PACKAGE} -u ${SOURCE}
ssm publish -d ${DIR}/${VERSION}/${COMPILER} -p ${PACKAGE}
#
# To remove (only do this if you are certain there are no user of thsi library!)
#ssm unpublish -d ${DIR}/${VERSION}/${COMPILER} -p ${PACKAGE}
#ssm uninstall -d ${DIR}/${VERSION}/${COMPILER} -p ${PACKAGE}

From a user's point of view, the library is now available at

. ssmuse-sh -d /ssm/net/cmdn/vgrid/5.0.0/${COMPILER}

5.3 Style ". s.ssmuse.dot"

As of v3.0.0 and upto v2.7.0 the Vertical Grid Descriptor package is release as an SSM package style ". s.ssmuse.dot". More recent version are released under style ". ssmuse-sh" see above. The top level of the Vertical Grid Descriptor package contains an ssm/ subdirectory, in which a Makefile is designed to simplify SSM packaging. The package configuration file ssm/config/config.mk should first be modified to update the release version number. Since the resulting SSM package is a _multi, the next step is to generate library, module and binary files with each supported compiler on each supported platform. In this example, the current directory is the top level of the Vertical Grid Descriptor package, and the compiler is assumed to have been pre-selected with the appropriate s.ssmuse.dot command (see code below, user may also dot the dot file setup.dot).

# On all platform
. setup.dot

For the Intel compiler on Linux Linux_x86-64 do

# On all platform
. setup_intel.dot

 gmake clean
 gmake all
 cd ssm
 gmake ssm

After this set of commands is completed for each compiler on each platform, the SSM package is complete. The package resides in the ssm/ subdirectory, with the name vgriddescriptors_VERSION_multi.ssm, where VERSION corresponds to the version set in the ssm/config/config.mk file at the beginning of the release process.

Copy or move the SSM package to your module depot (assuming that the depot directory has already been created).

 cp vgriddescriptors*.ssm ~/SsmDepot

Now the remaining steps of the installation procedure are common for all SSM packages. In this example, it is assumed that the user (MY_USER in the example) has logged into the afsgsrc account (the standard CMDN installer of the package, remember that all directories and files in the afsgsrc install directories chmod'd to u-w so that chmod manipulations will be necessary); however, this set of steps could also be followed in the user's own SSM domain structure under ~/my_ssm_domains. Note that the bundle installation instructions may not be needed in the latter case.

 ### Install and publish the vgriddescriptors package ###
 package=vgriddescriptors_VERSION
 #cd ~/SsmDomains/vgrid
 cd ~/SsmBundles/vgrid
 s.ssm-creat -d $package -r ~afsgsrc/SsmDepot
 s.ssm-install ${package}_multi
 s.ssm-publish ${package}_multi
 ### Create a bundle for easy access ###
 #cd ~/SsmBundles/vgrid
 cp PREVIOUS_VERSION.bndl VERSION.bndl
 # edit VERSION.bndl to update version number to VERSION

Once these steps are complete, a user should be able to acquire the new release of the Vertical Grid Descriptor package using the standard SSM commands, with VERSION replaced by the version number just released.

 . s.ssmuse.dot CMDN/vgrid/VERSION

Once the installation is complete, the released version should be svn-tagged so that it is easily recoverable. These command should be run in the top level of the Vertical Grid Descriptor package on a platform that supports Subersion.

 make distclean
 cd ..
 svn cp svn://mrbsvn/others/vcoord/trunk svn://mrbsvn/others/vcoord/tags/release-VERSION \
        -m "Tagging the VERSION release of the vcoord project"

6 Vertical Coordinate Descriptor Versions

Different versions of the vertical coordinate record are supported for each kind of vertical coordinate. The kind value is the same as that used by convip, with each kind having the possibility of a set of version numbers that correspond to specific layouts of the vertical coordinate record. For example, kind=5 and version=2 corresponds to a staggered hybrid-level coordinate descriptor. If a non-staggered descriptor is required, then this could be implemented as kind=5 and version=1. The kind and version records supported are listed here.

For internal use by the Vertical Grid Descriptor module, the kind and version entries are mangled into a single unique vcode. The equation used to combine the values is currently vcode=kind*1000+version meaning that up to 1000 versions are supported for each vertical coordinate type. This value is written in the ig1 field, but it is not directly accessible to the user since any modification to it could change the validity structure of the vgrid_descriptor object.