NCL

This page provides general information about the NCL NCL scripted language. NCL is indexed on ZERO

Basics

How to run an NCL script on your local Linux machine or Mac OSX

An NCL script should be run on the command line as follows:

$ ncl script.ncl

Although NCL can be run interactively, this should only be used for quick testing only. Always run NCL scripts as above when doing actual science.

How to run an NCL script on the Pawsey supercomputers zeus and zythos

DO NOT run ncl on the command line on any of the Pawsey computers, as this may cause the systems to crash

We have our own local install of NCL on the Pawsey machines under /group/y98:

/group/y98/ncl-6.2.1
/group/y98/ncl-6.2.1-opendap

To use any particular version of ncl, you would need to add this to your .bashrc file (for example to use the 6.2.1 opendap enabled version):

export NCARG_ROOT=/group/y98/ncl-6.2.1-opendap
export PATH=$NCARG_ROOT/bin:$PATH

NCL currently runs on zeus, zythos and magnus however magnus uses a different application launcher to zeus/zythos, so the commands to launch scripts on the different systems will be different to reflect this. Please see the section below for using NCL on magnus.

When debugging or developing a script you can reserve a node so that you can effectively run NCL from the command line. Currently, this only works on the zeus/zythos system.

First reserve a node:

salloc --ntasks=1

Then you can run your script as follows:
srun --pty --unbuffered ncl test.ncl

And your script should run fine. After you have finished debugging, do not forget to exit from that node you have reserved. Just type the exit command:

exit

This will not exit you from zeus/zythos, but simply exit you from that node.

To submit an NCL script as an actual job to the queue using sbatch:

#!/bin/bash --login
#SBATCH --nodes=1
#SBATCH --time=00:01:00
#SBATCH --account=y98
# Launch the job.
srun ncl test.ncl >& test_batch_ncl.out

You can also pack multiple scripts to run on a single zeus/zythos node by following the example below:

#!/bin/bash -l
#SBATCH --account=y98
#SBATCH --nodes=1
#SBATCH --time=12:00:00
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=juliaandrys@gmail.com
#SBATCH --export=NONE
RUN_DIR="$SLURM_SUBMIT_DIR"

echo "Start `date`"

srun --mpi=pmi2 ncl A2_diagnostics_wrfhrly_miroc_1983.ncl &
srun --mpi=pmi2 ncl A2_diagnostics_wrfhrly_miroc_1984.ncl &
srun --mpi=pmi2 ncl A2_diagnostics_wrfhrly_miroc_1985.ncl &

wait

Each zeus node has 16 cores so you can pack up to 16 jobs on a node in parallel. Generally this many jobs will result in memory issues on the node but this will depend on the memory requirements of your script.

How to run an NCL script on the Pawsey supercomputer magnus

NCL does not currently run on a reserved node on magnus so any testing and debugging you need to do should be done on zeus/zythos.

To submit an NCL script as an actual job to the magnus work queue using sbatch:

#!/bin/bash
#SBATCH --account=y98
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:10:00
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=juliaandrys@gmail.com
#SBATCH --export=NONE
module swap PrgEnv-cray PrgEnv-intel
module load cray-netcdf
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export NETCDF=/opt/cray/netcdf/4.3.0/INTEL/130/
export NCARG_ROOT=/group/y98/ncl-6.2.1
export NCARG_USRRESFILE=$NCARG_ROOT/hluresfile
export MY_TMP=/scratch/y98/julia/tmp

TMPDIR=$MY_TMP aprun -n 1 $NCARG_ROOT/bin/ncl front-detection-julia.ncl

Enabling syntax highlighting for ncl when using vi/vim

Follow the instructions here. If a .vim/syntax directory does not exist, just create it in your $HOME directory, e.g:

mkdir –p $HOME/.vim/syntax

If a .vimrc file does not exist, just create one. Note that to view hidden files and directories, use "ls -lah"
If it's not working for some reason, it's possible you did not copy and paste properly, in which case, use this ncl.vim file:
/group/y98/jatinkala/ncl-vim/ncl.vim

Automatic header creating for NCL scripts.

You need to load various scripts at the start of an ncl script, depending on which function(s) you wish to use. Additinally, every ncl script needs to start with "begin" and end with "end". This can be automated following the instructions here.

Here is the header file I use personally:

~> cat ncl_header.txt 
:insert
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"

; File Name :

; Purpose :

; Creation Date :

; Last Modified :

; Created By : Jatin Kala (Jatin.Kala.JK@gmail.com)

begin
;--------------------- Inputs -------------------------

;------------------------------------------------------
; start code here

print("Script finished normally")
end ; this ends begin
.

Change Created by to your name!

And here is my .vimrc file (this assumes ncl_header.text file is in your $HOME directory:

~> cat .vimrc
" ~/.vimrc (configuration file for vim only)
" skeletons
function! SKEL_spec()
    0r /usr/share/vim/current/skeletons/skeleton.spec
    language time en_US
    let login = system('whoami')
    if v:shell_error
       let login = 'unknown'
    else
       let newline = stridx(login, "\n")
       if newline != -1
        let login = strpart(login, 0, newline)
       endif
    endif
    let hostname = system('hostname -f')
    if v:shell_error
        let hostname = 'localhost'
    else
        let newline = stridx(hostname, "\n")
        if newline != -1
        let hostname = strpart(hostname, 0, newline)
        endif
    endif
    exe "%s/specRPM_CREATION_DATE/" . strftime("%a\ %b\ %d\ %Y") . "/ge"
    exe "%s/specRPM_CREATION_AUTHOR_MAIL/" . login . "@" . hostname . "/ge"
    exe "%s/specRPM_CREATION_NAME/" . expand("%:t:r") . "/ge"
endfunction
autocmd BufNewFile    *.spec    call SKEL_spec()
au BufRead,BufNewFile *.ncl set filetype=ncl
au! Syntax newlang source $VIM/ncl.vim 
autocmd bufnewfile *.ncl so $HOME/ncl_header.txt
autocmd bufnewfile *.ncl exe "1," . 16 . "g/File Name :.*/s//File Name : " .expand("%")
autocmd bufnewfile *.ncl exe "1," . 16 . "g/Creation Date :.*/s//Creation Date : " .strftime("%d-%m-%Y")
autocmd Bufwritepre,filewritepre *.ncl execute "normal ma"
autocmd Bufwritepre,filewritepre *.ncl exe "1," . 16 . "g/Last Modified :.*/s/Last Modified :.*/Last Modified : " .strftime("%c")
autocmd bufwritepost,filewritepost *.ncl execute "normal `a"
" ~/.vimrc ends here

Basic scripting principles

  • All scripts should have an inputs section, and a main code section. The idea is that you only need to edit the inputs section and NOTHING else! This makes scripts re-usable!
  • Use names which are meaningful, ii, jj, x is not really helpful, but something like index_years is a Lot more helpful
  • Indent your code, especially loops and IF statements
  • Make useful comments throughout the code
  • Write code such that someone else should be able to understand what the code does, so they can quickly modify it to suit their purposes
  • Whichever text editor you use to write your NCL scripts, make sure you have enable text highlighting. It greatly helps to fix typos! See:
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
load "$NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"

; File Name : test.ncl

; Purpose :

; Creation Date : 13-11-2014

; Last Modified :

; Created By : Jatin Kala (Jatin.Kala.JK@gmail.com)

begin
;--------------------- Inputs -------------------------

;------------------------------------------------------
; do work here

end ; this ends begin
  1. NCL is indexed on Zero. Make sure you FULLY understand the implications of this. If one defines an array of integers going from 1 to 10, and incrementing by 1, eg., x = ispan(1,10,1), then the first element, 1, is x(0), the last element, 10, is x(9). The size of the array will be 10, but x(10) will be out-of-bounds. Do not do anything until you fully grasp this.
  2. When starting off with NCL, make sure you understand how to use the $ sign effectively. This will make code much shorter in length.
  3. As with any language, one need to become familiar with data-types, it is critical to understand how integers differ from float and double types. See: http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclDataTypes.shtml

    For example, pay attention to the maximum allowed value for integers in NCL is +/- 12147483647. If you something like this (a=0.01*10000000000000), the result will be non-sense!!! You would need instead (a=0.01*10000000000000.). Note the“dot/decimal” makes the number of type float and not integer.

  4. Every NCL script needs to have an “inputs section” and then the main code. The concept applies to all languages. The idea is only stuff in the inputs section needs to be modified, and nothing after! This make code much easier to re-use.
  5. If your NCL script is getting more than 500 lines long, you are doing something wrong. STOP and RE-THINK.
  6. You can create your own functions and procedures. Make sure everything, which belongs to a function, is passed to “local”. This ensures that anything defined within a function belongs to the function only. See already defined functions.
  7. “list_files” is a pre-defined NCL function. Do not use this syntax!! .e.g list_files = systemfunc(“ls *.nc”). This will return an error!!!
  8. When getting started with NCL, these are some of the most useful functions. Make sure you fully understand how to use them:
    • system
    • systemfunc
    • ind
    • where
    • do and end do loops
    • if-then-else statements
    • all the tochar, tostring, tofloat type functions,
    • dimsizes
    • print
    • addfile
    • dim_xxx_n (xxx = avg, stddev, etc)
    • ispan and fspan
    • spinti
    • new (to create arrays)
    • mask
    • There are many more, use google!
  9. Make use of the reference guide: http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/
  10. Learn how to properly write netcdf files in NCL: http://www.ncl.ucar.edu/Applications/method_2.shtml
  11. Plotting in NCL – everything on a plot is “tweakable” by setting the right resource. This is both good and bad because plotting scripts can sometimes become very long, with 30+ lines of “res@xxxx” definitions. Developing plotting functions can be very useful.
    • When adding a string to the label bar of a panel plot, at the bottom of the label bar, this causes the label bar to move up and mess with the x-axis. This can be fixed by setting: resP@pmLabelBarOrthogonalPosF = -0.025
    • To be continued ….

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License