Using cyclone

Connection and first steps

Requirements: To access GFI&SKD department servers via ssh, your computer will need to be attached to UiB-network. Connection via ssh is also open from Sigma2 resources, and sshfs is available to/from Sigma2. Otherwise, connecting from outside UiB or from private platform, you will need UiB-vpn or login via UiB login server. And your UiB account will need to be enabled for connection.

Use ssh to connect to one of the cyclones systems, cyclone1, cyclone2 and cyclone3, respectively:

ssh -Y <username>@cyclone1.gfi.uib.no
ssh -Y <username>@cyclone2.gfi.uib.no
ssh -Y <username>@cyclone3.gfi.uib.no  - (cyclone3 is reserved for GPU usage only)

or just simply:

ssh -Y cyclone1.gfi.uib.no
ssh -Y cyclone2.gfi.uib.no
ssh -Y cyclone3.gfi.uib.no  -  (cyclone3 is reserved for GPU usage only)

if you're connecting from a computer where you are logged in under the same user name (as usually the case with your UiB machine). When you are connecting for the first time, you will be asked to verify a signature. Reply "yes" to this question.

Home directories

All servers cyclone*.gfi.uib.no share same home directories. These home directories are separate from your standard UiB home directory. The content of your home directories will not be synced, so you will manually need to transfer your scripts to the new system.

Please be aware that your local home directory is not expected to exceed 30GB at any time - even in periods when sys.adms. do not practice quotas. If you need to store larger amount of data, please plan for using the GFI&SKD storage - also described in this wiki.

To monitor your home directory size, you can use these commands from cyclone command shell:

du -sh ~/

Or to include more details on both dot files/directories and non-hidden home directory subfolders:

cd; du -sh .[^.]* *

How to access software

Most/all scientific software is only available through EESSI software modules. Modules need to be loaded manually before the respective software can be used. We use software modules, because they provide are a clean and handy way to offer conflicting software or software versions: Instead of updating software and potentially breaking user scripts and programs new software is then made available in parallel to those that require it.

Start-up scripts

It might be handy to automatically load modules for software that you use frequently every time you login to cyclone. To accomplish that for bash, there are two relevant files, $HOME/.bashrc and $HOME/.bash_profile . .bash_profile should contain things that need to be defined at login time only, like PATH and other environment variables, startup programs, etc. You just need things once, not in every shell you open. In most cases, you also need the things from .bashrc in your login shell. That's why .bash_profile sources .bashrc as well, but .bashrc doesn't usually source .bash_profile. In contrast, .bashrc should typically contain things you want to set in every shell you open, like aliases, functions, etc. These are per shell session items that are not inherited from environment (source and detailed explanation).

An example .bash_profile for inspiration

module load CDO/2.2.2-gompi-2023b
module load MATLAB/2023a

alias rm="rm -i"
alias cp="cp -i"
alias erai="cd /Data/skd/share/Reanalysis/ERA_INTERIM/6HOURLY/"

if [ -f ~/.bashrc ]; then
   . ~/.bashrc
fi

An example .bashrc for inspiration

alias rm="rm -i"
alias cp="cp -i"
alias erai="cd /Data/skd/share/Reanalysis/ERA_INTERIM/6HOURLY/"

Available software

Below is a list of frequently used software and which module to load on the new cyclone. Relevant module names are printed like for MATLAB/2023a. Modules can be loaded by module load <module name>, and unloaded module unload <module name>. More details on the module system.

To list all available modules, use the command

module avail

All packages listed with this command should work, so please report to support@hpc.uib.no if they do not.

Data analysis and plotting

• CDO

  CDO, CDO/2.2.2-gompi-2023

• Dynlib

  See temporary solution for python. Requires in addition OpenBLAS/0.3.1-GCC-7.3.0-2.30 and ScaLAPACK/2.0.2-gompi-2018b-OpenBLAS-0.3.1.

• ghostview

  <no module required>

• grads

  <no module required>

• Matlab

  Matlab, Matlab/2023a, Matlab/2021b

• ncdump and other netCDF utilities

  netCDF, netCDF/4.9.2-gompi-2023b

• NCL

  NCL, NCL/6.5.0

• nco

  NCO, NCO/5.1.9-foss-2023a

• ncview

  ncview, ncview/2.1.8-gompi-2023a. Load only when needed! The package is currently known to break some GUI programs like gedit and gvim.

• python, plus selection of scientific packages

  First choose and load the python module for the python version that you want to use, either Python/3.11.5-GCCcore-13.2.0 for python 3.11.5, or Python/2.7.18-GCCcore-12.2.0-bare for python 2.7.18.

  Then, as a temporary solution, many scientific python packages are installed in the following virtual environments:

  source /Data/gfi/users/local/share/virtualenv/dynpie2/bin/activate for python 2.7.15.

  source /Data/gfi/users/local/share/virtualenv/dynpie3-2021a/bin/activate for python 3.7.0.

  For the pygrib module, ecCodes/2.7.3-foss-2018b is required in addition.

• R

  R , R/4.4.1-gfbf-2023b

Text editors

• emacs, gedit, gvim, vi(m), nedit, meld, spyder (Spyder comes available when Anaconda module loaded)

  All installed globally

Compilers and libraries

• blas

  OpenBLAS/0.3.1-GCC-7.3.0-2.30. Note: The library is called libopenblas, so you'll need to compile with -lopenblas.

• eccodes (replacement for grib_api)

  ecCodes, ecCodes/2.31.0-gompi-2023b

• gcc, gfortran

  GCC, GCC/13.2.0

• grib_api

  grib_api/1.24.0-foss-2018b

• HDF5 libraries

  HDF5/1.14.3-gompi-2023b

• lapack

  ScaLAPACK, ScaLAPACK/ScaLAPACK/2.2.0-gompi-2023b-fb. Note: The library is called libscalapack, so you'll need to compile with -lscalapack.

• MPI

  OpenMPI/4.1.6-GCC-13.2.0

• netcdf

  For C-libaries:netCDF, netCDF/4.9.2-gompi-2023b,

  For Fortran-libraries netCDF-Fortran/4.6.1-gompi-2023b, compile with option -I/vol/vb_apps/software/netCDF-Fortran/4.6.1-gompi-2023b/include

• proj

  PROJ,PROJ/9.3.1-GCCcore-13.2.0

Jupyter & Jupyterhub

Jupyter-lab and jupyter notebook can be initiated when Anaconda module has been activated:

ml Anaconda3
jupyter-lab --no-browser --port 888X  (unoccupied port, interval 8880 < X < 8951)
(Prepare for copy&paste of the output line  http://localhost:888X/lab?token=<unique-code>)

Then from your local client computer:

ssh -N -L 888X:localhost:888X <userid>@cyclone1.gfi.uib.no
(same portnumber, hostname= cyclone1 or cyclone2)

From your preferred browser on your local client computer, paste in the http://localhost url output line from jupyter-lab:

http://localhost:888X/lab?token=<unique code>

Your jupyter-lab is now supposed to run towards your local browser window.

You can change "jupyter-lab" to "jupyter notebook" in first startup if notebook is preferred.

Here is a step by step guide to setup and use Jupyter based on the dynpie3 environment.

File:Get started cyclone.pdf

Accessing the data from your own machines

Note: From your private machines you will need to be connected to the UiB VPN to be able to mount the following network drives. From UiB-client computers, UiB network connection is sufficient.

On MacOS you can mount folders through Finder > Go > Connect to Server .... In the dialogue window that pops up, enter

smb://uib.no;<user name>@klient.uib.no/FELLES/MATNAT/GFI

as the Server Address. You might want to replace gfi by skd to access the skd disks if that is your primary affiliation.

On Ubuntu this is very similar: Files > Go > Enter Location .... Enter the address:

smb://uib.no;<user name>@klient.uib.no/FELLES/MATNAT/GFI

and enter your password in the next pop-up window, while keeping "UiB" in the "Domain" field.

From Windows you can mount by navigating to File Explorer > Map network drive and enter
\\klient.uib.no\felles\matnat\gfi. Make sure to enable Connect using different credentials and fill in <user name>@uib.no and your PW.

From platforms with sshfs available, alternative mount of network drive with sshfs to a dedicated directory using sshfs (linux: install with sudo apt install sshfs):

mkdir ~/cyclone_gfi
From UiB network: sshfs -o follow_symlinks <user name>@cyclone2.uib.no:/gfi/ ~/cyclone_gfi/
From ext. networks, please connect to UiB vpn first.
Unmount with: fusermount -u ~/cyclone_gfi

For access to Sigma2 storage resources via sshfs access from cyclones, defining the following aliases can be convenient for mount and umount towards ~/sigma2 - and similar can be done from Sigma2-resource for access to cyclones storage resource:

alias mountsigma2="mkdir ~/sigma2; sshfs <userid>@<sigma2-host>:<sigma2_path> ~/sigma2 -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,follow_symlinks,uid=$(id -u),gid=$(id -g)"
alias umountsigma2="cd ; chmod go+rx ~/; fusermount -u ~/sigma2; chmod go-rx ~/; rmdir ~/sigma2"

Displaying GUIs on your screen

The ssh-login command at the top of this wiki enables trusted X11 forwarding (the -Y option), which allows you to display Graphical User Interfaces (GUIs) on your screen. Inclusion of -C option as well is an advantage for speeding up the X11 response. This way you may be able to run software on a high performance system in a more userfriendly way than by just operating via the terminal. To make this work properly you may have to edit the ssh configuration file (~/.ssh/config on linux and macos systems) by adding the following lines:

ForwardX11 yes
ForwardX11Trusted yes
Compression yes

Windows will require additional terminal software (for establishing an ssh-connection) with inclusion of an XServer (e.g. MobaXterm, WSL or VcXsrv) or NoMachine - and some still prefer Putty.

For Mac the solution should be pretty similar when using XQuartz - for Tahoe and later macos versions it may come necessary with the following initial setting for Xquartz, entered from your Mac command shell:

defaults write org.xquartz.X11 enable_render_extension 0

And it may come fortunate with later follow up toggling "Full-screen mode" in XQuarts settings -> Output.