Best practices when using cyclone

We, as users, share cyclone computing resources (CPU and memory). at the same time. Here are a few recommendations to make sure that everybody can use it most efficiently:

First and foremost, please monitor your resource usage

Useful commands are

   • htop and top: Monitor usage of CPU and memory usage for the entire system. CPU usage given here is percent of one CPU core. I.e., a usage of 800% corresponds to 8 CPU cores being fully used.
   • top -u <your_username>: Monitor CPU and memory usage for your programs only. 
   • finger <some_username>: Find out who runs a program.

Limit your CPU use

Often, one program running on cyclone will use one CPU with maximum 100% CPU. However, some software such as Matlab, some Python modules, some model simulations, etc, by default try to occupy the entire machine they run on (72 virtual CPU cores for cyclone). This drastically slows down the jobs of other users, especially if several such programs run at same time.

CPU usage on cyclone is limited by the operating system to 25 virtual CPU cores - approx 1/3 of total. Trying to use more than that will not only slow down execution for everybody else, but also for yourself!

Therefore, please limit the number of CPUs your programs use.

- For Matlab: LASTN = maxNumCompThreads(N) with N set to a maximum of 8 (preferably less) - the user allows for 8 physical cores and 16 virtual cores.

https://se.mathworks.com/help/matlab/ref/maxnumcompthreads.html

- For Python (especially when using Pandas) and OpenMP-parallelised programs (written in Fortran, C, or any other language): 1) In the shell, before starting the program: export OMP_NUM_THREADS=N 2) Alternatively, within python: import os os.environ["OMP_NUM_THREADS"] = "N"

Preferably, set N to a maximum of 8 (preferably less).

If these limits are too restrictive for your application, you may want to look at other options for running heavily parallelized jobs, such as Fram of the Norwegian e-infrastructure.

Limit your memory use

This is currently less critical, as cyclone has a large amount of memory. However, sometimes, if the job uses a lot of memory it can be the sign of something going wrong in your script. Therefore, make sure to control your memory and to clear up/delete all unused variables (to deallocate the memory). Try also to close Matlab or Python when the job is finished, in the evening or before leaving for the weekend.

There is a hard limit imposed by the operating system that no user can take up more than half of cyclone’s memory. If you attempt to violate this restriction, the offending process will be killed by the operating system.

Thank you for following these recommendations.