Frequently Asked Questions

Q: What are the HPC facilities available at IUCAA?
  To see available HPC facilities at IUCAA click here
 
Q: How do I get a cluster account?
  Faculties and student can directly request for cluster account where as, visitors and associates request should come through their collaborator faculty.
The visitor and associate request must be approved by Head of the Computing Facility, Prof. Sanjit Mitra.
To request a cluster account, please send an email to sysadm@iucaa.in
 
Q: How do I login to a cluster?
  You can login to cluster using ssh command.

Syntax:- ssh -X username@clustername or cluster IP address.

For example, You can login to Pegasus using following command,

External:- ssh -X username@pegasus.iucaa.in (Public access)

Internal:- ssh -X username@pegasus.ac.iucaa.in (IUCAA)

Please note that, from a Windows machine use PuTTY and set the remote server as pegasus.iucaa.in OR pegasus.ac.iucaa.in and username as your your_id and achieve the same
 
Q: How do I get list of free and used nodes?
  sinfo command gives gives list used and idle nodes,

Syntax:- sinfo
 
Q: How many cores I can use?
  It depends on the total number of the available cores. For more details please refer Pegasus cluster Policy
 
Q: Which compilers and application software are available on Pegasus?
  Please refer to compiler and application softwares in Pegasus
All the application softwares are available in the forms of modules, Which allows us to have different versions of the same application software
These modules can be loaded using the below commands: 'module avail' - Show available modules
'module add ' - Adds a module to your environment for this session
'module initadd ' - Configure module to be loaded at every login
'module list' - To list the previously loaded modules
'module unload ' - To unload previously loaded modules
Please note that, To avoid loading modules on each of the sessions add the module load command to your $HOME/.bashrc
 
Q: How do I setup a MPI environment on Pegasus?
  One can set up the MPI environment on Pegasus by loading respective OpenMPI modules. Use command "module avail" to show all available modules Available MPI modules are given below:

mpich-3.3.1

openmpi-3.1.2_gnu

openmpi-3.1.2_gnu_old

platform_mpi-9.1.4

Based on the requirements one can load the required modules using below command, For example : module load <MPI Module Name>
 
Q: Which job scheduler is installed on Pegasus?
  Pegasus uses Portable Batch System (PBS) for job scheduling.
For more details please refer PBS User Guide
 
Q: How do I run my computational job?
 

step 1 : you need to setup your working environment in order to fully utilize the system (refer FAQ 6).

step 2 : check what resources (e.g. system and software, hardware) are available and whether these resources are enough for you to run your job.

step 3 : write a PBS script, below is example PBS script to run MPI job a.out with 1 node and 32 processes.

Script name : test.pbs

#!/bin/bash
#PBS -l select=1:ncpus=32:mpiprocs=32 ## Specify Number of nodes and processors
#PBS -N job_name ## Specify the job name
#PBS -k oed ## Enable direct-write (error and output) to working directory
#PBS -o out_file.txt ## Specify the output file name
#PBS -e err_file.txt ## Specify the error file name
cd $PBS_O_WORKDIR ## Change path to current working directory
mpirun -np 32 ./a.out    

To execute above script run below command :
qsub test.pbs
click here to download the above script
 
Q: How do I check status of my job?
  Once the job is submitted to scheduler, PBS generates a new job id. Job_id allows us to track the progress of job.
One can list all the jobs submitted by user using below command,

Syntax:- qstat -u <username>

A job can be in PENDING, RUNNING, SUSPENDED or EXIT state. You can use bjobs command to get status of your job,

Syntax:- qstat <job id>
 
Q: How do I get list of all running, pending and existed jobs?
  You can use qstat command to get list of all type jobs. By default, displays information about your own pending, running and suspended jobs.

Syntax:- qstat

-f Specifies that a full status display is written to standard out.
-a All jobs are displayed in the alternative format (see Standard output). If the operand is a destination id, all jobs at that destination are displayed. If the operand is a job id, information about that job is displayed.
-e If the operand is a job id or not specified, only jobs in executable queues are displayed..
-i Job status is displayed in the alternative format. For a destination id operand, status for jobs at that destination which are not running are displayed. This includes jobs which are queued, held or waiting. If an operand is a job id, status for that job is displayed regardless of its state.
-r If an operand is a job id, status for that job is displayed. For a destination id operand, status for jobs at that destination which are running are displayed, this includes jobs which are suspended.
-n In addition to the basic information, nodes allocated to a job are listed.
-l In combination with -n, the -1 option puts all of the nodes on the same line as the job ID. In combination with -f, attributes are not folded to fit in a terminal window. This is intended to ease the parsing of the qstat output.
-s In addition to the basic information, any comment provided by the batch administrator or scheduler is shown.
-G Show size information in giga-bytes.
-M Show size information, disk or memory in mega-words. A word is considered to be 8 bytes.
-R In addition to other information, disk reservation information is shown. Not applicable to all systems.
-t Normal qstat output displays a summary of the array instead of the entire array, job for job. qstat -t expands the output to display the entire array.
-u Job status is displayed in the alternative format. If an operand is a job id, status for that job is displayed.
-Q Specifies that the request is for queue status and that the operands are destination identifiers.
-q Specifies that the request is for queue status which should be shown in the alternative format.
-B Specifies that the request is for batch server status and that the operands are the names of servers.
 
Q: How do I change resources of my job when it is running/pending?
  qalter command can be used to change resources of a running/pending job

Example:- qalter <flag-option> <parameter_value> <job_id>

Above command will change the job queue of given job id.
 
Q: How do I kill my job?
  qdel command can be used to kill running/pending job,

Syntax:- qdel <job id>
 
Q: How do I suspend my job?
  qhold command can be used to suspend pending job,

Syntax:- qhold <job id>

qsig command can be used to suspend running job,,

Syntax:- qsig -s suspend <job id>
 
Q: How do I resume a job suspended by qhold/qsig?
  qrls command can be used to release job suspended by qhold,

Syntax:- qrls <job id>

qsig command can be used to resume job suspended by qsig,

Syntax:- qsig -s resume <job id>
 
Q: How do I get list of available queues?
  qstat command can be used to get list of available queues,

Syntax:- qstat -Q
 
Q: How I can change my password on cluster?
  You can use yppasswd command to change your cluster account password.

Syntax:- yppasswd
 
Q: How can I access Mathematica on Pegasus?
  To access Mathematica on Pegasus please use the below command,

ssh -X <username>@192.168.11.236

Enter your cluster credentials when prompted for password
Above mentioned is the IP address of the graphics1 OR you can also ssh to graphics1 from login node directly.

ssh -X graphics1

Load the available Mathematica module, Then execute the below command on to terminal,
# mathematica
 
Q: How can I submit interactive jobs using PBS?
  One can submit interactive jobs using -I flag while submitting a job,

Syntax:- qsub -I <pbs_script_name>

If you wish to use X-windows display then use below mentioned command,

Syntax:- qsub -I -X <pbs_script_name>
 
Q: How can I block required nodes for a certain amount of time using PBS?
 

Syntax:- qsub -I -X -l select=8:ncpus=32:mpiprocs=32,walltime=12:00:00

Above command allows us to block 8 nodes for 12 hours, one can modify and change no.of nodes and walltime provided.
We can check which of the nodes are allocated to us by using below command,

Syntax:- qstat -f <job id>

Please note that, providing a walltime is very important or else the nodes will stay busy until we terminate the job manually.
 
Q: How can I access IDL on pegasus?
  To access IDL on Pegasus please use the below command,

ssh -X <username>@192.168.11.236 or 192.168.11.237

Enter your cluster credentials when prompted for password
Above mentioned are the IP address of the graphics1 & graphics2 OR you can also ssh to graphics1 & graphics2 from login node directly.

ssh -X graphics1 or graphics1

Load the available IDL module, then execute the below command on to terminal,
# idl
 
Q: Pegasus PBS queue policies
 

Default walltime : 1 Hour - If no walltime is provided in job submission script, then PBS sets a default walltime of 1 hour.

Deafult Maximum walltime : 48 Hours - Users will not be able to submit any jobs with the walltime more than 48 hours.

Maximum running and queued jobs based on the user groups,

Group Name Maximum No. of running jobs Maximum No. of queued jobs
Beginner 8 8
Intermediate 12 12
Expert 16 16
 
Q: What is the Default memory quota on Pegasus?
 

Default quota limits : 150 GB under Home(/mnt) and 500 GB in Scratch(/scratch) file system. The extension of the quota limit must be approved by the Head of the Computing Facility, Prof. Sanjit Mitra.
 
Q: How to transfer the files via Gateway?
  Please refer, Data transfer via Gateway
 
Q: How to access the Jupyter Notebook via Gateway?
  Please refer, Jupyter Notebook via Gateway
 
Q: Data download on Pegasus
  Pegasus has a dedicated data server to facilitate the public data transfer.
To access the Data server, please use the below command:

ssh -X <pegasus-username>@data.iucaa.in