fem_software.md

All packages described in this section, are organized in so-called modules. To list the available versions of a package and load a
particular, e.g., ANSYS, version, invoke the commands

```console
marie@login$ module avail ANSYS
[...]
marie@login$ # module load ANSYS/<version>
marie@login$ # e.g.
marie@login$ module load ANSYS/2020R2
```

The section [runtime environment](modules.md) provides a comprehensive overview
on the module system and relevant commands.
(Thanks to Benjamin Groeger, Inst. f. Leichtbau und Kunststofftechnik)).

1. Prepare an Abaqus input-file. You can start with the input example from Benjamin:
[Rot-modell-BenjaminGroeger.inp](misc/Rot-modell-BenjaminGroeger.inp)
2. Prepare a job file on ZIH systems like this
```bash
#!/bin/bash
### needs ca 20 sec with 4cpu
### generates files:
###  yyyy.com
###  yyyy.dat
###  yyyy.msg
###  yyyy.odb
###  yyyy.prt
###  yyyy.sim
###  yyyy.sta
#SBATCH --nodes=1               # with >1 node Abaqus needs a nodeliste
#SBATCH --ntasks-per-node=4
#SBATCH --mem=2048               # total memory
#SBATCH --time=00:04:00
#SBATCH --job-name=yyyy         # give a name, what ever you want
#SBATCH --mail-type=END,FAIL    # send email when the job finished or failed
#SBATCH --mail-user=marie@tu-dresden.de  # set your email
#SBATCH --account=p_number_crunch       # charge compute time to project p_number_crunch


# Abaqus has its own MPI
unset SLURM_GTIDS

# load module and start Abaqus
module load ABAQUS/2022
abaqus interactive input=Rot-modell-BenjaminGroeger.inp job=yyyy cpus=4 mp_mode=mpi
```
3. Start the job file (e.g., name `batch-Rot-modell-BenjaminGroeger.sh`)
```
marie@login$ sbatch batch-Rot-modell-BenjaminGroeger.sh      # Slurm will provide the Job Id (e.g., 3130522)
```
4. Control the status of the job
```
marie@login$ squeue --me     # in column "ST" (Status) you will find a R=Running or P=Pending (waiting for resources)
```
Since the MPI library that Ansys uses internally (Platform MPI) has some problems integrating
seamlessly with Slurm, you have to unset the enviroment variable `SLURM_GTIDS` in your
environment befor running Ansysy workbench in interactive and batch mode.
# SSH connection established using -CX
marie@login$ # module load ANSYS/<version>
marie@login$ # e.g.
marie@login$ module load ANSYS/2020R2
marie@login$ runwb2
marie@login$ # module load ANSYS/<version>
marie@login$ # e.g.
marie@login$ module load ANSYS/2020R2
marie@login$ srun --time=00:30:00 --x11=first [SLURM_OPTIONS] --pty bash
[...]
marie@compute$ runwb2
The software NICE Desktop Cloud Visualization (DCV) enables to
remotly access OpenGL-3D-applications running on ZIH systems using its GPUs
(cf. [virtual desktops](virtual_desktops.md)).
marie@gpu$ module load ANSYS
marie@gpu$ unset SLURM_GTIDS
marie@gpu$ runwb2
```bash
#!/bin/bash
#SBATCH --time=0:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --mem-per-cpu=1000M

unset SLURM_GTIDS              # Odd, but necessary!

# module load ANSYS/<version>
# e.g.
module load ANSYS/2020R2

runwb2 -B -F Workbench_Taurus.wbpj -E 'Project.Update' -E 'Save(Overwrite=True)'
#or, if you wish to use a workbench replay file, replace the -E parameters with: -R mysteps.wbjn
```
marie@login$ srun --nodes 1 --ntasks-per-node=4 --time=0:20:00 --mem-per-cpu=1700 --pty bash -l
marie@node$ module load ANSYS/2021R2
marie@node$ mapdl -smp -np $SLURM_NTASKS
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --mem=2000
#SBATCH --job-name=ansys_mapdl
#SBATCH --output=output_ansys_mapdl
#SBATCH --time=01:00:00

module load ANSYS/2021R2

# -smp use shared memory parallel version
# -b (batch mode)
# -np specify number of cpu's to use
# -j jobname

mapdl -smp -b -np $SLURM_NTASKS -j solution -i <input-file>
marie@login$ sbatch mapdl_job.sh
marie@login$ srun --nodes 4 --ntasks-per-node=4 --time=0:20:00 --mem-per-cpu=1700 --pty bash -l

# generate node list
marie@node$ NODELIST=$(for node in $( scontrol show hostnames $SLURM_JOB_NODELIST | uniq ); do echo -n "${node}:${SLURM_NTASKS_PER_NODE}:"; done | sed 's/:$//')

marie@node$ KMP_AFFINITY=none mapdl -machines $NODELIST
#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --mem=2000
#SBATCH --job-name=ansys_mapdl
#SBATCH --output=output_ansys_mapdl
#SBATCH --time=01:00:00

module purge
module load ANSYS/2021R2

# generate node list
NODELIST=$(for node in $( scontrol show hostnames $SLURM_JOB_NODELIST | uniq ); do echo -n "${node}:${SLURM_NTASKS_PER_NODE}:"; done | sed 's/:$//')

# -b (batch mode)
# -machines xxx   specify machines list for distributed Ansys
# -j jobname
setenv KMP_AFFINITY none

mapdl -b -machines $NODELIST -j solution -i <input-file>
marie@login$ sbatch mapdl_job.sh
Start the server process with 4 cores, 10 GB RAM and max. 8 hours running time using an
interactive Slurm job like this:

```console
marie@login$ module load COMSOL
marie@login$ srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=2500 --time=08:00:00 comsol -np 4 server
```
If you'd like to work interactively using COMSOL, you can request for an interactive job with,
e.g., 4 cores and 2500 MB RAM for 8 hours and X11 forwarding to open the COMSOL GUI:

```console
marie@login$ module load COMSOL
marie@login$ srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=2500 --time=08:00:00 --pty --x11=first comsol -np 4
```

Please make sure, that the option *Preferences* --> Graphics --> *Renedering* is set to *software
rendering*. Than, you can work from within the campus network.
Interactive working is great for debugging and setting experiments up. But, if you have a huge
workload, you should definitively rely on job files. I.e., you put the necessary steps to get
the work done into scripts and submit these scripts to the batch system. These two steps are
outlined:

1. Create a [job file](../jobs_and_resources/slurm.md#job-files), e.g.
```bash
#!/bin/bash
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=12
#SBATCH --mem-per-cpu=2500

module load COMSOL
srun comsol -mpi=intel batch -inputfile ./MyInputFile.mph
```
```bash
#!/bin/bash
#SBATCH --time=01:00:00       # walltime
#SBATCH --ntasks=16           # number of processor cores (i.e. tasks)
#SBATCH --mem-per-cpu=1900M   # memory per CPU core

module load LS-DYNA
srun mpp-dyna i=neon_refined01_30ms.k memory=120000000
```

Submit the job file named `job.sh` to the batch system via

```console
marie@login$ sbatch job.sh
```

Please refer to the section [Slurm](../jobs_and_resources/slurm.md) for further details and
options on the batch system as well as monitoring commands.