Skip to content
Snippets Groups Projects
Commit 25d41f38 authored by Iryna Okhrin's avatar Iryna Okhrin
Browse files

making the page less colorful + unstrucured section *Data Management*

parent a9643a83
No related branches found
No related tags found
2 merge requests!580Automated merge from preview to main,!529Getting-started page
Hide whitespace changes
Inline Side-by-side
doc.zih.tu-dresden.de/docs/access/misc/putty_download.png 0 → 100644
+ 0
− 0
View file @ 25d41f38
doc.zih.tu-dresden.de/docs/access/misc/putty_download.png

180 KiB

doc.zih.tu-dresden.de/docs/access/ssh_login_putty.md 0 → 100644
+ 96
− 0
View file @ 25d41f38
# Connecting with SSH
## Connecting from Windows with PuTTY
Homepage: https://www.putty.org
### 1. Download and install PuTTY
To download go https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
Picture putty_download.png
Pick the installer suiting best your current system and run it afterwards. Follow the instructions
and finish the process. You should be able to see the following interface after starting the PuTTYapplication.
Picture putty_config
### 2. Start a new ssh-seccion
2.1 Quick start
1) Open PuTTY and insert the IP or hostname into the specified textbox and change the port as
needed.
Picture putty_quickstart
2) Click "Open" to start your new session. A Terminal will open up as new window. This action will do
basicly the same as using the command `ssh taurus.hrsk.tu-dresden.de:22` on a linux system.
Picture putty_login
3) After inserting your ZIH-Login and password it will log you into the targeted system.
2.2 Configured session
You can preconfigure some values to save time while the login in the future
Step 1 - Define the connection target
We start again by defining the hostname or IP and the port we want to connect to. But this time
we wont just open the session right away.
Picture putty_config1
Step 2 - Set an username
Use the tree-navigation on the left side to navigate to `Connection`->`Data`. There you will find a
textbox, which allows to set an `Auto-login username`. After entering your username the
application will basicly perform the same action as the command `ssh user@taurus.hrsk.tudresden.de:22` on a linux system.
Picture_putty_config2
Step 3 - Set a SSH-key
**Note**: For being able to use a ssh-key to lo into a system, you have to register the key on the
system before! Simply add the public-key to `~/.ssh/authorized_keys` and use the following
format.
```console
# <key-type> <public key> <comment>
ssh-ed25519 <public key> myuser@mylocalhost
```
To configure the ssh-key to use, naviagte to `Connection-SSH-Auth`. You will see a Textbox for `Private key file for authentifikation`. Insert the path to your local key-file or brows it using the `Browse...` Button. This will do the same as the command `ssh -i .ssh/id_red25519 user@taurus.hrsk.tudresden.de:22` on a linux system.
Picture purry_config3
Step 4 - Enable X-Forwarding (optional)
To enable X-forwarding navigate to `Connection`->`SSH`->`X11`. You will find a checkbox for `Enable X11 forwarding`. Simply put the tick in the checkbox and continue.
Picture putty_x11
Step 5 - Store your configuration
To store your current built configuration go back to the `session`-tab. There you can set a custom
name to store your configuration as. Insert this name into the `Saved Sessions`-Textbox. The Click
the `Save`-Button.
Picture putty_save
Afterwards you will see the configuration in the Listbox below.
**Note**: You can start a configured session by doubleclicking its name inside the listbox.
**Note**: You can alter your saved configurations by selecting its name inside the listbox and clicking
the `Load`-Button. To save the changes you have to save it again under the same name. This will
overwrite the old configuration permanetly.
**Note**: To deleted a configuration you have to select the session inside the listbox and click the
`Delete`-Button. This will remove the configured session permanetly.
## SSH Key Fingerprints
The page [key fingerprints](key_fingerprints.md) holds the up-to-date fingerprints for the login
nodes. Make sure they match.
doc.zih.tu-dresden.de/docs/training/getting_started.md
+ 61
− 66
View file @ 25d41f38
......@@ -57,10 +57,9 @@ This aspect is considered for all the following recommendations.
## Accessing the ZIH HPC system
!!! caution
The ZIH HPC system can be accessed only within the TUD campus networks.
Access from outside is possible by establishing a [VPN connection](https://tu-dresden.de/zih/dienste/service-katalog/arbeitsumgebung/zugang_datennetz/vpn#section-4).
The ZIH HPC system can be accessed only within the TUD campus networks.
Access from outside is possible by establishing a [VPN connection](https://tu-dresden.de/zih/dienste/service-katalog/arbeitsumgebung/zugang_datennetz/vpn#section-4).
There are different ways to access the ZIH HPC system (which are described in more detail below),
depending on the user's needs and previous knowledge:
......@@ -72,6 +71,8 @@ command line knowledge is required
[Graphical User Interfaces (GUIs)](../access/graphical_applications_with_webvnc.md) and similar:
e.g. commercial software such as Ansys, LS-DYNA (are not covered here).
Next, the access are described step by step.
### JupyterHub
1. Access JupyterHub here [https://taurus.hrsk.tu-dresden.de/jupyter](https://taurus.hrsk.tu-dresden.de/jupyter).
......@@ -108,7 +109,9 @@ This is the starting point for many tasks such as running programs and data mana
Using an ssh key pair is benefical for security reasons, although it is not necessary to work
with the ZIH HPC system.
=== "Linux/Mac users"
=== "Windows 10 and heigher/Mac/Linux users"
If you are a Windows user, you need to install Windows Terminal first (more information [here](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701?activetab=pivot:overviewtab)).
1. Open a terminal/shell/console (`Ctrl+Alt+T`) and type in
```console
......@@ -118,36 +121,30 @@ This is the starting point for many tasks such as running programs and data mana
1. After typing in your password, you end up seeing something like the following image.
![Successful ssh login](misc/ssh-success-login.png)
=== "Windows users"
=== "Users of any version of Windows"
1. Start [Windows Terminal](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701?activetab=pivot:overviewtab)
(Windows 10 or higher). In older versions of Windows install and set up MobaXTerm or PuTTY.
Install and set up [MobaXTerm](../access/ssh_login_mobaxterm) or [PuTTY](../access/ssh_login_putty).
For more information explore the [access compendium page](../access/ssh_login.md).
1. Follow the Linux/Mac users tab.
## Data Management and Data Transfer
For more information, see the [Windows compendium page](../access/ssh_login.md#connecting-from-windows).
First, it shown how to create a workspace, then how to transfer data within and to/from the ZIH HPC system. Also take care of the file permissions while collaboration.
## Data Management and Data Transfer
### Create a Workspace
There are different areas for storing your data on the ZIH HPC system, called [Filesystems](../data_lifecycle/file_systems.md).
You will need to create a [workspace](../data_lifecycle/workspaces.md) for your data (see example
You need to create a [workspace](../data_lifecycle/workspaces.md) for your data (see example
below) on one of these Filesystems.
The filesystems have different [properties](../data_lifecycle/file_systems.md) (available space,
storage time limit, permission rights). Therefore, choose the one that fits your project best.
To start we recommend the Lustre filesystem **scratch**.
!!! example "Creating a Workspace on Lustre Scratch Filesystem"
!!! example "Creating a workspace on Lustre Scratch filesystem"
The following command creates a workspace
* command: `ws_allocate`
* on the scratch filesystem: `-F scratch`
* with the name: `test-workspace`
* a life time of `90` days
* an email is sent to `marie.testuser@tu-dresden.de` 7 days before expiration:
`-r 7 -m marie.testuser@tu-dresden.de`
```console
marie@login$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90
Info: creating workspace.
......@@ -155,64 +152,62 @@ To start we recommend the Lustre filesystem **scratch**.
remaining extensions : 10
remaining time in days: 90
```
To explain:
* `ws_allocate` - command to allocate
* `-F scratch` - on the scratch filesystem
* `test-workspace` - workspace's name
* a life time of `90` days
* `-r 7 -m marie.testuser@tu-dresden.de` - send a reminder to `marie.testuser@tu-dresden.de` 7 days before expiration
The path to this workspace is `/scratch/ws/marie-test-workspace`. You will need it when
transferring data or running jobs.
The path to this workspace is `/scratch/ws/marie-test-workspace`. You will need it when transferring data or running jobs.
Find more [information on workspaces in the compendium](../data_lifecycle/workspaces.md).
!!! hint "Distinction: Transferring data from/to vs. within the ZIH system"
Please note the different settings for transferring data, that require different approaches:
### Transferring data **within** the ZIH HPC system
1. transfer within the ZIH system
1. transfer to or from the ZIH system
The approach depends on the data volume: up to 100 MB or above.
1. The approach for transferring data within the ZIH system depends on the data volume:
a) *small data (up to 100 MB)* and b) *medium/large data (above 100 MB)*
???+ example "`cp`/`mv` for small data (up to 100 MB)"
???+ example "a) Small Data (up to 100 MB)"
Use the standard Linux command `cp` on the command line.
* Copy the file `example1.R` **from your ZIH home directory to a workspace**:
Use command `cp` to copy the file `example.R` from your ZIH home directory to a workspace:
```console
marie@login$ cp /home/marie/example1.R /scratch/ws/marie-test-workspace
marie@login$ cp /home/marie/example.R /scratch/ws/marie-test-workspace
```
Analagously use command `mv` to move a file.
Find [here more examples for the cp command](http://bropages.org/cp).
Moving files is done analagously by using the command `mv`.
???+ example "b) Medium/Large Data (above 100 MB)"
Use the datamover commands e.g. `dtcp`, `dtmv` or `dtwget`.
Find [here more examples for the `cp` command](http://bropages.org/cp).
???+ example "`dtcp`/`dtmv` for medium/large data (above 100 MB)"
* Copying the directory `/warm_archive/ws/large-dataset` from one Filesystem location to another:
Use command `dtcp` to copy the directory `/warm_archive/ws/large-dataset` from one Filesystem location to another:
```console
marie@login$ dtcp -r /warm_archive/ws/large-dataset /scratch/ws/marie-test-workspace/data
```
```console
marie@login$ dtcp -r /warm_archive/ws/large-dataset /scratch/ws/marie-test-workspace/data
```
Analagously use command `dtmv` to move a file.
More [details on the datamover can be found here](../data_transfer/datamover.md).
2. Export nodes are used for transfer of data between outside and inside the ZIH HPC system.
### Transferring data **to/from** the ZIH HPC system
??? example "Copy a file between local machine and the ZIH HPC system (both directions)"
For transferring all data volume, use `scp` on the command line.
???+ example "`scp` for transferring data from/to the ZIH HPC system"
* Copy the file `example1.R` **from your local machine to a workspace** on the ZIH system:
Copy the file `example.R` from your local machine to a workspace on the ZIH system:
```console
marie@local$ scp /home/marie/Documents/example1.R marie@taurusexport.hrsk.tu-dresden.de:/scratch/ws/0/your_workspace/
Password:
example1.R 100% 312 32.2KB/s 00:00``
```
marie@local$ scp /home/marie/Documents/example.R marie@taurusexport.hrsk.tu-dresden.de:/scratch/ws/0/your_workspace/
Password:
example.R 100% 312 32.2KB/s 00:00``
```
Note, the target path contains `taurusexport.hrsk.tu-dresden.de`, which is one of the
so called export nodes that allows for data transfer from/to the outside.
Note that the target path contains `taurusexport.hrsk.tu-dresden.de`, which is one of the
so called export nodes that allow for data transfer from/to the outside.
???+ example "`scp` to transfer data from the ZIH HPC system to local machine"
* Copy the file `results.csv` **from a workspace on the ZIH system to your local machine**:
Copy the file `results.csv` from a workspace on the ZIH HPC system to your local machine:
```console
marie@local$ scp marie@taurusexport.hrsk.tu-dresden.de:/scratch/ws/0/marie-test-workspace/results.csv home/marie/Documents/
......@@ -222,22 +217,22 @@ a) *small data (up to 100 MB)* and b) *medium/large data (above 100 MB)*
Furthermore, checkout other possibilities on the compendium for working with the
[export nodes](../data_transfer/export_nodes.md).
!!! caution
!!! caution "Terabytes of data"
If you are planning to move terabytes or even more from an outside machine into the ZIH system,
please contact the ZIH HPC support in advance.
please contact the ZIH HPC support (hpcsupport@zih.tu-dresden.de) in advance.
!!! caution "Permission rights are crucial in a collaborative setting"
### Permission rights are crucial in a collaborative setting
Whenever working within a collaborative setting, take care of the file permissions.
Esp. after creating and transferring data, file permission configuration might be necessary.
Whenever working within a collaborative setting, take care of the file permissions.
Esp. after creating and transferring data, file permission configuration might be necessary.
**By default, workspaces are accessible only for the user who created the workspace.**
Files created by a user in the project directory have read-only access for other group members
by default. Therefore, the correct file permissions must be configured (using `chmod`
and `chgrp`) for all files in the project home and the workspaces that should be fully
accessible (read, write, execute) to your collaborator group.
A first [overview on users and permissions in Linux can be found here](https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/Users_and_permissions).
**By default, workspaces are accessible only for the user who created the workspace.**
Files created by a user in the project directory have read-only access for other group members
by default. Therefore, the correct file permissions must be configured (using `chmod`
and `chgrp`) for all files in the project home and the workspaces that should be fully
accessible (read, write, execute) to your collaborator group.
A first [overview on users and permissions in Linux can be found here](https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/Users_and_permissions).
??? example "Checking and changing file permissions"
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung