diff --git a/doc.zih.tu-dresden.de/docs/access/security_restrictions.md b/doc.zih.tu-dresden.de/docs/access/security_restrictions.md index b43d631c07fc47bf55da932dbb0d11aca4cf2ecf..21767df64eabf9397240e5402899f25b8c5e10fa 100644 --- a/doc.zih.tu-dresden.de/docs/access/security_restrictions.md +++ b/doc.zih.tu-dresden.de/docs/access/security_restrictions.md @@ -20,7 +20,7 @@ The most important items for ZIH systems are: * **ssh from ZIH system** is only possible inside TU Dresden campus. (Direct SSH access to other computing centers was the spreading vector of the recent incident.) -Data transfer is possible via the [export nodes](../data_transfer/export_nodes.md). We are working +Data transfer is possible via the [dataport nodes](../data_transfer/dataport_nodes.md). We are working on a bandwidth-friendly solution. We understand that all this will change convenient workflows. If the measurements would render your diff --git a/doc.zih.tu-dresden.de/docs/access/ssh_login.md b/doc.zih.tu-dresden.de/docs/access/ssh_login.md index 059d487c4ab5e9184a10158dec8ed0bd22c48e53..bb4495b72bfde1278561d575a3526adfa7b74f62 100644 --- a/doc.zih.tu-dresden.de/docs/access/ssh_login.md +++ b/doc.zih.tu-dresden.de/docs/access/ssh_login.md @@ -77,7 +77,7 @@ in it (you can omit lines starting with `#`): ```bash Host barnard #For login (shell access) - HostName barnard.hpc.tu-dresden.de + HostName login1.barnard.hpc.tu-dresden.de #Put your ZIH-Login after keyword "User": User marie #Path to private key: @@ -87,9 +87,9 @@ Host barnard #Enable X11 forwarding for graphical applications and compression. You don't need parameter -X and -C when invoking ssh then. ForwardX11 yes Compression yes -Host taurusexport +Host dataport #For copying data without shell access - HostName taurusexport.hrsk.tu-dresden.de + HostName dataport1.hpc.tu-dresden.de #Put your ZIH-Login after keyword "User": User marie #Path to private key: @@ -104,8 +104,8 @@ Afterwards, you can connect to the ZIH system using: marie@local$ ssh barnard ``` -If you want to copy data from/to ZIH systems, please refer to [Export Nodes: Transfer Data to/from -ZIH's Filesystems](../data_transfer/export_nodes.md) for more information on export nodes. +If you want to copy data from/to ZIH systems, please refer to [Dataport Nodes: Transfer Data to/from +ZIH's Filesystems](../data_transfer/dataport_nodes.md) for more information on dataport nodes. ## X11-Forwarding diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md index 894626208947186e48ba7d08b439cf6aace48655..952b4f691ae5470b8bcad7b6d4120af50fa62f6e 100644 --- a/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md +++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md @@ -44,7 +44,7 @@ marie@login$ dtrsync -av /archiv/marie/results /scratch/marie/results !!! note "Listing files in archive" - The intermediate archive is not mounted on the login nodes, but only on the [export nodes](../data_transfer/export_nodes.md). + The intermediate archive is not mounted on the login nodes, but only on the [dataport nodes](../data_transfer/dataport_nodes.md). In order to list the user's files in the archive use the `dtls` command ```console diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md index 0b78f1cdac700aa3a9f3b901686f406463c29378..558b24d3c8b60130433e6585409a55a587984209 100644 --- a/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md +++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md @@ -92,7 +92,7 @@ require some data preparation (e.g. packaging), but serve different use cases: The intermediate archive is a tape storage easily accessible as a directory (`/archiv/<HRSK-project>/` or `/archiv/<login>/`) using the -[export nodes](../data_transfer/export_nodes.md) +[dataport nodes](../data_transfer/dataport_nodes.md) and [Datamover tools](https://doc.zih.tu-dresden.de/data_transfer/datamover/) to move your data to. For detailed information please visit the diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md b/doc.zih.tu-dresden.de/docs/data_transfer/dataport_nodes.md similarity index 62% rename from doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md rename to doc.zih.tu-dresden.de/docs/data_transfer/dataport_nodes.md index 2b3a3da9e005352b1c2165afa3ce184486b89e30..3b3b845fac73bd650398814cc007d1facd2e764c 100644 --- a/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md +++ b/doc.zih.tu-dresden.de/docs/data_transfer/dataport_nodes.md @@ -1,18 +1,30 @@ -# Transfer Data to/from ZIH Systems via Export Nodes +# Transfer Data to/from ZIH Systems via Dataport Nodes -To copy large data to/from ZIH systems, the so-called **export nodes** should be used. While it is +To copy large data to/from ZIH systems, the so-called **dataport nodes** should be used. While it is possible to transfer small files directly via the login nodes, they are not intended to be used that way. Furthermore, longer transfers will hit the CPU time limit on the login nodes, i.e. the process -get killed. The **export nodes** have a better uplink (10 GBit/s) allowing for higher bandwidth. Note -that you cannot log in via SSH to the export nodes, but only use `scp`, `rsync` or `sftp` +get killed. The **dataport nodes** have a better uplink (10 GBit/s) allowing for higher bandwidth. Note +that you cannot log in via SSH to the dataport nodes, but only use <!-- [NT] currently not available:`scp`, -->`rsync` or `sftp` (incl. FTP-clients like e.g. [FileZilla](https://filezilla-project.org/)) on them. -The export nodes are reachable under the hostname `taurusexport.hrsk.tu-dresden.de` (or -`taurusexport3.hrsk.tu-dresden.de` and `taurusexport4.hrsk.tu-dresden.de`). +The dataport nodes are reachable under the hostnames -Please keep in mind that there are different -[filesystems](../data_lifecycle/file_systems.md#recommendations-for-filesystem-usage). Choose the -one that matches your needs. +- `dataport1.hpc.tu-dresden.de` (IP: 141.30.73.4) +- `dataport2.hpc.tu-dresden.de` (IP: 141.30.73.5) + +Through the usage of these dataport nodes, you can bring your data to ZIH HPC systems or get data +from there - they have access to the different HPC +[filesystems](../data_lifecycle/file_systems.md#recommendations-for-filesystem-usage). +As of 11/2023, the following directories are accessible: + +- `/home` +- `/projects` +- `/data/horse` +- `/data/walrus` +- `/data/archiv` +- (`/data/software`) +- (`/data/old/home`) +- (`/data/old/software`) ## Access From Linux @@ -22,7 +34,7 @@ are explained in the following section in more detail. !!! important The following explanations require that you have already set up your [SSH configuration](../access/ssh_login.md#configuring-default-parameters-for-ssh). - +<!-- [NT] scp currently not available ### SCP The tool [`scp`](https://www.man7.org/linux/man-pages/man1/scp.1.html) @@ -32,35 +44,35 @@ in a directory, the option `-r` has to be specified. ??? example "Example: Copy a file from your workstation to ZIH systems" ```bash - marie@local$ scp <file> taurusexport:<target-location> + marie@local$ scp <file> dataport:<target-location> # Add -r to copy whole directory - marie@local$ scp -r <directory> taurusexport:<target-location> + marie@local$ scp -r <directory> dataport:<target-location> ``` For example, if you want to copy your data file `mydata.csv` to the directory `input` in your home directory, you would use the following: ```console - marie@local$ scp mydata.csv taurusexport:input/ + marie@local$ scp mydata.csv dataport:input/ ``` ??? example "Example: Copy a file from ZIH systems to your workstation" ```bash - marie@local$ scp taurusexport:<file> <target-location> + marie@local$ scp dataport:<file> <target-location> # Add -r to copy whole directory - marie@local$ scp -r taurusexport:<directory> <target-location> + marie@local$ scp -r dataport:<directory> <target-location> ``` For example, if you have a directory named `output` in your home directory on ZIH systems and you want to copy it to the directory `/tmp` on your workstation, you would use the following: ```console - marie@local$ scp -r taurusexport:output /tmp + marie@local$ scp -r dataport:output /tmp ``` - +--> ### SFTP The tool [`sftp`](https://man7.org/linux/man-pages/man1/sftp.1.html) (OpenSSH secure file transfer) @@ -69,24 +81,29 @@ use compression to increase performance. `sftp` is basically a virtual command line, which you could access and exit as follows. +!!! warning "Note" + It is important from which point in your directory tree you 'enter' `sftp`! + The current working directory (double ckeck with `pwd`) will be the target folder on your local + machine from/to which remote files from the ZIH system will be put/get by `sftp`. + The local folder might also be changed during a session with special commands. + During the `sftp` session, you can use regular commands like `ls`, `cd`, `pwd` etc. + But if you wish to access your local workstation, these must be prefixed with the letter `l` + (`l`ocal), e.g., `lls`, `lcd` or `lpwd`. + ```console # Enter virtual command line -marie@local$ sftp taurusexport +marie@local$ sftp dataport # Exit virtual command line sftp> exit # or sftp> <Ctrl+D> ``` -After that you have access to the filesystem on ZIH systems, you can use the same commands as on -your local workstation, e.g., `ls`, `cd`, `pwd` etc. If you would access to your local workstation -from this virtual command line, then you have to prefix the command with the letter `l` -(`l`ocal),e.g., `lls`, `lcd` or `lpwd`. - ??? example "Example: Copy a file from your workstation to ZIH systems" ```console - marie@local$ sftp taurusexport + marie@local$ cd my/local/work + marie@local$ sftp dataport # Copy file sftp> put <file> # Copy directory @@ -96,9 +113,11 @@ from this virtual command line, then you have to prefix the command with the let ??? example "Example: Copy a file from ZIH systems to your local workstation" ```console - marie@local$ sftp taurusexport + marie@local$ sftp dataport # Copy file sftp> get <file> + # change local (target) directory + sftp> lcd /my/local/work # Copy directory sftp> get -r <directory> ``` @@ -118,18 +137,18 @@ the local machine. ```console # Copy file - marie@local$ rsync <file> taurusexport:<target-location> + marie@local$ rsync <file> dataport:<target-location> # Copy directory - marie@local$ rsync -r <directory> taurusexport:<target-location> + marie@local$ rsync -r <directory> dataport:<target-location> ``` ??? example "Example: Copy a file from ZIH systems to your local workstation" ```console # Copy file - marie@local$ rsync taurusexport:<file> <target-location> + marie@local$ rsync dataport:<file> <target-location> # Copy directory - marie@local$ rsync -r taurusexport:<directory> <target-location> + marie@local$ rsync -r dataport:<directory> <target-location> ``` ## Access From Windows @@ -138,7 +157,7 @@ the local machine. Windows 10 (1809 and higher) comes with a [built-in OpenSSH support](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview) -including the above described [SCP](#scp) and [SFTP](#sftp). +including the above described <!--[SCP](#scp) and -->[SFTP](#sftp). ### GUI - Using WinSCP @@ -147,6 +166,7 @@ First you have to install [WinSCP](http://winscp.net/eng/download.php). Then you have to execute the WinSCP application and configure some option as described below. +<!-- screenshots will have to be updated-->  {: align="center"} diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/overview.md b/doc.zih.tu-dresden.de/docs/data_transfer/overview.md index 6e8a1bf1cc12e36e4aa15bd46b9eaf84e24171bc..3ba882cca01fea481507ff6172c76c63104adae6 100644 --- a/doc.zih.tu-dresden.de/docs/data_transfer/overview.md +++ b/doc.zih.tu-dresden.de/docs/data_transfer/overview.md @@ -1,6 +1,6 @@ # Data Transfer -## Data Transfer to/from ZIH Systems: Export Nodes +## Data Transfer to/from ZIH Systems: Dataport Nodes There are at least three tools for exchanging data between your local workstation and ZIH systems: `scp`, `rsync`, and `sftp`. Please refer to the offline or online man pages of @@ -8,9 +8,9 @@ There are at least three tools for exchanging data between your local workstatio [rsync](https://man7.org/linux/man-pages/man1/rsync.1.html), and [sftp](https://man7.org/linux/man-pages/man1/sftp.1.html) for detailed information. -No matter what tool you prefer, it is crucial that the **export nodes** are used as preferred way to +No matter what tool you prefer, it is crucial that the **dataport nodes** are used as preferred way to copy data to/from ZIH systems. Please follow the link to the documentation on -[export nodes](export_nodes.md) for further reference and examples. +[dataport nodes](dataport_nodes.md) for further reference and examples. ## Data Transfer Inside ZIH Systems: Datamover diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md index 3a400076e75bcc1631e740998749ebac0aedc84f..e5963be3f2ce6035e71d8bbcd79b239a0ad2854d 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md @@ -113,23 +113,21 @@ only from their respective login nodes. -## Login and Export Nodes +## Login and Dataport Nodes !!! Note " **On December 11 2023 Taurus will be decommissioned for good**." Do not use Taurus for production anymore. -- 4 Login-Nodes `tauruslogin[3-6].hrsk.tu-dresden.de` - - Each login node is equipped with 2x Intel(R) Xeon(R) CPU E5-2680 v3 with 24 cores in total @ - 2.50 GHz, Multithreading disabled, 64 GB RAM, 128 GB SSD local disk - - IPs: 141.30.73.\[102-105\] -- 2 Data-Transfer-Nodes `taurusexport[3-4].hrsk.tu-dresden.de` - - DNS Alias `taurusexport.hrsk.tu-dresden.de` - - 2 Servers without interactive login, only available via file transfer protocols +- Login-Nodes + - Individual for each cluster. See sections below. +- 2 Data-Transfer-Nodes + - 2 servers without interactive login, only available via file transfer protocols (`rsync`, `ftp`) - - IPs: 141.30.73.\[82,83\] + - `dataport[3|4].hpc.tu-dresden.de` + - IPs: 141.30.73.\[4,5\] - Further information on the usage is documented on the site - [Export Nodes](../data_transfer/export_nodes.md) + [dataport Nodes](../data_transfer/dataport_nodes.md) ## Barnard diff --git a/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md b/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md index fe871b3d9adf2c22f5a7ac84edb5b2a953522f50..c9495588d57e726c00a31da961508e5096439170 100644 --- a/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md +++ b/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md @@ -195,31 +195,31 @@ The approach depends on the data volume: up to 100 MB or above. transfer section. ### Transferring Data *To/From* ZIH HPC Systems - +<!-- [NT] currently not available ???+ example "`scp` for transferring data to ZIH HPC systems" Copy the file `example.R` from your local machine to a workspace on the ZIH systems: ```console - marie@local$ scp /home/marie/Documents/example.R marie@export.hpc.tu-dresden.de:/data/horse/ws/your_workspace/ + marie@local$ scp /home/marie/Documents/example.R marie@dataport1.hpc.tu-dresden.de:/data/horse/ws/your_workspace/ Password: example.R 100% 312 32.2KB/s 00:00`` ``` - Note, the target path contains `export.hpc.tu-dresden.de`, which is one of the - so called [export nodes](../data_transfer/export_nodes.md) that allows for data transfer from/to the outside. + Note, the target path contains `dataport1.hpc.tu-dresden.de`, which is one of the + so called [dataport nodes](../data_transfer/dataport_nodes.md) that allows for data transfer from/to the outside. ???+ example "`scp` to transfer data from ZIH HPC systems to local machine" Copy the file `results.csv` from a workspace on the ZIH HPC systems to your local machine: ```console - marie@local$ scp marie@export.hpc.tu-dresden.de:/data/horse/ws/marie-test-workspace/results.csv /home/marie/Documents/ + marie@local$ scp marie@dataport1.hpc.tu-dresden.de:/data/horse/ws/marie-test-workspace/results.csv /home/marie/Documents/ ``` Feel free to explore further [examples](http://bropages.org/scp) of the `scp` command - and possibilities of the [export nodes](../data_transfer/export_nodes.md). - + and possibilities of the [dataport nodes](../data_transfer/dataport_nodes.md). +--> !!! caution "Terabytes of data" If you are planning to move terabytes or even more from an outside machine into ZIH systems, diff --git a/doc.zih.tu-dresden.de/docs/software/gpu_programming.md b/doc.zih.tu-dresden.de/docs/software/gpu_programming.md index 9c25859d509a0d621a5b1d6413e00391dd29a1ca..84eb94c667e550c55e6f8c73cb3672a95c88b278 100644 --- a/doc.zih.tu-dresden.de/docs/software/gpu_programming.md +++ b/doc.zih.tu-dresden.de/docs/software/gpu_programming.md @@ -260,7 +260,7 @@ metrics and `--export-profile` to generate a report file, like this: marie@compute$ nvprof --analysis-metrics --export-profile <output>.nvvp ./application [options] ``` -[Transfer the report file to your local system](../data_transfer/export_nodes.md) and analyze it in +[Transfer the report file to your local system](../data_transfer/dataport_nodes.md) and analyze it in the Visual Profiler (`nvvp`) locally. This will give the smoothest user experience. Alternatively, you can use [X11-forwarding](../access/ssh_login.md). Refer to the documentation for details about the individual @@ -317,7 +317,7 @@ needs, this analysis may be sufficient to identify optimizations targets. The graphical user interface version can be used for a thorough analysis of your previously generated report file. For an optimal user experience, we recommend a local installation of NVIDIA Nsight Systems. In this case, you can -[transfer the report file to your local system](../data_transfer/export_nodes.md). +[transfer the report file to your local system](../data_transfer/dataport_nodes.md). Alternatively, you can use [X11-forwarding](../access/ssh_login.md). The graphical user interface is usually available as `nsys-ui`. @@ -361,7 +361,7 @@ manually. This report file can be analyzed in the graphical user interface profiler. Again, we recommend you generate a report file on a compute node and -[transfer the report file to your local system](../data_transfer/export_nodes.md). +[transfer the report file to your local system](../data_transfer/dataport_nodes.md). Alternatively, you can use [X11-forwarding](../access/ssh_login.md). The graphical user interface is usually available as `ncu-ui` or `nv-nsight-cu`.