diff --git a/README.md b/README.md index f23e803c9f0d0c8a32b0361bfb8d50bfc6a1ffb3..17a8f9baadd7a1f34655f311518f8661eea754eb 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # ZIH HPC Documentation This repository contains the documentation of the HPC systems and services provided at -[TU Dresden/ZIH](https://tu-dresden.de/zih/). +https://doc.zih.tu-dresden.de/ by [TU Dresden/ZIH](https://tu-dresden.de/zih/). ## Setup diff --git a/doc.zih.tu-dresden.de/docs/access/key_fingerprints.md b/doc.zih.tu-dresden.de/docs/access/key_fingerprints.md index fc73348f258ce2e8e4264ded0e7eb6fcfa29ea25..56daf41c37faa0f446228b6afb1938b69ad92444 100644 --- a/doc.zih.tu-dresden.de/docs/access/key_fingerprints.md +++ b/doc.zih.tu-dresden.de/docs/access/key_fingerprints.md @@ -7,6 +7,8 @@ ## Login Nodes +### Taurus + The following hostnames can be used to access ZIH systems: - `taurus.hrsk.tu-dresden.de` @@ -40,6 +42,76 @@ shown matches one of the table. In this case, the fingerprint matches the one given in the table. Thus, one can proceed by typing 'yes'. +### Barnard + +The following hostnames can be used to access ZIH systems: + +- `login1.barnard.hpc.tu-dresden.de` +- `login2.barnard.hpc.tu-dresden.de` + +All of these login nodes share common keys. When connecting, please make sure that the fingerprint +shown matches one of the table. + +#### login1.barnard.hpc.tu-dresden.de + +| Key type | Fingerprint | +|:---------|:----------------------------------------------------| +| RSA | SHA256:c3Gs+WH1STvogyR9jTAPH8vhc5Ue75XsYlyCFbLDhJU | +| RSA | MD5:73:dc:bc:79:dc:1b:bf:e8:cb:1b:f5:1f:5e:6d:64:19 | +| ECDSA | SHA256:8Coljw7yoVH6HA8u+K3makRK9HfOSfe+BG8W/CUEPp0 | +| ECDSA | MD5:63:9a:67:68:37:85:31:77:a4:e6:0b:da:8c:d9:2f:96 | +| ED25519 | SHA256:Ws/vbrp5e/Ay+fcVzhsL0jupjGkDdn1cJ+SX6gQB6Bs | +| ED25519 | MD5:7f:5c:e6:2b:6f:94:24:9b:0f:2f:1d:bc:40:6b:59:c7 | +{: summary="List of valid fingerprints for Barnard login1 node"} + +#### login2.barnard.hpc.tu-dresden.de + +| Key type | Fingerprint | +|:---------|:----------------------------------------------------| +| RSA | SHA256:ZYQvEUc493+BETiJZ/jwqp/rHCM+IW1Trf6FdcSrqX4 | +| RSA | MD5:f8:1f:b6:f2:02:81:b9:ee:ce:99:71:ee:35:ad:32:fc | +| ECDSA | SHA256:01KP44mEMzswqdrkgA7ViO62Atdk8pAib/pmLBgetyk | +| ECDSA | MD5:d9:33:27:26:00:c9:81:cf:bb:45:43:dc:05:e8:1f:43 | +| ED25519 | SHA256:BewwkydtP2riZPShvNzAOWm+dQtdOq535j7Vow1HbRQ | +| ED25519 | MD5:18:8b:cd:1e:2e:9a:6c:8c:ee:b5:c9:3e:68:a3:4a:3f | +{: summary="List of valid fingerprints for Barnard login2 node"} + +#### login3.barnard.hpc.tu-dresden.de + +| Key type | Fingerprint | +|:---------|:----------------------------------------------------| +| RSA | SHA256:+VSqQp+6LZrZXOHPuDhxd2ti9mam/gDLSbn5kH0S2UI | +| RSA | MD5:19:16:ce:34:0e:2c:5f:37:42:06:f7:55:7d:19:cf:1a | +| ECDSA | SHA256:qZbC5BDKrTvE3J6qgGJLQwxtjfYy6pmrI7teEjFnHiE | +| ECDSA | MD5:b1:19:a6:bf:9e:95:ce:ee:fd:ab:b3:ee:5e:d7:e0:a7 | +| ED25519 | SHA256:ATNHOAZNjWHAXMwTWgxMvB9DIZ5bZurneN4sBKGSsz8 | +| ED25519 | MD5:ee:cb:cc:ff:be:15:f2:e8:8e:ac:ef:da:a1:f9:48:33 | +{: summary="List of valid fingerprints for Barnard login3 node"} + +#### login4.barnard.hpc.tu-dresden.de + +| Key type | Fingerprint | +|:---------|:----------------------------------------------------| +| RSA | SHA256:IYpo+qHKOIs4TEftlDp63QlQr85xlcgbapfMsbCeZDE | +| RSA | MD5:a7:3d:c3:be:53:62:7f:fc:5a:b5:6b:ba:8c:83:6e:4c | +| ECDSA | SHA256:nnUzS1Zu9+yaXf8ayDIwmfXabPtyvdr5c3Hvp+/zXhs | +| ECDSA | MD5:69:f9:54:60:24:79:22:cb:7f:ba:d0:90:f5:0f:4a:5d | +| ED25519 | SHA256:1QXw+IC51iT55LiE/7JJEXL7Jm1GZjk+/7OjaYfWXUY | +| ED25519 | MD5:17:8c:ea:26:dc:f0:43:61:a8:4d:06:e3:8e:f7:27:29 | +{: summary="List of valid fingerprints for Barnard login4 node"} + +??? example "Connecting with SSH" + + ```console + marie@local$ ssh login1.barnard.hpc.tu-dresden.de + The authenticity of host 'login1.barnard.hpc.tu-dresden.de (172.24.95.28)' can't be established. + ECDSA key fingerprint is SHA256:8Coljw7yoVH6HA8u+K3makRK9HfOSfe+BG8W/CUEPp0. + Are you sure you want to continue connecting (yes/no)? + ``` + + In this case, the fingerprint matches the one given in the table. Thus, one can proceed by + typing 'yes'. + ## Export Nodes The following hostnames can be used to transfer files to/from ZIH systems: diff --git a/doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md b/doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md new file mode 100644 index 0000000000000000000000000000000000000000..3974a2524de36f7dd2b2fdb03f5a54fa3375f6a0 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md @@ -0,0 +1,109 @@ +# HPC Resources + +HPC resources in ZIH systems comprise the *High Performance Computing and Storage Complex* and its +extension *High Performance Computing – Data Analytics*. In total it offers scientists +about 60,000 CPU cores and a peak performance of more than 1.5 quadrillion floating point +operations per second. The architecture specifically tailored to data-intensive computing, Big Data +analytics, and artificial intelligence methods with extensive capabilities for energy measurement +and performance monitoring provides ideal conditions to achieve the ambitious research goals of the +users and the ZIH. + +## Login and Export Nodes + +- 4 Login-Nodes `tauruslogin[3-6].hrsk.tu-dresden.de` + - Each login node is equipped with 2x Intel(R) Xeon(R) CPU E5-2680 v3 with 24 cores in total @ + 2.50 GHz, Multithreading disabled, 64 GB RAM, 128 GB SSD local disk + - IPs: 141.30.73.\[102-105\] +- 2 Data-Transfer-Nodes `taurusexport[3-4].hrsk.tu-dresden.de` + - DNS Alias `taurusexport.hrsk.tu-dresden.de` + - 2 Servers without interactive login, only available via file transfer protocols + (`rsync`, `ftp`) + - IPs: 141.30.73.\[82,83\] + - Further information on the usage is documented on the site + [Export Nodes](../data_transfer/export_nodes.md) + +## AMD Rome CPUs + NVIDIA A100 + +- 34 nodes, each with + - 8 x NVIDIA A100-SXM4 Tensor Core-GPUs + - 2 x AMD EPYC CPU 7352 (24 cores) @ 2.3 GHz, Multithreading available + - 1 TB RAM + - 3.5 TB local memory on NVMe device at `/tmp` +- Hostnames: `taurusi[8001-8034]` +- Slurm partition: `alpha` +- Further information on the usage is documented on the site [Alpha Centauri Nodes](../jobs_and_resources/alpha_centauri.md) + +## Island 7 - AMD Rome CPUs + +- 192 nodes, each with + - 2 x AMD EPYC CPU 7702 (64 cores) @ 2.0 GHz, Multithreading available + - 512 GB RAM + - 200 GB local memory on SSD at `/tmp` +- Hostnames: `taurusi[7001-7192]` +- Slurm partition: `romeo` +- Further information on the usage is documented on the site [AMD Rome Nodes](../jobs_and_resources/rome_nodes.md) + +## Large SMP System HPE Superdome Flex + +- 1 node, with + - 32 x Intel(R) Xeon(R) Platinum 8276M CPU @ 2.20 GHz (28 cores) + - 47 TB RAM +- Configured as one single node +- 48 TB RAM (usable: 47 TB - one TB is used for cache coherence protocols) +- 370 TB of fast NVME storage available at `/nvme/<projectname>` +- Hostname: `taurussmp8` +- Slurm partition: `julia` +- Further information on the usage is documented on the site [HPE Superdome Flex](../jobs_and_resources/sd_flex.md) + +## IBM Power9 Nodes for Machine Learning + +For machine learning, we have IBM AC922 nodes installed with this configuration: + +- 32 nodes, each with + - 2 x IBM Power9 CPU (2.80 GHz, 3.10 GHz boost, 22 cores) + - 256 GB RAM DDR4 2666 MHz + - 6 x NVIDIA VOLTA V100 with 32 GB HBM2 + - NVLINK bandwidth 150 GB/s between GPUs and host +- Hostnames: `taurusml[1-32]` +- Slurm partition: `ml` + +## Island 6 - Intel Haswell CPUs + +- 612 nodes, each with + - 2 x Intel(R) Xeon(R) CPU E5-2680 v3 (12 cores) @ 2.50 GHz, Multithreading disabled + - 128 GB local memory on SSD +- Varying amounts of main memory (selected automatically by the batch system for you according to + your job requirements) + * 594 nodes with 2.67 GB RAM per core (64 GB in total): `taurusi[6001-6540,6559-6612]` + - 18 nodes with 10.67 GB RAM per core (256 GB in total): `taurusi[6541-6558]` +- Hostnames: `taurusi[6001-6612]` +- Slurm Partition: `haswell` + +??? hint "Node topology" + +  + {: align=center} + +## Island 2 Phase 2 - Intel Haswell CPUs + NVIDIA K80 GPUs + +- 64 nodes, each with + - 2 x Intel(R) Xeon(R) CPU E5-E5-2680 v3 (12 cores) @ 2.50 GHz, Multithreading disabled + - 64 GB RAM (2.67 GB per core) + - 128 GB local memory on SSD + - 4 x NVIDIA Tesla K80 (12 GB GDDR RAM) GPUs +- Hostnames: `taurusi[2045-2108]` +- Slurm Partition: `gpu2` +- Node topology, same as [island 4 - 6](#island-6-intel-haswell-cpus) + +## SMP Nodes - up to 2 TB RAM + +- 5 Nodes, each with + - 4 x Intel(R) Xeon(R) CPU E7-4850 v3 (14 cores) @ 2.20 GHz, Multithreading disabled + - 2 TB RAM +- Hostnames: `taurussmp[3-7]` +- Slurm partition: `smp2` + +??? hint "Node topology" + +  + {: align=center} diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/hdfview_memory.png b/doc.zih.tu-dresden.de/docs/archive/misc/hdfview_memory.png similarity index 100% rename from doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/hdfview_memory.png rename to doc.zih.tu-dresden.de/docs/archive/misc/hdfview_memory.png diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/i4000.png b/doc.zih.tu-dresden.de/docs/archive/misc/i4000.png similarity index 100% rename from doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/i4000.png rename to doc.zih.tu-dresden.de/docs/archive/misc/i4000.png diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/smp2.png b/doc.zih.tu-dresden.de/docs/archive/misc/smp2.png similarity index 100% rename from doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/smp2.png rename to doc.zih.tu-dresden.de/docs/archive/misc/smp2.png diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_profiling.md b/doc.zih.tu-dresden.de/docs/archive/slurm_profiling.md similarity index 96% rename from doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_profiling.md rename to doc.zih.tu-dresden.de/docs/archive/slurm_profiling.md index deb9e3331aa774d11d4f167225074475eff6383f..3ca0a8e2b6e4618923a379ed8bcec854256b7fbf 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_profiling.md +++ b/doc.zih.tu-dresden.de/docs/archive/slurm_profiling.md @@ -70,7 +70,8 @@ More information about profiling with Slurm: ## Memory Consumption of a Job If you are only interested in the maximal memory consumption of your job, you don't need profiling -at all. This information can be retrieved from within [job files](slurm.md#batch-jobs) as follows: +at all. This information can be retrieved from within +[job files](../jobs_and_resources/slurm.md#batch-jobs) as follows: ```bash #!/bin/bash diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/lustre.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/lustre.md index 615282e59c23aa93844116a5a58939274bf5f12f..17394c63f91dc6536cc6351a91d52f6a972cb278 100644 --- a/doc.zih.tu-dresden.de/docs/data_lifecycle/lustre.md +++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/lustre.md @@ -180,7 +180,7 @@ Useful options: To list your personal filesystem usage and limits (quota), invoke ```console -marie@login$ lfs quota -h -u $LOGIN <filesystem> +marie@login$ lfs quota -h -u $USER <filesystem> ``` Useful options: diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md index 3db16f94c248bfb726f6f80a207446434a69b29d..924d98077b2489ba5f2516f3e21fe49004747ad2 100644 --- a/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md +++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md @@ -28,10 +28,10 @@ times. ### List Available Filesystems To list all available filesystems for using workspaces, you can either invoke `ws_list -l` or -`ws_find -l`, e.g., +`ws_find --list`, e.g., ```console -marie@login$ ws_find -l +marie@login$ ws_find --list available filesystems: scratch (default) warm_archive @@ -44,7 +44,7 @@ beegfs The default filesystem is `scratch`. If you prefer another filesystem (cf. section [List Available Filesystems](#list-available-filesystems)), you have to explictly - provide the option `-F <fs>` to the workspace commands. + provide the option `--filesystem=<filesystem>` to the workspace commands. ### List Current Workspaces @@ -67,7 +67,7 @@ overview of some of these options. All available options can be queried by `ws_l === "Certain filesystem" ``` - marie@login$ ws_list --filesystem scratch_fast + marie@login$ ws_list --filesystem=scratch_fast id: numbercrunch workspace directory : /lustre/ssd/ws/marie-numbercrunch remaining time : 2 days 23 hours @@ -135,7 +135,7 @@ overview of some of these options. All available options can be queried by `ws_l ### Allocate a Workspace To allocate a workspace in one of the listed filesystems, use `ws_allocate`. It is necessary to -specify a unique name and the duration of the workspace. +specify a unique name and the duration (in days) of the workspace. ```console ws_allocate: [options] workspace_name duration @@ -154,31 +154,54 @@ Options: -c [ --comment ] arg comment ``` -!!! example +!!! example "Simple workspace allocation" + + The simple way to allocate a workspace is calling `ws_allocate` command with two arguments, + where the first specifies the workspace name and the second the duration. This allocates a + workspace on the default filesystem with no e-mail reminder. ```console - marie@login$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90 + marie@login$ ws_allocate test-workspace 90 Info: creating workspace. /scratch/ws/marie-test-workspace remaining extensions : 10 remaining time in days: 90 ``` -This will create a workspace with the name `test-workspace` on the `/scratch` filesystem for 90 -days with an email reminder for 7 days before the expiration. +!!! example "Workspace allocation on specific filesystem" + + In order to allocate a workspace on a non-default filesystem, the option + `--filesystem=<filesystem>` is required. + + ```console + marie@login$ ws_allocate --filesystem=scratch_fast test-workspace 3 + Info: creating workspace. + /lustre/ssd/ws/marie-test-workspace + remaining extensions : 2 + remaining time in days: 3 + ``` + +!!! example "Workspace allocation with e-mail reminder" -!!! Note "Email reminder" + This command will create a workspace with the name `test-workspace` on the `/scratch` filesystem + with a duration of 90 days and send an e-mail reminder. The e-mail reminder will be sent every + day starting 7 days prior to expiration. We strongly recommend setting this e-mail reminder. - Setting the reminder to `7` means you will get a reminder email on every day starting `7` days - prior to expiration date. We strongly recommend to set this email reminder. + ```console + marie@login$ ws_allocate --reminder=7 --mailaddress=marie.testuser@tu-dresden.de test-workspace 90 + Info: creating workspace. + /scratch/ws/marie-test-workspace + remaining extensions : 10 + remaining time in days: 90 + ``` !!! Note "Name of a workspace" - The workspace name should help you to remember the experiment and data stored here. It has to - be unique on a certain filesystem. On the other hand it is possible to use the very same name - for workspaces on different filesystems. + The workspace name should help you to remember the experiment and data stored here. It has to + be unique on a certain filesystem. On the other hand it is possible to use the very same name + for workspaces on different filesystems. -Please refer to the section [section Cooperative Usage](#cooperative-usage-group-workspaces) for +Please refer to the [section Cooperative Usage](#cooperative-usage-group-workspaces) for group workspaces. ### Extension of a Workspace @@ -186,7 +209,7 @@ group workspaces. The lifetime of a workspace is finite and different filesystems (storage systems) have different maximum durations. A workspace can be extended multiple times, depending on the filesystem. -| Filesystem (use with parameter `-F <fs>`) | Duration, days | Extensions | [Filesystem Feature](../jobs_and_resources/slurm.md#filesystem-features) | Remarks | +| Filesystem (use with parameter `--filesystem=<filesystem>`) | Duration, days | Extensions | [Filesystem Feature](../jobs_and_resources/slurm.md#filesystem-features) | Remarks | |:-------------------------------------|---------------:|-----------:|:-------------------------------------------------------------------------|:--------| | `scratch` (default) | 100 | 10 | `fs_lustre_scratch2` | Scratch filesystem (`/lustre/scratch2`, symbolic link: `/scratch`) with high streaming bandwidth, based on spinning disks | | `ssd` | 30 | 2 | `fs_lustre_ssd` | High-IOPS filesystem (`/lustre/ssd`, symbolic link: `/ssd`) on SSDs. | @@ -205,7 +228,7 @@ remaining extensions : 1 remaining time in days: 100 ``` -Mail reminder settings are retained. I.e., previously set mail alerts apply to the extended +E-mail reminder settings are retained. I.e., previously set e-mail alerts apply to the extended workspace, too. !!! attention @@ -221,28 +244,57 @@ marie@login$ ws_extend -F scratch my-workspace 40 it will now expire in 40 days **not** 130 days. -### Send Reminder for Workspace Expiry Date +### Send Reminder for Workspace Expiration Date + +We strongly recommend using one of the two provided ways to ensure that the expiration date of a +workspace is not forgotten. -Send a calendar invitation by Email to ensure that the expiration date of a workspace is not -forgotten +#### Send Daily Reminder + +An e-mail reminder can be set at workspace allocation using ```console -marie@login$ ws_send_ical -F scratch my-workspace -m marie.testuser@tu-dresden.de +ws_allocate --reminder=<N> --mailaddress=<mail> [...] ``` +This will send an e-mail every day starting `N` days prior to the expiration date. +See the [example above](#allocate-a-workspace) for reference. + +If you missed setting an e-mail reminder at workspace allocation, you can add a reminder later, e.g. + +``` +marie@login$ ws_allocate --name=FancyExp --duration=17 +[...] +marie@login$ ws_allocate --name=FancyExp --duration=17 --reminder=7 --mailaddress=marie@dlr.de +--extension +``` + +This will reallocate the workspace, which counts against your maximum number of reallocations (Note: +No data is deleted, but the database entry is modified). + +#### Send Calender Invitation + +The command `ws_send_ical` sends you an ical event on the expiration date of a specified workspace. This + calender invitation can be further managed according to your personal preferences. The syntax is + as follows: + + ```console + ws_send_ical --filesystem=<filesystem> --mail=<e-mail-address> --workspace=<workspace name> + ``` + ### Deletion of a Workspace To delete a workspace use the `ws_release` command. It is mandatory to specify the name of the workspace and the filesystem in which it is located: ```console -marie@login$ ws_release -F scratch my-workspace +marie@login$ ws_release --filesystem=scratch --name=my-workspace ``` -You can list your already released or expired workspaces using the `ws_restore -l` command. +You can list your already released or expired workspaces using the `ws_restore --list` command. ```console -marie@login$ ws_restore -l +marie@login$ ws_restore --list warm_archive: scratch: marie-my-workspace-1665014486 @@ -257,9 +309,9 @@ beegfs: ``` In this example, the user `marie` has three inactive, i.e., expired, workspaces namely -`my-workspace` in `scratch`, as well as `foo` and `bar` in `ssd` filesystem. The command `ws_restore --l` lists the name of the workspace and the expiration date. As you can see, the expiration date is -added to the workspace name as Unix timestamp. +`my-workspace` in `scratch`, as well as `foo` and `bar` in `ssd` filesystem. The command +`ws_restore --list` lists the name of the workspace and the expiration date. As you can see, the +expiration date is added to the workspace name as Unix timestamp. !!! hint "Deleting data in in an expired workspace" @@ -275,7 +327,7 @@ It performs the following steps once per day and filesystem: - Check for remaining life time of all workspaces. - If the workspaces expired, move it to a hidden directory so that it becomes inactive. -- Send reminder Emails to users if the reminder functionality was configured for their particular +- Send reminder e-mails to users if the reminder functionality was configured for their particular workspaces. - Scan through all workspaces in grace period. - If a workspace exceeded the grace period, the workspace and its data are deleted. @@ -295,7 +347,7 @@ warm_archive: 2 months), you can still restore your data **into an existing work Use ```console -marie@login$ ws_restore -l -F scratch +marie@login$ ws_restore --list --filesystem=scratch scratch: marie-my-workspace-1665014486 unavailable since Thu Oct 6 02:01:26 2022 @@ -305,12 +357,12 @@ to get a list of your expired workspaces, and then restore them like that into a workspace 'new_ws': ```console -marie@login$ ws_restore -F scratch marie-my-workspace-1665014486 new_ws +marie@login$ ws_restore --filesystem=scratch marie-my-workspace-1665014486 new_ws ``` -The expired workspace has to be specified by its full name as listed by `ws_restore -l`, including -username prefix and timestamp suffix (otherwise, it cannot be uniquely identified). The target -workspace, on the other hand, must be given with just its short name, as listed by `ws_list`, +The expired workspace has to be specified by its full name as listed by `ws_restore --list`, +including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified). The +target workspace, on the other hand, must be given with just its short name, as listed by `ws_list`, without the username prefix. Both workspaces must be on the same filesystem. The data from the old workspace will be moved into @@ -381,7 +433,7 @@ the following example (which works [for the program g16](../software/nanoscale_s # Allocate workspace for this job. Adjust time span to time limit of the job (-d <N>). WSNAME=computation_$SLURM_JOB_ID - export WSDDIR=$(ws_allocate -F ssd -n ${WSNAME} -d 2) + export WSDDIR=$(ws_allocate --filesystem=ssd --name=${WSNAME} --duration=2) echo ${WSDIR} # Check allocation @@ -424,7 +476,7 @@ For a series of jobs or calculations that work on the same data, you should allo once, e.g., in `scratch` for 100 days: ```console -marie@login$ ws_allocate -F scratch my_scratchdata 100 +marie@login$ ws_allocate --filesystem=scratch my_scratchdata 100 Info: creating workspace. /scratch/ws/marie-my_scratchdata remaining extensions : 2 @@ -453,7 +505,7 @@ this is mounted read-only on the compute nodes, so you cannot use it as a work d jobs! ```console -marie@login$ ws_allocate -F warm_archive my_inputdata 365 +marie@login$ ws_allocate --filesystem=warm_archive my_inputdata 365 /warm_archive/ws/marie-my_inputdata remaining extensions : 2 remaining time in days: 365 @@ -499,7 +551,7 @@ to others (if in the same group) via `ws_list -g`. in the project `p_number_crunch`, she can allocate a so-called group workspace. ```console - marie@login$ ws_allocate --group --name numbercrunch --duration 30 + marie@login$ ws_allocate --group --name=numbercrunch --duration=30 Info: creating workspace. /scratch/ws/0/marie-numbercrunch remaining extensions : 10 @@ -555,9 +607,14 @@ wrong name. Use only the short name that is listed after `id:` when using `ws_li ---- -**Q**: Man, I've missed to specify mail alert when allocating my workspace. How can I add the mail -alert functionality to an existing workspace? +**Q**: I forgot to specify an e-mail alert when allocating my workspace. How can I add the +e-mail alert functionality to an existing workspace? + +**A**: You can add the e-mail alert by "overwriting" the workspace settings via + +```console +marie@login$ ws_allocate --extension --mailaddress=<mail address> --reminder=<days> \ + --name=<workspace-name> --duration=<duration> --filesystem=<filesystem> +``` -**A**: You can add the mail alert by "overwriting" the workspace settings via `ws_allocate -x -m -<mail address> -r <days> -n <ws-name> -d <duration> -F <fs>`. (This will lower the remaining -extensions by one.) +This will lower the remaining extensions by one. diff --git a/doc.zih.tu-dresden.de/docs/index.md b/doc.zih.tu-dresden.de/docs/index.md index c22ef202a4408ab09d938219fa9be8b896cd7ae1..64802daa9ae83761fb147961185fec3322880f0b 100644 --- a/doc.zih.tu-dresden.de/docs/index.md +++ b/doc.zih.tu-dresden.de/docs/index.md @@ -31,8 +31,10 @@ Please also find out the other ways you could contribute in our ## News +* **2023-11-16** [OpenMPI 4.1.x - Workaround for MPI-IO Performance Loss](jobs_and_resources/mpi_issues/#openmpi-v41x-performance-loss-with-mpi-io-module-ompio) +* **2023-10-04** [User tests on Barnard](jobs_and_resources/barnard_test.md) +* **2023-06-01** [New hardware and complete re-design](jobs_and_resources/architecture_2023.md) * **2023-01-04** [New hardware: NVIDIA Arm HPC Developer Kit](jobs_and_resources/arm_hpc_devkit.md) -* **2022-01-13** [Supercomputing extension for TU Dresden](https://tu-dresden.de/zih/die-einrichtung/news/supercomputing-cluster-2022) ## Training and Courses @@ -41,4 +43,9 @@ We offer a rich and colorful bouquet of courses from classical *HPC introduction [Training Offers](https://tu-dresden.de/zih/hochleistungsrechnen/nhr-training) for a detailed overview of the courses and the respective dates at ZIH. -* [HPC introduction slides](misc/HPC-Introduction.pdf) (Nov. 2022) +* [HPC introduction slides](misc/HPC-Introduction.pdf) Sep. 2023 + +Furthermore, Center for Scalable Data Analytics and Artificial Intelligence +[ScaDS.AI](https://scads.ai) Dresden/Leipzig offers various trainings with HPC focus. +Current schedule and registration is available at the +[ScaDS.AI trainings page](https://scads.ai/transfer-2/teaching-and-training/). diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/architecture_2023.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/architecture_2023.md new file mode 100644 index 0000000000000000000000000000000000000000..b0d23e2e789719ed0ff95a84f8f1056753cbb60c --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/architecture_2023.md @@ -0,0 +1,58 @@ +# Architectural Re-Design 2023 + +With the replacement of the Taurus system by the cluster `Barnard` in 2023, +the rest of the installed hardware had to be re-connected, both with +Infiniband and with Ethernet. + + +{: align=center} + +## Compute Systems + +All compute clusters now act as separate entities having their own +login nodes of the same hardware and their very own Slurm batch systems. The different hardware, +e.g. Romeo and Alpha Centauri, is no longer managed via a single Slurm instance with +corresponding partitions. Instead, you as user now chose the hardware by the choice of the +correct login node. + +The login nodes can be used for smaller interactive jobs on the clusters. There are +restrictions in place, though, wrt. usable resources and time per user. For larger +computations, please use interactive jobs. + +## Storage Systems + +### Permanent Filesystems + +We now have `/home`, `/projects` and `/software` in a Lustre filesystem. Snapshots +and tape backup are configured. For convenience, we will make the old home available +read-only as `/home_old` on the data mover nodes for the data migration period. + +`/warm_archive` is mounted on the data movers, only. + +### Work Filesystems + +With new players with new software in the filesystem market it is getting more and more +complicated to identify the best suited filesystem for temporary data. In many cases, +only tests can provide the right answer, for a short time. + +For an easier grasp on the major categories (size, speed), the work filesystems now come +with the names of animals: + +* `/data/horse` - 20 PB - high bandwidth (Lustre) +* `/data/octopus` - 0.5 PB - for interactive usage (Lustre) +* `/data/weasel` - 1 PB - for high IOPS (WEKA) - coming soon + +### Difference Between "Work" And "Permanent" + +A large number of changing files is a challenge for any backup system. To protect +our snapshots and backup from work data, +`/projects` cannot be used for temporary data on the compute nodes - it is mounted read-only. + +Please use our data mover mechanisms to transfer worthy data to permanent +storages. + +## Migration Phase + +For about one month, the new cluster Barnard, and the old cluster Taurus +will run side-by-side - both with their respective filesystems. You can find a comprehensive +[description of the migration phase here](migration_2023.md). diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard_test.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard_test.md new file mode 100644 index 0000000000000000000000000000000000000000..1529565f8555712da22f15e16141d8be3ad7d301 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard_test.md @@ -0,0 +1,163 @@ +# Tests on Barnard + +All HPC users are invited to test our new HPC system Barnard and prepare your software +and workflows for production there. For general hints please refer to these sites: + +* [Details on architecture](/jobs_and_resources/architecture_2023), +* [Description of the migration](migration_2023.md). + +We value your feedback. Please provide it directly via our ticket system. For better processing, +please add "Barnard:" as a prefix to the subject of the [support ticket](../support/support). + +Here, you can find few hints which might help you with the first steps. + +## Login to Barnard + +All users and projects from Taurus now can work on Barnard. + +They can use `login[2-4].barnard.hpc.tu-dresden.de` to access the system +from campus (or VPN). [Fingerprints](/access/key_fingerprints/#barnard) + +All users have **new empty HOME** file systems, this means you have first have to... + +??? "... install your public ssh key on the system" + + - Please create a new SSH keypair with ed25519 encryption, secured with + a passphrase. Please refer to this + [page for instructions](../../access/ssh_login#before-your-first-connection). + - After login, add the public key to your `.ssh/authorized_keys` file + on Barnard. + +## Data Management + +* The `/project` filesystem is the same on Taurus and Barnard +(mounted read-only on the compute nodes). +* The new work filesystem is `/data/horse`. +* The slower `/data/walrus` can be considered as a substitute for the old + `/warm_archive`- mounted **read-only** on the compute nodes. + It can be used to store e.g. results. + +These `/data/horse` and `/data/walrus` can be accesed via workspaces. Please refer to the +[workspace page](../../data_lifecycle/workspaces/), if you are not familiar with workspaces. + +??? "Tips on workspaces" + * To list all available workspace filessystem, invoke the command `ws_list -l`." + * Please use the command `dtinfo` to get the current mount points: + ``` + marie@login1> dtinfo + [...] + directory on datamover mounting clusters directory on cluster + + /data/old/home Taurus /home + /data/old/lustre/scratch2 Taurus /scratch + /data/old/lustre/ssd Taurus /lustre/ssd + [...] + ``` + +!!! Warning + + All old filesystems fill be shutdown by the end of 2023. + + To work with your data from Taurus you might have to move/copy them to the new storages. + +For this, we have four new [datamover nodes](/data_transfer/datamover) that have mounted all storages +of the old and new system. (Do not use the datamovers from Taurus!) + +??? "Migration from Home Directory" + + Your personal (old) home directory at Taurus will not be automatically transferred to the new Barnard + system. **You are responsible for this task.** Please do not copy your entire home, but consider + this opportunity for cleaning up you data. E.g., it might make sense to delete outdated scripts, old + log files, etc., and move other files to an archive filesystem. Thus, please transfer only selected + directories and files that you need on the new system. + + The well-known [datamover tools](../../data_transfer/datamover/) are available to run such transfer + jobs under Slurm. The steps are as follows: + + 1. Login to Barnard: `ssh login[1-4].barnard.tu-dresden.de` + 1. The command `dtinfo` will provide you the mountpoints + + ```console + marie@barnard$ dtinfo + [...] + directory on datamover mounting clusters directory on cluster + + /data/old/home Taurus /home + /data/old/lustre/scratch2 Taurus /scratch + /data/old/lustre/ssd Taurus /lustre/ssd + [...] + ``` + + 1. Use the `dtls` command to list your files on the old home directory: `marie@barnard$ dtls + /data/old/home/marie` + 1. Use `dtcp` command to invoke a transfer job, e.g., + + ```console + marie@barnard$ dtcp --recursive /data/old/home/marie/<useful data> /home/marie/ + ``` + + **Note**, please adopt the source and target paths to your needs. All available options can be + queried via `dtinfo --help`. + + !!! warning + + Please be aware that there is **no synchronisation process** between your home directories at + Taurus and Barnard. Thus, after the very first transfer, they will become divergent. + + We recommand to **take some minutes for planing the transfer process**. Do not act with + precipitation. + +??? "Migration from `/lustre/ssd` or `/beegfs`" + + **You** are entirely responsible for the transfer of these data to the new location. + Start the dtrsync process as soon as possible. (And maybe repeat it at a later time.) + +??? "Migration from `/lustre/scratch2` aka `/scratch`" + + We are synchronizing this (**last: October 18**) to `/data/horse/lustre/scratch2/`. + + Please do **NOT** copy those data yourself. Instead check if it is already sychronized + to `/data/walrus/warm_archive/ws`. + + In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in + `dtrsync -a /data/old/lustre/scratch2/ws/0/my-workspace/newest/ /data/horse/lustre/scratch2/ws/0/my-workspace/newest/` + +??? "Migration from `/warm_archive`" + + We are preparing another sync from `/warm_archive` to `The process of syncing data from `/warm_archive` to `/data/walrus/warm_archive` is still ongoing. + + Please do **NOT** copy those data yourself. Instead check if it is already sychronized + to `/data/walrus/warm_archive/ws`. + + In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in + `dtrsync -a /data/old/warm_archive/ws/my-workspace/newest/ /data/walrus/warm_archive/ws/my-workspace/newest/` + +When the last compute system will have been migrated the old file systems will be +set write-protected and we start a final synchronization (sratch+walrus). +The target directories for synchronization `/data/horse/lustre/scratch2/ws` and +`/data/walrus/warm_archive/ws/` will not be deleted automatically in the mean time. + +## Software + +Please use `module spider` to identify the software modules you need to load.Like +on Taurus. + + The default release version is 23.10. + +## Slurm + +* We are running the most recent Slurm version. +* You must not use the old partition names. +* Not all things are tested. + +## Updates after your feedback (state: October 19) + +* A **second synchronization** from `/scratch` has started on **October, 18**, and is + now nearly done. +* A first, and incomplete synchronization from `/warm_archive` has been done (see above). + With support from NEC we are transferring the rest in the next weeks. +* The **data transfer tools** now work fine. +* After fixing too tight security restrictions, **all users can login** now. +* **ANSYS** now starts: please check if your specific use case works. +* **login1** is under construction, do not use it at the moment. Workspace creation does + not work there. diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md index 538296b4ea52aee6c99f132811af8112803adcf9..bf5f25146730bceb6b442bd20d4e08e73e0863fc 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md @@ -81,7 +81,7 @@ For machine learning, we have IBM AC922 nodes installed with this configuration: ??? hint "Node topology" -  +  {: align=center} ## Island 2 Phase 2 - Intel Haswell CPUs + NVIDIA K80 GPUs @@ -105,5 +105,5 @@ For machine learning, we have IBM AC922 nodes installed with this configuration: ??? hint "Node topology" -  +  {: align=center} diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md new file mode 100644 index 0000000000000000000000000000000000000000..c888857b47414e2c068cac78f9ca9804efb056b5 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md @@ -0,0 +1,81 @@ +# HPC Resources + +The architecture specifically tailored to data-intensive computing, Big Data +analytics, and artificial intelligence methods with extensive capabilities +for performance monitoring provides ideal conditions to achieve the ambitious +research goals of the users and the ZIH. + +## Overview + +From the users' perspective, there are separate clusters, all of them with their subdomains: + +| Name | Description | Year| DNS | +| --- | --- | --- | --- | +| **Barnard** | CPU cluster |2023| n[1001-1630].barnard.hpc.tu-dresden.de | +| **Romeo** | CPU cluster |2020|i[8001-8190].romeo.hpc.tu-dresden.de | +| **Alpha Centauri** | GPU cluster |2021|i[8001-8037].alpha.hpc.tu-dresden.de | +| **Julia** | single SMP system |2021|smp8.julia.hpc.tu-dresden.de | +| **Power** | IBM Power/GPU system |2018|ml[1-29].power9.hpc.tu-dresden.de | + +They run with their own Slurm batch system. Job submission is possible only from +their respective login nodes. + +All clusters will have access to these shared parallel filesystems: + +| Filesystem | Usable directory | Type | Capacity | Purpose | +| --- | --- | --- | --- | --- | +| Home | `/home` | Lustre | quota per user: 20 GB | permanent user data | +| Project | `/projects` | Lustre | quota per project | permanent project data | +| Scratch for large data / streaming | `/data/horse` | Lustre | 20 PB | | + +## Barnard - Intel Sapphire Rapids CPUs + +- 630 diskless nodes, each with + - 2 x Intel Xeon Platinum 8470 (52 cores) @ 2.00 GHz, Multithreading enabled + - 512 GB RAM +- Hostnames: `n[1001-1630].barnard.hpc.tu-dresden.de` +- Login nodes: `login[1-4].barnard.hpc.tu-dresden.de` + +## AMD Rome CPUs + NVIDIA A100 + +- 34 nodes, each with + - 8 x NVIDIA A100-SXM4 Tensor Core-GPUs + - 2 x AMD EPYC CPU 7352 (24 cores) @ 2.3 GHz, Multithreading available + - 1 TB RAM + - 3.5 TB local memory on NVMe device at `/tmp` +- Hostnames: `taurusi[8001-8034]` -> `i[8001-8037].alpha.hpc.tu-dresden.de` +- Login nodes: `login[1-2].alpha.hpc.tu-dresden.de` +- Further information on the usage is documented on the site [Alpha Centauri Nodes](alpha_centauri.md) + +## Island 7 - AMD Rome CPUs + +- 192 nodes, each with + - 2 x AMD EPYC CPU 7702 (64 cores) @ 2.0 GHz, Multithreading available + - 512 GB RAM + - 200 GB local memory on SSD at `/tmp` +- Hostnames: `taurusi[7001-7192]` -> `i[7001-7190].romeo.hpc.tu-dresden.de` +- Login nodes: `login[1-2].romeo.hpc.tu-dresden.de` +- Further information on the usage is documented on the site [AMD Rome Nodes](rome_nodes.md) + +## Large SMP System HPE Superdome Flex + +- 1 node, with + - 32 x Intel Xeon Platinum 8276M CPU @ 2.20 GHz (28 cores) + - 47 TB RAM +- Configured as one single node +- 48 TB RAM (usable: 47 TB - one TB is used for cache coherence protocols) +- 370 TB of fast NVME storage available at `/nvme/<projectname>` +- Hostname: `taurussmp8` -> `smp8.julia.hpc.tu-dresden.de` +- Further information on the usage is documented on the site [HPE Superdome Flex](sd_flex.md) + +## IBM Power9 Nodes for Machine Learning + +For machine learning, we have IBM AC922 nodes installed with this configuration: + +- 32 nodes, each with + - 2 x IBM Power9 CPU (2.80 GHz, 3.10 GHz boost, 22 cores) + - 256 GB RAM DDR4 2666 MHz + - 6 x NVIDIA VOLTA V100 with 32 GB HBM2 + - NVLINK bandwidth 150 GB/s between GPUs and host +- Hostnames: `taurusml[1-32]` -> `ml[1-29].power9.hpc.tu-dresden.de` +- Login nodes: `login[1-2].power9.hpc.tu-dresden.de` diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/migration_2023.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/migration_2023.md new file mode 100644 index 0000000000000000000000000000000000000000..3a6749cff0814d2dbf54d53288fbcaa7fcb85818 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/migration_2023.md @@ -0,0 +1,82 @@ +# Migration 2023 + +## Brief Overview over Coming Changes + +All components of Taurus will be dismantled step by step. + +### New Hardware + +The new HPC system "Barnard" from Bull comes with these main properties: + +* 630 compute nodes based on Intel Sapphire Rapids +* new Lustre-based storage systems +* HDR Infiniband network large enough to integrate existing and near-future non-Bull hardware +* To help our users to find the best location for their data we now use the name of +animals (size, speed) as mnemonics. + +More details can be found in the [overview](/jobs_and_resources/hardware_overview_2023). + +### New Architecture + +Over the last decade we have been running our HPC system of high heterogeneity with a single +Slurm batch system. This made things very complicated, especially to inexperienced users. +To lower this hurdle we now create homogenous clusters with their own Slurm instances and with +cluster specific login nodes running on the same CPU. Job submission is possible only +from within the cluster (compute or login node). + +All clusters will be integrated to the new Infiniband fabric and have then the same access to +the shared filesystems. This recabling requires a brief downtime of a few days. + +[Details on architecture](/jobs_and_resources/architecture_2023). + +### New Software + +The new nodes run on Linux RHEL 8.7. For a seamless integration of other compute hardware, +all operating system will be updated to the same versions of OS, Mellanox and Lustre drivers. +With this all application software was re-built consequently using GIT and CI for handling +the multitude of versions. + +We start with `release/23.10` which is based on software reqeusts from user feedbacks of our +HPC users. Most major software versions exist on all hardware platforms. + +## Migration Path + +Please make sure to have read [Details on architecture](/jobs_and_resources/architecture_2023) before +further reading. + +The migration can only be successful as a joint effort of HPC team and users. Here is a description +of the action items. + +|When?|TODO ZIH |TODO users |Remark | +|---|---|---|---| +| done (May 2023) |first sync /scratch to /data/horse/old_scratch2| |copied 4 PB in about 3 weeks| +| done (June 2023) |enable access to Barnard| |initialized LDAP tree with Taurus users| +| done (July 2023) | |install new software stack|tedious work | +| ASAP | |adapt scripts|new Slurm version, new resources, no partitions| +| August 2023 | |test new software stack on Barnard|new versions sometimes require different prerequisites| +| August 2023| |test new software stack on other clusters|a few nodes will be made available with the new sw stack, but with the old filesystems| +| ASAP | |prepare data migration|The small filesystems `/beegfs` and `/lustre/ssd`, and `/home` are mounted on the old systems "until the end". They will *not* be migrated to the new system.| +| July 2023 | sync `/warm_archive` to new hardware| |using datamover nodes with Slurm jobs | +| September 2023 |prepare recabling of older hardware (Bull)| |integrate other clusters in the IB infrastructure | +| Autumn 2023 |finalize integration of other clusters (Bull)| |**~2 days downtime**, final rsync and migration of `/projects`, `/warm_archive`| +| Autumn 2023 ||transfer last data from old filesystems | `/beegfs`, `/lustre/scratch`, `/lustre/ssd` are no longer available on the new systems| + +### Data Migration + +Why do users need to copy their data? Why only some? How to do it best? + +* The sync of hundreds of terabytes can only be done planned and carefully. +(`/scratch`, `/warm_archive`, `/projects`). The HPC team will use multiple syncs +to not forget the last bytes. During the downtime, `/projects` will be migrated. +* User homes (`/home`) are relatively small and can be copied by the scientists. +Keeping in mind that maybe deleting and archiving is a better choice. +* For this, datamover nodes are available to run transfer jobs under Slurm. Please refer to the +section [Transfer Data to New Home Directory](../barnard_test#transfer-data-to-new-home-directory) +for more detailed instructions. + +### A Graphical Overview + +(red: user action required): + + +{: align=center} diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/architecture_2023.drawio b/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/architecture_2023.drawio new file mode 100644 index 0000000000000000000000000000000000000000..59ba4ad3b2f8b9387de2ede22079055aa3c9b864 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/architecture_2023.drawio @@ -0,0 +1 @@ +<mxfile host="Electron" modified="2023-06-05T08:54:45.184Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/20.8.10 Chrome/106.0.5249.199 Electron/21.3.5 Safari/537.36" etag="-HTyYpm7KikI7TLmpzGH" version="20.8.10" type="device"><diagram name="Seite-1" id="tXkwnWAt3AUjisLzzv1-">7V3bdttGsv0aP8YLV10eaYF2MCOQZkROwrzMkiEFJimLPhQVEvj6U7uqG3dZsi3bzLhmTRISFMFGdXXVrkv3fuGffdi/2Vx+fJ+sr65vXnjO1f6FH73wvOPQo3/jQi4XToJjuZBtFldyya0uXCyKa3PRMVfvF1fXd40/3K7XN9vFx+bFdH17e51uG9cuN5v1rvlnf61vmr/68TK77ly4SC9vuld/X1xt35ur7tFp9cGv14vsvfnpE88834dL+8fmSe7eX16td7VL/vCFf7ZZr7fy6sP+7PoGsrNyke+9fuDTcmCb69vtU75w9vrm7Wb074k33Jz/Hvz12/qPf/33l0Du8vflzb15YLrB5SZ9v/j7ml5dLe5WZvTb3IqEHuQjXqb5zeL26nrjv/Bf7d4vttcXHy9TfLAjTaBr77cfbuidSy/fre/pL6/O35UXLtNVtsHV8f2WbnNtrt+JAvgvfdfxg4CkGZ4c0ZMd0Wd/rW+3RkFIO+hv6ecWtxm9xbuuOIyE/r7ebK/3tUtGPG+u1x+ut5uc/sR8Gh75L71QvmXUNXBP5P2umvxjc+P3tWn3jCAvjbpl5c2rGaEXZlL6J2j85z77M1tk/nL87rfZb5fOJDz/5eSkZ4ZaE3J9ezWAqtO72zVEWRf9RkRv5bvdrFfXZ+ub9Ya/65+enp29fl1+YnUcc3q9X2z/MN/D6zkk/TI076K9ETy/yc2bvxY3N7W7u+8u3WvvU7Nzt77fpNefEIBdPtdXjYXancTaDIVOd4bstc31zeUW2t2wFD3TZn7h7XpBQ6505PTkZXDU0BE3cJp3kWcyX6yvyNa9jhyncaOwdZ/t5Sa73nbuw3pUPviXq9bxSVeTSMgX5u16s32/zta3lzfD6mpLnaq/OV+vP5qLy+vtNjcL9fJ+u27qo9WqSpPmtU/6taqjtV50DOG1tVY0/HKzbS0HvvZ6AfHIzTsLhq7UPm9p8ZFzeeoef5UWHxuzIlP6iT/0wyeq+5P1+KvMz/Gp6sjh6cjRQenIiaM6cng64h+Wjrj/FB2hKdnkJezBmxruwdvqa/zuZ9St4KB068jvC2KuLreX9J+XL1/+PAEMacGj0ctRDzb2Tr7R1Hg/ybKvL+4vX4A2v/HY+vMOavn1BaQ6x187x4flvj1f5/j55/iw3KhNYekcP+ccH1Y4b7ObOsfPOceHFY77/7C03nfGTk9NJ3+fybIQvwpdbtbZ4vbl+4/py+39L1eb67ur69uXV9edOa1mzHk8fHmG2CJwjl42ayOnbie4KFPh9ejCd76V8LyO9O7u3nckRQ+4bYqjGaub8LoeUJtLlzeL7JbepiS2a7r+CuJapJc3A/PBh8XVFa+fPvk3Z+g5piBwW1Pget058PsCvG82BUedKaD7fNysl9fp9q4zFf+zkbd7etqYGP/U6UxMb+T9reqG7nHfxLynu/88k+KdOgc2KX3FXM1Ufc7UfKtMlXvaMzVHN1sjjMa0HP3f/dp+8IuIcUB/4Dof9ywd+zm9yuS/tZYKe9d3m6+65+5y8+G/5q7/RVOLuS39gozX/vTPolFHQVujvI5GnfhdjQq+1WIPfmysVUPhjzVtlLFWLbya1z97INbid2+vNwsSGOBRD6jnPpJHKhkP96Q8qBZP7hp5NB6wfvJA4gHbLvd9leY55+xLFPB7zPPpc8/z03p6/Cc2Bz1XU48VSM2PZbsX4Sv3F+9FGL0E1uDgsgdsfFZEuVlvSUxrBEG/nD5X/5138vK4GeB4fnXpsTjT82x09PzL8kht+Y+z5U9d4oeVh+smJzi1w2sxwFp8d7m5vdxcHexydFuIygu6GL1/IX6rbEPwY8tQP/cy9P+ZiOrohyCqL9EZneavmGabjP6nTPPTWvCfo1HsO6jCszvefmzthSdNbH36fRvmu31mlUdndH158/H94cLr0i8fjD8Pf2zn8D/ONgdPXZAnB2WbQ82E/QN05rCip+7GxJat3dAPrQ/W1vrOwdlajZ1+3CJ8amf9oRnuxxbhx/XuenN6uKvw5NBWod39fvir8DAWztFTF85hRaPdrpi7Dx9PXi7vbxaHGx8ERwe3Wn5sS/LP7bP8f2om6B8TVH4DpflHpJV+XF7p6xqru/mfBUGhE8dxfzlx/GMAonp3j3TgHHhO6Cho2vywLKLWrb4Tdq3+N2vrDf6HM/atXbtXl9cnf6XfyoA/OfQ/LAPud2N/DjmwztwjUrvedXbw1dSTVmdD6JaHDT261L5dZ0PwY3eL/OQQ68l5gQPzhN3EAE0tsgKOd9q/PA88UxB4h+cHj35mIPvDl+Y/NPPgd1MPAKnHcJ7H7ukDzvPAk+nBQbrO464VXN5/zEmT39+/Y2F2BX1Ye/o858B29J30HeXStoGVNUhvLu/uFmlbAz954KHD/+vYGfcrzNiDM/G48Xii7fg+ZxsGx+0tnu0TCZ96smF3p2I7R/iNWzVO+g42/W6K9OUKcfzU8wK+j0Z4x8+jD96x32tovpc2dC31T7ZR+Kibyv+u+4RPurDk/Xb7sbtD+H9mDrzAP6wZ6G4I3tzlt2kXo9ylH/93pyW0B3fbaeniyG81Lcs/N8GVG175V6//Mz47+/tiEo368HpH+Nhr+fHpz14ehn75zt7B+aRM2s0q7kmPrvZsnfSfIYjpF8oTvPdnYeU1dptu8XjH3YDRKGpXwJ+esMdV7keJr3uoSEd6dhvw4gOff//q8u6jnJ7/12IPob4yH0S8Od0fyFvv9d3f2Qvv1Z4E6529jU6Dd7/vsqsP/8lT7+bvd0tnkVwEu3jx6ubdh9Hff765uf+zcBb/+c9vb86n2SL+deTMfw9X89/3f7/7MDuNP4R/x8sTut/Zcl9cfUiz6zfu3bvbhD5571z9Ojg6z0/9Kz+9vyqS+3f+v27Pi3iXRIO/U//PW/oV//L335zLyFmMpiv86vrP329uL3+dnMbLmZOcxXLn3fyP39bxmz8/vnuzO40XiZOsXr/5bRYv4jf0lx+cLFmm3vlyth9H8/A8Gt4nxcwdT2fZPB9sk+ngPokSfxSld8kZ3mf3yTSlv13djc+C3aiY50nevM6vowm9zrJ3Z4PdeTTIkrNdMToL/GQ5343OXi3jaHA3vgjC0TTzRmeD4Hw5LEbRJKffp9cDZxTNiphfD326X9i4/maH++3iKKFxD8KkSPb4DRrvPqExJUXqjC/wOX8/GE2H7vmZg+/T6yxsXd/zc0QTN8lf3WFcCcZ85nTGez5d3Y+nc3e0nGXn/LwTf7Sc03UZG40ziOV+9Oyz/ehit+2Txfm0Oe7rxattHGV0j3Q/ovufRzPnfLlyRstJEL9J6O/T+9E0CUcLeia6f5IH3mg62Y/Odh7kM1rGXus6XvujYuIki9ozRTH9/SSkeXZxX5Khf47nWw78OJrT8wwd+j7NX3I/KlJ/PKW/wWtoU+t6Oa/LFN8Lkiim34CcMNYJyTPDb8qYopVP8vTMXHok/4J+1xlPU8iOXqc01lVtXiHrzli3ozwgvcz2o8VgO1oE+yRa5ckCMuZxkW7EmdyvdW/Sq/LeZ05O99nTOHYJ5PnrOnt3Af2dk/wnNLeZa3SJf490yeg9yyFPitUdfht6j3HUr9fHUdN7+ruAxkN6fiF6RPf1kojkng8gkz3m9jyqyUHk4ybL1vVSPrHP3yN5mrHS+1mBeeTfrM3Z+XRu1uW8df0r5E6yG+W9cndH0dA5F11s69/W6IpH44GuHMVvnIx+n+4x2dNcFLR2i/F04ojOD3e0Rgr6zJc1U7cFWFN4pknrulmHBcZePVNSTOg3YrIvQ8iK5oXXMf3WMMNaTXicO2MHUjwnr+FRMQzIjtSul89SkBx2tA6KBHYGcirt5Ir1vm4D8P1xlOSji0GfbaCxz+v6su2M98wJyR7gGfcx2zFap8Xcq2xXQrplbXDmJtNZZZPqsoBNq49bdJ/0L7un56PvQYdJX5ZpmMhvb0cX0NW0SKztgVymMdnKHdmoGdnQ+b51Ha9zWv/Q95rtGYawbwn5GaNXLut4lOZxNIHNIH0f3rEcSe8xt/x6OstpfTSuV7YHn6dOwrYGa2tG+j0o6G/E9pgxjaDDZ6TD0Yr0rHm99758HXo5pN8u5wDj9ZJi4NDz4tmdccQ2hvQ0dni91ORQ3XsSwF9Utj/esc2OyOdZ20/zQl4d/oRssXkW+b0dybcw+o/vubLGIa+MXg9b12tjqet/+1lYTjPy3XEGuYym5FfPZN7IjgWjUnawUfXr5Xzm/PkyDWi+Wf9ZT6axsT88bz6Ng+TDYw3ps9Z1Hiutwcxp2p+OnrAOjpYJ+eEBz+MI62fRlG8phyLjddGjg1vRlzn9flLaH5p7+l4SwPeRf6TXZM+Wmf8Ev0tjmgW9fnc587p+F/IW2yZ+dxaOp7D1bb87J1lNevyuXP9Cv+uNp/G+x+/S9aHTtf+NsRr7vxL5d/wu8FCv3yWZzJ/f75LNGJ/1+V0ZR8vvwr44Fr+Z5wiTrt8VOXT9bks+n+t3ac5ojfb43S+Te8PvNuTe9LtN/ev1u/SMJPeYngXPmtA9kpC+U9mdIsnrdofWY2HsTk5rtnXdrOVoFfTYnSIpsrrd2WPcZh3n5IfMWqXx1Wxz5UeG9WcRu1NMduOzmt2JErJNjPftmCAL2Oiwda3vnjv2oxeE9ckvjkusz2P1emy9PGevrU8CyOYb2PqGzGu2/vNlXrP1TZlXtr5X5k+29as+O0+6X7MFdTtfJEHDzhMWkvsQHpjOH5B5GlT2v1fmHvmBooqthk7CPmSGuJDsNDAYYaOcZV/GjKNc7HwN+wFDUbyTFq3rBnfN83ED4ySkK2kODGKeCbiN7HBKeghslgArm/iYfE4BWbXjZrle2rGC7Ei0ovkcGjtf4uJQ5F/HfIyLC4tfmzizdd8KZ2L91HGmZ/S+B2fieftxJulX+AjOzGUcFNfCDtEckx/2SpzJOraSdXzmwK4SdoVPJltH6wd/J/54UsCOVjFUA2OS/OcFzyvsPWQAm7kYwL75NJq6D6j9Bu67ChtxLcZH8z6eJkbuE9ZpkwfBd3PSIfhXuvdkR8/miw+OaaywFT33ZDtBMl8mRanz1Ri3ZLtpXgeQdxWXiyzsb1XXK1lDn0Oy1yGvT+NX2R/A9uAZjK3EuqI1Rf4qC2StML4hXzS7o98Gjt4lF9b/zkPSne7vsY1pPAN+h/DzgOL65J7WUZEs53ldps37GdxT3g/jJJ0DvjGxlOgCjRM4ge9D9r1IemRdXu+5L9uYhj7I869ofmBjeKyEH1K/R95kb2gd5eyD23q2Fd2Yk59d1fI3cYB8EV0nLJndkS3bETbdj8S+yz3JntUwDeMBwTTxfty6bnx8XrdtFaaJEYs2cgn0fcE0yB2ZnEFSzMpcAutD/XoL09D85TVMQ7owCSpMMwlZFjxWspMXDXxpYv2knT/rjJX1bTnAWPtwpPVxHRyZLGs+7tlwZEPmdRz5+TKv5W/qMq/nb/pk/hk4kuzOrDd/07xvhSMJd3o1HEnjHcB/emauq3kju93WvU/Lfb4f5TWbXpBNp99i+VF8RfiC4szZU3IHIb32e3MHyLF2cwe7cTSs5w5ofEO3P3cweyB3MPvi3AHynn1YkuPiix4sGWX75KKOJecU02YW2wSC4zGumQd51fB0hVObcti2xsw4nuzYjsdK9rey8/wstBYSo/ssC8zFHedBIJe8eb0+lprud55FYnFaCxcDzCGti4lTYlHCBFZGDYyK65XuN/AZ6QnmkmST1uQu+K4t93ps/4Vyr2NK5PHdfkzZlEM/pkxc/B7s2WjBciJ8NJTf5pgBMXfmVH5WcIv42YkzNn52FCXIG7D/STCeHj+blHlL9l/k/2gNMFaguC1imwp7lLXvh+stP0v2YmXl7ohMJb/KtnJp4kuMcTqzGKmNPbxeH1sM9k/BNInIvHPfxrM/K6apZF3HNF8ga8L1cYU7F9Zmr9z6HDTxY4URjLyRx/JKeReZD/8ELELP5lZYyV7vuW8NQyZVfrjCYIyRCOvSPxV+IV0vZdC4d97U8xYWE7kTtodvGCJfGBiMTVh+ZmpTCWJZGgvnvdnm83Nf7PJzrCNasya+DQk7uByPFjQ/UdPOoG5KuulaH0sypzVJcQ/5GLYziEPkuzl8Ye136N7AnJPafA73tL5D5MWN7AvUF8nOsY7wd5GTI59G9/YRA4j9z3K2O7AjRey07Ut7jPLsc4o/V7ARpHeoocZid8wzig2q33ceNGz6dEj2ZoU8fc2mz3ZkT0i3ZjTmic0R4BloXSauyUeyTEn2yFvSbyOvOTe/tyJ/MLnr+T340j3iR1ofFh/gvtC9fcz3IewQwW61ZT1E3uAhWQfQjZHJD/DcRTHyCPhNjDMgX+ow1lvApsNGNa9/ucyhe4O95OdT2AHXyKZ17z6501qgf8Smx3+/zdYn6ZvXzuXZq9Xbi38hb5CLDxoi90GxMOLeOB+zHCn+Jj+IPAs9bwh9YpwjNpFwVkbzkUL3cl6PEc0v55gQJ0FmFIdQfIRaFtkzwtcDwlUsgz3joAVyLIThYIeKSc72bBFQDJ84fI3WI2pmdJ9dwrYlprU2CNluLed+ws86o+/S33HcYeXHcyz1Irof4Ww8G+w03XsQGoxIeG4WSO6YcAU9Bz0T2bOM7CD0Gz0C81Dq/nPod5hIrmcvNRKuy3mYg/F04Jl5hO1zjPxh38kfZg7WBem6M4oy4Gb6TrKjMfnGzwe0/gv46jE+4/XKuSToJOncai8xIa0X0VN7rRz3O7FJhAEGro0vz7lWR7ECvaffI7+XZpJXzLgOSHFELvoFH0Ax4pJxE9n/VNZXRH5pOrwTmU1846+wNkk3TL2EfC1fh+9ATkDmlNZOavNgW5ob+Bg/ET9Z6ZDY3YK/SzhqdCF6RfEW5p18H2RC+sNxtviE2md990K8RPM3kFoG3wtzOClGYoORl3Flbmguionx5bT2uFYCXZyg7gUcRL6TbCd9n3QhZB9vvkO/R/MzQS2EZUw2iPQFWFX0Wr4/h410uc8k4nomyTuDjvI9SYY+59QEz5Hdx3s7b/g8dcXGYswzX75PNjEi3ef7T3YjiSPvyR/uTX+GuS+PNSdMueOcLda2XHPIDngJcmUX+DzLZawTiruG+bmMMSzreKSHCWOc1JU1VcmPXxdpaHKLyNXtEZuwPMhns7yXg9DkRmmMZPP4Wvlb7DeRk2O5RqnLmIxlSD43wn3IFpvvkwwC5HBLGfJYh5A7xtqUobGftI4N/l8BK9HfrqyPydmnFGxPd7yGpOZG9n/FuVPkN6GvtKaArSh2hA2iuS9SmfNC8oK83guuidJ9aF0tB5zLI513E5E52ecE683YrqHFA8DshdhY5BbngcgCaxVzPzevYWey1mvkHCc2pmY5J7zGyc5M+XNgTsRJsL+IZXHNRy11zHhlBb/I10iurKMjyJVteLpH7kiegfzGBfwX6m2w4fxcgeSCEauS3yYdkLWJPCxhKtsnALtOesu+kmNX6ALH3PRbKebF5boMzYvYtgTfl3EWqcsxCuOXQQFdoO+GiPMknxV7oks0B+QzBBfWZMtzUcpwyzZgShg277UZpB9pwLmYCD4SuA9jMX1hC+j/EDky2NC95O2RPyFdQx8BbHE0yyUWSsh+rPaiKwn3QlW6MjFyTj2zFmFTpR6as48IjQ7hmXCPrr7QuiA7ZOusFJPWfLLk3Gw+CrUYp2/N0jpzTOwGvdmxnGH3CvQnIE4ekE0v1yzW3INrlvBKOL5ordlpDEzOv0V2A7XYe8RFdu5pLUkeB9i+WBEuz8o+NLH1qcHnZFuWs8DmuNAzA2yciI3YmbUF/703Y9iNGIfidyd0n0mtBsj+NiD9ysv7LaBbM8Fq+K6NnREbTmPBMBQjCMZi/Q8kRz8UPUIOFK/ZVmDtxvLb0SRnLIs5lX61Sk8NZrGYsra2MWcejV9wUMS9XPX5JF2ZGVtNvo3z7g/YYjvPS/ZZlR8T277Hc9b8mODUpayxEXp3zto2GJhkYuczQE60aYO5vgR74Ir+zwrbB0n3QT+Ej3w52+JlXM519Rm+R7qM+orpbUTsSzHPXvz1HPVqyA3zdcd6ECVOpVcJ5gq9QXguwhEkhwXbUNJPtnG0rsXGEfZHL1qlzxJP7Kp8LuJf9Crw653YyF0l/+5z1ObNyn0utdWWT0S/F+rbRtcMPl3tDK6wPmJb4S+s94GLPiuxmeRbBMt6qN9yvSvHs8WM1QT/zmyMXXAeX3o7CZuSna7yCyH5NGALqUEx3ogRS8FfmPHN7FoIGP9xXJE4X7AWXJ6fH7UW5G8oPkvaa2LPfutJa0KwncxHkhtcQjJKjT2b454P4xLMbTHYJew/IHuubXLOhf2HkXfCcfAK8TP8zI51l3xLQvHAiLE6etyGouPF0Bd/Dx2fGB1f5YJbVkXCfq+OVWGzy54RI58WHl7OUad0k7yLufHZCDWHhfWtMlfGLtNzxwZHrRxrg2kMEm8tkIcdNOe+5q+Nnu8Z+1l/z34m3SN2N7Uw3+o5/BqtwbaeN+Rc9fvxM4W2ng89R03NyHifCB5ArTMcMVaiNWbjoYLwO/yA+OZAxp8CG+dtPz6KYuDsQvDB0Oj+DH2NNp7aV3pe4hKutxAuCiq7MvwSH763cQLN0T6WOd0ZedI8ZqKz+M40MXYHNo/tIMWCcW7wEup2mcRANBbxqSHpUAB9bcQYX+XDKSZv+nCyLQN3tKj7cPkNztdXuSJPav3zgNZJyLqL+cqbeKiuk1ITQ0w8dGzMh94E8xp42BN5DaQ2gOfl+i7rL9bnXnxE5o0rvMnyLueV7cHKh46LDs0d9CsK/l2hNpGXuPmswrx0n0q3WDcMfo6kroeeBZI5bG1Z72DMMp2bXjjBLBKzDMOkxNYJfHRpp9me2d9hDJntYT/FTtmYgzEs2dW4wtcXgzoW3bb0eFuTn63FsfzLuGsp/tzoeanLNb9q/bCNYXiezm3cJ7qB2mnA9r/CY6X/Ti5sHD6rxeGc+4Stlv45E9Pz2rpAf/KwHjPv27H4CHEx8kqcD2hhlVK/5/tx2XMN3z40ceaMc6Mcu5/VY/msgF6VOj6VPBPNg4O1ajCog9prPU+QRJ2YXmw/8k9mzY7YXuHZM9gSuRdiEmOHZN1zrg222BlV9YAyn9G1MZhP1N8mjfmsxUyoCQGbeISzwxqmRsyDfKDBaZO9yIzHEAo+IFlMk5qNJWxexpUGSyCGyHkt+WTTLJYIgUXlb+O8gZvQh2BwU39sTf50OmzYX6Nzxi+vHOMzUfcJJUamWEtwQfs37JrfIdcjvmDiSf4evW0pfHMZ/0GfgIk/sd6lj571G31Ttg+HMRlyvqHBwQHH0RTPk61xzPotTD5QchSRXb9sa/CsiHkKxnXNWBq4LDdxd874o/acPDc2t1rirLlf9/1Sn8nukbMeXwx6/6a0xdG8hg8mFh84Es/34ODaGIzfJZs18MvnmHJtzeYqfJPXxZ6KoI5Ra7aPe7ZML17O+yDyWo60QNwyCUwuyOTv0oL3OVhMHCU1vzT3aj13e/SAYQ8Txee1/BJscuwI3hoIdmesMTQ4rb0O4lYeQOauxBpiy7CXCr6d543k04h3G2vAYGPYP/gEqf1hvsjOyjqzOSSTV+DYkZ5v6Er8VK4JU59qrAm6d4ZYscRxkktCn0d9PcxIJ4dhj/9zqmsce0m/HOeKMEcZfKzNb+VsB6KB01gHmPcoa/k8wnSSPy/Xn8Qms/2ouwbyJEp6/V1/PukBvX8UD08ew8PV75lYiuz9rqvrqP0O7ZqF/Gtx37xa91VOOWAc19HzmbH1xj+gLlBMzLVJEZtxj0V3bQ8AeiD2qKPa2N74wYxjgyn6uVe2b9P4u2Ertpk5JoeGnuuHY5sqfi9MLtNB3G5iG6eRj7Lxe9nDy/4sb+EMxAZk7yRGfwR/dGoAHOv3YA/+7WjumnXp8Bq+qMYtsiWfuZxJfgL5tYdrADnn44FD8xpuKCbe6KKJGXj/5jS2ca7k9StZljZUxgpZx4XoImpHSR0TyNrhXGijtsH3w/7PGk7Ix9CNDk5I2Z6MynWA+Na+hpxiW7e39RCTu85sPipvxao29s4fzkfZmgDnoxyyKd18lKzbpkzrNQE87xI5JfSmJ42agORmjJ3hdQmMbGO+JBh1agKSt67lee/EzpS4pcqBlHHkQzmQMjYk/LUi+x6b6xT7XFj7PIDvYftl87Uj5H4XqL+t+FqZ/8xpHiP0FeHvsW+XMTDhRMRDMfY0epAlfTdE73WjjrpATDARrNKQFedPWD9q9hn94vwedWT6+9zokdRGcF/0fBVJzU4JPrF7k7mOHQ3LWITnnfNGyP/ZeDpuYhMbD9u8uOQBuF/E6K34vrIWyjXar8hTcG+F8Tmr/Ugwr8/90guLQbi2XOXJar7oHfe2Neyl5Ken2Aed0m9Kn2THvnK/57BmX+cmv13aV8QsganzWvuKWnwgMWlv7qi0r2btsX1tXevkjth2LFq5I+yrxD699vVGjZVzfPvSH5o6w9NsK9cZmrYVPdHiO3zs7+M5MXWGBHXwIjWYL0bdtBhx/cXEKRfA3mlua3+J2NACvXEmB9qJxaq83sr40rnJp3Cs2a4P98VggnX5uWq6y7mcjv30pBdgYvXY+qz2a8lXl/rNMfHW5nCMfSjrbmPkqXrrObGpwSb++OvsLfI7zqhdg7X2tkD9uo6R0bsyKeehqsEOUc808c28kfdrYWSf+9qlfuk0bWuJObs12CrfX+YwbE9F43rZGzC3uSFfbHAamr6vQPJoTXueiG3jmKWWK6xymt3aatCsrTZ1tlXvLePML7S3Tp+9tTHUk+wtsGhe2Vt6pvCB+oetL3K8ENt5jmzcx3m1vvqHjcF2ybRjb5Fz8K1/qOxtF/uX9rayn+Ve/DF6J21dISr11OZky14M2Ar4F8GkA0/qX8DRc9/EdhR/Dpq4DjEiyzvxseeplo/C2qNnZUwYQu5Vbja+K7FPiQcEQ8lrYxs4n8H70VvXa3nhqLQFz2tzTS3wiXhWbMGijmeTCruWPS0T2B/T00LxtvSI1/LNRhfPTP4Y2EZ0CvlGE6+gFp+08Szbxp6elh338UmvRcOeog8J8W9vr8sna61DYwcGZEN6bW3RqZ0vM6/K4bHPxbkEtr5d5t1tjbORdy9t6byVd5/X/Z13Xq91Y99I084S3kl30iMieaR6PNmqe4jdiGp1yMpuNDDwE+oeXWxL352belQd13LsxLh25vTg2sDg2h3vGeR8ZOJXuHZmcW3eg2v9T+Bap7Kzs9DUHhH/5uiJNLYTe4/L3F6JVZs59nrd1vbm2F6XZs5t0ZtfL/M2sveQMAb2nPbj2vC8itvNtTS38dRYYpkuri3zDfOgrL9Nh363/paEvF+uintaeTWb/5deIMkZTArpL2zlxGQ/LeZjb/rmUd9l+zHG/mbZI4lcR1jlUcscN+KMO+mVHRr7NUe+RL4/XVl/z/4bNo30yTE1wlpOrN1PU/NpzdwxekRzE0/U8zycL+A+EevbBBfsJXc8Cc2aruGOIfK7XF+jdRfKWuKeJPbnFhdwTXNR640qr5k82ZRtmd2v7I7Q77vow1vAyKk3juZW/nvZU5CVeGvUj3s/1VNo44cKI4oPCdjPia0tOvkEqTs4j9cdKP5Z0nwVqZWxMzb5KvI9IuPlMDe5PAf9fnX/Kb27q72pbdLnaWBtrOSN5qhVOmat+MihNXInpo+kkzswOIBjhjIHyfUbf8z5e6yFYTenlbcxwKSQPKORF+vsJO/BAEUTAwxbvj42+60nj2AAyRWw3y7rabEr48tyGwN3flvmVOQXmb4Hk8ORtWp6gRaN3FM9FuCeS/Lx2AvRxliyh4h8neR76zpt6x6D/Vhqyg7nrWTOOna3xKx81lnH7vqmZltwH66xuxxTP253fVMHC+Vcraxjd817+AfT+0k4cBk/Fdei58Q3z0Q2MZUeHLKJsh8UmH21G/fb3KCKa2Pk6IvzT/fIob6+q+oY/fX1kWDZXbn/Xux1gHqYwbmetdfYP5WYGob0zD9Yw6jFZ2UNo94vUdUwTLzxRXU8wQYGb+Ecg+xO/F7SssG1WoX4vFKnOI+z5L4qjCvA33XzfBxTNGOEjt1t5cPF7gZVjn2+L3u5iyywvdxJcy1tyxyG4FvgxqKOF40/DAnHG7w527d6ucse9Qr3lrakmUeo5286edz6a8bZtTrHLLe5XdIlrhWJrzC9J9N50LTPjd6Thn02/lJ6T2zMWPA+y/7ek3b+Nqp65ht1B/i2Mi8WP2Sj0VciNppwnulTd/EsorvDcNSqOyTQP5PHkf3Cw6BbX6jsMtu4RdMuE5bwKrs8zyWGGu5trIPeGql7zPaS44L/Q/13aHoVZ671v3J+g/Rzx2XMw7V5W9/r1hnk7AjEcq7kHCp7knT7uQv08wrWQE6A7L/JheLsn9reE4430Ach+Bz1VlOni3g/rq0tmn0uqx10tFZThI3BntK8ZpOx3z20e135vIqHam9lrgpxYtlvaeOvQPp8+/IbxlZGfHZKGwvniLdKLMx4tNmLVtYye2s5A4l9TS9oWXtDLbew/YtcczI+dFDVlwmrkz2k55n5Mp8NP9rqGRjKPp4oDhq9szhDic88ZTte1Ht4xY7X62+DAn1AjRwg427JZbXwyr5Rg6v3Cnxerljqv4wV2vty7Loc2v5ZsTHR6pGeILs3h3vy805PEH0P+bGn9AR15GywNHqUra+tenrmODvCK+36cvhpPI09JcuZzSPb8UnPsqxju49E8DTr0JPzyI3aZ28f4aLRq+y0anVl/1XfHh6z1uv1unBc4ut6P6H0X5U95J1+WdG/sp8CubGCe99sb0yjli05udj21AJX7Nm3y5kYNR1Psfadp+Tk+MwwzsnF36j+UeaRv3CPGWS7qtal1HQLrM3mHrPaMzd1uNUPYOr0Zy39xR53Y0PJhuWmNuKXdbIuLnF6cIk7uuirgSTfUnerXFy3xzuXWMnINm/rLfcN1ms5O873ULxmY0XOXTXlgFhxH/+6XpwXwb/f/vrq/dWbLPszchbcH8dzMsOZDHfo2WEsXaSF3ctP6yegv5E+Zj5DjO1Ezj2zxTyUHA35DPJHZn+lx2cOXch5Ebw3X/YDYJy7srdQzvYpZP9ybM4K4P1KeSLnsLE+yBkwQ9PnaPaiy55K7MULTIyPPJVrzkrCeVkuY17Zhw/fZfpw567MM/ZeTgrGYJyjmTljswcZuUbOwefAR9IvOWI/jzM2sN+a5BOtoFfYi4V92u55eabPvHVdcpz0XdujELJ8sK+M8PkY/QrTmVvVoTJX6jGz8jyqBL3gsDFyLpzL+6bNWSX0XfRl8tkqfMZCeSbeivdf8m8WnFvyOW5m27AyOTD0uSIftkKv0d720I+nc7PnnPSw4HMGCosPJB42vesRzrXj/dbAlehpx9oIuT5MuAdn+mJ/ce2MTPgc9EzidwLeO0HjQC1jLHuP5WwyOdcbe5VRI8qr3HXmjWqYflTWULCXdm7i/pndbwzbTP5y4pmzGoHZfM7x8JktKx91Cd5ThzOBYJMlbuE4Hme1VGdZDM2ZRZWOjYEBTZ6Z9NKVHifZQw4czzUynG3L/eU412GV89lSnJuhOE3uwefQ8Rl/cl6p17puahOr3O6TH/O9sG+N5maZMcYeFWVuPRjJmYlulfPHOk123LvCOoI4ZN7UET7LZWZx7p7xLf2d1PcSOVNjGtu9UiSvGcf8Cdv7CZ/HN+Kzngc2xpV1HGEvhMGR5brD+GeIa7CWgRn9hM+dytD3FpizI8vxs+6zftpzddGfirMbzfgLjt/NuuBeNDP+1PaXFSLP2K/6nJHvSRzZozXYyjmTKewF1jlsHWMePksmQv2E98ND9/b12iJ600eF1L3J5nL+XPokY47/0RNrZBaOuK4xMPmBhi7v+OzAM97rSr8tff7I/XEPDOocUQqsn4kOTzAes9cZ54kg7zTZ05rcc/6V7QHON0WOhvdPV7YZPY9LnKWVcv13xHhpnoldTk1PxsDHvpJEzj3IpXe8ZZOjWcB1EbOf1s4522PukcZ8w/7jPJa59JdH1Toy+sDnn/M6ilIvqXKaYucuAu7HHMveMWuHfakFc64fdjiU/FcGnQpNPo3PKsM5/s3rVZ3axmqylia5nJWOOYtxNp5j5iyw51+QHpl9HUMn4XPwSMYXO5ynyDks1MtLO16dN+EnbOfncpaXrFV3ZPe+RvXP7f1jV3L4jEtdWcvD/ajyeTs+I4nswTn7GPSVJr6pKeMcy1DObmN7b85LmGE/MWqIoTmXgWQwkDOW5NwPb2z34k1ZT4xvxxksfDYP5E/zRjZR1pOL+hXNL8WTk4Y/xlkZI8RpnOeQGuGodZ3XP/bV2HOBxVfSHE1C23tj8rfYm8N7B8dsM7F+4VuGiKcCc0aFuQ6dnHvSo4e+j/LcNYflUsgZgtBz5N46OEJ8fsB7yQXrEn6JWUbYD0Jjs7makP3ZxasIZx3iTG/kVlBX5PnFvokizaAbfI4E53FW/H4U8X6LIinP2EnMWV3lmXDAkD5ytjg7QOyQqQ3CHxcrORNjit/NEF+7UvNJ5cw20ukRn8GP/XipradiHQHj+LIPR/ADzpcCPh9zDWWWm/Nme64bn8N9avNynJAD9uzL+fbAyKiDJOzvE+RJOecAHZoAnwE7wL+SjAamBjd3uF4SDQKTV78jf4ZaKGxBNkYtjM+Piv2GfeFcCGN+Ezule86RIs+CPjzpmXO5/kjYmGupFBdgj4Y5iwZ5kr30ZKfYO+2f2300iHkXnMOHrc0FX804b1dyAWCPKvcT4JwxWrML4YnZMffL2wwsNvj/N2DtyRtUSnUOH+/l8XEPjc83o4bqMnZ9f2qok0Ojhjo5YGqo4wfEfjjUUP6pckMpN5RyQyk3lHJDKTeUckMpN5RyQyk31EC5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhamNRbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZKlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbKrNnyik3lHJDKTeUckMpN5RyQyk3lHJDKTeUckMpN1RvDHan3FDKDaXcUMoNpdxQ9Z4W5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhoqVG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtK1rxyQyk3lHJD3Sk3lHJDKTfUZ9P2HAw3VOD8eG4o8/nhcEMF7uNC+VHcUOWEHS43VOApN5RyQyk3lHJDKTeUckMpN5RyQyk3lHJDDZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBuqNhblhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbqhEuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhMnumnHJDKTeUckMpN5RyQyk3lHJDKTeUckMpN5RyQ/XGYHfKDaXcUMoNpdxQyg1V72lRbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuqFi5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaFkzSs3lHJDKTfUnXJDKTeUckN9Nm3P4XBDBT+eGyo4OjRuqPCAuaGCB8R+QNxQR8oNpdxQyg2l3FDKDaXcUMoNpdxQyg2l3FAD5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhqqNRbmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbKlFuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbqjMnimn3FDKDaXcUMoNpdxQyg2l3FDKDaXcUMoNpdxQvTHYnXJDKTeUckMpN5RyQ9V7WpQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBsqVm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKFnzyg2l3FDKDXWn3FDKDaXcUJ9N23M43FDHP54bKgwOjRvq5IC5oY4fEPsBcUOdKjeUckMpN5RyQyk3lHJDKTeUckMpN5RyQw2UG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbqjYW5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm6oRLmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG0q5oTJ7ppxyQyk3lHJDKTeUckMpN5RyQyk3lHJDKTeUckP1xmB3yg2l3FDKDaXcUMoNVe9pUW4o5YZSbijlhlJuKOWGUm4o5YZSbijlhlJuKOWGUm4o5YZSbqhYuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbSrmhZM0rN5RyQyk31J1yQyk3lHJDfTZtz8FwQ4XOj+eGOnIPjBsqdB8Xyo/ihion7HC5oUJPuaGUG0q5oZQbSrmhlBtKuaGUG0q5oZQbaqDcUMoNpdxQyg2l3FDKDaXcUMoNpdxQyg2l3FDKDaXcUMoNpdxQtbEoN5RyQyk3lHJDKTeUckMpN5RyQyk3lHJDKTeUckMpN5RyQyk3lHJDKTeUckMlyg2l3FDKDaXcUMoNpdxQyg2l3FDKDaXcUMoNldkz5ZQbSrmhlBtKuaGUG0q5oZQbSrmhlBtKuaGUG6o3BrtTbijlhlJuKOWGUm6oek+LckMpN5RyQyk3lHJDKTeUckMpN5RyQyk3lHJDKTeUckMpN5RyQ8XKDaXcUMoNpdxQyg2l3FDKDaXcUMoNpdxQyg0la165oZQbSrmh7pQbSrmhlBvqs2l7DocbKuzQ9lxfZdcX5u16s32/zta3lzfD6uqriheJiY/Kvzlfrz+ai8vr7Ta/WBS4yeX9dt3kSrreL7Z/gFbpZWjezQ3JEl5H+/qbvPbm7fVmQc99vbHXbkkGf9TfzKu74m11K35n79UkZ3rh+aenafrXX+Unv5sJ8fmbV4PNZr2rWJzoyusFxMz3epA06259v0mvPyV7Q9i0vdxk1+a74z/32Z/ZIvOX43e/zX67dCbh+S+emX3MzCfVanN9c7ld/H3dGMfz64zfw/30+nKTvsdPe6+3oH16iAwqzW8WpDwb/3FGrXeiZufvyguX6Spj5Rvfb+k21+b6neiZ/9J3HT8ITrzj8OTI85wj+uyv9e3W6KHLUw+artvsscl7who2n4ZH3suwwW3mn3qddX1c/lF9WXsnXz9Fv72+XJ/Hb95d/XH5wXt96y7cIvvF/xlX9WMrtWfVn529ft1Z9d5TV3Wv6N2DX9SfGnZzTb+nm/93fXP1E61mp8lT2LeWj5yelRx8/bycvb55uxn9e+INN+e/B3/9tv7jX//tmZcPa3qcF+Erl1Z5GL184R3dbCHODb3K8Ap0ei/ff0xfvnz5dZSGm/WWNG99S29/OX0uAR/7DQF7YWkZayJ2nR5r+RwgqFfGXerC72AtPx9pPGy/Sstbt7ulFX7A8pZWtjKs8/pnD1pZetc22U8yl5/S70fNpXv63ObSfPXtekFjrohKnZOXxy1/3lY8eSjzxUr3OvcKj04bNwpcr3kjeerOjViJy0f6cr3+Idj+R+v1V6viN9cwP2wDxi9WML/trZzvqmBHPxRm/ihj95wa/iRt7cePTzWcx99HrT3PeR6lJuk0rabzfZXaMnx/gtm4O99Xl3fvS6WuKSuuv73ckvLc8hXP8XvjjSYua8PcB7Xk0UmtYaqwB7baa18598FpE9T5Nof1uXPfuVHwXB6T3m7W6239zwnvvk/WV9f4i/8H</diagram></mxfile> \ No newline at end of file diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/architecture_2023.png b/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/architecture_2023.png new file mode 100644 index 0000000000000000000000000000000000000000..bc1083880f5172240dd78f57dd8b1a7bac39dab5 Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/architecture_2023.png differ diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/migration_2023.png b/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/migration_2023.png new file mode 100644 index 0000000000000000000000000000000000000000..4ec6dc7bf0c9b37c02c3a5058fd998e6376ec42b Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/jobs_and_resources/misc/migration_2023.png differ diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/mpi_issues.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/mpi_issues.md index ccb34da378591594991ab746915fd90e9847920b..95f6eb58990233e85c5dfa535e0c1bde0c29ade6 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/mpi_issues.md +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/mpi_issues.md @@ -2,7 +2,33 @@ This pages holds known issues observed with MPI and concrete MPI implementations. -## Mpirun on partition `alpha`and `ml` +## OpenMPI v4.1.x - Performance Loss with MPI-IO-Module OMPIO + +OpenMPI v4.1.x introduced a couple of major enhancements, e.g., the `OMPIO` module is now the +default module for MPI-IO on **all** filesystems incl. Lustre (cf. +[NEWS file in OpenMPI source code](https://raw.githubusercontent.com/open-mpi/ompi/v4.1.x/NEWS)). +Prior to this, `ROMIO` was the default MPI-IO module for Lustre. + +Colleagues of ZIH have found that some MPI-IO access patterns suffer a significant performance loss +using `OMPIO` as MPI-IO module with OpenMPI/4.1.x modules on ZIH systems. At the moment, the root +cause is unclear and needs further investigation. + +**A workaround** for this performance loss is to use "old", i.e., `ROMIO` MPI-IO-module. This +is achieved by setting the environment variable `OMPI_MCA_io` before executing the application as +follows + +```console +export OMPI_MCA_io=^ompio +srun ... +``` + +or setting the option as argument, in case you invoke `mpirun` directly + +```console +mpirun --mca io ^ompio ... +``` + +## Mpirun on partition `alpha` and `ml` Using `mpirun` on partitions `alpha` and `ml` leads to wrong resource distribution when more than one node is involved. This yields a strange distribution like e.g. `SLURM_NTASKS_PER_NODE=15,1` diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md index d3e9674ced87e57dc6f45ccafccd71383c86d921..20a542d3abed3cd59b299c5d6560bc451f3eead0 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md @@ -1,7 +1,7 @@ # HPC Resources and Jobs -ZIH operates a high performance computing (HPC) system with more than 60.000 cores, 720 GPUs, and a -flexible storage hierarchy with about 16 PB total capacity. The HPC system provides an optimal +ZIH operates high performance computing (HPC) systems with more than 90.000 cores, 500 GPUs, and +a flexible storage hierarchy with about 20 PB total capacity. The HPC system provides an optimal research environment especially in the area of data analytics and machine learning as well as for processing extremely large data sets. Moreover it is also a perfect platform for highly scalable, data-intensive and compute-intensive applications. @@ -58,12 +58,12 @@ automatically select a suitable partition depending on your memory and GPU requi **MPI jobs:** For MPI jobs typically allocates one core per task. Several nodes could be allocated if it is necessary. The batch system [Slurm](slurm.md) will automatically find suitable hardware. -Normal compute nodes are perfect for this task. **OpenMP jobs:** SMP-parallel applications can only run **within a node**, so it is necessary to include the [batch system](slurm.md) options `-N 1` and `-n 1`. Using `--cpus-per-task N` Slurm will start one task and you will have `N` CPUs. The maximum number of processors for an SMP-parallel -program is 896 on partition `julia`, see [partitions](partitions_and_limits.md). +program is 896 on partition `julia`, see [partitions](partitions_and_limits.md) (be aware that +the application has to be developed with that large number of threads in mind). Partitions with GPUs are best suited for **repetitive** and **highly-parallel** computing tasks. If you have a task with potential [data parallelism](../software/gpu_programming.md) most likely that @@ -71,7 +71,9 @@ you need the GPUs. Beyond video rendering, GPUs excel in tasks such as machine simulations and risk modeling. Use the partitions `gpu2` and `ml` only if you need GPUs! Otherwise using the x86-based partitions most likely would be more beneficial. -**Interactive jobs:** Slurm can forward your X11 credentials to the first node (or even all) for a job +**Interactive jobs:** An interactive job is the best choice for testing and development. See + [interactive-jobs](slurm.md). +Slurm can forward your X11 credentials to the first node (or even all) for a job with the `--x11` option. To use an interactive job you have to specify `-X` flag for the ssh login. ## Interactive vs. Batch Mode diff --git a/doc.zih.tu-dresden.de/docs/misc/HPC-Introduction.pdf b/doc.zih.tu-dresden.de/docs/misc/HPC-Introduction.pdf index 955c681758a3830afa88f053d7342809c9785b26..3d2e0beb8eb0ce083218e7fd28a7e2993bbea191 100644 Binary files a/doc.zih.tu-dresden.de/docs/misc/HPC-Introduction.pdf and b/doc.zih.tu-dresden.de/docs/misc/HPC-Introduction.pdf differ diff --git a/doc.zih.tu-dresden.de/docs/misc/migration2023.drawio b/doc.zih.tu-dresden.de/docs/misc/migration2023.drawio new file mode 100644 index 0000000000000000000000000000000000000000..9679fffb77fa898a7e01fb939c258ec991808b6a --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/misc/migration2023.drawio @@ -0,0 +1 @@ +<mxfile host="Electron" modified="2023-06-22T11:44:28.169Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/20.8.16 Chrome/106.0.5249.199 Electron/21.4.0 Safari/537.36" etag="Y2qdBmcmrf7lYsppJ_dS" version="20.8.16" type="device"><diagram name="Seite-1" id="-EQ8Qw6o7c4L1Kgc9zBM">7b1bd+LIsrb7a3qMvS+6hkCSXb6kDFXN/Cwo2jB70jffwDKWMRi8ABdIF/u373gjMqXUwYd216nniu61emLZCBGZGREZERnPL/75/fHTdvZwG22u56tf2t718Re/+0u7/T7w6b+4kMqFVuu0JVeS7eLaXCsuXC6yubnomauPi+v5rvSH+81mtV88lC/Gm/V6Hu9L12bb7eZQ/rObzar8qQ+zZF67cBnPVvWrfyyu97f2a5ycFb/4bb5Ibs1Hv2+fyi/uZ/aPzTfZ3c6uNwfnkt/7xT/fbjZ7eXV/PJ+vIDwrF3nfxyd+mz/Ydr7ev+oNib/5+Oefp91o++unXw/xye3q4de23OXLbPVovvB+9rh93K02yWL9S/gh+PXkl5Aun6zoMz5cbelVglfyVwv6i9DzWu/eveO/8n79BQ9x8j+P+FIfxvxHxc/P3eb06dv8vrmfb153l/dP36WzeridvXyX3f3D+9p7//W4WrzivfcreoSnPp9+6dyAJ8Q+tbNsu3lcX88xUB79+nC72M8vH2YxfnugdUXXbvd0A7/bopez1SJZ0+vV/AY3u1msVueb1WbLt/JvQvyL65v13rku/9D13X67Wc6d35zwP/ljuTPKTLIv8+1+fnQumRn2aU5Ds9+m9Cfmt+1TM9vtevfMzwdn8ZyYa7fOusn/cGYWbJLfu5jT9MJM678wxYNWTdzza1rj5sfNdn+7STbr2apXXP1QXL3YbB6M4O/m+31qVNTscb8pD8v8uNj/x3k9xWi+C81P3aMZXP4htT+s6Rv+x/3BeRd+LN7GPxXvu+5AudGP6816Llc+LiAY/n15PlVmyElwevL+tGkm+K3AD4PnZsJu87iN589I+8To6Nk2me+f+TvfaB4MxbMTaztfzfaLL2V1/NUnyUlND9Jtbunmv7TPeRl/fNhu7sjA7PILu83N/kASemYxt15ezD/16s2t8Eurt/3NFm9bF+/3W7yn/8zFe9q0eFePJKE51unuOl+zV/N5ckNL+COM9H/xug1+9Lptn7xh3caP2y+5/H/yRVwZ/6vW9fWN1zTOLe/UP5v/rWX5/pXLsh3+VMsyCFR3fz/d/dpJ8pPp7vfP6+54O9vHt//Nqvr0R6tqG6LRZfo9lunZP3OZhu/fMEnKq/RvTBnv+06ZeDXb7RZxadY0KJJvavNfO09C76eaJ2dN6px2yff/d7aNb/Hx/72q3H9trOubqfJWq0H8Nuj4YbZdz7bXz4UxbdS39WvQGPXl3zkh1urvd/PtlwUNnRMF/YaBzpubdhw3DfL1ydVJ+JUGOWy/MaB5+s3s9d9Txd4PsN6tr2u9365VW+1/pvm1z13Wq9ez/YzDlNvdV9arP2Rp/fioRXCqS+vNS8v/hy4t/5mltYn3mwekMP/5i+vHh/LfEhLUxSWTNPiHLq6gaXH9iLzaj3EW3//oRdcKawOwnh8g13S3n98bqffXN4v14mq2pt2B91v397Z1ch3pkwj2ZRGXRWlmvCt3c8m68TEJdk7XP0Cgi3i26phf3C+ur3m1N41pJVRDGz2rMRoqYP76iHnlETupD1guDHfA/G82YPVMNEqWnhuwj93f/5eMVjsMXxqt1vvwe45WPfXYieP5w362jusKrG4frme721yjOYOF659nexqAtYjOK6IZtgytXRP3kxJ+0Ro4AgwbZru99mqjYT7h82ZBT+IETMrD59uf7S3EDpp3FUNTu9HJmff8jcRO1m7EY5x/7b+R3axr1dp2zOYu2v9EO3ZScR6bFlr7e9qxdn2h3Sy2uz0rxnWMOc66cHND/4lm+C70bz2z8ff1oglG/Q2t+BUGKHhfXklNWaTva7fsjUtLoiL83e3sAS8X91x0+2G2e5CS3ZvFEdL5YH7R5ZXkd+RHWklfkl/aH46ooDz/3D0Lrv44JNf3/07j9urL1Z23iC6DQ3/xYXV1P/jy56fV45+Zt/j3v3//dDFOFv3fBt70j3A5/eP45ep+cta/D7/0797T/c7vjtn1fZzMP7V2V+uIfnPrXf/WOblIz/xrP368zqLHK/9f64usf4i6nS+x/+eaPsWf/fG7N+t6i8F4iU/d/PnHaj37bXTWv5t40Xlf7nyY/uf3Tf/Tnw9Xnw5n/UXkRcuPn36f9Bf9T/SX914S3cXti7vJcdidhhfd3mOUTVrD8SSZpp19NO48Rt3IH3TjXXSOn5PHaBzT3y53w/PgMMimaZSWr/Pr7oheJ8nVeedw0e0k0fkhG5wHfnQ3PQzOP9z1u53d8DIIB+OkPTjvBBd3vWzQHaX0+fS64w26k6zPr3s+3S8sXf90wP0O/W5Ez90Joyw64jPoeY8RPVOUxd7wEr/n9weDca91ce7h/fQ6CSvXj/w9uqNWlH7Y4bkiPPO5V3vei/HycTietgZ3k+SCv+/IH9xN6bo8Gz1n0Jf70XefHAeXh32TLC7G5eeeLz7s+92E7hEfB3T/i+7Eu7hbeoO7UdD/FNHfx4+DcRQOFvSd6P5RGrQH49FxcH5oQz6Du367ch2v/UE28qKF8526ffr7UUjj3MJ9SYb+Bb7fXcfvd6f0fXoevZ/GL3ocZLE/HNPf4DVmU+V6Pq53Md4XRN0+fQbkhGcdkTwTfKY8U3fpkzzbZizbJP+MPtcbjmPIjl7H9KxLZ1wh69qz7gdpQPMyOQ4Wnf1gERyj7jKNFpAxPxfNjX4i96vcm+ZVfu9zL6X7HOk5DhHk+dsmubrE/J2S/Ec0tknLzCX+PJpLZt6zHNIoW+7w2Zj3eA73uvsczrynvwvoeWieX8o8ovu2oy7JPe1AJkeM7UXXkYPIpxXdVa7n8un7/D6Sp3lW+nmSYRz5M50xuxhPzbqcVq7/DbmT7AZpo9xbg27Pu5C5WJ1/ezNX2vQ8mCsn/U9eQp9P9xgdaSwyWrvZcDzyZM73DrRGMvqdL2vG1QVYU/hOo8p1sw4zPHvxnaJsRJ/RJ/3Sg6xoXHgd02f1EqzViJ/zYPRAjO/Ja3iQ9QLSI871/LtkJIcDrYMsgp6BnHI9ueR57+oAvH/YjdLBZadJN9CzT935sq8977kXkj7Adzz2WY/ROs2m7UJ3RTS3rA5OWtF4UugkVxbQae5zy9yn+Zc80vej92EO03y5i8NIPns/uMRcjbPI6h7IZdwnXXkgHTUhHTo9Vq7jdUrrH/Pd0T29EPotIjtj5lWL53g3TvvdEXQGzffejuVI8x5jy6/Hk5TWR+l6oXvw+9iLWNdgbU1ofncy+hvRPeaZBpjD5zSHu0uaZ+Xrjffl65iXPfrsfAzwvO0o63j0ffHdvWGXdQzN077H68WRQ3HvUQB7Uej+/oF1dpdsntX9NC5k1WFPSBeb7yKfdyD5Zmb+430tWeOQV0Kve5XrzrO487/6XVhOE7Ld/QRyGYzJrp7LuJEeCwa57KCj3Ov5eKb8+7s4oPHm+c/zZNw3+ofHzafnIPnws4b0u8p1flZag4lX1j+1ecJzcHAXkR3u8DgOsH4WZfnmcsgSXhcNc3Av82VKnx/l+ofGnt4XBbB9ZB/pNemzu8R/hd2lZ5oEjXb3btKu213IW3Sb2N1JOBxD11ft7pRkNWqwu3L9jXa3PRz3jw12l673vLr+Lz2r0f9LkX/N7sIfarS7JJPp17e7pDOG5012V56jYnehXzzrv5nvEUZ1uytyqNvdinz+qt2lMaM12mB33yb3kt0tyb1sd8vzr9Hu0nckuffpu+C7RnSPKKT3FHoni1JX79B6zIzeSWnNVq6btdxdBg16J4uyxNU7Rzy3Wccp2SGzVun5HN1c2JGe+11E72Sjw/Dc0TvdiHQT+/v2mSAL6Oiwcq3pnge2o5fk65NdHOa+Pj9ru0HXy/ds1PVRANl8A11fkrmj6/+6zB1dX5Z5oesbZf5qXb9s0vM09x1d4Or5LApKep58IbkP+QPj6RMyj4NC/zfKvE12ICv2Vj0vYhsywb6Q9DR8MPKNUpZ9vmccpKLnHd8PPhTtd+Ksct34XdN0WPJxIporcQofxHwn+G2kh2Oah/DNIvjKZn9MNieDrKr7Zrme67GM9Eh3SePZM3o+94tDkb/r87FfnFn/texnVu5b+JlYP66f2TbzvsHPxPdt9jNpfoUv+JmpPAfta6GHaIzJDrdzP5Pn2FLW8bkHvUq+K2wy6TpaP/g7scejDHq02EOVfEyS/zTjcYW+hwygMxcd6Defnsa1Ac5n4L7LsLSvxfPRuA/HkZH7iOe0iYPgvSnNIdhXuvfoQN/NFxvcp2eFrmi4J+sJkvldlOVzvnjGPeluGtcO5F3sy0UW9rOK64WsMZ9D0tchr09jV9keQPfgOxhdiXVFa4rsVRLIWmH/hmzRZEefDT/6EF1a+zsNae7UP491TOk74HPIf+7Qvj56pHWURXfT1JVp+X7G78nvh+ekOQf/xuylZC7Qc8JP4PuQfs+iBlnn1xvuyzqmNB/k+y9pfKBj+FnJf4j9BnmTvqF1lLINrs6zvcyNKdnZpRO/6QeIF9F18iWTHemyA/mmx4Hod7kn6TPHp2F/QHya/nFYuW5sfOrqtsKn6WMvWool0PvFp0HsyMQMomySxxJ4PrjXKz4NjV/q+DQ0F0ZB4dOMQpYFPyvpycuSf2n2+lE1flZ7Vp5vdx08a5MfaW1czY+M7hwb99X8yJLMXT/yr8vcid+4MnfjN00y/wt+JOmdSWP8pnzfwo8kv7Pt+JH0vB3Yz7YZ62LcSG9X597zcp8eB6mj0zPS6fRZLD/aX5F/QfvMyWtiByG99htjB4ix1mMHh2G358YO6Pl6rebYweSJ2MHkzbEDxD2bfEneF182+JLd5Bhdur7klPa0ifVtAvHj8VyTNuTl+NOFn1qWw77yzOzHkx478LOS/i30PH8XWguRmfssC4zFjuMgkEtavu4+izP3a99F9uK0Fi47GENaFyMv90XJJ7AyKvmouF7M/ZJ/RvMEY0myiR25i39Xlbu7t3+j3F2fEnH8VrNPWZZDs08ZtfB50GeDBcuJ/KOefDbvGbDnTrzCzorfInZ25A2NnR10I8QN2P5EeJ4GOxvlcUu2X2T/aA2wr0D7ti7rVOijpHo/XK/YWdIXSyt3T2Qq8VXWlXdmf4lnHE+sj1T1PdqNNjbrHF/j00Qi89p9S9/9q/o0haxdn+YNsia/vl/4nQurs5ctdwzK/mPhIxh5I47VzuWdJT7sE3wR+m6twley1xvu6/iQUREfLnww9pHI16X/L/wXmuu5DEr3TsvzvOKLidzJt4dt6CFeGBgfm3z5iclNRdjL0rNw3Jt1Pn/vy0N6gXVEa9bsb0PyHVq8H81ofLplPYO8Kc3NlrWxJHNak7TvIRvDegb7EHlvClvofA7dGz7nyBnP3pHWd4i4uJF9hvwi6TmeI/xexOTIptG9fewBRP8nKesd6JGs71X1S/UZ5btPaf+5hI6geYccal/0jvmOooPc+06Dkk4f90jfLBGnd3T65ED6hObWhJ55ZGME+A60LqOWiUeyTEn2iFvSZyOuOTWftyR7MNo1fB5s6RH7R1of1j/AfTH3jn2+D/kOXeitqqx7iBs8JesAc2Ng4gM8dt0+4gj4TDxnQLbUY19vAZ0OHVW+/naZY+51jhKfj6EHWkY2lXs3yZ3WAv2/6PT+l8/J5n386aM3O/+w/Hz5L8QNUrFBPcQ+aC+MfW8/HbIcaf9NdhBxFvq+IeYT+zmiE8nPSmg8Ysy9lNdjl8aXY0zYJ0FmtA+h/RFyWaTPyL/ukF/FMjiyH7RAjIV8OOihbJSyPlsEtIePPL5G6xE5M7rPIWLd0qe11glZb91N/Yi/64TeS3/H+w4rPx5jyRfR/cjPxneDnqZ7d0LjI5I/Nwkkdkx+BX0P+k6kzxLSg5jfqBGYhpL3n2J+h5HEeo6SI+G8XBtjMBx32mYcofs8I3/od7KHiYd1QXPdG3QT+M30nuhAz+QbOx/Q+s9gq4f4Ha9XjiVhTtKcWx5lT0jrReapvZY/95XoJPIBOi27v7zgXB3tFehn+jyye3EiccWE84C0j0hlfsEG0B7xjv0m0v+xrK8u2aVxbycyG/nGXmFt0tww+RKytXwdtgMxARlTWjuxjYPtaWxgY/xI7GQxh0TvZvxe8qMGlzKvaL+FcSfbB5nQ/OF9ttgE53dN98J+icavI7kMvhfGcJQNRAcjLtOSsaGxyEbGltPa41wJ5uIIeS/4QWQ7SXfS+2kuhGzjzXvo82h8RsiFsIxJB9F8ga8q81reP4WObHGdSZfzmSTvBHOU70ky9DmmJv4c6X38bMcNv49bomPxzBNf3k86sUtzn+8/OgxkH/lI9vBo6jPMfflZU/IpDxyzxdqWax7pgXaEWNklfp+k8qwj2nf10gt5xjDP49E8jNjHiVuypgr58essDk1sEbG6I/YmLA+y2Szvu05oYqP0jKTz+Fr+WWw3EZNjuXbjFvtkLEOyuV3ch3SxeT/JIEAMN5chP2sPcsezlmVo9CetY+P/L+Er0d8urY1J2aZkrE8PvIYk50b6f8mxU8Q3MV9pTcG3or0jdBCNfRbLmGcSF+T1nnFOlO5D6+quw7E8mvOtSGRO+jnCejO6q2f9AfjsmehYxBangcgCaxVjPzWvoWeSymvEHEd2T81yjniNk54Z8+/hc2KfBP2LvSyu+cilDtlfWcIu8jWSK8/RAeTKOjw+InYk34HsxiXsF/Jt0OH8vQKJBWOvSnab5oCsTcRhyaeydQLQ6zRv2Vby3hVzgffc9FkxxqXFeRkaF9FtEd4vz5nFLd6jsP/SyTAX6L0h9nkSz+q3ZS7RGJDNEL/QkS2PRS7DPeuAMfmwaaPOoPkRBxyL6cJGwu/Ds5i6sAXmfw8xMujQo8TtET+huYY6Auji7iSVvVBE+mN5lLkScS1UMVdGRs5x26xF6FTJh6ZsI0Izh/CdcI/6fKF1QXrI5llpT+rYZIm52XgUcjFe05qldeaZvRvmzYHlDL2XoT4B++QO6fR8zWLNPblmyV8Jh5eVNTvuwyfnzyK9gVzsI/ZFduxpLUkcB759tiS/PMnr0ETXx8Y/J91yNwlsjAs1M/CNI9ERB7O2YL+P5hkOA/ZD8bkjus/IyQGyvQ1ofqX5/RaYWxPx1fBeu3fG3nDcFx+G9gjiY/H8DyRG35N5hBgoXrOuwNrty2d3Ryn7shhTqVcr5qnxWaxP6axtjFmbnl/8oC7XcrnjSXNlYnQ12TaOuz+hi+0437HNKuyY6PYjvqdjx8RPvZM1NkDtznlVB8MnGdnxDBATLetgzi9BH7Rk/k8yWwdJ90E9hI94Oeviu34+1sXv8D6ay8ivmNpG7H1pz3MUez1Fvhpyw3jteB50I6+YVxHGCrVB+F7kR5AcFqxDaX6yjqN1LTqOfH/UohXzWfYThyKei/0vahX49UF05KGQf/17OONm5T6V3GrFJqLeC/ltM9eMf7o8GL/C2oh94X9hvXdaqLMSnUm2RXzZNvK3nO9K8d367KuJ/zuxe+yM4/hS20m+KenpIr4Qkk2DbyE5KPY3+thLwV6Y55vYtRCw/8f7ish7w1po8fj8qLUgf0P7s6i6Jo5st161JsS3k/GIUuOXkIxio8+muOfTfgnGNuscIrYfkD3nNjnmwvbDyDviffAS+2fYmQPPXbItEe0HBuyro8atJ3M86/li7zHHR2aOL1PxW5ZZxHbP9VWhs/OaESOfij98N0WeshWldZ8bvxsg57CwtlXGyuhl+t5940ctPauD6Rlkv7VAHLZTHnvHXpt5fmTfz9p7tjPxEXt3kwvz7TyHXaM1WJ3nJTkX9X78nUKbz8c8R07NyPgYiT+AXGc4YF+J1pjdD2Xkv8MOiG0O5Plj+MZp1Y4Pun342Zn4Bz0z9yeoa7T7qWMxz3O/hPMt5BcFhV7pvcWGH+0+gcbo2JcxPRh50jgmMmfxnnFk9A50HutB2gv2U+MvIW+XyB6InkVsakhzKMB8Le0x/pYNpz152YaTbum0BgvXhstncLy+iBW1Jdc/DWidhDx3MV5p2R9y56TkxLAn7nl2z4faBPMa/nBb5NWR3AC+L+d3ef5ifR7FRiTtYeFvsrzzcWV9sPQxx2UOTT3UK4r/u0RuIs395vPC56X7FHOL54bxn7uS10PNAskcujbPd7DPMp6aWjjxWWTP0guj3LeOYKNzPc36zH4O+5DJEfpT9JTdc7APS3q1X/jXlx3XF91X5vHekZ/NxbH8833XndhzM8/zuezYVWuH7R6Gx+nC7vtkbiB3GrD+L/yx3H5Hl3YfPnH24Rz7hK6W+jmzp+e1dYn65J67Zz5W9+ID7IsRV+J4QMVXyef39DjMa65h23tmnznh2Cjv3c/dvXySYV7lc3wscSYaBw9r1figHnKvbpwg6tb29KL7EX8ya3bA+grfPYEukXthT2L0kKx7jrVBF3uDIh+QxzPqOgbjifzbqDSezp4JOSH4Jm3ys0PHp8aeB/FA46eNjiIzfoZQ/AOSxThydCz55vm+0vgS2EOkvJZ80mnWlwjhi8rf9tOS34Q6BOM3Ne+tyZ6OeyX9a+acsctLz9hM5H1C2SPTXkv8gupn2DV/QKxHbMGoLfF71LbFsM35/g/zCT7xM+td6uh5fqNuytbhsE+GmG9o/OCA99G0nydd45n1m5l4oMQounb9sq7Bd8WeJ2O/rryXhl+Wmn13yv6H8z15bGxsNfezpr5r+yU/kzwiZj287DT+Ta6Lu1PHPxhZ/8CT/XyDH+w8g7G7pLM6fv49xpxbs7EK38R1caYicH1UR/dxzZapxUv5HETqxEgz7FtGgYkFmfhdnPE5B+sTdyPHLk3bTs3dETVgOMNE+3MnvgSd3PfE3+qI786+Rs/4adV10K/EAWTscl9DdBnOUsG287iRfEr73dIaML4x9B9sguT+MF6kZ2Wd2RiSiSvw3pG+X68l+6d8TZj8VGlN0L0T7BVzP05iSajzcNfDhOZkL2ywf15xjfdeUi/HsSKMUQIba+NbKeuBbscrrQOMezep2Dzy6SR+nq8/2ZtMjoP6GkijbtRo75rjSU/M+xf94dFL/nDxeWYvRfr+UJ/ryP327JqF/J1937RY90VMOWA/rjbPJ0bXG/uAvEA2MtdGWd8891Dmrq0BQA3EEXlUu7c3djDhvcEY9dxLW7dp7F2vsreZeCaGhprrp/c2xf49M7FMD/t2s7fxSvEou3/Pa3jZnqUVPwN7A9J3skd/wf+o5QB4r9/ge/Bnd6ctsy49XsOXxXOLbMlm3k0kPoH42tM5gJTj8fBDU8dvyEbtwWXZZ+Dzm+O+3edKXL+QZa5D5Vkh634mcxG5o8j1CWTtcCy0lNvg++H8p+MnpEPMjZqfELM+GeTrAPtb+xpy6tu8vc2HmNh1YuNRaWWvavfe6dPxKJsT4HiURzqlHo+SdVuWqZsTwPe9Q0wJtelRKScgsRmjZ3hdwke2e74oGNRyAhK3duK8O9Ezud9SxEDyfeRTMZB8b0j+15L0e99cp73PpdXPHdge1l82XjtA7HeB/NuSr+Xxz5TGsYu6Ivw9zu2yD0x+IvZDfZxpbEOW9N4QtdelPOoCe4KR+ColWXH8hOeHo59RL84/I49Mf5+aeSS5EdwXNV9Z5Ogp8U/s2WTOY3d7+V6Ex53jRoj/2f10v+yb2P2wjYtLHIDrRcy8FduX50I5R/s34hRcW2FszvI4EJ/X53rphfVBOLdcxMkcW3TFtW0lfSnx6THOQcf0mVInWdOvXO/Zc/Tr1MS3c/2KPUtg8rxWvyIXH8ietDF2lOtXs/ZYv1au1WJHrDsWldgRzlXinF71einHyjG+Y24PTZ7hdbqV8wxl3YqaaLEdPs738ZiYPEOEPHgWG5+vj7xpNuD8i9mnXML3jlOb+4tEh2aojTMx0NperIjrLY0tnZp4Cu81q/nhpj2Y+Lr8vZy5y7Gcmv5sSy3AyM5ja7OqryVenc9v3hPvbQzH6Ic87zZEnKoxn9M3OdjIH/49fYv4jjeo5mCtvs2Qv3Z9ZNSujPJxKHKwPeQzzf5mWor7VXxkn+vaJX/plXVr7nPWc7BFvD+PYdiaitL1vDZgamNDvujgODR1X4HE0cr6PBLdxnsWJ1ZYxDTrudWgnFstz9lKvjffZ75R33pN+tbuoV6lb+GLpoW+pe8UPpH/sPlF3i/07Th37b6P42pN+Q+7BztE45q+RczBt/ah0Ld13z/Xt4X+zM/iD1E7afMK3Xye2phsXosBXQH7Ij5ppy35L/jRU9/s7Wj/2Sn7ddgjsrwjH2eenHgU1h59V/YJQ8i9iM32d7nvk/sD4kPJa6MbOJ7B59Er1524cDfXBV9X55pc4Cv9WdEFC9efjQrfNa9pGUH/mJoW2m9LjbgTbzZz8dzEj+HbyJxCvNHsV5CLj6r+LOvGhpqWA9fxSa1FSZ+iDgn738Zal2dzrT2jBzqkQxp1bVbLnd8l7SKGxzYXfQlsfjuPu9scZynunuvSaSXuPnXtXfvCzXXj3EhZz5K/Ex+kRkTiSO5+spL3EL3RdfKQhd4o+cCvyHvUfVt679Tko1y/lvdO7NdOvAa/NjB+7YHPDHI8MvILv3Zi/dq0wa/1n/FrvULPTkKTe8T+N0VNpNGdOHucx/ZyX7UcY3fztrY2x9a6lGNui8b4eh63kbOH5GPgzGmzXxteFPt2cy1O7X5qKHuZul+bxxumQZ5/G/f8ev4tCvm8XLHvqcTVbPxfaoEkZjDKpL6wEhOT87QYj6Opm0d+l/XHEOeb5YwkYh1hEUfNY9zYZ+ykVrZn9NcU8RJ5/3hp7T3bb+g0mk+eyRE6MbFqPY1j08qxY9SIpmY/4cZ5OF7AdSLWtolfcJTY8Sg0a9rxO3qI73J+jdZdKGuJa5LYnlu/gHOaC6c2Kr9m4mRj1mX2vHJrgHrfRZO/BR85bg+7Uyv/o5wpSHJ/a9Ds9z5XU2j3D4WPKDYkYDsnujarxRMk7+C9nHeg/c8djVcWWxl7QxOvItsjMr7rpSaW56Hez7WfUru7PJrcJv0+DqyOlbjRFLlKz6wVHzG0UuzE1JHUYgfGD+A9Qx6D5PyNP+T4PdZCrx7TSqs+wCiTOKORF8/ZUdrgA2RlH6BXsfV9c9569IIPILECttt5Pq3fkudLUrsHrn22jKnIr2vqHkwMR9aqqQValGJP7l6Aay7JxuMsRNXHkjNEZOsk3uvOaZv36ByHklP2OG4lY1bTu7nPyr3OanrXNznbjOtwjd7lPfXLetc3ebBQ+molNb1rfoZ9MLWf5Afe9V/r16LmxDffiXRiLDU4pBPlPCh89uVh2Kxzg2Jf20eMPrt4vkYO+fVDkcdozq8PxJc95OfvRV8HyIcZP7dt9TXOT0UmhyE180/mMJz9WZ7DcOslihyG2W+8KY8nvoHxt9DHINmJ3YsqOtjJVYjNy+cUx3HuuK4KzxXg7+pxPt5TlPcINb1biYeL3g2KGPv0mNdyZ0lga7mj8lra5zEM8W/hN2auv2jsYUh+vPE3J8dKLXdeo174vbkuKccR3PhNLY7rvmY/28lzTFIb26W5xLkisRWm9mQ8Dcr6uVR7UtLPxl5K7YndM2Z8zrK59qQav+0WNfOlvANsWx4X6z+lo1FXIjqa/DxTp97Cd5G52wsHlbxDhPln4jhyXrgX1PMLhV5mHbco62XyJdqFXp6msofqHe1eB7U1kveYHCXGBfuH/G/P1CpOWtb+Sv8Gqefu53sezs3b/F49zyC9I7CXa0nModAnUb2eO0M9r/gaiAmQ/jexUPT+cc6e8H4DdRDinyPfavJ0XT6Pa3OL5pzL8oA56uQUoWNwpjR1dDLOu4f2rCv3q3gq95bHqrBPzOst7f4rkDrfpviG0ZVd7p1S9YVT7LdyX5j90XItWp7LbMzldGTva2pB89wbcrmZrV/knJOxoZ0iv0y+OulD+j4TX8azZEcrNQM9OcfT7Qel2ln0UOKep6zHM7eGV/S4m3/rZKgDKsUA2e+WWFbFXzmWcnBurcBfixVL/pd9heq5HLsue7Z+VnRMd/lCTZA9m8M1+WmtJojeh/jYa2qCanI2vjRqlK2tLWp6pugd0c71+l3veX8aZ0ruJjaObJ9PapZlHdtzJOJP8xx6dRy5lPtsrCNclGqVvUquLq+/ajrDY9a6m68Lh7l/7dYTSv1VXkNeq5eV+ZfXUyA2lnHtm62NKeWyJSbXtzW18CuObNulJ4Yzx2Osfe81MTnuGcYxuf43yn/kceQ3njGDbJfFupScboa1WT5j5nzn8hyu1AOYPP15Zf7ijLvRoaTDUpMb8fM8Wd0v8Rr8ktbgsikHEn3LuVvE4uo13qnslYxs0+q85bpBN5dz4HgP7dfsXpFjV2U5YK947P+2WVxkwf/5/NuH2+tPSfJn11twfRyPyQQ9GXao2WFfOosze5af1k9AfyN1zNxDjPVEyjWz2TSUGA3ZDLJH5nxlm3sOXUq/CD6bL+cB8JyHvLZQevtkcn65b3oF8HmlNJI+bDwfpAdMz9Q5mrPocqYSZ/ECs8dHnKpleiWhX1aLfV45hw/bZepwpy0ZZ5y9HGXsg3GMZuINzRlkxBo5Bp/CP5J6yQHbefTYwHlrkk93iXmFs1g4p926yHv6TCvXJcZJ77U1CiHLB+fKyD8fol5hPGkVeaikJfmYSd6PKkItOHSM9IVr8blp06uE3ou6TO6twj0W8p54Sz5/yZ+ZcWzJ530z64aliYGhzhXxsCVqjY62hn44npoz5zQPM+4zkFn/QPbDpna9i752fN4afiVq2rE2Qs4Pk9+Dnr44X+z0yITNQc0kPifgsxP0HMhlDOXssfQmk77eOKuMHFFaxK6T9sDx6Qd5DgVnaadm3z+x542hm8lejtqmVyN8Np9jPNyzZekjL8Fn6tATCDpZ9i28j0evlqKXRc/0LCrm2BA+oIkz07xsSY2TnCGHH885MvS25fpy9HVYptxbimMztE+Te3AfOu7xJ/1K25XrJjexTO05+SHfC+fWaGzuEvaxB1keWw8G0jOxVcT8sU6jA9eu8BzBPmRaniPcy2Vi/dwj+7f0d5Lfi6Snxrhvz0qRvCa8549Y34+4H9+Aez137B5X1nEXZyGMH5mvOzz/BPsarGX4jH7EfacS1L0Fpndk/vw893l+2r66qE9F70bz/Bnv38264Fo08/yxrS/LRJ59v6hzRrwn8uSMVmcvfSZj6Ausc+g69nm4l0wX+RM+D4+5d3Rzi6hNH2SS9yady/FzqZPs8/4fNbFGZuGA8xodEx8ozeUD9w4857Ou9NlS54/YH9fAIM/RjeHrJzKHR3gec9YZ/UQQdxodaU0eOf7K+gD9TRGj4fPThW5GzeMdemnFnP8dsL80TUQvx6Ymo+PjXEkkfQ9SqR2v6OTuJOC8iDlPa8ec9THXSGO8of/Rj2Uq9eXdYh2Z+cD9z3kddeN2VMQ0Rc9dBlyPOZSzY1YP+5IL5lg/9HAo8a8Ecyo08TTuVYY+/uXrRZ7a7tVkLY1S6ZWOMeujN55nxiyw/S9oHplzHT0v4j54JOPLA/opcgwL+fJcjxf9JvyI9fxUennJWm0N7NnXrvt7e/9+S2L47Je2ZC33joPC5h24RxLpgwu2MagrjXyTU0Yfy1B6t7G+N/0SJjhPjBxiaPoykAw60mNJ+n60h/Ys3pjnibHt6MHCvXkgfxo30omynlrIX9H40n5yVLLH6JUxwD6N4xySIxxUrvP6x7ka2xdYbCWN0Si0tTcmfouzOXx2cMg6E+sXtqWH/VRgelSY65iT07bU6KHuI++75rFcMukhiHmO2FvNjxCbH/BZcvF1yX/ps4xwHoSezcZqQrZnlx+66HWInt6IrSCvyOOLcxNZnGBucB8JjuMs+edBl89bZFHeYycyvbrynnDwIX3EbNE7QPSQyQ3CHmdL6Ykxxucm2F+3JOcTS882mtMD7sGP83ixzadiHcHH8eUcjvgP6C8F/3zIOZRJavrNNlw3Nofr1Kb5c0IOOLMv/e3hIyMPErG9jxAn5ZgD5tAI/hl8B9hXklHH5OCmHudLup3AxNV3ZM+QC4UuSIbIhXH/qL5f0i8cC2Gf3+yd4iPHSBFnQR2e1My1OP9IvjHnUmlfgDMaphcN4iRHqcmOcXbav7DnaLDnXXAMH7o2Ff9qwnG7nAWAM6pcT4A+Y7RmF8KJOTD75XMCig3+72sQfE6C03eWT2RpgQ38W7/97vT0e2J8mgjiivFRjI9ifBTjoxgfxfgoxkcxPorxUYyPYnwU46MYH8X4KMZHMT6K8VGMj2J8FOOjGB/F+CjGRzE+ivGRdJ9ifJ7wKRXjU7KxivFRjI9ifBTjoxifnWJ8FOOjGB/F+NSPUijGRzE+ivFRjI9ifMqt5RTjoxgfxfgoxkcxPorxUYyPYnwU46MYH8X4KMZHMT6K8VGMj2J8bFtdxfgoxkcxPorxUYyPYnwU46MYH8X4KMZHMT6K8bHxDMX4RIrxUYyPYnysX6sYH8X47BTjoxgfxfgoxkcxPorxuVCMj2J8FOOjGB/F+CjGRzE+ivFRjE8xrorxUYyPYnx2ivFRjI9ifBTjoxgfxfgoxkcxPhf/5Rgf/6yO8Tn9CTA+7RrGBzCe+w19M7q83lzPdzWuz3bzuL4GwKfrkXAOt4v9/PJhFuO3h+3sga7d7u/pMbotQH9Wi2RNr1fzG/oyH24Wq9X5ZrXZ8q38m5ubdhzT9d1+u1nOnd9cn1ydhCdfR/hh2ytJ3vfrkm+deN9T7r7ikxSfpPgkxScpPknxSYpPUnyS4pMUn9RRfJLikxSfpPgkxScpPknxSYpPUnyS4pMUn6T4JMUnKT5J8UnOsyg+SfFJik9SfJLikxSfpPgkxScpPknxSYpPUnyS4pMUn6T4JMUnKT5J8UmR4pMUn6T4JMUnKT5J8UmKT1J8kuKTFJ+k+KTEtl1TfJLikxSfpPgkxScpPknxSYpPUnyS4pMUn9S4B9spPknxSYpPUnyS4pPcmhbFJyk+SfFJik9SfJLikxSfpPgkxScpPknxSYpPUnyS4pMUn9RXfJLikxSfpPgkxScpPknxSYpPUnyS4pMUnyRrXvFJik9SfNJO8UmKT1J80ssEnxO/hk9qgvh8d3xSWMP4zPhdv30+x1febu7m8X5HL5P5Hg8Qx/Mdftxv6D8fZtv1bHtdA//QBy8edvOX2UpfQbBBOyxJNfAa0EjtZrGG30ysJzWxPu7mWwhudj17IEmerACTutrSqwSv9vRcAFbt4u3iYV9HVr1apDUO1cePPv2T/+YPI5b2V8KCnZbJVMFJXfxnYV30wTcT/emTot/Pd5jDdeGv5wdIe3OzP5As/jGyzwX7zNT/VrL/9/a3287Z5nPXv3/IHg+bu0EU/dpq1WV3ncwvzY+b7f52k2zWs1WvuPqhuHqx2TwYSd7N9/v0cpHhbbNHUjYlOc+Pi/1/QG97F5qfpoblhtfdo/tDan5wuW3X85vZI8+BZDu7XtAI1H8TP26/MCKu0FP4LqWhahi93eZxG8+fm5/v5Q/3sy1U6ktku/rgb+er2X7xpfwkX38ZvX9yGT1s5w+8TKrLaLa+5sGfbbHMGOrW9u4XJOP9YrOmv5rdY8Gsr3YPPDrxHJrQozV3/39n2/gWX+qfsvbCqt5rMjuthsV3+q0UX1A35ZsVBuTjJYZh87jez695/syuf92sV2lN1vTV988Jdb1ZzysryVyqMBIhxkU8W3XM5fvF9fXqqTEswxi/hkfwvgJLbBiatvc9YYnBWW1oxOQ0Dc1hSzL6Kcdmu9nLSva7Z19prE5a5bFqv//RYxW+xYaV5fQ3LJp9PS1Zt2aLRlLfpvmb8IPzLvxYvI1/Kt533dluN4diitCVjwuI6fkl+KJts/70K21bzaL+KGNnn7u2Puluz1qnYtxbL1uon4IcGzbZqSZybPubLbC6Mtyl65iuXOG5/+z/9g2UX0wSnG9/uGlqn1S8hrOG0fC/p7o7qU/9TgzHbLYmSWAS0Pf2/r8AzzifLxs2pjVtcj3b3eZrwhkzXP8829M4rPlK22v2zl4p9Rd1hyPTsEGk9tqrVYz5hM+bxXrvDKlXWWDVoIJoTfOuYrRqNzo5e+FGolVrN+Jhz7/222dCywvfYPlK26TvZvjyKRevZrvdIi7ZsGcDSy/asJPTV9qws5Ofyoa1vJN/zPAZv8V753lBxXch8b/kvdBPn+fbBYkNGv1nmhGnP9eMsPHmf9CMKM+Gk3/uXGi92kx9n8lwUg+L7meP28fdapMs1r+EH4JfT34Juw2Baf6rBf3Fqee13r17x3/l/cq+wcn/PG7wR7/TU26KH5+7y/un79JZPdzOXr7L7v7hfe29/3pcLV7xXpqm4YenPp9+6dzgSVffe9nVr+y5q55/iH9xfVNEHum6/NO0Izjhf76SD1qJXJ2Er9wRtLxv5YSe1idnMRJ5kunpsbVzuPVr0DiH+XfOvKv+fjffflnQUDpT4xuO/g/Z9716lL9ZfPJ9PSfWtNM40Z3GUzuNysI9a711p/E+fP5G33in8b4eAagN9u529oCXi/tZwhv53cM8hjxvFkeM2Afziy6nGfyO/Nj+uPuS0EQ6QpWff+6eBVd/HJLr+3+ncXv15erOW6Amqb/4sLq6H3z589Pq8c/MW/z7379/uhgni/5vA2/6R7ic/nH8cnU/Oevfh1/6d1ITcMyu7+Nk/qm1u1pH9Jtb7/q3zslFeuZf+/HjdRY9Xvn/Wl+gMrzb4foB+hR/9sfv3qzrLdBlkH7e/PnHaj37bXTWR8XReV/ufJj+5/dN/9OfD1efDmf9ReRFy4+ffp/0F/1P9Jf36CEfo74T9VV8djfKJuDLSd0k1yZHlpm6d1m3zJPNpqkwZKoM3JE9U2b5UOjX7Ufo72FqaYaoSR8nqPkJuP9MF31GDIu2ywzGgM8Wc62gcz3nFkVt5kNk0TGvXzEcYDm/xu+3fZgC7mHA5ylL1w0Xd1RlX9aeF3WA6A0lPcTxfUe+qY3kZzM9BRq4vWVZXIzLzy1nA5KAz05y7yn0TMKZkVHwCoY9+I/tynXD8xx5dYY9emlOWwXD3jBTawz7mM/D1Xmmcv2NDHuMZbuBYe9Lr58qA7P0rIalnvDZ+jp7NPakLq/h3ui78LXZozTv+UxDjT0qz1Fhj+IsVXtg+Ys5j7bOHpUe/XX2aFk+f5U9GnPtZBN79E1yLzHsS3IvM+zL86+RYY9+NeZMCHoRo8+pZ85wHAwj2zdcO0cXJNKPgHu/1HUEzoSVuIDCyM7yfohdXsf0WT3Ukx65Z+rC8sBxDrDTwAnH9fy7lNnYkFOuJ825HkcH4P3oSYtzOw26wc/rxcpM7+J5hemN73jsNzK9I1NbV2d6l2TRzPSm+ZfgzAnqL7l2MMI5Dcv0fp7/il49x8p1w5/n+V7lv+I8mMt/RV+NtJn/OnqC/zpydM9f478OUV+/KF9vvC9fx7zspahBN2MgvNqs48kZkYlXnJ3oe7xeHDkU9x4FeX+egkOKnq+Bcw4SvZBhT5ifYc7htaQXlpxZtWxZWeMjOZN116tcd56lxMWsfBeW04TPDDMbb9zxBpZ5BiZrLjvoKPd6Pp6p1DUzd05435gn0v/EjpsvfCx+VvRGqVzPubJeWf/U5sleeLXRYWB6+wywfhZl+eZycPiylTloznGAsRg5/BycX4rAAgGDEWe9PPQveIXdxVmUoNHu4lxdze5C3qLbxO4a1nnN7jIPssHuyvU32l0+i9Rgd5kJ2cSedp7V6P/lYdDM/G6hNruJ+T3AGZuvzvyetp5gfrfyGvES85vG8zy3u/ge4INU7a7IoW53K/L5y8zvEGdkm7jcb5J7ye6W5F62u+X512h36TsG3IeHv2skZ78Xjt7JotTVO7QeM6N3LOPduW7WctdhjRd6J8v7ConeAbc9MesY52zNWu21Xd1c2JGe+11E72Sjgz2bY87/CO+nzPZmPnkT77tyz4I7TXZxmPv6/KztBl0v37NR10eBy53/irq+JPMSl/uvytzR9WWZF7q+Ueav1vXLJj3vRa4ucPV8VvScYT3PDLoa57sic5fz3SjzCue75wm7lPsfHJlt3WW21525J+8Z+WyB2b8Z3y+QnitxVrlu/K5pOiz5ODijEafS44i/k+llFIc4WwQ+C/oyiN9HNgdnPWr7Zrme67FsinOBWd67ppv7xcImLPl87Bdn1n8t+5mV+xZ+Jvftc/zMtpn3DX4mvm+znwn+1At+ZirPgT68MbO6h+hfYP1MnmNLWcfCFse5eU96hHUC6StRMNiLPVTJxxSOuTkL6DDMhaedRa4NcD4D912GpX0ts79xzt+e/RnxnDZxEJd9nnEvhDH5DHyffiB9lBvu6XC/X8NZN+edm/jtviPrr8hZTw5RA2e99HkOZ918h4Jbzn1ecTYb/V6K71++n/F7ykz7glcO/cJzgZ6Tz8bF0g85ixpknV9vuK/Dtc/PtjIPPoi6wrkEF5J7ONXljd5PWcFeL82zfZULLzoGzDnwXaYB81RwRhznLEW/yz3lLN/e9QfEp+kfh5Xrxsanrm4rfJr+0bK07fPR+8WnQezIxAz4THjX8Q/c6xWfhsYvdXwa9IYKCp9mFLIs+Flj8Itc/9Ls9aNq/Kz2rIZXfxg2+5HWxtX8yOjOsXFfzY8sydz1I/+6zJ34jStzN37TJPO/4Ef66G3WFL8p37fwI8EFdfxIet4O7GfbjHUxbjjDWpl7z8t9ehykjk4HT53PP5L8uC8jevZPXhM7kHOITbEDxFjrsQP0snFjB230gW6OHUyeiB1M3hw7QNyzyZfkffFlgy/ZTY6mV6HxJafol2F9m7wfA93LML4a/NSyHPaVZzZngKMDP6swkY2e5+/SMr1f90YWzIGT3rgTw/QqrrvP4sz92neRvTj6WnTMedSRl/uiY+5pVvdRcb2Y+yX/bMqsZMgmduQu/l1V7u7e/o1yd33KFvPCGn3KshyafUqc6cT64T7g0veR5qdhF3qy5068ws6K3yJ2duQNjZ1Fn5WBicNHBb+sZGejPG7J9ovsX5SIr4AztKxToY+S6v1wvWJnSV8srdw9kanEV1lX3pn9JZ5xPLE+UtX3aDfa2KxzfI1PE4nMa/ctffev6tMUsnZ9mjfImtm4uT+wsDp72XLHoOw/Fj6CkTfiWO1c3uiTn0lPTPpurcJXstcb7uv4kFERHy58MOnjhp5efuG/9Jml03DvtDzPK76YyB0sM+HCZui5w+ONHqUmNxVhL0vPspQz/dxbjr63MASwZgxnGvyZuMX7UfQQ6Jb1jPTUWbasjSWZGzYX96rG2LXNe7m3q/M5LekHNHLGE31V+iHi4kb2GfKLpOd4jvB7u33Tow99CxD7ga6Q/oy2j0NVv1SfUb77NJC+CX3mIwqnrviOooPc+06Dkk7nvlRLxOldHuhhKFypjHnb4kPiO6DPVMvEI1vSXxX6l/vU+TlPAj1eutw7pvp5sKXoSQaugfUPcF/MvSP3p+U+PtBbVVmjR92Tsg4wN/IeZhi7bj8VThbr94BsqZczkDLoqPL1t8scc69zNOf1oQdaRjaVezfJPUG/V190ev/L52TzPv700Zudf1h+vvwX4gap2CD0wD/QXhj7XvQ4Yf2AHgVH5qZZTn3RywO9JcCKNL3WI5/7o3GMCfsk6ftD+oVzWaTPyL/mHg6QwZH9IGZh94QTnZle48yHBuMVfR178H/RGx89R03/V/SzdZkWTm/4PK7FY9w2/bUC7mMhOp/uzbyJtvSzmAQSO0aPfumfNkRPtUx6UUOXSt6f+9iEhud5lBwJ5+XQjykAd8iMI3Sf4aR2oN/Rv8kzfTm8AXgH3J8iOgzHsW/sfEDrPxNeTM+sV44lJZZpYnqUHAwr0l7Ln/tKdJLt31n0rOyOWoYpJH0yOLaTeMIKsr1QRtJz8i7Oe/dKb0yyS2Pu94Xet76xV1ib6dDmUcjWmt5mGccEZEwz9PHKe39mzMr2I7GTxRwSvZvxe9EryfS4GnC/jB56AtU41s7vmu6F/dIjeCK238yA6zxG2cD22Xqyr1/0N/r6CV9c+vr1j/2i/2Kr3NevkbMBnwo9mnJeifT1kz64Zc5GxH2pbF9K6dX3ZF/KlHxK2zO6ZVi8zKyIpK8f/d7yXPJ+w1XOhumnFD/P0Kj174tE3uCwGv5tte9klaWBvKxhSjdwi7gPH/d/eZLR7srQ7eecSh9z9OF3+jmn0lunc6xzi5ae9OsFe6HKiRuhH6HDiWMGjPB6hM1A8u1YLmYrZ2xUelc398W2HOcSZyPvjVt+XWIJsZyFE5NY9hd8zrZwQZbYy3K/bORShW2wzKS/sfQaNkwQ6FfLYw0a+EZhmW+EvSp6jkmvq4Zew23MW+ljgz1JwbAaIHacOnyEnPezlOcEE3FRY32Gwpo1rM+C5V1ivBV9snMZ7lkHjJnT0KQzaH7EAcdiurCR8PvwLJOChzwGo1548xK3575f3BOI+2N2J6nshbhP7dHtNe4wBVtV5jzqMQbjUaXfes7rrs+XnHVRsLrz7zOeuvGo13Lnbc9J9KV0mC35mg2iRtaY4d8IX6i8ZsF9votsn9hjwb8ZlXt/v4k7X+uLD/t9HFq+pmEh5IyZPAfI9hYs3jS/H/rOdSdelPfcNHtn7A3HJV5oie/FHA/DXBnitTDOgsgwW4RJa8ZU6tUOT7Mi87Wdcs9H7pM64r52A8u1kvEM0FPP5Z+8zDPqmx6oxo6Jbj8a9m+V1Zn1n2R1Ci8q7/t72dD3l+OhvZbM/4nt7Vcw122/67t+PtbF7/A+msvjnh3rvC9zqf81yUr6EqLnVuS5XCXL/IjK7Fman6zjpA8z970Gd7ncA1562ecsOWHxlXvAF/Kvfw9n3BrZablNRL2X6X19ED5pAzO7YEwYvdEBn91lEAk7bcyMA45fCCvI+r8Tu8fODC8HtZ3km04yJ74Qkk2DbyE5KPY3mFPeHto+wlxnwmshMLxRn/3xv74WCm7qj1gL8jfpoBtV14TbV/iFNSG+nekrnBq/5IA+56LPpmG9F7bjl5xz/1fb5zqU2kj2hw+mF7TDV15i/8ycA567T/GVC/6nF2UjM8ctZwC8+RKPt9b/WuRT8YfvpshTtqK07nPn/L2cLfEcK/stTFbsCR17X2NXoV9qv9av35nnJTkX9X78ncKih38f+VzfyPgYpZZdRftS9pXQW3TZxEKxDDX4xmnVjjsslKPtlf0KhpplkASFXum9xYYf7T4BfehNH/6DkSeNY+L0+rfMxpzfgR7sab3XOz3LZZlh18jveJMNn7QHdbZNK+99/QTbhsa+Lbn+aUDrxPSsToT94fhD7pw0vYm5J77d86E2oV/mhnJ/Svks5oOaa+B5LQ0bgVlAhW9WMG6Oog+WPua4zKEp+uQb/3eJ3ITLMc19XrBkXBZQ7j93Ja+HmgVw8UjX5vkOywCMcj5KkuSs1dy3jmCjcz0dLRq5dpaBZ/YchgPIMQ/jX192XF+0ibGUc1cLdlyS5PuuO7HnZp7nc9mxqznLxmG3ezknImeqCLvgeU7ExNmH57yrghNh2AWv4UQMhMNseqpXfJWC0Xi0vVRp7j7aXujoi4nYKO/dzw/lHv3oC2rneM6Z4B7lB+OD1njFUfcJdhviT3m//pyr1+L+483cYtsv3xsU+YBnGG4YT+TfRqXxdPZMyAnBNwHHNXR8aux5EA80ftroaPrY4xnCgeFNSS9oq2N77WJfaXwJ7CFyrlPOYA/hi8rf9tOS34Q6hLcwp+6sXWaWTSJrqhMODRfY+AXVz7Br/oBYj9iCUVvi98L0LZiHScE5e3q9Wz5GhQPDPlnGXMicVzfimDfpGstSy0w8UGIUXbt+Wdfgu2LPkw3Pa3tp+GVpI8eSbasTW3V5MI7tt8wpxKyFS1z/G4fn4/gHI+sfeLKfb/CDnWewrDUwevPvMebcmo1V+JYhh32f66M6us/01+X8esrnIMpcH+gIy1a1fLeMzznkzLXIsUvTgu0D1skC82ZE/keFk9nte4ZlIr47+xo946dV10GNu8ZjV+Gu4SwVbHuZu2b2u6U1kHMyCwapjBd6H3f2TgzJsqUsc7gl+6d8TZj8VGlNWM5TmceGPuql9TDhXuUN9s8rrhkmJurlOFaEMUq4f/VFwcpm+1FaBxj3blKxeXV+tOxNJsdBfQ2kUTdqtHfN8aQn5v2L/vDoJX+4+Lyc5cr9+itzHbnfnl2zoXAVirxHwQh/ihVj5/lkV+JXIS+Qc+1GhhvJDFrPqQHwmF9217F7gpxVXGPlda2961X2NhPPxNBQc/303qbYv1uenId9u9nbeNGiYf+e1/CyPUsrfkaJQfOC/1HLAfBe/1Ws7emxztomm3k3eS1ru8Ib47xFe3BZ9hn4/KblbyNOellhDhodatg+yGUb/g1yR5HrE+wqrO1i/3uJmMHU9RPSIeZGzU+IWZ80cwaFe27y9jYfYmLXiY1HpZW9apVF1RCPsjkBjkeBTdPIoarxX2uMxwp70OQEJDZj9AyvS/jIds8XMS+1zpRcunHeneiZ3G8pYiD5PvKpGMgzfO0KD9ZlbA+EiQ1fkK9VGNu+6LuOZ2I54I14BWN7ahnbWQNjO3iGsd1y9DPqxfln5hqB1WDi95wbwX1R85VFjp4S/8SeTRZ+RS/fi+Q87rupcMNrrD+HPWXj4hIHyKKCj1llbLf/XpyCayuMzVkeB+Lz+lwvvbA+COeWiziZY4uYqVvWlxKfHuMcdEyfKXWSNf3K9Z49R79OTXw716/YsxhOSa5fmXMle9LG2FGuX83aY/1aufYV2Wkc4zvm9tDkGV6nWznPUNatqIkW2+ELV7PIM+R8s5yxK3yTfJ9iOdAm9xc1M7TLOds8rrc0tnRq4ik5b6ymQyt7sN1f4Ka1pRZgZOdxwXktv5Z4dT6/Dc/PxHCMfsjzbkPEqRrzOYaXR/7u8O/pW8R3vEEjU7fH/EzmdhT6lmQ0ysehyMEyN97sb6aluN/TbOLEK+vW3Oes52CLeH8ew7A1FaXrZeYfYkO+6OA4NHVfgcTRyvpcGIWyZ3FihUVMs55bDcq51fKcreR7833mG/Wt16Rvcw73a/QtfNG00Lf0ncIn8h+HnAPctbUGkrN5JWP7EI1r+hYxB9/ah0Lf1n3/XN8W+jM/iz9kJtzInmey87TMhc45lZNAfFJmuxwlzyV8VIxxlHbKft0LnMqB1A2EkHsRm83Z7WnhD4gPVeYQI57B59Er16vceNYFX1fnmlzgK/1Z0QUL15+NCt81r2kZQf+Ymhbab0uNuBNvNnPR8JQjZgJLTUuEWqqCaV31Z1k3NtS0HLiOrxvV9CnqkLD/bax1eTbX2jN6oEM6pFHXZhfV3Pld0i5ieGxzmQtl8tt53N3mOGvcbNal00rcferau/aFm+vGuZGynm2B5yg1IhJHcveTlbxHYJl+eR6y0BslH/gVeY+6b0vvzRnwjl/Leyf2aydeg18bGL9WePccjwTzzPq1E+vXpg1+rf+MX+sVenYSmtyjxyzRcd/GzZk3b2N7ua9ajrG7eVtbm2NrXcoxt0VjfD2P28jZQ/IxcOa02a8NL4p9u7kWp3Y/NZS9TN2vzeMNhptZ5mU7+bco5PNyxb6nElez8X+pBZKYwSiT+sJKTEzO02I8jqZuHvld1h9DnG+WM5KIdYRFHDWPcWOfYbikPaO/poiXyPvHS2vv2X5Dpw3B4ErdOGZsdL5bT+PYtHLsuC3cORs7zuM8HC/gOhFr28QvOErseBSaNe34HT3Edzm/RusulLXENUlsz61fwDnNhVMblV8zcbIx6zJ7Xrk1QL3vosnfgo8ct4fdqZX/Uc4UJLm/NWj2e5+rKbT7h8JHNOx6tnOia7NaPEHyDt7LeQdwGMFMi62MvaGJV5HtERnf9VITywPL0Hftp9TuLo8mt+mB3Wh1rMSNpshVGtbx0kcMrRQ7MXUktdiB8QN4z5DHIDl/4w85fo+10KvHtNKqDzDKJM5o5FVnse/y2oOSD9Cr2Pq+OW89esEHkFgB2+08n9ZvyfMlqd0D1z5bxlTk1zV1DyaGI2s1Z4Dva99Z9gJcc0k2PhQefMnHkjNEYCBKXMaZ0zbv0TkOJafMzFnDI6/p3dxn5V5nNb3rm5xtxnW4Ru/ynvplveubPFgofbWSmt41P8M+mNpP8gPv+q/1a1Fz4pvvRDoxlhoc0olyHjRijuywWecGxb62jxh9dvF8jZzPjOQ8j9GcXx+IL3vIz9+Lvg6QDzN+btvqa5yfikwOQ2rmn8xhOPuzPIfh1ksUOQyz33hTHk98A+NvoY9BshO7F1V0sJOrEJuXzymO49xxXRWeK8Df1eN8vKco7xFqercSDxe9GxQx9ukxr+XOksDWckfltdTEaM9cf9HYw5D8eONvTo6VWu68Rr3we3NdUo4juPGbWhzXfc1+tpPnmKQ2tktziXNFYitM7cl4GpT1c6n2pKSfjb2U2hO7Z8z4nGVz7Uk1ftstauZLeQfmHdu4WP8pHY26EtHR5OeZOvUWvovM3V44qOQdwDc1uaudnBfuBfX8QqGXWcctynqZfIl2oZenqeyhwLXN9y8mVzg5Soyrw1zr4bhnahUnLWt/pX+D1HP38z0P5+Ztfq+eZ5DeEdjLtSTmUOiTqF7PnaGeV3wNxARI/5tYKHr/OGdPeL+BOgjxz5FvNXk65od2bG7RnHNZHjBHnZwidAzOlKaOTsZ599CedeV+FU/l3vJYFfaJeb2l3X8FUufbFN8wurLLvVOqvnCK/VbuC3eXtVq0PJfZmMvpyN7X1ILmuTfkci33ucs5J2NDO0V+mXx10ofg2fsyniU7WqkZ6Mk5nm4/KNXOoocS9zxlPZ65Nbyix938WydDHVApBsh+t8SyKv7KsZSDc2sF/lqsWPK/7CtUz+XYddmz9bOiY7rLF2qC7NkcrslPazVB9D7Ex15TE1STs/GlUaNsbW1R0zNF74h2rtfves/70zhTcjexcWT7fFKzLOvYniMRf5rn0KvjyKXcZ2Md4aJUq+xVcnV5/VXTGR6z1t18XTjM/Wu3nlDqr/Ia8lq9rMy/vJ4CsbGMa99sbUwply0xub6tqYVfcWTbLj0xnDkeY+17r4nJcc8wjsn1v1H+I48jv/GMGWS7LNal5HQzrM3yGTPnO5fncKUewOTpzyvz94450mZ+YB/Efomf58nqfonX4Je0BpdNOZDoW87dIhZXr/FOZa9kZJtW5y3XDbq5nAPHe2i/ZveKHLsqywF7xWP/t83iIgv+z+ffPtxef0qSP7veguvjeEwm6MmwQ80O+9JZnNmz/LR+AvobqWPmHmKsJ1Kumc2mocRomDvvmfOVbe45dCn9IvhsvpwHwHMe8tpC6e2TyfnlvukVwOeV0kj6sPF8kB4wPb/GfE/5LF5g9viIU7VMryT0y2qxzyvn8GG7TB3utCXjjLOXo4x9MI7RTLxhwZE/cAw+hX8k9ZIDtvPosYHz1iSf7hLzCmexcE67dZH39JlWrkuMk95raxRClg/OlZF/PkS9wnjSKvJQSUvyMZO8HxX47LwvMox3PjdtepXQe1GXyb1VuMdC3hNvyecv+TMzji35vG9m3bA0MTDUuSIetkSt0dHW0A/HU3PmnOZhxn0GMusfyH7Y1K530deOz1vDr0RNO9ZGyPlh8nvQ0xfni50embA5qJnE5wR8doKeA7mMoZw9lt5k0tcbZ5WRI0qL2HXSHjg+/SDPoeAs7dTs+yf2vDF0M9nLUdv0aoTP5nOMh3u2LH3kJfhMHXoCQSfLvoX38ejVUvSy6JmeRcUcG8IHNHFmmpctqXGSM+Tw4zlHht62XF+Ovg7LlHtLcWyG9mlyD+5Dxz3+pF9pu3Ld5CaWqT0nP+R74dwajc1dwj72IMtj68FAeia2ipg/1ml04NoVniPYh0zLc4R7uUysn3tk/5b+TvJ7kfTUGPftWSmS14T3/BHr+xH34xtwr+eO3ePKOu7iLITxI/N1h+efYF+DtQyf0Y+471SCurfA9I7Mn5/nPs9P21cX9ano3WieP+P9u1kXXItmnj+29WWZyLPvF3XOiPdEnpzR6uylz2QMfYF1Dl3HPg/3kukif8Ln4TH3jm5uEbXpg0zy3qRzOX4udZJ93v+jJtbILBxwXqNj4gOluXzg3oHnfNaVPlvq/BH74xoY5Dm6MXz9RObwCM9jzjqjnwjiTqMjrckjx19ZH6C/KWI0fH660M2oebxDL62Y878D9pemiejl2NRkdHycK4mk70EqteMVndydBJwXMedp7ZizPuYaaYw39D/6sUylvrxbrCMzH7j/Oa+jbtyOipim6LnLgOsxh3J2zOphX3LBHOuHHg4l/pVgToUmnsa9ytDHv3y9yFPbvZqspVEqvdIxZn30xvPMmAW2/wXNI3Ouo+dF3AePZHx5QD9FjmEhX57r8aLfhB+xnp9KLy9Zq62BPfvadX9v799vSQyf/dKWrOXecVDYvAP3SCJ9cME2BnWlkW9yyuhjGUrvNtb3pl/CBOeJkUMMTV8GkkFHeixJ34/20J7FG/M8MbYdPVi4Nw/kT+NGOlHWUwv5Kxpf2k+OSvYYvTIG2KdxnENyhIPKdV7/OFdj+wKLraQxGoW29sbEb3E2h88ODllnYv3CtvSwnwpMjwpzHXNy2pYaPdR95H3XPJZLJj0EMc8Re6v5EWLzAz5LLr4u+S99lhHOg9Cz2VhNyPbs8kMXvQ7R0xuxFeQVeXxxbiKLE8wN7iPBcZwl/zzo8nmLLMp77ESmV1feEw4+pI+YLXoHiB4yuUHY42wpPTHG+NwE++uW5Hxi6dlGc3rAPfhxHi+2+VSsI/g4vpzDEf8B/aXgnw85hzJJTb/ZhuvG5nCd2jR/TsgBZ/alvz18ZORBIrb3EeKkHHPAHBrBP4PvAPtKMuqYHNzU43xJtxOYuPqO7BlyodAFyRC5MO4f1fdL+oVjIezzm71TfOQYKeIsqMOTmrkW5x/JN+ZcKu0LcEbD9KJBnOQoNdkxzk77F/YcDfa8C47hQ9em4l9NOG6XswBwRpXrCdBnjNbsQjgxB2a/fE5AscH/fRVIdnD6zrL8LM6pARzrt99ZkNZ3Qcee1dGxivFRjI9ifBTjoxgfxfgoxkcxPorxUYyPYnwU46MYH8X4KMZHMT6K8VGMj2J8FOOjGB/F+CjGRzE+ivFRjI9ifBTjoxgfxfjk+kUxPorxUYyPYnwU46MYn0gxPorxUYyPYnwU46MYH8X4KMZHMT6K8VGMj2J8FOOjGB/F+CjGRzE+ivFRjI9ifBTjoxgfxfjkz60YH8X4KMZHMT6uLVKMj2J8FOOjGB/F+CjGRzE+ivEpYniK8VGMj2J8FOOjGB/F+CjGRzE+ivFRjI9ifBTjoxifSDE+ivFRjI9ifBTjoxgfxfgoxkcxPorxUYzPTjE+ivFRjI9ifBTjoxgfxfi8DuPjn9UwPqftnwDjE9YwPpt1slmsk1/aJ7P7B/ry66vdA8vgZEVP8+FqS68SvGJqT9u7XyTb2X6xWdf4P/T5i4cdyD+H28V+fvkwi/Gbw3aGG97u7+lRuy16udtvN8v5+Wa12fI7/Y8fffon/80fRkDtrzMWQeiVBiJ8Xx+IlufVhyH4ZsNw0kBTOvmfxw3kvKUP2RQ/1gditUkW6/rlxS/hh9N37979EnZrQ7PdPK6vwWDqei8Pz2y1SNb0ejW/wX1vFquVM1Y3NzftOG4axeuTq5Pw5CthsFrlMTv162N22jBk4TcbstNnhmy2eridvXHI3v/XDlkDtuz7DtlZbcged/Ptji7Fq/ls/evjQ31I6JOu5vPkZvePUW/hSUW9nfxo9dby6ovlZn6gC+vN9XzHj8Z3uZrTfx4fyLLQRKcHWeO/i/V+DhODS9XR2W9wE77VbBtjGOL943b+9rH6GuYleFlVtdoN8j/7dvKvz3wR2sdLWHDSLCLw7Xx2/ethSxLCYDzuIcTV5jAneXv/z+Dj5f9bEyzJZP/cbF9v1vOKAjKXrIqKSdL0Af4HSHgRz1Yd84v7xfX16qkhK2vD7WYvLojfPfO+kvbyK6N42jCKT7hr324htVq1gbQqbHY9e9g3rBB6MAzfLt4uHvb/HC0WnIYl+b8P6vK33ut3kn37Sdnv5zuslbr0ZZXtNjf7w+zvqKXvLHw/LCa2lX/rlfL3v5n8m5xkRY4qclSRo4ocVeSoIkcVOarIUUWOKnJUkaOKHFXkqCJHFTmqyFFFjipyVJGjihxV5KgiRxU5qshRRY5KaaIiR5/wKRU5WrKxihxV5KgiRxU5qsjRnSJHFTmqyFFFjtaPfStyVJGjihxV5KgiR8ttsBU5qshRRY4qclSRo4ocVeSoIkcVOarIUUWOKnJUkaOKHFXkqCJHLQJEkaOKHFXkqCJHFTmqyFFFjipyVJGjihxV5KgiR208Q5GjkSJHFTmqyFHr1ypyVJGjO0WOKnJUkaOKHFXkqCJHLxQ5qshRRY4qclSRo4ocVeSoIkcVOVqMqyJHFTmqyNGdIkcVOarIUUWOKnJUkaOKHFXk6MV/OXL0JKgjR8MfjxxttZrIicrxUY6PcnyU46McH+X4KMdHOT7K8VGOj3J8lOOjHB/l+CjHRzk+yvFRjo9yfJTjoxwf5fgox0c5PsrxkXyfcnye8CmV41OyscrxUY6PcnyU46Mcn51yfJTjoxwf5fjUz1Iox0c5PsrxUY6PcnzKveWU46McH+X4KMdHOT7K8VGOj3J8lOOjHB/l+CjHRzk+yvFRjo9yfGxfXeX4KMdHOT7K8VGOj3J8lOOjHB/l+CjHRzk+yvGx8Qzl+ETK8VGOj3J8rF+rHB/l+OyU46McH+X4KMdHOT7K8blQjo9yfJTjoxwf5fgox0c5PsrxUY5PMa7K8VGOj3J8dsrxUY6PcnyU46McH+X4KMdHOT4X//s4Pu9bPwPH532N43OzWM/wxl26jul/6L6rx91+y2SeeDvbx7d11A991OJhB8jP4Xaxn18+zGL85rCdPdC12/09PVW39XVE2T71SnJseSd1QbY8ry7G4JuJse29jEOar6872+3mQD+tN2vI6nq2uwUIiSXjCAnXP8/2+/l2zVfank9XaQw2y/kf5juSgD5sN4/ra76B95xo59fJ/FnBOnILG8Rmr23nq9l+8WVeunmTLM0nfN4s6EmKcTsJK+MWVsZjt3ncxnPztmJIanc6eV+dAdU77WfbZL6v3YkHN//if2e8Ww3jffI/jxsS2YeHzWG+PSt+plcr/O/Vll4leLXaJIt1/TJNgPDDu3fvfgm7tdlTHusXVtlstUgwdVbzG9z3ZrFanW9Wmy3fyr+5uWnHcT6jnN9cn1ydhCdfSeO1yoP0vgFbdvrMbPsGi7StzDJllimzTJllyixTZpkyy5RZpswyZZZ1lFmmzDJllimzTJllyixTZpkyy5RZpswyZZYps0yZZcosU2aZ8yzKLFNmmTLLlFmmzDJllimzTJllyixTZpkyy5RZpswyZZYps0yZZcosU2ZZpMwyZZYps0yZZcosU2aZMsuUWabMMmWWKbMssb0OlVmmzDJllimzTJllyixTZpkyy5RZpswyZZY17sF2yixTZpkyy5RZpswyt6ZFmWXKLFNmmTLLlFmmzDJllimzTJllyixTZpkyy5RZpswyZZb1lVmmzDJllimzTJllyixTZpkyy5RZpswyZZbJmldmmTLLlFm2U2aZMsuUWfY2ZtnpT8Asa4c1js9uP3/49Sr9Ff9Lv3l8uJ7t5wwv+4DbrffzZDvbLzbrH0ou80+rBKwGcba8sAGK9Axt628K86QmzPHscfu4M4wqb76P331TGlWIf3F9s9471+WfJkrVCf/zbWByZw2UqkaWXPvbTe/T2ojsb+e7OSP5aILf7/jx+E5XfPX2EXC0682BpNypjRVJYl8ekLJEDYzOHRZzqTJwkOoinq065vL94vp69dTKeSWb7i+M1Mn7dyft8mC164PVPmsYrG+oi+r8xD0vn8Uv4YfQ81oW5+b9yurIMuFkjbmMuP8tKyx837DCTr7vCjurDdr1Ync/W+9Xi3VCv9jcfINVFJMM59ufYB1VxqOBnnn2vU2638TPtGtjS5+yeQNOEUvw9L8Gp+i9wnf4vjxF/zkI5mz1cDt746C9/68dtHyi/7hBq0Mw1/MDXfh4Sf+5JwnvScJsqiojc1jQE9Nj0lNiWB7m+LufUktuN3tx9v3u2VfSmiee9+7krLICgwZLdtZ655/VB/TboYd9v2EVKtVUqaZKNVWqqVJNlWqqVFOlmirVVKmmSjVVqqlSTZVqqlRTpZoq1VSppko1VaqpUk2VaqpUU6WaKtVUqh+VavqET6lU05KNVaqpUk2VaqpUU6Wa7pRqqlRTpZoq1bR+slyppko1VaqpUk2ValrutK1UU6WaKtVUqaZKNVWqqVJNlWqqVFOlmirVVKmmSjVVqqlSTZVqaikjSjVVqqlSTZVqqlRTpZoq1VSppko1VaqpUk2VamrjGUo1jZRqqlRTpZpav1appko13SnVVKmmSjVVqqlSTZVqeqFUU6WaKtVUqaZKNVWqqVJNlWqqVNNiXJVqqlRTpZrulGqqVFOlmirVVKmmSjVVqqlSTS/+66mm7SrVtOWd/QRYUz9QkI+CfBTkoyAfBfkoyEdBPgryUZCPgnw6CvJRkI+CfBTkoyAfBfkoyEdBPgryUZCPgnwU5KMgHwX5KMjHeRYF+SjIR0E+CvJRkI+CfBTkoyAfBfkoyEdBPgryUZCPgnwU5KMgHwX5KMgnUpCPgnwU5KMgHwX5KMhHQT4K8lGQj4J8FOST2AZgCvJRkI+CfBTkoyAfBfkoyEdBPgryUZCPgnwa92A7BfkoyEdBPgryUZCPW9OiIB8F+SjIR0E+CvJRkI+CfBTkoyAfBfkoyEdBPgryUZCPgnz6CvJRkI+CfBTkoyAfBfkoyEdBPgryUZCPgnxkzSvIR0E+CvLZKchHQT4K8nkbyKcV/gwgn7AB5HPyP48b+uQPD5vDfHtW/EyvVvjfqy29SvBqtUkW6/rl+9Uv4Yd37979EnZrWKDt5nF9Df5P16O/PNwu9vPLh1mM3x62swe6drun9/vdFphBq0Wypter+Q3ue7NYrc43q82Wb+Xf3Ny045iu7/bbzXLu/Ob65OokPPk6Yxe2g3enVQjTaX3s7CV34MJvN3AnSmBSApMSmJTApAQmJTApgUkJTEpgUgJTRwlMSmBSApMSmJTApAQmJTApgUkJTEpgUgKTEpiUwKQEJiUwOc+iBCYlMCmBSQlMSmBSApMSmJTApAQmJTApgUkJTEpgUgKTEpiUwKQEJiUwRUpgUgKTEpiUwKQEJiUwKYFJCUxKYFICkxKYEtu5TQlMSmBSApMSmJTApAQmJTApgUkJTEpgUgJT4x5spwQmJTApgUkJTEpgcmtalMCkBCYlMCmBSQlMSmBSApMSmJTApAQmJTApgUkJTEpgUgJTXwlMSmBSApMSmJTApAQmJTApgUkJTEpgUgKTrHklMCmBSQlMOyUwKYFJCUyvoPiE79+dtCsUn7OfgcB0+gyB6e5xtZg9B2Da3T+8/+9ELLX8H45Yeq+IJUUsKWJJEUuKWFLEkiKWFLGkiCVFLHUUsaSIJUUsKWJJEUuKWFLEkiKWFLGkiCVFLCliSRFLilhSxJLzLIpYUsSSIpYUsaSIJUUsKWJJEUuKWFLEkiKWFLGkiCVFLCliSRFLilhSxFKkiCVFLCliSRFLilhSxJIilhSxpIglRSwpYimxrdkUsaSIJUUsKWJJEUuKWFLEkiKWFLGkiCVFLDXuwXaKWFLEkiKWFLGkiCW3pkURS4pYUsSSIpYUsaSIJUUsKWJJEUuKWFLEkiKWFLGkiCVFLPUVsaSIJUUsKWJJEUuKWFLEkiKWFLGkiCVFLMmaV8SSIpYUsbRTxJIilhSx9DbEUiv8roil//nj9nZ9fd6fp7+t/fX78Xzypf9ruwHj83H1uNtvmcSzu/6lTTemT/94NZ8nNzt68e7du2eoSq2XqUpVjFKIf3F9s9471+WfJrzSCf/zdQbGPw1Ko3LWMCit9kl9SNrfakiC2pDcLNaz1SJbrJM65IrZSW3vfpFsZ/vFZl0bGvrExcNu/vKw1MT88WNpAP4wAml/JcG3wirUKqxL3mugWgXfSvJhTfLb+a/x7GrFkvc2N3Xxx5v7h8c9rRUvxqqZb3dvH4CvoWZOvPJsbjfN5m9ECvv39rfbztnmc9e/f8geD5u7QRT92qrJ9PPtbAeBtZ4Eht2s5sfOdrs5kFDm62vzshuvZrvdIi4Lrcxzc1XL9fxm9sjDRUvjejEvlEvxG3mE+XUyLwm8NgZ1gTsCDZ+R53a+omX5pXz7JiGbT/i8WdAH5+NJC6CySCrjtNs8buO5eVcxVC/eKKxalP1sm8z3tRvxmOdf++3ToG5n7DR4mhun0+DJ0Tv9StPg/dn3nQb+k9PAZy+jhnC83Wzx+Nebw3q/uKc/PJndQ22ur3YP+cC9Fe+43ezFZvrdMy+/W03bvmoi2DcEFQV8cvKu3WDXrK0rOXntd+E30sJ1juZ2l67jusCv8F3+7P9Wt2K0OC7NjzQmt5tkQ05Jr7haWYLF31xsNg9G4Hfz/T69XGS4yexxvykPx/y42P/HeT3Frd6F5qfu0dyZf0gbVjq5LK2rWWvebnJmPO+k1/n4F9Y6fXteVM9I1Xhqsmae+buwecq8Win8raGvu5N0m4ft5m4e7+vOyg915P/WyssBqDnB1mvwfU4alPTX8OSfHvfvI/rXomj/lojD9k8n4jqGdr3Z3s/wvs3D/KktUeFMrDfrOe+idre56B0x4/rn2Z78+jVfgZ1q2g+Vdd9P705Udgmt9lvdypPT8IU7fWOH4uxJhyKoDbv6lbWJ4J29qwyg92bXsn6vrzcZ6MftBpjv4s9JJd5Gm+s5/uL/Bw==</diagram></mxfile> \ No newline at end of file diff --git a/doc.zih.tu-dresden.de/docs/misc/migration2023.png b/doc.zih.tu-dresden.de/docs/misc/migration2023.png new file mode 100644 index 0000000000000000000000000000000000000000..6f74947fa72520a8d3273a2e58ef17aeb478d745 Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/misc/migration2023.png differ diff --git a/doc.zih.tu-dresden.de/docs/software/cicd.md b/doc.zih.tu-dresden.de/docs/software/cicd.md new file mode 100644 index 0000000000000000000000000000000000000000..7294622a292acb18d586ccee3108332f6e555272 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/software/cicd.md @@ -0,0 +1,126 @@ +# CI/CD on HPC + +We provide a **GitLab Runner** that allows you to run a GitLab pipeline on the ZIH systems. With +that you can continuously build, test, and benchmark your HPC software in the target environment. + +## Requirements + +- You (and ideally every involved developer) need an [HPC-Login](../application/overview.md). +- You manage your source code in a repository at the [TU Chemnitz GitLab instance](https://gitlab.hrz.tu-chemnitz.de) + +## Setup process + +1. Open your repository in the browser. + +2. Hover *Settings* and then click on *CI/CD* + +  + { align=center } + +3. *Expand* the *Runners* section + +  + { align=center } + +4. Copy the *registration token* + +  + { align=center } + +5. Now, you can request the registration of your repository with the + [HPC-Support](../support/support.md). In the ticket, you need to add the URL of the GitLab + repository and the registration token. + +!!! warning + + At the moment, only repositories hosted at the TU Chemnitz GitLab are supported. + +## GitLab pipelines + +As the ZIH provides the CI/CD as an GitLab runner, you can run any pipeline already working on other +runners with the CI/CD at the ZIH systems. This also means, to configure the actual steps performed +once your pipeline runs, you need to define the `.gitlab-ci.yml` file in the root of your +repository. There is a [comprehensive +documentation](https://gitlab.hrz.tu-chemnitz.de/help/ci/index.md) and a [reference for the +`.gitlab-ci.yml` file](https://gitlab.hrz.tu-chemnitz.de/help/ci/yaml/index) available at every +GitLab instance. There's also a [quick start +guide](https://gitlab.hrz.tu-chemnitz.de/help/ci/quick_start/index.md). + +The main difference to other GitLab runner is that every pipeline jobs will be scheduled as an +individual HPC job on the ZIH systems. Therefore, an important aspect is the possibility to set +Slurm parameters. While scheduling jobs allows to run code directly on the target system, it also +means that a single pipeline has to wait for resource allocation. Hence, you want to restrict, +which commits will run the complete pipeline, or which commits only run a part of the pipeline. + +### Passing Slurm parameters + +You can pass Slurm parameters via the [`variables` +keyword](https://gitlab.hrz.tu-chemnitz.de/help/ci/yaml/index#variables), either globally for the +whole yaml file, or on a per-job base. + +Use the variable `SCHEDULER_PARAMETERS` and define the same parameters you would use for [`srun` or +`sbatch`](../jobs_and_resources/slurm.md). + +!!! warning + + The parameters `--job-name`, `--output`, and `--wait` are handled by the GitLab runner and must + not be used. If used, the run will fail. + +!!! tip + + Make sure to set the `--account` such that the allocation of HPC resources is accounted + correctly. + +!!! example + + The following YAML file defines a configuration section `.test-job`, and two jobs, + `test-job-haswell` and `test-job-ml`, extending from that. The two job share the + `before_script`, `script`, and `after_script` configuration, but differ in the + `SCHEDULER_PARAMETERS`. The `test-job-haswell` and `test-job-ml` are scheduled on the partition + `haswell` and partition `ml`, respectively. + + ``` yaml + .test-job: + before_script: + - date + - pwd + - hostname + script: + - date + - pwd + - hostname + after_script: + - date + - pwd + - hostname + + test-job-haswell: + extends: .test-job + variables: + SCHEDULER_PARAMETERS: -p haswell + + + test-job-ml: + extends: .test-job + variables: + SCHEDULER_PARAMETERS: -p ml + ``` + +## Current limitations + +- Every runner job is currently limited to **one hour**. Once this time limit passes, the runner job + gets canceled regardless of the requested runtime from Slurm. This time *includes* the waiting + time for HPC resources. + +## Pitfalls and Recommendations + +- While the [`before_script`](https://gitlab.hrz.tu-chemnitz.de/help/ci/yaml/index#before_script) + and [`script`](https://gitlab.hrz.tu-chemnitz.de/help/ci/yaml/index#script) array of commands are + executed on the allocated resources, the + [`after_script`](https://gitlab.hrz.tu-chemnitz.de/help/ci/yaml/index#after_script) runs on the + GitLab runner node. We recommend that you do not use `after_script`. + +- It is likely that all your runner jobs will be executed in a slightly different directory on the + shared filesystem. Some build systems, for example CMake, expect that the configure and build is + executed in the same directory. In this case, we recommend to use one job for configure and + build. diff --git a/doc.zih.tu-dresden.de/docs/software/containers.md b/doc.zih.tu-dresden.de/docs/software/containers.md index b93971011acbc7bf55eb790b00f57d15a16fc982..e40242e9a6531512965693e2a610c46e8eff02ef 100644 --- a/doc.zih.tu-dresden.de/docs/software/containers.md +++ b/doc.zih.tu-dresden.de/docs/software/containers.md @@ -165,12 +165,13 @@ https://github.com/singularityware/singularity/tree/master/examples. !!! hint As opposed to bootstrapping a container, importing from Docker does **not require root - privileges** and therefore works on ZIH systems directly. + privileges** and therefore works on ZIH systems directly. Please note, that the singularity commands + are only available on the compute nodes and not on the login nodes. You can import an image directly from the Docker repository (Docker Hub): ```console -marie@login$ singularity build my-container.sif docker://ubuntu:latest +marie@compute$ singularity build my-container.sif docker://ubuntu:latest ``` Creating a singularity container directly from a local docker image is possible but not @@ -247,7 +248,7 @@ There are some notable changes between Singularity definitions and Dockerfiles: A read-only shell can be entered as follows: ```console -marie@login$ singularity shell my-container.sif +marie@compute$ singularity shell my-container.sif ``` !!! note @@ -259,7 +260,7 @@ marie@login$ singularity shell my-container.sif automatically and instead set up your binds manually via `-B` parameter. Example: ```console - marie@login$ singularity shell --contain -B /scratch,/my/folder-on-host:/folder-in-container my-container.sif + marie@compute$ singularity shell --contain -B /scratch,/my/folder-on-host:/folder-in-container my-container.sif ``` You can write into those folders by default. If this is not desired, add an `:ro` for read-only to @@ -287,7 +288,7 @@ While the `shell` command can be useful for tests and setup, you can also launch inside the container directly using "exec": ```console -marie@login$ singularity exec my-container.sif /opt/myapplication/bin/run_myapp +marie@compute$ singularity exec my-container.sif /opt/myapplication/bin/run_myapp ``` This can be useful if you wish to create a wrapper script that transparently calls a containerized @@ -328,20 +329,20 @@ singularity build my-container.sif example.def Then you can run your application via ```console -marie@login$ singularity run my-container.sif first_arg 2nd_arg +marie@compute$ singularity run my-container.sif first_arg 2nd_arg ``` Alternatively you can execute the container directly which is equivalent: ```console -marie@login$ ./my-container.sif first_arg 2nd_arg +marie@compute$ ./my-container.sif first_arg 2nd_arg ``` With this you can even masquerade an application with a singularity container as if it was an actual program by naming the container just like the binary: ```console -marie@login$ mv my-container.sif myCoolAp +marie@compute$ mv my-container.sif myCoolAp ``` ### Use-Cases @@ -353,6 +354,6 @@ binary-distributed applications didn't work on that anymore. You can use one of 7 container images (`/scratch/singularity/centos7.img`) to circumvent this problem. Example: ```console -marie@login$ singularity exec /scratch/singularity/centos7.img ldd --version +marie@compute$ singularity exec /scratch/singularity/centos7.img ldd --version ldd (GNU libc) 2.17 ``` diff --git a/doc.zih.tu-dresden.de/docs/software/misc/menu12_en.png b/doc.zih.tu-dresden.de/docs/software/misc/menu12_en.png new file mode 100644 index 0000000000000000000000000000000000000000..a46d3b590299ef478d3afb077b45fe8697480570 Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/software/misc/menu12_en.png differ diff --git a/doc.zih.tu-dresden.de/docs/software/misc/menu3_en.png b/doc.zih.tu-dresden.de/docs/software/misc/menu3_en.png new file mode 100644 index 0000000000000000000000000000000000000000..54356995d0a898fe8673d7c6d13ce57b8749df00 Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/software/misc/menu3_en.png differ diff --git a/doc.zih.tu-dresden.de/docs/software/misc/menu4_en.png b/doc.zih.tu-dresden.de/docs/software/misc/menu4_en.png new file mode 100644 index 0000000000000000000000000000000000000000..caebd286dd951d5865e5baf16596877e1e31c1d4 Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/software/misc/menu4_en.png differ diff --git a/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md b/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md index d7a90eb4bf0af3fe417fe9e6c89d7c44a400be28..392124c49f16e5fc4c2dcb1774782d70180682c6 100644 --- a/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md +++ b/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md @@ -128,6 +128,8 @@ However hereafter we have an example on how that might look like for Gaussian: #SBATCH --ntasks=1 #SBATCH --constraint=fs_lustre_ssd #SBATCH --cpus-per-task=24 + #SBATCH --mem-per-cpu 2050 + # only 2050 MB RAM for haswell, as Gaussian somehow crashes if we try using the full 2541 of it # Load the software you need here module purge diff --git a/doc.zih.tu-dresden.de/docs/software/performance_engineering_overview.md b/doc.zih.tu-dresden.de/docs/software/performance_engineering_overview.md index 0807db9ac0b6474b9a8051639fbad6466a03df09..b8c79afa6c2ab2b224eed7361b84a1e8a57501ae 100644 --- a/doc.zih.tu-dresden.de/docs/software/performance_engineering_overview.md +++ b/doc.zih.tu-dresden.de/docs/software/performance_engineering_overview.md @@ -44,7 +44,6 @@ software performance engineering or application performance engineering within s | [Perf](#perf-tools) | Produce and visualize [profile](#profile) | easy | medium | low | (no)[^2] | | [PIKA](#pika) | Show performance [profile](#profile) and [trace](#trace) | very easy | low | very low | no | | [Score-P](#score-p) | Create performance [trace](#trace) | complex | high | variable | yes | -| [Slurm](#slurm-profiler) | Produce and visualize simple [trace](#trace)| easy | low | low | no | | [Vampir](#vampir) | Visualize performance [trace](#trace) | complex | high | n.a. | n.a. | [^2]: Re-compilation is not required. Yet, to obtain more details it is recommended to re-compile with the `-g` compiler option, which adds debugging information to the executable of an application. @@ -248,19 +247,6 @@ Many raw data sources are supported by Score-P. It requires some time, training, and practice to fully benefit from the tool's features. See [Score-P](scorep.md) for further details. -### Slurm Profiler - -!!! hint "Easy to use performance visualization of entire batch jobs" - -The [Slurm Profiler](../jobs_and_resources/slurm_profiling.md) gathers performance data from every -task/node of a given [batch job](../jobs_and_resources/slurm.md). -It records a coarse-grained [trace](#trace) for subsequent analysis. -[Instrumentation](#instrumentation) of the applications under test is not needed. -The data analysis of the given set of system metrics needs to be initiated by the user with a -command line interface. -The resulting performance metrics are accessible in a simple graphical front-end that provides -time/performance graphs. - ### Vampir !!! hint "Complex and powerful performance data visualization of parallel applications" diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml index 3e424e9894e8ee37e91de55edfd3ecc4d679a239..48a697afe5fede6bdadb7e4d033081e3d5a9f4bb 100644 --- a/doc.zih.tu-dresden.de/mkdocs.yml +++ b/doc.zih.tu-dresden.de/mkdocs.yml @@ -54,6 +54,7 @@ nav: - Singularity for Power9 Architecture: software/singularity_power9.md - Virtual Machines: software/virtual_machines.md - GPU-accelerated Containers for Deep Learning (NGC Containers): software/ngc_containers.md + - CI/CD: software/cicd.md - External Licenses: software/licenses.md - Computational Fluid Dynamics (CFD): software/cfd.md - Mathematics Applications: software/mathematics.md @@ -92,7 +93,6 @@ nav: - Produce Performance Overview with Perf: software/perf_tools.md - Track Slurm Jobs with PIKA: software/pika.md - Record Course of Events with Score-P: software/scorep.md - - Profile Jobs with Slurm: jobs_and_resources/slurm_profiling.md - Study Course of Events with Vampir: software/vampir.md - Measure Energy Consumption: software/energy_measurement.md - Compare System Performance with SPEChpc: software/spec.md @@ -101,6 +101,11 @@ nav: - Overview: jobs_and_resources/overview.md - HPC Resources: - Overview: jobs_and_resources/hardware_overview.md + - New Systems 2023: + - Architectural Re-Design 2023: jobs_and_resources/architecture_2023.md + - Overview 2023: jobs_and_resources/hardware_overview_2023.md + - Migration 2023: jobs_and_resources/migration_2023.md + - Tests 2023: jobs_and_resources/barnard_test.md - AMD Rome Nodes: jobs_and_resources/rome_nodes.md - NVMe Storage: jobs_and_resources/nvme_storage.md - Alpha Centauri: jobs_and_resources/alpha_centauri.md @@ -112,7 +117,6 @@ nav: - Partitions and Limits: jobs_and_resources/partitions_and_limits.md - Slurm Job File Generator: jobs_and_resources/slurm_generator.md - Checkpoint/Restart: jobs_and_resources/checkpoint_restart.md - - Job Profiling: jobs_and_resources/slurm_profiling.md - Binding and Distribution of Tasks: jobs_and_resources/binding_and_distribution_of_tasks.md - User Support: support/support.md - Archive: @@ -125,7 +129,9 @@ nav: - Platform LSF: archive/platform_lsf.md - BeeGFS Filesystem on Demand: archive/beegfs_on_demand.md - Jupyter Installation: archive/install_jupyter.md + - Profile Jobs with Slurm: archive/slurm_profiling.md - Switched-Off Systems: + - Overview 2022: archive/hardware_overview_2022.md - Overview: archive/systems_switched_off.md - Migration From Deimos to Atlas: archive/migrate_to_atlas.md - System Altix: archive/system_altix.md