Contribution Guide

In the following, it is outlined how to contribute to the HPC documentation of TU Dresden/ZIH and which rules should be followed when adding to this project. Although, this document might seem very long describing complex steps, contributing is quite easy - trust us.

Contribute via Issue

Users can contribute to the documentation via the issue tracking system. For that, open an issue to report typos and missing documentation or request for more precise wording etc. ZIH staff will get in touch with you to resolve the issue and improve the documentation.

Reminder: Non-documentation issues and requests need to be send as ticket to hpcsupport@zih.tu-dresden.de.

Preparation

Contributions can be done via editing the repository through GitLab's web interface or following the git-based workflow. Both ways are described in the following.

Fork and Clone Repository

All contributing starts with forking the repository to either gitlab.hrz.tu-chemnitz.de or any other git service, e.g., gitlab.com, github.com or your personal preference. Now, create a local clone of your fork

# SSH based method
~ git@gitlab.hrz.tu-chemnitz.de:LOGIN/hpc-compendium.git

# HTTP based method
~ https://gitlab.hrz.tu-chemnitz.de/LOGIN/hpc-compendium.git

Install Dependencies

TODO: Describtion

~ cd hpc-compendium/doc.zih.tu-dresden.de
~ pip install -r requirements.txt

TODO: virtual environment TODO: What we need for markdownlinter and checks?

Contribute via Web IDE

GitLab offers a rich and versatile web interface to work with repositories. To fix typos and edit source files, just select the file of interest and click the Edit button. A text and commit editor are invoked: Do your changes, add a meaningful commit message and commit the changes.

The more sophisticated integrated Web IDE is reached from the top level menu of the repository or by selecting any source file.

Other git services might have an aquivivalent web interface to interact with the repository. Please refer to the corresponding documentation for further information.

Contribute via Local Clone

mkdocs Rocks

As mentioned, this documentation bases on markdown files which are translated into static html files using the tool mkdocs. All markdown source files are located in the docs subfolder. The file mkdocs.yml is the single configuration file for the project from layout up to document structure and extensions. The navigation section nav in mkdocs.yml specifies the order, titles and nesting of the documentation pages.

In principle, mkdocs is not mandatory on the local system to contribute to the project. But it also provides a builtin development server that allows to serve the documentation, i.e. it can preview the updated documentation locally before committing the changes to the repository.

To make use of mkdocs, it is necessary to have two commands in mind

~ mkdocs serve - Start the live-reloading docs server.
~ mkdocs build - Build the documentation site.

Preview Using mkdocs

Invokemkdocs serveto build and preview the documentation. The documentation is automatically rerendered and reloaded if the system detects updates (great!). By default, the builtin web server hosts the documentation at http://127.0.0.1:8000.

~ cd /PATH/TO/hpc-compendium/doc.zih.tu-dresden.de
~ mkdocs serve
INFO    -  Building documentation...
INFO    -  Cleaning site directory
INFO    -  Documentation built in 0.08 seconds
[I 210127 13:33:30 server:335] Serving on http://127.0.0.1:8000
INFO    -  Serving on http://127.0.0.1:8000
[I 210127 13:33:30 handlers:62] Start watching changes
INFO    -  Start watching changes

Open http://127.0.0.1:8000 with a web browser to preview the local copy of the documentation.

Build Static Documentation

To build the documentation, invoke mkdocs build. This will create a new directory named public which holds the generated static html/jss/css files. This command is used to build the documentation within the CI/CD pipeline. Thus, it should exit without error.

~ cd /PATH/TO/hpc-compendium/doc.zih.tu-dresden.de
~ mkdocs build
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /PATH/to/hpc-compendium.git/doc.zih.tu-dresden.de/public
INFO    -  Documentation built in 0.09 seconds

Git Workflow

It is crucial to keep your branch synchronized with the upstream repository when you are working locally on the documentation. At first, you should add a remote pointing to the official documentation.

~ git remote add upstream-zih git@gitlab.hrz.tu-chemnitz.de:zih/hpc-compendium/hpc-compendium.git

Now, you have two remotes, namely origin and upstream-zih. The remote origin points to your fork, whereas upstream-zih points to the original documentation repository at GitLab Chemnitz.

$ git remote -v
origin  git@gitlab.hrz.tu-chemnitz.de:LOGIN/hpc-compendium.git (fetch)
origin	git@gitlab.hrz.tu-chemnitz.de:LOGIN/hpc-compendium.git (push)
upstream-zih  git@gitlab.hrz.tu-chemnitz.de:zih/hpc-compendium/hpc-compendium.git (fetch)
upstream-zih	git@gitlab.hrz.tu-chemnitz.de:zih/hpc-compendium/hpc-compendium.git (push)

Next, you should synchronize your main branch with the upstream.

~ git checkout main
~ git pull upstream main

At this point, your main branch is up-to-date with the original documentation of HPC compendium.

Making Changes and Merge Requests

It is good git-practise to only use the main branch for synchronization with the upstream, not for changes, as outlined in the previous subsection. In order to commit to this documentation, create a new branch (a so-called feature branch) basing on the main branch and commit your changes to it.

~ git checkout main
~ git checkout -b <FEATUREBRANCH>
# Edit file1, file2 etc.
~ git add <file1> <file2>
~ git commit -m <COMMIT MESSAGE>
~ git push origin <FEATUREBRANCH>

The last command pushes the changes to your remote at branch FEATUREBRANCH. Now, it is time to incoporate the changes and improvements into the HPC Compendium. For this, create a merge request to the main branch.

Important Branches

There are two important branches in this repository:

• Preview:
  • Branch containing recent changes which will be soon merged to main branch (protected branch)
  • Served at todo url from TUD VPN
• Main: Branch which is deployed at doc.zih.tu-dresden.de holding the current documentation (protected branch)

If you are totally sure about your commit (e.g., fix a typo), it is only the following steps:

1. Synchronize branches
2. Edit the markdown file of interest
3. Commit change
4. Push commit to your fork and probably new branch
5. Pose Merge Request

Checks

We have several checks on the markdown sources to ensure for a consistent and high quality of the documentation. These checks are run within the CI/CD pipeline and changes are only deployed to the HPC compendium, if the checks are passed. Thus, we highly recommend running the checks locally before committing and posing a merge request.

• Markdown linter
• Check internal and external links
• Check code and command examples

Markdown Linter

The markdown linter client helps to keep the markdown source code clean and consistent.

Installation

~ npm install markdownlint-cli

The configuration is stored in .markdownlint.json. The tool markdownlint can be run in dry or fix mode. The dry mode (default) only outputs findings, whereas in fix mode it resolves basic errors directly in the markdown files.

~ cd doc.zih.tu-dresden.de/
~ markdownlint [--config .markdownlint.json] [--fix] docs/index.md
docs/index.md:8:131 MD013/line-length Line length [Expected: 130; Actual: 138]

Before committing, invoke the script util/lint-changes.sh which calls the markdownlint tool for all (git-)changed markdown files.

~ sh util/lint-changes.sh
hpc-compendium-2.0-gitlab-tudzih git:(master) ✗ sh util/lint-changes.sh
README.md:6 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]
README.md:7 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 3]
README.md:21 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]
README.md:22 MD022/blanks-around-headings/blanks-around-headers Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Why is this not at gitlab.hrz.tu-chemnitz.de?"]
[8< 8<]

Check Links

No one likes dead links. Therefore, we check the internal and external links within the markdown source files. For that, the script util/check-links.sh and/or the tool markdown-link-check can be used.

Installation

~ npm install markdown-link-check

~ cd doc.zih.tu-dresden.de/
~ markdown-link-check docs/index.md

FILE: docs/index.md
[✖] http://141.76.17.11/hpc-wiki/bin/view/Compendium
[✓] https://docs.olcf.ornl.gov/
[✓] https://nersc.gitlab.io/
[✓] https://www.mkdocs.org/
[✓] https://docs.gitlab.com/runner/
[✓] https://docs.gitlab.com/ee/user/project/pages/
[✖] CONTRIBUTE.md

7 links checked.

ERROR: 2 dead links found!
[✖] http://141.76.17.11/hpc-wiki/bin/view/Compendium → Status: 0
[✖] CONTRIBUTE.md → Status: 400

TODO: When and how to run util/check-links.sh?

Check Code and Commands

1. All code blocks and commands should be runnable from a login node.

TODO: Implement Issue #9

Check Pages Structure

TODO: Create rules Issue #5

Content Rules

New Page and Pages Structure

The pages structure is defined in the configuration file mkdocs.yaml.

docs/
  - Home: index.md
  - Application for HPC Login: application.md
  - Request for Resources: req_resources.md
  - Access to the Cluster: access.md
  - Available Software and Usage:
    - Overview: software/overview.md
  ...

To add a new page to the documentation follow these two steps:

1. Create a new markdown file under docs/SUBDIR/NEW.md and put the documentation inside.
2. Add SUBDIR/NEW.md to the configuration file mkdocs.yml by updating the navigation section.

Make sure that the new page is not floating, i.e., it can be reached directly from the documentation structure.

Markdown

1. Please keep things simple, i.e., avoid using fancy markdown dialects.

• Cheat Sheet
• Style Guide

1. Do not add large binary files or high resolution images to the repository. See this valuable document for image optimization.

Code Blocks and Commands

• Use ticks to mark code blocks and commands, not italic font.
• Specify language for code blocks, refer to highlight.js for supported languages.

Writing Style

TODO Guide Issue #14

• Capitalize headings

Spelling and Technical Wording

To provide a consistent and high quality documentation, and help users to find the right pages, there is a list of conventions w.r.t. spelling and technical wording.

• Language settings: en_us
• I/O not IO
• Slurm not SLURM
• Filesystem not file system

TODO: Put into file

TODO: Implement checks Issue #13

Code Blocks and Command Prompts

Showing commands and sample output is an important part of all technical documentation. Thus, some rules have to be followed.

1. Prompts: It should be clear from the prompt, where the command is run (e.g. Taurus, specific partition or local machine).

• Taurus / HPC systems of TUD in general: taurus$
• Specific kind of node or partition: tauruslogin$, taurus-ml$ taurus-rome$etc.
  • TODO: Remove prefix taurus?
• Local machine: localhost$
• No output: Omit prompt (copy-paste)
• With Output: Add prompt (make clear what is the command and what is the output)

Data Privacy and Generic User Name

Where possible, replace login, project name and other private data with clearly arbitrary placeholders. E.g., use generic login marie and project name p_marie.

taurus$ ls -l
drwxr-xr-x   3 marie p_marie      4096 Jan 24  2020 code
drwxr-xr-x   3 marie p_marie      4096 Feb 12  2020 data
-rw-rw----   1 marie p_marie      4096 Jan 24  2020 readme.md

Mark Omissions

If showing only a snippet of a long output, omissions are marked with [...].

Mark Placeholders

Stick to the Unix rules on optional and required arguments, and seclection of item sets.

• <required argument or value>
• [optional argument or value]
• {choice1|choice2|choice3}