Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
ecdea94a
Commit
ecdea94a
authored
2 years ago
by
Jan Frenzel
Browse files
Options
Downloads
Patches
Plain Diff
Corrected spelling and lint in utilities.md.
parent
60e02331
No related branches found
No related tags found
2 merge requests
!652
Automated merge from preview to main
,
!649
Add sections about ratarmount and pragzip
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
doc.zih.tu-dresden.de/docs/software/utilities.md
+6
-7
6 additions, 7 deletions
doc.zih.tu-dresden.de/docs/software/utilities.md
doc.zih.tu-dresden.de/wordlist.aspell
+7
-1
7 additions, 1 deletion
doc.zih.tu-dresden.de/wordlist.aspell
with
13 additions
and
8 deletions
doc.zih.tu-dresden.de/docs/software/utilities.md
+
6
−
7
View file @
ecdea94a
...
@@ -124,15 +124,14 @@ marie@login3$ ssh login4 tmux ls
...
@@ -124,15 +124,14 @@ marie@login3$ ssh login4 tmux ls
marie_is_testing: 1 windows (created Tue Mar 29 19:06:26 2022) [105x32]
marie_is_testing: 1 windows (created Tue Mar 29 19:06:26 2022) [105x32]
```
```
## Working with Large Archives and Compressed Files
## Working with Large Archives and Compressed Files
### Parallel Gzip Decompression
### Parallel Gzip Decompression
There is a plethora of gzip tools but none of them can fully utilize multiple cores.
There is a plethora of
`
gzip
`
tools but none of them can fully utilize multiple cores.
The fastest single-core decoder is igzip from the
The fastest single-core decoder is
`
igzip
`
from the
[
Intelligent Storage Acceleration Library
](
https://github.com/intel/isa-l.git
)
.
[
Intelligent Storage Acceleration Library
](
https://github.com/intel/isa-l.git
)
.
In tests, it can reach ~500 MB/s compared to ~200 MB/s for the system-default gzip.
In tests, it can reach ~500 MB/s compared to ~200 MB/s for the system-default
`
gzip
`
.
If you have very large files and need to decompress them even faster, you can use
If you have very large files and need to decompress them even faster, you can use
[
pragzip
](
https://github.com/mxmlnkn/pragzip
)
.
[
pragzip
](
https://github.com/mxmlnkn/pragzip
)
.
Currently, it can reach ~1.5 GB/s using a 12-core processor in the above-mentioned tests.
Currently, it can reach ~1.5 GB/s using a 12-core processor in the above-mentioned tests.
...
@@ -161,8 +160,8 @@ src/tools/pragzip --help
...
@@ -161,8 +160,8 @@ src/tools/pragzip --help
The built binary can then be used directly or copied inside a folder that is available in your
The built binary can then be used directly or copied inside a folder that is available in your
`PATH`
environment variable.
`PATH`
environment variable.
Pragzip is still in development, so if it crashes or if it is slower than the system gzip,
Pragzip is still in development, so if it crashes or if it is slower than the system
`
gzip
`
,
please
[
open an issue
](
https://github.com/mxmlnkn/pragzip/issues
)
on Git
h
ub.
please
[
open an issue
](
https://github.com/mxmlnkn/pragzip/issues
)
on Git
H
ub.
### Direct Archive Access Without Extraction
### Direct Archive Access Without Extraction
...
@@ -189,7 +188,7 @@ pip install ratarmount
...
@@ -189,7 +188,7 @@ pip install ratarmount
```
```
Ratarmount is still in development, so if there are problems or if it is unexpectedly slow,
Ratarmount is still in development, so if there are problems or if it is unexpectedly slow,
please
[
open an issue
](
https://github.com/mxmlnkn/ratarmount/issues
)
on Git
h
ub.
please
[
open an issue
](
https://github.com/mxmlnkn/ratarmount/issues
)
on Git
H
ub.
There also is a library interface called
There also is a library interface called
[
ratarmountcore
](
https://github.com/mxmlnkn/ratarmount/tree/master/core#example
)
that works
[
ratarmountcore
](
https://github.com/mxmlnkn/ratarmount/tree/master/core#example
)
that works
...
...
This diff is collapsed.
Click to expand it.
doc.zih.tu-dresden.de/wordlist.aspell
+
7
−
1
View file @
ecdea94a
personal_ws-1.1 en 4
64
personal_ws-1.1 en 4
71
Abaqus
Abaqus
ACL
ACL
ACLs
ACLs
...
@@ -129,6 +129,7 @@ GPUs
...
@@ -129,6 +129,7 @@ GPUs
gres
gres
GROMACS
GROMACS
GUIs
GUIs
Gzip
Hackenberg
Hackenberg
hadoop
hadoop
haswell
haswell
...
@@ -306,6 +307,8 @@ png
...
@@ -306,6 +307,8 @@ png
PowerAI
PowerAI
PowerShell
PowerShell
ppc
ppc
pragzip
Pragzip
pre
pre
Pre
Pre
preload
preload
...
@@ -323,6 +326,7 @@ Pthreads
...
@@ -323,6 +326,7 @@ Pthreads
pty
pty
PuTTY
PuTTY
pymdownx
pymdownx
PyPI
PythonAnaconda
PythonAnaconda
pytorch
pytorch
PyTorch
PyTorch
...
@@ -331,6 +335,8 @@ queue
...
@@ -331,6 +335,8 @@ queue
quickstart
quickstart
Quickstart
Quickstart
randint
randint
Ratarmount
ratarmountcore
reachability
reachability
README
README
Rechenautomat
Rechenautomat
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment