Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
ecdea94a
Commit
ecdea94a
authored
2 years ago
by
Jan Frenzel
Browse files
Options
Downloads
Patches
Plain Diff
Corrected spelling and lint in utilities.md.
parent
60e02331
No related branches found
No related tags found
2 merge requests
!652
Automated merge from preview to main
,
!649
Add sections about ratarmount and pragzip
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
doc.zih.tu-dresden.de/docs/software/utilities.md
+6
-7
6 additions, 7 deletions
doc.zih.tu-dresden.de/docs/software/utilities.md
doc.zih.tu-dresden.de/wordlist.aspell
+7
-1
7 additions, 1 deletion
doc.zih.tu-dresden.de/wordlist.aspell
with
13 additions
and
8 deletions
doc.zih.tu-dresden.de/docs/software/utilities.md
+
6
−
7
View file @
ecdea94a
...
...
@@ -124,15 +124,14 @@ marie@login3$ ssh login4 tmux ls
marie_is_testing: 1 windows (created Tue Mar 29 19:06:26 2022) [105x32]
```
## Working with Large Archives and Compressed Files
### Parallel Gzip Decompression
There is a plethora of gzip tools but none of them can fully utilize multiple cores.
The fastest single-core decoder is igzip from the
There is a plethora of
`
gzip
`
tools but none of them can fully utilize multiple cores.
The fastest single-core decoder is
`
igzip
`
from the
[
Intelligent Storage Acceleration Library
](
https://github.com/intel/isa-l.git
)
.
In tests, it can reach ~500 MB/s compared to ~200 MB/s for the system-default gzip.
In tests, it can reach ~500 MB/s compared to ~200 MB/s for the system-default
`
gzip
`
.
If you have very large files and need to decompress them even faster, you can use
[
pragzip
](
https://github.com/mxmlnkn/pragzip
)
.
Currently, it can reach ~1.5 GB/s using a 12-core processor in the above-mentioned tests.
...
...
@@ -161,8 +160,8 @@ src/tools/pragzip --help
The built binary can then be used directly or copied inside a folder that is available in your
`PATH`
environment variable.
Pragzip is still in development, so if it crashes or if it is slower than the system gzip,
please
[
open an issue
](
https://github.com/mxmlnkn/pragzip/issues
)
on Git
h
ub.
Pragzip is still in development, so if it crashes or if it is slower than the system
`
gzip
`
,
please
[
open an issue
](
https://github.com/mxmlnkn/pragzip/issues
)
on Git
H
ub.
### Direct Archive Access Without Extraction
...
...
@@ -189,7 +188,7 @@ pip install ratarmount
```
Ratarmount is still in development, so if there are problems or if it is unexpectedly slow,
please
[
open an issue
](
https://github.com/mxmlnkn/ratarmount/issues
)
on Git
h
ub.
please
[
open an issue
](
https://github.com/mxmlnkn/ratarmount/issues
)
on Git
H
ub.
There also is a library interface called
[
ratarmountcore
](
https://github.com/mxmlnkn/ratarmount/tree/master/core#example
)
that works
...
...
This diff is collapsed.
Click to expand it.
doc.zih.tu-dresden.de/wordlist.aspell
+
7
−
1
View file @
ecdea94a
personal_ws-1.1 en 4
64
personal_ws-1.1 en 4
71
Abaqus
ACL
ACLs
...
...
@@ -129,6 +129,7 @@ GPUs
gres
GROMACS
GUIs
Gzip
Hackenberg
hadoop
haswell
...
...
@@ -306,6 +307,8 @@ png
PowerAI
PowerShell
ppc
pragzip
Pragzip
pre
Pre
preload
...
...
@@ -323,6 +326,7 @@ Pthreads
pty
PuTTY
pymdownx
PyPI
PythonAnaconda
pytorch
PyTorch
...
...
@@ -331,6 +335,8 @@ queue
quickstart
Quickstart
randint
Ratarmount
ratarmountcore
reachability
README
Rechenautomat
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment