For the latter, please make sure that you are on a datamover node and **not** on a login node. Depending on the number of files, the tar bundle process may take some time.
For the latter, please make sure that you are on a [Datamover node](../data_transfer/datamover.md)
and **not** on a login node.
Depending on the number of files, the tar bundle process may take some time.
We do not recommend to compress (e.g. gzip) the archive, as this will decrease the read performance substantially
We do not recommend to compress (e.g. Gzip) the archive, as this can decrease the read performance substantially
e.g. for images, audio and video files.
Once the tar archive has been created, you can mount it on the compute node using `ratarmount'.
All files in the mount points can be accessed as normal files or directories
in the filesystem without any special treatment.
...
...
@@ -269,9 +269,17 @@ Note that the tar archive must be mounted on every compute node in your job.
!!! note
Mounting an archive for the first time can take some time because Ratarmount has to create an index of its contents to access it efficiently.
The index, named `.<name_of_the_archive>.index.sqlite`, will be placed
in the same directory as the archive if the directory is writable,
otherwise ratarmount will try to place the index in your home directory.
This indexing step could be done in a separate job to save resources.
It also prevents conflicting indexing by more than one process at the same time.