Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
1cc590a5
Commit
1cc590a5
authored
1 year ago
by
Alexander Grund
Browse files
Options
Downloads
Patches
Plain Diff
Update instructions for custom EB environments
parent
d891cb89
No related branches found
Branches containing commit
No related tags found
2 merge requests
!1021
Automated merge from preview to main
,
!1016
Update instructions for custom EB environments
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
doc.zih.tu-dresden.de/docs/software/custom_easy_build_environment.md
+32
-10
32 additions, 10 deletions
...dresden.de/docs/software/custom_easy_build_environment.md
doc.zih.tu-dresden.de/wordlist.aspell
+1
-0
1 addition, 0 deletions
doc.zih.tu-dresden.de/wordlist.aspell
with
33 additions
and
10 deletions
doc.zih.tu-dresden.de/docs/software/custom_easy_build_environment.md
+
32
−
10
View file @
1cc590a5
...
...
@@ -98,15 +98,12 @@ marie@compute$ module load EasyBuild
This can be either done via environment variables:
```
console
marie@compute$
export
EASYBUILD_
ALLOW_LOADED_MODULES
=
E
asy
B
uild
\
marie@compute$
export
EASYBUILD_
CONFIGFILES
=
/software/util/etc/e
asy
b
uild
.d/barnard.cfg
\
export EASYBUILD_DETECT_LOADED_MODULES=unload \
export EASYBUILD_BUILDPATH="/tmp/$
{
USER
}
-EasyBuild
${
SLURM_JOB_ID
:-}
"
\
export EASYBUILD_SOURCEPATH="$
{WORKSPACE}/sources"
\
export EASYBUILD_INSTALLPATH="$
{
WORKSPACE
}
/easybuild
"
\
export EASYBUILD_SUBDIR_MODULES="modules" \
export EASYBUILD_MODULE_NAMING_SCHEME="HierarchicalMNS" \
export EASYBUILD_MODULE_DEPENDS_ON=1 \
export EASYBUILD_HOOKS="/software/util/easybuild/ebhooks.py"
export EASYBUILD_SUBDIR_USER_MODULES= \
export EASYBUILD_BUILDPATH="/dev/shm/$
{
USER
}
-EasyBuild
${
SLURM_JOB_ID
:-}
"
\
export EASYBUILD_SOURCEPATH="$
{WORKSPACE}/sources:/software/util/sources"
\
export EASYBUILD_INSTALLPATH="$
{
WORKSPACE
}
/easybuild
"
```
Or you can do that via the configuration file at
`$HOME/.config/easybuild/config.cfg`
.
...
...
@@ -119,7 +116,7 @@ marie@compute$ eb --confighelp > ~/.config/easybuild/config.cfg
Edit this file by uncommenting the above settings and specifying the respective values.
Note the difference in naming as each setting in the environment has the
`EASYBUILD_`
prefix
and is uppercase, while it is lowercase in the config.
For example
`$EASYBUILD_
MODULE_NAMING_SCHEME
`
above corresponds to
`
module-naming-scheme
`
For example
`$EASYBUILD_
DETECT_LOADED_MODULES
`
above corresponds to
`
detect-loaded-modules
`
in the config file.
Note that you cannot use environment variables (like
`$WORKSPACE`
or
`$USER`
) in the config file.
...
...
@@ -128,6 +125,12 @@ use of EasyBuild and could be forgotten.
You can also combine those approaches setting some in the config and some in the environment,
the latter will take precedence.
The first variable
`$EASYBUILD_CONFIGFILES`
makes sure the settings used for installing all other modules
on the cluster are used.
I.e. that config file is read before the custom one in your
`$HOME`
.
By that most of the configuration is already set up.
\
But of course e.g. the installation path needs to be set by you.
The configuration used can be shown via:
```
console
...
...
@@ -136,7 +139,7 @@ marie@compute$ eb --show-config
This shows all changed/non-default options while the parameter
`--show-full-config`
shows all options.
The hierarchical module naming scheme (used on
Barnard
) affects e.g. location and naming of modules.
The hierarchical module naming scheme (used on
our systems
) affects e.g. location and naming of modules.
In order for EasyBuild to use the existing modules,
you need to use the "all" modules folder of the main tree.
But likely only the "Core" subdirectory is set in
`$MODULEPATH`
.
...
...
@@ -150,6 +153,23 @@ marie@compute$ echo $MODULEPATH
/software/modules/rapids/r23.10/all:/software/modules/rapids/r23.10/all/Core:/software/modules/releases/rapids
```
Take care to adjust the path to the release you use.
I.e. in the above example the module
`release/23.10`
was loaded resulting in
`/software/modules/rapids/r23.10/all/Core`
on this cluster.
For the
`module use`
command you take that (from
`$MODULEPATH`
) and only strip of the
`/Core`
.
Or you can use this one-line command to do it automatically:
```
console
marie@compute$
ml use
$(
echo
"
$MODULEPATH
"
|
grep
-oE
'(^|:)[^:]+/Core:'
|
sed
's|/Core:||'
)
```
Finally, you need to tell LMod about your modules:
```
console
marie@compute$
export
ZIH_USER_MODULES
=
$EASYBUILD_INSTALLPATH
```
**Step 7:**
Now search for an existing EasyConfig:
```
console
...
...
@@ -177,6 +197,7 @@ Using the variable from step 6:
```
console
marie@compute$
module use
"
${
EASYBUILD_INSTALLPATH
}
/modules/all"
marie@compute$
export
ZIH_USER_MODULES
=
$EASYBUILD_INSTALLPATH
marie@compute$
export
LMOD_IGNORE_CACHE
=
1
```
...
...
@@ -184,6 +205,7 @@ marie@compute$ export LMOD_IGNORE_CACHE=1
```
console
marie@compute$
module use
"/data/horse/ws/marie-EasyBuild/easybuild/modules/all"
marie@compute$
export
ZIH_USER_MODULES
=
/data/horse/ws/marie-EasyBuild/easybuild
marie@compute$
export
LMOD_IGNORE_CACHE
=
1
```
...
...
This diff is collapsed.
Click to expand it.
doc.zih.tu-dresden.de/wordlist.aspell
+
1
−
0
View file @
1cc590a5
...
...
@@ -209,6 +209,7 @@ LINPACK
linter
Linter
lmod
LMod
LoadLeveler
localhost
lsf
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment