Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
6ded0628
Commit
6ded0628
authored
3 years ago
by
Martin Schroschk
Browse files
Options
Downloads
Patches
Plain Diff
Fix links
parent
5e0258d8
No related branches found
Branches containing commit
No related tags found
3 merge requests
!322
Merge preview into main
,
!319
Merge preview into main
,
!199
Fix links
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md
+7
-6
7 additions, 6 deletions
doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md
with
7 additions
and
6 deletions
doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md
+
7
−
6
View file @
6ded0628
# BIG DATA FRAMEWORKS: APACHE SPARK, APACHE FLINK, APACHE HADOOP
<span
class=
"twiki-macro RED"
></span>
**
This page is under
construction
**
<span
class=
"twiki-macro ENDCOLOR"
></span>
!!! note
This page is under construction
[
Apache Spark
](
https://spark.apache.org/
)
,
[
Apache Flink
](
https://flink.apache.org/
)
and
[
Apache Hadoop
](
https://hadoop.apache.org/
)
are frameworks for processing and integrating
...
...
@@ -90,17 +91,17 @@ launch your jobs in the background using batch jobs**. For that, you can
conveniently put the parameters directly into the job file which you can
submit using
`sbatch \[options\] \<job file>`
.
Please use a
[
batch job
](
S
lurm
)
similar to the one attached:
Please use a
[
batch job
](
../jobs_and_resources/s
lurm
.md
)
similar to the one attached:
[
example-spark.sbatch
](
misc/example-spark.sbatch
)
.
## Apache Spark with
[
Jupyter
](JupyterHub) n
otebook
## Apache Spark with Jupyter
N
otebook
There are two general options on how to work with Jupyter notebooks on
Taurus:
\<
br />There is
[
jupyterhub
](
J
upyter
H
ub
)
, where you can simply
Taurus:
\<
br />There is
[
jupyterhub
](
../access/j
upyter
h
ub
.md
)
, where you can simply
run your Jupyter notebook on HPC nodes (the preferable way). Also, you
can run a remote jupyter server manually within a sbatch GPU job and
with the modules and packages you need. You can find the manual server
setup
[
here
](
D
eep
L
earning
)
.
setup
[
here
](
d
eep
_l
earning
.md
)
.
### Preparation
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment