Skip to content
Snippets Groups Projects
Commit 34cde209 authored by Jan Frenzel's avatar Jan Frenzel
Browse files

Removed example-spark.sbatch as it is included in

big_data_frameworks_spark.md.
parent a2e7a941
No related branches found
No related tags found
3 merge requests!412Manual attempt to merge preview into main,!402Solved issue-194. Added missing information regarding usage of SSH config for...,!400Improved big_data_frameworks_spark.md
...@@ -119,12 +119,13 @@ short test runs, it is **recommended to launch your jobs in the background using ...@@ -119,12 +119,13 @@ short test runs, it is **recommended to launch your jobs in the background using
that, you can conveniently put the parameters directly into the job file and submit it via that, you can conveniently put the parameters directly into the job file and submit it via
`sbatch [options] <job file>`. `sbatch [options] <job file>`.
Please use a [batch job](../jobs_and_resources/slurm.md) with a configuration, similar to the example below: Please use a [batch job](../jobs_and_resources/slurm.md) with a configuration, similar to the
example below:
??? example "spark.sbatch" ??? example "spark.sbatch"
```bash ```bash
#!/bin/bash -l #!/bin/bash -l
#SBATCH --time=01:00:00 #SBATCH --time=00:05:00
#SBATCH --partition=haswell #SBATCH --partition=haswell
#SBATCH --nodes=2 #SBATCH --nodes=2
#SBATCH --exclusive #SBATCH --exclusive
......
#!/bin/bash
#SBATCH --time=00:03:00
#SBATCH --partition=haswell
#SBATCH --nodes=1
#SBATCH --exclusive
#SBATCH --mem=50G
#SBATCH -J "example-spark"
ml Spark/3.0.1-Hadoop-2.7-Java-1.8-Python-3.7.4-GCCcore-8.3.0
function myExitHandler () {
stop-all.sh
}
#configuration
. framework-configure.sh spark $SPARK_HOME/conf
#register cleanup hook in case something goes wrong
trap myExitHandler EXIT
start-all.sh
spark-submit --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_2.12-3.0.1.jar 1000
stop-all.sh
exit 0
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment