diff --git a/doc.zih.tu-dresden.de/docs/archive/deep_learning.md b/doc.zih.tu-dresden.de/docs/archive/deep_learning.md index f00b82d4df2caf3a066b229517c3bdfe3c57455e..7747d6f83c7fe1466532e0171a07001cf6998f62 100644 --- a/doc.zih.tu-dresden.de/docs/archive/deep_learning.md +++ b/doc.zih.tu-dresden.de/docs/archive/deep_learning.md @@ -57,7 +57,7 @@ Test case: Keras with TensorFlow on MNIST data Go to a directory on ZIH system, get Keras for the examples and go to the examples: ```Bash -git clone https://github.com/fchollet/keras.git'>https://github.com/fchollet/keras.git +git clone https://github.com/fchollet/keras.git cd keras/examples/ ``` @@ -79,8 +79,7 @@ module load modenv/scs5 # load scs5 environment module load Keras # load Keras module module load TensorFlow # load TensorFlow module -# if you see 'broken pipe error's (might happen in interactive session after the second srun -command) uncomment line below +# if you see 'broken pipe error's (might happen in interactive session after the second srun command) uncomment line below # module load h5py python mnist_cnn.py @@ -105,14 +104,14 @@ validate on 10000 samples Epoch 1/12 val_loss: 0.0268 - val_acc: 0.9911 Test loss: 0.02677746053306255 Test accuracy: 0.9911 ``` -## Datasets +## Data Sets -There are many different datasets designed for research purposes. If you would like to download some +There are many different data sets designed for research purposes. If you would like to download some of them, first of all, keep in mind that many machine learning libraries have direct access to -public datasets without downloading it (for example -[TensorFlow Datasets](https://www.tensorflow.org/datasets). +public data sets without downloading it (for example +[TensorFlow data sets](https://www.tensorflow.org/datasets). -If you still need to download some datasets, first of all, be careful with the size of the datasets +If you still need to download some data sets, first of all, be careful with the size of the data sets which you would like to download (some of them have a size of few Terabytes). Don't download what you really not need to use! Use login nodes only for downloading small files (hundreds of the megabytes). For downloading huge files use [DataMover](../data_transfer/data_mover.md). @@ -120,14 +119,14 @@ For example, you can use command `dtwget` (it is an analogue of the general wget command). This command submits a job to the data transfer machines. If you need to download or allocate massive files (more than one terabyte) please contact the support before. -### The ImageNet dataset +### The ImageNet Data Set The [ImageNet](http://www.image-net.org/) project is a large visual database designed for use in visual object recognition software research. In order to save space in the filesystem by avoiding to have multiple duplicates of this lying around, we have put a copy of the ImageNet database (ILSVRC2012 and ILSVR2017) under `/scratch/imagenet` which you can use without having to download it -again. For the future, the ImageNet dataset will be available in `/warm_archive`. ILSVR2017 also -includes a dataset for recognition objects from a video. Please respect the corresponding +again. For the future, the ImageNet data set will be available in `/warm_archive`. ILSVR2017 also +includes a data set for recognition objects from a video. Please respect the corresponding [Terms of Use](https://image-net.org/download.php). ## Jupyter Notebook @@ -223,7 +222,7 @@ You get a message like that: /home/<zih_user>/.jupyter/jupyter_notebook_config.json ``` -I order to create an SSL certificate for https connections, you can create a self-signed +I order to create an SSL certificate for secure connections, you can create a self-signed certificate: ```Bash @@ -232,8 +231,7 @@ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mykey.key -out mycer Fill in the form with decent values. -Possible entries for your Jupyter config (`.jupyter/jupyter_notebook*config.py*`). Uncomment below -lines: +Possible entries for your Jupyter config (`.jupyter/jupyter_notebook*config.py*`). ```Bash c.NotebookApp.certfile = u'<path-to-cert>/mycert.pem' c.NotebookApp.keyfile = @@ -318,7 +316,7 @@ Jupyter server (example above) you need to change the name of the configuration **Q:** - I have an error to connect to the Jupyter server (e.g. "open failed: administratively prohibited: open failed") -**A:** - Check the settings of your Jupyter config file. Is it all necessary lines uncommented, the +**A:** - Check the settings of your Jupyter config file. Is it all necessary lines not commented, the right path to cert and key files, right hashed password from .json file? Check is the used local port [available](https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers) Check local settings e.g. (`/etc/ssh/sshd_config`, `/etc/hosts`). diff --git a/doc.zih.tu-dresden.de/docs/software/big_data_frameworks_spark.md b/doc.zih.tu-dresden.de/docs/software/big_data_frameworks_spark.md index 6ede1221eb298c306ec663af3f4dc335a7ae8dc4..9b727401644f1791dd999511fde1c6c8fa49cbad 100644 --- a/doc.zih.tu-dresden.de/docs/software/big_data_frameworks_spark.md +++ b/doc.zih.tu-dresden.de/docs/software/big_data_frameworks_spark.md @@ -202,14 +202,14 @@ You are now ready to spawn a notebook with Spark. Assuming that you have prepared everything as described above, you can go to [https://taurus.hrsk.tu-dresden.de/jupyter](https://taurus.hrsk.tu-dresden.de/jupyter). In the tab "Advanced", go -to the field "Preload modules" and select one of the Spark modules. +to the field `Preload modules` and select one of the Spark modules. When your Jupyter instance is started, check whether the kernel that you created in the preparation phase (see above) is shown in the top right corner of the notebook. If it is not already selected, select the kernel `haswell-py3.6-spark`. Then, you can set up Spark. Since the setup in the notebook requires more steps than in an interactive session, we have created an example notebook that you can use as a starting point -for convenience: [SparkExample.ipynb](misc/SparkExample.ipynb) +for convenience: [Spark-Example](misc/SparkExample.ipynb) !!! note diff --git a/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md b/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md index 907df266cdadfc6e5d2cac86c053167cb2e56efe..2d0708ea52d3fc2acfed38df01eea65e6efc56c0 100644 --- a/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md +++ b/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md @@ -12,6 +12,7 @@ a research group and/or teaching class. For this purpose python virtual environm For more details see [here](python_virtual_environments.md). The interactive Python interpreter can also be used on ZIH systems via an interactive job: + ```console marie@login$ srun -p alpha --gres=gpu:1 -n 1 -c 7 --pty --mem-per-cpu=8000 bash #Job submission on alpha nodes with 1 gpu on 1 node with 8000 Mb per CPU marie@alpha$ python