Skip to content
Snippets Groups Projects
Verified Commit 2eb7dc31 authored by Alexander Dunkel's avatar Alexander Dunkel
Browse files

Minor fixes and improvements for all notebooks (2024 finalization)

parent 591275f5
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
md/01_raw_intro.md
+ 31
− 27
View file @ 2eb7dc31
......@@ -36,18 +36,14 @@ jupyter:
**1. Link the workshop environment centrally from the project folder at ZIH:**
Select the cell below and click <kbd>CTRL+ENTER</kbd> to run the cell. Once the `*` turns into a number (`1`), the process is finished.
Select the cell below and click <kbd>CTRL+ENTER</kbd> to run the cell. Once the `*` (left of the cell) turns into a number (`1`), the process is finished.
```python
!/projects/p_lv_mobicart_2324/intro_env/bin/python \
-m ipykernel install \
--user \
--name intro_env \
--display-name="01_intro_env"
!cd .. && sh activate_workshop_envs.sh
```
<div class="alert alert-warning" role="alert" style="color: black;">
<details><summary style="cursor: pointer;"><strong>Refresh the browser window afterwards with <kbd>F5</kbd></strong></summary>
<details><summary style="cursor: pointer;"><strong>Refresh the browser window afterwards with <kbd>F5</kbd> and select <code>01_intro_env</code> in the top-right corner.</strong></summary>
Refresh the browser window afterwards with <kbd>F5</kbd>, so that the environment becomes available on the top-right dropdown list of kernels.
</details>
</div>
......@@ -271,8 +267,13 @@ query_url = f'https://www.instagram.com/explore/locations/{location_id}/?__a=1&_
```
<div class="alert alert-warning" role="alert" style="color: black;">
<details><summary><strong>Use your own hashtag</strong></summary>
Optionally replace "park" with another hashtag above
<details><summary><strong>Use your own location</strong></summary>
Optionally replace "1893214" with another location above. You can search on <a href="https://instagram.com">instagram.com</a> and extract location IDs from the URL. Examples:
<ul>
<li><code>657492374396619</code><a href="https://www.instagram.com/explore/locations/657492374396619/beutlerpark/">Beutlerpark, Dresden</a></li>
<li><code>270772352</code><a href="https://www.instagram.com/explore/locations/270772352/knabackshusens-underbara-bad/">Knäbäckshusens beach, Sweden</a></li>
<li>You name it</li>
</ul>
</details>
</div>
......@@ -328,6 +329,9 @@ if response.status_code == 429 \
or '<!DOCTYPE html>' in response.text:
print(f"Loading live json failed: {response.text[:250]}")
else:
# write to temporary file
with open(OUTPUT / f"live_{location_id}.json", 'w') as f:
f.write(json_text)
json_text = response.text
print("Loaded live json")
```
......@@ -342,15 +346,8 @@ else:
</div>
Optionally, write to temporary file:
```python
if json_text:
with open(OUTPUT / f"live_{location_id}.json", 'w') as f:
f.write(json_text)
```
If the the url refers to the "login" page (or status_code 429), access is blocked. In this case, you can open the link using your browser and Instagram-Account to manually download the json and store it in Jupyter (left side) in `our/sample.json`. If such a file exists, it will be loaded:
<div style="width:500px">
If the the url refers to the "login" page (or status_code 429), access is blocked. In this case, you can open the link using your browser and Instagram-Account to manually download the json and store it in Jupyter (left side) in <code>our/sample.json</code>. If such a file exists, it will be loaded:</div>
```python
if not json_text:
......@@ -369,12 +366,12 @@ if not json_text:
</details>
Otherwise, ff neither live nor local json has been loaded, load sample json from archive:
Otherwise, if neither live nor local json has been loaded, load sample json from archive:
```python
if not json_text:
sample_url = tools.get_sample_url()
sample_json_url = f'{sample_url}/download?path=%2F&files=live_1893214.json'
sample_json_url = f'{sample_url}/download?path=%2F&files=1893214.json'
response = requests.get(url=sample_json_url)
json_text = response.text
......@@ -414,7 +411,7 @@ You can find the media under:
```python
for ix in range(1, 5):
display(str(json_data["native_location_data"]["ranked"]["sections"][ix]["layout_content"]["medias"][0])[0:200])
display(str(json_data["native_location_data"]["ranked"]["sections"][ix]["layout_content"]["medias"][0])[0:100])
```
Where `[ix]` is a pointer to a list of three media per row. Below, we loop through these lists and combine them to a single dataframe.
......@@ -448,17 +445,21 @@ Display
df.transpose()
```
See an overview of all columns/attributes available:
See an overview of all columns/attributes available at this json level:
```python
from IPython.core.display import HTML
display(HTML(f"<details><summary>Click</summary>{[col for col in df.columns]}</summary>"))
display(HTML(f"<details><summary>Click</summary><code>{[col for col in df.columns]}</code></summary>"))
```
Get the first item (from list of images) with `df['layout_content.medias'].iloc[0]`.
<div class="alert alert-info" role="alert" style="color: black;">
You can have a peer at all attributes of the the first item (from the list of images) with <code>df['layout_content.medias'].iloc[0]</code>.
</div>
We want to extract URLs of images, so that we can download images in python and display inside the notebook.
Extract URLs of images:
First, extract URLs of images:
```python
url_list = []
......@@ -514,6 +515,9 @@ def image_grid_fromurl(url_list: List[str]):
Use the function to display images from "node.display_url" column.
All images are public and available without Instagram login, but we still blur images a bit, as a precaution and a measure of privacy.
```python
image_grid_fromurl(
url_list)
......@@ -1178,7 +1182,7 @@ gv_layers.opts(
hooks=[set_active_tool])
```
```python jupyter={"outputs_hidden": true}
```python
gv_layers.opts(
projection=ccrs.GOOGLE_MERCATOR,
title=df.loc[0, "name"],
......@@ -1290,7 +1294,7 @@ Steps:
<!-- #region -->
**Contributions:**
- **2021 Workshop**: Multigrid display was contributed by Silke Bruns (MA), thanks!
- **2021 Workshop**: Multigrid display was contributed by Silke Bruns (MA), many thanks!
```python
plt.subplots_adjust(bottom=0.3, right=0.8, top=0.5)
ax = plt.subplot(3, 5, ix + 1)
......
md/02_hll_intro.md
+ 15
− 11
View file @ 2eb7dc31
......@@ -43,21 +43,20 @@ Open these notebooks through the file explorer on the left side.
<div class="alert alert-warning" role="alert" style="color: black;">
<ul>
<li>For this notebook, please make sure that <strong>"02_hll_env"</strong> is shown on the
<li>For this notebook, please make sure that <code>02_hll_env</code> is shown on the
<strong>top-right corner</strong>. If not, click & select.</li>
</ul>
</div>
Link the environment for this notebook, if not already done:
```python
!/data/horse/ws/s7398234-p_lv_mobicart_2324/workshop_envs/hll_env/bin/python \
<details style="margin-left: 1em;"><summary style="cursor: pointer;"><strong>Link the environment for this notebook, if not already done.</strong></summary>Use this command in a notebook cell:
<pre><code>
!/projects/p_lv_mobicart_2324/hll_env/bin/python \
-m ipykernel install \
--user \
--name hll_env \
--display-name="02_hll_env"
```
</code></pre>
</details>
</div>
<div class="alert alert-info" role="alert" style="color: black;">
<details><summary><strong>Steep learning curve ahead</strong></summary>
......@@ -1262,8 +1261,9 @@ fig.savefig(
**Save the Notebook**, then execute the following cell to convert to HTML (archive format).
```python
!jupyter nbconvert --to html \
--output-dir=./out/ ./02_hll_intro.ipynb \
!jupyter nbconvert --to html_toc \
--output-dir=../resources/html/ ./02_hll_intro.ipynb \
--template=../nbconvert.tpl \
--ExtractOutputPreprocessor.enabled=False >&- 2>&-
```
......@@ -1305,3 +1305,7 @@ root_packages = [
'matplotlib-venn', 'xarray', 'panel']
tools.package_report(root_packages)
```
```python
```
md/03_tagmaps.md
+ 40
− 16
View file @ 2eb7dc31
......@@ -16,7 +16,7 @@ jupyter:
<div style="width: 100%;display: flex; align-items: top;">
<div style="float:left;width: 80%;text-align:left;position:relative">
<h1>Part 3: Tag Maps Clustering and Topic Heat Maps</h1><br>
<p><strong>Workshop: Social Media, Data Analysis, &amp; Cartograpy, WS 2022/23</strong></p>
<p><strong>Workshop: Social Media, Data Analysis, &amp; Cartograpy, WS 2023/24</strong></p>
<p><em><a href="mailto:alexander.dunkel@tu-dresden.de">Alexander Dunkel</a>
<br> Leibniz Institute of Ecological Urban and Regional Development,
Transformative Capacities & Research Data Centre & Technische Universität Dresden,
......@@ -44,9 +44,18 @@ Open these notebooks through the file explorer on the left side.
<div class="alert alert-warning" role="alert" style="color: black;">
<ul>
<li>Please make sure that <strong>"03_tagmaps_env"</strong> is shown on the
<li>For this notebook, please make sure that <code>03_tagmaps_env</code> is shown on the
<strong>top-right corner</strong>. If not, click & select.</li>
</ul>
<details style="margin-left: 1em;"><summary style="cursor: pointer;"><strong>Link the environment for this notebook, if not already done.</strong></summary>Use this command in a notebook cell:
<pre><code>
!/projects/p_lv_mobicart_2324/tagmaps_env/bin/python \
-m ipykernel install \
--user \
--name tagmaps_env \
--display-name="03_tagmaps_env"
</code></pre>
</details>
</div>
......@@ -125,6 +134,7 @@ Define output directory
```python
from pathlib import Path
OUTPUT = Path.cwd() / "out"
TMP = Path.cwd() / "tmp"
```
Temporary fix to prevent proj-path warning:
......@@ -152,6 +162,8 @@ from tagmaps.classes.shared_structure import ItemCounter
%autoreload 2
```
Import tools used across different notebooks
```python
import sys
......@@ -161,10 +173,6 @@ if module_path not in sys.path:
from modules import preparations
preparations.init_packages()
from modules import tools
# enable ignore shapely deprecation warnings
import warnings
# from shapely.errors import ShapelyDeprecationWarning
# warnings.filterwarnings("ignore", category=ShapelyDeprecationWarning)
```
## Load Data & Plot Overview
......@@ -1063,6 +1071,11 @@ hv.save(
Data interchange is critical. Export Contours and other data used here as Shapefiles (UTM Zone 33N).
```python
from datetime import date
today = str(date.today())
```
**Contours**
For the contourf matplotlib format, we use [flopy](https://github.com/modflowpy/flopy) export.
......@@ -1072,10 +1085,16 @@ import flopy
from flopy.export.utils import export_contourf
epsg_output = tm.clusterer[TOPICS].crs_proj.split(":")[1]
# write a prj file first;
# to be used in the export_contourf function
prj_file = TMP/ f"{epsg_output}.prj"
with open(prj_file, "w") as prj:
epsg_wkt = tools.get_wkt_prj(epsg_output)
prj.write(epsg_wkt)
export_contourf(
str(OUTPUT / '2021-07-12_KDE.shp'),
CF,
epsg=epsg_output)
str(OUTPUT / f'{today}_KDE.shp'),
CF, prj=prj_file)
```
**Alpha Shapes**
......@@ -1102,10 +1121,6 @@ import fiona
from fiona.crs import from_epsg
from shapely.geometry import mapping, Polygon, Point
from datetime import date
today = str(date.today())
# Define a polygon feature geometry with one attribute
schema = {
'geometry': 'Polygon',
......@@ -1161,8 +1176,9 @@ write_shapefile(
## Create Notebook HTML
```python
!jupyter nbconvert --to html \
--output-dir=./out/ ./03_tagmaps.ipynb \
!jupyter nbconvert --to html_toc \
--output-dir=../resources/html/ ./03_tagmaps.ipynb \
--template=../nbconvert.tpl \
--ExtractOutputPreprocessor.enabled=False >&- 2>&-
```
......@@ -1238,10 +1254,18 @@ tools.clean_folders([INPUT, Path.cwd() / "02_Output"])
</div>
```python tags=[]
```python
root_packages = [
'python', 'geoviews', 'holoviews', 'ipywidgets', 'descartes', 'hvplot',
'flopy', 'mapclassify', 'memory_profiler', 'python-dotenv', 'shapely',
'tagmaps', 'matplotlib', 'sklearn', 'numpy', 'pandas', 'bokeh', 'fiona']
tools.package_report(root_packages)
```
```python
!conda list --name worker_env --explicit > ../resources/03_tagmaps.txt
```
```python
```
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung