Skip to content

Commit

Permalink
feat: Update all Static-Storage sources to mapdata (#114)
Browse files Browse the repository at this point in the history
I'm really proud/happy for this change. Finally, it's coming all
together.

These files were 1-1 ported from their old representation to mapdata.
Still, they have more useful information to the map system, like levels.
Labels have unique, but readable feature ids as well, while services
continue to have numbered unique ids.

Combat locations were built from the ground up, with a new script,
replacing the old, hand-made data. This is done by extracting data from
the content book dumps. For backwards-compatibility, I'm adding Grind
Spots and Uth Shrines from the old data (these are still mostly relevant
locations in-game).
Changes for combat locations:
- Include levels for every location (except the Grind Spots and Uth
Shrines)
- Include corrupted dungeons (missing previously)
- Include lootrun camps! These were added almost a year ago, but we
still have not had a location on the map. This changes now.
- Merge caves with combat locations. We used to have a separate data
source for them, but there is no reason to keep them separate.

---------

Co-authored-by: kristofbolyai <[email protected]>
  • Loading branch information
kristofbolyai and kristofbolyai committed Jun 4, 2024
1 parent dcd848f commit 4521950
Show file tree
Hide file tree
Showing 8 changed files with 12,275 additions and 2 deletions.
6 changes: 5 additions & 1 deletion Data-Storage/raw/content/content_book_dump.json
Original file line number Diff line number Diff line change
Expand Up @@ -20661,7 +20661,11 @@
"3 Galleon\u0027s Graveyard Fragments",
"Various Items"
],
"location": null
"location": {
"x": -587,
"y": 39,
"z": -3522
}
},
{
"type": "dungeon",
Expand Down
15 changes: 15 additions & 0 deletions Data-Storage/urls.json
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,21 @@
"md5": "38bb5299f66016271f5bf3a1f49ec0ae",
"url": "https://raw.githubusercontent.com/Wynntils/Static-Storage/main/Data-Storage/major_ids.json"
},
{
"id": "dataStaticMapdataCombatLocations",
"md5": "22e909d5a81accad983af78d547ee695",
"url": "https://raw.githubusercontent.com/Wynntils/Static-Storage/main/Reference/combat_locations_mapdata.json"
},
{
"id": "dataStaticMapdataPlaces",
"md5": "feec214529dcf318085d3978098f87c8",
"url": "https://raw.githubusercontent.com/Wynntils/Static-Storage/main/Reference/places_mapdata.json"
},
{
"id": "dataStaticMapdataServices",
"md5": "c84ea9dfea44549d579c376fc904be35",
"url": "https://raw.githubusercontent.com/Wynntils/Static-Storage/main/Reference/services_mapdata.json"
},
{
"id": "dataStaticMaps",
"md5": "3d1825eee8011ca34fce25a5a3597fc0",
Expand Down
69 changes: 69 additions & 0 deletions Generators/extract_combat_locations.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#!/bin/bash

# Get the base directory
BASE_DIR="$(cd $(dirname "$0")/.. 2>/dev/null && pwd)"

# Set the content and output directories
CONTENT_DIR=$BASE_DIR/Data-Storage
OUTPUT_DIR=$BASE_DIR/Reference

# Create the output directory if it does not exist
mkdir -p $OUTPUT_DIR

# Function to map and transform data
transform_data() {
jq 'def map_category(type; name):
if type == "bossAltar" then "boss-altar"
elif type == "lootrunCamp" then "lootrun-camp"
elif type == "dungeon" then
if (name | startswith("Corrupted ")) then "dungeon:corrupted"
else "dungeon"
end
elif type == "raid" then "raid"
elif type == "cave" then "cave"
elif type == "Rune Shrines" then "shrine"
elif type == "Grind Spots" then "grind-spots"
else type
end;
map({
featureId: (map_category(.type; "") + "-" + (.name | gsub(" "; "-") | gsub("[^a-zA-Z0-9\\-]+"; "") | ascii_downcase)),
categoryId: ("wynntils:content:" + map_category(.type; .name)),
attributes: (if .requirements.level then {
label: .name,
level: .requirements.level
} else {
label: .name
} end),
location: (.location // .coordinates)
})'
}

# Read, transform, and write the JSON data from the primary source
primary_data=$(cat $CONTENT_DIR/raw/content/content_book_dump.json | jq '[
.dungeon[], .raid[], .bossAltar[], .lootrunCamp[], .cave[]
]' | transform_data)

# Read, transform, and write the JSON data from the secondary source, filtering for 'Rune Shrines' and 'Grind Spots'
secondary_data=$(cat "$CONTENT_DIR/combat_locations.json" | jq '[
.[] | select(.type == "Rune Shrines" or .type == "Grind Spots") |
{type, locations} | (.locations[] |
{
name: .name,
coordinates: .coordinates,
type: .type
}) + { type: .type }
]' | transform_data)

# Combine primary and secondary data
combined_data=$(echo "$primary_data" "$secondary_data" | jq -s add)

# Write the combined data to the output file
echo "$combined_data" > $OUTPUT_DIR/combat_locations_mapdata.json

# Calculate md5sum of the new cave data
MD5=$(md5sum $OUTPUT_DIR/combat_locations_mapdata.json | cut -d' ' -f1)

# Update urls.json with the new md5sum for dataStaticMapdataCombatLocations
jq '. = [.[] | if (.id == "dataStaticMapdataCombatLocations") then (.md5 = "'$MD5'") else . end]' < $BASE_DIR/Data-Storage/urls.json > $BASE_DIR/Data-Storage/urls.json.tmp
mv $BASE_DIR/Data-Storage/urls.json.tmp $BASE_DIR/Data-Storage/urls.json
38 changes: 37 additions & 1 deletion Generators/extract_labels.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#
MYDIR=$(cd $(dirname "$0") >/dev/null 2>&1 && pwd)
TARGET="$MYDIR/../Reference/places.json"
TARGET_MAPDATA="$MYDIR/../Reference/places_mapdata.json"

# This file contain additional labels that are not provided by upstread.
MISSING="$MYDIR/../Data-Storage/map-labels-missing.json"
Expand Down Expand Up @@ -87,11 +88,46 @@ case 14:
cat "$TARGET.tmp" "$MISSING" | jq -s '{labels: .}' | jq --sort-keys ".labels|=sort_by(.name)" > "$TARGET"
rm "$TARGET.tmp"

# Calculate md5sum of the new maps.json
# Calculate md5sum of the new places.json
MD5=$(md5sum $TARGET | cut -d' ' -f1)

# Update urls.json with the new md5sum for dataStaticPlaces
jq '. = [.[] | if (.id == "dataStaticPlaces") then (.md5 = "'$MD5'") else . end]' < $MYDIR/../Data-Storage/urls.json > $MYDIR/../Data-Storage/urls.json.tmp
mv $MYDIR/../Data-Storage/urls.json.tmp $MYDIR/../Data-Storage/urls.json

echo Finished updating "$TARGET"

jq '.labels[] | {
featureId: ("labels-" + (.name | gsub(" "; "-") | gsub("[^a-zA-Z0-9\\-]+"; "") | ascii_downcase)),
categoryId: ("wynntils:place:" + (if .layer == 1 then "province"
elif .layer == 2 then "city"
else "town-or-place" end)),
attributes: {
label: .name
},
location: {
x: .x,
y: 0,
z: .z
}
} + (if .level != null then {level: (
if .level | test("^\\d+$") then
(.level | tonumber)
elif .level | test("^\\d+-\\d+$") then
(.level | split("-")[0] | tonumber)
elif .level | test("^\\d+\\+$") then
(.level | gsub("\\+"; "") | tonumber)
else
null # Default value in case of unexpected format
end
)} else {} end)
' < "$TARGET" > "$TARGET_MAPDATA"

# Calculate md5sum of the new places_mapdata.json
MD5=$(md5sum $TARGET_MAPDATA | cut -d' ' -f1)

# Update urls.json with the new md5sum for dataStaticMapdataPlaces
jq '. = [.[] | if (.id == "dataStaticMapdataPlaces") then (.md5 = "'$MD5'") else . end]' < $MYDIR/../Data-Storage/urls.json > $MYDIR/../Data-Storage/urls.json.tmp
mv $MYDIR/../Data-Storage/urls.json.tmp $MYDIR/../Data-Storage/urls.json

echo Finished updating "$TARGET_MAPDATA"
50 changes: 50 additions & 0 deletions Generators/extract_services.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ TARGET_DIR=$(cd $(dirname "$0")/.. >/dev/null 2>&1 && pwd)/Reference
cd $TARGET_DIR

TARGET="services.json"
TARGET_MAPDATA="services_mapdata.json"

# Download the json file from Wynncraft API
wget -O markers.json.tmp "https://api.wynncraft.com/v3/map/locations/markers"
Expand Down Expand Up @@ -89,3 +90,52 @@ MD5=$(md5sum $TARGET_DIR/$TARGET | cut -d' ' -f1)
# Update urls.json with the new md5sum for dataStaticServices
jq '. = [.[] | if (.id == "dataStaticServices") then (.md5 = "'$MD5'") else . end]' < ../Data-Storage/urls.json > ../Data-Storage/urls.json.tmp
mv ../Data-Storage/urls.json.tmp ../Data-Storage/urls.json

jq '
def to_feature_id(type; index):
(type | ascii_downcase | gsub(" "; "-") | gsub("[^a-zA-Z0-9\\-]+"; "")) + "-" + (index|tostring);
def map_type(type):
{
"Alchemism Station": "profession:alchemism",
"Armour Merchant": "merchant:armor",
"Armouring Station": "profession:armoring",
"Blacksmith": "blacksmith",
"Booth Shop": "booth-shop",
"Cooking Station": "profession:cooking",
"Dungeon Scroll Merchant": "merchant:dungeon-scroll",
"Emerald Merchant": "merchant:emerald",
"Fast Travel": "fast-travel",
"Housing Balloon": "housing-balloon",
"Item Identifier": "identifier",
"Jeweling Station": "profession:jeweling",
"Liquid Merchant": "merchant:liquid-emerald",
"Party Finder": "party-finder",
"Potion Merchant": "merchant:potion",
"Powder Master": "powder-master",
"Scribing Station": "profession:scribing",
"Scroll Merchant": "merchant:scroll",
"Seaskipper": "seaskipper",
"Tailoring Station": "profession:tailoring",
"Tool Merchant": "merchant:tool",
"Trade Market": "trade-market",
"Weapon Merchant": "merchant:weapon",
"Weaponsmithing Station": "profession:weaponsmithing",
"Woodworking Station": "profession:woodworking"
}[type];
[.[] | .type as $type | .locations | to_entries | .[] |
{
featureId: to_feature_id($type; .key),
categoryId: ("wynntils:service:" + map_type($type)),
location: .value
}
]
' < $TARGET > $TARGET_MAPDATA

# Calculate md5sum of the new gear data
MD5=$(md5sum $TARGET_DIR/$TARGET_MAPDATA | cut -d' ' -f1)

# Update urls.json with the new md5sum for dataStaticMapdataServices
jq '. = [.[] | if (.id == "dataStaticMapdataServices") then (.md5 = "'$MD5'") else . end]' < ../Data-Storage/urls.json > ../Data-Storage/urls.json.tmp
mv ../Data-Storage/urls.json.tmp ../Data-Storage/urls.json
Loading

0 comments on commit 4521950

Please sign in to comment.