Directories and Metadata using JSON and Mustache

[tajmone]

@thoni56, based on the previous discussions on the List of ALAN Games on IFDB thread on the Alan Google Group, I think that a good development strategy for this repository that would be simple to start with, yet flexible enough to allow future growth without breaking things, could be to:

1. Organize games into a simple directory-based classification system that splits game into: ALAN-2 vs ALAN-3 games; the game's locale; "game slug", i.e. a very-short version of the title (e.g. cloak for "Cloak of Darkness"), which is often already used in the actual storyfile name.

   I.e. adopting the following naming convention(s):

   /alan-2/
       /de/
       /en/
           /<game-slug>/
   /alan-3/
       /de/
       /en/
           /<game-slug>/
   

2. In each game folder, add a JSON file with the game meta-data (title, author, locale, pub. date, alan-version, IFDB link, etc.) following some guidelines provided in the repository documentation (if not a proper JSON Schema).

   These JSON files could then be used in the future for various applications, ranging from the creation of a real database (e.g. for an app using SQLite) to the repository automation itself.

   E.g. once the archive has been populated, I was thinking that we could exploit Ruby and Rake to merge all the different JOSN files into a single in-memory JSON file, and then use mustache templates to automatically generate:

   • A README.md in each adventure folder, presenting the game info (like a library card).
   • A full list of all the games, to be added in the main README.md file.
   • A docs/index.html file with the games list (download links and all) by title, by author, etc., as an single-page HTML website served via GHPages.

I've used mustache templates before to automate the creation of repository contents and/or documentation, and found them to be very easy to work with, especially in Ruby since it's mustache's native-implementation language. The Ruby mustache implementation comes with some extra power which can make its use even simpler, especially with Rake.

The idea of adding a simple JSON file to each game folder seems a reasonable maintenance burden, since contributors are only asked to focus on that specific game, as opposed of having to fiddle with a global JSON games data.

Initially we could simply focus on using this metadata to create the README docs of each folder, to ensure nice previews on GitHub. As the project grows, we can start to add more mustache templates and add the GHPages website, etc. In any case, change of mind about the metadata fields and structure only requires minor adjustments to the mustache templates and the toolchain, since its a very simple (yet powerful) logic-less templating system.

What I like most about this approach is these JSON files could be used for so many other things, including their translation to other JSON Schemas so that they might be exported to other tools (databases, etc.). If any such third party tools was to request us to add extra JSON keys to help better support their product, their addition wouldn't harm our project since it's all "background data" for the toolchain.

How do you see this proposal? Would it work well with the online database project you've set up? (i.e. in terms of sharing data in either or both directions)

Any suggestions?

[thoni56]

First I think we need to set the context and declare the need we are thinking to fulfill, so we are on the same page. Please comment if there is an aspect that I've missed.

1. If we do not trust IF Archive/DB to host games "forever" then we need to store the actual games here
2. If we think what's lacking is a better search mechanism, using GitHub would no give us that
3. If there is extra metadata that we think is essential to add, it needs to be stored somewhere
4. There is currently no natural place for source code for games

Given those and the discussion on the list, I felt that item 2 was the most urgent to fulfill.

I'm not at all fond of using flat file structures to store multi-dimensional structured data, so I immediately thought "database".
A quick way to do that was to dump the information for the games that you had gathered into a simple database.

And, as I think we can trust the community to keep the IF Archive/DB up for a long time, and at least give a heads up when it needs to be taken down or move, the database can be seen as a "front-end" to the IF Archive/DB. It is easy to export/download the data in .csv format available for all visitors. We can easily add other meta-data, like locale, etc. and it would almost instantly be possible to search, filter and order according to that new data. If we should duplicate metadata that is already in the IF Archive is probably a balance, is it important for searching, then yes, otherwise, no.

Caspio turns out to be very simple to use if you've tried Microsoft Access or LibreOffice Base. And I'm sure there are, and will be, a number of similar alternatives. The free plan gives us everything we need.

(The DB is, and probably should stay, a simple one-table implementation. I was tempted to break out an "author" table, but decided against normalizing it because a simple .csv is easier for anyone to import into their application of choice, and also author information is more complex, as you mentioned in your latest group post. Free text searches sometimes rules ;-), especially since I amended the filtering to use "contains" instead of "equals" for the title and author fields.)

So my thinking would be to use this repo as a "backing store" where we could periodically dump the exported data from Caspio for safe-keeping. Nothing more, nothing less. (If GitHub has a built-in .csv viewer that would be wonderful, giving some structure for the visitor to this repo without any work for us.)

Remains 4) the "Alan Games Source Code Repository". Perhaps for this purpose a structure like you suggest would be appropriate. Again I'm thinking that the "Alan Games DB" would then have links here in the same way as it links to the actual games in the IF Archive. Then the only structure here that would make sense to me is the v2/v3 division since if you are browsing this repo for source examples you are looking for either v2, or more probable, v3. If you want the source for a particular game, you'd probably go directly from the games page in the DB.

[thoni56]

I definitely believe that the IF Archive is the best storage for any IF related assets, since its sole goal is the preservation of the IF legacy, and it's a non-profit organization recognized by the government.

The IF Archive does also store adventures' source code, if available, so that's not an issue either.

Good. I didn't notice that. It has actually been years since I seriously browsed the IF Archive...

My main argument[s]

• modernisation
• single "page"
• creating collections
• complete source

I think those are good arguments and also explains clearly the difference between this and the IF Archive, usable vs. archive, if you will. We should be very clear about this in our descriptions and communications around this repo.

And as long as we have nothing to add for a game, refer to the IF Archive. Then this repo would grow as we modernize, tweak and make each game compilable.

Bear in mind that the number of ALAN games is not huge, and I don't think it will grow much in the next decade either, so we'll be speaking of 60 to max 100 games at all times.

Yes. 57 at this point ;-)

Having said that, it's worth mentioning that I am at odds with databases — I fail to grasp their organizing logic (even though I've studied all the "normal forms" over and over gain) and struggle hard when it comes to modelling a database. So, since I don't particularly enjoy working with databases I don't have much practical experience with them (nothing much beyond maintenance work I had to do for one reason or the other).

You are thinking about relational databases like MSQL, MySQL etc. They tend to be overused, sometimes as a persistent store for only half-structured data. To use those you need to have normalizable data, and you should never work with them directly, only through you application.

Their main use is when you have Giga/TeraBytes of data that you need to query. If that data can be normalized you can build efficient indices over the tables that can increase lookup performance thousand-fold. With 57 records that will never be the case, linear search will do fine ;-)

Although MS Access, LibreOffice Base and Caspio allows for creating relational data there is no need to do that:

The DB is, and probably should stay, a simple one-table implementation. I was tempted to break out an "author" table, but decided against normalizing it because a simple .csv is easier for anyone to import into their application of choice [...]

I agree. The more articulated data could be stored here, using JSON or YAML files that go with each adventure in its folder (as mentioned before). I don't remember which is the native format supported by mustache for Ruby (i.e. JSON or YAML), since different implementation use one or the other, but I'd go for the one that works best with mustache.

So my thinking would be to use this repo as a "backing store" where we could periodically dump the exported data from Caspio for safe-keeping. Nothing more, nothing less.

I think that it would be a good idea to dump here these backups, but I obviously see benefits in populating the repository too.

Perhaps we can find some format that would be useful both ways. Although I think I'd prefer to add searchable attributes/fields to the "search database" and export, rather than the other way around, because of how export/import/gui-design tend to work in these tools.

Maybe adding the "slug" to the data in the DB would facilitate its propagation to the game folders in some form.

(If GitHub has a built-in .csv viewer that would be wonderful, giving some structure for the visitor to this repo without any work for us.)

It indeed does, as long as there are no formatting errors — for more info see Rendering CSV and TSV data. Here's an example of how it looks like:

https://github.com/goodby/csv/blob/master/example/user.csv

but I have no idea whether URLs are rendered as clickable links or not (from the above example I can see that emails are not).

As you can see from the data I uploaded, we get nice presentation, but URL:s are not recognized and auto-turned into links. Marking and right-clicking "follow link" in your browser is fairly easy, though.

Remains 4) the "Alan Games Source Code Repository". Perhaps for this purpose a structure like you suggest would be appropriate. Again I'm thinking that the "Alan Games DB" would then have links here in the same way as it links to the actual games in the IF Archive.

I think that links should always point to the IFDB game page, ...

Sorry, meant to say "IFDB". ... have links here in the same way as it links to the actual games in the IFDB

[tajmone]

Yes. 57 at this point ;-)

But that's mostly just the English games. If we add to this the Spanish and Italian demos the number is well over 60, and I guess there are more English demo and tutorial adventures that are simply not included — for some reason, no one deemed it worth adding to the IFDB implementations and translation of Cloak of Darkness, but we might want to do this, at least for the "pure alan" (no-library) implementations — cloak is an example of an adventure that would have to be updated fairly often, since each library has its own cloak demo, which gets updated with the library.

Perhaps we can find some format that would be useful both ways. Although I think I'd prefer to add searchable attributes/fields to the "search database" and export, rather than the other way around, because of how export/import/gui-design tend to work in these tools.

CSV should be up to the task, if we keep things simple enough. JSON/YAML offer the benefit of structured data and collections, and are still considered simple and popular formats (JSON at least is, whereas YAML is harder to parse).

I agree that the online DB should self-sufficient in this respect, and just export this data. Any needed changes should be done manually on the DB directly, since at the import-end tools have ample freedom to manipulate the data as they please.

Maybe adding the "slug" to the data in the DB would facilitate its propagation to the game folders in some form.

I'm not entirely sure if this might help the DB. We probably won't end up having a "bullet proof" consistent way of coming with adventures slugs, but rather compromise between adopting those native game shorthands that were good enough to use and custom ways of shortening long titles. Our main issue is going to be adventures with spaces in their filename, which in some contexts are going to be problematic. The slug alone is only going to be useful if we're sure that it's shared by both folder AND storyfile names, but this might not be the case (unless we take the liberty of renaming storyfiles, but I'm not sure authors would be happy with that, they tend to protective of such choices).

As you can see from the data I uploaded, we get nice presentation, but URL:s are not recognized and auto-turned into links. Marking and right-clicking "follow link" in your browser is fairly easy, though.

In any case, AsciiDoc can import CSV files as table contents, which means that any URL will become a link at preview/conversion time, but also that some fields might contain simple formatting markup. The downside is that GitHub disables all AsciiDoc preprocessor directives in online previews, so in order to benefit from this a repository would have to coalesce all the sources in order to create a README.adoc which end users can benefit from. That's why I think that using local JSON files and mustache templates is the most flexible solution for a wide range of uses of this data.

A Side Note on DBs

You are thinking about relational databases like MSQL, MySQL etc. They tend to be overused, sometimes as a persistent store for only half-structured data.

Yes I meant those, since this is generally what's understood when you mention database without further specifications — although there are many other types, like no-SQL, etc. In fact, no-SQL key-value databases (e.g. redis) is what most people really need most of the time, but these tools are not as widely known as MySQL, MariaDB, etc.

The Internet has probably been the main cause of databases overuse and abuse(s), where everyone seems to need a full-fledged and powerful relational database as MySQL (now MariaDB), regardless of the number of uses and pages. I personally only every used db-driven tools (like PHP-Nuke, WordPress, etc.) when there was a real need for it, like multi-user multi-level access for editing contents, since databases add a huge overhead to a website, make its service slower, and open the door to all sort of vulnerabilities. I'd more often just use a custom engine based on mod_rewrite .htacess rewrite rules and dynamic fetching of contents snippets from disk.

To use those you need to have normalizable data, and you should never work with them directly, only through you application.

The "normalizing" aspect of databases is where I get lost. I simply tend to overthink it and deem it necessary to create to many independent tables, keys, etc. — i.e. I fail to see things in a simple way, but I guess this comes with practical experience. But there are other "grudges" too, like when I see OODB — as if the OOP paradigm (or hype) hasn't done already enough damage in the programming world (speaking of uses and abuses).

Their main use is when you have Giga/TeraBytes of data that you need to query. If that data can be normalized you can build efficient indices over the tables that can increase lookup performance thousand-fold. With 57 records that will never be the case, linear search will do fine ;-)

Indeed, although SQLite might be an exception to this, since it's designed to be used as a standalone engine by small-to-big applications, and does without the server-client model. In fact, it's been used many times over as a means to store proprietary data-formats for applications projects, etc.

But all those issues aside, my main grudge with databases is that I realize they strongly coupled with the idea of server-client. I've tinkering for a long time with the idea of creating an IF catalogue to discover, download and share text adventures and their assets (feelies, etc.), in the form of a distributed database where each user can add its own entries and then share them with others. But I soon realized that without a central server or authority it would be hard to enforce uniqueness of certain keys (e.g. authors) and prevent ending up with redundant entries for a same author. As mentioned on the ALAN list, there simply isn't enough data about game authors to grant a unique-ID mechanism. Even attempting solutions like fingerprinting wouldn't work due to the lack of info (e.g. like Git's hashing method to uniquely identify blobs, etc.).

So, ultimately this tool would most likely have to operate via some double-standard, separating the "officially endorsed" data from user-contributed data, and somehow integrate the latter via updates. But this undermines the idea of a distributed and serverless tool, although a possible workaround could be to allow multiple authoritative data sources to coexist, and adopt a cryptographic signature mechanism to allow authors to channel their "official" updates.

I've been wondering whether these limitations are due to the ways databases have been used historically, or if they are intrinsic to the nature of cataloguing challenges and unrelated to the relational database model and software paradigms. But I can definitely see how the server-client mentality is pivotal to how databases work. This is also true for the Internet, but a decentralized Internet is not only possible but tools have been created for that goal; it's just that all the Internet Economy of big corporations revolves around the server-client model, so money is what ultimately shapes the Internet, its tools and how we use them. Maybe for databases is the same? I simply don't know.

[thoni56]

Maybe adding the "slug" to the data in the DB would facilitate its propagation to the game folders in some form.

I'm not entirely sure if this might help the DB. We probably won't end up having a "bullet proof" consistent way of coming with adventures slugs, but rather compromise between adopting those native game shorthands that were good enough to use and custom ways of shortening long titles. Our main issue is going to be adventures with spaces in their filename, which in some contexts are going to be problematic. The slug alone is only going to be useful if we're sure that it's shared by both folder AND storyfile names, but this might not be the case (unless we take the liberty of renaming storyfiles, but I'm not sure authors would be happy with that, they tend to protective of such choices).

I was thinking that each game directory would be named as the "slug" and the "metadata" from the DB could then easily be made to populate some "metadata file" in that directory, just as a start and making it simple. What files in it would be called would not matter then, and that information could be populated manually, or the two data "sets" merged using some scripting, like I think you are suggesting.

I just don't want us to overdo, or, for that matter, overthink, this. Keep It Simple and evolve overtime. What would be the simplest first thing we could do here? Is there a game we'd like to collect first, given the arguments you presented? Let's do that, with the ideas and the vision still in mind, but not starting with the tooling. Then we can always change structure and add tooling as we go.

[tajmone]

I just don't want us to overdo, or, for that matter, overthink, this. Keep It Simple and evolve overtime. What would be the simplest first thing we could do here? Is there a game we'd like to collect first, given the arguments you presented? Let's do that, with the ideas and the vision still in mind, but not starting with the tooling. Then we can always change structure and add tooling as we go.

Agreed. I've started to pick various games from the list, from different authors, download all their related assets and look into them, so we can have a better view of what type of contents we're speaking of, and how various authors have organized them in their games distribution packages.

Since every author has his/her own approach when it comes to releasing games (some include WinArun binaries, other the ALAN sources, etc.), I'm trying to work out both the minimum baseline of essential files involved as well as the full spectrum of possible files, so that we can adopt an approach that works well with every adventure.

As I'm doing that, I'm also jotting down notes on whether an adventure is being stored at the IF Archive or not, so that I'll be able to submit those that aren't (i.e. which are stored at private websites, which is where their IFDB link is pointing for downloads).

The sampling process is probably going to take some days, since I want to cover both old ALAN-2 games and more recent ALAN-3 games, and I'm already seeing that some distributions contain unexpected interesting extra assets, like elaborate HTML walkthroughs containing adventure info, maps and transcripts, created by third parties, and might find other assets like game-map project files for IF mapping tools, etc.

I'll update you once I have a better picture of the contents we'll be working with.

[thoni56]

That look really interesting! Looking forward to seeing this evolve based on your findings!

[thoni56]

As you can see I experimented with exporting data from Caspio and it turns out that the CSV displays as a table by GitHub, which is great. It would have been even better if the links would have been recognized too, but you can't have everything.

It's possible to export as XML, Excel or even Microsoft Access as well, if we think that would be better (but I think not).