Parse nyan API objects in converter instead of hardcoding them

[heinezen]

Required skills: Python, Cython, nyan, maybe C++

Difficulty: Medium

Currently, the asset converter stores the openage nyan API as hardcoded nyan objects. It would be much better if the converter could read the API objects from the filesystem with a parser. There would be two ways to do this. Either we implement a new Python parser for nyan or we extend libnyan's already existing C++ parser with a Python interface (related issue in nyan repository).

Further reading:

• Current implementation of nyan API loading
• Converter architecture
• Converter workflow

[sarcXD]

Hi, I can try taking a look at this. Should I discuss this in matrix chat or here is fine?

[heinezen]

@sarcXD You can discuss it here, then it doesn't get lost in the chat :)

[sarcXD]

understood, I'll take my time going through this in detail, and post any questions that come up

[sarcXD]

Hi @heinezen I have been going through the mentioned links and the game code and I have a couple of questions.

1. What exactly does the api loader file have, are those nyan objects that are related to the game state? I see that this is used by the converter processor to load the objects that are created in the nyan_api_loader.py file to a dataset.nyan_api_objects. What do these nyan_api_objects do exactly? I have read nyan is useful for modding, are they just for modding or are they being used as loading game files as well.
2. Looking at this initially, I had thought that for all the objects that are created manually in the nyan_api_load.py file, we needed to make the process abit more dynamic. The simplest example I thought of for this, was say those objects were stored in a file, something simple like a json, the functions would then just read the file and created objects from them.
3. Reading Python API nyan#64 and maybe overthinking slightly I got abit confused. So now I am not sure that point 2, is exactly what we wanted. Is there an actual object in the file system, that contains all these objects we are creating in the nyan_api_loader.py, and what you want is to be able to parse them, or am I just overthinking it over here.

Apologies if any of these questions isn't clear, I have gone throught the architecture overview, the workflow docs, the issue, and browsing the converter code, and since I am fairly new to this, most of the things are unclear to me, so I am just trying to get as firm of an understanding of what we need and how this works (for this issue) as possible.

[heinezen]

What exactly does the api loader file have, are those nyan objects that are related to the game state? I see that this is used by the converter processor to load the objects that are created in the nyan_api_loader.py file to a dataset.nyan_api_objects. What do these nyan_api_objects do exactly? I have read nyan is useful for modding, are they just for modding or are they being used as loading game files as well.

The nyan objects defined in the file are the engine API for the gamestate. This blogpost describes how it works in general, although not everything written there is up-to-date.

In short: Data for a game is defined by inheriting from the API objects. All API objects map to functionality in the engine (e.g. unit abilities). When you convert a game with the converter, you will see a bunch of .nyan files in the created folders. These contain the data for the game.

Looking at this initially, I had thought that for all the objects that are created manually in the nyan_api_load.py file, we needed to make the process abit more dynamic. The simplest example I thought of for this, was say those objects were stored in a file, something simple like a json, the functions would then just read the file and created objects from them.

nyan has its own file language that should be parsed, so we don't need JSON. See here for the language spec. The converter currently also outputs files in this language when converting a game. Solving our issue would involve an implementation that can also load the objects defining the engine API from file into the converter (instead of hardcoding the object creation).

Is there an actual object in the file system, that contains all these objects we are creating in the nyan_api_loader.py, and what you want is to be able to parse them, or am I just overthinking it over here?

Yes, the objects in nyan_api_loader.py should be defined in a folder in this repository and then parsed by a function to create dataset.nyan_api_objects.

[heinezen]

@sarcXD I realized that #1313 (comment) might need more explanation for how nyan is integrated into the engine :D

When discussing nyan, we have to differentiate between its two components: nyan the database (aka nyandb), and nyan the language/notation. nyandb stores and updates the gamestate during a game in an object oriented data model. The database is initialized by loading nyan objects from files, written in the nyan notation language.

To give the objects an ingame purpose, the engine also exposes its internal interface via nyan. This is the engine API (or more precisely: the modding API) that is loaded by nyan_api_load.py. The meaning of the objects is described in our docs and in the blogpost I linked. For example, engine.util.game_entity.GameEntity is used to define a unit/building/ingame object in the game by declaring which parameters are necessary. When a game wants to add a new unit type, it inherits from engine.util.game_entity.GameEntity and sets values for the parameters.

For this ussue, it's not so important to understand how this works internally in the nyandb during a game. We only want to load the nyan objects exposed by the engine/modding API into the converter, so that the converter can access the definitions and parameters made by these nyan objects. For example, the converter converts a unit type from AoE2 by first creating a nyan object that inherits from engine.util.game_entity.GameEntity. Then, it fills in the necessary parameter values by looking them up in the AoE2 dataset.

[TheJJ]

I think the best way would be to implement this feature in nyan, since also has nearly all the necessary parts.
Basically nyan would need a feature to provide api information from nyan file(s): parsing it, and returning the structure/types/... conveniently.

The downside of this approach is that we would depend on libnyan at convert time, which wouldn't be too bad, but makes the converter depend on C++ code.

---

The other approach is to parse nyan in Python, so extend the converter to handle nyan-files. That would mean reimplementing a lot of features in py, but we'd remain independent of the c++ libnyan.

---

If I had to choose, I'd go for the first approach, since less (or rather no) redundant implementation. And we wanted to add a Python API to nyan anyway: SFTtech/nyan#64.

[sarcXD]

Hiya, sorry I was afk, basically got busy but I'm picking this up again.