Convert usermods to static libraries #4480
Conversation
Redesign the usermod system so that usermods are implemented as PlatformIO libraries instead of headers. This permits them to call for dependencies, and eliminates the compiler flags for enabling each one, allowing the build cache to behave better. The usermod list is built using some linker magic to construct a static list in ROM memory. This eliminates the need for wasting SRAM on something fixed at build time.
Look for 'usermod_v2_x' as well. This could be removed later if we clean up the folder names.
Borrowed library definition from @netmindz's work on wled-dev#4476.
|
Annoyingly, PlatformIO requires locally-defined properties to be prefixed with 'custom_' ; I'd've liked it to just be 'usermods' but no such luck. :( |
platformio.ini
Outdated
| board_build.partitions = ${esp32.default_partitions} ;; default partioning for 4MB Flash - can be overridden in build envs | ||
| # additional build flags for audioreactive - must be applied globally | ||
| AR_build_flags = -D sqrt_internal=sqrtf ;; -fsingle-precision-constant ;; forces ArduinoFFT to use float math (2x faster) | ||
| AR_lib_deps = kosme/arduinoFFT @ 2.0.1 ;; for pre-usermod-library platformio_override compatibility |
There was a problem hiding this comment.
If we are updating to be a library, why do we still need the old AR_lib_deps info as well?
There was a problem hiding this comment.
Compatibility with old platformio_override files that reference it. If we remove it, git pull breaks platformIO completely until you clean all references out of your overrides.
There was a problem hiding this comment.
If folk have platform_override that is trying to use AR, is having AR_build_flags that that doesn't have the old define just going to cause confusion? They will think they have the flags, but now that's not enough to actually enable
There was a problem hiding this comment.
Yes, this would be a breaking change that will need to be staged appropriately. Unfortunately if your platformio file is outright invalid, PlatformIO in VSCode doesn't tell you what's wrong, it just blankly refuses to operate with no error message. I found it more useful to at least keep the tool operating while I was correcting my build definitions, but that's just my $0.02.
There was a problem hiding this comment.
I'll have a bit of a play around with this over the next few days
It looks like a great start, the key things in my mind however are what extra changes we can make both in terms of code, documentation and release management to help make the transition as painless as possible
One quick question - how essential is the rename from .h to .cpp ?
| @@ -1,482 +0,0 @@ | |||
| #include "wled.h" | |||
There was a problem hiding this comment.
Is it better to delete this file or retain but with message about the new format - What would be the best way to highlight the changes to developers of the may forks?
There was a problem hiding this comment.
Good question! I'd figured the deletion will create an obvious merge conflict, so it's clear that something must be done. Personally I'd rather not leave a .cpp file that the build tooling will have to grovel over only for it to do nothing.
Yes. Unfortunately it looks like the docs aren't in the code proper (beyond the examples), which is a little frustrating when different versions might have different approaches. :(
Some .cpp file has to instantiate the module object; and this must be in the library, or you're back to all the problems with |
How about a migration script that looks for an existing .h of a usermod, moves with git and then automatically added the extra static field and call to register? |
|
Are you aware of the build failure when using the audioreactive usermod? @willmmiles |
Not impossible. but nontrivial - the script would have to pick out the class name and any constructor arguments. We'd probably need to call on |
platformio.ini
Outdated
| @@ -424,23 +420,23 @@ build_flags = ${common.build_flags} ${esp8266.build_flags} -D WLED_RELEASE_NAME= | |||
| board = esp32dev | |||
| platform = ${esp32.platform} | |||
| platform_packages = ${esp32.platform_packages} | |||
| custom_usermods = audioreactive | |||
| build_unflags = ${common.build_unflags} | |||
| build_flags = ${common.build_flags} ${esp32.build_flags} -D WLED_RELEASE_NAME=\"ESP32\" #-D WLED_DISABLE_BROWNOUT_DET | |||
| ${esp32.AR_build_flags} | |||
There was a problem hiding this comment.
Is there any way to move the build_flags used for the usermod into it's library.json?
There was a problem hiding this comment.
In most cases, yes.
In the specific case of AR, the build flags aren't actually for the AR module, they're for the FFT library dependency. Unfortunately, the only way to affect some dependent library's build is to do it at the project level. PlaformIO doesn't let one library define flags for one of its dependencies. https://community.platformio.org/t/setting-flags-defines-for-building-a-librarys-dependency/36744
There was a problem hiding this comment.
Hm, maybe there is a way, using platformio hook scripts in the library. Working on it.
There was a problem hiding this comment.
There are certainly usermods that use more typical build_flags that affect the usermod itself, but perhaps these still need to stay in the main platformio.ini if they are actual optional flags the user chooses to set
There was a problem hiding this comment.
Fixed! It turns out there is a way though platformio hook scripts. 8527d23
Yup, looking in to it - seems like some sort of case sensitivity thing. |
I'm just knocking up a quick PoC of a script. If nothing else it will help show how many of the usermods in the official AC repo do not follow proper naming convention etc |
Yup, a breaking API change like this is a good opportunity for cleanup. Honestly I'd figured on just going through them by hand, there aren't so many it's a huge problem. With a solution like this in hand, I was hoping to ultimately create a "core" modules folder with the modules that this team is directly supporting; and then work towards migrating anything with a "WLED_DISABLE_X" flag in to becoming a core module instead. (Yes, I recognize that will likely require some expansion of the module API to handle specialized timing and polling requirements; I think that's probably a good thing, as if we have one use case for it, there may be others.) Further, the linker section approach demonstrated here for the static usermod list could also be used later to make static library-based FX or bus type lists, too, allowing modules to include new FX or bus types without needing any RAM to build a runtime data structure. One step at a time, though! |
|
I didn't want to mess up your main PR with my attempts at automatic migration, so I've popped into it's own PR for now #4481 These are the ones that don't follow any common naming WARNING: usermods/ADS1115_v2/ADS1115_v2.h missing |
|
Fixed the naming of a few, so now just WARNING: Artemis_reciever possibly still v1 usermod |
I think it might be because we are missing the |
Confirmed. If I hack wled.h to define LOROL_LITTLEFS the problem goes away. Do you have a wled_custom.h or something locally? |
- Check after including wled.h - Use WLED_DISABLE_MQTT instead of WLED_ENABLE_MQTT
A better solution for cross-module communication is required!
Define a couple pins, leave a note where the usermod list comes from.
This is now managed centrally.
|
Known issues in the current state:
|
Don't blast the path of any mentioned library - parse only the tree of the actual build deps.
Only include paths for the base system deps, not those of other usermods.
...it's been 3 years, and it's easier than cleaning up the readme.
It'd be better to not propagate the 'v2' suffix any further. This is the standard flavor of usermods now.
Describe the new usermod enable process, and update sample platformio_override.ini stubs.
Use a custom setup script to check for the dependencies and pass along the required compile flags to the module; also split the object definitions for the target modules from their source so as to allow #including them.
|
Good to see you are still chipping away at this tricky problem. We will definitely be in a much better situation once you complete the work. With regards to your question regarding inter module communication, it is true that they are are some issues with that design of UM data, which we may choose to address at some point, but as it's the best we have currently, then all usermods should comply with the interface and use UM data rather than any other unsupported mechanism |
It wasn't so much a qustion as an observation of the current state of affairs. If mod developers are finding the existing UM data approach too complex or unworkable for their use cases, I'm not going to judge for finding a local escape vs attempting an ecosystem-wide API upgrade. (Which, if we want to take on, we should do soon-ish -- since this PR is already a breaking change, might as well get 'em all done at once!) I consider such changes out of scope for this PR, though. Better to support all the existing code as cleanly as possible, and update the APIs as something that can be reviewed separately. |
This mod includes a header from the Adafruit Unified Sensor library inherited by its target sensor libraries. This isn't reliably picked up by PlatformIO's dependency finder. Add an explicit dep to ensure build stability.
The "arduino-pixels-dice" library needs the ESP32 BLE subsystem, but doesn't explicitly depend on it.
|
@willmmiles & @netmindz for usermods that interact (i.e. call each other) it would be best to include simple API to exchange a few variables or provide events. Perhaps best using JSON as it is flexible.
No other usermods interact. As (almost) all usermods employ |
|
One more thing regarding EDIT: I've prepared a git patch that moves all audio effects into usermod file if you are interested. It gets rid of |
I agree. It also seems like there's a common pattern in "sensors to MQTT" type usermods too. I think putting in a simpler framework for publishing and subscribing to sensor data might be a good plan, and maybe even some common code for routing things from the data API to MQTT. ..but ultimately where I'd like to go is a general facility for routing sensor data (and yes, even audio data) to FX parameters, maybe even with a little formula parser to set min/max/scale/combine/etc. This is still a half-baked idea and would need a lot of fleshing out to handle corner cases, but I want to keep it in mind as a long term goal. I don't think we want to round-trip through JSON for all sensor data, though. That's enough of a hassle (and CPU cost) that I'd expect people to find shortcuts to avoid it, just like you did for the audio data.
I did put in the "gyro surge" partner for MPU6050; it basically routes the net angular rate to the global brightness. They use the um_data API to communicate, though. |
This allows the common newline syntax in platformio
Minor compile correctness tweak
For some reason, building it seems to consume 300kb of SRAM?? Probably there's still something wrong with the configuration.
These can be static locals instead; allows these usermods to build and link together.
Redesign the usermod intergration so that usermods are implemented as PlatformIO libraries instead of headers. This permits them to call for dependencies, and eliminates the compiler flags for enabling each one, allowing the build cache to operate consistently regardless of the selected modules.
The usermod list is built using some linker magic to construct a static list in ROM memory. This eliminates the need for wasting SRAM on something well known at build time.
To use this integration:
.hshould be moved to a.cpp, with a static instantiation of the module class at the endREGISTER_USERMOD()to add it to the compiled-in usermod listcustom_usermods = audioreactive animartrixbuilds in the "audioreactive" and "animartrix" usermodsThe
custom_usermodsproperty is implemented via a platformio hook script that constructs the required symlink lines and injects them in to the build environment.Currently I have converted 'auto_save', 'audioreactive', and 'animartrix' (based on #4476) to library-type as proof-of-concept.