Magicmirror: Discussion: Simplify the core.

[MichMich]

Since I open sourced MagicMirror², I have received many awesome pull requests adding a lot of nice features to the Magic Mirror core. I am very grateful for that! Unfortunately, this comes with a downside: the application gets a lot more complicated.

The reason I open sourced MagicMirror² was because it is a perfect starter project for newcomers. Getting their feet wet in javascript and software development in general (see manifesto). With the current complexity, this project is slowly moving away from this.

I also noted that the amount of bash code is increasing, while the main intention of version 2 was to try to do everything in 1 program language: javascript. (Opposed to v1, which was a combination of languages like PHP, Javascript en in some cases Python).

Looking at other OS projects, there are some beautiful examples on how the project can be focused at what it does best. One example I like to mention is OctoPrint/OctoPi. Software to control your 3D printer. While @foosel does a great job to maintain the core application (OctoPrint), the packaged Rasberry Pi image (OctoPi) is maintained in a different repo by a different user (@GuySoft).

If we use that same approach there are a lot of things we can take out the core repo, and let other users start their own repo's to accomplish the same features while separating concerns.

In my opinion, the following parts could initially be taken out of the core repo, and taken over by other repo's:

• Installer script
• Update script
• Start Script
• PM2 integration
• Docker Package

Of course, we won't take out these features as long as no other solutions are in place. As an install and update replacement, we need to write a good manual.

The added benefit of this approach is that the release schedule of the above features is no longer tied to the quarterly release schedule of MagicMirror².

With the new approach I think it's good to set a few guidelines to which the future versions should adhere:

• Javascript, HTML, CSS only.
• No compile scripts (ie. webpack, sass, less)
• Focus (only) on Raspberry support.
• Install/Run/Update using default commands: npm install, git pull, etc. Which will help people to learn and understand these commands.

Since this project would have never been such a success without the help of the community, I want to discuss this approach before taking any action. So I'm looking forward to your input.

Please let me know what you think with an open mind.

Thanks! Michael.

EDIT: Provisional TO DO List

Based on the discussion below, I made a provisional to do list which could be the base for the simplification of this repository. The to do list is not set in stone, and could change based on the feedback in this issue.

• [X] Remove installer scripts (make sure an external repository is available as a replacement).
• [X] Remove update scripts.
• [X] Remove start script.
• [X] Separate Client & Serveronly commands as regular npm commands: npm run start & npm run server.
• [X] Cleanup folders (remove /installers & /splashscreen and remove all root .sh files). Where necessary, replace shell code with javascript code.
• [ ] Implement handler for incompatible modules.
• [X] Create separate documentation site and repo.
• [x] Improve documentation on how to install and update. This should reflect the procedure without an installer/updater, but it should also mention the available third party installers/updaters.
• [X] Make an overview of the hard- and software requirements.
• [ ] Make an overview of the supported (tested) hardware and operating systems.

Feel free to share your ideas and feedback!

[rejas]

Very much agree with your reasoning and your overview of parts that dont have to be part of the core MM package (although I never looked into the "non-html/js/css" stuff since I am more of a frontendler and just happy abotu scripts that do all the setup stuff :-)

As for the guidelines on what future versions should adhere too, I think it is missing on what browsers beside the electron engine should be supported.

[sdetweil]

for serveronly you can use any browser that might work.. most do.. as electron is built on chromium,. the chromium_browser for the platform is used.

the difficulty with the objective

Install/Run/Update using default commands: npm install, git pull, etc. Which will help people to learn and understand these commands.

is that is not all that is needed.. while the core may have been updated, there are many many modules that now have their own intertwined dependencies on the electron version , grpc version and more.. change the core, and they are dead... the users don't know, they just installed..

the good news is MM has reached a wide non-developer audience.. its also the bad news

and additional thought.. the install on pi0 and 1 has become very complicated as the older OS versions (jessie and stretch) are not getting updates.. npm install leaves a mess.
I don't mind the above being separated, but how can they be referenced from the core to save users all these troubles..

the vast majority of the users are on pi devices.. but many more are coming.

[MichMich]

Removing the installer from the core does moet mean there won’t be an installer. It can be a separate repo/application that does exactly the same. By splitting the two, we can have more focus. Advanced users won’t need an installer. Beginners can download the installer if they want.

This way the core becomes much more maintainable and we won’t run into issues like #1858.

I personally think it’s better to focus on a better reporting system which handles updated packages. This way the users will learn why the magic mirror won’t start so they can choose to update or remove specific modules.

Regarding the browser support: we should focus on electron (chromium) this way it will also work in chrome. Old browsers can’t be supported due to new JS features.

[rejas]

Would be good to write that explicitly down: MM is developed for electron, but if you want to use mm in serveronly mode, here is what your browser should at least support.

[khassel]

• Focus (only) on Raspberry support.

I would focus on docker images as primary installation method.

Mainly on arm32v7 (raspi), but other linux architectures are no problem. I think many users first try the magicmirror with a serveronly-setup without a real mirror on e.g. ubuntu.

The official docker image provided by @bastilimbach - only serveronly - has the incredible pull count of nearly 3 Million !!!

There were discussions concerning the docker setup in 2017,
finally the decision was to build only a serveronly-setup.

But you can run also run magicmirror as docker image on a raspi with electron (npm start).

You only need to install docker (and if desired docker-compose) on your pi.
Installation or update of the magicmirror is done with one statement (docker pull ...).

I'm running such a setup now for more than 2.5 years with no problems.
For updating modules, I'm using the remote control modul.

With such an approach you could determine to support only the docker installation
and there is no need to provide install or update scripts anymore (whereas everyone is still free to do a manual install ...).

[MichMich]

@khassel I understand what you are saying, en indeed the pull count is impressive. But docker isn’t something beginners start with. For me personally I value the simplicity of a simple js project more than the sheer amount of users that simply install and run the script. If that was the case, I would just distribute a ready to use raspberry pi image.

If you take a look at the list of contributors, a lot of them made their fist ever GitHub contribution in the MagicMirror² project. This is what sets the project apart. This is what drives me.

If we slink MagicMirror² down to a simple js project that can either be start as a simple electron app or as a server only process, wouldn’t it be easy for someone else to make and distribute a nice docker wrapper?

In that case we can focus on the quality of the core, while others focus on the ease of use via a docker image.

This would also allow multiple approaches: docker, installer apps, ready to use images, etc.

[MoreLinuxDev]

@MichMich, I like the idea simplifying the core and focus on one programming language.
I think the project should focus on current (supported) Raspberry PI's only. Yes that leaves out some older PI's and maybe the PI 0, but that's ok. Running MM on old PI's or the PI 0 is something an expert would try, but an beginner just buy's an starter kit and gives it a try.

What I often ran into during my use of the MM over the last 1.5 year or so is that modules are a one shot create and for get by someone who created it, but doesn't maintain it in the future. Modules then fail due to a dependency on the MM version or on some external API that has been updated.

So what would be great as part of the core MM would be a system that blocks old and unsupported modules without breaking MM.

Also keeping everything on github would be preferred. I mention this because the 2.10.0 updated gave me some extra work with the update script. As the 2.10.0 was released and you wrote "git pull && npm install", but in that same topic @sdetweil wrote to use his update script, and then pointing to his Dropbox.

To me, being more admin then developer, that raised a few red flags as I just don't blindly run something from someones Dropbox. No offense @sdetweil, that is just me being cautious.

@sdetweil -Server / -client only is not something where an beginner starts with. They buy a Raspberry starter kit and give it a go with a normal monitor connected. If this then works out of the box, the the spark might ignite and they might build an actual mirror. If the core becomes to complex and failing at the first steps of setting up the PI, we would loose the beginner as there are so many other possible PI projects that one can try.

@khassel, making the core in docker makes it unnecessary complex. A beginner buy's a starter kit and gives it a go that specific PI. No other apps are running there. So putting MM in docker would be a overkill as the PI itself is the container. Doing MM in docker is again an expert thing.

[MichMich]

Yes that leaves out some older PI's and maybe the PI 0, but that's ok. Running MM on old PI's or the PI 0 is something an expert would try, but an beginner just buy's an starter kit and gives it a try.

I agree. I think this is comparable to other software. For example: iOS13 won't run on a iPhone 4. If we keep trying to make it work on all devices possible, we are holding back the development of the framework.

If someone really wants to use an older raspberry, they can just use an older release of the framework.

So what would be great as part of the core MM would be a system that blocks old and unsupported modules without breaking MM.

That was exactly my thought: if you start the mirror, and one of the modules gives an error. you get a (1 minute) alert stating: "Module X seems to have problems and is disabled by mm2. Please check logs/date_module_x.log for more info." - This shouldn't be too complicated to incorporate and solves a lot of issues.

Regarding the -serveronly/-client versions: if a user knows he wants to use the server-only version. He is probable able to find out he should run npm run server-only.

In the end I think al current possibilities (installers, docker, etc) can remain. The only difference is they can be split up in multiple repositories, managed by different users. This enables different release schedules and makes it much more manageable for me to maintain in the long run.

The documentation will reflect the possibilities:

Manual installation:
- Use the following steps ...

Automatic installation
- Take a look at one of the following repositories ...

[roramirez]

Hi all!,

There main problem here is because the all script and maintenance of installer system to be a tricky, many variety of devices is a lot of cost keep compatibles with all their.

Actually at now, the scripts bash installer is a lot spaghetti code (without offenses) and trouble to tests.

At the beginning of MM was use a X Raspberry model to build a (DIY) Awesome Mirror but at now there a many version of Raspberry and many environments where should can run the software. Imagine this in a next years. Keep old and next future version will going keep our effort to give maintenance the "installer code" system instead to the core.

So, at to all this, I think we need keep the installer like is at this moment for a new release and make an announcement for the next release as 3.0.0 where the all code related a configuration and installer will be remove.

[sdetweil]

I agree that it needs maint. But installer is a big word. For example, if u separate, then we need control of package.json and the runtime env, and the Travis stuff too.

As an example, we currently need to upgrade electron for new pi, and buster. But cannot. We shouldn't have to lag 6 months

This gets harder and harder to separate.

I didn't create the installer. But the original no longer worked. As this latest also won't at some time without maintenance.

[MichMich]

I agree that it needs maint. But installer is a big word. For example, if u separate, then we need control of package.json and the runtime env, and the Travis stuff too.

The installer as it currently is, could be a in a separate repo. As a matter of fact, you shared new versions of the upgrade script and installer (if I'm not mistaken) on the forum straight from your dropbox. So the installer doesn't need to be in the same repo. Correct me if I'm wrong, but I get the feeling you are against this idea. What are the cons in your opinion? Maybe we can solve those cons in another way.

As an example, we currently need to upgrade electron for new pi, and buster. But cannot. We shouldn't have to lag 6 months.

This is a whole separate issue and has no effect on the installer. I agree we need to fix this Travis issue. As a matter of fact, this is a great example of why I think we need to simplify the core: let us focus on electron and the framework, while others put resources in de development of installers etc.

This gets harder and harder to separate.

This proves my point. The longer we wait. The harder it becomes. Thats why I started 2020 with this discussion.

I didn't create the installer. But the original no longer worked. As this latest also won't at some time without maintenance.

An other reason why I want to remove it from the core. It isn't maintained properly. I don't have the resources to maintain it. If we separate it from the core, someone else can take that responsibility.

In my opinion, I think there is nothing wrong with setting a requirements list to run MagicMirror². This should include stuff like: node version 10 or higher and Raspbian version X.X.X. The requirements list could even limit the types of supported Raspberries (hardware).

It's not that it won't run outside that requirements list, it's just not guaranteed.

@roramirez I agree with your reasoning, but I don't think we should release it as a new major version. It's not a major paradime shift. It's not an API change. In my opinion it could just be 2.11.0, 2.12.0 or 2.13.0.

Whenever we switch to a complete new setup (for instance single file components) which will break existing modules, we should switch to 3.0.0.

Also note that the core could still include features to allow external repo's to work. For example, the core could still include features like defining an external config.js path, so that it can work in a docker container.

Edit: formatting.

[khassel]

An other reason why I want to remove it from the core. It isn't maintained properly. I don't have the resources to maintain it. If we separate it from the core, someone else can take that responsibility.

with a separation new versions or bugfixes of the installer are possible at any time, they are not bound at a mm release anymore.

Also note that the core could still include features to allow external repo's to work. For example, the core could still include features like defining an external config.js path, so that it can work in a docker container.

bad example 😜, docker can mount any path outside the container, so there is no need for a special path for the config.js ...

[MichMich]

with a separation new versions or bugfixes of the installer are possible at any time, they are not bound at a mm release anymore.

Exactly!

bad example 😜, docker can mount any path outside the container, so there is no need for a special path for the config.js ...

Whoops! Well, you get the point. 😅

[MoreLinuxDev]

bad example stuck_out_tongue_winking_eye, docker can mount any path outside the container, so there is no need for a special path for the config.js ...

And that's why docker isn't for beginners, as experts don't even always get it :smile:

[cowboysdude]

@MoreLinuxDev I disagree on the take on the modules. What modules are initially is made by a user to create something they want and is shared by that user. My understanding with MM is still the same... The user should be taking an active part in the development of their own mirror. All of this is done for free and given away as such. The problem with 'supporting' modules from what I've seen in the last couple years is that it goes from 'being nice and sharing' to a full time job because user a wants the module to do this and module b wants the module to that. It's not that they aren't being maintained it's more like here is something for free, like MM, and the user should want to take an active part. I started with MM when there really was no installer script and I learned through trial and error how to solve my own problems and that 1st mirror is still running :) Until this I didn't even know javascript..... but I asked and learned and modified modules to fit my own needs.. the repo's are free....
But overall I agree back to roots is the best thing to do here. This was built to run on a 35.00 board but it's getting too heavy for that.

[sdetweil]

in preparation, I have created a separate repo for these files.
https://github.com/sdetweil/MagicMirror_scripts
and changed the topic links

also, collecting the pr fixes there

@MoreLinuxDev thanks for the note about concern for dropbox.. now moved to github..
I was trying to be responsive to users for updates as we have been testing these for the last 6 months.

[MoreLinuxDev]

@sdetweil, thanks for all the work put into the scripts.

@cowboysdude Yes that is the way you do it (many thanks for that) and the way it should be done, but reality is different. A nice example is the 3th party voice control area there are several modules abandoned.

Currently in the module area it's a bit wild west. The only way to figure out if a module would run is to install it and hope. Would be great if there is a way to find out if a module would be working under the new 2.10.0.

The way I do it right now is to have a dev "mirror" ( a Pi hooked-up to a monitor) and a real prod mirror. When a new MM version arrives I first do the dev, and see what breaks. If al goes fine I do the prod mirror.

Nowadays it mostly goes quite well, but that's because I reduced my mirror to the really essential needed modules that I know are actively maintained. The unmaintained modules I have abandoned as well.

I found this out the hard way when I wanted to get my MM voice controlled. So from my point of view it would make MM more user (beginner) friendly if the was a system in place were old and non-working modules are recognized and stopped from breaking things.

[sdetweil]

@MoreLinuxDev oh I wish it was just the MM core code that was the problem with updates..

we have a rapidly changing node/npm base,
an electron base where we are woefully behind, but latest doesn't work
the dependencies are also not being kept up with the rest
and new hardware which requires updates to the above (PI 4 runs buster only, which requires new node and electron)

oh, and then there are the modules..

I have three modules (one public) and 2 of the three have broken cause of node, electron and dependency changes.

[MichMich]

Since we are starting to get a consensus on the desired directions, I've edited the opening post and added a provisional TO DO List. Open for feedback and let me know if I forgot something.

Not that this TO DO list only covers the simplification procedure. It goes without saying that all other ToDo's (like upgrading Electron) remain.

Thanks!

[rejas]

What are your thoughts on having all repos (core, scripts, installers,) under one organizational account? Or are you fine with the core being on your account, the installer on someone elses?

[MichMich]

I prefer the later. I want to stay focused on the core. Others can focus on the many ways of installing/usage. Especially since there isn’t just one way to install it.

One person could maintain the installer. Someone else could maintain a docker image. And who knows there someone willing to maintain an .img file.

This way the possibilities of installing the mirror won’t be limited to one “preferred” method.

[tbouron]

Hi @MichMich, I thought I would add my 2c since you are asking for it :)

Regarding the stack, I understand were you are coming from. Typescript would be nice to use IMO but the latest versions of JS are nice enough. I would argue that webpack is completely transparent to new comers once it's in place so using it doesn't introduce overhead. Same applies to preprocessors really. Although that requires to learn about mixins and special syntax, but new comers might not know CSS so is learning less or Sass much of deal at that point?

I am a committer for few Apache projects and most of them are structured the way you describe: each part has a git repo, and the main repo just gather all these. Now I like this approach a lot. This is nice indeed as the MagicMirror core will be able to live its own life. I would also go all the way down and separate each "official" module (each one being a git repo) as dependencies of the core.
However, this approach can also be a burden (talking from experience) especially when you want to perform changes that impact different parts: PRs are multiplied and you cannot do that in one go. This is something to bear in mind. That being said, I don't think it's an issue for MagicMirror given the nature of the project.

What I'm more concerned about is the follow-up of the "distribution channel". Right now, the installation is packaged with MagicMirror but if tomorrow this is separated, that can died for any reason: no time, no maintainers, etc. True, it won't be "too much" of an issue as you can still install it manually and do all the scripting yourself, but this is a real risk that can impact users.
Don't get me wrong, I really wish it will actually expand the scope and we will see various different packaging like:

• docker
• raspbian (and other distro) package
• PI image
• etc

Hope that make sense.

[MichMich]

@tbouron Thanks for your feedback!

As much as I like TypeScript, Less/Sass myself, I really want to stay away from it. It is a big step for newcomers to learn. Especially the tooling necessary. Most contributors started by changing a small line of code and seeing direct results. For example: there are a lot (way more than you would expect), kids classes teaching kids on how to program with the MagicMirror² as an example. Letting kids change lines of code in a text editor and see the results only works if they don't need to run precompilers.

Also keep in mind that the "best stack" has changed a lot in the past couple of years (since I initially released the project). If I had used those types of preprocessors back then, we would still be using them is this project today, even though a lot better alternatives are available.

I also honestly think that due to the simplicity of the application, adding complexity like webpack would not add a lot of value. Don't get me wrong: I love them in all of my projects in my professional day job. But for this project KISS really is key.

For that same reason I will keep the default modules inside this one repo. That way starters can clone one repo and start playing. It keep things simple for them. And better yet: it keeps things simple for me to maintain.

Regarding your concern on the unmaintained installer scripts: I absolutely agree. But that problem is even bigger if the they are part of the core AND unmaintained. In my opinion we should work to make the manual installation as easy and smooth as possible (by writing good documentation). That way an installer might become obsolete one day.

Thanks again for your feedback!

[bastilimbach]

@MichMich Unfortunately I just now had the time to contribute to this discussion. I very much agree with your decision to modularise MagicMirror into separate modules like core, installer/upgrader, etc.

Just to be clear you don't want to separate the core in such a way that it is consumed as an NPM dependency, right? Let's say you publish this project under mm-core which just exposes the app.js and a separate module uses this to create the server.

One concern that I have is regarding the quarterly releases. As I'm the maintainer of the docker-MagicMirror I faced one problem with regards to keeping my repo up to date with the current MagicMirror release. We currently don't have an automatic way to detect when a new MM release is published therefore I'm forced to create a daily cronjob on a Continuous Integration system (currently Travis CI) which pulls the new code every day and publishes a new docker image regardless of when a new release is published. This is needed to keep my docker image up to date and provide the newest MagicMirror release.
I could imagine that an official Linux distribution (something like OctoPi) would also face this problem. The maintainer would be forced to check if a new release of MM is published/ready and update the distro manually. (Usually, just a retrigger of a build, which could also be done automatically)

Btw as these are major changes to the whole project we need to increase the version to 3.0.0 as we are breaking the compatibility be getting rid of serveronly etc.

[sdetweil]

@bastilimbach
Btw as these are major changes to the whole project we need to increase the version to 3.0.0 as we are breaking the compatibility be getting rid of serveronly etc.

we are not getting rid of this at all.. we are just using it more where we need it

[bastilimbach]

@sdetweil in the Provisional TO DO List @MichMich mentioned the removal of serveronly

[sdetweil]

@bastilimbach ah, didn't read that., too busy bailing
he didn't mention removing, but moving the folder out of the MM root and adding an npm script command entry for it

[MichMich]

Just to be clear you don't want to separate the core in such a way that it is consumed as an NPM dependency, right? Let's say you publish this project under mm-core which just exposes the app.js and a separate module uses this to create the server.

I understand the value of doing that. But for simplicity sake I want to keep the core in one repo (for now).

One concern that I have is regarding the quarterly releases. As I'm the maintainer of the docker-MagicMirror I faced one problem with regards to keeping my repo up to date with the current MagicMirror release. We currently don't have an automatic way to detect when a new MM release is published therefore I'm forced to create a daily cronjob on a Continuous Integration system (currently Travis CI) which pulls the new code every day and publishes a new docker image regardless of when a new release is published. This is needed to keep my docker image up to date and provide the newest MagicMirror release.
I could imagine that an official Linux distribution (something like OctoPi) would also face this problem. The maintainer would be forced to check if a new release of MM is published/ready and update the distro manually. (Usually, just a retrigger of a build, which could also be done automatically)

As you say, this is the same for many apps (like OctoPi). Maybe a solution would be to run a update in the initial start of the docker image / os .img? Just a simple git pull && npm install should eventually be enough.

Btw as these are major changes to the whole project we need to increase the version to 3.0.0 as we are breaking the compatibility be getting rid of serveronly etc.

I won't take out server only. I'll only streamline the commands to npm run start and npm run server. I will keep /serveronly for backwards compatibility. I'll change this in the provisional TO DO list.

[khassel]

One concern that I have is regarding the quarterly releases. As I'm the maintainer of the docker-MagicMirror I faced one problem with regards to keeping my repo up to date with the current MagicMirror release. We currently don't have an automatic way to detect when a new MM release is published therefore I'm forced to create a daily cronjob on a Continuous Integration system (currently Travis CI) which pulls the new code every day and publishes a new docker image regardless of when a new release is published. This is needed to keep my docker image up to date and provide the newest MagicMirror release.

I think daily updates are too much. I use the subsription on the releases for mail notification, and I look into the new release info before updating (using release tags instead of branch master). You could may automate something with git describe --abbrev=0 --tags which delivers the latest tag on the current branch.

[MichMich]

I took the first steps by creating a separate docs site:
https://docs.magicmirror.builders

These docs come from a separate repo: https://github.com/MichMich/MagicMirror-Documentation, and are updated when ever I merge a PR in that repo.

I've quickly added all the separate docs into that site, but not that it's far from finished/polished.

Feel free to send PR's.

[MichMich]

I've just released a minor release which includes links to the new docs site. This way readme won't link to a broken installer.

[MichMich]

@sdetweil Can you tell me (or send a PR) with the files I can remove from the core now that your repo contains the installers?

[sdetweil]

@MichMich PR #1874

[MichMich]

Thanks!

[bastilimbach]

@MichMich I created PR #1875 which updates the GitHub issue template to make sure that issues that are not related to the core are posted in the respective GitHub repos.
For now, this includes my docker image repo and the installer from @sdetweil.

[sdetweil]

i'll have to fix the issue template.. text says 'official' installer, while its actually an alternative, automated installer.

the 'official install' is npm install

[MichMich]

I agree with you @sdetweil. I've updated the issue template accordingly.

[chriscn]

Could the whole project be moved to a Organisation? Would make the organisation easier, similarly to what OctoPrint has done?

[bastilimbach]

@chriscn Already answered here: https://github.com/MichMich/MagicMirror/issues/1860#issuecomment-570939842

[MichMich]

I’ve removed the run-start script and cleaned up the install commands. Starting the mirror can now be don by using one of the following two commands:

• npm run start
• npm run server

Using ‘DISPLAY=:0` is no longer necessary. If the DISPLAY variable isn’t set :0 will automatically be used.

I tested the develop branch successfully on a Pi4 as well as a Pi3. Please let me know if any of you run into issues.

Building a error catcher for problematic modules is up next.

[sdetweil]

@MichMich will this untrack run-start.sh?

you need to announce armv6l no longer supported or works

that anyone with a working solution on a pi 0, pi 1, or armv6l will need to do some work after upgrade to get the thing to run again.
and anyone trying to install, must build something like the new run-start to get it to work at all.

[guysoft]

@MichMich Hey,
Sorry to enter the show late.
So you want to make a CustomPiOS distro for MagicMirror? I could help with that.
Its really easy to create and then have an automatic build for it. I can even donate my build server to make nightly build servers.

Can you point me to the current install script? What do you need to run to install the current version?
I can see how to install it, but then I suppose you want it to start on boot or something like that?

You can have the UI start on boot using CustomPiOS's gui module. Similar to what is used in FullPageOS.

[MichMich]

@sdetweil Indeed Pi 0 and 1 are no longer officially supported. Will add that to the release notes. Of course anyone that wants to use a pi 0 or 1 kan just continue to use 2.10.

I’ll add extended release notes to this release anyway.

@guysoft I don’t intent to build something like that. But anyone is free to start their own repo which does this.

I personally prefer a bare installation because this helps people to understand how it works.

[guysoft]

@MichMich I can help with the initial repo, but I don't think it would be high on my maintenance list, also I have no way to test it since I have no MagicMirror. Did the same for https://zynthian.org and just finished https://photoprism.org/ . Created a repo and a build.

Its my experience that people learn less when, for example, they have to install NodeJS and electron first. They are more likely to contribute code when they know exactly where it goes. Your call, but IMHO won't consider installation a learning and understanding phase.

[sdetweil]

@guysoft as for the learn less... i have scripts that automate the core functions or install and upgrade.
https://github.com/sdetweil/MagicMirror_scripts, the point being to get them up and running, improve initial acceptance and help retain them.

also, you don't need a mirror... it will run on just about anything.. pi, Windows Ubuntu, Mac, linux, ...
the install script will deal with all the difference issues

[guysoft]

@stefsims Again, I can write an initial repo, but I am not going to do all of it.
Those scripts are a start, but I would suggest basing it upon the gui module in CustomPiOS. Its a module that installs and configures all of the X stuff that I see your script does. It would mean less code to maintain in the long run.

I can write a CustomPiOS repo that runs your script, and we can see what comes out the other side.
Its my experience that if you take this as a joint effort, then it would last better. Its just I am maintaining already OctoPi, FullPageOS, ZynthianOS, RealtimePi and recently PhotoPrismPi and just started working on MtigOS. And there are a few others. So I would like to see some commitment before I start working on this and then get requests to fix stuff. I you try and I improve CustomPiOS documentation as a result.

[cowboysdude]

@sdetweil Indeed Pi 0 and 1 are no longer officially supported. Will add that to the release notes. Of course anyone that wants to use a pi 0 or 1 kan just continue to use 2.10.

I’ll add extended release notes to this release anyway.

@guysoft I don’t intent to build something like that. But anyone is free to start their own repo which does this.

I personally prefer a bare installation because this helps people to understand how it works.

No, it ONLY helps people who understand and know programming on how it works. It does not help the 'basic' guy/gal who is just starting out. I have stuck with this since it started and it took me 4 full days of 'figuring' it out before it ever worked once. Having the installer makes it easier for 'novice' users or aren't we interested in those people? It's only logical to have an installer because it's actually helping the project grow.

Personally me and a few other people gave up any hope of seeing this progress because there really is none unless you count progress as programmer friendly only and that's why you've lost some very good people that no longer bother to even come to the forums or here anymore. I mean look at other forum you set up....it's basically just an extension of the main forum with people asking the same questions there hoping you, yourself, may actually respond, which you rarely if ever talk to the 'little people'. Making it easier makes more sense. Lets face it this is about as bare bones as it gets..... while I've watched others like Evan Cohen's mirror progress we've not seen the same here. You have basically a full time VERY experienced programmer at your discretion and you are choosing to basically tell him 'Too bad go start your own.'

Not to mention that when someone asks questions of any of you directly you have chosen to just ignore them so please don't say it's because that way people can see how it work because that's just not a valid reason when many of them have ZERO idea what programming really is. I could care less if you ever talk to me.....frankly I think you are a rude person. But this move is not by any means a 'smart move'........ Against my better judgement I donated 100 US to this project as I wanted to see it mature but it's really not and that's disappointing.

[MichMich]

@cowboysdude I'm not sure if or how I should respond to your comment. Calling people rude isn't helping anyone. Especially when talking about open source software. Keep in mind that any work I have done for the past 6 years on the MagicMirror² project was completely voluntarily. I have no obligations to you or anyone else. The fact that you donated to the project doesn't change that.

That being said: I think it's important to explain that I'm absolutely not against the idea of "automatic installers", "ready made raspberry images" or anything else that makes life easier for beginners. It's just that I can't manage all these components myself. Pulling them out as external repositories allows the maintainers to maintain those components without my intervention. @sdetweil's hard work on the installer is a perfect example.

In the past years the project has grown a lot. This came with a huge downside: poor maintainability. If I let the project continue the way it was, it would eventually crash and burn. More important: I would completely lose the drive and interest to continue the project. By splitting up the project, this all can be prevented and will eventually improve the quality of the core. Of course this means we have to get trough some rough patches. And there certainly will be some moments of discussion and incomprehension. But keep in mind It's only to keep the project running.

I'd also like to point out the forum is not intended to ask me for help. It's a place where users can help each other and share their knowledge. Answering all questions I get would be more than a day job. I already have a day job. It pays my mortgage and has paid for the first 5 years of the MagicMirror² project. My time is limited. So I rather spent it on improving and releasing updates of the framework than answering individual questions.

As said before: donations don't give me the obligation to help individuals out. That's not the idea behind an open source project like MagicMirror². Donations are used to pay for the costs of running the project. I don't want to give anyone the impression I'm doing this for the money. If that's what anyone thinks, they couldn't have been further from the truth. Therefor I'd like to make an exception and return your donation. Please send me an email (michael at xonaymedia.nl) to confirm your PayPal adres and I'll transfer the money.

In the future. I recommend you think before you call someone rude. Everyone is fighting their own battles. Calling names won't make things magically better. Especially not at times like these.

Kind regards,
Michael.

[Kuhlemann]

Is there a (community-made) list or a forum thread for modules which are known to break by MM release 2.11.0?
This would make things a lot easier. ;-)

[trinibouy]

Hi everyone I have recently gotten into coding and I am still very much wet behind the ears. The magic mirror was supposed to be an easy project to give me some confidence and boost morale. I dont mind following coding and finding a way to load everything onto the raspberry pi however I cannot see anything that could help me right now. I am totally lost and a bit off putting.

I am willing to put in the work and find things out but it seems to me right now that unless i spend the next 5 years working on things I wont be able to participate in the joy of coding and physically see somethings manifest.

I would really appreciate if someone could point me in the right direction to continue my coding experience to build the magic mirror.

Thank you in advance

[sdetweil]

@trinibouy this should be in the forums and not here..

documentation for module development is here
https://docs.magicmirror.builders/development/introduction.html

you can use my Sample Module which implements all the stuff (not nunjuks templates)
https://github.com/sdetweil/SampleModule

like any project, some things are easy and some are hard. early learning on coding and design will make it a little bigger hill to climb. (more stuff to learn at once)

on the forums, come to the module development section
https://forum.magicmirror.builders/category/6/development

[guysoft]

@trinibouy If you feel like starting a CustomPiOS-based image for MagicMirror I would be happy to support you making and maintaining one.

You can start with this:
https://github.com/guysoft/CustomPiOS/wiki/Building#quickstart---i-want-to-build-a-new-distro-using-docker

You will need to install docker on your machine, google is your friend for that, feel free to ask questions.

[sdetweil]

there are 2 docker distributions already

[guysoft]

@sdetweil if Magicmirror has a docker image then making a flashable image makes life easier.

You could just use the docker module in CustomPiOS. You can place a docker-compose.yml in the folder /boot/docker-compose/magicmirror and it will automatically pull and start the docker-compose file on boot. You can test it with the MtigOS image that has it installed:
https://github.com/guysoft/MtigOS

Just take the image released there, and change the files in the /boot/ folder as described above, should work if you have the right docker-compose.yml. I don't mind setting a repo with it if you can give me a working yaml file.

[khassel]

you can use this yml-file for testing, just rename it to docker-compose.yml.

You may have to adjust the left sides of the volume-mounts for config and modules and the timezone env var TZ.

[guysoft]

>

I can see its missing

1. X running

2. All this stuff outside the image:

   • ../mounts/config:/opt/magic_mirror/config
     
     
     
     • ../mounts/modules:/opt/magic_mirror/modules
     
     
     • /etc/localtime:/etc/localtime:ro
     
     
     • /tmp/.X11-unix:/tmp/.X11-unix
     
     
     • /opt/vc:/opt/vc/:ro
     
     

[khassel]

don't know anything about MtigOS ...

You can take a look at the requirements here for running MM with docker in debian buster on a pi.

X11 should be installed in buster, you need docker and docker-compose.

The config.js and modules are mouted out of the docker image, for testing only you can omit them, the MM will start with default config:

• ../mounts/config:/opt/magic_mirror/config
• ../mounts/modules:/opt/magic_mirror/modules

Only needed for getting localtime from the pi:

• /etc/localtime:/etc/localtime:ro

Needed for X11

• /tmp/.X11-unix:/tmp/.X11-unix

This is a special case for getting system temperature, you can omit this:

• /opt/vc:/opt/vc/:ro

[khassel]

for further discussions we may should open a new issue ...

[guysoft]

@khassel I feel like this is approaching my comment at: https://github.com/MichMich/MagicMirror/issues/1860#issuecomment-592153475 .
Again, I am kind uncomfortable putting the time an effort to a project I can't really test without a mirror, and without anyone else willing to put the time too to maintain it.

Your answer only opens up more design questions:

• If you are installing the GUI, what else should run with it? The GUI session needs some start script? xterm?
• The docker daemon should start after the GUI has started, so it can access it (otherwise I don't understand how it would start a GUI application from the container) - where do you set that?

I could create an initial image build script that takes this in to account, but I need someone to stand up to maintain it.

[sdetweil]

@guysoft >can't really test without a mirror,

you don't need a 'mirror', this thing will run on anything, including windows. macos, gallium on chromebook, ubuntu, zorin, raspian of course,

[guysoft]

@sdetweil Its my experience that if you don't have the hardware you can't really fully grasp what people are doing with your software.

[guysoft]

I got board arguing with you, here:
https://github.com/guysoft/MagicMirrorOS

Its a distro script that has docker, a gui, and unclutter installing.

It should start the GUI and then docker with the docker-compose.yml that @khassel supplied.

It should take an hour to build on my Jenkins and then a few to upload it. So it hopefully that should happen if there are build errors, which I will check for tomorrow.

[khassel]

did already some tests with MtigOS, will take a look at MagicMirrorOS after work ...

[guysoft]

Cool, @khassel thanks, I flashed it but didn't boot it yet. It has the X server installed which MtigOS hasn't. Also Docker should start after the GUI.

[guysoft]

It boots in to a login screen, let me see if I can fix that

[guysoft]

Ok, fixed something with the init scripts, it seems like the mirror is running but its a blank screen,

Another build is in progress.

here is the log:

> [email protected] start /opt/magic_mirror
> DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-04-30 11:03:22.763] [LOG]    Starting MagicMirror: v2.11.0
[2020-04-30 11:03:22.790] [LOG]    Loading config ...
[2020-04-30 11:03:22.800] [LOG]    Loading module helpers ...
[2020-04-30 11:03:22.803] [LOG]    No helper found for module: alert.
[2020-04-30 11:03:22.901] [LOG]    Initializing new module helper ...
[2020-04-30 11:03:22.902] [LOG]    Module helper loaded: updatenotification
[2020-04-30 11:03:22.904] [LOG]    No helper found for module: clock.
[2020-04-30 11:03:23.612] [LOG]    Initializing new module helper ...
[2020-04-30 11:03:23.613] [LOG]    Module helper loaded: calendar
[2020-04-30 11:03:23.614] [LOG]    No helper found for module: compliments.
[2020-04-30 11:03:23.615] [LOG]    No helper found for module: currentweather.
[2020-04-30 11:03:23.616] [LOG]    No helper found for module: weatherforecast.
[2020-04-30 11:03:23.667] [LOG]    Initializing new module helper ...
[2020-04-30 11:03:23.668] [LOG]    Module helper loaded: newsfeed
[2020-04-30 11:03:23.669] [LOG]    All module helpers loaded.
[2020-04-30 11:03:23.897] [LOG]    Starting server on port 8080 ... 
[2020-04-30 11:03:23.911] [INFO]   You're using a full whitelist configuration to allow for all IPs
[2020-04-30 11:03:23.918] [LOG]    Server started ...
[2020-04-30 11:03:23.919] [LOG]    Connecting socket for: updatenotification
[2020-04-30 11:03:23.920] [LOG]    Connecting socket for: calendar
[2020-04-30 11:03:23.921] [LOG]    Starting node helper for: calendar
[2020-04-30 11:03:23.922] [LOG]    Connecting socket for: newsfeed
[2020-04-30 11:03:23.923] [LOG]    Starting module: newsfeed
[2020-04-30 11:03:23.924] [LOG]    Sockets connected & modules started ...
[2020-04-30 11:03:24.223] [LOG]    Launching application.

[sdetweil]

@guysoft usually means the vendor and fonts npm installs did not get done
cd ~/MagicMirror/vendor
npm install
cd ../fonts
npm install

[guysoft]

@sdetweil This is the point I am stopping. Again I DONT WANT TO MAINTAIN THIS.

I have given you the platform and code and I am willing to bring it to the point it boots. I am using the TV of my girlfriend place to test it indeed boots, I have no spare Pis or screens to lend to this. From there you are on your own. maintaining stuff like the npm packages can be done inside the docker container.
I can even run nightly builds against the repo or any other CustomPiOS distro you like and even provide the hosting, but I don't want to maintain this.

I am getting this to the point you get a gui and the repo boots. I am also happy to help with CustomPiOS related issues, but thats all I am willing to lend a hand with. I just have too much on my plate and this is volunteer work.

[sdetweil]

@guysoft i hear you, but I don't want to maintain it either..
my install scripts were removed from the main repo, because they require frequent ongoing maintenance (more than the once/quarter MM release cycle) and it was felt this didn't contribute to the core objective.

[guysoft]

Ok, the current nightly build (only on at the moment).

1. Boots
2. Reaches a xterm, and then you need to wait, the script pulls the latest docker image.
3. starts magic mirror, which shows a black screen on the HDMI, the output from docker logs mm is:

start magicmirror

> [email protected] start /opt/magic_mirror
> DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2020-04-30 16:15:16.044] [LOG]    Starting MagicMirror: v2.11.0
[2020-04-30 16:15:16.052] [LOG]    Loading config ...
[2020-04-30 16:15:16.057] [LOG]    Loading module helpers ...
[2020-04-30 16:15:16.059] [LOG]    No helper found for module: alert.
[2020-04-30 16:15:16.106] [LOG]    Initializing new module helper ...
[2020-04-30 16:15:16.107] [LOG]    Module helper loaded: updatenotification
[2020-04-30 16:15:16.108] [LOG]    No helper found for module: clock.
[2020-04-30 16:15:16.489] [LOG]    Initializing new module helper ...
[2020-04-30 16:15:16.489] [LOG]    Module helper loaded: calendar
[2020-04-30 16:15:16.490] [LOG]    No helper found for module: compliments.
[2020-04-30 16:15:16.490] [LOG]    No helper found for module: currentweather.
[2020-04-30 16:15:16.491] [LOG]    No helper found for module: weatherforecast.
[2020-04-30 16:15:16.516] [LOG]    Initializing new module helper ...
[2020-04-30 16:15:16.517] [LOG]    Module helper loaded: newsfeed
[2020-04-30 16:15:16.517] [LOG]    All module helpers loaded.
[2020-04-30 16:15:16.635] [LOG]    Starting server on port 8080 ... 
[2020-04-30 16:15:16.641] [INFO]   You're using a full whitelist configuration to allow for all IPs
[2020-04-30 16:15:16.647] [LOG]    Server started ...
[2020-04-30 16:15:16.648] [LOG]    Connecting socket for: updatenotification
[2020-04-30 16:15:16.649] [LOG]    Connecting socket for: calendar
[2020-04-30 16:15:16.649] [LOG]    Starting node helper for: calendar
[2020-04-30 16:15:16.650] [LOG]    Connecting socket for: newsfeed
[2020-04-30 16:15:16.650] [LOG]    Starting module: newsfeed
[2020-04-30 16:15:16.651] [LOG]    Sockets connected & modules started ...
[2020-04-30 16:15:16.717] [LOG]    Launching application.

If I connect to: http://magicmirroros.local:8080/ I can see this:

If you update the docker image it will pull the latest one when the device boots.

I think I have given you guys a good start, I am not maintaining this, I am willing to accept and manage PRs.

[khassel]

@guysoft thanks, this looks very good! But I missed one volume mount which is the reason for the black screen. Will make a PR in your repo later or tomorrow.

[guysoft]

Ok, so we simplified it. You flash the image. It works. It updates on boot.
thanks @khassel for code and testing.
https://github.com/guysoft/MagicMirrorOS

[TheTyrius]

@MichMich With the effort to separate concerns into multiple repositories, what would you say about hosting an "awesome-magicmirror" repo besides (and preferably (?) under the same namespace as) mm-core?
For anyone unfamiliar with this type of list here would be an example: https://github.com/vuejs/awesome-vue#resources
In this list users would be able to find links to other repos that contain installers, updaters, ...
Listed modules could also be annotated with "last updated on" or "supported mm² version" and more.

This would provide users with a nice entry-point into the ecosystem and aid both discoverability and exploration.

Let me know what you think. In any case I want to express my thanks for the ongoing efforts of anybody who is involved, as I've only recently discovered the whole topic and am excited to dive in :)

[guysoft]

Update, we made a first stable release:
https://github.com/guysoft/MagicMirrorOS/releases/tag/0.1.0

[MichMich]

🎉 Great work!

[rejas]

Cool, will check it out very soon hopefully. My pi3 crashed hard again (starts only with a kernel panic now) so I will see how settingh it up with your docker image works out.

[rossburton]

Focus (only) on Raspberry support.

Is this still a design goal?

If so, please don't. I totally agree with moving installers, docker images, updates and so on from the core. The core is basically a dashboard framework, and a dashboard framework can run anywhere. My prototype is running on a x86 NAS that I connect to with Safari in a fullscreen window on my Mac, and most likely will never actually run on a Pi at all. Make a Pi-specific OS or installer or whatever, but leave the core agnostic.

[guysoft]

@rossburton It should not be hard to make a docker container both for x86 and armf (if someone maintains it AFAIK).
The docker image MagicMirrorOS is running should also run on any ARM device. So not RaspberryPi spesific, but it is architecture specific. anything that uses Arm32v7, Orange Pi and such should also work and CustomPiOS supports building an image for them too.

[rossburton]

I don't care about the docker images. My concern is that at the top of this page it says that "future versions should ... focus (only) on Raspberry support". I don't believe the actual MagicMirror code should be specific to any particular hardware.

[guysoft]

Also another update - it seems that in under 10 days nearly 100 people installed MagicMirrorOS. That's a really impressive installation rate for a new distro. Here are the anonymous statistics gathered by the statistics module in CustomPiOS:

[TheTyrius]

Here are the anonymous statistics gathered by the statistics module in CustomPiOS

(disclaimer: please do not take as offence of any kind!)
Is this an opt-in feature? Or selectable during install? At a glance I see no indication of this module in the MagicMirrorOS repo.

[guysoft]

@TheTyrius You can disable it, the code is Free Open Source software. I am also as you can see publishing the information I gather. Its also GDPR complainant because I scramble all personal information including the Raspberry Pi Ids.
I keep this because its the only way I can prioritise which distro is actual being used out of the 8 I am maintaining. None of the 90 people using it told me that they are using it. They don't have to, but it means I am in the dark otherwise.
You are welcome to fork it and maintain it yourself. It is in fact in my interest that code I write in my free time as a volunteer would be understand by other people such as yourself.
This module is enabled in the config file and is documented here: https://github.com/guysoft/CustomPiOS/wiki/Modules#usage-statistics

[deg0nz]

Hi, just a quick question about the versioning for these changes discussed here:
Shouldn't version 2.11.0 actually be 3.0.0? Because, if I understood SemVer right, you have introduced a lot of changes that break things and have incompatibilities to older versions. So following to SemVer this should actually be 3.0.0 then...

Note: This is just a thought, please correct me if I'm wrong. But maybe this would also help people to get aware of the drastic changes that were made here.

Edit: Arrghh... nevermind, I just read this comment. Go on. Nothing to see here!

[sdetweil]

@deg0nz only thing I know that broke (unintentional) was the socket stuff.. everything works unchanged

modules need to adjust to new electron version, but not specifically a version problem

[MichMich]

I'm closing this issue for now. If anyone feels this should remain open, feel free to let me know. Trying to cleanup the issue list.