Rpi-jukebox-rfid: Discussion: Future of Phoniebox

[ZyanKLee]

To go deeper into the side-tracked topic about the code future on #763 I open this issue (as we are lacking an alternative communication method).

Yesterday I had a short look into the build, install and update scripts of LibreElec.tv - it left me barely any wiser after an hour reading through all the scripts, testing the build script and trying to figure how the heck the actual update of a LibreElec system works. (not the download but the flashing of the new version - what does it and where is that piece of code?)

Anyway: as far as I can see this project can be divided into some more or less separate sub-projects.

• Installer
• Webinterface (incl mpd-controller)
• Sound System (MPD / Mopidy)
• RFID-Controller
• Optional components
  
  
  
  • Streaming Services (Spotify, GMusic, etc)
  
  
  • Buttons / Rotary Encoders (USB/GPIO)
  
  
  • Soundcards (USB/GPIO)
  
  
  • Others
  
  

... did I miss something vital?

Questions:

• Is it ok to have all those parts in one repo or is a github organisation with multiple projects better?
• How can we organize the project without one of those parts constantly interfering with the others?
• What could a git workflow look like that allows us to maintain a stable version (currently: master branch) and development of new code / rewrites of existing code that can deviate in large parts from the stable version?
• Do we need another communication method beside GH Issues?
• With @MiczFlor as the sole maintainer of this project: how can we help him not to get burned up in the process? Is Micz trusting anyone enough to help him out with MR approvals and such?
• Is there a "masterplan" on what Micz / we envision for the phoniebox to look like in a year or so? (core-features, UI, addons, etc)
• Is the effort that would be needed for the definition of a formal plugin system worth the probably enhanced stability of the whole system?

[ZyanKLee]

I'd like to notify and invite at least the six most recent contributors (Micz has been mentioned in the post above) to this discussion:
@andreasbrett @s-martin @patrickweigelt @splitti @veloxidSchweiz @xn--nding-jua

[s-martin]

Questions:

• Is it ok to have all those parts in one repo or is a github organisation with multiple projects better?

I think one repo with sub directories for each component is fine. Phoniebox isn’t that big and I think different repos adds another level of complexity, which could be hard to integrate in a working product.

• How can we organize the project without one of those parts constantly interfering with the others?

Different sub directories with maybe directories for common stuff could work. Adding (automated) unit tests for each component could be added more easily as a separation in components forces clear interfaces.

• What could a git workflow look like that allows us to maintain a stable version (currently: master branch) and development of new code / rewrites of existing code that can deviate in large parts from the stable version?

I think the current workflow is fine in general. Alternatively we could use GitHub Flow, which is very lightweight. Master is always deployable and each feature/fix branches from master and after review/test is merged in master.

• Do we need another communication method beside GH Issues?

I think something like Gitter, Slack or similar could help for quick questions and discussions of ideas.

• With @MiczFlor as the sole maintainer of this project: how can we help him not to get burned up in the process? Is Micz trusting anyone enough to help him out with MR approvals and such?

I think, if we try to add more tests like @veloxidSchweiz has already started (and automate them with Github Actions), we become more confident that changes don’t break the product, so @MiczFlor doesn‘t need to review and test everything so thoroughly.

• Is there a "masterplan" on what Micz / we envision for the phoniebox to look like in a year or so? (core-features, UI, addons, etc)

Yes, I’m really interested what plans or ideas exist.

• Is the effort that would be needed for the definition of a formal plugin system worth the probably enhanced stability of the whole system?

Difficult question :)

[xn--nding-jua]

Hi,

because you asked me to give my opinion:

I think one repo with sub directories for each component is fine. Phoniebox isn’t that big and I think different repos adds another level of complexity, which could be hard to integrate in a working product.

I agree with @s-martin to keep the Phoniebox in one single Github-repo. In the past weeks I've created two pull-requests and was able to understand the functionality of the code within some evenings. If the code is splitted into several individual repositories, for people like me (with low/middle skills in php but some ideas) it could be harder to contribute to Phoniebox. Keep in mind that the project is focused on parents and childrens.

If a more complex version is nescesary, maybe the repo can be forked to a "PhonieBoxPro" with enhanced functionality? It may then be justified to create a distributed repository.

By the way: thank you for this wonderful project!

[MiczFlor]

Hi guys, this is the right thread, just at the right time. I took a little time to write up a few thoughts regarding Phoniebox. Possibly this should - eventually - become a small manifesto about the idea, progress, community and future of phoniebox?

I believe the few posts above already illustrate the nature of the Phoniebox community. Generally speaking: I am very much open to the idea of sharing responsibility, giving access to merge and plan together. From the past (years!) of experience, I have seen great minds and coders come and go. So all in one place is a good idea, to make sure the project actually has a future. (I am also happy to have a Phoniebox meetup of developers in Berlin, if there are enough around. Any interest?)

https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Architecture_HowWeWork

Phoniebox consists of code, a lively community and inspiring designs

A Phoniebox...

1. ... is easy to install.
2. ... out of the box plays local audio files and can be controlled with RFID cards and through a web app.
3. ... is not complete until fitted into a unique, jaw-dropping and eye-opening box. Everyone builds an original.

How the Phoniebox community works

1. Phoniebox should inspire people to tinker with technology.
2. Contributors are welcome!
   
   
   
   • more than code: feature requests, filing bugs, engaging in the discussion are all contributions.
   
   
3. The Phoniebox code lives in one place.
   
   
   
   • making the future of Phoniebox independent from individual developers.
   
   
   • contributors come and go. And that's ok.
   
   
4. The discussion around the future of Phoniebox is open to all.
5. Sharing responsibility.
   
   
   
   • while the project is comparably small, not one person understands all the code there is.
   
   
   • peer reviews and merging pull requests are responsibilities to be shared with the community.
   
   
   • peer support keeps the project alive (and is working very well, thanks to the community).
   
   
6. Contributing means great responsibility: major rewrites and refactoring are only taken on if the one or more developers plan to see it through to the bitter end...

The Phoniebox code base

1. The Phoniebox code base is free and open (currently available on github).
   
   
   
   • Open meaning:
   
   
   • the source code can be accessed and read by anyone who choses to do so.
   
   
   • the code base is open to contributions by other developers to improve stability and add features.
   
   
2. the code base is open to interface with new hard- and software and services (soundcards, RFID readers, displays, smart homes, GPIO buttons, barcode scanners, Spotify, Google Music, ...).
3. Where possible, configuration beats hard coded solutions.
4. Configuration should not be more complex than a hard coded alternative script would be.
5. Phoniebox thrives towards a unique API (currently the shell scripts pretend to be that).
6. Phoniebox incorporates many languages (bash, php, python, js) if need be.
7. Phoniebox depends on APIs which are out of our control (e.g. Spotify integration).
8. The architecture, structure and coding convention of the code are open to improvements (naming conventions, folder structures, etc.).

[MiczFlor]

From my more abstract statement above, I deduce the following action points:

1. stability: the develop branch should become stable first, before we embark on reqrites and configuration
2. merging pull requests. Who would want to get access to that privilege - and if only temporary
3. configurable components: yes! In practical terms, I suggest that at least two or more people should commit themselves to drawing up an idea, suggesting it to the community and coding it :)
4. In order to keep Phoniebox in one place, improving documentation, structure, architecture would be good

... what did I miss?

[s-martin]

I'd like to add:
Testing and test environment.

Adding first unit tests was already done in #763 (can't wait until that is merged). This could be improved step by step, maybe even for integration tests with hardware (mockups).
In #738 docker files were added, so this will be a big help.

If we can write a wiki article how to set up a test environment on real hw or docker, this could help many people and improve overall test coverage.

[Flowdeeps]

I think one repo with sub directories for each component is fine. Phoniebox isn’t _that_ big and I think different repos adds another level of complexity, which could be hard to integrate in a working product.

I agree with this. If you look at how Google organises something like the NSynth project which has four different very definitely interconnected but also very different components (hardware schematics, audio pipeline, interface software, firmware, and PCB…) it's all separate folders each with a README but a root level README giving the overall information.

https://github.com/googlecreativelab/open-nsynth-super/

[Flowdeeps]

6. Contributing means great responsibility: major rewrites and refactoring are only taken on if the one or more developers plan to see it through to the bitter end...

This is why you force contributors to make forks with feature branches and create pull requests based on their suggested changes. I've seen some quite correct suggestions about master being always deployable and only merging after manual and automated pipeline acceptance testing. 👍

[MiczFlor]

@s-martin great energy, a wiki article would be great. And how would you write Testing and test environment. into the above list. Here my suggestion:

https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Architecture_HowWeWork

How the Phoniebox community works

...

1. Contributors are welcome!
   
   
   
   • more than code: feature requests, filing bugs, engaging in the discussion and testing are all contributions.
   
   
   • testing is a selfless and noble form of contribution.
   
   

...

The Phoniebox code base

1. running existing tests is a must do, writing new tests is for kudos.

[Flowdeeps]

@MiczFlor I would love to join a Phoniebox group. Have you considered starting a meetup? It might be a good fit for the Embedded Linux or Linux Audio groups that already exist here in Berlin and have an active userbase. I could potentially do an introduction to some local hardware folks if you're interested. Alternatively there are the always excellent C-Base and XHain hackspaces who might be interested in hosting an event in exchange for exposure and beer money through the door.

[splitti]

First of all: sorry for my late answer, my time in the early year is really expensive, but i look forward to invest the time i can give to this Project.

Questions:

• Is it ok to have all those parts in one repo or is a github organisation with multiple projects better?

The basic Installation should be one repo in my opinion. I would like to see more repos, that can be attached like modules, maybe also as an installer. I also think, that config files, which will be edited manually, and also Shortcuts, Music, Playlists and so on should be seperated from the basic github repo in the installation. That makes it much easier to get an update working... But this is only my point of view.

• How can we organize the project without one of those parts constantly interfering with the others?

First of all, we should think of a stable and actual repo. @MiczFlor this is in german, because i don't find the words in english: Das Projekt hier ist der Hammer und ich bin Dir dafür mega dankbar. Aber es ist in dieses Repo so viel rein geflossen, dass es sehr schwer zu überschauen ist. Nimm das nicht als Kritik, sondern eher als Verbesserungsvorschlag, dass man Modulbasiert verschiedene Repos anbietet.

• What could a git workflow look like that allows us to maintain a stable version (currently: master branch) and development of new code / rewrites of existing code that can deviate in large parts from the stable version?

The main problem is, that there must be a main developer and tester for each step. One problem is also, there are maybe only two contributers, but both checking in to develop. Maybe it is better to make a repo like develop_username_timestamp? I am not sure, this is not easy for @MiczFlor to handle.

• Do we need another communication method beside GH Issues?

If there were different developers that organize the developent: yes. At this stage, i think no, because contributers are coming and going. There a just a handful of developers here about months.

• With @MiczFlor as the sole maintainer of this project: how can we help him not to get burned up in the process? Is Micz trusting anyone enough to help him out with MR approvals and such?

Ask @MiczFlor

• Is there a "masterplan" on what Micz / we envision for the phoniebox to look like in a year or so? (core-features, UI, addons, etc)

This would be a nice plan! But you need main contributers always working for this project, not just for a couple of weeks.

• Is the effort that would be needed for the definition of a formal plugin system worth the probably enhanced stability of the whole system?

Do you have an example.

So, i orderd three more Pis, i will build three more boxes to test. I hope, the project goes on and i try helping as much as i can.

[veloxidSchweiz]

Hi,
Thanks @ZyanKLee for starting this discussion. I think this is a goo opportunity to go forward.
@MiczFlor: I like your 'HowWeWork', this would have helped me when I started/thought about working on that project.

Here some personal opinions from my experiences. I am not a full-stack developer (whatever that means), but I already worked in some software projects from different perspectives.
The company I am workin in, is building electronic devices which requires some quite advanced software.
What I learned there is that, first of all testing is one of the key components to have a code which is always deployable. This includes good unit tests but also a bunch of system tests.
We experienced having one mono-repo seems to be the best way to guarantee the testing the code. This does not mean that the code should be separated in different modules, The modules could, for example, be maintained in different folders. Setting up multiple build-chains/different test chains for each module seems to be an overkill for me and does not really simplify the workflow.
One important step I see is that one should think about the interfaces. I.e. how is the interface between the php code, the gpio control ,....
For the testing the deployment I like the idea of using docker and/or ansible.
Some more comments on testing:

• I think testing is one of the best possibilities to have a certain 'trust' in the changes you do. It helps to understand breaking changes, as this will typically be visible by a change in the test.
• As we are using hardware it is important that one has system tests. I personally try always to work without actual hardware and try to mock the behaviour, then when i have implemented the functionality i will validate it with actual hardware and adjust the unittests to actually represent the behaviour as good as possbile.

I am not an expert on running docker on RPis, but if there is possiblity to containerize the functionalities this would probably allow a very easy deployment.... One would run separate containers for each subject of concern and it would be quite easy to update single compontents... here i would suggest to have one for the html interace, one for the mpd (or one of his spotify sisters), one for the control of the RFID reader, one for the gpio handling (maybe one separate for the oled display).... By only 'mounting' the music- and setting-directory, the system is very likley to be good upgradable.
Even though docker is quite lightweight and does not cost many resources, I am not sure what would be the effect on systems based on RPiZero.... Such an idea is in my opinion a very nice option, especially as most communcation happens by mpc, bbut it would need some tests for evaluation.

This is only my personal view and I am open for such a discussion. Having such an open-source-project in which many people invest a marginal time of their free time is really great and I love this projects, If I can show my acknowledgement by contributing this is even better ;-)

[MiczFlor]

Thanks for the input, this is really cool.

Regarding the code repo: I agree with ...

• @s-martin ("I think one repo with sub directories for each component is fine.")
• @Flowdeeps ("it's all separate folders each with a README but a root level README giving the overall information.") and
• @veloxidSchweiz ("The modules could, for example, be maintained in different folders. Setting up multiple build-chains/different test chains for each module seems to be an overkill for me and does not really simplify the workflow.")

I know what it's been like to answer questions regarding other peoples forked repos who once posted a link to their file and then changed it a few weeks later to fit their specific, it didn't work for others and suddenly I had to take the heat and try to awake the contributor from the dead (who did great work, and meant to do good, and did). Then suddenly everybody was annoyed for no reason. I prefer people creating pull requests and merge to develop - then they want to be proud of their code, comment their code and I know it is tested (enough) even if I don't have the hardware (ok, sure @veloxidSchweiz this is not the type of testing you are talking about :)

... and would like to restructure the folders and rename the files to make it easier for new people to dive into the deep end. In terms of components or functionality, and following @ZyanKLee list in the first post, I think these are the "folders" we need - not sure how to name them... (what am I missing?)

• Installation
• Local audio files and folders (currently under shared, together with the shortcuts for RFID tags)
• Web App (currently htdocs Webinterface incl mpd-controller)
• Sound System configuration and howto (MPD / Mopidy)
• RFID-reader hardware
• GPIO buttons and rotary (USB/GPIO : WARNING that GPIO buttons might be taken by soundcards :)
• Soundcards (USB/GPIO : WARNING that GPIO buttons might be taken by buttons :)
• Displays (WARNING that GPIO buttons might be taken by buttons or soundcards :)
• External services (Streaming Services, Spotify, GMusic, etc)
• Bash scripts triggered by web app (php), RFID chip (python) or services (This currently(!), as I said before, functions like an API. everything eventually calls a bash script)
• Services (on startup, daemons .service files)

Any suggestions for naming and folder structure? I started this document in the wiki, please put suggestions there (the wiki might be the more collaborative editing space)
https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Architecture_FoldersFiles

[ZyanKLee]

First of all: I'm glad this discussion was so well received and on time. :)
Thanks to all of you for all those great comments and ideas.

After @MiczFlor's latest post I thought about how to go on from here with his plan while avoiding any interruption to the installation of new Phonieboxes and came up with this idea:

First of all, I think that _new_ github feature "projects" might come in very handy to organize the planned steps and workpackages without us cluttering up the main issues too much.
If @MiczFlor would create a project by the name of "reorga" (or something else along those lines), we can cut this massive undertaking in small manageable chunks and keep track of them in the project.

To also separate the code from the "legacy" system, I would suggest a new long-living branch with a similar name; i.e. "reorga".
Then master and develop could go on as they do with the maintenance of the current code. In the meantime reorga would be used as base-branch for everything related to the reorganization; with its own feature branches for each reorga ticket ( - ideally :laughing: ).
After we are satisfied with its status, it will be merged to master (replacing everything in there) and develop will be refreshed from the new base as well.
New functionality would ideally only be added to the reorga - but I'm not sure if this is feasible as the reorga might run some time.

This is an attempt to illustrate the git flow:

The reorga branch should be set as protected - similar to develop and master to avoid commits being pushed directly.

[ddbkva]

maybe you can change the phoniebox software so that it can be integrated into the Logitech Media Center.
Squeezelite runs very well on a Raspberry pi.

[MiczFlor]

(I pinned this thread in github, so it's at the top now, in case you miss it)

Re @ZyanKLee suggestion for structure and workflow, I was being pushy and started the components folder... you can see it in the develop branch already.
Inside the file CONTRIBUTING.md I added a few lines, which read like this (please comment):

---

Naming conventions

• Files & folder names

  • all lower case
  • separate words with dashes - (less keystrokes, better autocomplete recognition, in HTML links dashes can not be confused) not camel/PascalCaps or underscores
  • be descriptive in your wording (e.g. raspberry, not juicy-red-thing)
  • move from general to specific (e.g. food-fruit-raspberry, not raspberry-food-fruit)
  • unique and clear product IDs (e.g. MAX7219)
    
    
    
    • the product ID should be written as is (no lowercase)
    
    
    • the product ID should come last in a descriptive name (e.g. dot-matrix-module-MAX7219)
    
    
  • be consistent and look at existing examples before you invent something new

• README.md

  • written in capital letters, so it's easier to spot
  • every new folder of a component deserves a README.md file

Structure of files and folders

Inside the root folder or the repo, these folders are important:

• scripts
  
  
  
  • this folder should contain only actively used scripts (controlling playout, rfid tiggers, etc.)
  
  
  • some possible services and features might live in the components directory (see below)
  
  
  • if one or more scripts are needed for the activation of a component (like daemons), they should be copied to the scripts directory during installation / activation
  
  
  • WHY? By copying, changes will NOT affect the github repo and make it easier for users to modify their components
  
  
• components
  
  
  
  • contains sub- und subsubfolders for additional features, services, hardware
  
  
  • subfolders are for categories (e.g. displays, soundcards) and are plural, even if there is only one
  
  
  • subsubfolders are specific hardware, services, features, protocols, etc.
  
  

[ZyanKLee]

Hey @ddbkva
Please create a new issue for this feature request and explain there a little more about how you envision this integration.

[ZyanKLee]

• Local audio files and folders (currently under shared, together with the shortcuts for RFID tags)

this should not be in the project folder in my opinion. We should rather fall back to common directories from the desktop world /home/pi/Music, /home/pi/Playlists, etc to avoid deleting them on updates or clean re-installs.

[s-martin]

this should not be in the project folder in my opinion. We should rather fall back to common directories from the desktop world /home/pi/Music, /home/pi/Playlists, etc to avoid deleting them on updates or clean re-installs.

Added reference #797

[ZyanKLee]

should we move

• shared/shortcuts/ to ~/.phoniebox/shortcuts/ or ~/.phoniebox/rfid-cards/
• settings/global.conf to ~/.phoniebox/global.conf

?

[s-martin]

I think that would make sense. We have to check, if there are other config files, too. @veloxidSchweiz added some config files in #763, too.

[MiczFlor]

• Local audio files and folders (currently under shared, together with the shortcuts for RFID tags)

this should not be in the project folder in my opinion. We should rather fall back to common directories from the desktop world /home/pi/Music, /home/pi/Playlists, etc to avoid deleting them on updates or clean re-installs.

I like the idea of separating the media from the code. And I doubt that people would use their Pi for Phoniebox AND some other audio solution that would require the system style folders /home/pi/music. Is there any advantage in creating media folders like /home/pi/PhonieMedia/or the like?
Using the ~/.phoniebox/shortcuts/ is possible. Currently, samba still allows access to the audio folder so that you can push files via the file browser. I use this sometimes also for quick backups. Should we point sambe to such a personal, hidden folder instead?

[veloxidSchweiz]

I like the idea of using the default music folder. I actually started to just symlink the Music folder into the phonbox shared folder. This is for me already so much cleaner and seems to work ( I had quite some problems at the beginning with changing the folder..)

if I do not see an advantage of creating a separate audio folder and to put the music in the standard place makes sense for me.. A valid question is where to put playlists and shortcuts... Any other opinnion on that?

@MiczFlor , I started to use ~/.config/phoniebox maybe we should use only one location, for configs or similiar things... We can also switch to use ~/.phoniebox instead...

@s-martin , @ZyanKLee: How do you like the idea of having separate docker containers for each concern? (GPIO, RFID, html, mopidy,....)