Dietpi: META | Developer Entry Point?

[jakimfett]

Currently, there doesn't seem to be a place where the development pipeline (edit, build, test, tweak, repeat) is documented, or really even defined.

There's small bits scattered around, but there doesn't appear to be a "getting started" doc anywhere, although automation is mentioned.

I've got a couple of raspberry pi units (an original-ish, a Zero W, and a second gen B+ I think?) that I'd like to use on my local network, but there's enough stuff I want to tweak that jumping into dev seems like the easiest answer given my scripting tendencies...

Is there a build script or doc I missed seeing somewhere?

[Fourdee]

@jakimfett

Hi,

Image build script is located: https://github.com/Fourdee/DietPi/issues/1285#issue-280771944
https://github.com/Fourdee/DietPi/blob/master/PREP_SYSTEM_FOR_DIETPI.sh

Currently, there doesn't seem to be a place where the development pipeline (edit, build, test, tweak, repeat) is documented, or really even defined.

In short:

• Send all PR's to dev branch
• If your PR contains code changes (eg: install change for Nextcloud), it should be tested on SBC/VM on Debian Stretch before requesting review.
• PR's also need to be considered for any patching changes that need to made for existing systems (eg: if updated NextCloud install requires a reinstall or config update).
• Once PR merged, if additional testing is required we'll make a note on the PR and complete additional testing.

Basically, go for it, make any changes, send a PR. We'll explain in the PR as needed.
If unsure/missing info, please let me know.

[msdos]

Should this last bit of information added by @Fourdee be added to https://github.com/Fourdee/DietPi/blob/dev/BRANCH_SYSTEM.md#steps-to-use-the-dietpi-dev-branch?

[MichaIng]

It fits better to https://github.com/Fourdee/DietPi/blob/dev/CONTRIBUTING.md from my point of view, but would overload it perhaps. My suggestion is to start using the Wiki and define/add such information with more details there. Then add a link to the above md to the Git coders section.

[jakimfett]

It seems to me like having CONTRIBUTING.md contain a sentence or two on the dev process, and then links to next steps, would make sense. Definitely want to keep the file concise, tho.

Thoughts on using an in-repo knowledge management (eg, something markdown based, in a docs folder) rather than locking into Github's wiki system?

Very much looking forward to playing with this.

[msdos]

@jakimfett IIRC, some projects use https://readthedocs.org because you can host your documentation in a docs folder and, when pushing to it, automatically update it using web hooks.

requests, a highly used library in the Python world uses this approach.

https://readthedocs.org/projects/requests/
https://github.com/requests/requests/tree/master/docs

[MichaIng]

Ideas I have is to use the GitHub wiki mainly for developer and super user information:

• Add an image creation procedure for every device (added partly when there is time). A start can be taken from: https://github.com/Fourdee/DietPi/issues/1505
• Add deep core system information about which Linux/GNU/Debian features we use/require, how we in case implement/change them and which additional core features we add (bashrc.d, cron.minutely, postboot.d and such).
• Add details about DietPi boot scripts and init/service handing.
• Add info about our coding style. Wrap up info/results from: https://github.com/Fourdee/DietPi/issues/1510
• Add info about our PR/dev/testing/beta procedure, which covers the OT.

Much work, however I hope I can do a start with this soon.

Usual user information on the other hand can be found on the forum: https://dietpi.com/phpbb/

[FredericGuilbault]

Things like CONTRIBUTING.md and getting started kinda can't be writen by core devs.

You need noobs to know what noobs need to know. Core devs have a lots of things taken for granted without realising it.

[MichaIng]

Jep I 100% agree. However we reply on pull requests or questions/hints. Beside we do our best to explain things as detailed as possible.

An open/ongoing task is still the wiki which I want to turn into a power user/contributor knowledge base, including the main topics of Debian and Linux derivatives on general to allow understanding how things work and how/why we handle on DietPi.

[jakimfett]

@msdos and @MichaIng the point I was (attempting) to make is that using an external doc source (eg, Github Wiki, forums, readthedocs, etc) is imo just adding an unnecessary external dependency.

Rolling the docs into the repo, and setting up a downstream presentation of docs (eg, hooks to update readthedocs) are two discrete steps, and I'm...hoping that I can nudge things towards avoiding the second bit, especially since I'm attempting to build my SBC infra with minimal human intervention.

My current workflow looks something like:

• Set up Laminar CI and get prerequisites building properly
• Burn an official DietPi image to target system media or VM
• Harden target system (fail2ban, iptables, 2fa/sso/keyshare)
• Bring SaltStack online
• Build customized DietPi image from locally tweaked secondary/virtualized system
• Write to an SD card
• Differentiate resulting system(s) as necessary for my various use cases, eg gitea or murmur.

Am attempting to automate as much of this as possible, because I dislike solving the same problem multiple times, and dogfooding does wonders for revealing the kinks in your development/deploy pipeline.

===

Things like CONTRIBUTING.md and getting started kinda can't be writen by core devs.

@FredericGuilbault disagree.
IMO core devs are the only ones who have visibility on how users on the bleeding edge of the use spectrum can acceptably contribute.

Not everyone is able to write clear docs, but I've participated in enough collaborative projects to know that unless the process of generating those docs is closely monitored by the people who are going to have to put up with whatever PR bs shows up in their queue, it's better when core devs are involved in the process.

Without core dev's feedback in places like this, I'd just be swinging blind, and my main focus, in submitting this issue and interactions here, is to get the isolated knowledge (eg https://github.com/MichaIng/DietPi/issues/2263#issuecomment-441243668 and https://github.com/MichaIng/DietPi/issues/2263#issuecomment-440076742) rolled into something concise and accessible for so-called "power users" and sysadmins like myself who want to start using this amazing project in their home labs (or staging, if they're really brave).

[MichaIng]

@jakimfett
Thanks for your input. To reduce manual user input, we have currently two possibilities to run custom scripts, actually three: https://github.com/MichaIng/DietPi/blob/dev/dietpi.txt#L114-L124

• /boot/Automation_Custom_PreScript.sh, if present, runs before any other firstrun setup steps are done, especially before network is up. So special hardware, core system and network setup needs can be done by placing a script at this location.
• /boot/Automation_Custom_Script.sh respectively the chosen URL target script runs after dietpi-software firstrun installs/setup have been done (APT access/time sync/network assured), which can as well be automated: https://github.com/MichaIng/DietPi/blob/dev/dietpi.txt#L53
• Placing scripts in /var/lib/dietpi/postboot.d/ (ext4 partition access required) run as very last step before login, but persistently, not just on first boot.

The only thing that needs to be done manually always is the initial login. There is an idea/request to allow auto-login as well on first boot, or run even dietpi-software installs without prior login: https://github.com/MichaIng/DietPi/issues/2520

• But needs testing, probably a larger project than it seems on first view.

About image creation:

• I am currently writing and testing a script to automate image creation: https://github.com/MichaIng/DietPi/pull/2693/files
• It shrinks file system and partition, so it needs to be assured that dietpi-fs_partition_resize.service is enabled to resize on target system.

---

About docs:

• Not sure what you mean by "unnecessary external dependency".
• Of course we cannot or do not want to place all kind of info into the DietPi code on every system, but only those which are relevant for end users to execute/use the related scripts.
• Every details about first run setup etc. totally make sense to have externally (on GitHub/forum) since you want/need to read them before the system is running.
• Also all development docs make sense online since you usually develop things outside a particular target system.
• So only minimal required info within the scripts on the system, larger power user/dev/contributor/firstrun setup info online totally make sense, doesn't it? Also this is common practice across nearly all FOSS software.
• The only thing I am currently not 100% happy with is the separation between forum and GitHub. Would be nice to have everything in one place. But this is due to the beginning of DietPi and how forum and GitHub usage has developed itself. Currently end-user orientated info is available on the forum while the aim is to add more power user/dev/contributor relevant info on GitHub, which somehow makes sense since GitHub is still a developer platform by it's core.

---

About who should write docs: You are both right!

• Of course only the developers know the product until it's core, so all docs at least need to be reviewed by them, developer/contributor relevant info even can only written by them.
• But when it comes to having things understandable for others, of course devs might have a too deep view/knowledge to remember writing even basic background info and all single steps down. But this highly depends on the actual guy who is reading it. If something is not clear/detailed enough a feedback/contribution loop can solve it. But this is usual process of every documentation, regardless of who was writing it.

---

Finally yeah indeed there is still much documentation missing, if one needs more then just installing DietPi on ones single SBC at home and install a cloud, media server/player, web application or such. The initial aim was to make this things "just" run without much hassle. But for things beyond, there is deeper Linux/GNU/Debian knowledge required and manual work, as in every other server case.

• Best approach is always to write every manual steps down into shell/bash scripts, after verified. And I hope the above mentioned automation systems help to apply them easier on multiple systems.

[MichaIng]

I mark this as closed as we have a dedicated documentation page and repository now: https://github.com/MichaIng/DietPi-Docs
We're constantly adding more content there, including things that are currently in the main repo wiki. Suggestions and contributions are highly appreciated of course, there is still much to do and much more that can be done optionally 🙂.

[jakimfett]

@MichaIng thank you so much for the attention and progress on this.