Semanticmediawiki: 2.4.0 release

[JeroenDeDauw]

Tracking issue for the 2.3 release.

• Release notes
• Open issues for 2.4
• 2.4 issues already closed
• Changes since 2.3

See also: 3.0 release

RC1 target: June 20th

• [x] Finalize rel notes
• [x] Update version number (2.4.0-rc.1) (see https://github.com/dbrock/semver-howto)
  
  
  
  • [x] SemanticMediaWiki.php
  
  
  • [x] COMPATIBILITY.md
  
  
• [x] Create tag
• [x] Create tarballs
• [x] Run composer update for SMW sandbox
• [x] Announce via mail
• [x] Do a release tweet on Twitter

Release target: July 9th

• [x] Finalize rel notes (include contributors (git shortlog -sn 2.3.0...2.4.0))
• [x] Update version number (2.4.0)
  
  
  
  • [x] SemanticMediaWiki.php
  
  
  • [x] README.md
  
  
  • [x] COMPATIBILITY.md
  
  
• [x] Check Special:Version
• [x] Create branch
• [x] Create tag
• [x] Create tarballs
• [x] Update release template
• [x] Run composer update for SMW wiki
• [ ] Announce on wiki --> via https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1716
• [x] Update version on wikidata, en, de, mw and wikiapiary
• [x] Touch LocalSettings.php
• [ ] Announce via mail --> via https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1716
• [ ] Do a release tweet on Twitter --> via https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1716

[mwjames]

Preliminary notes

• 2.4 requires to runupdate.php because of changes in #1335, #1369

[JeroenDeDauw]

@mwjames @kghbln how about doing this in the next few weeks? RC target in 2 weeks, release target in 4? Then we can finally bump to 3.x alpha and get rid of the compat cruft.

[mwjames]

how about doing this in the next few weeks? RC target in 2 weeks, release target in 4?

Today, I squashed several bugs and features but there are some minor tasks on my list to be done before 2.4 gets released.

[mwjames]

... and we passed the 5000+ Travis job

[kghbln]

how about doing this in the next few weeks? RC target in 2 weeks, release target in 4?

There are things on which I collaborate sort of with James I will have to look at too soon.

Currently we are under heavy feature fire, which is good! Will be anther great release. :)

[mwjames]

It will be some time before I'm happy with 2.4 therefore no release should be expected for either Feb or Mar.

[JeroenDeDauw]

Looks like there is a big pile of improvements for this release already, and it's been half a year since the last feature release.

[mwjames]

Looks like there is a big pile of improvements for this release already,

I'm not really finished (I want the multilingual property label stuff done somehow this time around and not have it rot again a year or so) but currently a bit hung up on the FullTextSearch support side project for Blob values.

and it's been half a year since the last feature release.

I mean, if you think a release is somehow necessary then go ahead but I wont be working on 3.0 any time soon (I'm just not really looking forward spending time on code clean-up) and I will most likely continue with 2.5.

I'm a bit worried about the smw.org documentation status though which is getting worse by the release.

[JeroenDeDauw]

We can always do a 2.5 release yes. Doing a release soon does not prevent anything else from being released soon after, as we can just do another release. While there is no critical need to do a release, the longer we wait, the more unused potential we have sitting on master, not benefiting the users.

[JeroenDeDauw]

The features section of the release notes looks like it needs a bunch of love before we can do a release.

[mwjames]

I want the multilingual property label stuff done somehow this time around and not have it rot again a year or so) but currently a bit hung up on the FullTextSearch support

Maybe its time to start kicking the 2.4 tires. Fulltext (@kghbln which would require MariaDB 10.0.5+ on the sandbox) and multilingual property labels will most likely come with 2.5.

• I would recommend to move to MW 1.23 as min. requirements (== LTS) for SMW 2.4 which leaves room for people to use PHP 5.3 + LTS release (and we can remove some legacy stuff in SMW 2.5, especially all the 1.19 skipped tests).
• SMW 2.4/2.5 to use PHPUnit ~4.7/4.8 (I tried 5.3 which was a nightmare slow + several breaks)
• SMW 3.0 should mark the entry point for 1.27 (LTS) + 5.5 to avoid any controversy with the user base

features section of the release notes looks like it needs a bunch of love

I was too lazy to plant some flowers.

[JeroenDeDauw]

I would recommend to move to MW 1.23 as min. requirements (== LTS) for SMW 2.4 which leaves room for people to use PHP 5.3 + LTS release (and we can remove some legacy stuff in SMW 2.5, especially all the 1.19 skipped tests).

As master still works with MW 1.19, it makes more sense to me to do a last release supporting 1.19 now and bumping the requirements right after. That still allows us to remove legacy stuff in 1.25. How does that sound?

SMW 3.0 should mark the entry point for 1.27 (LTS) + 5.5 to avoid any controversy with the user base

This means supporting PHP 5.3 till SMW 2.5 has been released (or rather, till the last pre-SMW 3.0 release is made). As you seem to want to cram a lot of stuff into 3.0, I'm thinking this might well be another year of supporting 5.3. By which time PHP 5.5 will be well unsupported. Being more conservative with bumping PHP requirements than WMF is kinda... I'd want to drop support for these ancient versions ASAP, not in the 2.4 release, though in the next one. Then again, I'm not actively working on SMW, so not my call. I've already made the bump to 5.5 with Maps now, and imagine most actively worked on MW extensions have done the same. The Wikibase stuff did. (And my current project is PHP 7)

[mwjames]

As master still works with MW 1.19, it makes more sense to me to do a last release supporting 1.19 now and bumping the requirements right after.

+1

By which time PHP 5.5 will be well unsupported. Being more conservative with bumping PHP requirements than WMF is kinda...

Let's play this by the ear. It is just to make an early announcement that 2.4 is the last official supported MW 1.19 with the next most likely to require MW 1.23 (as LTS) while SMW 3.0 is aiming towards PHP 5.5 as min. requirement.

I've already made the bump to 5.5 with Maps now

I think this is fine but I don't want to have a discussion with people why we choose a different PHP requirement compared to MW for a software like SMW.

[JeroenDeDauw]

We don't need to have such a discussion (except that we are having one right now ;p), as we can just change it and be done with it. We did this before, with switching to 5.3 min, which IIRC we did before MW.

[mwjames]

1435 and CachedPropertyValuesPrefetcher should account for some subjective performance improvement on a page view and with the help of [0] a user can turn a standard table into a dynamic result output.

[0] http://sandbox.semantic-mediawiki.org/wiki/Whats_Nearby/Dynamic_table_%28nolocation%29

[JeroenDeDauw]

I was too lazy to plant some flowers.

I'm not sure what you mean by that. Thing is, we can't really do a release without the release notes being in better shape, and while I can fix some grammar and clarify some things, I can't make the current stuff fluffy enough as I do not really know what was done.

[kghbln]

Maybe its time to start kicking the 2.4 tires. Fulltext (@kghbln which would require MariaDB 10.0.5+ on the sandbox)

@mwjames Sandbox is now at PHP 5.6.19 (from 5.4.45) and MariaDB 10.0.24 (from 5.5.48)

I also need to get the wiki docu done...

[mwjames]

2.4 + 1.26.2 + PHP7 are part of the test run now.

[mwjames]

2.4 (preliminary): Tests: 3534, Assertions: 11461, Coverage ~83%

Now, that we crossed the 80% coverage we should strive for not dropping below that line.

The 80% may not indicate that our confidence level is higher than in comparison to 2.3 or 2.2, instead we can only confirm that at least 20% of the code is not tested and therefore requires human intervention to filter, test, and stress the software.

While some may argue that % coverage is meaningless [0] it is still a confidence indicator that helps to determine whether a piece of software has sufficient production quality or not in terms of expected vs actual results [1].

Some developers may find writing tests a daunting task or caring for coverage irrelevant to the software process but I do believe that writing explicit tests expresses the willingness and commitment of a project to the product and for those who use it.

Most of our integration tests are written in JSON and not involve specific implementation details which is expected to help fight against regressions and provide consistency on expected results from a higher order level. Yet, some old unit tests may cause "... excessively long changes to tests ..." [2] due to legacy reasons but I don't think that we fall into the [3] category for the sake of coverage.

[0] http://blog.ploeh.dk/2015/11/16/code-coverage-is-a-useless-target-measure/
[1] [expected vs actual](https://camo.githubusercontent.com/13b82ae40b32b9c560c0b213f4fe358bd8d01593/68747470733a2f2f7261772e6769746875622e636f6d2f6477796c2f6c6561726e2d7472617669732f6d61737465722f696d616765732f526f6c6c2d426c6f636b732d546f696c65742d536561742e6a7067)
[2] http://martinfowler.com/bliki/TestCoverage.html
[3] http://martinfowler.com/bliki/AssertionFreeTesting.html

[JeroenDeDauw]

That’s really cool, thanks so much for caring.

[gesinn-it]

@mwjames , @JeroenDeDauw what do you think, how far are we away from a 2.4 release?

[JeroenDeDauw]

I have time this weekend to poke at things. My main concern is that the release notes and docs might not be sufficient to communicate all the new awesome.

[mwjames]

Given that:

• you guys @gesinn-it @Fannon have worked with it for sometime
• @kghbln didn't complain :-) about a broken system caused by SMW-core itself, and
• now that 1.27-rc1 was brought to light

I'd say it's time to put it in a box, wrap a label around it, and we all move on.

@gesinn-it @Fannon If you help skimming the RELEASE NOTES then I think @JeroenDeDauw and @kghbln would be delighted to move this release forward.

That’s really cool, thanks so much for caring.

Tests: 3673, Assertions: 11984, Skipped: 2. equals ~86% coverage

[kghbln]

now that 1.27-rc1 was brought to light

It's 1.27.0-rc.0 though I was not able to update sandbox as desired on the past weekend due to mucha other problemos. Hope to do it this weekend.

[JeroenDeDauw]

@gesinn-it three weeks or so? RC-1 in about a week, and then release two weeks after that.

[kghbln]

Sounds good to me.

[mwjames]

as per https://twitter.com/WikiAhoi/status/740481812737236992:

Would be so nice to have a "human-readable" version of the list in simpler language for non-natives/users!?!

Users are invited to make it more "human-readable" (whatever this entails) but this project is about contributions hence we should not expect a deux ex-machina that fixes it.

[kghbln]

Would be so nice to have a "human-readable" version of the list in simpler language for non-natives/users!?!

Hey persons, do not make me ready. :|

[mwjames]

I hope no one is waiting on me to say go go go.

Trivia

• 855 commits to master since the SMW 2.3.0 (released Oct 25, 2015)
• Tests: 3685, Assertions: 12014, Skipped: 2

• HHVM failure is caused by the microseconds bug

[JeroenDeDauw]

I did not have time to do stuff last weekend or earlier this week. Definitely have time in the coming days tho, so will get this moving.

[JeroenDeDauw]

Users are invited to make it more "human-readable" (whatever this entails)

Describing the new capability in a non-technical way, from the users perspective, together with some minimal hints of the syntax, and a link to the docs if they exist. And for the highlights this would be a bit more verbose, perhaps a "paragraph" each, kinda like we used to do way back https://www.semantic-mediawiki.org/wiki/Semantic_MediaWiki_1.6.0#Changes

Example https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/1656

this project is about contributions hence we should not expect a deux ex-machina that fixes it.

The point @Zabien (WikiAhoi) made is that users cannot make things that are not understandable by users understandable for users, as they don't understand them by definition. Once there is basic understanding, people can improve wording and grammar, can add examples, and perhaps extend the description based on their experiences.

I get that you are not enthusiastic about such documentation work, though I fear that without it most users will never know about or understand most of the things you are adding/improving.

[mwjames]

The point @Zabien (WikiAhoi) made is that users cannot make things that are not understandable by users understandable for users, as they don't understand them by definition.

I do understand the sentiment and I would think that we have improved considerable in comparison to other releases since most issues are accompanied by a sandbox reference to illuminate the effect or if not the opposite of the intended change.

Users interested in improving the documentation (and yes, RELEASE NOTES are documentation as well) are invited (and yet, I fail to see real participation) to browse a particular item that seems unclear and if definitions or functionality are irrevocable arbitrary, to raise the voice and seek clarification. But a general call for making things more "human-readable" does not necessarily force my participation.

I get that you are not enthusiastic about such documentation work, though I fear that without it most users will never know about or understand most of the things you are adding/improving

Bare bone docs are in place and awaiting to see an affectional support either on the sandbox or smw.org but I think it is a wrong assumption to expect that everything is handed on a gold plate. This is a project that eagerly seeks contribution from various angles, and documentation is just one of them.

[Fannon]

I agree with @mwjames here. Writing good user documentation is a task in itself. Many changes are (briefly) documented in the release notes and/or the sandbox. This itself may be a bit thin, but it should be enough as a starting point.

The real issue is (imho) that there needs to be someone that has both the skill and the willingness to makes a (rather big!) time invest to write good documentation.

[JeroenDeDauw]

should be enough as a starting point.

I myself run into the problem of the docs that are there not being sufficient. Some of them are straight forwards enough and can be addressed in little time. Others require me to spend a significant time in trying to understand what they are about and which details are relevant. And then there are those I simply do not get at all.

I assume the same is true for most people that have experience with SMW, which leaves only the first category of points having a real hope of being given any love.

most issues are accompanied by a sandbox reference

To me this seems like the wrong way to go about things. Mentioning an issue ID in the release notes is not understandable to users. You can link it of course. Though having users read notes on PRs and GitHub issues is really not ideal. They can then follow yet another link to the sandbox, though the examples there appear, based on my limited experience, a bad substitute for an explanation of what the thing is. Examples are nice once you understand that, however forcing interpretation via reverse engineering examples is not.

You don't have to take my word for it though. What can be empirically observed is that people are not contributing. You can say the release notes are fine, and that if users want them better, they should stop being lazy and help. Or you can ask users what their problems are and what they would like to see improved, and then discuss how you can collaborate in a way that works for everyone involved.

[mwjames]

Examples are nice once you understand that, however forcing interpretation via reverse engineering examples is not.

Well, I could go on arguing about this but nothing that brings us or anybody here forward. My final word on this topic is either people care enough to actually try and figure out what it is about and show some compassion instead of just "I have to expect that ..." and ask if something is unclear or well ...

[JeroenDeDauw]

@mwjames We'll have to agree to disagree on that then.

@mwjames What's the status of MW support? I take it we support 1.26 and 1.27 now?

[JeroenDeDauw]

I've created the tag. Shall I go ahead and do the usual tarball on SourceForge thing? Or does someone want to do the new GH thing proposed in https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1609? Once that is done I'll send a mail and do the tweet.

Fluffing up the release notes before that happens is of course nice. Some minor stuff in https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/1682

[JeroenDeDauw]

I will do the SourceForge thing tomorrow and then send the email. If someone does the GH thing, I can include that in the mail.

[Zabien]

Thanks for picking that up and considering it! I very much appreciate your hard work put into every new release. Understandably, documentation is prio B or less – first things first. I also totally agree on the wiki way of "see something improvable, go ahead, improve it". (Besides, the documentation is not in that bad of a state as it seems I was stating.)

Still, documentation/release notes are basis for marketing. It's what people see first.

What I had in mind was something short similar to “Creative Commons”-for legally binding long “terms & conditions”. Hardly anybody reads them thoroughly, but they are there in full, when details are needed. Same goes for documentation, considering it should be useful for beginners and devs alike.

I would love to see that – and tried – but can’t really do it: Non-dev, non-native.
Doesn't qualify it as a bad idea, though.

Without sentiments, let me add some practical thoughts:

1. Git(hub) is not so much fun for non-devs. Anything above a direct link is a hurdle.
2. The Github issues are not linked from the smw.org website in the release notes, so when I’m not on Github anyway, I will hardly find the issue. (Should I?) Should they be linked?
3. Some of the topics in the release notes (e.g. here) could be linked to internal pages. Could this be part of improvement?
4. Do non-devs actually have to read past the first big headings (Positional units, display precision)? If I don’t understand what follows, will I miss out on a new function for my day-to-day workflow or some major security issues? I guess not?
5. I like translating documentation into German like it’s done at translatewiki.net (via the Translate extension). To open up the documentation for non-natives, is this extension worth being considered? (Sorry, Karsten, again…)
6. Maybe a list of "low hanging fruits for documentation" with tedious tasks for monkeys like me could make participation even easier? (Complex list on Wikipedia, but you get the idea)
7. If it’s new features (not improvements, bug corrections), I’d love to see it introduced even more prominently, along these made-up lines for Twitter…

_Currency values are not easy to handle… now you can default the position of the currency unit.
https://www.semantic-mediawiki.org/wiki/Help:Special_property_Corresponds_to_

_Want to show only 2 or 3 digits after the comma in a number? Or peculiarly 10? You’re now in control of that: https://www.semantic-mediawiki.org/wiki/Help:Special_property_Display_precision_of_

Sorry, if too off-topic. Tirade over.

[s7eph4n]

@JeroenDeDauw If all that needs to be done is creating the release and uploading the zip/tar balls I can do that.

[JeroenDeDauw]

@s7eph4n that'd be much appreciated. There is a little script at build/release/build_tarballs.sh that can help. This is what I used for SourceForge before. Note that the name of the release should be 2.4.0-RC1. Also note that the tag has already been created, and that no new one should be added. Especially not 2.4.0, which would be quite a headache to rectify in a semver.org complain way.

@Zabien some quick replies:

Do non-devs actually have to read past the first big headings [...] I guess not?

Everything except Internal changes has as audience the userbase, so should be understandable by them. The list of bug fixes is presumably less interesting to most users, so I don't think it's all that important to put much time into that. New features and enhancements on the other hand, I imagine most users would want to read through. The highlights are a somewhat random pick out of that list, so it certainly contains a lot of cool new things.

If it’s new features (not improvements, bug corrections), I’d love to see it introduced even more prominently, along these made-up lines for Twitter…

I was actually thinking about something that would help with this significantly yesterday. Created a new issue for that: https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1685

[Zabien]

@JeroenDeDauw , thanks!

The highlights are a somewhat random pick out of that list, so it certainly contains a lot of cool new things.

True! Maybe the thought could go towards a general extensive feature overview with highlights grouped into topics (and links to their release version) – instead of polishing up the release notes too much. There are good starts at Getting started or here. But I'll leave it to this and try to get in medias res when finding the time.

Your idea in #1685 is interesting for several reasons, though I'm not sure if a different release pace would solve a documentation question.

[s7eph4n]

@JeroenDeDauw Done: 2.4.0-RC1

[JeroenDeDauw]

Thanks @s7eph4n! I'm heading off for today now, so will do the release mail first thing tomorrow.

[JeroenDeDauw]

Mail and release tweet send

[mwjames]

Blocker:

• [x] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1699
• [x] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1690#issuecomment-228169408
• [x] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1695#issuecomment-230190575

[JeroenDeDauw]

Am I understanding correctly that the release is still blocked on #1690?

[JeroenDeDauw]

Ping @mwjames

[mwjames]

Ping @mwjames

Let's finish this.

[JeroenDeDauw]

Thanks for creating the tarballs @kghbln. Is it not possible to put the + character in the file name? Semantic.MediaWiki.2.4.0.dependencies.7z implies its just the dependencies of SMW, and not SMW itself. Semantic MediaWiki 2.4.0 (+dependencies).7z is nicer.

[kghbln]

Is it not possible to put the + character in the file name? Semantic.MediaWiki.2.4.0.dependencies.7z

Did not see this. Good point. Actually that's what we are getting when just uploading the files we create. I have now amended the script to cater for the new situation. See pull request https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1722

Also re-uploaded the files with the new names.

[kghbln]

--> via https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1716