Sonic-pi: How to make Sonic Pi more user-friendly for visually impaired / blind users?

[hzulla]

We're looking for users who are visually impaired / blind. Please comment and discuss here.

Please let the developers of Sonic Pi know how you're using your computer (hardware and software tools, e.g. screen readers) and how Sonic Pi works on your setup. What works, what doesn't work well?

Can you tell us about other open source software that works well for you and where the developers can learn a few tricks to improve Sonic Pi's accessibility?

Thanks & welcome!

[ogomez92]

We need to somehow replace the scintilla editor or enable its accessibility settings. I dont have experience with qt development but maybe someone else can fix this, it's really not that complicated.. It's just the scintilla editor that is causing problems. :)

[yaxu]

After a bit of googling I found that openscad seems to use qscintilla and have made it accessible in version 2014.03 for screenreaders. I found one example:
https://github.com/openscad/openscad/pull/674

[hzulla]

@ogomez92 could you nevertheless please describe the system you're using?

[hzulla]

Are you able to build Sonic Pi from source, so that you can test accessibility patches on a development branch?

[ogomez92]

Not really, how do I go about doing that?

[hzulla]

What operating system and what accessibility enhancements for it are you using?

[ogomez92]

windows, NVDA. though the improvements are the same for all major screen readers. Oh and I also will be using a mac soon

[hzulla]

@yaxu nice find. But they removed it again here: https://github.com/openscad/openscad/commit/ba8446e8bfbf770926a13b2d312187077adcb368 as current Qt is supposed to ship with screen reader support by default.

I tried it with my current Ubuntu build using Qt 5.4.2 and the orca screen reader, but that wouldn't work, no speech output from that Sonic Pi build. Then again, I'm not familiar enough with Orca...

[hzulla]

(Well, what we really need is a developer who knows about both Qt and its accessibility features...)

[ogomez92]

I don't think it's qt, qt is fine, we need a replacement for scintilla, or a command line setting to replace the edit control for a non scintilla based thing.

[hzulla]

Afaik, Qscintilla only uses Qt widgets and we need to activate the accessibility settings of Qt accordingly so that Qscintilla uses them automatically.

[indiejane]

I am working with a student and, as we can't find a screenreader that works with the RPi, he accesses it over ssh using Putty with the NVDA screenreader so if Sonic Pi could be accessed directly via the command line, that would be useful, although sonic-pi-cli is a step in the right direction.
I've tried using the windows version of Sonic Pi with the JAWS screenreader and the Mac version with Voiceover. The main problem was that neither set-up read the content of the work area in the GUI.

[hzulla]

I tried using Orca on Xubuntu and found it difficult to run. It uses Pulseaudio which gets blocked once jackd is started by Sonic Pi's supercollider.

While it's possible to get all running together, it's another hurdle to pass. Also, too, no tts output from the Sonic Pi Window in that setup, so something is missing in SP.

It would be worth finding a good working Qt application that works with a screen reader on Linux to learn from its source. Do you know any?

[ogomez92]

No..... using the cli interface is a step backward. If everyone else can use the gui, why can't we?
And don't even bother with orca... It's probably not going to work. I'd focus on windowsmac first.

[hzulla]

@ogomez92 - I'm on Linux and RPi is Sonic Pi's primary development target. I'd want to improve the accessibility situation for all three OS - Win / Mac / Linux.

[jscholes]

I'm using Windows 7 with NVDA as my screen reader. I don't mind how I get code into Sonic Pi, whether it's via a CLI interface or using the GUI. But at the moment neither is really an option.

As has been mentioned, the main problem is with the Scintilla-based editor, which can't even be focussed from the keyboard. Tab traversal doesn't work for the other controls either, but there's not much point going into reams of detail about that when the core of the product isn't usable.

I also tried the work of a couple of other developers to implement Sonic Pi control via OSC, including a Ruby gem and some Erlang code, as well as a Python-based OSC client. But none of these seem to work on Windows. So I find myself completely locked out.

@ogomez92 pointed out, quite rightly so, that accessibility on Linux is not a good quantifier for the level of screen reader access provided by an application in general. Blind users of Linux are definitely in the minority, and I would go as far as to argue that if you focus on Linux accessibility support first and forrmost, you will be wasting your time. Not least because finding blind users to test your implementation will be a challenge.

[hzulla]

A couple of things to note.

Sonic Pi is cross-platform. None of the platforms - Win, OS X, Linux - are second class citizens. Sonic Pi's GUI is using Qt, which is cross-platform. Qt has accessibility features, which are also cross-platform, but most likely we're not using them right yet. So if we fix something on one platform, we're most likely fixing it on all.

I'm not focusing to fix accessibility on Linux, it's just that I'm developing on Linux and as the starter of this thread, that's where I'm testing this. I'm not visually impaired and therefore the correct setup of a computer for that user group is very new to me. Also I'm told that there is fairly large user base of visually impaired Linux users, so I wouldn't scoff at those.

QScintilla is a drop-in editor widget for use by Qt applications. I'm still pretty sure that it's not QScintilla's fault that accessibility doesn't work in Sonic Pi, but that we just don't setup Qt accessibility correctly in Sonic Pi's Qt configuration yet. The fact that screen reading fails for everything in Sonic Pi and not just for the editor is a strong sign for this.

What we need for Sonic Pi is developers who are familiar with the accessibility features of Qt (I'm not, but willing to learn) or a pointer to other Qt-based applications that are friendly to visually impaired users, so that we can learn from their code.

That said, we definitely do want to make Sonic Pi more friendly to visually impaired users, we're just lacking the manpower and/or experience with this particular challenge yet.

[alpha2013]

Hi!
I'm a linux blind user about 4 years and I consider that blinux blind users need to be support and it's not a reason because we are few. Sonicpi need to be accessible.
Now it's true that qt as accessibility features but they have some work to do because the accessibility features from qt is not perfect far of that. Now for scintilla it's true that scintilla gtk or qt is not accessible linux or not scintilla is effectively a problem.

[hzulla]

@alpha2013 thanks for commenting. Could you please describe your computer setup and the tools you use to make it accessible? I'd love to understand what you do as a blind user so that I can have a similar setup to test with during development.

[alpha2013]

is this possible to have a version of sonicpi for linux running on a pc?
I'm a ubuntu user with orca screen reader and I want to build a video
game console based on a raspberypi and this console will use sounds only
no images it will be a game console for blind or someone who want to
play with his or her ears.

Le 2016-04-09 16:03, Hanno Zulla a écrit :

@alpha2013 https://github.com/alpha2013 thanks for commenting. Could
you please describe your computer setup and the tools you use to make
it accessible? I'd love to understand what you do as a blind user so
that I can have a similar setup to test with during development.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/samaaron/sonic-pi/issues/962#issuecomment-207845733

[hzulla]

@alpha2013 Yes. You can run Sonic Pi on desktop Linux. However, we are aware that it isn't as accessible to blind users as we want it to be. Also, I don't know how blind Linux users use their computers, so I have no idea how to test and develop for better accessibility.

Sonic Pi is developed for Windows, OS X and Linux. While we focus on Raspbian with our Linux development, there are binaries for Debian and Ubuntu as well.

[ogomez92]

I think they just use orca on a gnome desktop :)
never used linux myself but I do use windows with NVDA and I've used a mac.

On 4/9/16, Hanno Zulla [email protected] wrote:

@alpha2013 Yes. You can run Sonic Pi on desktop Linux. However, we are aware
that it isn't as accessible to blind users as we want it to be. Also, I
don't know how blind Linux users use their computers, so I have no idea how
to test and develop for better accessibility.

Sonic Pi is developed for Windows, OS X and Linux. While we focus on
Raspbian with our Linux development, there are binaries for
Debian
and Ubuntu as well.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/samaaron/sonic-pi/issues/962#issuecomment-207848187

[alpha2013]

Can you check the kodi project kodi have a kind of self screen reader to
use it perhaps will be able to take that kind of screen reader to use in
sonic-pi.

Le 2016-04-10 03:24, ogomez92 a écrit :

I think they just use orca on a gnome desktop :)
never used linux myself but I do use windows with NVDA and I've used a
mac.

On 4/9/16, Hanno Zulla [email protected] wrote:

@alpha2013 Yes. You can run Sonic Pi on desktop Linux. However, we
are aware
that it isn't as accessible to blind users as we want it to be. Also, I
don't know how blind Linux users use their computers, so I have no
idea how
to test and develop for better accessibility.

Sonic Pi is developed for Windows, OS X and Linux. While we focus on
Raspbian with our Linux development, there are binaries for

Debian
and Ubuntu as well.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/samaaron/sonic-pi/issues/962#issuecomment-207848187

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/samaaron/sonic-pi/issues/962#issuecomment-207939311

[dumbledad]

I've got a blind kid and his friend in the lab mid April, ostensibly to show me what they have been doing with Project Torino https://hxd.research.microsoft.com/work/torino.php

I was thinking that if I get time it might be great to get them started on SonicPi. What's the current state-of-play on this? They'd be Windows 10 based. Are we still blocked by QScintilla / Qt or are screen readers and accessibility tools working now?

[samaaron]

@dumbledad as far as I'm aware the situation is unchanged.

However, I did recently read that the latest QScintilla has made some improvements w.r.t. accessibility. However, that will require me figuring out how to rebuild the GUI with the latest Qt and QScintilla on Windows. Unfortunately I'm thrashing around looking for funding again which is burning a lot of my time and energy.

One short term approach would be to get the prototype web interface working again which I assume wouldn't suffer from such problems.

[SunderB]

I don’t have much experience with html and JavaScript development, but I’d like to help get the html interface working as much as I can. 🙂 I hope it helps with accessibility.

Would it be better to get the existing code working or to remake the backend of the html interface?

[samaaron]

@SunderB getting the existing code working should be very possible - it's just basic HTML and ClojureScript :-)

[SunderB]

I’m busy most of next week, but I’ll try and work on it a little the week after.

I’m trying to find where the IP address and port of the sonic pi server are specified. Is this the right place?

ws.cljs:

(defn mk-ws []
  (reset! ws (js/WebSocket. (str "ws://" hostname  ":8001"))))

[SunderB]

Wait, from the name of that it looks like it's the port of the web server that it makes.

[SunderB]

I've made a pull request: https://github.com/samaaron/sonic-pi/pull/1859, I hope this is a good start to getting the HTML interface working. 🙂

[atti01]

Hello, This is Attila from Germany.
I am myself totally visually impaired, and i am very interested in the accessibility of the qt gui of sonic pi.
As a hint on how to proced i might refer you to john melas, who developed the motif tools for yamaha motif series synthesizers using the qt framework, or alternatively to björn rasmussen who developed the voice conferencing solution teamtalk, whose qt gui implementation makes big progress on windows in conjunction with the nvda screen reader. Unfortunately i am not a programmer, but this might change due to this nice interdisciplinary approach, sonic pi offers...
Hope some more blind folks might chime in, since i posted about sonic pi in a german blind music producers whatsapp group one hour ago.
I'll definitely stick to this topic and highly apreciate the developers openness to accessibility Issues.
Best Regards, Attila.

[samaaron]

@atti01 Hallo! Wie Gehts?

Getting Sonic Pi working well for visually impaired users is a big focus for me. I'm very much aware that things are a complete mess right now and would love to tackle things.

The way I see it there's a few major issues:

1. For the Qt GUI, no work has been done on improving accessibility using Qt's standard tools
2. For the Qt GUI, the text editor component has had no accessibility options. However, looking at the developer logs for the component (qscintilla) it seems this has now been resolved, so I'm looking to get this integrated into a beta as soon as possible.
3. For the HTML GUI, no work has been done to move this beyond a very basic and unsupported prototype.

I'd like to work on all three issues. I'm pretty confident of knowing how to do 2 and 3, but 1 is still a mystery to me. Ideally we'd have a Qt expert who understands accessibility issues to help us out. However, these people you mention, John and Björn might be great contacts to get advice from. Do you happen to know either of then or have their contact details?

Have you tried the Sonic Pi Qt GUI yet? Do you have a list of things that are immediately a problem that I could look into and try and resolve for a beta release for you to play with? Which platform do you use? Windows or macOS?

[bmustill-rose]

All

I've been giving Phil from Riverbank Computing (the author of QScintilla) some help regarding his accessibility work although I have next to no QT development experience so my input has been more test oriented.

As it currently stands, version 2.10.3 of QScintilla does indeed contain some accessibility related code although my (Windows based) testing suggests that it's broken. In fact, if you attempt to use a program that has adopted 2.10.3 in combination with a screen reader that supports IAccessible2 (NVDA for example) the program in question will start consuming large amounts of CPU & become unresponsive shortly after you start writing something & you will find that you have no choice other than to kill its process.

Baring the above in mind, I'd suggest that the version of QScintilla in use on master be kept at 2.10.2 or similar until the issues have been resolved for a couple of reasons:

• Screen reader users who are unaware of Sonic Pis accessibility problems may attempt to run it only to find that their computer becomes unresponsive.
• People who are not completely dependent on a screen reader E.G. people who have some useful vision but who require a screen reader in certain situations may currently be able to use SP to a certain extent. If a version is released that uses 2.10.3 it would (IMO) be a regression for them due to the unresponsiveness problem.

I should add that it's been a pleasure working with Phil so far & I'm thrilled that he's showing continued interested in making his component accessible. I'm sure that we'll have accessible QScintilla soon - we're just not quite there yet.
I'll comment when I have any updates but in the mean time I'll be happy to answer any accessibility-related questions anyone may have as I'd love to be able to use SP without having to rely on the gem.

[bmustill-rose]

2.10.4 (released earlier today) fixes the above issue.

[samaaron]

@bmustill-rose thanks for letting me know - I was about to ask this exact question :-)

[Vindoes]

@bmustill-rose Does the 2.10.4 update fixes all isues regarding the use of a screenreader like JAWS or NVDA with Sonic Pi? Thank you.

[bmustill-rose]

Definitely not! The 2.10.4 release will in theory make the text area accessible to a screen reader that supports IAccessible2 and nothing more. It does however remove the blocker on figuring out what accessibility issues SP has. Are you looking at this from a user or developer point of view?

[Vindoes]

Too bad! I'm looking at this from a teacher point of view. I'm searching for an application that is accessible for blind children so they also can learn the basics of coding.

[yonghuming]

believe you can do well

[mjcheverie]

@samaaron Hi! I am a teacher of blind students in Los Angeles, and just recently discovered Sonic Pi. I am also an avid user of OpenSCAD in behalf of my students. I am wondering if the new version of QScintilla (2.10.4) has made any difference in the accessibility on the editing window of Sonic Pi to screen readers like NVDA or Voiceover. I have a very talented blind student who is anxious to start using Sonic Pi. Thanks!

[mjcheverie]

@samaaron Hello again! A few days ago I learned about FoxDot, which I am sure you are familiar with. One very attractive feature for blind users is that the coding can be done in a command window. The command window is accessible to screenreaders and refreshable Braille displays, which means that a blind user can code in real time. Is it be possible to do this in Sonic Pi? FoxDot communicates with SuperCollider; I'm wondering if Sonic Pi has that capability. I am very impressed and enjoy very much your commitment to education. This is reflected in your tutorials for Sonic Pi. Unfortunately, FoxDot is a bit more difficult to learn for my students. Your tutorials are excellent! I would very much like my students to be able to use Sonic Pi.

[obscurerichard]

I have a PR that might help a little bit with this (#2170) - it has editor tab switching using meta keys 0 through 9. I made this for myself but it would help with navigating between code windows, which I find really awkward without additional keyboard shortcuts.