Julia: better printing of sparse matrices?
Compare:
julia> A = Diagonal(rand(1000))
1000×1000 Diagonal{Float64,Array{Float64,1}}:
0.0341907 ⋅ ⋅ ⋅ … ⋅ ⋅ ⋅
⋅ 0.758919 ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ 0.700876 ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ 0.453502 ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ … ⋅ ⋅ ⋅
⋮ ⋱
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ … ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ 0.489115 ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ 0.791967 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.164329
julia> sparse(A)
1000×1000 SparseMatrixCSC{Float64,Int64} with 1000 stored entries:
[1 , 1] = 0.0341907
[2 , 2] = 0.758919
[3 , 3] = 0.700876
[4 , 4] = 0.453502
[5 , 5] = 0.560549
[6 , 6] = 0.832382
⋮
[995 , 995] = 0.270777
[996 , 996] = 0.0664044
[997 , 997] = 0.689441
[998 , 998] = 0.489115
[999 , 999] = 0.791967
[1000, 1000] = 0.164329
Why can't a SparseMatrixCSC be printed in the same way as the structured sparse-matrix types? It would be a heck of a lot more readable.
PS. On an unrelated note, I also notice that the SparseMatrixCSC output isn't setting the :typeinfo property of the IOContext, so it screws up e.g. #30575. Update: This is fixed by #30589.
stevengj
All 66 comments
Not kidding, I had the exact same thought earlier today while reviewing some sparse code!
rfourquet
on 4 Jan 2019
Note that for a matrix with density p, the probability of having all k shown entries to be zero is (1-p)^k which for p small behaves as exp(-k p). Assuming k~200 (about the number of 0.0 that julia fits in my terminal), in an arguably realistic scenario for an NxN matrix with N=10^6, p=5/N this probability is ~ 0.999. I think that in a good fraction of real cases all shown entries would be zero. So I don't think that the current representation is so bad. ;-)
abraunst
on 4 Jan 2019
I think that in a good fraction of real cases all shown entries would be zero.
True, but the diagonal is quite often filled though.
fredrikekre
on 4 Jan 2019
I also don't like the current printing. Another possibility that we discussed a few years ago was to print a text spy plot instead of the numerical values. Sparse matrices are often quite large and it can often be more interesting to see the distribution of non-zeros than a few of the elements.
andreasnoack
on 4 Jan 2019
@abraunst, in real sparse matrices it's pretty rare to have a random distribution like that. One option would be to print a portion starting at the first nonzero entry.
stevengj
on 4 Jan 2019
I think people working on PDEs and such have the structured sparse matrices, but the folks working on graphs usually don't have that structure.
I do agree that what we do today should be replaced and I also like the idea of the text spy plot.
What if the portion around the first nonzero entry is also extremely sparse? (you could easily have just 1 nonzero in there).
ViralBShah
on 4 Jan 2019
We could use Braille patterns to print a fairly compact text spy plot: https://en.wikipedia.org/wiki/Braille_Patterns
StefanKarpinski
on 4 Jan 2019
Yes, similar to https://github.com/Evizero/UnicodePlots.jl
stevengj
on 4 Jan 2019
Off-topic - I would love to ship UnicodePlots in stdlib. That would also give us these spy plots for little effort.
ViralBShah
on 4 Jan 2019
Using output like Diagonal when all (or most?) of the matrix fits on the screen, then switching to a text spy plot for larger matrices would be ideal.
JeffBezanson
on 4 Jan 2019
Another possibility for the Diagonal-like form is to print the first and last nonzeroes together with their column and row indices, but in matrix form. Something similar to
julia> sparse(A)
1000×1000 SparseMatrixCSC{Float64,Int64} with 1000 stored entries:
_______|11 ______ 14 _______ 102 ______ 932 ____
27 |⋅ 0.758919 ⋅ ⋅
96 |0.389739 ⋅ ⋅ ⋅
203 |⋅ ⋅ 0.700876 ⋅
701 |⋅ ⋅ ⋅ 0.453502
with perhaps some form of color highlighting (or bold, or italic) for the column/row indices
pablosanjose
on 7 Jan 2019
Another possibility for the
Diagonal-like form is to print the first and last nonzeroes together with their column and row indices, but in matrix form. Something similar to
Seems promising, although it is not exactly clear what "first" and "last" indices are. For example, what would be the output for sparse(1:n,n:-1:1,fill(1,n),n,n) (antidiagonal) or
EDIT: sorry, the antidiagonal shape is easy
abraunst
on 7 Jan 2019
Seems promising, although it is not exactly clear what "first" and "last" indices are.
I think it wouldn't be difficult to define this in a sensible way. For example take the N non-zeroes closest to the four corners of the matrix, and create a list of their respective rows and columns, removing duplicates.
pablosanjose
on 7 Jan 2019
As soon as you put the reader in charge of interpreting row and column indices, I don't see much advantage over what we are currently doing. The visual layout doesn't actually add anything if it bears no relationship to the arrangement of the matrix.
stevengj
on 7 Jan 2019
It does have a few advantages:
- It's more visually obvious which are the row and which are the column indices
- It's more compact since it can potentially show many more non-zeros in the same space
- It gives some visual sense of the shape of the data (same row, same column, etc.)
The remaining problem is choosing which rows and columns to select.
StefanKarpinski
on 7 Jan 2019
Personally, to me, the spy plot is the thing I like seeing when I have a sparse matrix. I would love UnicodePlots in stdlib and just have this at my fingertrips (but I can always have that with a few lines of code and installing a package)
ViralBShah
on 7 Jan 2019
The thing about using a spy plot for the default output is that it's backwards from all our other structured matrices — they use dots to represent locations with _no_ stored value. This becomes particularly apparent if we switch between the two output types depending upon matrix size.
So I also like where @pablosanjose's idea is going.
mbauman
on 7 Jan 2019
The thing about using a spy plot for the default output is that it's backwards from all our other structured matrices
- I doubt that this will be confusing in practice
- Sparse matrices are also very different from the structured matrices in that they don't have a fixed structure. The structure of a sparse matrix is probably the most important thing to know about it.
Using a pattern similar to the structured matrices would become something like
10974×10974 Symmetric{Float64,SparseMatrixCSC{Float64,Int64}}:
1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … ⋅ ⋅ ⋅
⋅ 2.27861e7 -2.66358e-7 -32711.2 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ -2.66358e-7 3.94135e5 4.33531e-7 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ -32711.2 4.33531e-7 1.84354e7 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ … ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … ⋅ ⋅ ⋅
⋮ ⋮ ⋱
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.80032e7 27001.2 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … 5080.43 3.58957e7 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ -1.17899e5
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 2.34176e6 1.03842e8 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … 3.6964e6 4.14671e7 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ -6.21564e6 3.76573e7 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ -66123.9
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … -1924.62 -5001.35 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.3521e8 -5050.15 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ -5050.15 1.3521e8 ⋅
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.05842e6
and a text spy plot (from UnicodePlots) would look like (except that the character width seems to be off on GitHub).
Sparsity Pattern
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
1 │⣟⠿⣜⠓⠦⣄⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ > 0
│⠙⢷⡈⠛⢶⣠⣄⠙⠳⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ < 0
│⠀⠀⠩⣄⡈⢻⣿⣯⡳⣿⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠈⠛⠲⠻⢶⡻⡿⣜⡻⢧⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⢬⡙⠷⣍⠳⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠳⢮⣻⢦⣿⢶⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⣯⡻⣯⣝⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠲⣏⣷⣾⣛⢦⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢮⣹⠾⣌⡷⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⣮⡻⡷⣜⢷⢤⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢳⣜⠻⢧⣘⡳⠄⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢧⠙⠻⢦⡈⠻⢦⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢷⣄⠹⠿⣦⣀⠹⢶⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠷⣤⡈⠙⢶⣤⡈⠙⠖⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢲⣄⠉⠻⣦⣆⡨⠹⣶⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠲⣦⣒⡙⠷⢦⣐⡋⠷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⢯⣌⢛⡿⣿⣛⡷⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⠻⣞⣿⡿⣞⡿⢦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠻⢮⣻⣷⣟⢷⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⣿⣛⣷⣟⣲⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠿⣿⣿⣿⣟⢦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠿⣿⣷⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢿⡿⢿⣯⡿⢦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⢾⣯⣿⣾⣯⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠺⣿⢿⣯⣿⣶⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠷⣟⣿⣾⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⣿⡿⣿⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠻⣿⣿⣷⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⣿⣿⣶⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠿⣿⣿⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢿⣿⣷⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢿⣿⣷⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⢻⣽⢾⣽⠦⢤⣀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢻⣬⠷⢧⣈⣿⢦⡀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠳⣷⣯⣿⣿⣯⣀⠀│
10974 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠙⢿⣿⣷│
└────────────────────────────────────────────────────────────────────────────────────────────────────┘
1 10974
nz = 428650
Found my way here through discourse. Let me know if I can help. I am also not opposed to simple copy&paste of specific code excerpts. Note though that I am not the original author of the spy function in UnicodePlots; that would be @dpo
Evizero
on 8 Jan 2019
I would be delighted to see spy (and all of the amazing UnicodePlots really) find its way into base!
dpo
on 8 Jan 2019
I'm afraid that what's going to happen here is that many people are insisting on wanting a spy plot for this, and as a result better sparse matrix printing is going to get coupled with bringing UnicodePlots or some subset of it into Base or stdlib. However, that won't and shouldn't happen. If a package was a sports car, then being in stdlib—or worse, Base—is like it having to drag a a ten ton trailer around behind it all the time. It brings all development to a near complete halt. If you want spy plots, install UnicodePlots and use the spy function. Let's keep this issue focused on improvements to sparse matrix printing that don't require bring the development of UnicodePlots to a standstill.
StefanKarpinski
on 8 Jan 2019
(Part of) my point was that the code needed for producing a braille-character spy visualization exists and is decently straighforward. So if the wish is to go towards that route, then there is the option to factor it out of UnicodePlots and into something self contained that can live whereever (instead of reinventing the wheel)
Evizero
on 8 Jan 2019
If we can get a simple printing of just the spy plot part, I would be fine with that. It's hard to tell from the outside how straightforward it is or isn't.
StefanKarpinski
on 8 Jan 2019
To see if the Diagonal-type printing could work I've written a small package (https://github.com/pablosanjose/SparseShow) to test it out. It selects and prints a subset of non-zeroes close to the corners. It's just a proof of concept to see if that approach gains any traction, feel free to ignore this if you think we should go in a different direction.
The non-zero selection is currently not very efficient, in the sense that there is an intermediate conversion of the sparse matrix s to indices and values with findnz(s) (but it could be done in a better way without so much allocation, and a bit more code). The complete list of nonzeroes is then sorted according to a distance function dist (Metropolis distance from corners at the moment, can be changed to whatever).
This is the kind of rough output it generates (on a small-size repl)
julia> using SparseShow, SparseArrays
julia> sp = sprand(100, 100, .05)
100×100 SparseMatrixCSC{Float64,Int64} with 509 stored entries:
[59 , 1] = 0.893654
[3 , 2] = 0.604112
[46 , 2] = 0.760531
⋮
[41 , 100] = 0.345492
[82 , 100] = 0.506576
[85 , 100] = 0.709841
julia> spshow(sp)
8×5 Array{Union{Spdef, Float64, Int64},2}:
2 4 5 96
1 ⋅ 0.375638 0.0951865 ⋅
3 0.604112 0.360448 ⋅ ⋅
6 ⋅ ⋅ ⋅ 0.504533
91 0.170411 ⋅ ⋅ ⋅
92 0.990792 ⋅ ⋅ ⋅
99 ⋅ ⋅ 0.36217 ⋅
100 ⋅ ⋅ 0.600704 ⋅
(Ignore for the time being the summary Array{Union{Spdef, Float64, Int64},2})
Before putting any real work into this, do you think this kind of thing is interesting? There seems to be a 50/50 split in the post above
pablosanjose
on 9 Jan 2019
Certainly an improvement. I think it is interesting and good to take it all the way.
ViralBShah
on 9 Jan 2019
Ok. One difficulty I've hit is how to place the dotted ellipses at arbitrary rows/columns, other than the middle of the matrix. As far as I can tell, the Base machinery (Base.print_matrix) doesn't allow me to choose their position, correct? Also, it doesn't allow me to prepend the display of a matrix with an index column/row (what I did up there is an inelegant hack).
pablosanjose
on 9 Jan 2019
I like it. One issue is that it makes it look like this is _all_ of the data in the sparse matrix. If there are non-zeros between some rows or columns, I think that it should print ⋮ and ⋯ in between. The non-zero selection should be done such that you get at most a single row and/or column ellipsis.
StefanKarpinski
on 9 Jan 2019
Yes, see comment above. The correct spot for the ellipsis is not the middle of the displayed matrix, however. It should be chosen as the split between indices i<N/2 and i>N/2 (where N is the size along the relevant axis). Do we need to extend Base.print_matrix to allow for this? It seems like quite a complicated function.
pablosanjose
on 9 Jan 2019
The non-zero selection should be done such that you get at most a single row and/or column ellipsis.
That is how it's done in my code. Only complete clusters around each of the four corners are displayed.
pablosanjose
on 9 Jan 2019
Ok, that sounds good. Then I guess the only required change is to print ellipses wherever there a rows or columns for which no values are printed.
Do we need to extend Base.print_matrix to allow for this? It seems like quite a complicated function.
Good lord, no. I originally perpetrated that horror show and am somewhat shocked that it has survived this long. Please don't feel any need to cram more functionality into it.
StefanKarpinski
on 9 Jan 2019
Good lord, no. I originally perpetrated that horror show and am somewhat shocked that it has survived this long. Please don't feel any need to cram more functionality into it.
Haha, that's a relief! I'll look into creating a custom print_matrix specifically for this type, and try to submit a WIP PR in the next few days, if work allows. I'll watch this issue for further reactions/feedback/decisions.
pablosanjose
on 9 Jan 2019
Braille spy plot PoC:
Base.show(o::IO,::MIME"text/plain",m::SparseMatrixCSC)=(w=min(m.n,160);h=min(m.m,120);P=
fill(5<<11,(w+3)÷2,(h+3)÷4);P[end,1:end-1].=10;zip(findnz(m)...).|>u->(x=(u[2]-1)*w÷m.n;
y=(u[1]-1)*h÷m.m;P[1+x÷2,1+y÷4]|=Int(")*,h08H¨"[4x&4+y&3+1])-40);print(o,join(Char.(P))))
Result:
julia> sprandn(30,30,0.3)
⠊⢤⡠⢀⢆⣁⡄⣰⢔⠿⡀⡄⢈⠁⠆
⢮⠂⡋⠥⠂⡈⢀⢂⠘⠅⢱⠺⢒⠠⡱
⠋⢁⡁⡀⠨⠊⡀⢶⢆⠐⡤⠼⢬⢈⠣
⡪⡀⠫⢧⠏⢕⡡⢁⠃⣢⠃⠃⣐⠰⠌
⢦⠙⡀⢅⠀⡃⠀⠨⠀⡐⢬⣄⠰⢂⠁
⠔⠈⠄⢈⠀⠠⡙⢀⠊⢠⢭⡦⠄⡆⠧
⣀⢈⠂⢁⠂⠑⢙⠀⡐⡈⠢⠈⣜⠄⡃
⠃⠐⠂⠈⠂⠙⠁⠃⠐⠒⠈⠀⠈⠀⠐⠀
julia> sparse([(z=(x-80)/52+(y-50)/57im;(1:40).|>_->z=z^2-.8+.156im;abs(z)<2) for y=1:100,x=1:160])
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣐⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣤⣒⣢⠠⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢴⡟⠿⠿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢿⡀⣖⠲⣦⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣧⢆⣄⣄⣳⣄⣤⡀⠀⠓⠚⠁⢘⢿⣷⣷⠀⡠⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠐⠛⠀⣿⣿⣿⣿⣿⣦⡀⡀⠀⡴⣿⣿⡿⡀⠒⣎⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢺⢚⠤⣄⣀⢤⠄⠀⠀⠀⠀⠀⠀⠰⡠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣇⣻⣾⣷⣿⣿⣿⣷⣶⡷⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⢄⣀⡁⡁⣿⣿⣶⣷⣐⠀⠀⠀⠀⢄⢶⣶⣿⣿⣿⣿⠿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣴⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⠀⠀⠀⠀⠀⡠⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣥⣿⣿⣿⣿⣿⣿⣿⣿⣷⡂⠀⠀⠠⣰⣿⢿⠛⠗⠙⠹⠇⢻⣿⡿⠛⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣥⠁⠀⠀⠀⠀⠀⠀⠀⠀⣒⢤⣠⠀⠀⡞⡍⡋⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⢀⠆⣴⣤⠀⠀⣾⠟⠋⠻⡟⣿⣿⣿⣿⣿⣿⡂⠀⠀⢨⡿⠁⠀⠀⠀⠀⠀⠀⠀⣨⡿⢠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡞⠀⠀⠀⠀⠀⣀⣃⣶⣷⣤⣾⣿⣿⣆⠈⠁⣿⡄⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⣠⡀⠟⠛⣿⣿⠀⢾⢠⣤⠀⠀⢘⡿⢿⣿⣿⣿⠍⠀⠀⢐⣿⠁⣠⣶⣶⣀⡀⠀⠀⠀⠀⠀⢾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣹⠀⠀⠀⠀⠤⣼⣿⢿⣿⢿⣿⢿⣿⣿⣿⣼⣷⣿⣷⡽⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠘⠸⢦⠍⢰⣿⣿⡄⠈⠉⣸⠁⠀⠤⣶⣿⣿⣯⣰⠒⠀⠀⠐⢹⣦⣿⡿⠉⣿⡄⠀⠀⠀⠀⢰⣼⠉⠙⣿⣿⣿⣿⣿⣿⣧⠾⠀⠀⠀⠀⣰⣿⡟⠉⠁⠀⠀⠈⠘⢿⡛⣿⣿⣿⣿⣿⣿⡅⠀⠀⠀⠄⣀⢠⠄⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠉⢟⢹⡿⠷⠛⠋⠀⣰⣿⣿⣿⣿⣿⡿⣠⠀⠀⠀⠀⠈⠉⠀⣀⣿⡃⠀⠀⠀⠀⢠⣴⣤⣶⣿⣿⣿⡟⠋⢙⣉⠀⠀⠀⠀⠀⣾⡇⠀⣠⣤⣀⠀⠀⠀⠈⢸⣿⣿⣿⣿⡿⠟⠀⢠⡶⠞⠿⣿⣴⣧⠄⢀⠀⠀⠀
⠀⠀⠀⠀⠀⠉⠀⠀⠁⠀⠀⠀⡽⣿⣿⣿⣿⣿⣷⣾⣷⢠⣀⢀⣀⢤⣴⡿⠟⠀⠀⠀⢀⣸⣋⣿⣿⣿⣿⣿⣿⣷⣤⣼⠙⠀⠀⠀⠀⠀⣹⣧⣼⣿⡏⢻⣔⠀⠀⠐⢚⣨⣿⣿⠟⣋⠁⠀⢼⣠⢤⠀⢹⣿⡟⠠⠍⣻⡰⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⡝⣿⡟⠿⢹⣿⣿⣿⣿⣿⣿⣿⣿⡿⠉⠁⠀⠀⠀⣸⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠃⣀⡀⠀⠀⠀⠀⠘⠛⠋⠀⣼⠗⠀⠀⣠⣽⣿⣿⣿⣿⡷⡀⠀⠈⢉⡸⠃⢸⣿⡷⢶⠇⠈⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠹⡇⡤⡀⠻⢿⣿⢋⡙⠟⠃⠆⠀⠀⠀⠀⠀⢰⣻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡋⠸⣯⣀⠀⠀⠀⠀⠀⢀⢀⣼⡯⠀⠀⠀⣺⣿⣿⣿⣿⣿⣷⣿⣦⣶⣿⠃⠀⠈⠙⠁⠃⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠠⡦⡵⠃⠀⠈⠈⠑⡂⠀⠀⠀⠀⠀⠀⠀⠀⢌⣽⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣾⣿⣿⡆⢿⣴⣔⣷⣾⣿⢟⠈⠀⠀⠀⢚⣿⣿⣿⣿⡿⡿⣿⣿⢏⠅⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠏⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡘⠉⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⡛⠃⠁⠀⠀⠀⠐⠘⢟⢻⣿⡇⢄⢄⠀⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢘⡟⠛⡿⣿⣿⡟⡿⠻⡆⣿⣿⣿⣿⣿⣿⡿⠿⠿⠿⠏⠘⠀⠀⠀⠀⠀⠀⠀⠉⠁⠀⠉⢱⠺⠂⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⡓⡂⢸⣿⣿⢟⠁⠀⠀⠈⠻⣿⣿⣿⣿⣇⣰⢖⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠀⠘⠟⢿⣷⡀⢴⡒⣆⠀⠈⠁⠙⠆⠁⠁⠋⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⠒⠃⣸⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢿⣿⣷⣟⠡⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠈⣒⡊⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠑⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
maxbennedich
on 11 Jan 2019
:boom:
JeffBezanson
on 11 Jan 2019
Wow! That some mindblowing stuff!
While much less spectacular, here is an update of the other approach, now with ellipses. I ended up writing an alternative to the complicated print_matrix in Base, which I think is more transparent than the old (and slightly more general, in that it allows for table headers).
Small demo:
julia> using SparseArrays, SparseShow
julia> sparseshow(sprand(1000,1000,.002))
1000×1000 SparseMatrixCSC{Float64,Int64} with 2042 stored entries:
____|2________|6________|7________|9________|19________|38_______ … |974______|990______
1| ⋅ ⋅ ⋅ ⋅ ⋅ 0.157993 ⋅ ⋅
3| ⋅ ⋅ ⋅ ⋅ 0.177 ⋅ ⋅ ⋅
10| ⋅ ⋅ ⋅ ⋅ 0.171668 ⋅ ⋅ ⋅
13| ⋅ 0.920032 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
29| ⋅ ⋅ 0.100892 ⋅ ⋅ ⋅ … ⋅ ⋅
136| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.619486 ⋅
142| ⋅ ⋅ ⋅ ⋅ ⋅ 0.348234 ⋅ ⋅
163| ⋅ ⋅ ⋅ ⋅ 0.0266989 ⋅ ⋅ ⋅
285| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.819157 ⋅
398| ⋅ 0.213834 ⋅ ⋅ ⋅ ⋅ … ⋅ ⋅
453| ⋅ 0.552437 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
461| ⋅ ⋅ ⋅ 0.909883 ⋅ ⋅ ⋅ ⋅
489| ⋅ ⋅ ⋅ ⋅ 0.381582 ⋅ ⋅ ⋅
⋮ ⋮ ⋱
577| ⋅ ⋅ ⋅ ⋅ ⋅ 0.909606 ⋅ ⋅
632| ⋅ ⋅ ⋅ 0.370542 ⋅ ⋅ … ⋅ ⋅
718| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.540304 ⋅
738| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.209349 ⋅
756|0.870332 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
806| ⋅ ⋅ ⋅ 0.56719 ⋅ ⋅ ⋅ ⋅
830| ⋅ ⋅ ⋅ 0.627342 ⋅ ⋅ … ⋅ ⋅
895| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.161652
966| ⋅ 0.282836 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
988|0.413226 ⋅ ⋅ 0.56368 ⋅ ⋅ ⋅ ⋅
990| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 0.837156 ⋅
991| ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ … ⋅ 0.733607
995|0.844924 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
While there are some bugs to be ironed out, I think the state of this might be sufficient to make a decision. Despite having devoted some time to SparseShow, I'm really seduced by that minimal spy plot.
pablosanjose
on 11 Jan 2019
That Julia set spy plot is delightful! I do worry about how one would see actual values in a sparse matrix. Would we print actual values if the whole thing fits on screen so that you can take a slice of a matrix in order to see actual values? Or would you have to do scalar indexing to get values?
StefanKarpinski
on 13 Jan 2019
Braille spy plot PoC:
Base.show(o::IO,::MIME"text/plain",m::SparseMatrixCSC)=(w=min(m.n,160);h=min(m.m,120);P= fill(5<<11,(w+3)÷2,(h+3)÷4);P[end,1:end-1].=10;zip(findnz(m)...).|>u->(x=(u[2]-1)*w÷m.n; y=(u[1]-1)*h÷m.m;P[1+x÷2,1+y÷4]|=Int(")*,h08H¨"[4x&4+y&3+1])-40);print(o,join(Char.(P))))
I'd like to point out this:
julia> sprand(10000,10000,0.01)
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀
I don't think it's terrible in this particular case (in the end, there's no way to show this amount of information in one screenful), but I do worry about the fact that many useful cases could fall in this category.
abraunst
on 13 Jan 2019
You could do a density plot: you divide the matrix into MxN bins (where M,N are determined by the display size) and compute the sum of log1p(abs(A[i,j])) for all of the i,j in each bin. Then plot the color (or grayscale) in each bin corresponding to this magnitude, rescaled to [0,1]?
stevengj
on 13 Jan 2019
Then plot the color (or grayscale) in each bin corresponding to this magnitude, rescaled to [0,1]?
That would be really nice, but is it at all possible to do that at the Braille-dot level? I would have thought you can only shade a full character, right?
pablosanjose
on 13 Jan 2019
I'd like to point out this
Yes, I agree that's a bit odd. It's how spy is implemented in both UnicodePlots and MATLAB. An alternative is to sample the matrix at each screen "pixel":
Base.show(o::IO,::MIME"text/plain",m::SparseMatrixCSC)=(w=min(m.n,40);h=min(m.m,30);
print(o,join(Char(v>0 ? (c=5<<11;for x=0:1,y=0:3 (p=(2v+x)*m.n÷w)≤m.n&&(q=(4u+y)*m.m÷h)≤
m.m&&m[q,p]≠0&&(c|=Int(")*,h08H¨"[4x+y+1])-40);end;c) : 10) for v=0:(w+1)÷2,u=1:(h+3)÷4)))
It's not perfect either since it misses entries:
julia> A = sprand(10000,10000,0.001)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠠⠀⠀⠀⠀⠀⠀⠀
julia> nnz(A)
99824
But maybe it's preferred since at least it preserves the density? Compare to the previous implementation (and UnicodePlots and MATLAB):
julia> sprand(10000,10000,0.001)
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠀
It would also solve a problem which I think several of the implementations above suffer from (including SparseShow if I'm reading the code correctly) -- having a complexity ~O(nnz). That doesn't seem acceptable for a show implementation? Example:
julia> @time A = sparse(bitrand(10000, 10000));
4.769406 seconds (50 allocations: 1.694 GiB, 8.19% gc time)
julia> @time spy(A)
5.335417 seconds (82 allocations: 2.666 GiB, 6.57% gc time)
Compared to (with the above implementation):
julia> @time show(stdout, MIME"text/plain"(), A)
...
0.000283 seconds (20 allocations: 1.672 KiB)
...the implementations above suffer from (including SparseShow if I'm reading the code correctly) -- having a complexity ~O(nnz).
That's correct for SparseShow: I do a complete conversion of the sparse matrix to I,J,V decomposition with findnz. However that is not really necessary: one could write an iterator that yields the non-zeroes at i,j ordered by their metropolis distance dist(i,j) to the corners, without allocating the I,J,V arrays. I tried this, but the code complexity was really high, and I couldn't really find a compact and elegant way to write this iterator. I do think that code simplicity is very important here (as stated by @StefanKarpinski, for example), so your untra-lean PoC spy code is really appealing to me. If we could color each of those single dots with different levels of gray representing nnz density, as pointed out by @stevengj, I'd say that would be the ideal solution here.
pablosanjose
on 13 Jan 2019
Another option is to use this kind of approach. It uses ▀ (U+2580, 'Upper Half Block') instead of Braille dots.
EDIT: The idea is to control both the background and the foreground color (16-grayscale in 256-color terminals) to represent two quasi-square pixels with one character.
pablosanjose
on 13 Jan 2019
That would be really nice, but is it at all possible to do that at the Braille-dot level? I would have thought you can only shade a full character, right?
True. A simpler option would just divide the matrix into MxN bins and put a braille dot for any bin that contains any nonzero entry. But I guess that's what @maxbennedich already proposed?
For sparse matrices, there is also the argument that only the nonzero pattern really matters, not the magnitude of the nonzeros, so a dot for nonzero (regardless of magnitude) might be the best option anyway.
stevengj
on 13 Jan 2019
A simpler option would just divide the matrix into MxN bins and put a braille dot for any bin that contains any nonzero entry. But I guess that's what @maxbennedich already proposed?
It's what my first suggestion resulted in, and what I've seen other spy implementations do. However, all implementations I've seen, do this by iterating and plotting each individual non-zero entry, which I think is problematic. But now that you rephrase the problem this way... yes obviously that's a better way to solve it! It should be fairly straightforward and efficient to figure out if a given bin contains any nonzero (by analyzing colptr and rowval).
maxbennedich
on 13 Jan 2019
For sparse matrices, there is also the argument that only the nonzero pattern really matters, not the magnitude of the nonzeros, so a dot for nonzero (regardless of magnitude) might be the best option anyway.
I agree. After all we want to use the same method to show sparse matrices of non-numeric types too. +1 to showing their density, however, as you said, not just whether there is any nonzero in a given bin. It's true that that requires O(nnz) complexity (must collect information from all nonzeros), but I don't agree that is a problem. If you ask to see a humongous sparse matrix, well, that's your responsibility, right? I don't think it will take more that a fraction of a second in any realistic case (though that is just a guess). Moreover, this kind of density sampling could be done with a single allocation of the (small) output matrix, and a sequential (cache friendly) scan of the sparse matrix.
I'm really interested in the ▀ (U+2580, 'Upper Half Block') approach using grayscales, as an analogue sort of the Braille spy plot. I think the result could be truly useful and beautiful, and probably quite simple to code. Actually, my only concern is that the U+2580 symbol does not fill the whole upper half block of a character in my terminal, but leaves a small empty bit above the block (font-specific I'd assume). It quite spoils the whole procedure (see e.g. the upper edge of the pixel matrix in this example). Is there a way to avoid that?
EDIT: cross post with @maxbennedich!
pablosanjose
on 13 Jan 2019
When the sparse matrix is very large the trouble is that the entire spy plot output can tend to become non-zero. Another way to represent it might be to split the matrix into M×N bins and count how many non-zeros there are in each and print dots only for bins have more than the median number of non-zeros per bin. That way what you're plotting is representative of the non-zero density.
StefanKarpinski
on 13 Jan 2019
Another PoC using nonzero densities encoded in grayscale half-blocks:
using SparseArrays
sparsedensityshow(S::SparseMatrixCSC, gamma = 1.0) = sparsedensityshow(stdout, S, gamma)
function sparsedensityshow(io::IO, S::SparseMatrixCSC, gamma = 1.0)
(m, n) = size(S)
canvassize = (displaysize(io) .- (4, 0)) .* (2, 1)
bin = maximum(((m, n) .+ canvassize .- (1, 1)) .÷ canvassize)
colorsize = (round(Int, (m ÷ bin + 1)/2) * 2, n ÷ bin + 1)
density = fill(0.0, colorsize)
rows = rowvals(S)
for col in 1:n
for ptr in nzrange(S, col)
(i, j) = (rows[ptr] - 1) ÷ bin, (col - 1) ÷ bin
bintotal = (i == colorsize[1] - 1 ? rem(colorsize[1] - 1, bin) : bin) *
(j == colorsize[2] - 1 ? rem(colorsize[2] - 1, bin) : bin)
density[i + 1, j + 1] += 1/bintotal
end
end
for row in 1:2:colorsize[1]
for col in 1:colorsize[2]
printpixel(io, density[row, col], density[row + 1, col], gamma)
end
resetpixel(io); print(io, "\n")
end
return nothing
end
printpixel(io, fg, bg, gamma) = print(io, "\e[38;5;",
232 + round(Int, (23 * bg^gamma)), ";48;5;",
232 + round(Int, (23 * fg^gamma)), "m", "▄")
resetpixel(io) = print(io, "\e[39;49m")
This gives this result (EDIT: embedded a bitmap directly here).
It's not rendered correctly in Github (no fancy ANSI codes, I guess). The contrast of bins can be adjusted with parameter gamma. By using a lower-half block "▄" character instead of an upper-half block we can avoid the problem mentioned above. An sprand(10000,10000,0.01) is rendered in 0.1 seconds in my machine.
pablosanjose
on 14 Jan 2019
When the sparse matrix is very large the trouble is that the entire spy plot output can tend to become non-zero.
Indeed. Do you think it's common in practice that every bin will include non-zeros, or would most matrices tend to have a more defined structure?
Another way to represent it might be to split the matrix into M×N bins and count how many non-zeros there are in each and print dots only for bins have more than the median number of non-zeros per bin. That way what you're plotting is representative of the non-zero density.
Interesting idea. Are you not concerned about iterating over all non-zeros? For large matrices, it could mean that show becomes a bit sluggish.
I've created two more Braille based PoCs; one that prints a dot for any non-zero in the bin (as above, but much more efficient), and one that prints a dot if the count is above the median. Code available here, and some sample output here:
julia> show_any_nonzero(sprandn(500,1000,0.5); maxw = 24)
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
nz = 250016
julia> show_above_median_nonzero(sprandn(500,1000,0.5); maxw = 24)
⠸⠶⠒⢀⢲⢤⠐⣓⢍⣰⢦⡖⢸⣦⠴⢠⢔⡹⠴⣜⠤⠐⣲⣽
⣉⡹⡚⠒⣘⢉⢠⣗⢨⣜⣓⣣⢨⣛⣱⠐⡑⢓⢄⡛⣿⢜⣃⢛
⠘⠜⢢⢨⣿⢵⠸⠟⣼⢸⢆⢿⢴⡞⡴⠽⡽⠘⣶⠰⡺⠘⣯⡗
⠒⣶⢐⠒⢃⠴⣰⣲⣷⢰⢚⣠⢠⣦⡾⢖⢄⡕⢠⣁⣌⠀⢾⢾
⢁⢑⣣⢨⣏⡻⢂⡓⡍⢨⣛⣙⣘⠦⢞⢘⢛⠛⣔⡘⡛⠛⡙⡂
⠱⢗⢝⠰⣒⢦⠜⣫⡛⢨⣁⣥⠩⢷⣻⠺⢖⣋⣰⡞⡏⠙⢿⡅
nz = 250433
For the following, the median was 0, so the results were identical for both implementations:
julia> show_any_nonzero(my_sample_matrix; maxw = 32)
⢿⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠹⣿⣿⣶⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠘⢿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠻⣿⣿⣿⣿⣶⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠈⢻⣿⣿⣿⣿⣧⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠈⠿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣿⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣿⣿⣿⣿⣶⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢻⣿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣿⣿⣄⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠛⢿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣿⣿⣷⡀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⣿⣿⣆⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⢿⣷⡄⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠻⣦
nz = 137821
julia> show_any_nonzero(sparse([(z=(x-93)/61+(y-65)/74im;(1:60).|>_->z=z^2-.8+.156im;abs(z)<2) for y=1:130,x=1:184]); maxw = 24)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣼⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⢠⣴⣤⡿⣿⣦⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⢠⣼⣿⣶⡀⣴⣿⣿⣿⣿⣿⣿⣷⠀⠀⣄⠀⣤⠀⠀⠀⠀
⣤⣿⣾⡟⣿⣿⠣⣿⣥⠉⢻⣿⣿⣿⠏⢠⣾⣿⣿⣿⣦⢀⣀⠀
⠈⠹⠟⢹⣿⣿⣦⣭⣿⠀⣾⣿⣿⣿⠀⣿⣶⡆⣽⣿⣷⣿⣿⣦
⠀⠀⠀⠈⣻⡟⢻⠛⠁⢸⣿⣿⣿⣿⣷⣤⣾⠗⣿⣿⣿⡟⠛⠁
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢙⣿⣿⣛⢿⣿⣿⠃⠀⠈⠛⠃⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⣿⡇⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
nz = 4729
Performance:
julia> A = sprand(10000, 10000, 0.5);
julia> @time show_any_nonzero(A; maxw = 100);
...
nz = 50004593 0.005193 seconds (41 allocations: 93.609 KiB)
julia> @time show_above_median_nonzero(A; maxw = 100);
...
nz = 50004593 0.348106 seconds (43 allocations: 406.188 KiB)
When the sparse matrix is very large the trouble is that the entire spy plot output can tend to become non-zero.
I think this is fine. If you have an sprand matrix where the nonzeros are spread out equally over the whole matrix with sufficient density, the correct result is for the spy plot to show nonzeros everywhere. (A density plot would show the same thing.) This, combined with the density of nonzero entries (which should be printed in the summary line), is a reasonable summary of the sparsity structure.
The suggestion for showing only counts above the median is interesting, and I'm not completely opposed, but there is also the possibility that it might be more confusing to explain.
stevengj
on 14 Jan 2019
A potential problem with the graphical approaches: Poor Unicode support in Windows command prompt. Braille characters don't display with the default font (have to change to "MS Gothic"), and I wasn't able to get the half blocks to display with any font. It seems to work out of the box in Cygwin.
Would this rule out the graphical approaches?
Then again, built-in functions ∘ (function composition) and ∈ (in) among others also display as garbage in Windows command prompt.
maxbennedich
on 14 Jan 2019
Yet another reason to finally address #7267.
stevengj
on 15 Jan 2019
The suggestion for showing only counts above the median is interesting, and I'm not completely opposed, but there is also the possibility that it might be more confusing to explain.
Why not combine the two: Use braille for signaling any nonzeros and color shading (per braille character) for signaling nonzero density (median normed)?
@maxbennedich for me it seems to be often way faster to just loop over all nonzeros and count into buckets instead of the searchsortedfirst thing, since the number of nonzero rows for one column is rather small and often even less than the plot height. As long as the matrix is not very row-dense, this might not be that bad.
stev47
on 15 Jan 2019
Why not combine the two: Use braille for signaling any nonzeros and color shading (per braille character) for signaling nonzero density (median normed)?
That would be nice. ~However, one problem with this is that the Braille shading should not be a grayscale color, but rather a transparency alpha channel, since you don't know what the background color of the terminal is (it is user-dependent). Very few terminals (any?) allow control of the character alpha. In the half-block approach you control both background and foreground colors, so this is not an issue, as long as your terminal supports at least 8-bit colors.~
Scratch that. You can just override the background color of the Braille characters.
pablosanjose
on 15 Jan 2019
I like the coloring approaches, but am concerned it might look strange depending on the color scheme used in people's terminals.
One way to solve the density plot without using colors is to use sampling. Consider a matrix with random density increasing linearly from 0 to 1 towards the right. The "any nonzero" approach renders it like this:
julia> show_any_nonzero(sparse([rand() < c/10000 for r=1:1000, c=1:10000]); maxw=80)
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
nz = 4999242
The "above median" approach renders it like this:
julia> show_above_median_nonzero(sparse([rand() < c/10000 for r=1:1000, c=1:10000]); maxw=80)
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⢻⣽⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠠⢐⠿⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢰⣹⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠐⢘⣹⣺⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
nz = 5001619
A sample based approach (such as in this post above) looks better:
julia> show_sampled_nonzero(sparse([rand() < c/10000 for r=1:1000, c=1:10000]); maxw=80)
⠀⠀⠀⠀⢀⠀⠈⠀⡁⠀⠈⠈⡅⠤⠀⠠⢄⠁⠙⠄⠈⢰⡛⡀⢂⠰⠄⡤⢬⡀⠘⠈⣔⢌⡻⢇⣀⡥⠉⡗⢪⠵⡓⣯⣿⢠⢯⣣⣁⠁⣽⢏⣟⣝⡡⣧⣧⡯⠯⢿⣫⣿⢟⢾⣻⢿⣿⣿⣿⡿⣷⡿⣟⣿⣯⣿⣿⣿⣿⣿
⠀⠀⠀⠀⠐⢀⠀⠄⠐⠠⠀⣄⠕⠂⢀⡄⠀⢀⠁⠁⠃⠘⠖⠉⡑⠾⡀⣊⠋⠹⢢⣽⡑⢏⢀⣰⣄⣇⣏⡮⡋⢻⣛⢢⣄⣝⣖⡷⠮⡿⣣⣕⣯⣡⡙⡎⠥⣾⣟⡷⡻⢽⣿⢁⣖⣷⣷⣷⣿⣷⣽⣿⣿⣿⣟⣿⣿⣻⣿⣿
⠀⢀⠠⠀⠀⠌⠠⡂⠀⠀⠀⡐⠀⠐⡀⢁⠐⢒⠈⢩⠥⣋⠀⠨⢓⣅⠡⠂⠈⢋⠳⣣⠦⡂⡁⣋⠁⣠⣼⣲⡉⠐⢹⣨⣽⡲⠥⣣⢿⣉⣾⣋⢿⠯⣗⢓⡽⢾⢌⣹⠯⢺⢛⣿⡻⣸⣷⣿⣙⡿⣫⣿⣏⣽⣿⣾⣿⣿⣿⣿
⠀⠀⠀⠄⠀⠀⠄⢂⢠⠠⠀⠀⠀⢂⠀⠈⠪⠠⠄⢨⠀⢀⣅⠢⢥⠕⣀⠠⡬⢆⠳⡀⠘⠯⠁⠰⠷⣪⠳⣄⢭⢦⠆⢳⡯⢅⣘⣿⡹⣵⡽⣷⠽⣘⢳⢣⡾⠿⡹⣿⡹⢯⣬⢭⢛⣟⣻⣽⣷⣿⣾⣿⣯⣿⣿⣿⣹⣿⣿⣿
nz = 5000152
However, this approach might also be quite confusing since it could miss non-zeros. I also don't know if this makes a difference for real-world sparse matrices.
@maxbennedich for me it seems to be often way faster to just loop over all nonzeros and count into buckets instead of the searchsortedfirst thing
I think you're right for matrices with a small/medium amount of non-zeros, but for larger matrices this is not the case. For a 10k x 10k matrix with 50 million random non-zeros, my half-optimized code loops over and buckets all non-zeros in ~0.35 seconds on my system, or 20 CPU cycles per non-zero. IMO, this is not acceptable for a show implementation. For example, if you're working in Atom and using the Workspace view, _every single command you do in the REPL will call show on all variables_, and thus incur a 0.35 second complete freeze, with one CPU pegged at 100 %. Compare to the searchsortedfirst implementation (show_any_nonzero) which does the same thing in ~0.005 seconds (see benchmark above).
I updated the gist I linked above with a show_any_nonzero_loop_nz method that loops over all non-zeros; feel free to take a look and comment in case you had a better implementation in mind.
maxbennedich
on 15 Jan 2019
For a 10k x 10k matrix with 50 million random non-zeros [...]
To me that is a rather dense matrix. For row-dense matrices searchsortedfirst along the columns is indeed faster. If one wants to optimize printing a large dense SparseMatrix, one could get information about row-density through diff(S.colptr) and use the appropriate method.
stev47
on 16 Jan 2019
Ah, I see what you mean. Good point.
maxbennedich
on 16 Jan 2019
I wonder where we finally ended on this one. It would be nice to get something in. Or should these things go into UnicodePlots or elsewhere?
ViralBShah
on 20 Feb 2019
If there's interest in a Braille-based plot, I'm happy to volunteer some time to do a proper implementation. Before proceeding, I have these two doubts:
- Font may not support Braille characters, in particular Windows command prompt.
showmight become slow for large, dense matrices (example above: 10k x 10k matrix with 50 million entries could render in 0.35 second on a consumer laptop). A fix would be to use a sample-based approach, i.e. don't look at every nonzero, but sample the matrix at a bunch of indexes to create the plot. Faster, but less accurate. Or a hybrid between the two approaches.
maxbennedich
on 20 Feb 2019
Using sixels for plotting might also be an idea, see also e.g. lsix. Terminal support might vary, but at least it's font-independent.
stev47
on 29 Mar 2019
I think I'd prefer to go with @pablosanjose's approach where we print the non-zeros. That is less of a departure from what we've done and continues to show actual values in the matrix. I would also be in favor of having built-in spy functionality for matrices in general, but I think that can be added separately.
StefanKarpinski
on 29 Mar 2019
I still don't see @pablosanjose's approach as much of an improvement over the current output. spy-like output, on the other hand, would be a huge improvement.
stevengj
on 29 Mar 2019
Whatever the final decision, I think it should not rely on processing all non-zeros of the sparse matrix. That would likely bog down the display of really huge matrices that often arise in real world use. show should be snappy. Hence, I think the spy-plot should just do partial density sampling. Same for the gray-scale version using half-blocks.
pablosanjose
on 29 Mar 2019
This is a clear case of the perfect being the enemy of the good. Regardless of how much people may love spy plots for sparse matrices, we don't show any other kinds of data structures using that kind of visualization. No one is saying that we shouldn't have spy but people insisting on that being the default way to show sparse matrices has prevented any improvement from happening here.
StefanKarpinski
on 29 Mar 2019
That sounded a little bit too salty. My point is that having two options here is preventing any progress and printing sparse matrices as spy plots by default is controversial and unusual enough for it not to actually happen, leaving us where we are with no improvement at all.
StefanKarpinski
on 30 Mar 2019
Well, I do see @stevengj's argument: why is it better to show four matrix corners than just the beginning and end of first and last columns?
But I would point out that the alternative sparseshow style does have at least two advantages over the current show: it makes better use of screen real state (uses all the available REPL width, and can thus show "more" information), and can potentially also reuse column/row indices when more than one shown matrix element share them. Whether those advantages are actually worth the change is a matter of debate, I guess.
pablosanjose
on 1 Apr 2019
Do we have a PR for the spy like printing?
ViralBShah
on 31 Oct 2019
Yes we have #33821. Everybody is kindly invited to review.
dkarrasch
on 15 Nov 2019
Related issues
dpsanders
·
3Comments
musm
·
3Comments
i-apellaniz
·
3Comments
tkoolen
·
3Comments
sbromberger
·
3Comments
Most helpful comment
Braille spy plot PoC:
Result: