cowlicks (Posts about gsoc)

Wrapping Up Google Summer of Code

Blake Griffith — Mon, 23 Sep 2013 21:52:13 GMT

My Summer has been over for a while, but GSoC is officially ending this week. And what do I have to show?

SciPy's sparse matrices now support boolean data.
Sparse matrices now support boolean operations (!=, <=, etc.).
NumPy universal functions (ufuncs) can now be overridden for compatibility.
Sparse matrices uses this ufunc override functionality to do sensible things when acted on by ufuncs.
Support fancy indexing.
Axis arguments for min and max methods.
various other things: speed ups, small features, test coverage, bugs fixes, code clean ups, and a few backports.

But I think that I have benefited personally from this experience much more than SciPy has. I learned so much this summer, a short list of things I know vastly more about than I did before this summer:

PYTHON. Jeez looking at code I wrote before this summer is painful.
The Python and NumPy C API, before this summer I was a total C noob. Now I can scratch together a python API in C.
SciPy.
NumPy, numpy.frompyfunc is awesome.
Git, before this summer I could commit thing to my own repos and that was pretty much it. Git hooks are awesome.
In general, development workflow.

I grown attached to SciPy and am excited I can make meaningful contributions to that are actually helping people. But contributing more will have to wait untill after the GRE, grad school applications, and my senior thesis. There are a few things I would like to add down the road to incorporate into SciPy 1.0. e.g. 64bit indices, change spares matrices to emulate numpy.array rather than numpy.matrix. I would also like to see if I can apply ufunc overrides in a few other places that mix numpy.array and scipy.sparse like scikit-learn.

Here is a link to git log -p upstream/master --author="Blake Griffith" in numpy and scipy.

week 4 - Sparse Matrix Inequalities

Blake Griffith — Fri, 21 Jun 2013 10:32:39 GMT

This week I worked on adding support for inequalities to sparse matrices. The pull request is here (still pending). I also modified, the c++ routines used to produce a boolean output, pull request here. This is so these inequalities don't have to be cast to bool in python. Once both of these are accepted the inequalities will be using the c++ routines that produce boolean output.

These are separate pull requests but they are related and one will need to be rebased once the other is accepted. Or I may combine them.

Each of the four basic inequalities has its own particular quirks which make it more or less efficient with sparse matrices. Here we consider A to be a sparse matrix. The cases where B is a scalar, or sparse are considered. (By scalar we effectively mean a matrix the same size as A with every element as the scalar value B.

`A < B`

For scalar B, this operation is only efficient if B < 0 otherwise the resulting matrix will be dense.

For sparse A and B this is also efficient.

`A > B`

For scalar B the efficiency is the opposite of the less than operator. That is B > 0 has efficient output. Similarly to < this operation is efficient with other sparse matrices.

`A <= B`

This operation is pretty much useless. The only case which make this efficient is when B < 0. But this case is already covered by <, and < can be used efficiently with sparse matrices. So why use <=? I'll discuss the pros and cons of having these non-strict inequalities in a moment.

Another disadvantage of non strict inequalities like these is that they raies NotImplementedError when comparing with 0. This is because of the nature of the c++ routines. When comparing every pair of elements they don't consider cases where both are zero.

`A >= B`

Like <= this operation is only efficient for one case, when B > 0. And also like <=, it is not very useful.

Why have non strict inequalities?

I can't really think of uses for >= and <= but they might exist, I added them because they were easy to implement once I had done > and <. In practice, they don't slow the usage down. But removing non - strict inequalities removes 24,990 lines of code.

Implementation

The inequality operations are implemented as c++ routines, these are wrapped to provide a python interface using SWIG, then the various inequalities are added appropriately.

C++

The inequalities routines were implemented by reusing existing routines like csr_binop_csr. Overrides for inequalities had to be added to complex_wrapper.h too.

The binop routines had to altered, another class was added to their templates, T2, for the type of data out. In theory this could be something other than bool.

SWIG

To handle boolean output data a new swig macro was added to create typemaps that have npy_bool_wrapper as this output class. I then use these to instantiate the boolean operations.

Python

Here the associated python special methods corresponding to the inequalities were added in compressed.py. These operations can be used with a scalar, dense, or sparse matrix. Numpy style broadcasting is not implemented.

To produce a bool output by default for boolean comparisons, I altered the _binopt function to pass the c++ routines a matrix to use for output with a bool dtype.

Problems

When testing the inequalities with dense matrices, I could not get the proper behavior with a dense matrix on the left hand side. Previously, with != and == I modified the __bool__ method but that did not work in this case. This is because of the same problem I will be dealing with in the next stage of my proposal, interactions with numpy ufuncs.

Week 1 - bool dtype support

Blake Griffith — Sun, 02 Jun 2013 06:05:49 GMT

I got accepted to Google Summer of Code! Thank you everyone who has helped me get this far, and Amber for making me a congratulatory google cake!

On my proposal timeline I gave myself 2 weeks to add support to sparse matrices for dtype=bool. So far, dtype=bool works implementation was not too hard, and most of my time was spent figuring out how SWIG works. However there is still a lot of work to do in testing bool support. This will be more time consuming than it should be because the sparse test suite is messy.

bool dtype implementation

sparsetools

The sparsetools/ directory contains several *.cxx, *.h, *.i, and *.py files. The header files (*.h) contain C++ routines these are wrapped by SWIG according to instructions in the *.i interface files. SWIG uses the header and interface files to generate python *.py files C++ code in *.cxx files. Once these *.cxx files are compiled and linked the functions they define can be called by the generated *.py files.

Where do I add bool support?

Contained in sparsetools/ is a file called complex_ops.h which translates numpy's complex types to C++ classes. Well, that is what I want to do, except for numpy's bool types instead! My new file bool_ops.h is much simpler though. It basically does

typedef npy_int8 npy_bool_wrapper;

Yep that's it, (for now anyway). This file basically says to treat the npy_bool_wrapper type like the npy_int8. Because boolean algebra is almost integer algebra.

npy_bool_wrapper is used in numpy.i to define a typemap from numpy's npy_bool type to our npy_bool_wrapper.

In sparsetools.i we have to declare the new data type and add it to the INSTANTIATE_ALL macro which is used in all the *.i files to call the functions we want from the *.h files.

What else?

Adding bool to supported_dtypes in sputils.py prevents it from upcast. Then tests need to be added, but this isn't too easy. The test suite does not have a way to take some general data or dtype. So I am going through adding tests one-by-one where it seems appropriate. I'm also trying to add tests for other dtypes because there don't seem to be any tests for anything other than float64 and int64.

While I was doing this I discovered a few bugs relating to how sparse handles uint dtypes.

Once I'm done adding tests and fixing what ever does not pass, I'll move on to the next stage in my proposal. Adding support for boolean operations.