Changes between Version 6 and Version 7 of MaskedArrayAlternative

Timestamp:

02/10/08 09:23:51 (5 years ago)

Author:

jarrod.millman

Comment:

--

MaskedArrayAlternative

@@ -1 @@
-= An alternative implementation of MaskedArray =
-
-'''Note: the new implementation of MaskedArray is now available in the scipy sandbox. '''
-
-== History ==
-
-As a regular user of MaskedArray, I (Pierre G.F. Gerard-Marchant) became increasingly frustrated with the subclassing of masked arrays (even if I can only blame my inexperience). I needed to develop a class of arrays that could store some additional information along with numerical values, while keeping the possibility for missing data (picture storing a series of dates along with measurements, what would later become the {{{TimeSeries}}} package). 
-
-I started to implement such a class, but then quickly realized that any additional information disappeared when processing these subarrays (for example, adding a constant value to a subarray would erase its dates). I ended up writing the equivalent of {{{numpy.core.ma}}} for my particular class, ufuncs included. Everything went fine until I needed to subclass my new class, when more problems showed up: some attributes of the new subclass were lost during processing. I identified the culprit as {{{MaskedArray}}}, which returns masked ndarrays when I expected masked arrays of my class. I was preparing myself to rewrite {{{numpy.core.ma}}} when I forced myself to learn how to subclass ndarrays. As I became more familiar with the {{{__new__}}} and {{{__array_finalize__}}} methods, I started to wonder why masked arrays were objects, and not ndarrays, and whether it wouldn't be more convenient for subclassing if they did behave like regular ndarrays.
-
-The new {{{maskedarray}}} is what I eventually come up with. The main differences with the initial {{{numpy.core.ma}}} package are that {{{MaskedArray}}} is now a subclass of {{{ndarray}}} and that the {{{_data}}} section can now be any subclass of {{{ndarray}}}. Apart from a couple of issues listed below, the behavior of the new {{{MaskedArray}}} class reproduces the old one. Initially the {{{maskedarray}}} implementation was marginally slower than {{{numpy.ma}}} in some areas, but work is underway to speed it up; the expectation is that it can be made substantially faster than the present {{{numpy.ma}}}.
-
-
-Note that if the subclass has some special methods and attributes, they are not propagated to the masked version: this would require a modification of the {{{__getattribute__}}} method (first trying {{{ndarray.__getattribute__}}}, then trying {{{self._data.__getattribute__}}} if an exception is raised in the first place), which really slows things down. 
-
-== Main differences ==
-  * The {{{_data}}} part of the masked array can be any subclass of ndarray (but not recarray, cf below).
-  * {{{fill_value}}} is now a property, not a function.
-  * in the majority of cases, the mask is forced to {{{nomask}}} when no value is actually masked. A notable exception is when a masked array (with no masked values) has just been unpickled.
-  * I got rid of the {{{share_mask}}} flag, I never understood its purpose. 
-  * {{{put}}}, {{{putmask}}} and {{{take}}} now mimic the ndarray methods, to avoid unpleasant surprises. Moreover, {{{put}}} and {{{putmask}}} both update the mask when needed.
-  * if {{{a}}} is a masked array, {{{bool(a)}}} raises a {{{ValueError}}}, as it does with ndarrays.
-  * in the same way, the comparison of two masked arrays is a masked array, not a boolean
-  * {{{filled(a)}}} returns an array of the same subclass as {{{a._data}}}, and no test is performed on whether it is contiguous or not.
-  * the mask is always printed, even if it's {{{nomask}}}, which makes things easier (for me at least) to remember that a masked array is used.
-  * {{{cumsum}}} works as if the {{{_data}}} array was filled with 0. The mask is preserved, but not updated.
-  * {{{cumprod}}} works as if the {{{_data}}} array was filled with 1. The mask is preserved, but not updated.
-
-== New features ==
-  This list is non-exhaustive...
-  * the {{{mr_}}} function mimics {{{r_}}} for masked arrays.
-  * the {{{anom}}} method returns the anomalies (deviations from the average)
-  * the {{{stdu}}} and {{{varu}}} return unbiased estimates of the standard deviation and variance, respectively.
-
-== Using the new package with numpy.core.ma ==
-  I tried to make sure that the new package can understand old masked arrays. Unfortunately, there's no upward compatibility.
-For example:
-{{{
->>> import numpy.core.ma as old_ma
->>> import maskedarray as new_ma
->>> x = old_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
->>> x
-array(data =
- [     1      2 999999      4      5],
-      mask =
- [False False True False False],
-      fill_value=999999)
->>> y = new_ma.array([1,2,3,4,5], mask=[0,0,1,0,0])
->>> y
-array(data = [1 2 -- 4 5],
-      mask = [False False True False False],
-      fill_value=999999)
->>> x==y
-array(data =
- [True True True True True],
-      mask =
- [False False True False False],
-      fill_value=?)
->>> old_ma.getmask(x) == new_ma.getmask(x)
-array([True, True, True, True, True], dtype=bool)
->>> old_ma.getmask(y) == new_ma.getmask(y)
-array([True, True, False, True, True], dtype=bool)
->>> old_ma.getmask(y)
-False
-}}}
-  
-== Using maskedarray with matplotlib ==
-By default matplotlib still uses numpy.ma, but there is an rcParams setting that you can use to select maskedarray instead.  In the matplotlibrc file you will find:
-
-{{{
-#maskedarray : False       # True to use external maskedarray module
-                           # instead of numpy.ma; this is a temporary
-                           # setting for testing maskedarray.
-}}}
-
-Uncomment and set to True to select maskedarray everywhere.  Alternatively, you can test a script with maskedarray by using a command-line option, e.g.:
-
-{{{
-python simple_plot.py --maskedarray
-}}}
-
-== Masked records ==
-  Like {{{numpy.core.ma}}}, the {{{ndarray}}}-based implementation of {{{MaskedArray}}} is limited when working with records: you can mask any record of the array, but not a field in a record. If you need this feature, you may want to give the {{{mrecords}}} package a try (available in the {{{maskedarray}}} directory in the scipy sandbox). This module defines a new class, {{{MaskedRecord}}}. An instance of this class accepts a {{{recarray}}} as data, and uses two masks: the {{{fieldmask}}} has as many entries as records in the array, each entry with the same fields as a record, but of boolean types: they indicate whether the field is masked or not; a record entry is flagged as masked in the {{{mask}}} array if all the fields are masked. A few examples in the file should give you an idea of what can be done. Note that {{{mrecords}}} is still experimental...
-
-== Optimizing maskedarray ==
-
-=== Should masked arrays be filled before processing or not ? ===
-  In the current implementation, most operations on masked arrays involve the following steps:
- * the input arrays are filled
- * the operation is performed on the filled arrays
- * the mask is set for the results, from the combination of the input masks and the mask corresponding to the domain of the operation.
-
-For example, consider the division of two masked arrays:
-{{{
-#!python
-import numpy
-import maskedarray as ma
-x = ma.array([1,2,3,4],mask=[1,0,0,0], dtype=numpy.float_)
-y = ma.array([-1,0,1,2], mask=[0,0,0,1], dtype=numpy.float_)
-}}}
-
-The division of x by y is then computed as
-{{{
-#!python
-d1 = x.filled(0) # d1 = array([0., 2., 3., 4.])
-d2 = y.filled(1) # array([-1.,  0.,  1.,  1.])
-m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m = array([True,False,False,True])
-dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
-result = (d1/d2).view(MaskedArray) # masked_array([-0. inf, 3., 4.])
-result._mask = logical_or(m, dm)
-}}}
-
-Note that a division by zero takes place. To avoid it, we can consider to fill the input arrays, taking the domain mask into account, so that:
-{{{
-#!python
-d1 = x._data.copy() # d1 = array([1., 2., 3., 4.])
-d2 = y._data.copy() # array([-1.,  0.,  1.,  2.])
-dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
-numpy.putmask(d2, dm, 1) # d2 = array([-1.,  1.,  1.,  2.])
-m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m = array([True,False,False,True])
-result = (d1/d2).view(MaskedArray) # masked_array([-1. 0., 3., 2.])
-result._mask = logical_or(m, dm)
-}}}
-Note that the {{{.copy()}}} is required to avoid updating the inputs with {{{putmask}}}.
-The {{{.filled()}}} method also involves a {{{.copy()}}}.
-
-A third possibility consists in avoid filling the arrays:
-{{{
-#!python
-d1 = x._data # d1 = array([1., 2., 3., 4.])
-d2 = y._data # array([-1.,  0.,  1.,  2.])
-dm = ma.divide.domain(d1,d2) # array([False,  True, False, False])
-m = ma.mask_or(ma.getmask(x), ma.getmask(y)) # m = array([True,False,False,True])
-result = (d1/d2).view(MaskedArray) # masked_array([-1. inf, 3., 2.])
-result._mask = logical_or(m, dm)
-}}}
-Note that here again the division by zero takes place.
-
-A quick benchmark gives the following results:
-  * {{{numpy.ma.divide}}}  : 2.69 ms per loop
-  * classical division     : 2.21 ms per loop
-  * division w/ prefilling : 2.34 ms per loop
-  * division w/o filling   : 1.55 ms per loop
-
-So, is it worth filling the arrays beforehand ? Yes, if we are interested in avoiding floating-point exceptions that may fill the result with infs and nans. No, if we are only interested into speed...
-
-
-
-
-== Thanks ==
-  I'd like to thank Paul Dubois, Travis Oliphant and Sasha for the original masked array package: without you, I would never have started that (it might be argued that I shouldn't have anyway, but that's another story...).
-  I also wish to extend these thanks to Reggie Dugard and Eric Firing for their suggestions and numerous improvements.
-
-
-=== Revision notes ===
-  * 08/25/2007 : Creation of this page
-  * 01/23/2007 : The package has been moved to the SciPy sandbox, and is regularly updated: please check out your SVN version!
-
-
-
+[[ReST(/trunk/numpy/ma/README.txt)]]