Some issues for INS to consider in planning the work to support a transition to pysynphot:

Suggested work items

What:	Available to be worked when:
Resolve existing discrepancies with synphot	Now
Categorize ETC RT test cases according to known synphot accuracy	Now
Fill out ETC RT test cases for better coverage (and categorize as above)	Now
Resolve discrepancies with ETC RTs using pysynphot	As soon as they are run (shortly after 17.2 release)
Resolve discrepances with synphot RTs, once translated	After they are translated (??)
Define test cases for remaining functionality	Now
Resolve discrepancies with test cases	As development proceeds

Why do we expect differences between synphot and pysynphot?

Pysynphot is not simply a translation of synphot from SPP to Python. It is a re-implementation, taking advantage of modern language constructs to permit flexibility that did not exist in SPP.

• Pysynphot is double precision; synphot is single precision
• Differences in how and when wavelengths are sampled, interpolated, rebinned, and combined
• Synphot has no equivalent of pysynphot's analytic spectra
• Possibly different implementations of:
  • integration
  • interpolation
  • extrapolation
• Different order of operations

It's known that synphot does not always produce the "right" answer because of precision issues and poor handling of sampling around very narrow lines. It's also possible that synphot has more long-undetected bugs than the one we found last spring in efflam.

=> synphot can be considered brass standard, but not gold standard

Proposal:

• For imaging and broadband filters: use synphot as sanity check, then INS resolves discrepancies & (hopefully) approves new answers
• For spectroscopy, narrow lines, high redshifts: start from first principles when producing test cases
• Limit the effort spent in understanding the differences between synphot and pysynphot
• Spend time upfront collaborating to identify a limited, prioritized set of test cases that will probe the parameter space of (spectrum, obsmode, units, functionality) for each build
  • then fill out the parameter space for more test coverage as confidence is established

Pysynphot tests and test cases

Pysynphot tests are organized slightly differently than either synphot or the ETC.

• For both synphot and the ETC, a single "test case" specifies multiple input parameters (synphot: task name and task parameters; ETC: user inputs on the form) and multiple outputs (synphot: output files created as well as multiple output values calculated; ETC: multiple output values calculated). In order for a single "test" to pass, all the significant output products associated with the test case must pass. ("Significant" is defined by the test creator, and may vary from test to test.)

• For pysynphot, the tests are more finely grained, and there is a difference between a "test case" and a "test". A "test case" specifies, for example, a spectrum and an obsmode. There may then be multiple "tests" associated with each test case; each test compares a single calculated value (scalar or array) to the reference value for that test case. Thus, the test cases cover the parameter space of input parameters, and the tests cover functionality. For a given test case, some tests may pass while others fail.