IPython Public API

This page will contain information about what we consider to be the public IPython API, how to best expose it, etc. Historically ipython hase exposed many extension points, but not necessarily in the most organized manner. Recent changes to the stable branch have improved the situation, but as we refactor the core, we should really organize this as cleanly as possible. This page (or group of pages, if it grows) should serve as a central reference for all this material, both while developing and in the future for ipython extension writers.

Note: in following with Enthought's recent code guidelines, we will implement an api.py file at the top-level of the new ipython package, listing explicitly all of the extension points. This will probably be based on today's ipapi.py.

Here I will keep listed various cases submitted by users of things they actually have needed in real-world use of ipython. Making sure that our future API satisfies the needs of those who so far have actually used ipython in their projects is the best way to ensure we develop a practical tool.

The PyRAF shell

Todd Miller, from the StSCI, submitted the following notes from his experience (the attached file ipython_api.py below contains his code to implement this:

I added support for IPython to our analysis package, PyRAF, as we discussed in March or April just before you left on (extended) travel. I wanted to feed back some of the features of IPython that I used as input to any future API created for IPython.

I used these features of IPython, some of which are probably considered "private":

1. I found it necessary to hook init_realine due to order of evaluation issues:

InteractiveShell.init_readline
InteractiveShell.init_readline = pyraf_init_readline

2. I used ipapi.get() to get and API object and the (ab)used it by pulling out the InteractiveShell?:

_ipython_api = IPython.ipapi.get()
IP = _ipython_api.IP

3. I used lsmagic() to get a list of magic functions:

IP.lsmagic()

4. I used expose_magic to define my own magic functions for PyRAF:

_ipython_api.expose_magic("set_pyraf_magic", set_pyraf_magic)

5. I assigned my own prefilter to InteractiveShell? as a way of passing a hidden parameter via bound method:

InteractiveShell.prefilter = foo_filter

I later called InteractiveShell._prefilter() to continue filtering in IPython:

InteractiveShell._prefilter(IP, line, continuation)

6. I accessed IPython's AutoFormattedTB class to handle PyRAF tracebacks.

IPython.ultraTB.AutoFormattedTB

I set a custom exception handler using the InterativeShell.set_custom_exc():

IP.set_custom_exc((Exception,),  showtraceback)

I clear the custom exception chain like this to remove my PyRAF exception handler:

IP.custom_exceptions = ((), None)

7. I used a custom readline completer like this:

IP.set_custom_completer(completer)

I accessed InteractiveShell.matchers directly to remove my completer:

IP.Completer.matchers

The PyMAD shell

The !PyMAD project, by Fréderic Mantegazza, is an IPython-based console for remote instrument control. For this project, Fréderic required a number of extensions which were added to the core of IPython. This page? lists some of these more sophisticated extensions, which can be useful for others using IPython as a base component of a custom interactive environment.

In all the following, ipshell is a IPShellEmbed instance.

Catch custom exceptions

The solution is to use the new set_custom_exc() method of IP bject (ie ipshell.IP object):

def pymadHandler(self, exceptionType, exceptionValue, traceback):
    """ This is a special handler to handle custom exceptions in IPython.
    """

    # Handle PyMAD exceptions
    if issubclass(exceptionType, PyMADError):

        # We just print the exception on the PyMAD message stream
        MessageLogger().error(exceptionValue.standardStr())

    # Handle Pyro exceptions
    elif issubclass(exceptionType, Pyro.errors.ConnectionClosedError):
        MessageLogger().critical("Connexion with server lost. Please restart it...")
        if self.code_to_run_src.find('.') != -1:
            code = self.code_to_run_src.split(".")
            obj = eval(code[0], self.user_ns)

            # We rebind the object to the remote server...
            obj.adapter.rebindURI()
            MessageLogger().warning("Server found again. Running last command...")

            # ...and try to execute the previous code
            eval(self.code_to_run_src, self.user_ns)
        else:
            MessageLogger().warning("Could not automatically rebind to server. Better restart console")

    # Handle AttributeError exception
    elif exceptionType is AttributeError:
        if self.code_to_run_src.find('.') != -1:
            code = self.code_to_run_src.split(".")
            obj = eval(code[0], self.user_ns)
            if isinstance(obj, Pyro.core.DynamicProxyWithAttrs):
                MessageLogger().error("%s has no attribute %s" % (code[0], code[1]))
            else:
                self.showtraceback()
        else:
            self.showtraceback()

    # Others
    else:
        self.showtraceback()
        print "\n\n"
        print "*** Unknown exception ***"
        print "Exception type :", exceptionType
        print "Exception value:", exceptionValue
        print "Traceback      :", traceback
        print "Source code    :", self.code_to_run_src

ipshell.IP.set_custom_exc((PyMADError, Pyro.errors.ConnectionClosedError, AttributeError), pymadHandler)

Custom matchers for completion

As PyMAD uses remote objects, completion only shows the client Pyro proxy. So we create a new matcher which get the object (from the text param), call a special method on this object which returns all available attributes (in fact, only these we want to show to the user). Here is the code used:

def proxy_matches(self, text, state):
    """ Get the attributes of a remove Pyro object.
    """

    # Another option, seems to work great. Catches things like ''.<tab>
    m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)

    matches = []

    if m:
        expr, attr = m.group(1, 3)
        try:
            obj = eval(expr, self.namespace)
            if isinstance(obj, Pyro.core.DynamicProxyWithAttrs):
                words = obj.getAvailableAttributes()
                n = len(attr)
                if words:
                    for word in words:
                        if word[:n] == attr and word != "__builtins__":
                            matches.append("%s.%s" % (expr, word))

        except NameError:
            pass

        except Pyro.errors.ConnectionClosedError:
            MessageLogger().critical("Connexion with server lost.")

    return matches

ipshell.IP.set_custom_completer(proxy_matches)
ipshell.IP.Completer.merge_completions = False

Documentation handlers

In the same way as matchers, get the docstring from the remote object instead of the client one when using obj? and obj.method? syntax. This can be done by adding a getdoc() method to the object. But it only works with the first syntax. New exception handlers

Here, the idea is to be able to present different kind of exceptions in different ways. Some will only print a simple message, some others will print the entire traceback (maybe a modified traceback). This is done in the point 1. Prevent objects from being deleted by del keyword

This can be done with pre-filters, but I didn't tried yet. Dynamic Prompt

Works fine using the PEP 215 syntax. Multi-lines prompt are supported:

argv = ["-pi1",
        "Ki=${globals()['Spectro'].Ki} Kf=${globals()['Spectro'].Kf}\n\
        mode=${globals()['Spectro'].driveMode} flipper=${globals()['Spectro'].flipperBeam}\n\
        PyMAD@${os.path.expandvars('$PYRO_NS_HOSTNAME')}>>> ",
        "-po",
        " ",
        "-profile",
        "pymad"]

Command-line interpreter

The idea is to have IPython interprets code has if it was entered through keyboard (ie make difference between magic commands and normal python code). Can be done with the ipshell.IP.runlines(). Here is a magic do:

def magic_do(self, parameter_s=''):
    """ do magic command.

    This magic takes a job file as parameter.
    The job file can contain both magics and python syntax. The only
    restriction in this last case is to add a blank line after the end
    of a block. For example:

    rd a6
    for pos in (10., 20., 30.):
        dr a6 $pos

    rd a6
    """
    myParser = ILLParserTPG()
    cmd = 'do ' + parameter_s
    try:
        command = myParser(cmd)
        fileName = command['fileName']
        file_ = PrivoxyWindowOpen(fileName, "r")
        lines = file_.read()
        Logger().debug("%%do_ %s\n%s" % (fileName, lines))
        self.shell.runlines(lines)

    except SyntacticError:
        MessageLogger().error(sys.exc_info()[1])

InteractiveShell.magic_do = magic_do
    del magic_do

Note that you must have the var multi_line_specials set to 1 in you ~/ipythonrc config file.

Other hooks

List here other extension points...

• Compiled code transformations. This will let users do things like the following (trick by A. Martelli on c.l.py).

Daniel <4daniel@gmail.com> wrote:

...

Ideally I'd like to have a way to tell the interpreter to use Decimal

by default instead of float (but only in the eval() calls). I understand the performance implications and they are of no concern. I'm also willing to define a single global Decimal context for the expressions (not sure if that matters or not). Is there a way to do what I want without rolling my own parser and/or interpreter? Is there some other alternative that would solve my problem?

Says Alex:

What about:

c = compile(thestring, thestring, '<eval>')

cc = new.code( ...all args from c's attributes, except the 5th
                          one, constants, which should instead be:
                          decimalize(c.co_consts)...)

i.e.

cc = new.code(c.co_argcount, c.co_nlocals, c.co_stacksize, c.co_flags,
               c.co_code, decimalize(c.co_consts), c.co_names,
               c.co_varnames, c.co_filename, c.co_name,
               c.co_firstlineno, c.co_lnotab)

where

def decimalize(tuple_of_consts):
  return tuple( maydec(c) for c in tuple_of_consts )

and

def maydec(c):
  if isinstance(c, float): c = decimal.Decimal(str(c))
  return c