The IPython Kernel

Date: 11/28/2005

Questions

Do we use a CR-LF terminated protocol or a length prefixed one?

Shall the protocol be based on a plain text syntax or python dictionaries?

Do we want to be able to telnet to kernels?

What are the design questions about sending custom and large objects?

Security problems with pickle?

Is it worth look at PB for some things?

Do we want the kernel to be customizable by the user? Yes

The translator and completer The type of kernel

Do we want IPython to need to be installed on each client machine? No

Overview

The purpose of this page is to document the current state of the IPython kernel and the related refactoring of IPython itself. The design is evolving rapidly and hopefully this page can be updated along with that evolution.

A Brief History

Fernando and I have gone through many stages of thinking about the idea of an IPython kernel. The history of our thinking gives relevant backgound to the current design of the kernel. Here is a brief history:

1. We first thought about having a 3 layer design with i) a frontend that handles all the UI, ii) the kernel interface layer and iii) the kernel. Our thought was that i) and ii) would run in one process and talk to iii) over the network. In this model we imagined that there could be multiple kernels running in different processes and even on different hosts. But, we assumed that the frontend/kernel interface layer could be repointed to any kernel. That is, the kernel's all appeared to be the same to the frontend. We gave this design up because users want to be able to execute GUI code in the kernel. At the time, we thought that the users GUI code must exist in the frontend (which has access to the GUI). So we next came up with...

2. Have a single process that has i) all the frontend UI code and ii) the "master" IPython kernel. This way (so we thought), the master kernel could run GUI code. There would still be computational kernels, but these would be slaves to the master kernel. But again, this design had a major problem. Because the notebook frontend is being written using wx, the user could only execute wx code in the master kernel. Thus, users could not make calls to any other GUI toolkit (Cocoa, tk, gtk, qt, etc.) from the wx frontend. Because we don't want to write multiple frontends to support each GUI toolkit, we had to do something else, which brings us to where we are today...

Current Design Overview

IPython will have a number of different layers:

1. The frontend, which will handle the UI and talk to the kernel interface layer. IPython will probably come with two frontends: the wx based notebook and a terminal based one.

2. The kernel interface layer (KIL). This layer will expose an interface for talking to the kernel over the network. This layer will exist in the same process as the frontend and will use Twisted and nonblocking sockets to talk to the kernel.

3. The kernel. The IPython kernel will be a fully functioning IPython instance that takes interactive commands over the network rather than from the command line. The kernel itself will consist of two processes running on the same machine, one public and one private. This is done to overcome the limitations of Python's global interpreter lock. See more details below.

Here is more detailed information about the different layers.

The Frontend

There will be a number of different frontends ranging from a simple terminal based one to a fully featured wx-based notebook one.

The Kernel Inteface Layer (KIL)

The KIL will use the defined network protocol to talk to the kernel over the network. It will provide a simple interface that the frontend can use to talk to the kernel. The KIL should use non-blocking sockets to allow for complete flexibility in controlling the kernel.

The IPython Kernel

The Kernel's Public Process

The kernel's public process will communicate with two other processes. First it will listen on a public port for incoming commands from the kernel interface layer. It will then place these commands in a work queue. It will also communicate with the kernel's private process in a blocking manner to submit things from the queue and gather the results. The overall picture is that the private process will do the computations serially and the public process will do everything else.

The Kernel's Private Process

The private process will listen on a private (local only) port for commands from the kernel's public process. It receives a command, does the work and then replies, all while blocking.

Most importantly, there will be different types of private process's cores to support users running GUI code. When the user starts the kernel, they will pick what type of private kernel process they want to use. There will be computational kernels that are designed for compute intensive tasks but that do not support calls to GUI toolkits. There will also be a set of GUI kernels that allow the user to call various GUI toolkits from within the kernel. But these GUI kernels will lack features that the computational kernel has. These GUI kernel's will implement the features that are currently found in Shell.py, but in a much cleaner way.

As we have implemented a basic kernel, we have realized that there is a deep and significant concurrency problem when running extension code that does not release the GIL. Basically the kernel completely dies. This is unacceptable for real world use. To solve this problem Fernando had the idea of having a two process model. I have toyed with this before (and rejected it), but I am reconsidering it. Here are some design ideas.

The kernel would consist of two parts: a non-blocking public process and a blocking private process.

The private process would be a Twisted server that takes command only from the local host and processes them. It blocks until each command is complete and then sends back the results to the caller. The process that calls this private blocking kernel should expect the private process to "go silent" while it is working. The only problem with this is how fatal errors could be distinguished from long running (blocking) calculations. This needs to be addressed.

The public process would provide a non-blocking interface to the private process. The public process would take commands and do one of two things with them. First the command could be placed in a queue. The queue would need to be written using Twisted's event loop and would reside in the public process. The public process would then watch this queue and send commands to the private process when they are ready. Second, the command might not need to be queued. In this case, the public process could try to handle the command immediately. If the command required contacting the private process, there should be a method for trying to connect and then properly handling failures if the private process is busy.

Other things that should be handled by the public process:

a. Sending results back to the result gatherers. This is ideal as the private computational process will not have to spend time doing this.

b. Forwarding. Sometimes commands need to be forwarded to other kernels. The public process could do this w/o any cost to the computational process.

Another benefit to this model. There will be absolutely no threads used in either process!!!!! This should dramatically improve the latency and make certain things _much_ easier to design. This is a _huge_ improvement in my opinion.

I could also see a nice security model emerging from this. Basically, the private process would be blocked to all parties except the public one. All the security stuff could be put into the public process. We could even have different public processes that use different security models.

Also, different GUI kernels could be implemented in a simple manner. The GUI specific stuff would exist in the private process. Thus the GUI kernels could still have a queue (which they don't now) as the queue is in the public process. The GUI would still be blocked when non-GIL-releasing extension code is running, but there is _no_ way around that problem as I see it.

Older Notes

Notes from a conversation (fperez/rkern, 9/8/05):

We want an interface which allows the user to easily have different sheets in a GUI which appear to have clean namespaces (imagine writing chapters in a book). There are two ways to do this: either each group of sheets sharing state talks to a different kernel in a different process, or they all talk to the same kernel, but each kernel maintains a list of user namespaces to execute in.

The second approach would require perhaps a bit more refactoring, since elements which are currenly kernel-global would have to be managed as lists, one per namespace. Bascially the kernel would have to in fact have internally 'mini-kernels' holding prompt counters, user namespace, etc. If we simply allow the opening of a new kernel as needed (which we'll likely need to do anyway, if we want to allow a single front end combining code from different GUI toolkits), it may be easier just to use a multi-process approach. I'm leaning in this direction...

Reply by bgranger 9/22/05:

If you want each sheet to have a clean namespace, you also probably want each sheet to talk to a different kernel. The reason is that if one sheet has a long running command, the other sheets will be locked up if they all share a single kernel. Thus I also think the multi-process/multi-kernel approach is better for this . . . and easier to implement.

A note on mmap: The python mmap module provides support for memory-mapped files across both *nix and Windows (here is a trivial little example). I could see the two-process model using sockets to send commands back and forth, while mmap is used to share the big data chunks (anything that can be pickled can be written to the mmap'd file) with less overhead. Just some thoughts...

A possibly useful reference for Qt threads:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/415311