Changeset 3060

Show
Ignore:
Timestamp:
02/19/08 18:09:34 (10 months ago)
Author:
bgranger
Message:

Basic work on updating the documentation.

Files:

Legend:

Unmodified
Added
Removed
Modified
Copied
Moved

  • ipython1/branches/ipython1-client-r3021/INSTALL

    r2820 r3060  
    11=========================== 
    2 IPython Kernel Installation 
     2IPython1 Installation 
    33=========================== 
    4  
    5 :Author:  Brian E. Granger 
    6 :Contact: ellisonbg@gmail.com 
    7 :Date: September 20, 2007 
    84 
    95.. contents:: 
    …… 
    128============ 
    139 
    14 IPython1 enables parallel applications to be developed in Python.  This INSTALL file describes the steps required to install ipython1.  For a description of ipython1, see the README file in this directory. 
     10IPython1 enables parallel applications to be developed in Python. This INSTALL 
     11file describes the steps required to install ipython1. For a description of 
     12ipython1, see the README file in this directory. 
    1513 
    1614Please let us know if you have problems installing ipython1 or any of its 
    17 dependencies. Currently we have tested ipython1 extensively with Python 2.4 
    18 and 2.5. It will not work on Python 2.3. 
     15dependencies. Currently we have tested ipython1 extensively with Python 2.4 and 
     162.5. It will not work on Python 2.3. 
    1917 
    2018IPython1 kernel requires four things: 
    2119 
    22 1. Setuptools 
    23 2. IPython 
    24 3. Zope Interface 
    25 4. Twisted 
     201. IPython 
     212. Zope Interface 
     223. Twisted 
    2623 
    2724It also has the following optional dependencies: 
    …… 
    32294. mpi4py (for MPI support) 
    3330 
    34 If you are comfortable using subversion and installing Python packages, you can probably just get and install these packages without further instructions If not, read on. 
     31If you are comfortable using subversion and installing Python packages, you can 
     32probably just get and install these packages without further instructions If not, 
     33read on. 
    3534 
    36 To get these packages, you will need Subversion (or svn). Many systems will 
    37 already have Subversion installed; if you need more information about 
    38 Subversion, see http://svnbook.red-bean.com/. 
     35To get these packages, you will need Subversion (or svn). Many systems will 
     36already have Subversion installed; if you need more information about Subversion, 
     37see http://svnbook.red-bean.com/. 
    3938 
    4039Installation Instructions 
    …… 
    4847    python setup.py install 
    4948     
    50 To install into the site-packages directory of your system Python you will need admin privileges.  More information about installing Python packages can be found in the Python documentation at http://docs.python.org/. 
    51  
    52 Setuptools 
    53 ---------- 
    54  
    55 IPython1 uses setuptools for installation.  Currently this is required, but  
    56 eventually, we will likely make this optional.  We recommend using the latest 
    57 development of setuptools (svn trunk). 
     49To install into the site-packages directory of your system Python you will need 
     50admin privileges. More information about installing Python packages can be found 
     51in the Python documentation at http://docs.python.org/. 
    5852 
    5953IPython 
    …… 
    8781--------- 
    8882 
    89 First, you should remove any old versions of ipython1 from your site-packages directory.  The latest prototype of the IPython kernel can be retrieved from the IPython Subversion repository with the command: 
     83First, you should remove any old versions of ipython1 from your site-packages 
     84directory. The latest prototype of the IPython kernel can be retrieved from the 
     85IPython Subversion repository with the command: 
    9086 
    9187:: 
    …… 
    9591If you want to do development on ipython1 and want to always run the svn version of ipython1, you should install using the command:: 
    9692 
    97     python setup.py develop 
     93    python setupegg.py develop 
    9894     
    99 Then each time you do an "svn up" you will not need to reinstall ipython1. 
     95Then each time you do an "svn up" you will not need to reinstall ipython1.  This requires 
     96that you have setuptools installed. 
    10097 
    10198Configuration 
    10299============= 
    103100 
    104 IPython1 has a configuration system.  When you run ipython1 for the first 
    105 time, the default configuration files are written you your ~/.ipython directory.  The basic config files are ipython1.core.ini, ipython1.kernel.ini and ipython1.notebook.ini.  Command line options override these configuration files. 
     101IPython1 has a configuration system. When you run ipython1 for the first time, 
     102the default configuration files are written you your ~/.ipython directory. The 
     103basic config files are ipython1.core.ini, ipython1.kernel.ini and 
     104ipython1.notebook.ini. Command line options override these configuration files. 
    106105 
    107106Testing 
    108107======= 
    109108 
    110 Once you have completed the installation of the IPython1 kernel you can run our test suite with the command: 
    111  
    112 :: 
     109Once you have completed the installation of the IPython1 kernel you can run our test suite with the command:: 
    113110 
    114111    trial ipython1 
    115112     
     113Or:: 
     114 
     115    nosetests -v ipython1 
     116     
    116117The trial command is part of Twisted and allows asynchronous network based 
    117 applications to be tested using python's unittest framework. Please let us 
    118 know if the tests do not pass.  The best way to get in touch with us is on the 
    119 IPython developer mailing list: 
     118applications to be tested using python's unittest framework. Please let us know 
     119if the tests do not pass. The best way to get in touch with us is on the IPython 
     120developer mailing list: 
    120121 
    121122http://projects.scipy.org/mailman/listinfo/ipython-dev 
    …… 
    125126=========== 
    126127 
    127 IPython1 includes MPI support, which enables the ipython1 engines to 
    128 pass data between each other using MPI.  Our recommendation is to use mpi4py  
    129 for MPI support, but ipython1 supports any code (Python or in a compiled language) that uses MPI.  The simplest way of getting started with MPI is to  
    130 install an MPI implementation (we recommend OpenMPI) and mpi4py (mpi4py.scipy.org) and then start the engines using:: 
     128IPython1 includes MPI support, which enables the ipython1 engines to pass data 
     129between each other using MPI. Our recommendation is to use mpi4py for MPI 
     130support, but ipython1 supports any code (Python or in a compiled language) that 
     131uses MPI. The simplest way of getting started with MPI is to install an MPI 
     132implementation (we recommend OpenMPI) and mpi4py (mpi4py.scipy.org) and then 
     133start the engines using:: 
    131134 
    132135    mpirun -n 4 ipengine --mpi=mpi4py 
    133136     
    134 This will automatically import mpi4py and make sure that MPI_Init is called 
    135 at the right time.  We also have built in support for PyTrilinos, which 
    136 can be used (assuming PyTrilinos is installed) by starting the engines with:: 
     137This will automatically import mpi4py and make sure that MPI_Init is called at 
     138the right time. We also have built in support for PyTrilinos, which can be used 
     139(assuming PyTrilinos is installed) by starting the engines with:: 
    137140 
    138141    mpirun -n 4 ipengine --mpi=pytrilinos 

  • ipython1/branches/ipython1-client-r3021/README

    r3053 r3060  
    22IPython1 README 
    33=============== 
    4  
    5 :Author:  Brian E. Granger 
    6 :Contact: ellisonbg@gmail.com 
    7 :Date: September 20, 2007 
    84 
    95.. contents:: 
    …… 
    128======== 
    139 
    14 This file gives an overview of IPython1.  IPython1 has a sophisticated and powerful architecture for parallel and distributed computing.  This architecture abstracts out parallelism in a very general way.  This enables ipython1 to support many different styles of parallelism including:: 
     10Welcome to IPython1.  We recommend the following as starting points for 
     11people new to IPython1:: 
    1512 
    16 1. Single program, multiple data (SPMD), message passing applications. 
    17 2. Task farming. 
    18 3. General Multiple Program, Multiple Data (MPMD) parallelism. 
    19 4. Data parallel. 
    20 5. Combinations of these approaches. 
    21 5. Custom user defined approaches. 
    22  
    23 For installation instructions, see the INSTALL file in this directory. 
    24  
    25 The IPython1 architecture consists of three components: the IPython Engine, the IPython Controller and various Controller Clients. 
    26  
    27 IPython Engine 
    28 -------------- 
    29  
    30 The IPython Engine is a Python instance that takes Python commands over a 
    31 network connection.  Eventually, the IPython Engine will be a full IPython 
    32 interpreter, but for now, it is a regular Python interpreter.  The Engine can also handle incoming and outgoing Python objects sent over a network connection.  When multiple Engines are started, parallel and distributed computing becomes possible.  An important feature of an IPython Engine is that it blocks while user code is being executed.  Read on for how the Controller solves this problem to expose a clean asynchronous API to the user. 
    33  
    34 IPython Controller 
    35 ------------------ 
    36  
    37 The IPython Controller provides an interface for working with a set of Engines.  At an abstract level, the Controller is an object and process to which IPython Engines can connect.  For each connected Engine, the Controller manages a queue of commands in incoming/outgoing Python objects.  While the Engines themselves block when user code is run, the Controller hides that from the user to provide a fully asynchronous interface to a set of computing resources.  Because the Controller listens on a network port for Engines to connect to it, it must be started before any Engines are started.   
    38  
    39 The Controller also provides a single point of contact for users who wish to 
    40 utilize the Engines connected to the Controller.  There are different ways  
    41 (ways=interfaces) of working with a controller.  Currently we have two 
    42 default interfaces to the controller:  RemoteController and TaskController. 
    43  
    44 Additional interfaces can be added to to enable other styles of parallelism. 
    45  
    46 Controller Clients 
    47 ------------------ 
    48  
    49 For each controller interface, there is a corresponding client.  These clients 
    50 allow users to interact with a set of engines through the interface that 
    51 the controller exposes. 
    52  
    53 Getting Started 
    54 =============== 
    55  
    56 To use IPython1 for parallel computing, you need to start one instance of the 
    57 controller and one or more instances of the engine.  The controller and each 
    58 engine can run on different machines or on the same machine.  Because of this, 
    59 there are many different possibilities for setting up the ip addresses and 
    60 ports used by the various processes. 
    61  
    62 Starting the controller and engine on your local machine 
    63 -------------------------------------------------------- 
    64  
    65 This is the simplest configuration that can be used and is useful for testing 
    66 the system and on machines that have multiple cores and/or multple CPUs.  The 
    67 easiest way of doing this is using the ``ipcluster`` command:: 
    68  
    69     ipcluster -n 4 
    70      
    71 This will start an IPython controller and then 4 engines that connect to the  
    72 controller.  Lastly, the script will print out the Python commands that you 
    73 can use to connect to the controller.  It is that easy. 
    74  
    75 Underneath the hood, the ``ipcluster`` script uses two other top-level scripts 
    76 that you can also use yourself.  These scripts are ``ipcontroller``, which 
    77 starts the controller and ``ipengine`` which starts an engine.  To use these 
    78 scripts to start things on your local machine, do the following. 
    79  
    80 First start the controller:: 
    81  
    82     ipcontroller & 
    83      
    84 Next, start however many instances of the engine you want to use:: 
    85  
    86     ipengine & 
    87  
    88 *WARNING*: the order of the above operations is very important.  You *must* 
    89  start the controller before the engines, since the engines connect to the 
    90  controller as they get started. 
    91  
    92 On some platforms you may need to give these commands in the form 
    93 ``(ipcontroller &)`` and ``(ipengine &)`` for them to work properly.  The 
    94 engines should start and automatically connect to the controller on the default 
    95 ports, which are chosen for this type of setup.  The logs are fairly verbose 
    96 and you should see various messages about the engines being registered with the 
    97 controller.  You are now ready to use the controller and engines from IPython. 
    98  
    99 Starting the controller and engines on different machines 
    100 --------------------------------------------------------- 
    101  
    102 To start the engines and controller on different machines you simply need to 
    103 specify the ip addresses where the controller is running.  Let's say you want 
    104 to start the controller on host0 and the engines on hosts1 - host16.  First 
    105 start the controller on host0:: 
    106  
    107     host0> ipcontroller & 
    108      
    109 Next, start the engines on host1 - host16 specifying the where the controller 
    110 is running:: 
    111  
    112     host1> ipengine --controller-ip=host0 & 
    113      
    114 Repeat this on host2 - host16 and you should be ready to go. 
    115  
    116 Specifying custom ports 
    117 ----------------------- 
    118  
    119 The controller can be configured to use different ports that the default. 
    120 Both the controller and engines must be told to use the same port:: 
    121  
    122     ipcontroller --engine-port=10001              # listen on 10001 
    123     ipengine --controller-port=10001              # connect to 10001 
    124      
    125 Starting engines using mpirun 
    126 ----------------------------- 
    127  
    128 The IPython engine can be started using mpirun/mpiexec, even if the engines 
    129 don't call MPI_Init() or use the MPI API.  This is supported on modern MPI 
    130 implementations like OpenMPI.  On a system with MPI installed you can do:: 
    131  
    132     mpirun -n 4 ipengine --controller-port=10000 --controller-ip=host0 
    133      
    134  
    135 Using the Controller and Engines from a Client 
    136 ============================================== 
    137  
    138 The best source of documentation about how to use the controller and engines 
    139 is the docstrings of kernel modules.  Eventually, we will provide more  
    140 details in this readme. 
    141  
    142 The RemoteController client 
    143 --------------------------- 
    144  
    145 Once a controller and engines are running you can connect to it using the  
    146 `RemoteController` class.  This class provides a client for the MultiEngine 
    147 interface.  Don't worry about what this means for now, it will become 
    148 clear as you use it. 
    149  
    150 First import the ipython1 client and create a RemoteController instance:: 
    151  
    152     >>> import ipython1.kernel.api as kernel 
    153     >>> rc = kernel.RemoteController(('127.0.0.1',10105)) 
    154      
    155 Now you can see what engines are registered with the controller:: 
    156  
    157     >>> rc.get_ids() 
    158     [0,1,2,3] 
    159  
    160 If this works, you are ready to go.  Start next by looking at the docstrings 
    161 for the following commands: 
    162  
    163 1. execute 
    164 2. push/pull 
    165 3. scatter/gather 
    166  
    167 The TaskController client 
    168 ------------------------- 
    169  
    170 The TaskController interface to the controller presents the engines as a fault 
    171 tolerant, dynamic load-balancing system.  You can even have a 
    172 RemoteController and a TaskController simultaneously connected to the same 
    173 controller.  But, be warned, the tasks will share the namespace on the  
    174 engines with the RemoteController.  But, this is actually a feature that allows 
    175 you to do some really nice things. 
    176  
    177 First create a TaskController objects:: 
    178  
    179     >>> import ipython1.kernel.api as kernel 
    180     >>> tc = kernel.TaskController(('127.0.0.1',10113)) 
    181      
    182 Note that the port is different than the RemoteController.  Now you can  
    183 start submitting tasks by passing instances of kernel.Task to tc.run. 
    184  
    185 It is best to check out the examples of using the task farming stuff in  
    186 ipython1/doc/examples. 
     131. The top-level ``INSTALL`` files describes how to install IPython1. 
     142. Users who want to use IPython1 for parallel computing should start 
     15   with ``docs/ipython1_parallel_intro.txt``. 
     163. IPython1 developers should start with ``docs/ipython1_dev.txt`` 
    18717 
    18818 

  • ipython1/branches/ipython1-client-r3021/docs/ipython1_parallel_intro.txt

    r3058 r3060  
     1Overview 
     2======== 
     3 
     4This file gives an overview of IPython1.  IPython1 has a sophisticated and powerful architecture for parallel and distributed computing.  This architecture abstracts out parallelism in a very general way.  This enables ipython1 to support many different styles of parallelism including:: 
     5 
     61. Single program, multiple data (SPMD), message passing applications. 
     72. Task farming. 
     83. General Multiple Program, Multiple Data (MPMD) parallelism. 
     94. Data parallel. 
     105. Combinations of these approaches. 
     115. Custom user defined approaches. 
     12 
     13For installation instructions, see the INSTALL file in this directory. 
     14 
     15The IPython1 architecture consists of three components: the IPython Engine, the IPython Controller and various Controller Clients. 
     16 
     17IPython Engine 
     18-------------- 
     19 
     20The IPython Engine is a Python instance that takes Python commands over a 
     21network connection.  Eventually, the IPython Engine will be a full IPython 
     22interpreter, but for now, it is a regular Python interpreter.  The Engine can also handle incoming and outgoing Python objects sent over a network connection.  When multiple Engines are started, parallel and distributed computing becomes possible.  An important feature of an IPython Engine is that it blocks while user code is being executed.  Read on for how the Controller solves this problem to expose a clean asynchronous API to the user. 
     23 
     24IPython Controller 
     25------------------ 
     26 
     27The IPython Controller provides an interface for working with a set of Engines.  At an abstract level, the Controller is an object and process to which IPython Engines can connect.  For each connected Engine, the Controller manages a queue of commands in incoming/outgoing Python objects.  While the Engines themselves block when user code is run, the Controller hides that from the user to provide a fully asynchronous interface to a set of computing resources.  Because the Controller listens on a network port for Engines to connect to it, it must be started before any Engines are started.   
     28 
     29The Controller also provides a single point of contact for users who wish to 
     30utilize the Engines connected to the Controller.  There are different ways  
     31(ways=interfaces) of working with a controller.  Currently we have two 
     32default interfaces to the controller:  RemoteController and TaskController. 
     33 
     34Additional interfaces can be added to to enable other styles of parallelism. 
     35 
     36Controller Clients 
     37------------------ 
     38 
     39For each controller interface, there is a corresponding client.  These clients 
     40allow users to interact with a set of engines through the interface that 
     41the controller exposes. 
     42 
     43Getting Started 
     44=============== 
     45 
     46To use IPython1 for parallel computing, you need to start one instance of the 
     47controller and one or more instances of the engine.  The controller and each 
     48engine can run on different machines or on the same machine.  Because of this, 
     49there are many different possibilities for setting up the ip addresses and 
     50ports used by the various processes. 
     51 
     52Starting the controller and engine on your local machine 
     53-------------------------------------------------------- 
     54 
     55This is the simplest configuration that can be used and is useful for testing 
     56the system and on machines that have multiple cores and/or multple CPUs.  The 
     57easiest way of doing this is using the ``ipcluster`` command:: 
     58 
     59    ipcluster -n 4 
     60     
     61This will start an IPython controller and then 4 engines that connect to the  
     62controller.  Lastly, the script will print out the Python commands that you 
     63can use to connect to the controller.  It is that easy. 
     64 
     65Underneath the hood, the ``ipcluster`` script uses two other top-level scripts 
     66that you can also use yourself.  These scripts are ``ipcontroller``, which 
     67starts the controller and ``ipengine`` which starts an engine.  To use these 
     68scripts to start things on your local machine, do the following. 
     69 
     70First start the controller:: 
     71 
     72    ipcontroller & 
     73     
     74Next, start however many instances of the engine you want to use:: 
     75 
     76    ipengine & 
     77 
     78*WARNING*: the order of the above operations is very important.  You *must* 
     79 start the controller before the engines, since the engines connect to the 
     80 controller as they get started. 
     81 
     82On some platforms you may need to give these commands in the form 
     83``(ipcontroller &)`` and ``(ipengine &)`` for them to work properly.  The 
     84engines should start and automatically connect to the controller on the default 
     85ports, which are chosen for this type of setup.  The logs are fairly verbose 
     86and you should see various messages about the engines being registered with the 
     87controller.  You are now ready to use the controller and engines from IPython. 
     88 
     89Starting the controller and engines on different machines 
     90--------------------------------------------------------- 
     91 
     92To start the engines and controller on different machines you simply need to 
     93specify the ip addresses where the controller is running.  Let's say you want 
     94to start the controller on host0 and the engines on hosts1 - host16.  First 
     95start the controller on host0:: 
     96 
     97    host0> ipcontroller & 
     98     
     99Next, start the engines on host1 - host16 specifying the where the controller 
     100is running:: 
     101 
     102    host1> ipengine --controller-ip=host0 & 
     103     
     104Repeat this on host2 - host16 and you should be ready to go. 
     105 
     106Specifying custom ports 
     107----------------------- 
     108 
     109The controller can be configured to use different ports that the default. 
     110Both the controller and engines must be told to use the same port:: 
     111 
     112    ipcontroller --engine-port=10001              # listen on 10001 
     113    ipengine --controller-port=10001              # connect to 10001 
     114     
     115Starting engines using mpirun 
     116----------------------------- 
     117 
     118The IPython engine can be started using mpirun/mpiexec, even if the engines 
     119don't call MPI_Init() or use the MPI API.  This is supported on modern MPI 
     120implementations like OpenMPI.  On a system with MPI installed you can do:: 
     121 
     122    mpirun -n 4 ipengine --controller-port=10000 --controller-ip=host0 
     123     
     124 
     125Using the Controller and Engines from a Client 
     126============================================== 
     127 
     128The best source of documentation about how to use the controller and engines 
     129is the docstrings of kernel modules.  Eventually, we will provide more  
     130details in this readme. 
     131 
     132The RemoteController client 
     133--------------------------- 
     134 
     135Once a controller and engines are running you can connect to it using the  
     136`RemoteController` class.  This class provides a client for the MultiEngine 
     137interface.  Don't worry about what this means for now, it will become 
     138clear as you use it. 
     139 
     140First import the ipython1 client and create a RemoteController instance:: 
     141 
     142    >>> import ipython1.kernel.api as kernel 
     143    >>> rc = kernel.RemoteController(('127.0.0.1',10105)) 
     144     
     145Now you can see what engines are registered with the controller:: 
     146 
     147    >>> rc.get_ids() 
     148    [0,1,2,3] 
     149 
     150If this works, you are ready to go.  Start next by looking at the docstrings 
     151for the following commands: 
     152 
     1531. execute 
     1542. push/pull 
     1553. scatter/gather 
     156 
     157The TaskController client 
     158------------------------- 
     159 
     160The TaskController interface to the controller presents the engines as a fault 
     161tolerant, dynamic load-balancing system.  You can even have a 
     162RemoteController and a TaskController simultaneously connected to the same 
     163controller.  But, be warned, the tasks will share the namespace on the  
     164engines with the RemoteController.  But, this is actually a feature that allows 
     165you to do some really nice things. 
     166 
     167First create a TaskController objects:: 
     168 
     169    >>> import ipython1.kernel.api as kernel 
     170    >>> tc = kernel.TaskController(('127.0.0.1',10113)) 
     171     
     172Note that the port is different than the RemoteController.  Now you can  
     173start submitting tasks by passing instances of kernel.Task to tc.run. 
     174 
     175It is best to check out the examples of using the task farming stuff in  
     176ipython1/doc/examples.