Ticket #231 (new defect)

Opened 9 months ago

Last modified 9 months ago

ipython1 ipcontroller's memory usage

Reported by: yichun Assigned to: bgranger
Priority: normal Milestone:
Component: ipython1 Version:
Severity: normal Keywords: ipcontroller
Cc:

Description

IPython1's ipcontroller seems to be memory-heavy and this can became a bottleneck of applicaton. I am collecting an array of about 4 MB as my results using TaskController? to run tasks and get results. The memory usage of my ipcontroller can reach 400 MB.

Is there a point in the design that this is justified? Or I am missing something important here?

Change History

02/18/08 11:57:12 changed by bgranger

There are a number of things that control the memory consumption of the Controller:

1) The number of pending tasks/commands that are not yet being worked on. If you have 1000s of pending tasks, these will surely take up memory.

2) The number of completed tasks - in the task controller, all completed tasks are kept around currently. This is obviously a design problem. Eventually, we will move to a system that keeps the most recent N results in memory, and stores the rest to disk. But, do you really need to bring the result back through the controller? Can you save it to disk? Do you see this when you run many such tasks?

3) The number of engines connected and being used. We have seen cases where a user was pulling a 10 MB array from 64 engines. While 10MB seems small, 640 MB is not. Furthermore, if you are running the engines on the same machine as the controller, there is a short period of time where the arrays is both on the controller and the engine -> then your memory consumption is over a GB. The moral of the story is that you do have to be careful about moving data around.

Can you give us more information about exactly what you are doing - the controller should not be consuming 400 MB unless you are doing something that is memory intensive.

02/18/08 13:31:52 changed by yichun

I am doing embarrassingly parallel tasks in large number (441 task for a bunch and about 5*5*5*7 bunches -- on 22 to 30 nodes, so these add up to quite some memory, and maybe I did not restart the ipcontroller before re-launching my tasks. Each task supposedly only transmit back one float number to the controller, in total this will result a about 6M data in my controller in some configurations.

I used to write data file to disks but eventually changed to do as much as possible in engines.

It would be great if use can control some memory usage of the ipcontroller, say, to ask it forget (from memory) results of tasks n to n+1000, etc, etc..

Thanks for your answer. I might soon switch to optimization in the parameter space which will require much less memory.