I spent many happy hours yesterday investigating Idle's debugger. The source of my excitement was that Debugger.py, one of Idle's debugger files, is quite simple. Suddenly it seemed a matter of just a few days to give Leo a debugger. My strategy is to put traces inside Idle to see how it works, and to simultaneously create parallel copies that will hold the version that Leo will use.

This is a long post because it summarizes a lot of very picky work. It may come in handy for the pyxides project.


The following files comprise the bulk of Idle's debugger. All are located in Python's Lib/idlelib folder: Debugger.py, RemoteDebugger?.py, run.py, FileList?.py and PyShell?.py. PyShell?.py isn't really part of the debugger, but there are interactions between the debugger and Pyshell.py that matter. run.py contains code for running python code in a subprocess.

As we shall see, it appears that separate copies of these files will be needed for Leo. (I could be wrong about this, but for now using separate files is the simplest and clearest way forward.) The Leo version are called leo_Debugger.py, leo_RemoteDebugger?.py, etc.

FileList?.py represents the list of open editors corresponding to open Idle files. The debugger uses this (I think) to highlight the presently selected line when execution pauses while single stepping or when reaching a breakpoint. leo_FileList?.py will be a complete rewrite. Indeed, in Leo all derived files could be considered to be open during debugging, so highlighting the presently selected line involves

  1. finding the corresponding @file tree,
  2. finding the proper node and
  3. choosing the proper line.

In other words, leo_FileList?.py will turn out to be an extremely useful adaptor class.

Note that RemoteDebugger?.py must be a separate file because it becomes the basis for a separate process.

Can't use Idle's files unchanged

My initial plan was to use Idle's debugger files without change, but this probably isn't going to work. Just a few lines of code must change, but that causes great packaging problems. For example, run.py is almost independent of the debugger, but it contains the method Executive.start_the_debugger that contains the following code:

return RemoteDebugger.start_debugger(self.rpchandler, gui_adap_oid)

The Leo version of this file, leo_RemoteDebugger?.py, replaces:

import RemoteDebugger


import leo_RemoteDebugger as RemoteDebugger

A trivial change, but how does one 'override' this import statement? And if run.py can't be generalized, it means that the debugger files that use run.py must be different too.

Creating the remote debugger

One of the attractions of Idle's debugger is that it creates the debugger in a separate process from Idle itself. This causes some packaging problems, however. Idle creates a separate process by building the following command in ModifiedInterpreter?.build_subprocess_arglist:

command = "__import__('idlelib.run').run.main(%r)" % (del_exitf,)

In Idle this code is in Pyshell.py, but for now I've put all such code in leo_Debugger.py. Anyway, the Leo version of this is:

command = "__import__('leo_run').main(%r)" % (del_exitf,)

Again, a small change, but how does one override this? The following code in ModifiedInterpreter?.spawn_subprocess creates the new process:

args = self.subprocess_arglist
self.rpcpid = os.spawnv(os.P_NOWAIT, sys.executable, args)

Path problems

It not possible to override code in the new process using code in the old process. Furthermore, the new process apparently contains a 'clean' version of the sys module, including sys.path, so the Leo version of this code will throw and ImportError? unless leo_run is on sys.path, or in the Os PATH variable. I worked around this for now by appending Leo's src folder to sys.path in sitecustomize.py. Later, some better way will have to be found. This may seem like a small detail, but it cost me several hours of noodling yesterday. In particular, the location of a file like run.py can make a difference because of sys.path. And certainly the name of a file affects import statements (and __import__ statements passed to spawnv).

These kinds of details are what make reusing code so difficult. To be clear, I see nothing whatever that is dubious in the Idle code--the problem is just hard.

Avoiding Pyshell in Leo's debugger

There are some unpleasant references to PyShell? objects inIdle's debugger. The most extensive changes I made to Leo's debugger code involved replacing references to a PyShell? instance by references to a dummyShell instance. The dummyShell class consists of just a few ivars. This strategy may need modification.

Present status


Obsolete --edreamleo, Thu, 25 May 2006 07:33:09 -0700 reply

This page illustrates my old way of thinking. The big debugger aha says that it is best to run debuggers in a process completely separate from Leo if possible.