Summary of tools that I have used (a lot — UL — or a bit — UB) or should try to make use of (T) for Python development:
- Develop: Virtualenv (T), Netbeans (UL), WinPDB (UB)
- Document: Sphinx (UB)
- Test: Nose (UL), Fudge (T), Pylint (UB), doctest (UB), coverage (UL)
- Distribute: Pip (T), easy_install (UL)
- Deploy: Fabric (T)
Code
To develop Python modules, packages and applications, I have used many different editors and IDE’s: PythonWin, IDLE, Boa, SPE, Scite, Notepad++, WingIDE, Eclipse/PyDev, NetBeans, and a few others such as PyScripter etc that really didn’t work for me (too unstable, some essential feature missing, etc). I started with PythonWin and IDLE and moved on quite quickly to Boa, which even back in 2005 provided an integrated GUI designer for wxPython. However I found the UI menus to be really cluttered, and with a few other issues I was having with it, I moved on to SPE.
SPE was my favorite IDE for a couple years, its UI was clean and well-organized, it had nice capabilities and made use of pre-existing tools such as wxGlade, winpdb and pychecker, rather than re-inventing lesser wheels. But development on it stopped and there were a few bugs that were really annoying so I eventually looked for something else. Scite was ok for a while when I was using MinGW/MSYS as a development environment, it was a bit like in the good-ol-days in Linux. But once I had to move away from MinGW it was no longer adequate. I did try WingIDE, which is a commercial tool, it does seem very powerful, but I really like clean UI and wing’s I found to be very complex, almost like everything is in there at once, whether or not you plan on using it.
I eventually discovered Notepad++ which is perfect for quick development of Python scripts and offers syntax highlighting of so many languages that data files can also be displayed nicely etc. But it is not an IDE, it doesn’t have the notion of project so when I want to run the application I have to run it outside the editor, there is no refactoring etc.
After that I tried Eclipse with PyDev. This was rather nice but the free version of PyDev lacked a few essential features and I found it really annoying that the author would charge for those. Also, I find that the UI of Eclipse has been so abstracted, in order to cover to many different programming languages and use cases, that it is often a bit difficult to find what I need. For instance the help is sometimes too abstract, I would just want something specific to the language I am programming in. So I eventually looked for something else again.
For the past 6 months I have been using NetBeans. Its UI is great, it has nice capabilities, almost all the essential stuff. BUT it is quite slow to load (it can take a whole minute or more from double-click to the time I can see my first file), though UI is very fast other than that, and the code completion is not nearly as good as it was in SPE. SPE had a really neat code-completion algorithm in that it would cache the words you typed so when you would type a variable name it would propose that name and you just had to accept or keep typing or ESC. This was a real advantage for Python since in Python it is really difficult for an IDE to find what local variables are defined, so code completion never works on variable names. Also when it doesn’t know the type of an object it proposes the complete list of all methods and functions that exist in Python’s standard library, which is completely useless thing to do. But I really like it’s refactoring capabilities, it’s code navigation, it’s search/replace etc. It’s definitely very well designed, but the fact that it wasn’t designed for Python still shows as of the latest development release (Sep 11, 2009).
Document
I haven’t yet found the wholy grail of documentation systems. Like Doxygen for C++, there is Sphinx for Python. Sphinx uses the strategy that typically you want to create a hierarchy of HTML pages for various topics and you want to be able to refer and hyperlink to the source code, as well as be able to use some of the source code docs in the page docs. Note that Doxygen also handles Python docs, but not as nicely as Sphinx.
However, what I would really like is a documentation system that does the above, but also allows for the documentation to be edited online. For instance there could be two roles: editor and contributor. Editors could edit the comments (but not the code) online, and the changes would go straight to the version controlled repository. Contributors could edit the documentation and see what it would look like, and submit the change for review to the registered Editors.
This suggests that you would need a full-fledged CMS such as Django, Drupal, Joomla or such, with some plugins that would support doing source code checkouts, changes to the docs, re-generation of the HTML with results visible online, and if Editor then checkin the changes, if contributor then submit the changes to a list of people. The tool should allow the list of editors to be maintained, and contributors to be blocked from submitting. The tool would have to support the major version control systems such as Subversion, GIT, Mercurial, Bazaar, Darcs. And it would have to support the few different documentation systems that exist for Python, such as epydoc, sphinx, though initially perhaps it could start with just sphinx.
Heck, it would be awesome if such CMS could continuously run the unit tests with graph of success rate vs time, coverage % for various files vs time, etc. One could argue that this doesn’t belong in a distribution’s online docs, but actually it could be a nice way for users to get a sense of progress, as long as high-level numbers are shown (for instance the coverage % for complete package rather than for each module).
I guess I have to add this one to my TODO list of software development projects :}
Test
Nose: this regression testing framework is so easy to use and has a few really nice capabilities. For instance, I have a test folder that is a sibling to my source folder in my package. In that test folder are many files that contain the string “test” in the filename. Nose will find them all and execute each class method or function that has the string “test” in it, and provides a few neat functions like assert_equals and assert_raises.
Another nice feature of Nose is how it integrates with “coverage“, a python package for statemennt coverage analysis. This helps indicate if there are important sections of code that have not been exercised by unit tests. Note that this does not test logical / semantic coverage, just statement or syntax coverage, but just that is a huge benefit over just trusting your instincts.
Fudge: seems like a very capable and easy to use mock class library, but have yet to try it. It allows you to do things like “expectations” of method calls: a sequence of calls must be followed, or all methods in a set must be called.
Pylint: have used it off and on, can pick up on some good errors. Minor problem is that easy_install does not work for it, you must download logilab-common, logilab-astng and pylint, unzip all three, and run python setup.py install in each folder (in the order just listed for download).
WinPDB: For debugging I most use print statements in strategic locations in the code. I find that stepping through code is only useful once in a while when a bug is really difficult to figure out, but in general it’s easier to print variables in a few locations than to put breakpoints, starts the debug session, and get the debugger to print values, which is typically not trivial because using Python’s “print” is so easy and supports formatting etc. In any case, my favorite debugger, in the odd case that I do decide to use it, is WinPDB. Very powerful, and very good UI.
VirtualEnv: I’ve heard about VirtualEnv in a few places on the web and it is compelling, but I haven’t really had a need for it so far. VirtualEnv basically allows you to create a “mini python environment”, mini in the sense that it is an “extension” or “specialization”, in OO language, to the basic Python installation you have on your host machine. This can be useful to try out packages without putting them in the global site-packages folder, or to do development work. It is also very useful for having multiple versions of one package. For instance, say you had two GUI applications based on Wx, but one is based on an older version of Wx. You create a virtual env for each, install one application in each, and install the associated WxPython. Then the base Python install doesn’t have either Wx or the apps. Rather, each one has it’s “virtual Python environment”.
Distribute
I have been using easy_install/setuptools for a couple years for my projects involving Python. For the kind of project that I have been working on, it has been a great tool to have. So I thank its author for making it available to everyone, and I thank the creators/maintainers of the Python Program Index, without which easy_install would be less practical.
Every once in a while I check the “landscape” to see if there are any new and up-and-coming tools that I should be aware of. In the past while, I have seen posts from people annoyed about easy_install/setuptools, though the use cases mentioned have never applied to me so I couldn’t relate much. So sure enough, there is a new packaging/distribution tool emerging, called Pip. It apparently does most of what setuptools does, but seeks to keep only the best, and discard the “warts” that some expert users have been complaining about. So I’ll have to give it a try.
Deploy
In browsing the “distribute” landscape, I found a post by Alex Clemesha about two other related tools, virtualenv and Fabric. Fabric is apparently useful for deployment in that it easily runs local and remote commands. It is based on SSH so as long as you have an SSH service running on Windows (such as FreeSSH), I expect it to work on Windows too, but I’ll have to try it out.
