Jim Kingdon on Programming

count.count

2011-12-07T20:32:00.001-08:00

Sometimes you pick a programming idiom because it is what you are familiar with, because you think it is expected, or because it expresses clearly what the code you are writing is trying to do. Other times, it is just too hard to resist. Lately at work at least two of us have seen .count.count in our rails3 code, and at first were sure it must be a typo. The real story is more fun than that, see the nerdfeed blog for more.

Using active record in rails migrations

2011-07-28T08:45:00.000-07:00

Most rails developers have probably sooner or later run into the problem: if your migrations refer to active record classes and the active record classes change out from under the migration, old migrations won't work as desired any more. Whether this is a big problem or a minor annoyance depends on how often you run migrations, how many databases you have (typically one for each developer and one or more you deploy to), etc, but I've seen the problem even over the course of three developer machines and a day or two, as some refactoring made people unable to update their code and then run a only-slightly-older migration.

One solution, advocated in the "Data migrations" section of Code review: Ruby and Rails idioms is just to fall back to writing migrations in SQL, bypassing active record (with the exception of the low-level parts of active record which connect to the database). This has two problems. The first is that active record doesn't help you a lot with this kind of low-level SQL construction. The example in that block post uses string interpolation to construct SQL, which they can get away with in that example (because the columns are integers) but which blows up as soon as the quoting isn't correctly handled (in a migration, this is probably just a bug rather than a security hole, but search "SQL injection" if you are unfamiliar with the problems). The second problem is that active record just is a more expressive way to manipulate data. How many people use script/console rather than script/dbconsole to look around the database?

My recommended solution, also advocated in How to use models in your migrations (without killing kittens), is to define the classes within the migration. There's an example in that blog post, but the short summary is that if, for example, your migration wants to refer to Vendor, you put "class Vendor < ActiveRecord::Base; end" within the migration class. In some cases you might need to define a few has_many or belongs_to relationships (make sure to set class_name to refer to the migration-specific class), but the interesting (and surprising to me) thing is that I've found that in practice you don't need a whole lot of them. Just to give a few examples of what this gets you, think of things like calling find_or_create_by_name to skip creating a record if it already exists, or looking up an object by name and then using its ID in a subsequent SQL statement. If you are thinking "but I can do that in SQL", then I'm not sure I should try to convince you. But if you are thinking "yeah, that is easier / more-concise / more-readable in active record" then defining your classes in the migration gets you both this, and also lets you run migrations even after your code has continued to evolve.

Celebrate tau day with a few proofs

2011-06-28T03:00:00.000-07:00

What better way to celebrate Tau Day than by trying your hand at writing a few proofs? Wikiproofs is a wiki which anyone can edit, and the goal is to build a library of proofs. They are written in a formal language, so the web site can check their correctness, and that is what makes it good for the tau day exercise: you get feedback about whether your proof is correct from the site as you go. The tau day exercises are intended to be of modest length and difficulty (so you can get them done on tau day) and explain everything you need to know about wikiproofs. Go to Wikiproofs tau day to play.

Help me proofread tau day exercises

2011-06-17T19:52:00.000-07:00

In honor of tau day (a math holiday celebrated on June 28 every year), I have written some exercises at wikiproofs.org tau day. If you have a little time to go to that page and try to go through the exercises, I'd appreciate your feedback on writing style, whether they are too easy or too hard, whether they were too long or too short, and any other suggestions. I'm hoping this will be a fun game/exercise for anyone interested in math but I am trying to make it accessible to someone with no experience in formal proofs.

If you are here to read about programming, I intend to keep writing about that too, but the math proofs have been a good part of my hobby activities lately. It is kind of like programming anyway (processed by a computer, needs to follow pretty specific rules to work, can be addictive in some of the same ways).

Scripting math proofs with RHilbert

2011-05-15T05:21:00.000-07:00

Formal math projects like wikiproofs prove mathematical theorems in a way that a computer can verify. There could be several motivations for this, including finding/preventing errors in proofs, helping learners to understand a proof, or exploring the consequences of assuming a different set of axioms.

Wikiproofs and related projects like metamath require that the person writing the proof spell it out pretty explicitly. For example, if you have 1 + 2 = 3 and you need 2 + 1 = 3, you'll need to explicitly transform one into the other. Other proof systems, like coq and isabelle, have a fairly powerful prover which can notice that you have 1 + 2 = 3 and a + b = b + a and combine those to prove 2 + 1 = 3.

Enough background. What I've been playing with lately is a project I just started and which I am calling Hilbert. This is a marriage of a metamath-like proof engine (in this case hilbert-kernel) and a generic scripting language (in this case Ruby). Writing a full prover in Ruby is of course one direction this could eventually head, but I was thinking more in terms of simpler kinds of automation (perhaps there could be a routine called "commute as needed" which would be able to turn "1 + 4 = 3 + 2" into "4 + 1 = 3 + 2" by noticing the left hand side needs to be flipped and the right hand side doesn't). I'm hoping this system will be easy for people who find Ruby more comfortable than coq or isabelle (I might count myself among them, although of course I reserve the right to learn coq or isabelle some day). It also may have other benefits, like making it easier to develop hilbert-kernel itself (for example by running hilbert-kernel tests).

Update 20 May 2011: this is under active development, but the change which needed an update above is that I renamed the project from RHilbert to hilbert.

Ruby rescue gotcha

2011-03-22T13:40:00.000-07:00

Today in the codebase at my day job, I found a particularly cute bug related to "rescue" in Ruby. This isn't a particularly unknown gotcha–I've read about it on the net at least once–but this is a particularly sweet (or devious) manifestation, and as far as I can tell it was purely accidental, not a contrived example.

The following example uses rspec, but the key thing is begin; 2 / 0 rescue NoMethodError; end versus begin; 2 / 0; rescue NoMethodError; end–what is the difference between these two statements?


require File.expand_path('spec_helper', File.dirname(__FILE__))

describe "enigma" do
  it "fails, but why" do
    lambda { begin; 2 / 0 rescue NoMethodError; end }.
      should raise_error(ZeroDivisionError)
  end

  it "passes, but why" do
    lambda { begin; 2 / 0; rescue NoMethodError; end }.
      should raise_error(ZeroDivisionError)
  end
end

I'll post the answer in a comment in a few days; feel free to post your answers as comments if you wish.

Levitation-perl and deleted files

2011-03-03T17:48:00.000-08:00

Ever since I started using levitation-perl, I was curious about how it handles deleted files. Well, I found out (the hard way). The symptom was that I did a routine merge and a lot of files showed up as conflicts (including ones which hadn't been edited, on the wiki or on the git side, for a long time). This is usually a symptom of git not being able to find a reasonable common ancestor.

Turned out that any files which had been deleted in mediawiki caused what amounts to a rewrite of the git history (as if those files had never been created). This is not specific to mediawiki/levitation; the same kind of thing would happen in pure git if you ran git filter-branch or similar mechanisms to delete a file from git and make sure it was erased from the history (see for example Rewriting History or The trouble with firmware).

The consequence of the rewritten history for my merge was that the common ancestor was very old (prior to when the deleted file was first created, about two years ago in my case), rather than a few days old as would be the case without the history rewrite.

How did I get my merge done? In my specific case, the content of the deleted files was nothing sensitive, so I was fine with having them remain in the git history. If they needed to stay gone from the history, I would probably have needed to follow RECOVERING FROM UPSTREAM REBASE in the git-rebase manpage.

My solution was to create a merge commit whose parents are the two corresponding commits in the rewritten and non-rewritten history. Since the git repository I was using is public, you can follow along. The commands here are somewhat edited (I've snipped out my dead ends and multiple invocations of gitk to see what I had at each step).


[browse gitk history to find that 681b7936 is one of the commits
with message "add domain and Domain"]
$ git checkout 681b793629d88729b919f40d0862884147db0d8d
Note: checking out '681b793629d88729b919f40d0862884147db0d8d'.

You are in 'detached HEAD' state. . . .
HEAD is now at 681b793... add domain and Domain
$ git checkout -b withdeletedfiles
Switched to a new branch 'withdeletedfiles'
$ gitk levitation/master&
[browse history to find that a6d0885... is the commit with message
"add domain and Domain"]
$ git checkout a6d0885
Note: checking out 'a6d0885'.

You are in 'detached HEAD' state. . . .

HEAD is now at a6d0885... add domain and Domain
$ git checkout -b withoutdeletedfiles
Switched to a new branch 'withoutdeletedfiles'
$ git merge -s ours withdeletedfiles
Merge made by ours.
$ git diff --stat -w withdeletedfiles withoutdeletedfiles
 Main/W/P/.3a/WP:INDEX          |    2 --
 Wikiproofs/S/u/b/Subject index |   30 ------------------------------
 2 files changed, 0 insertions(+), 32 deletions(-)
$ git checkout master
Switched to branch 'master'
$ git merge withoutdeletedfiles
Removing Main/W/P/.3a/WP:INDEX
Removing Wikiproofs/S/u/b/Subject index
Merge made by recursive.
 Main/W/P/.3a/WP:INDEX          |    2 --
 Wikiproofs/S/u/b/Subject index |   30 ------------------------------
 2 files changed, 0 insertions(+), 32 deletions(-)
 delete mode 100644 Main/W/P/.3a/WP:INDEX
 delete mode 100644 Wikiproofs/S/u/b/Subject index
[1]+  Done                    gitk
$ gitk&
[1] 3233
$ git show 01a25538177dbe768e130aa94f7d49be11a63733
commit 01a25538177dbe768e130aa94f7d49be11a63733
Merge: 4ba316c e908dc8
Author: Jim Kingdon 
Date:   Tue Mar 1 20:40:57 2011 -0500

    Merge branch 'withoutdeletedfiles'

$ git diff --stat -w e908dc8..master
 Interface/S/e/t/Set theory                         |   13 +-
 Main/O/u/t/Out lines                               |  311 +++++++++++
[and the rest of the diffs I'd expect from the wiki to git]
 22 files changed, 4626 insertions(+), 47 deletions(-)
$ git diff --stat -w 4ba316c..master
 Main/W/P/.3a/WP:INDEX          |    2 --
 Wikiproofs/S/u/b/Subject index |   30 ------------------------------
 2 files changed, 0 insertions(+), 32 deletions(-)
$ git push
$

Four hour agile subcommittee meeting

2011-02-26T04:35:00.000-08:00

Don't know whether to file this under "what happens when something becomes a buzzword", "people who don't get it", or "baby steps, baby steps", but today at standup (for an organization not traditionally agile, but whose avowed intention is to become agile), someone casually mentioned the words "four hour agile subcommittee meeting". It is one of the most wonderfully oxymoronic phrases I've heard in a while.

Tracking a mediawiki wiki in git

2011-01-16T07:13:00.000-08:00

There has been a lot of interest lately in git-backed wikis, like gollum, git-wiki, or gitit. It is appealing to both make it easy for casual editors to make a change (without cloning anything, installing git software, having to push, etc), but also provide a git repository for things like browsing and editing the content offline, being able to merge changes, and the like. The appeal strikes me as particularly strong if the content of the wiki is software (as is envisaged for wheat) or something software-like, such as mathematical proofs (as in wikiproofs).

So a wiki which is backed by git is cool, but what if the wiki is using mediawiki (perhaps the most popular wiki software, in use at wikipedia and other sites)? One might want the various features and extensions of mediawiki, or one might be looking to git-ize a wiki which has been around longer than this whole git-based wiki trend got going. One solution I've been playing with is
levitation-perl.

You first of all get an XML dump file from the wiki (most mediawikis produce these once a day and make them available for download). Then you download levitation-perl and install perl and some packages from
CPAN (as documented in levitation-perl's README and slightly elaborated at my fork). Then you can run levitation-perl (as described in the README, again) to import all the mediawiki changes into an empty git repository. Then you can push this git repository someplace public, clone it locally, and do all the good git stuff you are used to.

At least for me, moving changes from git to the wiki is manual, although that is perhaps in part a function of how wikiproofs works–the changes are checked for correctness as you edit the pages on the wiki, and I haven't yet bothered to try to install the correctness-checking software locally.

When the wiki changes and you want to get all those changes into git, get a new XML dump and run levitation-perl again. You don't need to run it in an empty git repository this time–I just run it in the git repository that I used for the previous levitation-perl run (on the other hand, when I want to check out files, I clone this repository elsewhere).

Levitation-perl hasn't gotten much attention lately, I think due to technical problems using it for something as large as wikipedia (and perhaps policy questions of what to use it for on wikipedia). But none of that really affects my use of it on wikiproofs, because wikiproofs is way smaller than wikipedia, and because my goals are mostly just wanting a way to work on wikiproofs when I don't have good internet
connectivity.

Writing an Eclipse plug-in

2008-10-10T19:20:00.000-07:00

I recently started playing around with writing an Eclipse plug-in, and I thought I should share some first impressions about how easy it was.

The motive was to more easily play with metamath, a system to automatically verify (not write) mathematics proofs. Unless I've missed something, there is little activity on development environments for math proof systems, but it seems to me the need for good tools (like eclipse) is at least as great for math proofs as for software.

In all cases this was based on the Eclipse which ships with Fedora 8, although I'm not aware of anything Fedora-specific. I started with the eclipse help files for "Plug-in development environment". It was relatively easy to create a project seeded with one of the example plug-ins which ships with eclipse (in my case, first the hello world one, and then com.example.witheditor which seemed like the most relevant to the plug-in which I was trying to write, which at least at the start will be a few simple decorations on top of text editing, similar to an emacs major mode or one of the syntax coloring modules for vi, gedit, etc). Generally, the help files walked me through all I needed to get started. I was somewhat puzzled with "how do I get back to the Overview once it is closed" until I figured that opening the plugin.xml file gives you a specialized view, including Overview, with links to click and forms to edit.

The fact that Fedora ships with the source code to the standard Eclipse classes that a plug-in needs to hook into, combined with the good Eclipse features for navigating Java code, made things really easy. I was pleasantly surprised the first time I control-clicked on an Eclipse method and got not only the arguments I needed to pass in, but also the commented source code. Within a few hours, I had turned the example plugin into something which was at least starting to understand metamath syntax. Not bad considering that this includes a fair bit of experimenting (e.g. playing with foreground and background colors) and learning about the platform. I tend to find that Java and Eclipse make it easier to explore a large unfamiliar codebase, compared with a language like Ruby (where exploring large unfamiliar codebases, like Rails, has been a daily activity in my recent job), but I also give credit to developers of the Eclipse plug-in system. The example pointed me to the relevant parts of the plug-in libraries, and the well-commented source code of the libraries themselves helped me poke around to figure out what pieces would do what I want. Another huge win is the way that the eclipse plug-in development environment just worked out the box. There was no messing around with CLASSPATHs, jars, ant and similar rigamarole: just go to the Overview page and click on "launch" and you are running.

Error reporting was a problem: one of my first edits to the example passed a bad value to a constructor in a constant (this is the RGB constructor in IXMLColorConstants if anyone is following along). There was a dialog box referring to an error log, but I have no idea where this error log might live (apparently not in my plug-in project). Ideally, the plug-in development environment would have somehow showed an exception with a stack trace, or something of the sort.

My plug-in so far can be found at mmclipse.

Simple Design and Testing conference, suburban Chicago, Sep 12-14

2008-08-28T22:07:00.000-07:00

I probably should have mentioned this a while ago, but coming up in about two weeks is the 2008 Simple Design and Testing conference. As with other open space conferences, the format is similar to Birds of a Feather (BoF) sessions and ranges from free-wheeling discussion to something slightly more closely approaching a presentation. Topics typically center around things like agile development and object oriented design. The conference is free and registration is pretty simple - write a wiki page called a "position paper" saying what you want to learn and/or contribute. Mine was ErectorRubyRenderingLibrary.

Parse XML with saxophone

2008-05-22T22:13:00.000-07:00

Programmers often want to parse an XML document and perform some actions as we do (for example, build up an in-memory data structure, write data to a database, print output to the console, etc).

For the most part, there have only been two well-known ways of doing this. The first is to read the XML document into a DOM, which is an in-memory tree representing the document. Then you walk the DOM tree doing your own processing. This is usually a pretty convenient way to go. But it has several downsides, the most obvious and probably biggest of which is that if the XML document is too big to fit in memory it won't work.

The second approach is a streaming API such as SAX or xmlpull. SAX calls you whereas you call xmlpull, but either way you are getting a low-level stream of events (start tag, end tag, and text being the most important). For simple tasks, this isn't so bad. But when you have to assemble data from a few different parts of the document, you need to set variables to keep track of what you've gotten and what you are waiting for, and it is possible to make yourself a pretty big pile of spaghetti. My co-workers and I had such a problem recently, and our answer was a small library, which we call saxophone. Saxophone sits on top of SAX, and converts the raw stream of events into calls to handlers, and lets the handlers return data which get passed to other handlers. The current implementation is in ruby, although the ideas should port to other languages if anyone has such a need.

Here's a simple example. We want to parse a web feed and print out each of the titles. That is, there will be, buried somewhere in the XML, something like <title>My Dog has Fleas</title> and we want to print out "My Dog has Fleas" and likewise for every other occurrence of a title tag.

The full example can be found in the examples directory of saxophone, but the key part is:


parser = Saxophone.new(
  :title => lambda { | element | puts element.all_text() }
)

This is saying that any time you see a "title" tag, call this handler, and provide it all the text directly under title via the all_text() method.

This example doesn't demonstrate everything. Handlers can return values, which are then available to higher handlers. Handlers also have access to attributes directly.

If people want to hear more about it, I can write some more documentation and examples for some of these other features. But the key thing here is that we have a fairly concise way to (a) match the parts of the XML that we care about, and (b) synthesize the results of those matches into larger data structures (but only as far as we want - after we have gotten everything we need to, for example, write a database row, we can return that memory to the system). This is all done in a streaming fashion. That is, we don't need to store the whole document, or the whole result, in memory.

The whole thing kind of reminds me of XSLT or XQuery. In know in XSLT for sure, and probably XQuery, there are some tasks that turn out to be really awkward, and that's probably true of saxophone as well. But saxophone seems like a good match for a few of the problems we've had. And of course, having a library within your regular programming language (in this case Ruby) can also be a plus over a special-purpose language.

Saxophone is free software, available here as part of the pivotalrb project on rubyforge, which contains a variety of open source code released by the ruby/java consulting firm Pivotal. I and my pair wrote saxophone as part of a Pivotal project.

Draft of Erector talk

2008-04-03T07:25:00.000-07:00

I'm giving a talk on erector in about a week at the Washington, DC Ruby User's Group. Here's a draft of my talk; please provide feedback so I can improve the talk (for example, as comments at the bottom of this page).

Ruby's flexible and minimal syntax makes it well-suited to writing libraries for various tasks

Erector applies that to HTML (or XML) rendering

(start up erector in script/console and go through the following examples)


class Foo < Erector::Widget
  def render
    html
  end
end
Foo.new().to_s


Erector::Widget.new() do
  html
end.to_s


Erector::Widget.new() do
  p "foo"
end.to_s


Erector::Widget.new() do
  text "hello"
end.to_s


Erector::Widget.new() do
  p do
    text "hello"
    b "world"
  end
end.to_s


Erector::Widget.new() do
  a :href => "a.html" do
    text "world"
  end
end.to_s


Erector::Widget.new() do
  a "world", :href => "a.html" do
end.to_s


Erector::Widget.new() do
  text '<>&'
end.to_s

Hooking erector to rails:


class WelcomeController <
  ApplicationController

  def index
    render :text =>
      Views::Welcome::Show.new().to_s
  end

end

Structuring views with the usual Ruby techniques: especially inheritance and methods

Responsibilities of a view mechanism:

quote output

balance start/end tags

Comparison with ERB and Markaby, quoting:

ERB: no, must call h

Markaby: for strings, not blocks

Erector: yes*

* except when you call raw

Comparison with ERB and Markaby, tag balancing:

ERB: no

Markaby: yes

Erector: yes*

* except when you call raw

Comparison with ERB and Markaby, subpages:

ERB: partials, helpers

Markaby: partials, helpers

Erector: inheritance, methods

Similar libraries in other languages:

Java: webiyo.sf.net

If there is time:

Calling helpers from erector

Writing helpers in erector

Hello world type example for jruby and mayfly

2008-02-20T18:06:00.000-08:00

Given all the interest in the ruby programming language, it is natural to ask whether rails applications (or ruby applications more generally) could write their tests with mayfly.

As a first step, I figured out how to run mayfly under jruby. Here's what I did.

First, I installed jruby JRuby 1.1 RC 2as directed. Then, I put the mayfly 0.3 jars in $JRUBY_HOME/lib (there are other ways to tell jruby where to look for them, but this seemed like the simplest).

Then, I put the following in hellodb.rb:


include Java
import 'net.sourceforge.mayfly.Database'

d = Database.new()
d.execute("create table foo(x integer)")
d.execute("insert into foo(x) values(5)")
puts d.rowCount("foo")
d.execute("insert into foo(x) values(6)")
puts d.rowCount("foo")

Then running jruby will invoke mayfly:


$ jruby hellodb.rb
1
2

The next step would be to figure out how to get Active Record talking to Mayfly.
Haven't tried that yet.

Profiling with gprof (success on a short test program)

2008-02-11T21:01:00.001-08:00

When last we discussed profiling Mayfly, I was profiling with JIP. Brian Slesinsky, in a comment to that article, told me that he has found that JIP has a per-method cost which shows up in the profiling data (so that method calls appear to be more expensive than they really are). He suggested the NetBeans profiler.

Well, I was looking into how to install and use NetBeans (NetBeans, unlike Eclipse, does not ship with Fedora), and hadn't gotten much of anywhere until I had a lot of time (on a flight), and was left with seeng whether I could get anywhere with the tools which I have already installed. That means gcj and gprof. I got gprof working fine on a short test program (where it correctly identified the bottleneck), but didn't (yet) succeed in running it on mayfly.

I suppose if people would find them helpful, I could upload my test programs and build scripts, but basically they boiled down to:


gcj --main=Profilee -pg Profilee.java
./a.out
gprof >profiler

Getting something like this invoked from ant is largely a boring but more or less straightforward matter (although it wasn't clear to me how the classpath relates to the class and java files specified on the command line). But at least one of my invocations led to a runaway linker which made my machine swap for quite a while before I finally gave up. So although it is premature to declare victory on this just yet, I did want to report on my success with the test program.

Avoid needless generality

2007-12-13T13:38:00.000-08:00

I've been asking people what Mayfly features would be most useful to them. I suppose I shouldn't be surprised that one of the answers I got was roughly "we use the syntax CONCAT(a, b, c) for string concatenation in MySQL and it would be nice if Mayfly had it instead of making us write it as a || b || c". This fits right into the theory that often the most important features are the silliest (as I already discussed regarding case sensitive table names).

Anyway, having decided to work on CONCAT, my first instinct was to provide hypersonic-compatible stored procedures, which allow the user to define their own CONCAT (well, the two argument version anyway; I'm not sure hypersonic has stored procedures with variable numbers of arguments). I started implementing stored procedures, and it became clear that there were plenty of corner cases (picking which overloaded method to call, error handling, type coercion, and of course the variable number of arguments issue).

So not only did it seem easier to just add a CONCAT built-in, it is closer to what the user really wants. Although defining a concatenate method and registering it with a CREATE ALIAS command isn't hard, there's no particularly compelling reason to make people do this. Figuring out where to put the CREATE ALIAS invocation may also be more of a pain than it sounds, in a system like Mayfly with no static state shared between objects.

There's a good chance I (or some other Mayfly contributor) will eventually get around to implementing the general stored procedures (this feature can be a handy way to make SQL code more portable between Mayfly and non-Mayfly databases, in cases where Mayfly doesn't yet have all the built-in functions that the other database has). But jumping right to the general feature runs a risk of either (a) designing a feature which is supposed to be general, but which isn't really suited for any use case other than the original specific one, which could have been implemented in a way which is much more straightforward for the programmer and the user, and/or (b) designing a general feature which is too complex, too slow, takes too much time to implement, etc, relative to what is really needed. So there's a lot to be said for special-purpose solutions.

Simple Design and Testing Conference

2007-12-03T17:59:00.000-08:00

Had the chance to talk about Mayfly at the Simple Design and Testing Conference 2007 conference last weekend. As with other open space conferences I've been to, you don't know quite how a session is going to turn out, but my notes include the sample code which I showed people.

Hibernate Annotations and You Ain't Gonna Need It

2007-09-18T15:11:00.000-07:00

In the last post I showed a Hibernate demo. I wrote it to show how to hook Mayfly to Hibernate, but it also was my first experience with Hibernate Annotations. I have long been hoping that Hibernate Annotations would ease the pain of hibernate, specifically by eliminating the need to keep looking back and forth between XML mapping files, Java code, and database schemas.

Basically, my experience to date is quite positive. The key thing to note about the demo is all the things which aren't there.

This starts with configuration. There is no hibernate.cfg.xml file, no database username and password, no JDBC URL and driver (I guess some of those would come back if we were talking to a database like MySQL as opposed to Mayfly, but the hibernate.cfg.xml wouldn't).

Next, and more importantly, look at Foo. Here we just need an instance variable for each database column, an annotation @Entity at the start, and an annotation @Id on the primary key. Since the instance variables are named the same as the columns, we don't need to specify a mapping between the two. There's no XML mapping file. I also don't bother with getters and setters. I can always add them later (that's why refactoring browsers have "encapsulate field"), and using them only where they are providing some value avoids a huge amount of clutter. (Or to put it more provocatively, "You Ain't Gonna Need Getters and Setters").

Just getting everything into Foo.java, rather than splitting it between Foo.java and Foo.hbm.xml, is a huge win. You can't just control-click over to the XML file (in a browser like Eclipse), and even if you could you'd be looking back and forth rather than just having everything in one place as with annotations.

So positive impressions confirmed. Definitely plan to give annotations a try on a larger project next time I have a choice.

Demo of Mayfly and Hibernate

2007-09-18T14:55:00.000-07:00

In April I wrote a bit about hooking Mayfly to Hibernate. Well, I finally got around to writing a demo showing how to hook it all up in a small self-contained example.

The demo is only a few dozen lines of code, but instead of repeating the whole thing here, I'll point out some interesting bits. Each test creates its own Database object, so there is no shared state to clear out between tests. The test then calls the openConnection on Database to get a JDBC connection, which the FooPersistence class then passes to the openSession method of Hibernate.

Here I configure Hibernate by instantiating an AnnotationConfiguration, tell it the dialect (Hibernate can't figure this out from the connection, because it needs the dialect before the openSession call), and registering Foo as an entity. If you configure Hibernate via XML mapping files, that would work too.

That's pretty much all that is needed. The dialect file is currently part of the demo, so copy it from there.

Profiling mayfly

2007-07-30T07:27:00.000-07:00

One of the key goals for an in-memory database for unit testing is that it should be fast. And mayfly is faster than the other databases I've tried for some things, like creating or altering tables. However, it is some 3x slower than hypersonic for inserting rows ("insert into foo(a,b) values(3,4)" kinds of commands). I've been meaning to profile it for a long time now, but I finally got around to it.

First I added the JIP jar to the project and added a profile-test target to build.xml. Because JIP doesn't work with gcj, I ran it under Sun java. JIP writes a text file report, and I worked off that.

The results surprised me. They showed that 70% of the time was being consumed in the lexer and parser. Conventional wisdom is that lexing and parsing just isn't your big bottleneck in a compiler these days, but perhaps that is more for an optimizing compiler which spends more time in code generation and optimization passes. I inlined a few short methods which were bottlenecks (at the cost of a small amount of code duplication, but not so much as to be really shocking, and remember I was careful to only do this for the bottlenecks). This was also a surprise: that method invocation seemed to be such a cost. Given my limited knowledge of java internals, that sort of makes sense, but it kind of seems like a step backwards, in the sense that in the C/Pascal/etc days we made so much effort to make method invocation fast, and tell people they didn't need to make their code ugly to avoid method calls. (At the risk of belaboring the obvious, even if method calls are expensive you still don't need to make your code (very) ugly to avoid them: it is only a handful of invocations which are actually going to make a difference in your run-time, and the profile shows you which ones).

I was also able to streamline the non-parsing part of the code, mostly by taking out some extra steps (for example, transforming a column name to a Column and back to a name more times than needed). Some of that had built up through a series of changes which had left in vestiges of previous ways of doing things. So cleaning this up left the code simpler and clearer, as well as faster.

Other changes, like changing Row to be a HashMap rather than a List, didn't seem to help at all (or even hurt slightly). It has been conceptually a map for some time now, but apparently those linear searches were not particularly more expensive than the many calls to hashCode you get with the map. I guess the fact that we don't expect more than a few dozen columns in a table is responsible.

So what is next when I look at this again? For the lexer, I may have run out of obvious ideas (given that it has duties like tracking the line and column numbers of every token, and I don't see giving up that feature, which provides good error messages). For the parser, there is a lot of expression handling machinery that is involved in parsing the "3" in "insert into foo(a) values(3)". Unless I think of a better way, having the top-level expression parser look for a literal followed by something like "," or ")", and going into a fast-path special case might be worth it. I know that looks like a kluged-up wannabe bottom up parser, but I've been happy enough with recursive descent in other ways, that I have trouble seeing switching back to a parser generator. As for the execution (building up rows, modifying the tables, etc), I'd have to look at the profile more. Although I've seen some hot spots, fixed them (and perhaps created others), I don't have as much of an intuitive feel for what is slow here as I do for the lexing and parsing.

Database upgrades: SQL versus code

2007-05-29T12:02:00.000-07:00

When last I wrote about database upgrades, one of the design designs was whether to have a database upgrade be a hunk of code (as in rails and other systems I've worked with), or an SQL script (as in the non-automated upgrade scripts that were checked in to MIFOS prior to December's work).

For the first cut (last December), I went with SQL, on the theory that (1) it might be easier for people to understand, especially sysadmins and others who wouldn't necessarily read much of the Java code, (2) many of the cases which came to mind, such as adding a column or adding a table, could be done this way, and (3) if the automated upgrade runs into trouble, it may be easier to run some scripts one at a time, perhaps with changes, rather than trying to mess around with Java (again, for some people). But I did have in mind, even then, that I might need to add code upgrades at some point.

Well, we found a case where the SQL scripts don't work. The MIFOS database contains tables which store things like strings which are displayed to the user. MIFOS ships with a whole bunch of these ("loan", "client", etc), but such strings can also be added by the microfinance institution. In adding them (at least the way our database is currently set up), one needs to assign at least one ID which is not referenced directly from the Java code, but which also is referenced elsewhere in the database. Although there are variants of SQL which have variables and the like (PL/SQL, PL/pgSQL, etc), I don't think MySQL has those kinds of extensions (and trying to turn SQL into a procedural language is somewhat awkward anyway). So the solution will be to implement Java upgrades. I have in mind keeping the ability to do SQL upgrades (that is, each upgrade is either a java upgrade or a SQL one). That is largely to ease the transition (we have about 19 upgrades already, and won't need to convert them over all at once). We'll also see whether writing upgrades in SQL, in those cases where it is possible, ends up being appealing or just a source of confusion.

Test speedups

2007-04-26T17:00:00.000-07:00

The MIFOS tests (ApplicationTestSuite, or pretty much all the junit tests) run in 2600 seconds on my laptop. When you are checking in several times a day, that adds up to a lot of time spent waiting for tests. It also discourages good habits like running the tests frequently to find problems early, checking in frequently, certain kinds of experiments (for example "if I clean up this ugly code, will anything break?") and the like.

There are a few reasons for this. First of all, I'll agree with something that Eric Du or Li Mo (I forget which) said a month or so ago, which was that too many of our tests are integration tests (test many classes) as opposed to unit tests (test a small number of classes). This is very much a case-by-case thing, so I guess I'll just mention the case which came up today: I was testing generateId in AccountBO. And it turns out there is no need to access the database to test this method (see AccountBOTest).

I've also known for a long time that many of the tests get caught in a familiar trap of writing many objects to the database (I need a client, and a client needs a group, and a group needs a center....). Or of creating objects via the database when creating them in-memory would work just fine, be faster, and avoid problems with getting rid of them at the end of the test.

But I was surprised at the speedups around getUserContext(). Someone (sorry, I tried finding out who in the archives but I didn't find it) posted some numbers to the MIFOS mailing list saying that
replacing TestObjectFactory#getUserContext (which involves several database calls) with something faster (TestUtils#makeUser() is the usual choice) cut the run-time of a certain test in half (or something - the number varies from one test to the next and once I convinced myself that the speedup is significant I haven't really been measuring things further).

Unfortunately, globally changing getUserContext to makeUser doesn't quite work - some tests fail that way. But one of my projects lately has been going through tests and changing all those that can be changed.

Getting tests to run fast can take work (especially if you don't have the luxury of writing fast tests from the start of a project). But tests that run slowly don't tend to get run.

Mayfly and Hibernate

2007-04-03T18:35:00.000-07:00

On MIFOS, we are successfully using Mayfly and Hibernate together, but there are some catches (and some future work - for me and/or other volunteers - in terms of making this work better).

First of all, MIFOS is currently on Hibernate 3.0beta4. I suspect later versions of Hibernate work too, but it would be nice to download Mayfly and Hibernate and try it out on some kind of "hello world" situation.

Next, there's the Hibernate dialect. The one we are using in MIFOS is checked in to MIFOS as MayflyDialect. It would be nice to submit this to Hibernate as a patch. I have been meaning to do this, but just haven't gotten around to it. For a while, I thought it might be changing frequently, but that hasn't been true lately.

Anyway, enough of the boring stuff. The interesting part is whether we can hook up the features of Mayfly which distinguish it from Hypersonic and the rest. For example, let's take the feature of wanting to give each test a fresh database. Suppose we have a static DataStore which contains all the tables and perhaps some data which all tests should start with (in MIFOS, getStandardStore() in DatabaseSetup ). Now, in Hibernate one typically creates a SessionFactory once (not on every test, as it is expensive to make one), and the SessionFactory has the JDBC URL built in. So how do we give a new Database for each test while being able to re-use the SessionFactory? Well, what I've done so far is open my Session with the SessionFactory#openSession(Connection connection) method. I'm probably best off just pointing to an example: testGetAllParameters in ReportsPersistenceTest.

So, anyone found better techniques? This is a good subject for collaboration, not just because it is a way to share the work, but also because everyone's way of setting these things up tends to be slightly different.

Web security: avoiding HTML injection

2007-03-15T17:49:00.000-07:00

A shockingly high percentage of web applications have various kinds of security holes (or just bugs), and one of the biggest causes is failing to quote strings before putting them into an HTML page. See for example LWN: Cross-site scripting attacks.

Most people have now figured out this issue to the point of realizing that your application needs to quote text as it outputs it and providing a function to escape HTML, such as the PHP htmlspecialchars or any number of locally written versions such as the MIFOS xmlEscape in MifosTagUtils.

However, requiring people to remember to call it is error prone. A better approach is that HTML is one thing, and strings are another. So inserting a string into an HTML document will quote it. A few template systems do this (tinytemplate, Amrita, probably a few others). Most HTML-generating libraries (builder, DOM, XmlStreamWriter, etc) do it. If you are using an older template system, like JSP, ASP, PHP, etc, start thinking about how to migrate to something which is secure by default. Here's an article about these issues in Rails (recommending builder instead of rhtml).

Recursive descent parsers

2007-02-14T10:41:00.000-08:00

Many of us have occasional, rather than frequent, need for parsing a complicated text format (such as a programming language - I'm talking about things more complicated than, say, XML or comma separated values). So often the first step is trying to remember how parser generators work (a parser generator being a tool which takes a grammar and produces a parser, such as byacc, ANTLR, or SableCC). The latest one through this process is Martin Fowler. Through most of my career I've occasionally struggled with ANTLR and yacc, and only recently have come to the conclusion that I'm best off where many of us started: with a hand-written recursive descent parser.

If I lose you with any of the parser jargon, you might need to look at the Dragon Book, but I'm trying to keep it to a minimum.

Anyway, recursive descent is good because:

* You can accomplish everything in your Integrated Development Environment, or pre-existing ant build file. Parser generators, as with other kinds of generated code, impose extra steps every time you modify the grammar (extra manual steps and/or extra build scripting).

* Can accept a wide range of grammars. The only catch is that you need to eliminate left recursion, but in my
experience the solution of a while or do-while loop ends up being even more elegant than the left-recursive grammar you started with.

* Very nice to unit test. Your unit test can call into any production of the grammar. With most parser generators, you can only easily call the top-level production, and then you need to dig around in the tree it returns to find what you are supposed to assert on.

* Easy to understand and debug.

* Gives you complete flexibility to decide how you want to handle trees/actions/etc (often, building up domain/command objects from within the grammar will work well, with no extra tree layer. Depends on the problem I suspect).

* Provides refactoring and navigation tools for the grammar for free. What productions refer to this rule? Just hit "find usages". How do I make a separate rule from part of this rule? Just hit extract method (and you know it will still work - the ANTLR/yacc transformation which looks like extract method is not correctness-preserving in my bitter experience).

Anyway, if you are still with me and want to see how this works, the next step is probably to look at the SQL parser that I wrote this way for Mayfly: the tests (some of them) are at ParserTest and the parser is in Parser.

Mayfly starts to get options, sooner than I thought it would

2007-02-10T19:15:00.000-08:00

OK, a few weeks ago I mused about whether Mayfly was going to need some options (in that case, to set how it handles data types). Well, most of the reasons I imagine for options still remain in the future: "some day we may need/want this". But I did start adding an option to Mayfly. What was the first option? Some profound difference of philosophy about the data model which SQL should present? Maybe the ever-popular "should SQL be more relational?" or the subtle and deep issues around handling of SQL NULL?

No, nothing like that.

It is for case sensitive table names.

The SQL standard, and all databases I know except one, say that table names are case insensitive.

I say, "all databases except one". MIFOS of course is using that one (MySQL). And it is worse than "MySQL makes table names case sensitive". MySQL makes table names case sensitive only if file names are case sensitive (typically Unix). The practical effect of this is that half our team has case insensitive filenames and half has case sensitive ones, and the first group is often accidentally breaking the build (but only for the second group).

I thought a bit about various solutions, and there's a lot to be said about just having Mayfly run in case sensitive mode (on all platforms). So yeah, the CVS version of Mayfly has a method in Database called tableNamesCaseSensitive. Give me a few more days and most of Mayfly should honor it (large parts already do).

Yet another example of how it is hard to anticipate what will really be important (along with familiar cases like prioritizing software features only once you see what users look for and miss when they try to use a prototype, or only doing performance tuning once you have measured where the bottlenecks are).

Follow your nose

2007-01-29T12:15:00.000-08:00

Today's tale concerns what happens when we see code smells, and the twisty path we sometimes follow between getting a whiff of something, and reaching the real problem, and/or solution.

It started innocently enough. I was going through the compiler warnings from Eclipse, cleaning them up. Most of these fixes are improvements but not especially big or difficult. For example, getting rid of an unused variable, running the "organize imports" tool, adding missing @Override annotations, etc. But then I saw one which was clearly pointing to something bigger. A simplified version of the code with the warning is:


int FLAT_INTEREST = 1;
int DECLINING_INTEREST = 2;
createLoan(FLAT_INTEREST);

and the warning was because DECLINING_INTEREST was never used. Now, even without going through code archeology I can pretty much guess the sequence of events:

* Developer one creates a method


createLoan(int interestType)
// 1=flat, 2=declining

and various calls to it of the form


createLoan(1);
createLoan(2);

* Developer two sees some of these calls, and in the process of trying to read this code, or write something similar, wants to make the code better say what the parameter means:


int FLAT_INTEREST = 1;
createLoan(FLAT_INTEREST);

She (or the next developer) also figures out that "flat" is as opposed to "declining" and that two means declining. Hence the:


int DECLINING_INTEREST = 2;

All of this is all well and good. Sure, it is easy to find fault with the idea that local variables are a good place for these constants, but the local variable is better than what we had before - createLoan(1) - and we should be willing to improve code clarity one step at a time, rather than trying to solve everything at once.

A next small step could be to turn these local variables into constants in some central place, but I instead looked at the implementation of createLoan. It was something along the lines of:


createLoan(int interestType) {
  return new Loan(
  InterestType.fromInt(
    interestType));
}

That is, this int was getting turned into an
enum anyway. Well, once we see that there is already an enum, we realize most of the job is already done, and we end up with something like the following:


createLoan(InterestType type) { . . . }

createLoan(InterestType.FLAT);

Well, actually there were too many callers to convert everything in one step, but I converted some, and kept the createLoan(int) method as a transitional aid, as I've described previously.

(For those people wanting to see the actual code in MIFOS, the code which had had the warnings was getLoanAccount in CollectionSheetHelperTest, the method which I called createLoan above really is createLoanOffering in TestObjectFactory and the enum is InterestType).

Reviewing code: in-person versus written

2007-01-22T08:41:00.000-08:00

It has become a fairly widespread belief (although somewhat less often practiced), that software benefits from having other developers look at it. Such code review is designed to find flaws, seek out forgotten corner cases, improve code style, and any number of other improvements.

One of the key questions is whether to do this in writing or face to face. Most open source projects rely on written feedback, with the most two common forms being (a) send a patch to a mailing list, and get a description of what could be improved (the mozilla code review system is particularly formalized), or (b) check the code in, and other developers look at it there (if not as a deliberate effort to review it, then in the course of their own work). By contrast, a face to face review can be anything from a formal event which sits a bunch of developers in a conference room with the code on a projector, or the back-and-forth of pair programming, or anything in between ("hey, could you look at this?").

As it turns out, this morning I got two unrelated examples of how the written method can be frustrating.

The first story starts a week and a half ago. A new MIFOS developer had checked in some code. Not surprisingly for a new developer, it had a variety of problems both large and small. I sent an email describing some of the ones which seemed most important and/or easiest to find (oh, and reverted the checkin because it broke the build, but fortunately that's much easier with subversion than it would have been with CVS). Today the developer checked in a fixed version. It certainly is improved (I'm pleased to say the unit tests pass). I noticed a variety of smaller things. Now I have to figure out which ones to fix myself and which ones to complain about. In the second case am I going to wait another week and a half to get a reaction? And how do I try to calibrate how good is "good enough" and what we should worry about later? Now, those are valid questions no matter how the suggestions are delivered. And a one-week feedback loop is better than waiting many months, until the developers thought they had something which was "finished". But it is certainly harder to come to an understanding, or build up a team set of practices, if each little point is going to require a number of back-and-forth emails or phone calls.

My second example is a much smaller one, which actually makes it easier to see the point without getting hung up in larger issues of how you manage a project. A well-known columnist wrote an online article which had a typo in one of the examples. I wrote in and said "you have an extra curly brace on the last line; perhaps you copy-pasted it from a few lines up". He wrote back and said "no, the curly should be there" (thinking of the instance a few lines up, which was similar in syntax). This caused me to pause a bit. "Well, I'm pretty sure I'm right, but is it worth worrying about?" and even "how can I express this most clearly, because obviously what I said at first needed too much re-reading for a busy author to do", and so on. Well, I did write the second email ("no, the one at the end" or however I said it) and the author wrote back that I was right and fixed it (no doubt feeling at least a touch sheepish).

Not an especially big deal. But I contrast it with what would have happened if this had happened while pair programming. I would have said "oh, there's an extra curly". Perhaps I would have just taken the keyboard and removed it. But the worst case would be closer to "no, it matches the one up here" "not that curly, this curly (pointing with finger/mouse/etc)" "oh yeah, you're right (reaches for keyboard and deletes it)". Not only would it have gotten fixed faster, but with a minimum of tension (not that pair programming is immune from tension, but that's a subject for another article).

Is SQL a strongly typed language?

2007-01-10T11:22:00.000-08:00

A strongly typed language can mean different things, but here I look at how SQL stacks up against some of the definitions of strong typing (and to keep it practical, what different SQL implementations do and do not let you do).

* Static typing as opposed to dynamic typing: SQL is statically (strongly) typed in this sense; each column has a type.

* compile-time checks for type constraint violations: Here we need to define the difference between compile-time and run-time in SQL. Basically I would call "compile-time" to be the call to Connection#prepareStatement and "run-time" to be the call to PreparedStatement#executeUpdate. An alternate (probably mostly equivalent) definition is that a "compile-time" check happens even if there are no rows to operate on. A browse through the Mayfly testsuite for "rowsPresent" flags will show cases in which SQL implementations differ on whether a particular check is compile-time or run-time, although the popularity of query optimizers tends to mean that checks happen at compile time (in those cases where I've checked; most of the Mayfly acceptance tests don't distinguish the two cases).

* complex, fine-grained type system: SQL is more fine-grained than systems of the "everything is a string" variety (there are different syntaxes for '5', 5, 5.0, and x'5'), but only recent versions of SQL try to add things like structure/record types.

* omit implicit type conversion. The databases I've tested (Mayfly,MySQL,Postgres,Derby, and Hypersonic) all refuse to read 'foo' as zero (if looking for an integer). All tested databases allow storing an integer into a DECIMAL column (that is, you can say INSERT INTO FOO(x) VALUES(5) and you don't need to say 5.0 even if x is of type DECIMAL). There are also plenty of cases in between (for example, INSERT INTO FOO(x) VALUES(9) into a string column, which works in Hypersonic, MySQL and Postgres, but not Derby or Mayfly).

Anyway, I could go on, either with gory details of what does and does not work (for which I'm better off just having you look at the the Mayfly acceptance tests), or with more general philosophy on types (which dictates what kinds of cases to look for), but I'll cut to the chase: What will be most useful for Mayfly users?

For now, I am generally leaning in the direction of making Mayfly picky - it seems better to catch any errors early (when writing/running tests), rather than later (when trying to deploy on different databases, for example). It is my experience so far that MIFOS doesn't seem to play loose with the type system (which is probably mainly a reflection on Hibernate), so I feel somewhat vindicated in this judgment. As with many things for Mayfly, I realize there are other situations (most notably, if you want to Mayfly-ize an existing application without having to modify it). So the question is whether Mayfly should be opinionated software? I'm generally of the mind that software works best with a clear idea of what it is supposed to do, and that software which tries to accomodate every possible answer to "how should X work?" tends to just get a bunch of poorly-thought-out configuration choices, none of which end up being quite right (for any given situation, or opinion). On the other hand, I'm assuming that Mayfly sooner or later will have some kind of "opinion manager" where you can pick, say, "please enforce the practices considered best by the mayfly developers", or "maximize MySQL compatibility" or even define your own (much like the code formatting configuration in IDEA or Eclipse).

Whether strong typing is a case for options, I don't know, however. Fear says that of course things would break and we need an escape hatch. But I am beginning to wonder whether that breakage is small enough to just wait and see whether this becomes a problem.

new Integer(5) versus Integer.valueOf(5)

2007-01-04T11:40:00.000-08:00

Seems that findbugs warns you if you call new Integer(5) instead of the (new with Java 1.5) Integer.valueOf(5). The point of the latter is that it might return you an existing object rather than creating a new instance.

I'll get back to the Integer.valueOf case, but on the general topic of trying to avoid object creation, there has been a long and largely unhappy history of this in Java. For example, see Allocation is faster than you think, and getting faster.

To summarize the possible problems with object caching and pooling:

* One can accidentally end up sharing a mutable object where the simple design calls for an unshared object. One way to avoid this problem is just to use immutable objects, for example joda-time objects instead of java.util.Date objects. In the Integer.valueOf example, Integer is immutable, so we don't have this problem.

* Pooling almost always complicates the code. Not so much of an issue for Integer.valueOf, in the sense that the standard library has the extra code, we just need to figure out whether to call it.

* Object pools can cause synchronization bottlenecks. There are of course complicated solutions, like separate pools for each thread. In the Integer case, this is Integer.valueOf's problem (and the Sun J2SE 1.5 implementation solves it by just allocating a fixed size pool on startup).

* Object pools tend to increase memory consumption. Often the performance hit of chewing up extra memory (for a long time) will exceed the allocation/deallocation overhead (which may involve a short-lived object, the cheapest kind). Again, in the Integer.valueOf case that's someone else's problem not ours (and in the Sun J2SE 1.5 implementation anyway, the size of the object pool is fixed at JVM startup and won't change based on anything we do).

So having exhausted the usual arguments against object pools, I conclude that it is in fact a good thing to call Integer.valueOf instead of new Integer.

Open Source, Free or Libre?

2007-01-03T12:53:00.000-08:00

It might be just flamebait to even mention it, but those of us who are writing software which conforms to the Open Source Definition or the Four Freedoms (most often both) sooner or later need to decide whether to call it open source software, free software, or libre software, with the most controversy tending to surround whether people are trying to water down the meaning of open source (for example see What is open source?).

The only reason I bother to write about this is that Martin Fowler, in his Semantic Diffusion article, points out that this is completely par for the course when a concept is getting popular. He has some great examples of this (I am old enough to remember how for a time everything was described as "object-oriented").

So if open source is like the other terms Fowler discusses, there isn't a need to start a big panic. As long as we have a critical mass of people using the word open source to refer to software meeting the open source definition, the odds are good that the meaning won't drift too far and too permanently.

Database upgrades

2006-12-19T16:31:00.000-08:00

Well, my project for the last few weeks has been database upgrades. This concept has been popularized by rails and to a certain extent the Database refactoring book, but it is something that just about any project could do, without a whole lot of mechanism. It only took us (myself and Stelligent) a few days to implement an automated upgrade scheme.

I guess the best way to describe the design is just to point to the Wiki page.

Some of the key decisions were/are:

(1) sql vs java. Is an upgrade script just a .sql file, or is it written in a real programming language like Java/ruby/perl/etc? We went with the .sql approach because it seems like the Simplest Thing Which Could Possibly work, and it seems like introducing a programming language could get needlessly complicated and/or hard to understand. Having said that, I do realize that we may someday hit a situation where it is really awkward to process data via SQL. But I'd rather cross that bridge when we come to it, given how many upgrades so far have been very simple (insert default value into new column, create new table with no rows, etc).

(2) Whether to provide upgrade scripts for every checkin which changes the database schema, or on coarser boundaries like releases? The latter is
sufficient if the goal of this exercise is merely to allow production users to upgrade. Requiring developers to blow away their test data can be done, but the real question I have there is how is a developer supposed to know when they need to do this. I've been on projects where a significant amount of the chatter in the team room is "is anyone else seeing a failure x?" "rebuild your database". This doesn't really seem great even if everyone is in the same room, and on a distributed project like MIFOS, it seems to me quite important to at least detect an out of date database on developer machines (and once you've done that, might as well just do the upgrades).

(3) Whether to provide downgrade scripts. The purpose, I guess, is to let someone try out a new version of the software knowing they can always go back if it didn't work out. Given all we are trying to do in terms of having tests, continuous integration, etc, to make upgrades go smoothly without needing to roll back, I'm not sure how much effort to put into downgrades. On the other hand, it may be hubris to think downgrades will never be needed.

(4) What tests to build. The interesting tests in the MIFOS case are in LatestTest. There are tests of the upgrade scripts (checking against latest-*.sql), the downgrade scripts (checking each one undoes the corresponding upgrade script). We're also thinking about some tests which would test that the upgrade scripts properly upgrade existing user-supplied data (as opposed to just schemas and data supplied with MIFOS). But this post is getting long already, so I'll leave that for another time (or for people to ask about).

MIFOS programmer intern position in Washington, DC

2006-12-14T17:38:00.000-08:00

If you are a programmer looking for an internship in Washington, DC, check out Software Development Intern, Mifos. Much of the time would be spent pairing with me and/or working on some of the things I write about here.

SQL dumps and topological sorts

2006-12-14T13:52:00.000-08:00

A few weeks ago I wrote a SQL dump utility for Mayfly. That is, take a database and dump it out as SQL statements, capable of re-creating the database. I played around with it for a while, thought I had it in a fairly complete state, and didn't think about it much until I was ready to use it (for some database upgrade work which I should write about too, but not now).

Turns out I forgot a key step: try out the dumper on some reasonably large and/or real world data set. In this case, the data set is the MIFOS SQL schema (and master data checked in, which has various records which the application itself needs). And the test is a fairly simple, automated one: dump out the data and try to reload it. This test worked fine with some small SQL scripts I wrote for the Mayfly test suite, but when I dumped the MIFOS schema/data, it failed because the foreign keys were out of order. That is, in the schema, a foreign key has to refer to a table which already exists (constraining the order in which we dump CREATE TABLE statements), and a row has to refer to a row which already exists (constraining the order in which we dump INSERT statements).

Sounds simple, right? Just sort the tables, using a java.util.Comparator which returns -1 or 1 based on whether there is a foreign key between the two tables, right? I'm slightly embarrassed to admit that I actually implemented this, found that it worked on a few small test cases, noticed it failed on MIFOS ("gee, that's funny, I'll need to look into that"), and proceeded to other problems (my immediate need did not demand that the SQL file be reloadable into a database, just that it represent what is in the database).

So here's an exercise for the reader: what was wrong with my implementation? (feel free to answer in the comments section if you want).

Anyway, after some research on Wikipedia, I realized that what I needed was a topological sort (and, fortunately, that I didn't also need something like the Floyd-Warshall algorithm).

Are there libre Java topological sort libraries out there? I didn't look very hard, but didn't see any (on apache commons, I saw something in the "commons sandbox", but it wasn't clear that it was ever finished, or that it even is still there).

I took an hour or two to understand the topological sort algorithm in Wikipedia, and another two hours or so to implement it, and it will probably be another hour or two to have the SQL dump utility call it, so I don't really have a pressing need for a library any more. That is, unless I find maintaining my implementation ends up being a lot of work, which could happen although I suspect I'm over the hard part in just getting to this point. It is one of those algorithms which seems scary until you understand it, but pretty simple thereafter. It's only a hundred or so lines of code and about twice that for tests (depending of course on what you count as the topological sort code and what you count just as setting up various things to send into it).

I was a little bit surprised that a topological sort was so unfamiliar to me. I guess this just means that we don't need it all that often, although when we do need it, we need it bad. By "need it bad", I mean my suspicion is that no amount of ad hoc algorithm building of the sort of "well, these two tables are out of order, let's try switching them" is likely to converge on a working, fast-enough algorithm. Not that I tried the ad hoc approach (unless you count my Comparator misstep). I wouldn't say this example undermines my faith in agile practices like evoluationary design or "running code speaks louder than words" (see for example the line "Too much talk, not enough code. Type!" in the bowling score pair programming example). But it does indicate that there are times when it pays off to stop coding long enough to go read up on how others have solved similar problems.

Needles, haystacks, and log4j

2006-11-29T12:01:00.000-08:00

There was an interesting bug report that came in to the MIFOS mailing list recently.

Someone was posting that they couldn't start MIFOS (that's not the interesting part). The interesting part was: "then I am Getting so many errors, List of the Errors is as follows." and lots of log4j output (in fact, too much for the mailing list archive program to show it all, so you'll need to take my word for what was there).

Most of the log4j output which was INFO messages which meant nothing at all was wrong. The trouble started with a WARN which started "org.hibernate.cfg.SettingsFactory -
Could not obtain connection metadata"
and proceeded with a stacktrace. Then another 19 or so INFO messages (not related to the error, as far as I can tell). Then another WARN, this one even more cryptic than the last: "org.hibernate.util.JDBCExceptionReporter - SQL Error:
1045, SQLState: 28000". Then finally an ERROR which fairly directly said what was wrong: "org.hibernate.util.JDBCExceptionReporter - Access
denied for user 'root'@'localhost' (using password:
YES)".

In other words, this was a simple problem (the database user and password that had been supplied to MIFOS were not set up in MySQL) but the actual error message was buried in some 1500 lines of red herrings.

It's no wonder that software gets a reputation for being hard to install/configure/run, when tracking down the simple problems involves this level of looking for a needle in a haystack.

For MIFOS, the low-hanging fruit seems pretty clear: make sure the default log4j logging level is set to WARN (in fact, I would have changed this already, except I couldn't find where it is being set - which is another good log4j rant but one for another time). Then all those INFO messages wouldn't be there. Bonus points would be given for: (1) reporting the real error once instead of 3 times (probably best done within Hibernate), and (2) making it so that one can go to localhost:8080/mifos (that is, the URL which would have had the application, had it started) and see an error message (or at least a hint - like "application failed to start - see xxx for detail").

SQL DELETE of all rows not as easy as you'd think

2006-11-16T15:32:00.000-08:00

So clearing out the contents of a table from an SQL database is a relatively common operation. Tests might do it to start from nothing, or MIFOS's own testdbinsertionscript.sql does it so that the tests can have some sample data which is a bit different than what we supply for production.

Sounds simple, right? Just execute:

DELETE FROM TABLENAME

And in fact that works most of the time.

But there is a fairly common case in which things
might not be quite that simple. Suppose that each row of the table points to a parent. For example:

create table foo(id integer primary key,
name varchar(255),
parent integer,
foreign key(parent) references foo(id)
);
insert into foo values(1, 'Eve', null);
insert into foo values(10, 'Seth', 1);
insert into foo values(101, 'Enos', 10);

(For the non-SQL-aware, the FOREIGN KEY stuff just means what I said in words - that the parent points to another record in the table).

Now in this case suppose we try to delete a row:

delete from foo where id = 1

This should fail, and does, because to delete the record for Eve would leave the record for Seth pointing to nothing.

But now try:

delete from foo

If the database deleted the records one at a time, and applied all the usual rules, then it might fail (depending on in what order the database processes the records). In fact that is what you see in MySQL, and the developers of MySQL have offered a way around this by adding an ORDER BY to their DELETE statement.

Hypersonic is much like MySQL, except it seems not to honor any ORDER BY.

Postgres and Derby, on the other hand are smarter: they just will delete all the rows (I don't know whether they look at foreign keys as a group rather than row-by-row, or what, but the observation is that the delete Just Works).

Right now Mayfly is like Hypersonic/MySQL, without the chance to specify ORDER BY. I guess the Postgres/Derby behavior is the right one (although I'll have to think about how to implement it - if it were a simple change I would have just done it, rather than all this whining). Somehow ORDER BY doesn't feel right to me. It seems to be based too much on a model of how delete is to operate, and not enough on what result delete is supposed to produce.

For now, I worked around this in MIFOS by first clearing the parent pointers and then deleting the rows:

update foo set parent = null
delete from foo

That could get complicated if one were not allowing NULL in this column. But for this situation, it seems like a pretty painless workaround (this particular test data setup isn't a performance bottleneck, so there is no need to worry about that).

Press mention in newsforge

2006-11-14T08:27:00.000-08:00

I make no attempt here to log all the mentions of Grameen or even MIFOS (especially since the Nobel prize), but here's one in newsforge: Microfinance and open source: natural partners.

Newsforge is one of the better open source news sites. I mean, no one can match LWN's Weekly Edition for relevance and good writing, but most of the time that I click on a newsforge article, I end up informed. Just to pick another example from today, their article about the reaction to Sun's Java plans is spot-on. It points to some relevant mailing list threads and avoids getting caught up in the hype.

Testing equals and hashCode

2006-11-10T15:20:00.000-08:00

This isn't a post about whether it is a good idea to implement equals and hashCode in all your classes, and if so how (check all fields, check some kind of identifier, check fields except the boring ones, etc).

No, I'm assuming that you have decided to implement equals and hashCode, either because you like working that way, or because you are using a package like Hibernate which encourages/requires it.

So now the question is: being good test-driven developers that we are, how do we write the tests for our equals and hashCode methods? Many of us have probably read the javadoc for Object#equals (the so-called equals contract), and started out
writing things like:

assertTrue(one.equals(two));
assertTrue(two.equals(one));
assertFalse(one.equals(null));

etc.

And that's about right. But it seems like this is a framework waiting to happen (well, framework is probably too grandiose a word for something which probably doesn't need to be more than a hundred or so lines of code and just affects tests for equals and hashCode, but hey, people have called things frameworks for less).

Are there any good ones out there in Apache commons or the other usual places? I've seen some really bad ones, but generally have just ended up writing them myself. I'm enclosing the one I'm currently using in both Mayfly and MIFOS.

The one thing it doesn't do super-well is test transitivity. You can give it a bunch of things which should all be equals to each other, and it tests that they all are, but it doesn't do any transitivity tests for not-equals. I think it is pretty clear how to fix that: instead of just passing in a bunch A of things equals to each other, pass in several bunches: A, B, and C. Each object within A should be equals to the others in A, but none of the ones in B and C. Likewise for B and C (the mathematically experienced of you will recognized these "bunches" as equivalence classes). In fact, I started to implement this today, and I got a bit hung up on whether it reads as nicely as what I have now. Somehow, passing in Object[][] { new Object[] { a1,a2}} just seemed like too many levels of [] and {} and such. I don't know if my concern is justified.


public static void assertAllEqual(Object[] objects) {
    /**
     * The point of checking each pair is to make sure that equals is
     * transitive per the contract of {@link Object#equals(java.lang.Object)}.
     */
    for (int i = 0; i < objects.length; i++) {
        Assert.assertFalse(objects[i].equals(null));
        for (int j = 0; j < objects.length; j++) {
            assertIsEqual(objects[i], objects[j]);
        }
    }
}

public static void assertIsEqual(Object one, Object two) {
    Assert.assertTrue(one.equals(two));
    Assert.assertTrue(two.equals(one));
    Assert.assertEquals(one.hashCode(), two.hashCode());
}

public static void assertIsNotEqual(Object one, Object two) {
    assertReflexiveAndNull(one);
    assertReflexiveAndNull(two);
    Assert.assertFalse(one.equals(two));
    Assert.assertFalse(two.equals(one));
}

public static void assertReflexiveAndNull(Object object) {
    Assert.assertTrue(object.equals(object));
    Assert.assertFalse(object.equals(null));
}

Mayfly SQL Dump produces SQL that Mayfly can read

2006-11-08T15:55:00.000-08:00

My project of the last few days has been to write a dump utility so that Mayfly can output a database in SQL (similar to mysqldump and similar tools provided with most databases).

Mayfly's dumper can now output CREATE TABLE statements with all of Mayfly's current data types, and likewise INSERT statements for the rows.

So the milestone is that I can now take the standard MIFOS data from the unit tests (DatabaseSetup#getStandardStore()), give it to the dumper, load that dump file back into Mayfly, dump it again, and the first and second dumps will have identical contents.

Now, if the dump just leaves out parts of the data/metadata (as it currently does with constraints, auto-increment values, and binary columns), then this test won't complain (the first dump will omit something, and the reload will just load something different). But it still seems like the dumper might not be too far from finished: this test at least implies that the dumper doesn't blow up on anything in the MIFOS data/metadata, and doesn't generate any invalid SQL.

First MIFOS unit tests pass with Mayfly

2006-11-06T08:55:00.000-08:00

(This was actually from 2 Nov 2006)

So, one of my main projects lately (last 2 months or so) has been getting the MIFOS unit tests to work with an in-memory database. For a while the task was just to get Mayfly to read the MIFOS SQL files (mifosdbcreationscript.sql and mifosmasterdata.sql) - I could measure progress by how many lines into the script before Mayfly gave an error.

After that, the task was to get Hibernate to talk to Mayfly. This was considered successful when a simple Hibernate call could get an object from data which had been in the database (I later found out that there were other corners of Hibernate I needed to worry about).

Then there was running a MIFOS test (one of the existing unit tests, which have been running with MySQL until now). I started with FeePersistenceTest (chosen more or less at random).

First step was making it through the initialization code in TestCaseInitializer. This mostly just worked, but there was one interesting surprise. There was a join of 80 rows by 500 rows by 500 rows (written with implicit joins and WHERE, not INNER JOIN and ON), and that was too much for the naive "build the cartesian product first and then start applying WHERE conditions" algorithm that Mayfly had. Now, one can argue that a unit test should be whittling down its dataset, and that might be how we end up going, but one of my ideas for MIFOS and Mayfly is to see how far we can get while avoiding some of those familiar unit testing slimmings. (As another example, if I run into a piece of MySQL-specific SQL, I tend to rewrite the SQL to be portable, or add the feature to Mayfly, rather than build an abstraction layer which lets MIFOS generate different flavours of SQL). Anyway, back to joins. I built a simple query optimizer which got me past this.

Oh, yeah, and there was all the ALTER TABLE work I did so Mayfly could execute some/all of the Iteration*.sql files (as it turns out, I'm not sure I needed this quite yet, but I should soon).

So next various things failed as FeePersistenceTest created its test objects and such. I've been fixing those one at a time. In fact, I've
been beginning to worry about how much work might be left, given that I don't have any particularly good way to estimate how many of these features remain. Well, this morning I saw an odd symptom - instead of the usual 6 failing tests, I only saw 4. That's right, 2 had passed. Looking at what had failed, I saw 2 easy features to implement on
my laptop at lunch, and once I checked that in, all 6 were passing!

Now, when I tried running CenterBOTest (second test picked at random), there were a whole new set of failures. But still, to be over the FeePersistenceTest hump is quite exciting.

Enums are a good thing

2006-11-04T04:10:00.000-08:00

Yesterday I dove into the tests looking for something to clean up. I started with the NonUniqueObjectException we're getting in one test (and swallowing), but in the process of trying to look around to see what the two objects might be that make uniqueness not exist, I found other code smells.

So I'm looking at code which (simplified) looked something like:

createClient(Short.valueOf("3"), "A test client")

The pain involved in using short instead of int is the first glaring thing, but actually what that really should have been was an enum:

createClient(CustomerStatus.CLIENT_ACTIVE, "A test client")

If those two things look basically the same to you, I'd suggest thinking a little harder about where you are spending your brain power while reading/maintaining this code. Sure, once you've come up to speed you can probably remember that "3" here means active, but shouldn't you have the computer keep track of that? And if you are just learning this code, or forgot that detail, then "3" is totally mystifying - in fact what got me onto this tangent is that I was wondering whether it was an ID which, duplicated, had something to do with the Hibernate non-unique exception.

One more detail: how did I fix this? The createClient method had about 150 callers (fortunately with good test coverage). So I didn't want to fix them all at once. I created my new createClient:

createClient(CustomerStatus status, String name)

and had it call the old one (or maybe vice-versa, the point is having one call the other rather than a copy-paste, since it is so easy to look up the enum from the short, or vice-versa):

createClient(short status, String name)

I then started fixing up callers. I think I got to about 100 before I got bored. So I checked in the 100, and I can get to the other 50 some other day.

I suppose I could also turn this into a rant about how helpful Java's strong typing is, because with the enum I know (as I'm typing, thanks to Eclipse, not just at run-time) what that first argument to createClient is. But that's a debate which goes back at least to the 1960's. I'll just say that since we are paying the price (extra syntax, mainly) for compile-time types, we should get the payoff.