Yesterday I saw a drinks machine that had three selections – Tea, Coffee or Random (Tea or Coffee).  However, the machine was wired up wrongly so that each button does not give what it claims. If each drink costs 50p, what is the minimum that you have to put into the machine to work out which button gives which selection?

What I like about this is that it makes more natural several of the constraints of the original puzzle (a crate of apples, a crate of oranges, and a mixed crate):

• Why can’t I just peer into the crate and look at the contents?
• Why only draw one fruit at a time?
• Why do I care how many fruits I have to sample?

It also makes more sense of the effectively-unlimited supply of fruit that is implied by the question: if it’s the wiring that’s at fault and not the hopper of tea / coffee, you could refill it as many times as you want without affecting the wiring.

The puzzle, of course, hinges on the listener not paying attention to the crucial fact: the wiring for each button (or the labelling on each crate) is known to be wrong, not just unknown. It’s one of the neat things about this puzzle that an intelligent solver may well mishear it, but can easily prove it’s impossible if the wiring is entirely unknown. In point of fact, it’s a fairly simple to show that if it is possible, then the answer must be one: we can’t necessarily distinguish a random stream from a constant stream with any number of samples. Constructing a proof that one sample will suffice is fairly easy, given that there are only three possibilities of which to sample, and by symmetricity only two possibilities that differ.

Does this make a good interview question for a software developer? I’m not at all convinced it does. It feels like a meta-puzzle: you reason through it by knowing the implicit rules of the game of puzzle-writing (one obvious one being that the answer can’t be impossible, and it can’t rely on luck or “good enough” solutions like 1000 random samples). Good software engineering often involves judgment calls where such rules don’t exist, and great software engineering often involves rethinking the rules of the game entirely.

It seems odd that having immunity to a disease should drive up my costs for getting vaccinated for another disease. It seems downright bizarre that my making it easier (or at least, no harder) for the government to deliver the free service should mean I have to pay. It would be fashionable at this point to rail against the fundamental inefficiency of government-provided healthcare.

But I don’t think that’s what’s happening here. The mistake the NHS seem to be making here looks like the same sort of mistake that is made every day by thousands of software developers. The set of rules for Hepatitis has a bug in it (no pun intended).

It seems to me that in this case the bug is a leaky abstraction. The obvious goal is to provide immunity to both diseases, and a policy of providing a combined vaccine looks from a high level as if it fulfils that goal. When the details of the policy are implemented, an obvious refinement is not to provide a combined vaccine to someone who’s already had one shot (whether this is a matter of health or cost-saving I don’t know, but it doesn’t affect the argument).

Obviously this is an over-simplification; looking at the world through the eyes of a programmer, things naturally form into shapes and idioms that are common currency to the techie. But I think this there’s something to be gained from seeing the world this way. When people try and build rule sets, they get things wrong. Not because they are stupid, or unwilling, or corrupted, or bureaucratic, but simply because creating perfect rule sets isn’t something that humans are naturally capable of.

The punch line? Bob’s email hadn’t been working all morning, and nothing I sent had got through to him. Bob’s emails read to me as if he’d got my instructions and somehow failed to follow them correctly, so it looked to me like he was being inept. From Bob’s point of view, I was giving him stony silence that could only be explained by a lack of motivation.

I’m sure this kind of problem has been plaguing the interactions between technical and non-technical people for as long as there has been technology to bicker over. Users complain about bugs that don’t get fixed, while techies whinge that the users misdescribe their problems, foul things up further with misguided attempts to fix them and then fail to follow instructions when they get them. Why isn’t it going away as people get more tech-savvy?

Behaving correctly when you encounter a fault and reporting it in a productive way is a reasonably subtle task; Simon Tatham’s piece on how to report bugs effectively is a very approachable and effective guide, but it’ll probably only ever be read by the motivated. Even so, you might think that sheer trial and error would teach people that certain ways of reporting a bug get the solution they’re after more quickly than others.

Some people suggest that we techies are lacking in communication skills, and that we need to work harder at making ourselves understood in order to engage with the modern workplace. This isn’t entirely without merit, but I think it’s facile. I’ve been lucky enough to work with some very articulate developers, and none have been immune to this kind of misunderstanding.

I think there’s something deeper going on here, and I think it lies in psychology. The fundamental attribution error is the observed effect that people tend to attribute other people’s behaviour disproportionately to their character, neglecting the possibility of simple situational explanations. For example, I attributed Bob’s inability to solve his problem to his not taking the time to read my instructions properly, or being too poorly-skilled to put them into operation. All the time I was building up in my head a picture of Bob as an incompetent, over-confident bumbling oaf who shouldn’t be trusted to go near a keyboard—and all because his email wasn’t working.

But it goes the other way too. I’ve no doubt Bob had painted a picture to himself of me as an uncommunicative nerd who didn’t appreciate the importance of Bob’s project to the business. Neither of us even considered that there might be an alternative explanation.

The thinking that leads to the fundamental attribution error makes some sense, when you think about it. When you’re dealing with someone on a casual basis you can come to great harm by not making an assessment of their character, and assuming that any flaws you observe are likely to hurt you again in the future is a sound defensive position. This might lead to missed opportunities in casual acquaintances, but that could be a price worth paying.

This thinking is catastrophic, though, when it becomes an entrenched pattern that prevents the very nature of the problem from being properly understood. Blaming developers for being aloof, or users for being incompetent, only reinforces the stereotypes. Working on our communication skills only helps if the thoughts we communicate are based on the right model.

public class HigherLevelClass
{
   public event SomeDelegate MyEvent
   {
      get { return impl.MyEvent; }
   };

   private LowerLevelClass impl;

   /* ... */
}

Unfortunately this isn’t valid syntax, and at first glance that seems to be the end of it. However, I discovered today that the designers of C# have actually thought of this, and provided a way round it:

public class HigherLevelClass
{
   public event SomeDelegate MyEvent
   {
      add { impl.MyEvent += value; }
      remove { impl.MyEvent -= value; }
   };
}

Simple, once you know how. But this illustrates an interesting problem with keyword search when applied to programming techniques: I’ve searched for just this information on several occasions, but wasn’t able to find it until I hit on the magic phrase “event forwarding”. In my experience it’s pretty common to know how to describe what you’d like something to do, but not be able to find the same words as the person who knows how to do it.

I’m sure this is far from the only reason, but perhaps it goes some way to explaining why there are eight separate implementations of every single open source library type, each of which is partially complete.

I wonder whether one of the unspoken reasons for this hatred is that at one point or another almost everyone has ended up doing this sort of thing:

   builder.Add("SELECT foo FROM bar WHERE id = ");
   builder.Add(id.ToString());

   if (additionalConstraint)
   {
      builder.Add(" AND frobbable = 1 ");
   }

   /* ... ad nauseam ... */

SQL is a hard language to like: it’s never been properly standardised (or rather, it has, but the standard has never been implemented) meaning that you spend too much time worrying about compatibility. Its theoretical underpinning is poor, leading to constructions that are hard for the engine to optimise (meaning more manual work).

However, SQL is a language in its own right, and was never intended to be generated programmatically by another programming language. This shouldn’t come as a surprise, as I struggle to think of any programming language that has been designed to work in this way.

Using SQL from a decent command-line environment is a powerful tool and often a pleasure to use. By comparison, generating SQL programmatically is an abomination that would be worth of The Daily WTF were it not for the fact that nobody’s ever invented an API that offers the same flexibility.

Personally, I blame the vendors. Until RDBMSs can offer the same quality of optimisation that modern compilers can (that is, write in a high-level language and never even think about micro-optimisation) high-performance relational database access will remain a sea of vendor-specific optimiser hacks. Maybe there’s a theoretical reason why optimisers will never be this good, in which case perhaps we do need to abandon the relational model in practice. But let’s not pretend it has anything to do with SQL.

Problem 8 is as follows: Find the subsequence of 5 consecutive digits that yield the greatest product when multiplied together, in the 1000-digit number:

73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450

I was able to come up with an F# solution that is one line, plus a helper line to convert the string into a sequence of digits:

let str = "731<...>"

let digits = Seq.map (fun x -> int (Char.GetNumericValue x)) str

let maxproduct num list =
 Seq.max (Seq.map (fun x -> Seq.reduce (*) x) (Seq.windowed num list))

The value digits is just a list of the digits in the string, converted into integers. The function maxproduct works the obvious way: take every subsequence of five digits (Seq.windowed), multiply them together (Seq.reduce, applied to each element of the sequence with Seq.map) and then find the maximum (Seq.max).

The only reason this needs quite so little work is the existence of Seq.windowed in the standard library, which does exactly the right thing in turning a 1000-element list into 996 arrays of subsequences of consecutive digits.

I’m not sure I like ramming all the functions into one line, and I’m sure there must be a way to combine map and reduce without the lambda, which adds a lot of clutter. If this was real code, it would need quite a lot of work to make it readable. However, the standard library is a big win, because the process of ‘windowing’ a sequence is nicely separated from the code. It’s also nice (for toy problems like this, at any rate) that the program is pretty much a definition of the problem, with little thought being necessary as to how to do the processing.

The simplest explanation

Put simply, a “relation” is what SQL calls a table. If you learn nothing else about relational theory, at least understand this. This is an oversimplification of course, but it’s close enough to being true that if you don’t want to learn any theory, it will at least make the discussions of theorists more comprehensible.

The mathematical explanation

This isn’t the best way to understand what a relation is, but if you intend to have meaningful discussions with other practitioners, you will need to have a common understanding based on a definition that is unambiguous. I’ll therefore get the mathematical explanation out of the way here; if it doesn’t make much sense, return to it after the more intuitive description below. Though this is a relatively formal description, it doesn’t come close to being totally precise, and anyone who wants to know more is encouraged to investigate a book on the subject.

An attribute is a combination of a name and a  type identifier, where we can for the moment treat a type as being a (possibly infinite) set of values with some operators defined on it. Think of an attribute as being like a column definition.

A tuple is a set of distinct attributes, where each attribute is associated with one value that is an instance of the type for that attribute. The members of a tuple do not posess an inherent order, and tuples ordered in different ways for display purposes nevertheless represent the same tuple.

A relation consists of a heading and a body. The heading is a (possibly empty) set of attributes with distinct names. The body is a (possibly empty) set of tuples, each of which has the same set of attributes as the heading of the relation.

To put this in terms familiar to an SQL user: an attribute is analogous to a column definition, a tuple is analogous to a row and a relation is analogous to a table. Note that this is an over-simplification, mostly because we think of the rows and columns of a table as posessing an inherent order, and mathematical relations have no such order.

One other thing that bears stating at this point is that a relation is technically an immutable value, and is held in a mutable variable called a relvar. This is analogous to common programming practice where an integer like 5 is immutable, but held in a mutable integer variable. If you “insert a row into a table”, then actually you change the contents of that relvar from one relation to another. This distinction is rarely of relevance in discussing theoretical issues.

The intuitive explanation

Unless you’re already familiar with relational theory, that was probably all rather unclear, in which case the only vital things to take away are: columns are unordered, and rows are unordered. If you are familiar with relational theory, you’re probably angry at me for making so many mistakes, in which case please point them out in the comments.

So what does this mean in intuitive terms? A common intuitive feeling about tables is that they represent a list of entities, and indeed this understanding works nicely for simple cases. Take a table of salaried employees in FictoCorp:

The head of HR for this company might look at the table and say, “yep, those are my employees—I’d recognise ‘em anywhere.” As far as they’re concerned, each row in this table represents one of the employees they have to deal with. Furthermore, no row represents more than one employee, and there’s no employee of the company who doesn’t have a row.

It just so happens that FictoCorp (who have a lot of important customers in the netball industry) has a policy that all employees must play for one of the company’s netball teams. In order to keep track of this, the team captain keeps the following table in the company database:

We’ll simplify things by only displaying four employees, though obviously there would be more.

As an aside, netball has the nice property that the positions are named and unique; no player can be on the same team playing in the same position as another player. Therefore the combination of Netball Team and Position uniquely identifies a single employee. Obviously this constraint makes it impossible for FictoCorp to hire or fire people other than in unisex groups of 7 (in order that they can add or remove an entire netball team at once), but hey, it’s worth it for all the lucrative netball-industry contacts.

As far as the netball club captain is concerned, the entries in this table are the employees. Any employee will be in this table, and anyone in this table is an employee. So who is right, the HR manager or the netball club captain? Which table “holds” the employees? And if one table “is” the set of employees, what does that mean about the other table?

A digression

FictoCorp’s netball teams are so successful that the major league teams start to send talent scouts to their games. One day, the manager of a professional team rings up to enquire about hiring one of FictoCorp’s players.

“He was brilliant, we just have to have him … Any price, any price at all … His name? I don’t remember that, but he was definitely playing Wing Attack for your Men’s First team”

Luckily, with this information is all that is needed to identify that the player in question is Charles. The table of netball players worked equally well as a way of finding a player from their netball team and position as vice versa.

From the point of view of an outsider to FictoCorp, the table is a list of teams and playing positions, with the useful effect that the player’s name can be looked up. The talent scout’s view and the club manager’s view of the meaning of the table are different, but both are using the same table.

Resolving the ambiguity

The netball players table is neither a container of people, nor a container of playing positions. Both of these are extrinsic to the table: they will continue to exist if the table is deleted, though FictoCorp may no longer have the information it needs to get the necessary work done.

One way to think of the relation is in terms of the corresponding predicate: a function that takes a group of objects and produces a true or false value. An informal definition of the predicate for the netball players table might be:

There exists a player called X, who plays on team Y in position Z

If we substitute into this values from the table, we get true values from the function:

There exists a player called Alice, who plays on team W1 in position GA (true)

There exists a player called Charles, who plays on team M1 in position WA (true)

If we substitute in other values, we get false values from the function

There exists a player called Charles, who plays on team W1 in position WA (false)

You can think of this as a function on a 3-dimensional space, where one dimension is the list of every person in the world, one dimension is every netball team FictoCorp has and the final dimension is every possible position in a netball team:

The predicate is a function over this entire 3-dimensional space. The tuples (rows) in the relation represent points in this space for which the function evaluates to true. Tuples that could be in the table, but aren’t, represent points in this space for which the predicate evaluates to false.

Things to note:

• The predicate evaluates to true or false on every point in this space; nowhere in the space is the predicate undefined
• The predicate can’t be evaluated anywhere but points in this space; it would be meaningless to do so

In a sense, the predicate give the meaning of the table, and this meaning won’t change as we add and remove players from various teams. The tuples in the relation (the rows in the table) show us what is currently true in the real world. It is a goal of a well-maintained database that the facts implied by the table always remain a true representation of what is true in the real world, for drawing conclusions about the real world is the reason databases exist.

Objections to this model

One obvious objection to this model is that if people, salaries, netball team positions etc. are all extrinsic to the tables, how do we keep track of an entity that happens not to appear in any of the tables? If FictoCorp has a contractor called Edgar working for the company who isn’t in the employees table, and is excused from being in any of the netball teams, how do we keep track of this person?

The answer is that the database contains all the information we want to store, and nothing else. If the database system needs to be able to be used to answer questions about contractors, it will have a contractors table in which Edgar will appear. If for some reason FictoCorp doesn’t care to know what contractors it has relationships with, then Edgar will be a non-entity as far as the database is concerned.

The first thing to know about SQL and Relational Theory is that it’s largely a retread of Chris Date’s previous excellent book Database in Depth. The latter is a favourite of mine: extremely readable, yet with enough theoretical clout to change the way I looked at databases forever. The new volume carries over large chunks of the text from the older one, with some minor tweaks. As the name suggests, it brings in substantial additional material to link relational theory in with SQL, the only practical implementation of the model in current use.

In the preface, Date explains that the motivation for the new book was the realisation that practitioners weren’t able to figure out for themselves how to apply his theoretical ideas within SQL. Clearing up this difficulty is an admirable goal, and illustrates well that Date’s approach is practical and not meant as ivory-tower theory, but I can’t help but wonder if one of the reasons he didn’t state was that books sell better with ‘SQL’ in the title.

The additional material has resulted in a book that is roughly twice as long. This isn’t a problem in itself, though it does spoil one of the things I loved about “In Depth”: that it could be read in a couple of evening’s work by a sufficiently motivated person. The importance of making a book light enough that you can sit and read it on the sofa without looking like a database nerd should not be understated.

The prose remains clear and readable, and strikes a nice balance that makes it approachable to relative beginners while avoiding ever sounding patronising. Date’s style is precise to a fault, and some people will find it needlessly pedantic; nevertheless, there isn’t any pointless pedantry here, and if you stick with it you’ll learn why subtle distinctions need to be made.

So how useful are the new insertions on SQL? I find it difficult to tell. On the one hand, it makes it much easier to relate the ideas in this book to discussions of theory that actually occur in the real world, since SQL is the lingua franca. In the old book, it was certainly annoying to have all the examples written in Tutorial D, without a real specification of how the language works. On the other hand, Date’s examples in this book are still in a mythical beast called “Standard SQL”, of which no practical implementation exists. What is good practice in standard SQL might be impossible in your chosen implementation, or there might be a better way to achieve the same thing.

It’s certainly worth buying one of the two books here, but the choice of which is not as obvious. If you already own “In Depth”, the updated version probably isn’t worth buying. If you don’t, then “SQL and Relational Theory” is the thing to buy, unless you’re after a lighter and more portable read.

“Quite simply, not enough schools perform like the best. In fact, results tend to hover around the national average.” — A UK government minister explains education

On of Richard Stallman’s points was that software producers need to stop competing with each other in order to ensure our output is the best that it can possibly be. Ayn Rand argued that we need to compete with each other constantly in order to ensure our output is the best that it can possibly be.

The wish to make more things brilliant and less of them mediocre drives a lot of people, from the greatest entrepreneurs and inventors to whichever hapless minister produced the quotation with which I began this article (I was unable to find any source on the web, and I’m forced to admit it may have been apocryphal.)

But the method by which we make the world a better place is a fiercely contentious question, and many of the people on the extremes will, when push comes to shove, put ideology ahead of results. Nothing wrong with that, but it leaves those of us in the middle lacking clear role models when it comes to pursuing a more consequentialist approach. This is particularly difficult to do when we identify strongly with their goals, if not their ideologies.