Software People Inspiring

Codemanship's Code Smell Of The Week - Data Classes

 

A key goal of OO design is to minimise depdencies between classes by packaging data and behaviour as close together as we can. In practice, a good rule of thumb for class design is to put fields and methods that use those fields in the same classes. Data classes are classes which just have fields and no behaviour (besides simple getters and setters), and they break this rule of thumb, creating serious dependency issues in your code.

Here, Jason Gorman demonstrates how to refactor data classes by moving the methods (or parts of methods) that use fields of dta classes into the classes that contain those fields.



Download the source code from http://bit.ly/czsOHP

For training and coaching in refactoring, TDD and OO design, visit http://www.codemanship.com

Software Is Both Art & Science. Can We Move On Now?

 

Bonjour, mes enfants.

SEMAT has gotten me thinking. Any mention of "engineering" and "science" in software development seems to polarise opinion.

Undoubtedly, there's a large section of the software development community who believe those words simply do not apply. Software development is not a science. It's an art. Or a craft. Like basket weaving. It's not engineering. No sir!

And there's an equally large section of the community who believe the exact opposite. Software development is a science. It is engineering. We can apply scientific principles to shape predictable, well-engineered end products.

Of course, they're both wrong.

Anyone who dismisses the notion of any kind of scientific basis for software development is running away from reality. Everything that exists has a scientific basis. Even American Idol. You just have to understand it. That we don't fully understand the science of software does not mean that no such science exists or that we'll never understand. I just can't help being reminded of UFOlogists who claim that "science cannot be applied to the study of UFOs". What they mean, of course, is "I've got a good thing going here selling my unscientific ideas to these schmucks, and I'd like to keep it that way, thanks".

And anyone who believes that software development can be completely tamed by science is equally deluded. There are sciences - emerging in recent decades - that teach us that there are many things in the world that, while it's possible that we can fully understand them, we will never be able to control them. Chaos is science. And software development is mostly chaos. Any vision of "software engineering", where pulling lever A causes X and pulling lever B causes Y with certainty, is the product of naivity at the macro, project scale. Life just isn't like that. Clockwork might be, but life isn't. Software development is intractably complex and unpredictable.

In the real world, biology has come up with processes that deal effectively - but not predictably - with intractably complex problems. Evolution is one such process. Evolution solves complex, multi-dimensional problems by iterating through generations of potential solutions. That is science. We can understand it. But we cannot even begin to predict what solution a process of evolution will reach, or how long it will take to reach it.

The clockwork, Newtonian paradigm of "software process enginering" is fundamentally flawed. And anyone who believes that it's possible to attain "value" deterministically is deeply mistaken. "If we pull lever A, we'll ship an extra 10,000 units". Give me a break!

Flawed, too, is any notion that this means that there's no engineering at all to be done in creating good software. I doubt anyone would claim that making rock music is "engineering" - well, anyone sane, at least. But there is science that can be applied within this process.

It's possible mathematically to predict what effect the choice of software compressor and the compression settings used will have on the amplitude of a recording across a certain frequency range. Indeed, it is helpful in getting the best-sounding mix. There is such a thing as "audio engineering" within the music production process. Granted, it's chiefly a creative process. But there is useful science we can appky within in to help tweak the results closer to perfection.

Similarly, while software design is chiefly a creative process, it can be useful to know if the code we're writing "smells" in any significant way. Some code smells can be detected just by looking at the code and using our judgement. Others are more subtle and harder to spot, but just as damaging to maintainability. Code analysis tools, as they grow more sophisticated, can complement our "eye for good design" every bit as much as audio engineering tools complement our "ear for a good mix" and music composition aids can complement our "ear for a good tune".

The trick, I believe, is to find the balance and work within the limitations of science and engineering and creative disciplines. Trying to figure out the formula for "valuable software" is every bit as futile as looking for a formula for making "hit records". And relying entirely on your eyes and ears to refine an end product has severe limitations. For millenia, we've used tools and theory to tweak and refine all manner of creative end products. We even have a word for it: "machine-tooled". The fact that you can fit a computer more powerful than all of the computers in the world were 30 years ago in your breast pocket, and it looks good, is testament to this symbiosis of science and art.

We can continue to refine and extend our scientific understanding of code and coding through research and exploration, just like any science. And new and useful tools will emerge from that understanding that will make it possible for us to produce better quality code, for sure.

And we can also continue to develop and refine the art of software development. Through reflection, practice and sharing.

There is such a thing as "software engineering", but it has limited scope, just like "audio engineering". Specifically, it's limited to things we can predict and can control, like what happens to coupling and cohesion if we move a class from one package to another.

The bigger picture of "delivering value" is a complex human endevour, and creativity, judgement and more than a sprinkling of luck is all we have that we can bring to bear in any meaningful way at this level. We may be capable of understanding, with the benefit of hindsight, why Feature X was used more than Feature Y when the software was released. But then, with hindsight, we understand quite a lot about volcanos and hurricanes, too. These are things that can only be understood with hindsight, really. We don't see them coming until they're almost upon us, and we have two choices - stay and risk everything or get out of the way and live to fight another day.

In years to come, I'll probably notice more and more a difference between "hand-rolled" software and software that has been written with some help from "software engineering" tools - the more grown-up descendants of tools like XDepend and Jester.

But I sincerely doubt I will ever be able to tell at the start of a project whether the resulting software will enjoy success or not. Sure, I'll be able to look back on projects and say "hey, y'know what we got wrong?" But far in advance, the outcome is every bit as unknowable as a hurricane, volcanic eruption or hit record. So where things like requirements and processes and "enterprise architeture" are concerned, I'll stick with arts and crafts.

Value Is Not The Opposite Of Waste: Why I Don't Buy Into Process Improvement

 

If you've been a regular visitor to my blog since 2005, first of all, thank you. You may also know that I'm not the biggest fan of mechanistic or pseudo-scientific approaches to making teams better at creating software.

Which is why I don't buy into process improvement any more. At all. In all it's guises. TQM, BPR, Six Sigma, Lean. All old wives tales and hifalutin mumbo-jumbo, in my honest opinion.

Yes, there are the success stories that devotees and evangelists routinely point to. Usually in Asia. Mostly Toyota.

But anyone can find the exception that proves the rule. I can point to 80-a-day smokers who lived to be 100. In a wider sample population, there doesn't seem to be compelling evidence that process improvement makes a positive difference in the long term.

Commoditising "value" in this way, suggesting it can "flow" (and no doubt can be diced, sliced, weighed and stored for future use, like charcoal briquettes), seems very alien to me. "Value", in my mind, is a very complex and vague concept. There's fiscal value, of course. Profit. But businesses have been learning the hard way that such a one-dimensional view of what matters can lead to a very one-dimensional approach to management. Companies that are only interested in profits tend to be so at the expense of other kinds of "value", like satisfied customers, content employees, safe neighbourhoods, clean air, and so on.

Businesses have been learning, albeit very slowly and clumsily, that long-term success comes from chasing a richer, multi-dimensional and balanced set of outcomes. They are also learning that, like all multi-bodied problems, these outcomes interact and affect each other in complex ways. Who knew that improving the quality of your products could actually reduce costs, for example?

To suggest that this complex web of interconnected "things" can somehow "flow", to me, sounds as bizarre as proposing that "happiness" is rectangular for easy stacking.

The reality is so much richer and nuanced and unpredictable, of course. It is not like sorting out blockages in your plumbing, or hot-rodding an engine. A does not necessarily follow B.

The most damning indictment of process improvement is the widely-accepted fact that good developers tend to produce better software. All the tweaking and Six Sigma-ing and Lean-ing in the world won't make a piss-poor team appreciably better at delivering "value", any more than it could make an orchestra any better at playing Mozart.

Did it occur to anyone that the folks at Toyota just got better at making cars?

And it's not without it's unpleasant side-effects, either. Many methods focus on reducing "waste". Lean goes the whole hog and suggests that if we reduce waste, we improve the "flow of value". This has a very definite "profit vs. loss" feel to it. It is distinctly one-dimensional.

Complex systems need a fair dollop of waste. Waste might come in the form of unsuccessful prototypes, or a range of choices, or a level of redundancy. We only use 10% of our brains. Most of our DNA is "junk". Many species fail to flourish, ending up as food for the ones who do. Is anyone suggesting that removing 90% of my brain would make me smarter? Or that removing my junk DNA will make my childern healthier and stronger? Or that only successful organisms should be allowed to be born?

My point, laboured as it is, is that in many cases "waste" turns out to be there for a very good reason. In creative and innovative persuits, which are inherently novel and therefore unpredictable, who can say what will never be needed?

The adaptive capacity of a complex system necessitates some waste. Some choice. Some diversity. Some redundancy. Some slack.

In this respect, the likes of Six Sigma are effectively anti-Agile, if we interpret "Agile" to mean "responsive to change". If that's your goal, then focus on the people in your teams, and make plenty of room for their innate learning and adaptive abilities to work their magic.

You may now start throwing the furniture around.

Macs Are Better (Because They Suck)

 

I'm not a Mac user. I have never owned a Mac, and the only device in my possession that was manufactured by Apple was a present; an iPod Nano from my brother.

People constantly tell me that I should buy a Mac. They say the Mac is more stable than Windows. That it is more secure.

I say, yes. I completely agree. Macs very probably are more stable, and they very probably are more secure. I don't need to own a Mac to know that. It's simple statistics.

According to various folk I've known who worked at or with Microsoft over the years, the underlying source of the bulk of reported Windows bugs is third party device drivers. Essentially, device drivers are hosted by the OS. In that sense, they're like Enterprise Java Beans. They are designed to present a specific interface and run inside a container. To do this safely, they must satisfy all the behavioural rules - including rules about concurrency and timing, which is always a bit fiddly to get right - required by the container. And not all 3rd party device drivers satisfy all of the rules all of the time. Every new device driver presents a finite risk to the stability of the OS. The more device drivers your OS supports, the greater the accumulated risk. Put simply, Macs are more stable than Windows because they support far fewer third party devices. Less choice, greater stability.

And of course Macs are more secure than PCs. Far fewer people are trying to attack the security of Macs. Why go to the expense of writing viruses and wotnot for such a niche platform? And also, there are far more ways of attacking a PC because there are far more ways to write code that will run on a PC. Again, with a Mac, less choice means greater security.

Macs aren't more stable and more secure because they're better engineered. The real reason why Macs are more stable and more secure than PCs is because they're less popular and offer less choice to their users.

In other words, Macs are better because, actually, they suck.

Twitter Re-Educates Me On "Value"

 

Twitter is currently teaching me a valuable lesson in - er - value.

Whenever I try to log into the service at the moment, be it via the web site, from my mobile phone or via TweetDeck, it's 50/50 that I'll actually get a response from the site.

If I am successfully logged in, when I submit an update it's 50/50 again that it'll actually get successfully posted.

Add to that some "logic quirks" with the functionality itself and some very dodgy AJAX nonsense going on, and we have one demonstrably flakey Web 2.0

And yet current speculation from those in the know (though the crisis in our financial markets maybe suggests that these people know nothing) values Twitter at some ridiculous, astronomical figure. We're talking the GDP of a small, but developed, economy here.

I am still convinced that a team of 4-6 good developers could deliver Twitter pretty much as it is on the Google App Engine in less than a fortnight. And deliver a far more reliable and robust implementation at that.

But how does the relationship between reliability and value work in this case? It calls into question the simplistic predictable relationship - a commoditised version of "value", if you like - that we've been assuming when we talk about things like "value streams" and "sustainable delivery of value".

If Twitter runs on just a few thousand lines of code, and I suspect there's very little under the bonnet in reality - then, if the service is valued in the billions, each line of code could be carrying the weight of millions of dollars in equity.

Imagine writing a line of code knowing that! How much time and effort would put in to making it right, and into making it scale?

Except, of course, Twitter's developers had no idea when they wrote it. And neither, I suspect, do any of us when we're writing code. No idea at all.

Being Skeptical About Metrics != Dismissing All Metrics

 

There's a growing and vocal tranche within the software development community who believe that all software metrics are bad.

Curiously, they don't believe that metrics are bad when their doctor tells them that their blood pressure is abnormally high, or when their union representative tells them that their salary is below the national average for their job description.

But attach any kind of number to anything related to the business of writing software and it's a different story. Metrics, to some, are the Devil's work and must be exorcised from software development.

It's true that metrics can and often are abused in management. Decades of cooked books, skewed studies and notoriously optimistic "government statistics" have conditioned many of us to mistrust numbers. And rightly so. Whenever anyone presents you with a measurement, it's wise to be skeptical and question what it might mean and how it was collected. But dismissing all measurements as a rule is not skepticism. It's just as much an act of blind faith as believing every statistic you see.

Software development without metrics is woolly and very, very highly subjective. It's like the Olympics without stopwatches or cookery without a set of scales. Metrics debunkers sometimes paint a picture of the other extreme as being the reality of metrics - namely total reliance on arbitrary numbers, even when they fly in the face of our subjective reality.

I would not advocate blindly accepting what your metrics tell you. But I would definitely not advocate ignoring the numbers, either. If Max Planck had thought like that, we'd never have ended up with quantum mechanics, which - measured in terms of the accuracy of its predictions - is our most successful science. Sometimes the reality is what the numbers say it is, and not what our eyes or our instincts tell us.

And sometimes, and especially in very complex matters, simple measurements hide a more subtle truth, and our instincts serve us best - provided we've had the time to develop them to the point where they can be trusted. Estimating is a great example of something where the first instinct of a senior programmer is just as likely - if not more likely - to be right than any number derived through some kind of empirical COCOMO nonsense.

But that doesn't apply to every aspect of software development. If a coverage tool tells you that 50% of your code isn't executed during a test run then you know that 50% of your code isn't being tested at all. If a static analysis tool tells you that one of your methods has 500 lines of code and a cyclomatic complexity of 80 then you know it will be practically untestable and very difficult to change. If JDepend tells you that you have a package full of concrete classes that is very heavily depended upon then you know that making changes to code in that package is likely to have a bigger knock-on effect because of the lack of substitutability.

Sure. Absolutely. I won't deny that metrics throw up false positives and can sometimes send us on wild goose chases. But I'll tell you this: I have never seen a project team that delivered high quality software that wasn't monitoring quality using some kind of measurements. I suspect that's because testable goals tend to be easier to aim for.

By all means, be a metrics skeptic. Question every number you see. If a metric tells you something about reality, go look at reality and use your hard-earned judgement to see if you agree with the metric.

But dismiss all software metrics out of hand at your peril. Because I know that software teams who don't measure quality tend not to deliver very good software. Successful development teams use metrics. Of course, there are plenty of teams who don't measure anything and who think they're successful. There are plenty of people who think they have psychic powers, too.

Physicist Tests Journal of Cultural Studies - Finds Relativism Gone Mad!

 

Alan D. Sokal, a physicist at NYU, submitted a wordy article to a leading journal of cultural studies packed full of outrageous and totally unsupported claims suggesting that physical reality is a social construct and even that science should be led by a political agenda. And it got published. Without any corrections.

As Sokal puts it:

"What concerns me is the proliferation, not just of nonsense and sloppy thinking per se, but of a particular kind of nonsense and sloppy thinking: one that denies the existence of objective realities, or (when challenged) admits their existence but downplays their practical relevance. At its best, a journal like Social Text raises important questions that no scientist should ignore -- questions, for example, about how corporate and government funding influence scientific work. Unfortunately, epistemic relativism does little to further the discussion of these matters.

In short, my concern over the spread of subjectivist thinking is both intellectual and political. Intellectually, the problem with such doctrines is that they are false (when not simply meaningless). There IS a real world; its properties are not merely social constructions; facts and evidence DO matter. What sane person would contend otherwise? And yet, much contemporary academic theorizing consists precisely of attempts to blur these obvious truths -- the utter absurdity of it all being concealed through obscure and pretentious language."

One can't help wondering if many of our leading professional publications would be just as easily bamboozled by relativist nonsense. I have read articles that, at the time, I thought must surely be spoofs.

Software development abounds with outrageous and totally unsupported claims and the "epistemic relativism" Sokal accuses Social Text of championing. Most specifically, the pernicious and ultimately disastrous notion that your way is as good as my way is as good as their ways (so everybody gets to be right - hoorah for everybody!) and that, in software projects, there is no objective reality and all that matters is perspective and discourse. In less fancy language, we get to sit around yapping all day and everybody's opinion is equally valid - and evidence doesn't matter.

Indeed, as critics of metrics are fond of pointing out, reliance on evidence leads to oppression. I would argue that misinterpretation and/or misapplication of - often poor quality - evidence can and does lead to bad things in all kinds of organisational life. But I don't infer from this that seeking evidence is therefore always wrong (far from it), or that the reality of software and software development itself is somehow beyond empiricial understanding in certain key respects.

This is currently a very unfashionable view, and one for which I'm often chided by my peers. But I increasingly believe it to be important, and see a dark future for our profession if we continue down the slippery slope of woolly thinking that we're on.

Example Agile Quality Assurance Strategy

 

For the morbidly curious among you, here's a link to a (suitably anonymised) quality assurance strategy for what some might describe as an "Agile project" team.

There's much emphasis on defect prevention and "left-shifting" (the practice of moving testing further upstream in the design and development process), as well as on automation and the economy of scale that can be achieved.

There are metrics. Don't be afraid. They're only baby metrics, and haven't learned how to bite yet.

There's also some process guidance and a bt of innovation for incorporating non-functional quality requirements into a lightweight, iterative and - most importantly - test-driven development approach.

Musings On Testing Multithreaded Logic

 

I'm currently pondering the problem of multi-threaded logic and concurrent testing.

Threads are radioactive, man! You should try to stay well clear, and if you can't avoid them, handle with extreme caution.

You see, if I write a line of code, then there are only certain ways it can be wrong. If I write two lines of code, that doubles the ways it the code can be wrong. If I write three lines, then that pretty much doubles it again. Do you see where I'm going with this? Every new piece of logic can exponentially increase the probability of getting it wrong. Which is one very good reason to favour short and simple methods.

If we keep our methods short and simple, and test them effectively, then we can get a grip on our logic and produce reliable results.

But when we add the possibility of more than one thread of execution happening at the same time, then we get a literal explosion of possible combinations of interleaving of logic. If no data is shared between threads, we're probably okay. Which is why things like Java servlets and ASP.NET web forms are relatively "thread-safe", and dunderheads like me and you can get away with using this highly isolated concurrent processing.

But the moment threads start sharing data, we get problems. To illustrate this, I've tried to visualise an example interleaving of executable code statements from two threads both trying to transfer funds between the same accounts.



In this example, thread 1 checks the pre-condition for withdrawing money from the payer's account, and immediately after so does thread 2. Now both threads believe that account1 has sufficient funds to cover both transfers. Which, of course, it doesn't. Thread 1 calls the withdraw() method first, leaving only £20 in the account to cover thread 2's £50 withdrawal. But thread 2 is none the wiser, and calls withdraw() anyway. Oops!

A different interleaving of the code being executed by the two threads leads to a wholly different outcome.



There are several unique interleavings that can be achieved from just these few lines of code being executed by just two concurrent threads. If we wanted to ensure we get the multi-threaded logic right, I'd want to test all of them.

One of the real drawbacks of trying to test multi-threaded code is that the operating system (or virtual machine) decides the priority of threads and the interleaving of executable statements, and this is usually in competition with a whole bunch of running programs, all doing complex things. Effectively, you should not expect the same interleavings each time, and therefore you should expect your multi-threaded tests to be unpredictable and difficult to reproduce.

This is not very helpful, obviously. Automated tests should be totally predictable and reproducible. But how do we achieve this?

My proposal is to create a tool that reads your multi-threaded code for a test, and generates a set of Command classes to capture every executable step. It would build a sequence of commands that are semantically identical to the multi-threaded code, but that we can now explicitly choose when to execute. We can then create copies of this sequence, and calculate every possible interleaving for a given number of concurrent threads.

Not a trivial undertaking, I grant you. But quite do-able, in my opinion. Applicable test assertions could be checked before and after each concurrent step is executed.

It still needs a lot of thought, but I think this idea might have some merit, so I'm going to look into it a bit deeper.

Productive Teams Are Business Assets. Recognise Your Investment In Building Them.

 

So here's a joke. You may have heard it before, in which case please don't spoil it for the other readers.

Q. How many weeks does it take your average OO software development team to ramp up to the point where they're delivering tested code, visible features and have build and test automation up and running? From a standing start? Does that include implementing a Agile process from scratch with a new team of people who've never worked together before? In a new office? On a client site (so you have no control over your environment)? How long would it be before you were delivering real features to the business?

A. It depends.

My own experience has been that just to be squirting out working, tested code of a provably high design and implementation quality from a standing start is that it can take a couple of months or more.

The list of things you have to get in place for that to happen is far longer than most of us tend to acknowledge. Everything from network access to source control to build automation to stationary (those story cards don't order themselves) and whiteboards. Just getting the room in shape can be a week or three of team effort.

Then there's the approach. Contrary to what you may have been told, we don't all carry around a book called "How To Develop Software" and arrive on teams with exactly the same understanding and outlook as each other about how to go about the business of writing code. It can take a lot of work - a lot of talking and debating and compromising (if you're lucky enough to work with a team that compromises, of course) - to reach a consensus on key practices and processes that the team will work to. That can take quite some time, too - especially if you want your finished approach to actually work. (We never really had that problem with RUP. We knew we were probably going to fail. But what the heck?! The documentation was going to look superb!)

And then there's the architecture. Oh boy, is there the architecture?! How long have you got? Back in the old days when men had fancy moustaches and fought to the death over the affections of some emaciated, disease-ridden debutante (those were the days, right guys?) life was much simpler. You slapped the cad about the face with your glove, proposed pistols at dawn and proceeded to blow holes in each other before repairing for breakfast at Mrs Miggins Tea Shop. Simplicity itself.

These days we crowd around whiteboards and throw loaded patterns at each other, until one of them explodes and we end up with Mementos all over us. Then we repair to our workstations and glare at each other for the rest of the project.

Anyway, it's an horrific, costly and potentially very time-consuming process. We're lucky if, by the end of it all, we've managed to agree a single thing - like that the code should run on a computer...

Add to this a million and one little gotchas - everything from tool and framework licenses to setting up Wikis to figuring out where to get a decent cup of coffee round here.It all takes time. And that time has to come from somewhere.

To get a team to a point where they are able to deliver anything of any value, you must invest tens of thousands - maybe hundreds of thousands - of pounds. And it is an investment. A productive team is an asset that has to be built up.

If you fail to recognise the value of that asset, you will make the mistake that so many employers make, which is that productive teams aren't worth preserving. All too often I see teams being broken up and laid off after a successful delivery, when - in reality - the team itself is every bit as valuable a business asset as the software they've created. (Often moreso.)

OK. So the punchline still needs some work...