Is software an asset or an expense?

I had a brief conversation on Twitter with Camille Fournier () about costing for projects and development, and this is something that I’d been thinking about (but not writing about) since I was at ANZ bank about 2007 or 2008. Most of it was trying to explain/understand expectations about software development and delivery and “how much effort to put into projects”.

Even though I actually have a commerce degree, I don’t intend this observation to be a strict treatise on the accounting terms “asset costing” and “expense costing” but more about the general expectations set by considering the constructed software as an asset, or an expense.

The basic premise is that in general teams of software developers, in the absence of specific direction or rules will assume that the software being delivered will have the properties of an asset. The software will last for an extended period of time, it will be modified and updated and will not generally have a short defined end of life date.

However, in many cases the teams of people asking for software to be built will not always have the same view, and they may be well asking for systems that are developed in shorter time frames, and have a short defined end-of-life date. This generally is where the difference in expectation on project cost (and time) often comes from. We see this a lot in start-ups, where iterating and responding to new ideas trumps all possible long term benefits. With nothing right now, there is no future for the start-up.

Coupled with this particular problem, there is generally a mismatch between how much effort the demand team expects the supply team will take to construct a solution.

So, what should we do about it?

At REA, I’m starting to help our teams with this, and the first step is to get some greater definition about the expectation of not only what problem the system should help solve (“functional requirements”) but also the scope of what that system should participate in (“non-functional requirements”).

Generally these would be called something like SLA (service level agreements). How many users will it support, what is the uptime? What is the response rates? What are the transaction rates? These are pretty standard non-functional requirements that you’d see in systems descriptions. However, one key part that I’m trying to encourage people to think about is “how long should this last?”.

I think the first part of the problem is trying to get an understanding from the customers about what their goals are. Do they want to create some “disposable software” to solve this problem? Do they think that the solution to this problem should last for a long time and be enhanced over that time? Do they even understand there is a difference to the engineering required to do this?

Now, if we add to this the general trend towards microservices (smaller units of functionality that can be more easily replaced) maybe we are looking at a general shift in the way we write code for systems and how we might wish to think about, and set expectations for system development. Can we really think of software components as “write them to throw away”?

Certainly if we’re looking at treating the components or code like an expense, I think there’s a better chance. I also think that using expense based thinking for “research and discovery” might lead to more opportunities for faster iterating through ideas, knowing the expected use and lifespan of the work performed.

I’d also like to see more input from people who have tried similar ideas. It’s a slice of a topic about “software life-cycle management” that is more deserving of thought that most teams give.

You say you want a revolution..

“Well, you know. We all want to change the world”

My title on LinkedIn is “Reluctant Architect”. This should not be considered a reflection about how I feel about my job, the company I work at, or the work that I do. It’s more of a reflection about what the rest of the industry thinks of an Architect in the software sense.

Basically, the term has been completely hijacked and the use is completely wrong. For the most part architect in the computing industry is some old white guy who’s completely forgotten anything about software development (if they ever did any) and spends all day writing PowerPoint presentations on choosing WebSphere vs Weblogic. This is then inflicted on the organisation like stone tablets being handed down from the mountain.

I can’t find enough words to describe how much I disagree with the concept, the implementation and the horrors that are perpetrated by organisations that follow this model. It is the ultimate in disempowerment, it actively discourages teams from learning and puts power in the hands of people least capable of using it effectively.

So, while I’ve given a roasting to the way my industry has traditionally handled architecture, how, and what do I do differently?

My background is software development, and I’ve spent a lot of years working with, and mentoring teams, but without a doubt the biggest influence on my recent career has been becoming a parent. I have the most wonderful 8 year old boy, who has the same level of enthusiasm for life as his father along with the youthful confidence that everything he knows is right. At this point, you have to transition from “telling” to “experiencing”. No amount of me telling George that “eating that many lollies will make you feel sick” would convince him. So, short of doing things that would actually (or likely) kill him, I encourage him to safely explore his boundaries. Quite often there is joy in the discovery, and quite often there is cuddles and comforting words after a “learning experience”.

So, being an architect, and a reluctant one at that.

(From 1550s, from Middle French architecte, from Latin architectus, from Greek arkhitekton “master builder, director of works,” from arkhi- “chief” (see archon) + tekton “builder, carpenter”. An Old English word for it was heahcræftiga “high-crafter.”

This pretty much sums up what I feel about the role in general. I am an old white guy with many scars from building systems the wrong way, or seeing other teams build things the wrong way and I wasn’t quick enough to help them. I try very hard to build relationships with the technical staff across the organisation so I can influence their approaches and thinking without needing to actually tell them “do it this way”. This sounds all a bit unicorn and rainbows and holding hands summer-time walks on the beach, but I’d say there’s very few people at REA that have any doubt about my position on various topics, and what my likely response is if they test those boundaries.

Specifically, what does this look like in practice? Glad you asked! I’ll outline the process that I go through (and have done) at REA focussing on architectural change.

The history

When I joined REA nearly 4 years ago, there was a small number of large applications, there was strong coupling and releases were painful. We were tied strongly to data centres with applications running on racked tin. Applications made many assumptions about how close they were to each other (latency for example). Control of applications were “tier” based (back-end vs front-end) and there was contention across the organisation for product releases.

The strategy

Working with Rich, the main goal was to structure the organisation to allow faster releases and improve the quality of the systems (reduce coupling) to make this possible. There was a heavy investment into cloud computing (using Amazon) as the means to reduce contention in the development and testing area, with still having a pathway to multiple production deployment environments controlled by the business aligned organisational structures (we call them “Lines of Business”)

The vision

A dashboard for each Line of Business that shows all cloud providers in the world, their latencies and suitability for applications, including cost. The teams are able to deploy services across the globe, according to data transfer requirements, latency rules and follow the “cheapest” computing and storage options.


Yeah, something like that, but less Neo and more graphs.

The direction

We need to have our monolithic coupled applications split so that each Line of Business can deploy them independently. Our operational staff need to have visibility into the health of the applications without actually knowing where they are. The systems need to support increased speed of delivery of new function for each of the Lines of Business.

The final attribute is considered one of the driving reasons for these changes – so I’m going to focus on it in future sections. However, at this point most of the work that I do is making sure the technical leaders for the various Lines of Business understand the vision and the direction without interfering too much in the actual implementation.

There’s also a lot more to the ongoing strategy involved, but that’s probably another topic for another time.

The design

I strongly value autonomy and self-discovery by the teams. I think learning by doing is the most powerful approach and Open Source ecosystems have shown that the mutations from different development approaches (generally) improve the state of the art as the development teams learn from previous implementations.

In terms of the design of “the architectural direction and improvements” I’ll explain how I’m influencing the understanding and behaviour around application deployment, modularity and most importantly monitoring and fault tolerance.

I realise that “make application deployment, modularity etc, etc better” isn’t a desirable directive, because it’s not very useful and because in many cases people don’t have a clear idea what “better” is. For developers especially many of these concepts are quite foreign, so what I aim for is smaller fine grained directives that help to provide some gentle prodding for exploration in the right areas.

By doing this, what I’m trying to get teams to “work through” is the potential difficulties involved in implementing some of the architectural improvements in their specific contexts. If I actually knew the answers I’d probably work with the teams directly, but I rarely know “exactly how” teams should implement things. I’m blessed by being surrounded by an awesome group of developers and technical specialists that are very capable of implementing improvements in their own contexts, my role is to show them the path to take.

The implementation

Taking the example of “improve modularity and decoupling”. What is needed under these circumstances is independent services. However, a key part of the total system improvements, especially when relating to multiple independent systems is monitoring and fault tolerance (and investigation). REA Group use AWS for many of our system deployments, so some of this is ‘more important’ than dealing with racked tin, but the same principles should apply.

So, now we think a bit. What can I do at this point, and what principles can I impose on the teams to move in the right direction. One of the most expensive parts of software is the operation of running systems. Most of this is because the monitoring, logging and debugging tools are “left as afterthoughts”. I could say “make sure all systems have monitoring, logging and debugging according to the checklist defined in subsection 42, document 27b-6”. That sort of directive could sound familiar to many people, and is pretty much everything I despise about “enterprise architects”.

My directive was “remove ssh access to all deployed systems, nobody can log in”.

To say the response was incendiary was possibly an understatement. Nomex underwear is standard issue for my job, but it’s very interesting to see how often it’s needed. The other thing that interested me was “what roles gave what responses”.

For the most part, experience ops people (hi Cos, love your work) saw through my facade and knew what was up. They’re also generally used to working in constrained environments, and as a result have a huge toolbox to still effectively do their work. The other good news is that these wonderful people also become great advocates for improvement, because most of the burden of running systems fall in their laps.

Developers are predictably “y u no luv me” because what their main focus is to develop and deploy rapidly, debug issues locally and repeat. There’s probably a good reason for excluding some of these principles during development, but as I will repeat, unless the developers feel the pain themselves, it’s unlikely that changes are going to be made. All that does mean is that the operation team gets sad.

Why did I choose that particular course of action?

Well, it’s pretty controversial, so there’s lots of talk (and complaining) so people communicate with each other about how terribly unreasonable that I am, and how I don’t understand “the reality’ of software development. It’s visible and easy to do (don’t start sshd, easy) and should it turn out to be a retrograde step, it’s easy to change back.

The other benefits we see from this is that our systems start to become immutable – a property that I find particularly valuable in coding, and it transfers nicely to system components as well. This is a great thing in AWS land because we can just shoot the cattle and launch another one, and I know that nobody has been on the box “fixing” it.

By not being able to log into the box, we have to think hard about how we log and monitor our system components, especially important things like tracing transactions through the system. What sort of identifiers? What do our components do under failure conditions, network connections etc

The aftermath

There’s a school of thought that I should carefully explain my reasoning behind my decisions so it’s clear to everybody, and there is limited “heat and light”. There may be some merit to that, but my role is not a dictator, it’s an educator and a communication enabler. I don’t mind if the teams are all plotting to subvert me, or even getting together to bitch about how unreasonable I am, but the point is, they’re talking about it – with each other. That’s a big win. I love watching our internal IRC when somebody proposes “how about I use a bastion box to get to the instances” and there tends to be a few comments like “don’t say that, you’ll make Jon angry”, or “shhhh, don’t say it out loud”. That’s fine. It means that people are paying attention, and that even tongue in cheek comments like that make me feel like a difference is being made.

The second part is that I’m not always sure that I’m right. Sometimes I just go with “this seems like a good idea”. Like parenting with George, provided nobody is going to die (or projects fail spectacularly) then making these sorts of decisions and directions will gain valuable insight into our projects and systems, even if we start to go in the wrong direction.

The astute readers here (well, let’s face it, if you’re reading my blog, you’re probably already astute) will notice that I’ve only described a very thin slice for the implementation. Yes, that’s true. This is a big thing, it’s a long term view and to be honest, it’s sometimes disheartening to have to wait. It’s worth the wait, just need to hold firm and be confident that what you’re doing is the right thing. So don’t be confused that the descriptions above cover all that is needed, even from a pure “this is the direction” point. There’s probably 20-30 separate parts of the implementation that is being influenced at any point in time.

I’m looking for long term change with the work I do. Not short term fixes. I want teams to participate in this journey and not just be told what it looks like. There’s also significant parts of cultural change that form part of what I’m aiming for. People do get stuck “thinking the same way” and it’s my role to try and encourage them to think in a way that systems will be constructed in superior ways.

I hope this turned out to be of value to some people, I’m happy to discuss it further in public, or private emails. I’m very happy to help organisations understand how architecture can be better integrated with the development and operation of systems.

Once more unto the breach

One of the (very few) things that I completely love about using a Mac for software development is the integrated command line which does a pretty good job of approximating a Unix environment.  This is great for me, and supplements how I work nicely. 

To get a similar environment under Windows has always been a great goal, but somewhat hindered by the completely shit that is cygwin.  So, as I commented in my last entry, I have started using VirtualBox to provide a great scripting environment, but I’m still stuck in the mire of not having a decent terminal client to access the command line shell.

There are a number of alternatives, of which SecureCRT is probably the best (an awesome application that I’ve owned for many, many years) but it’s a commercial piece of software that many people may not feel the need to buy.  PuTTY is pretty good, but it’s only a marginally better application than the standard Windows console application.

Frustration over the weekend finally got to the point of doing something about it, and I investigated a number of alternatives including Console2, Xshell, MinGW and finally settled on using MobaXterm (  Great application with a number of awesome features that I’ve not completely finished exploring yet.  It comes as a free for personal use, with an upgrade to a professional version for $49 EURO.  I’ll keep using it for a while to see if I want to upgrade, but I might do it just to support the work.

Hope that helps some people become more productive.

Use of weapons

While I may rant on occasions about various desktop environments, I’m pretty neutral about them provided they are vaguely suited to use.  Now, I’m doing a whole bunch of stuff in Unix-land, and frankly either MacOS or Win7 it complete shit at dealing with this.  So, I’ve been looking at how I can do this without hurting my head too much.

My answer is “to the cloud”.  Actually no, that’s what everybody else is doing.  My real answer is “virtual machines”.  Modern desktop systems are for the most part powerful enough to run a VM with a minimum of effort, and you can create/clone/destroy them as often as you want.  This is the real joy of dealing with a VM.  Right now I want to spin up a dev environment that mirrors our CI view of the world.  So, I create a VM, I install stuff.  I *clone* the VM and start using the cloned VM.  I can completely screw around with it – and if I fuck it up (or when I’m finished with it) I just discard it – and I can grab the template VM and start again.

The other benefit is that I don’t have to worry about installing any form of UI on those VMs – because I can just create an SSH server, and ssh my way into the virtual machine and go to town.  I can use the “power” of the graphical display of my native desktop platform (with all the ability to cut and paste things into an email / a log / a whatever) while having a *real* Unix environment to work with.

Now, while the *real* Unix is compelling – the single biggest feature that makes it so desirable to use a VM is the create and destroy model.  I can even create a base template with all the packages installed, then use that to clone a couple of nodes and fire them up. 

I’ve been using VirtualBox on Windows and VMWare Fusion on MacOS.  Some of the reasons for these choices have been the ability to install some of the native tools, which make it easier to do cut and paste – and to share directories between the native and slave environments.   Some of this has been due to UI convenience (VBox on Windows I can run a fullscreen Ubuntu for example, but VBox doesn’t let me do that on MacOS – no support)

I think I’m going to standardise on using VirtualBox if I can work out the native directory sharing between the MacOS and Unix environment – but at this stage I may be using 2 different systems.

And then it all changed

Events happen in your life that whiz past.  Some you’re only briefly aware of, and others make you stop and watch.   Then there are events that stop you, and make you pay attention to them.  You ignore those events at your peril.

I spent tonight as I do most nights cuddling my little man to sleep.  It’s one of those things we do, and it’s one of the great times of my day.  Days that have been shitty all melt away as I spend anywhere from 5mins to 30mins just lying there, in the dark, with my son next to me.  Tonight was the night that I realised, with startling clarity, exactly what was important to me in my life.

I’ve spent 20years of my life trying to make other people do work in a better way.  I’ve covered for incompetent morons, I’ve spent countless thousands of hours trying to get organisations to understand why when I say something, it’s worth listening to.  Now, I just don’t give a fuck.  The people who don’t understand why shitty requirements end up with shitty software – you know what – you just keep writing those shitty requirements and I’ll just nod.  No unit tests?  Sure, you have a nice time there, I hope it doesn’t hurt you too much.  Shitty code with cyclomatic complexity through the roof?  Grats – at least you can use an IDE, your mum must be so proud you ignorant fuck.

It’s just not worth my energy dealing with you, and your shitty defensive attitudes about how you have to do it this way because there’s no other option.  I’m just going to smile and say thank you, and then think again about how I’m going to go home tonight and cuddle the most important thing in the world.

I feel like I’m selling out – but you know what, after 20years of fighting the good fight, I’ve not had a single organisation say to me “thanks Jon, you really made a difference”.  I’ve done it for me, for my colleagues and mostly for the people I try to mentor.  No longer are you going to leech my energy like vampires, I’m going to be able to save that for my son.

Those people who want to learn a better way, and those people who ask me for help – you’re always welcome, and I’ll spend time and energy with you, because you care.  The rest of you, wear your warm badge of ignorance with pride, march on down that lane of stupidity and I hope you get hit by a car.

I should add to this, that it’s not my current organisation that is bad, or even the worst in this respect, it’s just that some time in the last year I grew up, and don’t want to do things that are bad for me any more.

Maybe one day I’ll find an organisation that doesn’t clothe itself in ignorance, one that treats people like they’re trying to help, not hinder and when I do, I’ll blog long and hard about that because life is too short, and love is too important to be hindered by spending time being unappreciated.

I love you Georgie, thank you so much for saving me.

Architecture – it’s not all bad

I was having a chat with a colleague today about the role I’m playing on his project.  We were discussing the sorts of things that trouble him, and what I can do to help.  I’ve been musing on the “all architects suck ass” thing for a while, because I know that’s not true, and really in response to him I said;

“It’s not my responsibility to deliver your project, but it’s my responsibility to make sure _you_ can deliver your project.”

That’s my take on what my job is about.  There are projects to be delivered, there are issues that are external to a project, and it’s my job to make sure they go away and the project can be delivered appropriately.

Good is the enemy of great, but ‘great’ is not universal

I have lots of great discussions with Simon Harris, James Ross and Perryn Fowler.  Mostly because we’re all passionate about doing the best possible job we can.  Also because we like to poke fun at our own little pet loves and fanboi’isms.

Our latest set of shelling of each other has been about source code control systems.  Now, the actual item is pretty irrelevant, but it serves as a vehicle for concrete discussion of a more abstract topic.

Essentially for any particular problem, there is a solution which is “good enough”.   Now, for this particular case, I’ll posit that ‘SVN’ is good enough.  A great solution for a very competent team, which is distributed may be ‘Git’.  A great solution for a not so experienced team may be ‘SVN’. 

What?  How can you say that?  Well, it’s pretty simple really. 

Good covers all the basic requirements of the solution.  For source code control, let’s say that’s ‘saving various versions of my code and letting my peers share it in a reliable way’ (you can tell I’m not putting too much effort into this, but deal with it).  SVN covers this, and you really can’t go below this level of functionality – because then you’re not covering all your bases.

Next, we look at great, but ‘great’ for Simon and Perryn is very different to ‘great’ for me, and many of the teams I work with.  ‘Great’ is all about making that next step, and being as productive as possible.  Git (or whatever tool-de-jour) is going to be more productive for Simon, but it’s going to be an unmitigated disaster in the hands of inexperienced and software developers who really just don’t care that much.  So, moving ‘up’ to Git, will be a productivity decline for development teams who don’t have the capacity to deal with it.

I blogged a few years ago about similar problems with languages here and here