Software development is a mission-critical activity, providing many enterprises with a precious source of competitive advantage. It is an activity that has attracted some of the brightest stars in IT and the best are highly treasured.

At its heart, software development has always been a demanding environment, with new languages, platforms and methodologies appearing in a constantly changing market. Programmers routinely have a hard time, often working for customers who have only vague notions of what they want, but still need it yesterday.

The very first software development efforts were chaotic, based on the premise of “code and then fix”. Applications could develop without much of a plan, growing organically from short-term decisions and without a formal underlying structure. This methodology sufficed for one-off applications, with tightly prescribed functions that never needed to expand. Problems only arose when new features needed to be added or integration with other devices, platforms or applications was required.

This trial-and-error development method was prone to bugs, difficult to fix and complicated to test. More structured software development methodologies promised to make the process more disciplined, predictable and efficient.

But by emphasising planning, many companies found their software development process had become bureaucratic and their development cycles elongated. And still the accuracy of coding had not improved ­ – programmes still required extensive debugging, but now they came with a mass of documentation.

Over the past few years there has been a discernible shift towards automation, scripting and wiring up of components, says Keith Braithwaite, programmer and agile business unit leader at software developer Zuhlke.

“These techniques enhance programmers’ ability to write code by removing the overhead of too much housekeeping activity,” he says.

“However, the biggest improvement came from reviving the idea that systems are built mainly to serve their users and that therefore system builders need to collaborate closely with users and business sponsors rather than isolating them behind documentation.”

As a result, in the past few years there has been mounting interest in a new style of software development methodology, referred to as agile methods. Rather than being process driven, agile methodologies are people oriented, and designed to manage change within the process and the external environment. Agile methods are the antithesis of the over-engineered approach ­ – instead of long-term planning and massive amounts of documentation, agile focuses on delivering improvements in small, iterative stages.

While software development methods have changed, so too have the platforms and languages used by the developers. Remember the frantic search to bring Cobol programmers out of mothballs to deal with the millennium bug? Languages and programmers move on as technology changes and the focus of programming has shifted to the web.

Clive Howard, partner at internet consultancy Howard Baines, points to the growth of technologies and platforms over the past few years as contributing to the software developers’ dilemma. “Ten years ago there were very few technologies in web development and the only serious platform was the desktop browser,” he says.

Today there is a huge range of technologies to choose from ­ – .NET, PHP, Ruby on Rails and Java at the back end to front-end XHTML, JavaScript, Ajax and Silverlight. Together with the increasing variety of platforms, including browser and mobile gadgets, traditional desktop applications are becoming part of the web offering.

Better development tools are now delivering improved reliability and programmers are able to develop feature-rich, reliable, scalable applications that provide a great user experience quickly and easily.

“The learning curves are shortened and flattened out with new technologies often being very fast and easy to learn and implement,” says Howard. “For example, developers no longer need to write thousands of lines of cross-platform or browser-compliant code. By using Ajax they can simply drag and drop components into Visual Studio or Dreamweaver.”

Even with the introduction of componentised tools and vast programming libraries of routines and building blocks, software development is still no easy task. For some firms the overhead of managing a development team has become too much. Outsourcing is now commonplace. While this removes the need to engage expert coders, often the design and specification stages ­ – which can account for more than 50 per cent of the work ­ – remain within the firm. There is also the additional burden of managing an external supplier, whose objectives could contrast wildly with internal imperatives.

And while software development services are being offered globally, internal development shops will not disappear overnight, says Tony Baer, senior analyst at Ovum. “Outsourcing is more about managing the change in the division of labour. However, you get what you pay for and the lowest cost will not necessarily deliver the best results. The lower the cost of the coding skills base, the bigger the investment in management costs has to be made, which is why a proportion of development is coming back onshore and even back in-house,” he says.

Coding, however good, is always prone to error and never before has software testing been more important. Proof is in the way testing is treated ­ – no longer is it seen as a final activity, tacked on to the end of a development project if the budget allows it. Testing today is an integral part of the development process.

Some programming teams even design the testing procedures before embarking on the full specification. “The progression from testing to test early to test first to test driven is the most exciting movement in development I have seen in the past 10 years,” says Zuhlke’s Braithwaite. “Testers have become the users’ friend and developers are learning to love failing tests ­ – counting the test failures creates meaningful progress measures in the development process.”