During lulls in the night fighting in Baghdad’s Sadr City last year, as Muqtada al Sadr’s militia turned Baghdad’s biggest ghetto into the most booby-trapped war zone on earth, it used to look to me like someone was staging Macbeth in hell. With the dark air full of dust and smoke, human figures moved over the pavement like black ghosts while car lights swerved crazily through the smog.

The spectres around me were mostly involved in planting the homemade bombs known as improvised explosive devices, or IEDs—the insurgency’s main weapons in Iraq. The swerving cars were avoiding the Coke cans that indicated the buried bombs. And the youths hunched over the road pouring liquid into the dark bitumen would explain to me that it was kerosene they were dishing out: relatively viscous, it seeps into a road surface, and then, when lit, melts it, making digging easier. Thus the orange flames that flared all night along the boulevards. The ordinance most likely to be buried in the small pits then were 105mm howitzer shells that the guerillas called “Austrians,” after the country where the shells had been made. Wires led from detonation charges into the doorways of small, shabby mosques where other groups of teenagers in black stood around car batteries attached to the wires.I never had contact with the other side of this battle—coalition forces—back then, but an NBC cameraman I knew told me that from the inside of a 7th Cav Bradley, these young men on the streets looked like video game targets through the thermal night vision screens inside the American armoured vehicles. One night the Bradley my friend was travelling in was hit by twelve IEDs.

The IED might seem like a relatively low-tech piece of weaponry in a military epoch of lasers, unmanned drones and smart bombs. And it might appear a humble opponent for a US military establishment 3m strong that consumes $400bn a year. But it is the defining weapon of America’s war in Iraq, and it has been the focus of a battle of innovations and counter-innovations marshalling high-tech gadgetry and low-tech cunning on both sides.

When military historians write the annals of this struggle, they will remember it as the "IED war." The IED is responsible for 80 per cent of American casualties in Iraq, and it is unprecedented in modern warfare to find a conflict so dominated by a single weapon.

The most recent of the many American innovations deployed in the IED struggle has been the CROWS turret system, which allows a Humvee’s roof-mounted heavy weapon to be fired remotely from inside the vehicle’s protected shell by a soldier using a video screen monitor and a joystick.

Earlier this year I rode across central Iraq in a big loop from Camp Anaconda, near Baghdad, to Anbar province, the heart of the Sunni insurgency, in the first CROWS-mounted Humvee to be deployed into the war’s IED epicentre. At the beginning of the journey, waiting to leave the wire on a cold and starry night before driving out into the “red zone” beyond Anaconda’s perimeter, about ten soldiers from our convoy gathered around the vehicle that had the CROWS. It was a typical scene: the system causes a sensation wherever troops in Iraq see it for the first time.

With Dean Martin’s “That’s Amore” playing from speakers we had set up on the roof of the Humvee, the soldiers were standing around looking through the open back-left door of the vehicle, passing around a bag of crisps like any bunch of kids anywhere, watching a TV—which in a sense the CROWS is: its monitor is a 15” screen, fixed to the back of the driver’s seat. The gunner manning the CROWS, a former stevedore in a tough light infantry air assault unit, explained to the others that it had a 27x zoom in daylight, white-heat and black-heat night vision that can see five thousand metres in the dark, pre-set fire zones and so on.

“ But the best thing,” said Wade, “is that I’m not up there in the turret any more. I’m not gonna get shot at. And if an IED goes off, I’m safe inside the vehicle.” In a war in which most coalition casualties come from IEDs detonated by mobile phone calls or radio signals dialled in by the hands of terrorists whom the troops never see, this is what resonated with the other soldiers. “No more lost arms and split wigs,” said Wade’s sergeant, 41-year old Hector Rodriguez from the Bronx.

But Wade, Rodriguez and the others probably know that an innovation like the CROWS is rarely a permanent answer in a war of ingenuity like the one they are waging in Iraq. In the IED fight, both sides are innovating as quickly as they can. Marines training in California before deployment to Iraq are told about the IED battle: “Anything we do, Haji will take two weeks to get around.” The story began back in 2003 and 2004, when American soldiers were still driving around Iraq in unarmoured Humvees (today they aren’t allowed to leave the wire in one), and insurgents were burying artillery shells in the roadbeds.

When the armoured Humvees started to arrive, a more potent and directed blast was needed, so “Haji” started using anti-tank mines instead of artillery shells. Then the occupying troops got better at disrupting the emplacement of IEDs, and the insurgents, with less time, increasingly started to leave them by the roadside.

As coalition efforts to detect roadside bombs have improved, so has the sophistication of insurgents’ attempts to conceal them. Carcasses of dogs and donkeys, roadside ice stands and “discarded” kitchen gas canisters are all being used.

It’s like pest control, or the antibiotics business: one side develops a response, and the other side innovates around it, creating a pattern that resembles the mutations and life and death evolutions of biology, speeded up.

Anti-tank mines, for example, produce a blast that goes straight up, which means the target has to be directly above the device, and as the insurgents now frequently lack the time to dig them into tarmac roadbeds, they dig them into soft shoulders and medians where they have observed that coalition troops park or turn frequently. Now the soldiers are careful about where they turn and park, so the insurgents increasingly lure their targets down dirt roads. The insurgents used to use wires, like the ones I saw in Sadr City, to detonate the bombs. Once the Americans started spotting and following the wires, the enemy began to set up additional bombs along the wires. When the Americans started using armoured Combat Engineer units to follow the wires, the insurgents began to detonate via mobile phones, garage door openers and controls for children’s toys.

Trying to understand the insurgents’ ability to improvise, an American infantryman in Iraq once surmised to me that under the sanctions regime in Saddam’s day, Iraqis had had to learn to make do with such limited resources that today the insurgency draws on a public patrimony of small-scale ingenuity. It is a local, up-to-date version of a myth Americans have often applied to themselves. “American ingenuity,” it used to be called: a culture of tinkering and physical problem-solving that made MacGyver a major television icon for the generation who are now platoon leaders and company commanders in America’s military. It is part of the same American self-definition that credited Patton’s swift armoured advance through Europe in 1945 to the mechanical skills of a now-bygone generation of American farm kids and hot-rodders.

The CROWS is just one of the innovations America is deploying in the IED war. Last November, a hundred Talon robots started shipping out to Iraq, at a cost of $250,000 each. The 3’ x 4’ moon-rover type gadgets are used to blow up detected or suspected IEDs. (In my experience a round from the heavy gun on a Bradley is much more typical.) The Army’s single ZEUS, its first battlefield laser system, is another example. Mounted on top of a Humvee, it is now in Iraq after a successful tour in Afghanistan, zapping IEDs with a joystick-controlled invisible high-powered laser beam. Forget Star Wars: ZEUS looks like a big cardboard box with a small window in it.

On the American side, it’s not all high technology and massive procurement bureaucracies. Early in the war, individual platoons were armouring their own Humvees with whatever scrap metal they could find, and the wire cages built around their vehicles by the Stryker Brigade in Mosul have been effective against RPG rounds. Marine Captain Jonathan Kuniholm, a reservist and doctoral candidate at Duke University, is one of these troops whose ingenuity MacGyver or General Patton might recognize. With two weeks to go before deploying to Iraq last August, he and some fellow marines decided that the platoon Kuniholm led would benefit from having its own robot for identifying and destroying IEDs.

For $1200, Kuniholm’s engineering firm made a gizmo based on a remote-controlled Monster Truck platform. Called Bubba, it shipped out to Iraq in Kuniholm’s footlocker and is still being used in the field, along with a second model. Kuniholm’s firm is now involved in a secret multi-company project to supply the Pentagon with equipment related to the IED war. While he was still in Iraq—his right arm was blown off on New Year’s day by an IED hidden in an olive oil tin, and he is now back in North Carolina, switching his PhD to Prosthetic Research—Kuniholm received contacts from other soldiers in theatre who wanted to buy versions of Bubba on their personal credit cards.

With armour only a partial safeguard, interrupting the detonation of IEDs has also become a priority. At a cost of $56m, 1,440 Warlock radio-frequency jammers were scheduled to be delivered this summer, aiming to prevent mobile phone and radio devices from activating the explosives. Even better than interrupting the call would be to fry the bomb’s electronics altogether. The Naval Surface Warfare Centre in Virginia is working on a device called NIRF (Neutralizing Improvised Explosive Devices with RF) that aims to do so with microwave blasts of electromagnetic energy.

The Pentagon set up an IED task force in June of last year, and the army has had its own IED task force since April 2003—the month after the invasion of Iraq. The marines have projects of their own via the Office of Naval Warfare (ONR) and the Marine Corps Warfighting Laboratory. In May, Congress approved an additional $129m for the IED struggle. Jay Cohen, director of the ONR, has called America’s technological effort to address IEDs “a Manhattan-like project.” At the Naval Research Laboratory alone, 75 scientists are focused on the IED problem, according to Cohen. Cliff Anderson, a programme manager at the ONR, says that in over 30 years of working in the military innovations business, the IED effort is “one of the largest things I’ve seen the department of defence attempt to do in terms of people assigned to finding creative solutions” to a military problem.

Despite all of the resources America is able to bring to bear on the technological side of this war against old howitzer shells hidden in donkey carcasses, Anderson believes it is the kind of technological struggle that by its nature cannot have an end. “We are competing against other people,” he says. “You’re not trying to discover the principles of nature,” with their limited, definite and absolute characteristics. For this reason, says Anderson, “I don’t see an end in sight from a technical perspective.”

Soldiers also know that there is only so much that technology can achieve in warfare. Kuniholm says that even had Bubba been with him on the patrol that cost him his arm, the olive oil can IED would probably still have got him: “It wouldn’t have aroused our interest,” he says. “It points out the difficulty of dealing with this sort of threat—you can only do so much about it.”

His words reminded me of Rodriguez telling a bunch of younger soldiers gawking at his CROWS system, “Once Haji figures out this thing has a camera, he is going to try to take out the camera. If the camera goes, you need a spotter up there, and with the turret armour gone he doesn’t have any protection. And with nobody up top you can’t hear what’s going on.”

“ It’s like power steering,” said Rodriguez. “It’s easy, but sometimes it’s too easy.”