British Prime Minister David Cameron faces a test of his leadership at his Conservative Party's annual conference starting Sunday, after criticism from his lawmakers and a bravura showing by his Labour rival.

The Conservatives, behind in the polls midway through an uneasy five-year coalition with the smaller Liberal Democrats, are under fire for failing to drag Britain out of recession despite tough austerity measures.

But when Cameron addresses the conference on Wednesday, he will also be trying to woo rightwingers in his own party who want a return to core values and a referendum on membership of the European Union.

On the eve of the conference in the central English city of Birmingham, Cameron found himself dragged into a row on abortion after the health minister said he favoured lowering the limit from 24 weeks to 12.

Cameron is facing an unexpected challenge after a resurgent Ed Miliband, leader of the opposition Labour Party, confounded critics of his geeky style with a barnstorming speech at his party's conference last week.

Speaking without notes while walking around the stage, Miliband said he wanted to rebuild Britain as "one nation" -- stealing a slogan traditionally used by the Conservatives.

Miliband's unpopularity since he beat his older brother David to the Labour leadership has long been dubbed the Conservatives' secret weapon, and without it Cameron will have to work harder ahead of elections due in May 2015.

But there are challengers within Cameron's own party too.

British Prime Minister David Cameron gestures during a signing ceremony in Brasilia last month. Cameron faces a test of his leadership at his Conservative Party's annual conference starting Sunday, after criticism from his lawmakers and a bravura showing by his Labour rival

Boris Johnson, the ebullient Mayor of London and former schoolmate of Cameron at Britain's elite Eton College, is widely believed to be building a base for a possible leadership bid.

Johnson, a crowd-pleasing orator who is due to take the stage at the conference on Tuesday, has capitalised on his exposure during the London 2012 Olympic Games in July and August.

Foreign Secretary William Hague is due to speak on Sunday, and finance minister George Osborne on Monday.

British media have widely reported that Cameron's popularity is at an all-time low among his own lawmakers, with right-wing political magazine The Spectator saying last week he was "heading for a crash".

Cameron is certainly heading for a clash with rightwingers over Europe, an issue that has divided the Conservatives for decades, as they urge him to take a stronger stand against Brussels.

They will push him to seek a freeze in the EU budget when leaders meet next month and to oppose a proposed banking union in the eurozone, which they fear could hurt London's status as a financial hub.

British Opposition Labour Party Leader Ed Miliband takes part in a Questions and Answers discussion on the fourth day of the annual Labour Party Conference in Manchester, on October 3. Miliband confounded critics of his geeky style with a barnstorming speech at his party's conference last week

Former defence secretary Liam Fox, who quit amid a lobbying scandal in 2011, said last week that Cameron had to forge a stronger Conservative identity and "renegotiate" Britain's relationship with Europe.

Justice minister Chris Grayling called for more "EU veto moments" from Cameron, who angered other European leaders by keeping Britain out of the EU fiscal pact last year.

Foreign policy hardly got a mention during Miliband's speech, but with 9,500 troops in Afghanistan and Britain offering non-military support to Syria's rebels, Cameron may make some play of it.

Yet while Cameron is increasingly at loggerheads with his lawmakers, his party has been more than able to press the self-destruct button on its own.

New Health Secretary Jeremy Hunt sparked a furious row by admitting in The Times newspaper on Saturday that he backs a reduction in the time limit for abortions.

His comments were a distraction from what was expected to be a major effort by Cameron to focus during the conference on the government's success with the state-funded National Health Service.

Government Chief Whip Andrew Mitchell meanwhile reportedly pulled out of the conference after allegedly calling police at Downing Street "plebs" because they would not let him take his bicycle though the front gates.

Critics said the outburst reinforced the Tories' image of a party of the rich, led by so-called "toffs" educated at expensive private schools.