Karzai also admitted that he has not heard from the US president since Barack Obama moved into the White House last month.
After leading Afghanistan for over eight years, Karzai is now seen by many Western officials as part of the problem rather than one who can force through solutions.
Obama is expected to approve the deployment of about 30,000 extra US troops to Afghanistan in the next few days, despite fears in some circles that the mistakes of the Soviet Union are being repeated.
The deployment is seen as vital for securing presidential elections - only the second in Afghanistan`s history - set for August 20 after being postponed for three months largely because of security concerns.
The comments by Karzai to Al Jazeera come as speculation grows over his political fate.
Reporting from Kabul on Friday, Al Jazeera`s Hamish MacDonald said Holbrooke was due to meet Afghanistan`s defence and interior ministers before holding talks with Karzai on Saturday.
"Holbrooke will meet at least one of four men who could potentially take over from Karzai in elections to be held later this year," our correspondent said.
"At the moment, there is a perception that Karzai is losing favour not just at home but also internationally, so the fact that Holbrooke is meeting some of the possible replacements is significant."
Holbrooke is also expected to discuss with Afghan leaders a Russian offer to open up new supply lines into Afghanistan for international forces following several attacks on existing routes.
There are fears that the security situation is deteriorating in and around Kabul after the Taliban launched one of their most audacious raids on the city to date on Wednesday, killing 20 people in a co-ordinated assault on three government buildings.
Kabul says most of the opposition fighters are based in Pakistan`s remote tribal regions to the east of Afghanistan.
Holbrooke could ask officials why they cannot even secure government infrastructure in Kabul, our correspondent said.
"This is clearly something Holbrooke will be thinking of as he is not only responsible for Afghanistan, as far as the US is concerned, but also for Pakistan."
Holbrooke believes the security forces fighting the Taliban and al-Qaeda have to be stronger.
There are now 80,000 US and Nato soldiers in the country, backed up by the Afghan army and police.
But most Nato nations are reluctant to contribute more; they also do not want their forces in a frontline role.
In Washington on Thursday, Al Jazeera`s Anand Naidoo asked Robert Wood, the US state department spokesman, if the planned US deployment was likely to go ahead.
"The president has made it very clear that Afghanistan is going to be a priority and that we need to do more in terms of fighting the Taliban," Wood said.
Xenia Dormandy, a former South Asia section head at the US National Security Council, told Al Jazeera: "There is a debate because they truly do not know what the border Afghan policy is going to be over the coming years.
"They are still trying to decide what are going to be the objectives over the next year to two years. And until the new administration comes up with these new objectives, the question of how many forces and how much US forces you need is very much up in the air."