Partial referendum results show 58% voting no, and 42% supporting the plan.
"The worst option was chosen. The vote has split the nation in two," Prime Minister Johanna Sigurdardottir said on state TV.
It is the second time a referendum has rejected a repayment deal, and the case will now go to an international court.
Landsbanki ran savings accounts in the UK and Netherlands under the name Icesave and investors there lost 4bn euros (£3.5bn; $5.8bn).
When it collapsed in 2008, the British and Dutch governments had to reimburse 400,000 citizens - and Iceland had to decide how to repay that money.'Such a revulsion'
"The Icelandic nation has been put in a terrible situation," Reykjavik voter Helgi Sigurdsson told the Associated Press news agency.
"It has two choices - both are bad. Probably a lot of people stood for a long time holding the ballot slip."
Parliament had backed this deal, but President Olafur Ragnar Grimsson refused to sign it, leading to the referendum.Prime Minister Johanna Sigurdardottir: "Disappointing numbers"
A previous deal, imposing a tougher repayment regime, was rejected in a March 2010 referendum by 93% of voters.
Finance Minister Steingrimur Sigfusson appeared to rule out a third attempt to persuade voters to accept a repayment deal.
"One cannot really ask a whole nation to agree to such a revulsion," he said.
"I think we're getting a very clear sign from this referendum, that further negotiations are ruled out. No use in trying that again."
The issue will now be referred to an international court, the European Free Trade Association Surveillance Authority. a process which could take several years.
Backers of a "yes" vote had argued the repayment deal was the best way to resolve the issue in terms of cost and risk to Iceland.
The "no" camp said the Icelandic taxpayer was under no legal obligation to pay for a private bank's losses and that the deal would put a heavy burden on the nation.Longer period, lower interest
Under the terms of the rejected deal, Iceland would have paid the money back with 3.3% interest to the UK, and 3% to the Netherlands, over 30 years between 2016 and 2046.
Under the previous proposal, the money was to be paid back with 5.5% interest between 2016 and 2024.The issue may now have to go to the courts
The actual cost to the state is expected to be much less than the 4bn euros owed, as the government says most of the repayment will come from selling the assets of Landsbanki.
The government has said it does not expect the cost to exceed 50bn kronur (£168m).
Analysts say a resolution of the issue is vital to Iceland's prospects for recovery because it would allow the country to return to the financial markets to fund itself.
Solving the dispute is also seen as key to Iceland's chances of joining the EU.
Iceland's three main banks collapsed within days of each other in October 2008.
The government compensated Icelandic savers, but overseas customers faced losing all of their money.
The issue sparked a diplomatic row between Iceland and the UK, and created uncertainty over Iceland's economic recovery.
This post was made using the Auto Blogging Software from WebMagnates.org This line will not appear when posts are made after activating the software to full version.