British actress Kate Winslet was among those staying on Necker Island in the British Virgin Islands when a tropical storm broke out and lightning hit the house, said Sir Richard.
The Virgin Group boss, who was staying at a nearby property, said the fire had "completely destroyed" the building.
He said all 20 people in the house managed to escape without injury.
Sir Richard's 90-year-old mother Eve and his 29-year-old daughter Holly were also staying at the property, named the Great House, when the blaze broke out in the early hours of Monday.
"We had a really bad tropical storm with winds up to 90mph," said Sir Richard, who was staying at a nearby property with his wife Joan and son Sam, 25, at the time.
"A big lightning storm came around 04:00 and hit the house.
"My son Sam rushed to the house and helped get everyone out.
"Kate Winslet, her boyfriend and her family were there and Holly and some of her friends were also staying. My mother was there and they managed to get her out and she is fine.
"The main house is completely destroyed and the fire is not yet completely out. My office was based in the house and I have lost thousands of photographs which is very sad."Each bedroom at the Great House has its own balcony, four-poster bed and en-suite bathroom
Winslet, 35, who won an Oscar for her role in the 2008 movie The Reader, has a daughter Mia, 10, and a son, Joe, seven.
Sir Richard, 60, bought Necker Island in the early 1980s and began building the eight-bedroom Great House in 1982.
It was expected to be the venue for the marriage of Holly and shipbroker Freddie Andrews later this year and is used by Branson and his friends, as well as being hired out.
"It's very much the Dunkirk Spirit here," said Sir Richard.
"We want to rebuild the house as soon as we can."