Tunisia said it had informed Libya of its "extreme indignation".
Rebels and pro-Gaddafi troops were fighting for control of the border post between Dahiba and Wazin, which rebels captured a week ago.
Shells fired by pro-Gaddafi forces are said to have landed within Tunisia.
The government troops apologised to their Tunisian counterparts after briefly crossing the border, Reuters news agency reported.
But Tunisian authorities warned of a "dangerous military escalation".
"Shots fired at a populated area on Tunisian territory (are) a violation of Tunisia's territorial integrity and a violation of the security of the residents of this region", said a statement from the Tunisian foreign ministry.
"Given the gravity of what has happened ... the Tunisian authorities have informed the Libyans of their extreme indignation and demand measures to put an immediate stop to these violations."Misrata shelling
Fighting in western Libya, near the border with Tunisia, has intensified over recent days.
Rebels seized the border post a week ago, and pro-government forces have been trying to retake it.
Thousands of people have fled from Libya across the frontier to flee the violence.
Elsewhere, there were reports that government troops had retaken the southern city of Kufra following violent clashes.
In the capital, Tripoli, at least two loud explosions were heard late on Thursday as Nato jets flew overhead.
And in the western city of Misrata rebels reported continuing bombardments by pro-Gaddafi forces.
The rebel-held city has been besieged by forces loyal to Col Gaddafi for several weeks.
Those forces have retreated from the centre of Misrata, but rebels say they are shelling residential areas from the outskirts.
Earlier, a rebel commander and witnesses told reporters a that a Nato air strike in Misrata on Wednesday had killed at least 11 rebels.
A Nato official confirmed alliance aircraft had struck "a number of combat vehicles 10 miles south-east of Misrata port", adding that Nato could not "independently verify reports that these vehicles were operated by opposition forces".
He said there had been no Nato attack on any building in or around Misrata.
Send your pictures and videos to firstname.lastname@example.org or text them to 61124 (UK) or +44 7725 100 100 (International). If you have a large file you can upload here.
Read the terms and conditions
This post was made using the Auto Blogging Software from WebMagnates.org This line will not appear when posts are made after activating the software to full version.