The origins of Castel Vittorio lie in ancient times: many years ago the village had a fortified castle, lookout towers and its own walls.

Originally called Castel Dho, and later on Castelfranco (or Free Castle) it took the name of Castel Vittorio in 1862 in honour of King Victor Emmanuel II of Italy.

The first prehistoric settlements found in the Intemelia valley, at the foot of Mount Bignone, date from the Bronze Age.

The dry-wall fortifications scattered over the surrounding mountains, the so-called "castellari", predate Roman times.

The Romans settled primarily along the coast and around the agricultural plain surrounding Ventimiglia. They introduced the cultivation of vines and the gathering of olives, setting up the first farms and tracing the Via Aurelia.

In the early Middle Ages, frequent Saracen forays prompted some of the villagers in the area to seek refuge in fortified settlements further inland.

As the age of the Communes approached, castles and towers also began to appear along the plain and it was during this time that the Benedictine monks introduced olive cultivation.

Later on the Doria family, lords of Dolceacqua from 1270, started to dominate the political and socio-economic life of these frontier lands.

During the following centuries the House of Savoy began to establish its influence in the area, starting with Pigna and then with Seborga.

The narrow cobbled streets (the famous "carrugi") with their slopes, steps and covered passageways give an idea of how many sacrifices were made in everyday Mediaeval life. As well as meeting requirements of safety in the event of military attack, the stone houses had to defend the inhabitants against the cold, heat and even earthquakes. Initially built only on two storeys (stable and cellar on the ground floor, living quarters on the first floor), as time progressed and as families grew, they rose to as many as six storeys high.

Ubiquitous lovers of the picturesque, it was the English in the late 19th century who renamed these fascinating Mediaeval centres "Rock Villages". The dwellings seem stacked one upon the other, exploiting every inch of the available land.

Throughout the village of Castel Vittorio one finds traces of the original military buildings and medieval structures with their typical slate portals. The walled centre of the village enclosed the tiny square, town hall, church and oldest private houses joined by covered passageways, alleys, steeply sloping lanes and flights of steps.

The village's rivalry with neighbouring Pigna is exemplified in the legendary incident of 1727 when the inhabitants of Pigna stole the bell from CastelVittorio and when, to pay them back, the villagers of CastelVittorio took away the paving stones from Pigna’s town square.