The town is named after Simon van der Stel, an early governor of the Cape Colony. The land rises steeply from near the water's edge and the picturesque village is boxed in along the shoreline by the heights above. The small harbour itself is not a particularly good natural harbour and is protected from swells by a breakwater that was built with thousands of huge blocks of sandstone quarried out of the face of the mountain above. 

Simon's Town is now in effect a suburb of greater Cape Town. It is the terminus of a railway line that runs south from the central business district of Cape Town. In places the railway line hugs the steep eastern shore of False Bay quite spectacularly and in bad weather foam from some heavy swells will fly up and wet the trains! 

One of the best beaches on the Cape Peninsula, Boulders Beach, is located a few kilometres to the south of Simon's Town. Here small, secluded coves with white sandy beaches and calm, safe, warm, shallow waters are interspersed between huge rounded boulders of Cape granite that form low cliffs and small caves. 

There has been a colony of African penguins at Boulders Beach since 1985. There is no record of the birds having lived here prior to that date, so their decision to settle in an area already well-utilized by humans is remarkable. There are only three penguin populations on the mainland in southern Africa. (The others are on islands).