While the Oliphant Clan was mainly centered in the central highlands (in the counties of Perthshire, Angus, Fife etc.), lands were also held in the Lowlands. After David Olifard returned to Scotland with King David I, he was made Justicary of the Lothians, and was granted land in Roxburghshire, including Drumsagard (now known as Cambuslang, just west of Bothwell,) Smailholm and Crailing, along with the great barony of Bothwell. His son and grandson, both named Walter, lived in Roxburgh (the exact location of the homes are unknown, as they were probably made of wood and long since destroyed,) as Justicary's of the Lothians. The Lordship and lands near Bothwell (in Lanarkshire) passed through David's great grandaughter, to Walter of Moray (of Clan Murray), where he built Bothwell Castle, probably as an addition on the Oliphant fortification there. The "jury is still out" as to whether the great Donjon at Bothwell was built by the Oliphants or the Murrays. There are many Olivers who claim descent from these early Olifards of the Borders and do not accept that there is any connection with the Oliphant Clan. However this has been proved untrue because David Olifard's (Justicary of Lothians and Lord of Bothwell) son was a nephew to Osbert Olifard at Arbuthnott (which formed part of the feudal Barony of Bothwell), definitively linking the border Oliphants to the highland branches. Since the Oliphants were the heirs to the ancient lands held by the earliest Holifards/Olifards in Perthshire and Kincardine, tradtionally passed to the eldest son, it can be assumed that the Oliphants are the senior line. That any Olivers claiming descent from the same Olifards, means that the Olivers are descended from the younger sons of these early Olifards/Oliphants.
Most of the lands in Roxburgh held by Oliphants were passed through marriage or sold to the surrounding Clans, further pointing to the conclusion that the lands of Aberdalgie in Perthshire were those held the longest, as the Chiefs were reinstated there while the Border lands were given away as dowries.