The whole town fits in a half-day. Most visitors from Torre del Mar drive up mid-morning (10 minutes on the MA-3202), park near the Cervantes Theatre, and walk up through La Villa toward the fortress.
La Villa is the surviving old town — whitewashed houses, narrow alleys, the occasional patio open to the street. From there it's a steep ten-minute climb to the Fortaleza. The castle itself is mostly ruined walls, but the platform on top has the single best view in the Axarquía: Torre del Mar's beaches one way, the Sierra de Tejeda the other.
Coming back down, Santa María la Mayor is open most mornings. It was built on top of the old congregational mosque after Vélez-Málaga was taken by the Catholic Monarchs in 1487; the mudéjar tower is the giveaway. From there the Palacio de Beniel and the Cervantes Theatre are a short walk back toward the parking.
For lunch, the bars around the old town are honest, local and far cheaper than the Torre del Mar paseo. After lunch, most people either drive back to the beach, or detour through El Ingenio for any actual shopping the seafront can't cover.