Delays to work commencing on Cardiff’s St David’s Hall has left a cabinet member on the city council “beyond frustrated”.