The two shots above show the sort of growth that olive trees will put on during a year. The left shows a profusion of basal growth the other other shot show growth lower down on the trunk. Both these sorts of growth need to be removed so as to concentrate the trees energies on growth where you want it and more importantly on the fruit.
This growth is best removed with a hand held axe as shown. I tend to pick off 10 or so trees one day and then do the same the next as the weather is still quite hot.