Another disadvantage is that vector will move things around as the array expands an contracts and this may result in a delete of one of those pointers which will also delete the Object.
My understanding of the dangers of vectors is opposite to this, if you have a vector of pointers, vector<TYPE *> as you resize (reduce in size) the vector the pointers are destroyed but the object they point to is not. You have to handle the memory pointed to yourself (hence your suggestion to use a handle template).
A disadvantage of vector<TYPE> that you have not mentioned is that because of the vectors need to hold its memory contiguously as it would be held if it were an actual array if you insert data anywhere but at the end of the vector or if inserting at the end of an vector causes the array to need a larger capacity, or any other operation that causes the vector to need a larger capacity the contents of the array need to be copied around the array. If TYPE is large this can have an effect on performance. If TYPE has pointers to allocated memory in it and no copy constructor/assignment operator this can be disastrous.
And as an aside you quite often see it said that you can use a vector of vectors as a 2 dimensional array. However where as vectors holds its memory in the same layout as a 1 dimensional array a vector of vectors does not hold its memory in the same layout as a 2 dimensional array. This point would only be relevant if you where interfacing with some legacy code that still used arrays rather than vectors and I am not saying you couldn't program you way round it.
My point is that it is good to be aware of how the various containers hold their memory and what you can rely on and what you shouldn't rely on.