Had a similar thing in our back garden, but from your neighbours point of view. Had a large 40' sycamore which was literally 6' from their side wall. Now their house was built a long time after the tree was planted, so their foundations took it into account thank gawd.
I wanted the tree down...