Obvs dont know the specific rules of your house, but I had similar with the neighbours with a shared fence and the normal and most common land registry split, is that the fence on your right is theirs and the fence on your left is yours.....but that isn't always exclusively the case.
We couldnt...