Firstly, well played for letting it happen despite the rule: it is one of those ridiculous technicalities. However, technically you are right, and for the purpose of arguments over other unforeseen circumstances, I've found it safest to just always play the rules, as otherwise there's a case of "but you let that rule about this and that slide last game, and this is similar": sometimes they help you, sometimes they hurt you.
However, it seems unfortunate that his back two dwarfs are on the right - it's so close that by my count, if he had positioned them on the left, he'd have more in sight of your flank. For this reason alone I think I would have done the same and allowed the flank, even when sure of the rule.
Tbh, when I watched this I thought your Grave Guard would pop from this flank which would have been doubly annoying! I think it's nice it worked out as a draw.