With
all
of the other issues with dragon physics, this part stands out to be
the most plausible. Depending on the time period of the castle, the
materials used to build castles were wood, stone, and mortar.
The wood could easily be burned down with dragon's fire. The stone and
mortar would easily crumble to such a massive creature with plenty of
momentum, as well as claws.
google image