Aboard wooden sailing ships, the "devil" was the seam formed at the juncture of the covering board that capped the ship's sides and the deck planking. The seam was particularly difficult to caulk because it was so long, there was so little space in which to do so, and there was so little standing room between the devil and the sea. See also: between the devil and the deep blue sea.
"'Why, the devil, do you see,' said Jack, 'is the seam between the deck-planking and the timbers, and we call it the devil, because it is the devil for the caulkers to come at: in full we say the devil to pay and no pitch hot; and what we mean is, that there is something hell-fire difficult to be done--must be done--and nothing to do it with. It is a figure.'" -- Patrick O'Brian, The Mauritius Command