John Cheever's "The Swimmer".

John Cheever's story "The Swimmer" starts off reasonably enough. The protagonist, Neddy Merrill, is lounging about the swimming pool at the home of his friends, the Westerhazys, when a peculiar thought occurs to him: there are so many swimming pools between his current location and his own home eight miles away that he can literally swim home -- with a few jogs across back yards and intervening parkways. However, what begins as a whimsical exercise …

…a woman with whom he has been having an affair; she likewise alludes to the fact that he's borrowed money from her, and rudely refuses to give him any more. Finally reaching his own house in a state of complete exhaustion, he tries the garage doors, but "rust came off the handles onto his hands." The house is empty; his family is gone; and at last we learn how long Neddy's swim has actually been.