When I first bought my present (holiday) house, my initial thought was renting it out as a holiday house (I'm not much a fan of the high season myself, so would basically be able to make the most of the high income period, while having the house for my own pleasure off season).
After running some numbers, I decided to drop the idea. The raw income figures that I could have got looked good on paper, but after setting aside a stake for the rental agency, for taxes, plus accounting for general wear and tear, utility bills etc, plus having to furnish the house for holiday use and make everything child-proof rather than with my own personal stuff (antiques etc, which i don't want other people touching) it would basically have meant I wouldn't really have felt at home in my own house.
So I finally decided not to let the house.
Maybe if you're looking at a prime waterfront location or something that is otherwise highly sought after and for which you can charge high rents, the situation changes. But my place is not in that category. I know several people who have used their houses as holiday lets but came to regret it.