Ile de Rey is great. we spent 3 weeks there about 2 years ago in May, St Martin de Ray is the big town on the North side of the island.
Rich day trippers mostly, nice restaurants and the like..Savage ice cream by the harbour.
St Marie de Rey on the South side is nice too but way more residential.
There is a place Bois up by the big beach (Plage Gros Jonc I think) where all the campsites are, very Nice altogether. its supposed to be rammed in the high summer but rent a few bikes and you'll be away. It's flat and there are bike paths everywhere though the vineyards.
For small kids the Aquarium in La Rochelle is great. Also depending on the time of year a Top 14 match in La Rochelle was brilliant. Loads of atmosphere with big brass bands and Fireworks etc. You get resale tickets on their website but the season is over by summer