I think my optimal group size for PVE open world content, that is questing not instanced fixed-group dungeons, is 2 or 3. Doing this sort of content requires communication, there are usually diverse quest lines or activities available at a given moment so you need, collectively, to agree what to do in what order. Otherwise you end up with characters dying solo to a group of monsters because they took a left instead of a right, or in LoTRO people end up at different stablemasters.
I have always preferred questing together, even though I’ve done my fair share of solo play in some games. Some games support this playstyle better than others, Guild Wars 2 certainly does if you don’t mind helping each other on personal story missions. WoW used to until Cataclysm nerfed difficulty through the floor and added so much phasing it was too much hassle to stay in step. LoTRO is or at least was a shining example of plentiful challenging content for groups out in the open.
A key factor of group questing is that the group can set their own pace. I find just following a zerg of random players fun for a bit (in GW2 or Rift) it’s fun but gets tiring. Often there are lore snippets or stories to absorb and I hate being rushed on that, I also cannot bring myself to ignore vistas.
So a small questing group is best for me, it also allows us to play with combos, since getting any combos at all in a zerg is likely just random luck.
Pacing is important for longer-term enjoyment of any game, so don’t hesitate to take a break from the zerg!