I use visual separation a lot working approach (arrival). When vectoring to follow from the downwind or base, and particularly when filling a gap on final, it's easy to lose separation technically, even though the airplane will be in the gap once you're done. It's also handy when you're running visual parallel approaches but the airplanes aren't seeing the airport very far out (say, less than 10 miles). You need one airplane cleared for the visual approach or visual separation. Working at LAX during an arrival push, working arrivals to all 4 runways simultaneously... you will say "maintain visual" and "follow" (which is also visual separation) more times than you can count.
As for VFR aircraft, it depends on the airspace. Yes, a controller can definitely have a separation error with a VFR aircraft in B or C airspace, or in a TRSA. Visual separation is neither assumed nor automatic, even for VFR aircraft, as far as the controller is concerned. In that airspace where VFR aircraft must be separated, the controller must apply the separation standards or instruct the aircraft to "maintain visual separation," even if it's a VFR aircraft.
Per the .65, in the Visual Separation section (7-2-1): Separation requirements also apply to VFR aircraft when IFR, Class B, Class C or TRSA separation services are required.
Visual separation for VFR aircraft is also specifically mentioned in the sections for TRSA, B, and C airspace.
There is a difference between see-and-avoid and positive separation. VFR aircraft in those classes of airspace are guaranteed ATC radar separation (in most cases) so the "maintain visual separation" is supposed to impress on the pilot that ATC is relieving itself of that obligation.
(sorry for not completing this thought earlier... I realize the original question was about VFR aircraft)