It can be tough to leave.
That is why I keep as much "leave" time as possible at work.
The guys that snapped that picture pulled up while I was landing the fish and couldn't believe I left right after that. I know a fish like that was a "fluke" in the middle of the day but (here's another stolen one) You gotta be in it to win it. Now if there was a blitz going on, I would have to be a little late back to the office.
I've had days where I've caught three bass in the twenty minutes I can fish out of my forty-five minute lunch break. Fishing for an hour before work is a great way to start the day, and instead of driving home during rush hour, I'll go fish for another hour, or four, in the evening.
It isn't all just about being there when the fish are. The more time you get out, the more you have the opportunity to learn, and your skills will improve.