Scalawag,
You are right. You do need to know the area well. Best way would be to go out with someone who knows the area. However, I am sure you would be fine out there on your own. All you really have to do is take your time, pay close attention to your charts, and most of all, follow the locals carefully.
I would suggest, heading out there on nice calm weekday, so the traffic is light and you can take your time. I don't know what kind of boat you have, obviously a smaller boat would help. Once you get a handle on the bay and inlet while it is nice, visit the inlet on a rougher day. You will see the break and the holes I have spoke of.
Since you are not a Brookhaven resident, you would have to launch from the Smith Point Marina, County Permit required, and make the approximately four mile run down the bay to the inlet. As you can see, the bay is bouyed but unfortunately, the cuts to the inlet aren't. This is where the charts and local knowledge come in most of all.
The fishing is worth it.
Anything else I can do to help send me an e-mail.