Ah... the joys of being on a boundary!

You do just have to vary it depending on who you are talking to. I've got an SE23 postcode but live in Southwark so it can get interesting on occasion (e.g. when the police are figuring out which SNT team might be interested!)

In general, when talking to someone who knows South London, I say Forest Hill and reference the Horniman. If they look blank, I move to near Dulwich and then 'a bit further east than Brixton' if they are completely clueless.