Meaning

noun a port in western Israel on the Mediterranean; incorporated into Tel Aviv in 1950

originFrom Jaffa, an ancient port city in Israel from which the oranges were exported. The sense "impotent male" is a reference to being "seedless"; see seed. The cricketing sense, of a 'sweet ball' (from the bowler's perspective), is a metaphor further reinforced in Australian English where Jaffa is also a trademark for a popular lolly that is red and round.

Etymology from Wiktionary, CC BY-SA 3.0