By Ayebare Mariam
Kashiwa Reysol forward Olunga drilled in a late winner, having earlier made it 1-1 with a superb overhead kick and also heading against the crossbar.
Tanzania had gone ahead twice through Simon Msuva’s tap-in and Mbwana Samatta’s effort for a half-time lead.
Johanna Omolo flicked in a header at the near post for Kenya, before Olunga struck with just 10 minutes remaining.
The result moves Kenya level on three points with Senegal, who were beaten 1-0 by leaders Algeria, while Tanzania exit the tournament still searching for their first victory.
Kenya plays Senegal and Tanzania face Algeria in the final group games on Monday at 20:00.