In the days of the early church, all churches were initially under the direct oversight of apostles (some of whom were prophets) or their surrogates (e.g. Timothy and Titus). Only after a church would reach a sufficient level of maturity would the oversight be placed in the hands of local elders/pastors (Acts 14:23). Once a church is placed in the hands of local elders/pastors, the apostles may continue to have a voice in the assembly, but the oversight authority is with the local pastors. Even apostles must submit to the local leadership.
The problem today is that we have men called pastors who are gifted church planters (apostles), but they stay on too long and never turn the church over to a team of local elders. We also have men called pastors who do not meet the qualifications enumerated by Paul in 1 Timothy 3 and Titus 1. Both of these situations lead to many of the problems we see in single-pastor assemblies today.