In a saucepan, combine sugar, flour, and salt. Gradually stir in milk. Cook over medium heat, stirring constantly, until the mixture is thickened and bubbly. Cook and stir for 2 more minutes. Remove from heat.
Gradually stir about 1 cup of the hot mixture into the beaten egg yolks, then return all of the egg mixture to the saucepan. Cook and stir for 2 more minutes. Remove from heat and stir in butter and vanilla.
Layer half of the banana slices and half of the vanilla wafers in a serving dish. Pour half of the pudding mixture over the bananas and wafers. Repeat layers with remaining bananas, wafers, and pudding.
Cover and chill for at least 1 hour before serving.