In production systems, the buffer capacities have usually been assumed to be fixed during normal operations. Inspired by the observations from the real industrial operations, a novel concept of Adaptive Buffer Space (ABS) is proposed in this paper. The ABS is a type of equipment, such as movable racks or mobile robots with racks, which can be used to provide extra storage space for a production line to temporarily increase certain buffers’ capacities in a real-time fashion. A good strategy to assign and reassign the ABS can significantly improve real-time production throughput. In order to model the production systems with changing buffer capacities, a data-driven model is developed to incorporate the impact of buffer capacity variation in system dynamics. Based on the model, a real-time ABS assignment strategy is developed by analyzing real-time buffer levels and machine status. The strategy is demonstrated to be effective in improving the system throughput. An approximate dynamic programming algorithm, referred to as ABS-ADP, is developed to obtain the optimal ABS assignment policy based on the strategy. Traditional ADP algorithms often initialize the state values with zeros or random numbers. In this paper, a knowledge-guided value function initialization method is proposed in ABS-ADP algorithm to expedite the convergence, which saves up to 80% computation time in the case study.