In this research, the modeling of a solar renewable system along with the use of intelligent optimization method was discussed in order to increase the system performance and reduce the life cycle cost. A system with a combination of five units of PVT panel, heat pump, PEM electrolyzer, hot water storage source and fuel cell was designed to provide the required energy consumption of an 80-unit residential complex in Iran, including hot water, cooling, heating and electricity. The performance of the proposed system was analyzed by using the weather information of the coastal city of Bandar Lengeh in Hormozgan province. System modeling was done using a new approach by TRNSYS software. Optimization of this system to increase system performance and reduce life cycle cost was done by Design Expert software and using RSM. The three objective functions investigated included electricity production, PMV and LCC, and four decision variables including the number of solar panels, fuel cell capacity, heat pump capacity and electrolyzer capacity were selected. The optimization results showed that in the most optimal state with the number of 1425 solar panels, the capacity of 65.99 kW for the fuel cell, the capacity of 40.01 kW for the heat pump and the capacity of 52.5 kW for the electrolyzer are in the optimal state. The system can reach the production value of 725955.25 kWh per year in the optimal condition with a PMV of 0.43 and a LCC of 344014.63 $.