When we are young everything seems big and time is slow, when we are old everything is small and time is fast. Each consecutive moment is a decreasing percentage of your total remembered time.
I think I can explain this. When you are 5 years old. 1 year is 20% of your entire life. As you age each year is a smaller percent of your life and time seems to move faster (when measured from a year to year perspective). At 45, 1 year is 2.2% of your total life.
As a child every second is literally a larger chunk in relation to the amount of time you've lived. The longer you live, the smaller the amount of time in relation to your total time lived is. So it doesn't go faster, our perception of it changes. It is quite an interesting phenomena. I notice a lot of people when they reach an older age will often say "Wow time sure flies." But when I was a young child it seemed like I was a child for quite a long time.