A mental model is a person's understanding of how someting works based on their past experiences, perceptions, what other people have told them, etc.
Mental models are what people use to assist with figuring out problems that are placed before them.
With regards to design, a mental model applies to the representation of something, be it an object, an idea, etc.
Mental models have the ability to change as peoples' knowledge and experiences change.
Understanding the mental models for your target audience is a critical component to successful design.