Color agnosia is a medical or psychological condition that prevents a person from correctly associating hue names with common objects. The sufferer retains the ability of distinguishing hues. It is a specific form of agnosia

Agnosia

Agnosia is a loss of ability to recognize objects, persons, sounds, shapes, or smells while the specific sense is not defective nor is there any significant memory loss...

 and generally results from damage to the visual cortex

Visual cortex

The visual cortex of the brain is the part of the cerebral cortex responsible for processing visual information. It is located in the occipital lobe, in the back of the brain....

, often in V4 (as opposed to most other kinds of color blindness, which stem from problems with the photoreceptor cells).