Teachers, administrators, testing agencies, and researchers all need a valid, reliable method of assessing writing ability. Each group has turned to holistic ratings of writing samples as a reliable qualitative procedure for responding to the essential features of writing. Yet the validity of holistic ratings has never been convincingly demonstrated. This paper analyzes the implicit requirements for achieving reliable results from holistic ratings and argues that these conditions bring the validity of the ratings into doubt. The research evidence available suggests that even in carefully supervised rating sessions, holistic ratings may be unduly influenced by superficial features of writing samples. Those who use holistic ratings to evaluate writing ability need to give more serious attention to the validity of the scores that may result.