Understanding the visual knowledge of language models