William Casey, the former Reagan campaign manager and now CIA director, told the New York Times he wouldn't have touched the Carter briefing plans ''with a 10-foot pole.'' The Times noted that Mr. Casey's remarks on the controversy differed with the recollection of White House chief of staff James Baker, who has said the papers came from Mr. Casey. Casey said he might have given Mr. Baker some papers without studying them but found it unlikely he could have overlooked hundreds of pages of Carter materials.