The short, safest and most efficient answer to this question is yes. According to the national code, with relation to electrical panels in closets: NEC 240.24D Overcurrent devices shall not be located in the vicinity of easily ignitable materials, such as a clothes closet.
Encyclopedia of energy sources