We have performed a safety assessment of mercury and lead as possible hohlraum materials for Inertial Fusion Energy (IFE) targets, including for the first time a comparative analysis of the radiological and toxicological consequences of an accidental release. In order to calculate accident doses to the public, we have distinguished between accidents at the target fabrication facility and accidents at other areas of the power plant. Regarding the chemical toxicity assessment, we have used the U.S. DOE regulations to determine the maximum allowable release in order to protect the public from adverse health effects. Opposite to common belief, it has been found that the chemical safety requirements for these materials appear to be more stringent than the concentrations that would result in an acceptable radiological dose.