The second generation Boltorn-type hyperbranched polyester (HBP) with 16 hydroxyl at the end was synthesized by melting polycondensation reaction. It was based on pentaerythritol (B4) as a core molecule and 2,2-dimethylolpropionic acid (AB2) as monomer and p-toluenesulfonic acid as catalysis. To form the modified hyperbranched polyester GHBP1 and GHBP2, the hydroxyl was further modified by methacryloyl chloride and propionyl chloride introducing all or part of the double bond at the end of HBP. Adding different proportions of GHBP to the conventional dental restorative resin system of bisphenol A glycerolatedimethacrylate (Bis-GMA) and triethylene glycol dimethacrylate (TEGDMA), its influence on the volumetric shrinkage and double bond conversion was investigated. The results indicated that modification with GHBP lead to an obvious decrease in the volumetric shrinkage but a slightly change in the degree of conversion. GHBP2 exhibited better reducing volumetric shrinkage properties.