I don't think any type of caulk or foam is going go take the heat of the exhaust. Take a piece of sheet metal cut a more exact diameter hole. If you can take the exhaust off put the metal in place and re-install the exhaust. If you can't maybe you can slit the sheet metal. Similar to what I did on the front of my unit.
fiberglass insulation around the pipe and into the hole, then screw down the proper size escutcheon
or make your own as Shane suggested .
(Actually Shane's suggestion is what sparked the thought of a split flange escutcheon )
owens corning sells a fire retardent insulation with a foil type backing. you might want to check that out. along with using a tin circular coupler as a type of flashing.
you can go to the hard ware store and just get a furnace heat flange....they protect the roof from getting hot when the riser pipe comes off the furnace and exhaust out...it slips into the hole and protects the floor and the opening....