"...would have followed him to Jotunheim if he bade them." (War Of The Gods, p. 261)
Centuries later, Skafloc and Mananaan MacLir do make this journey. However, when Harald Hardrada attempts it, something different happens. His fleet finds only icebergs and a whale and must turn back. Poul Anderson's The Last Viking Trilogy, about Hardrada, is neither heroic fantasy nor historical fantasy but historical fiction. Jotunheim does not exist. Either the supernatural realms were always imaginary or they have somehow withdrawn as in Anderson's The Broken Sword or in JRR Tolkien's Middle Earth History. For further treatment of this idea, see here.