Anything to seal under door threshold?

Our front door has always been very cold in the winter, and the frame swells up until the storm door no longer closes. I finally took off the inside trim and, sure enough, whoever installed it didn't do anything to fill the gaps between the rough-in and the door itself. Now, after a can of Great Stuff (Door and Window) and two tubes of silicone (the first was old and blew up on me), all the visible gaps are sealed.

Then I started thinking: what about underneath the door? What does that look like and is there likely to be any sort of gap that needs to be sealed? The inside threshold is a strip of oak: any idea how that might be held on and how to get it off without damaging that or the hardwood floor? Other similar ones in the house have finish nails, but I don't see the nail heads on this one. The light isn't very good there, but do you know of anything else they may have done? Is there enough likelihood of a gap to make it worth taking off while I have the trim off?

 

Either the wood is swelling from being wet(no good), or there is frost heave taking place. Keeping water from penetrating around the door should stop both. Is there a basement or crawl space under/at the doorway?? I wonder what kind of wood is under the threshold, and the condition. Sounds something needs to be replaced.

 

I was working on the theory that the gaps between the rough frame and door frame were responsible for both the drafts and the buildup of moisture in the winter. I haven't seen any sign of rot or mold.

The threshold isn't an issue: the side frames bow in enough to prevent the door from closing, but not enough to see. I saw no rot or mold in either the rough-in or the door frame, from what I can see with the inside trim off. If this is caused by warm humid air escaping in the winter, is there likely to be anything under the threshold to seal, or does sealing around the other three sides of the frame cover it pretty well? I hesitate to experiment, since I can't see any easy way to get the threshold off without damage.

Underneath is a block wall down into the basement, so frost heave seems unlikely. The rim joist (behind the block) has about 3" fiberglass batting: nowhere near enough for Massachusetts. I do plan on pulling and replacing that if I ever get a couple more hours this summer. There is also a hole for the power to the AC that I need to make sure is sealed.

I'm not sure what else to look for, although if I could afford to foam in the walls I would.

 

Caulking/sealing any gaps around the outside can solve a lot of problems, at least for a while. But the swelling thing raises flags.