Nelson House The site of the Nelson House was home to Hendrickson’s Tavern in the 18th century. Thomas Jefferson and James Madison. Alexander Hamilton told stories over drinks there. The site today is recognized with a historic marker.