Congress is on Recess so why does our Congressman, Chris Smith, stay in Virginia?

Congressman Smith lives full time in Virginia. He has lived in Virginia since his second term as District 4 Congressman over 30 years ago. He maintains a small residence in Monmouth County NJ that meets the housing requirement of having a habitat in the state he Represents in order to keep his seat as our Congressman.  He raised his family in Virginia and his children received Virginia "In state" tuition.  When other Representatives visit their home states and meet with their constituents during recess, Chris Smith goes home to Virginia.  He does not come to NJ and he does not meet with his constituents.  Why do we put up with it?