Inbound text not associating to client

We are getting inbound SMS via Twilio and they are not being associated to the proper contact despite the phone numbers matching.  We are having to manually do the association to the contact.  Anyone have a fix or work around??