The immediate context of the remark is in the middle of a series of instructions about everyday troubles:
Honor widows who are truly widows. But if a widow has children or grandchildren, let them first learn to show godliness to their own household and to make some return to their parents, for this is pleasing in the sight of God. She who is truly a widow, left all alone, has set her hope on God and continues in supplications and prayers night and day, but she who is self-indulgent is dead even while she lives. Command these things as well, so that they may be without reproach. But if anyone does not provide for his relatives, and especially for members of his household, he has denied the faith and is worse than an unbeliever.
—1 Timothy 5:3-8 (ESV)
My reading of this section is that Paul has gotten on the topic of taking care of people who don't have family to take care of them and when he thinks of men who can take care of family, but don't, he reacts strongly. To him, such behavior is unacceptable for a believer. Since they think they are believers and yet do not practice very ordinary responsibility required both by human decency and by the faith, they are worse off than if they weren't believers in the first place. (Perhaps this is related to the concept of "sin that leads to death" in 1 John 5.)
Paul may also be talking about similar men in his second letter to Timothy:
But understand this, that in the last days there will come times of difficulty. For people will be ... lovers of pleasure rather than lovers of God, having the appearance of godliness, but denying its power. Avoid such people. For among them are those who creep into households and capture weak women, burdened with sins and led astray by various passions, always learning and never able to arrive at a knowledge of the truth. Just as Jannes and Jambres opposed Moses, so these men also oppose the truth, men corrupted in mind and disqualified regarding the faith. But they will not get very far, for their folly will be plain to all, as was that of those two men.—2 Timothy 3:1-2a;4b-9 (ESV)
My eye is drawn to the phrase, "having the appearance of godliness, but denying its power." In this case, the men seem like believers, but are actually subverting the faith from the inside. These men would be "worse than an unbeliever" because they are tricksters and lead people astray. An ordinary unbeliever might lead people astray too, but they wouldn't have the appearance of godliness.
Summary
Paul is likely talking about people living within the community of the faith (i.e., the church), but who are not truly believers.
For (much) more information, see The City of God by Augustine of Hippo.