In the 1820 Census there are only six Lime families listed. Three of them, reside in Pennsylvania (two in the county of Lancaster, an Adam & William, and one in the county of Dauphin, Michael). The other three are all unnaturalized citizens and neighbors in Fairfield, South Carolina. What's interesting about these three are that their names are James, James Sr., and William. I've attached the census image for all to see.
James Lime is 45+
his wife is 26-45
Of note: There is also a James LYME listed on the 1820 Census. He is the only LYME in the whole 1820 census. He is residing in Warren, Tennessee. He is listed as 26-45 with a wife who is between 16-26 and no children. I don't believe this could be our James as he was married in 1825.