In the 1880s fishermen began to leave Carteret County, N.C., and go to the mullet fishing grounds of Southwest Florida, where they made new homes in places such as Cortez and Punta Gorda.