David Friedman writes, "Many libertarians who favor free immigration in principle have serious reservations about its implications in practice." 
You can count me among those libertarians.
Although I strongly believe that people ought to be able to travel, work, and live wherever they like (imaginary lines in the sand notwithstanding), I'm aware that there are potential problems with an open border policy.
I'm still studying the issue.
1. David Friedman, The Machinery of Freedom: Guide to a Radical Capitalism (Chu Hartley, 2014), kindle location 5337.