If you have that map, the GM can’t throw an obstacle at you for moving around between locations on the map. If you don’t, he may. The general act of moving around does not require its own test.
My interpretation/explanation is that if you have an area mapped, you can skip the navigation test (Pathfinder above-ground, sometimes Dungeoneering underground) usually needed to “fast-travel” anywhere covered by the map. Because you don’t take a test, it doesn’t use a turn, so no accrued conditions or triggering twists, etc. Even without a map, you wouldn’t test just to walk to a location you can see.
Just to clarify, map in this case refers to an item in your gear that was purchased / drawn by a character, not a physical map that the players drew, correct?