It appears so, it was called other things by native people but the Dutch seem to be the first to call it Long Island in the 1600s. Many geographic features in the area have similar sort of names, like Short Hill, East River, West River, Indian Hill, Short Beach, Beaver Swamp, the colonists really weren’t very clever with their naming.
In my experience people in the Northeast US pretty much never refer to New York City as just ‘New York’. New York will mean the state, they will say ‘the city’ or ‘NYC’ to differentiate from the state. Or they will say upstate to mean the rest of the state outside of NYC metro area.