A fascinating knit of age-old tradition and modernity, this Gulf island draws you in with its wave of contrasts