My own thoughts, from the perspective of someone who's a bit behind due to IRL health issues but also really doesn't wanna be here but is forcing themself to go through it anyway for the mounts and such at the end (I'll try not to let that bias creep in TOO much):

1. Island Sprint/Return are redundant. In any other content that isn't just the main map, Return is modified for the content in question, taking you back to the start of a dungeon/raid, back to your home base in PvP or Eureka/Bozja-style zones, etc. Here, there's two different Returns to juggle, and using the wrong one puts me all the way back in Radz-at-Han rather than my island base camp--user error, to be sure, but it's user error due to 6+ years of muscle memory with my standard Return shortcut suddenly working against me for the first time. Feels like something could've been done about that. Also, as others have stated, having to keep re-upping Sprint (reduced timer or not) rather than just having it on a toggle like they did for PvP recently feels like a step backwards.
2. the orchestrion is kinda not worth. Maybe I'm just too used to the static playlists at my or my FC's houses that keep playing until I change them, but the idea that the base camp doesn't save my music selection when I leave and return is a bit annoying. I'm sure it's the same at inn rooms as well (dunno for sure, never played the orchestrion there), but I don't generally feel like taking time out of every island visit to get my radio working properly again, so I've learned to just tolerate/tune out the overly-twangy default track and move on with my day rather than have something I enjoy playing there.
3. the mode selection thing feels clunky to me. I'm sure they were trying to cut down on sub-menu pop-ups as much as possible here, but having to shift between "modes" to gather or capture in the wild, feed or pet my pastured animals, water or sow seeds...like I said, it all feels clunky to me, especially when crafting something is a separate thing you can do independent of all that. I think I honestly preferred the sub-menus from standard gardening over this.
4. lack of lore on cowries. This one's kind of a nitpick, I'll admit, but most currencies at least get some kind of passing explanation when they're introduced--Allagan tomestones get a nice long explanation of what they are, why the House wants them, and what you get out of trading them away, and wolf marks, ventures, scrips, and GC/allied/Centurio seals all get at least a passing "we use this currency for this content" explanation when you unlock the thing they get used for. Here, the first time I recall the NPCs mentioning the existence of cowries was when one of the mammets asked me to fork a few over for an expansion, which felt out of place in the game overall. And on that note...
5. paying your reward currency away. This feels, among other things, like a holdover from the games they're trying to emulate that doesn't feel at home to me in FF14 as a whole, and ultimately just seems to needlessly gatekeep your progress towards the end rewards further than it needs to. Imagine having to pay wolf marks to go into frontlines, or having to give up tomestones to unlock a dungeon or run roulettes. The idea that you buy the rewards from the island with the same currency you have to spend to make the island worthwhile in the first place doesn't feel good; I don't feel like I'm making progress towards my goals when my currency count ticks down in large quantities every time I unlock a new patch of land or upgrade my pasture/cropland but only ticks back up at a trickle when a workshop project finishes. It's one thing with the game's main economy to have to "spend gil to make gil" in most cases, but again, you don't need to spend tomestones to make tomestones, or spend wolf marks to make wolf marks, so I don't get why cowries are the exception where the game's non-gil currencies are concerned.
6. workshop allocation. I'm not a fan of the idea that the island essentially lets you "double dip" on your materials and potentially hinder your progress by assigning workshop tasks that you don't have enough mats for, or that you do have the mats for at the time but end up using elsewhere by accident as part of some other crafting/renovation task. Sure, you can check your allocations tab and see where/if you have any supply deficiencies, but I don't get why they couldn't just remove the mats from your inventory right when you assign your workshop crew a task. Not only does that feel simpler from a gameplay standpoint, it also feels smarter from a supply management standpoint to cross materials off your available-to-use list if you've allocated them to something already.
7. the workshop in general. This, honestly, is where the idea of the Island Sanctuary falls apart completely for me. As I said, I don't generally enjoy this sort of gameplay, but up until rank 4 I could switch myself into task mode and go out and gather what I needed for any given thing, treating the whole thing like a variation on a standard crafting/gathering list I'd do for gear or bardings or whatever else in the main game. Island exp was sparse as far as individual gathering nodes, crafting attempts, or animal captures went, but it came steadily enough by way of crafting or expanding facilities, which gave me something to focus my attention on as I went. Then rank 4 hit, and while I fully expected the leveling curve to kick in somewhere, I wasn't expecting it to dry up as badly as it has. Tasks come at a trickle now, and my only real source of both cowries (which, again, I can't spend yet to get the things I actually want from this place because I need around 7.5k for future island expansions and such) and island exp boils down to a glorified spreadsheet simulator and a supply/demand curve so imaginary that my customers aren't even worth animating. Sure, this is supposed to be casual/relaxing content, but this isn't even really under my control; all I can do is assign tasks to my shop 'bots as best I can and...wait. It's like sending my retainers or adventuring squad on a mission, except instead of knowing what I'm getting into ahead of time, the content just kinda sprung the idea on me after I spent the first few ranks expecting to be an active participant. That's barely even content at that point, it's just me waiting while the game plays itself (especially when I eventually reach the point where, from what I understand, automating the process entirely and just check in for the rewards every day or two becomes the most efficient way of doing things). Casual content can still be stimulating, and it can still provide ways for folks to go ham if they really want to--far as things like the actual AC or SV games go, my wife plays both just kinda doing whatever, while some of our FC mates get really in-depth about planning their facilities out and min/maxing their activities. Island Sanctuary doesn't seem interested in that--at best, I can spend maybe 20 minutes to an hour gathering things for new tools or new/expanded facilities at each new rank-up, and then I'm at the mercy of the workshop timer regardless of the amount of effort I want to invest. That, or spend hours monotonously grinding away at gathering nodes for 10 exp each, which is just as un-fun for me because the gains are tiny compared to what I need for a rank-up and the process itself is kinda aimless. I feel like they should have gone more in-depth and involved with the actual building side of things, giving folks a more constant stream of tasks to do at whatever pace they wanted in order to get through the leveling process, and relegated the workshop side of things specifically to earning cowries--even if that meant I eventually hit a wall and had to wait a bit because I leveled too quickly for my cowrie earnings to keep up and let me do any given expansion, I'd still feel more like an active participant in the content than I do now.

But again, I acknowledge that this content isn't for me and that I'm kinda just brute-forcing my way through it for the mounts and such. Doesn't mean I don't wish they'd done a few things differently to make it a more palatable process, though.