The only things that are considered the rare items are the items that say "an item most rare!" at the end of the message when your retainer brings them back. This is a separate loot pool from the other quick venture returns with specific items like cordials, dyes, the rare minions, and other things in it. Most of the "items most rare" are not actually that rare or worth much. Many "common items" that you don't need any special gear to get are actually quite valuable still. The message basically means nothing unless you specifically want something from that "items most rare!" loot pool.