How do I let someone use my trademark for a limited purpose?

You should have an experienced intellectual property attorney prepare a trademark licensing or assignment contract between you and your brother, and execute it before you take on your partners.  You will need to disclose this contract to  potential partners, and you and your attorney should discuss whether there is a need to include specific provisions in your partnership agreement that protect your brother’s interests.

A license is a contractual agreement whereby a licensor (you) permits a licensee (your brother) to use intellectual property (in this instance, trademark)  under express circumstances.  Here, it seems that you want to grant a trademark license to a sole licensee for the purpose of clothing sales, while you and your new partners retain the rights to use the trademark for all other business purposes.  Normally, trademark licenses are granted in exchange for “royalty” payments, however you can continue to take no money from your brother if that is your choice.  The contract will need to set a fixed term for the license, as well as the conditions under which it may be renewed for an additional period of time or terminated for breach of the license conditions.

The assistance of a qualified legal professional is necessary here – trademark rights can be lost through improper licensing or assignment.  For example, the contract should make it clear that you are monitoring your brother’s use of the restaurant’s trademark; if your trademark is licensed without adequate quality control or supervision by the trademark owner (you), your new partners could ask the Patent Office to cancel the trademark registration and/or go to court to challenge your brother license agreement.