How to Add a Vector Mask to a Layer in Photoshop CS6
Whereas layer masks let you create soft-edged masks in Photoshop CS6, vector masks create hard-edged masks defined by shapes created by a vector path on a layer. Vector-based shapes produce clean, smooth, and well-defined edges that are never jagged. And you can size and transform vector shapes without ever degrading the appearance of the element.
You create a vector mask when you create a shape with any one of the shape tools. You can also create a vector mask when you convert type to a shape. (Choose Layer→Type→Convert to Shape.)
To add a vector mask to layer, follow these steps:
Select the layer in the Layers panel and choose Layer→Vector Mask→Reveal All or Hide All.
Remember that you can’t add a vector mask to a background layer.
Choose Window→Paths. In the Paths panel, select the Vector Mask layer.
Select the Pen tool or grab any of the Shape tools.
On the vector mask, create the desired shape for your vector mask.
Notice how everything inside the path (represented by the white area on the vector mask thumbnail) is visible and everything outside the path (represented by the gray areas on the vector mask thumbnail) is hidden, or masked. If necessary, edit your path with the Direct Selection tool (white arrow). As you edit the path, the mask dynamically updates.Credit: ©iStockphoto.com/EXTREME-Photographer Image #18577493
Like layer masks, you can add vector masks only to layers, not backgrounds. If necessary, simply convert your background to a layer by double-clicking the background in the Layers panel.
If your vector mask is satisfactory, save your file and then close it.
Another way to add a vector mask is to select your desired layer, draw a work path with the Pen tool or one of the shape tools, and then select Layer→Vector Mask→Current Path.