New developing tissues such as the shoot apical meristem (SAM) and the axillary meristems with low transpiration require high zinc (Zn) levels for their active growth, but the molecular mechanisms underlying the preferential distribution of Zn to these tissues are poorly understood. We showed here that, OsZIP4, a member of the ZIP (ZRT, IRT-like protein), showed high expression in the basal node, tiller bud and shoot SAM at the vegetative stage, but higher expression was also found in the node I at the reproductive stage of rice (Oryza sativa). Immunostaining revealed that the OsZIP4 was localized at the phloem parenchyma cells and phloem transfer cells of enlarged and diffuse vascular bundles in both basal and upper nodes. Knockout of OsZIP4 resulted in significant decrease in development of the tiller buds, but increase in Zn levels were observed in the shoot basal region at tillering stage. Not only less Zn levels in the young panicles at panicle initiation (PT) stage but also in the flag leaf blade and in brown rice during the reproductive growth stage in the knockout lines as compared with the wild-type rice. A short-term experiment with the ⁶⁷Zn stable isotope showed that more Zn was accumulated in the nodes, whereas less Zn in the newly developed meristem tissues, indicating that OsZIP4 is essential for transport and distribute Zn out of nodes to the developing tissues. In short, OsZIP4 located in the node is responsible for unloading Zn from the phloem of enlarged vascular bundles, which is playing an important role in preferential distribution of zinc to the developing tissues in rice, which is critical for their development in rice.