Coiled tubing (CT) has been increasingly used in offshore oil and gas engineering. However, a number of issues should be solved to guarantee reliability and safety of the application. In this study, a mechanical model of CT-in-riser system is extracted from actual operations. Moreover, micro elements of the CT and riser are respectively analyzed using the infinitesimal method, and then a coupling model is derived. After differential equations are replaced by algebraic equations, the Newton iteration method is used to solve the coupling model. To validate above theory, an experimental platform based on the similarity laws is built. There are two types of experiments to be investigated, including the effects of fixed and floating boundaries, and the effects of three annular clearances. The results show that the force-out has a similarly increasing trend along the force-in under the fixed and floating boundaries, and the larger the annular clearances, the lower the transfer efficiency of the axial force. The maximum error between calculation results and experimental results is within ±5%, which validates the accuracy of the coupling model. The study will provide a theoretical support for CT's operations in offshore engineering.