This paper presents a dynamic association and natural interaction method for multi-displays composed of smart devices. Users can intuitively associate relations among smart devices by shake gestures, flexibly modify the layout of the display by tilt gestures, and naturally interact with the multi-display by multi-touch interactions. First of all, users shake their smart devices to create and bind a group for a multi-display with a matrix configuration in an ad-hoc and collaborative situation. After the creation of the group, if needed, their display layout can be flexibly changed by tilt gestures that move the tilted device to the nearest vacant cell in the matrix configuration. During the tilt gestures, the system automatically modifies the relation, view, and configuration of the multi-display. Finally, users can interact with the multi-display through multi-touch interactions just as they interact with a single large display. Furthermore, depending on the context or role, synchronous or asynchronous mode is available to them for providing a split view or another UI. We will show the effectiveness and advantages of the proposed approach by demonstrating implementation results and evaluating the method by the usability study.