The paper proposes a pose-based real-time system for inferring the engagement of a shopper with a retail shelf by recognizing some atomic actions of shelf-interaction. These actions include examining the shelf, reaching for an object, taking an object, reading a product’s label and placing it on a cart for check-out. A novel pose-representation that is robust to large intra-class variations while performing these retail actions, is proposed in this work. The paper also extends the framework to do real-time action segmentation, abnormal action detection and configurable privacy protection of shoppers. The abnormality detection also offers a scope for learning new un-modelled actions through crowdsourcing. Though the system currently relies on a Kinect sensor (RGBD) for computing the joints of the human body, the system can work with a combination of RGB surveillance camera and any 2D video-based pose-tracking algorithm. The system has an accuracy of 90% in recognizing the 5 actions considered in this work and exhibits a latency of about 1 sec w.r.t. real world action. This can have a huge potential in optimizing store resources and in improving the shopping experience of the customer.