Current smartphone authentication systems are deemed inconvenient and difficult for users on remembering their password as well as privacy issues on stolen or forged biometrics. New authentication system is demanded to be implicit to users with very minimum user involvement being. This idea aims towards a future model of authentication system for smartphones users without users realizing them being authenticated. We use the most frequent activity that users carry out with their smartphone, which is the calling activity. We derive two basics related interactions that are first factor being arm’s flex (AF) action to pick a phone to be near ones’ ears and then once getting near ear using second factor from ear shape image. Here, we combine behavior biometrics from AF in first factor and physical biometrics from ear image in second factor. Our study shows our dual-factor authentication system does not require explicit user interaction thereby improving convenience and alleviating burden from users from persistent necessity to remember password. These findings will augment development of novel implicit authentication system being transparent, easier, and unobtrusive for users.