Managing AI and Data Science Projects: Unveiling Secrets from Industry Experts for Real-world Success

Abstract

Data science projects are becoming increasingly critical in today’s data-driven landscape, yet effectively managing their complexities remains challenging, and this often results in the project's underperformance and or failure. This study provides comprehensive guidelines for critical aspects of data science project management - from initiation and implementation to scaling. Both primary and secondary data were gathered using mixed research methodology. Twenty-six (26) experts from international companies were engaged with a well-structured questionnaire, and selective companies successful in previous data science projects were approached to gather primary data. For secondary data, papers published in reputable journals were researched systematically to support claims from the primary data. However, based on an extensive literature review and critical sampling of opinions, significant challenges for data science projects identified include a lack of feedback-constrained timeframes, project complexity, the inability to meet high stakeholder expectations, the inability to enable informed decisions, and the iterative nature of data science is underscored, along with nuanced expositions on navigating the potential challenges of science data projects. Finally, other best practices identified through literature searches and case studies include regular audits, effective metadata management, data privacy and security, optimal cloud storage for cost-effectiveness, and computation acceleration. Further, recommendations were provided, which contributed to positioning this paper as a valuable guide for practitioners, researchers, organizations, and individuals in the field.