ISSN (Online): 2321-3418
server-injected
Engineering and Computer Science
Open Access

Transforming Data Warehousing with AI-Driven Innovations in Data Engineering

DOI: 10.18535/ijsrm/v9i2.ec02· Pages: 575-587· Vol. 9, No. 02, (2021)· Published: February 25, 2021
PDF
Views: 302 PDF downloads: 157

Abstract

Data warehousing has been identified as an important component of business intelligence because it provides the infrastructure that organisational databases require in order to support the storage, management and analysis of large volumes of data. However, with the exponential growth of data and the increasing complexity of the analytical workloads, conventional data warehousing systems may fail to address the modern business needs of most organizations. The adoption of AI in data engineering is therefore a revolutionary opportunity for resolving these challenges as it provides novel solutions for the scalability , automation and optimization of data warehouses.

This article aims at analyzing the impact of AI-derived advancements in actual data warehousing with emphasis placed on utilizing the machine learning algorithms, automation methods and practices, and powerful analytics for enhancing data integration, ETL processes and real-time decision support. Current advanced technologies like NLP, predictive analysis, and deep learning are defining new generation data processing capabilities to generate insights at a faster rate and with greater accuracy. These developments are especially important for organizations working with large data as they improve the capability to analyse a great deal of data in real time.

This research examines such factors and advantages, including lower operational costs, enhanced data quality, and faster data processing, demonstrated by case studies and examples from the data warehousing industry. Certain implications for practitioners of implementing AI-based data warehousing solutions are also discussed in the study, such as the problem of data privacy, talents’ shortages, and the integration of historical information systems. It is for this reason that this study finds its place in providing preliminary findings on the direction of data engineering as informed by AI technologies, data warehousing, and more broadly, data-centric organizations.

Keywords

Data WarehousingAI InnovationsArtificial IntelligenceMachine LearningData EngineeringProcessesData IntegrationBig DataReal-Time AnalyticsPredictive AnalyticsData LakesPipelinesCloud Data WarehousesScalabilityData QualityAutomationData Management

References

  1. JOSHI, D., SAYED, F., BERI, J., & PAL, R. (2021). An efficient supervised machine learning model approach for forecasting of renewable energy to tackle climate change. Int J Comp Sci Eng Inform Technol Res, 11, 25-32.Google Scholar ↗
  2. Mahmud, U., Alam, K., Mostakim, M. A., & Khan, M. S. I. (2018). AI-driven micro solar power grid systems for remote communities: Enhancing renewable energy efficiency and reducing carbon emissions. Distributed Learning and Broad Applications in Scientific Research, 4.Google Scholar ↗
  3. Joshi, D., Sayed, F., Saraf, A., Sutaria, A., & Karamchandani, S. (2021). Elements of Nature Optimized into Smart Energy Grids using Machine Learning. Design Engineering, 1886-1892.Google Scholar ↗
  4. Alam, K., Mostakim, M. A., & Khan, M. S. I. (2017). Design and Optimization of MicroSolar Grid for Off-Grid Rural Communities. Distributed Learning and Broad Applications in Scientific Research, 3.Google Scholar ↗
  5. Integrating solar cells into building materials (Building-Integrated Photovoltaics-BIPV) to turn buildings into self-sustaining energy sources. Journal of Artificial Intelligence Research and Applications, 2(2).Google Scholar ↗
  6. Manoharan, A., & Nagar, G. MAXIMIZING LEARNING TRAJECTORIES: AN INVESTIGATION INTO AI-DRIVEN NATURAL LANGUAGE PROCESSING INTEGRATION IN ONLINE EDUCATIONAL PLATFORMS.Google Scholar ↗
  7. Joshi, D., Parikh, A., Mangla, R., Sayed, F., & Karamchandani, S. H. (2021). AI Based Nose for Trace of Churn in Assessment of Captive Customers. Turkish Online Journal of Qualitative Inquiry, 12(6).Google Scholar ↗
  8. Khambati, A. (2021). Innovative Smart Water Management System Using Artificial Intelligence. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(3), 4726-4734.Google Scholar ↗
  9. JALA, S., ADHIA, N., KOTHARI, M., JOSHI, D., & PAL, R. SUPPLY CHAIN DEMAND FORECASTING USING APPLIED MACHINE LEARNING AND FEATURE ENGINEERING.Google Scholar ↗
  10. Joshi, D., Sayed, F., Jain, H., Beri, J., Bandi, Y., & Karamchandani, S. A Cloud Native Machine Learning based Approach for Detection and Impact of Cyclone and Hurricanes on Coastal Areas of Pacific and Atlantic Ocean.Google Scholar ↗
  11. Agarwal, A. V., & Kumar, S. (2017, November). Unsupervised data responsive based monitoring of fields. In 2017 International Conference on Inventive Computing and Informatics (ICICI) (pp. 184-188). IEEE.Google Scholar ↗
  12. Agarwal, A. V., Verma, N., Saha, S., & Kumar, S. (2018). Dynamic Detection and Prevention of Denial of Service and Peer Attacks with IPAddress Processing. Recent Findings in Intelligent Computing Techniques: Proceedings of the 5th ICACNI 2017, Volume 1, 707, 139.Google Scholar ↗
  13. Mishra, M. (2017). Reliability-based Life Cycle Management of Corroding Pipelines via Optimization under Uncertainty (Doctoral dissertation).Google Scholar ↗
  14. Agarwal, A. V., Verma, N., & Kumar, S. (2018). Intelligent Decision Making Real-Time Automated System for Toll Payments. In Proceedings of International Conference on Recent Advancement on Computer and Communication: ICRAC 2017 (pp. 223-232). Springer Singapore.Google Scholar ↗
  15. Agarwal, A. V., & Kumar, S. (2017, October). Intelligent multi-level mechanism of secure data handling of vehicular information for post-accident protocols. In 2017 2nd International Conference on Communication and Electronics Systems (ICCES) (pp. 902-906). IEEE.Google Scholar ↗
  16. Doddipatla, L., Ramadugu, R., Yerram, R. R., & Sharma, T. (2021). Exploring The Role of Biometric Authentication in Modern Payment Solutions. International Journal of Digital Innovation, 2(1).Google Scholar ↗
  17. Han, J., Yu, M., Bai, Y., Yu, J., Jin, F., Li, C., ... & Li, L. (2020). Elevated CXorf67 expression in PFA ependymomas suppresses DNA repair and sensitizes to PARP inhibitors. Cancer Cell, 38(6), 844-856.Google Scholar ↗
  18. Singu, S. K. (2021). Real-Time Data Integration: Tools, Techniques, and Best Practices. ESP Journal of Engineering & Technology Advancements, 1(1), 158-172.Google Scholar ↗
  19. Singu, S. K. (2021). Designing Scalable Data Engineering Pipelines Using Azure and Databricks. ESP Journal of Engineering & Technology Advancements, 1(2), 176-187.Google Scholar ↗
  20. Malhotra, I., Gopinath, S., Janga, K. C., Greenberg, S., Sharma, S. K., & Tarkovsky, R. (2014). Unpredictable nature of tolvaptan in treatment of hypervolemic hyponatremia: case review on role of vaptans. Case reports in endocrinology, 2014(1), 807054.Google Scholar ↗
  21. Shakibaie-M, B. (2013). Comparison of the effectiveness of two different bone substitute materials for socket preservation after tooth extraction: a controlled clinical study. International Journal of Periodontics & Restorative Dentistry, 33(2).Google Scholar ↗
  22. Roh, Y. S., Khanna, R., Patel, S. P., Gopinath, S., Williams, K. A., Khanna, R., ... & Kwatra, S. G. (2021). Circulating blood eosinophils as a biomarker for variable clinical presentation and therapeutic response in patients with chronic pruritus of unknown origin. The Journal of Allergy and Clinical Immunology: In Practice, 9(6), 2513-2516.Google Scholar ↗
  23. Gopinath, S., Janga, K. C., Greenberg, S., & Sharma, S. K. (2013). Tolvaptan in the treatment of acute hyponatremia associated with acute kidney injury. Case reports in nephrology, 2013(1), 801575.Google Scholar ↗
  24. Shilpa, Lalitha, Prakash, A., & Rao, S. (2009). BFHI in a tertiary care hospital: Does being Baby friendly affect lactation success?. The Indian Journal of Pediatrics, 76, 655-657.Google Scholar ↗
  25. Singh, V. K., Mishra, A., Gupta, K. K., Misra, R., & Patel, M. L. (2015). Reduction of microalbuminuria in type-2 diabetes mellitus with angiotensin-converting enzyme inhibitor alone and with cilnidipine. Indian Journal of Nephrology, 25(6), 334-339.Google Scholar ↗
  26. Gopinath, S., Giambarberi, L., Patil, S., & Chamberlain, R. S. (2016). Characteristics and survival of patients with eccrine carcinoma: a cohort study. Journal of the American Academy of Dermatology, 75(1), 215-217.Google Scholar ↗
  27. Swarnagowri, B. N., & Gopinath, S. (2013). Ambiguity in diagnosing esthesioneuroblastoma--a case report. Journal of Evolution of Medical and Dental Sciences, 2(43), 8251-8255.Google Scholar ↗
  28. Swarnagowri, B. N., & Gopinath, S. (2013). Pelvic Actinomycosis Mimicking Malignancy: A Case Report. tuberculosis, 14, 15.Google Scholar ↗
  29. Maddireddy, B. R., & Maddireddy, B. R. (2020). Proactive Cyber Defense: Utilizing AI for Early Threat Detection and Risk Assessment. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 64-83.Google Scholar ↗
  30. Maddireddy, B. R., & Maddireddy, B. R. (2020). AI and Big Data: Synergizing to Create Robust Cybersecurity Ecosystems for Future Networks. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 40-63.Google Scholar ↗
  31. Maddireddy, B. R., & Maddireddy, B. R. (2021). Evolutionary Algorithms in AI-Driven Cybersecurity Solutions for Adaptive Threat Mitigation. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 17-43.Google Scholar ↗
  32. Maddireddy, B. R., & Maddireddy, B. R. (2021). Cyber security Threat Landscape: Predictive Modelling Using Advanced AI Algorithms. Revista Espanola de Documentacion Cientifica, 15(4), 126-153.Google Scholar ↗
  33. Maddireddy, B. R., & Maddireddy, B. R. (2021). Enhancing Endpoint Security through Machine Learning and Artificial Intelligence Applications. Revista Espanola de Documentacion Cientifica, 15(4), 154-164.Google Scholar ↗
  34. Damaraju, A. (2021). Mobile Cybersecurity Threats and Countermeasures: A Modern Approach. International Journal of Advanced Engineering Technologies and Innovations, 1(3), 17-34.Google Scholar ↗
  35. Damaraju, A. (2021). Securing Critical Infrastructure: Advanced Strategies for Resilience and Threat Mitigation in the Digital Age. Revista de Inteligencia Artificial en Medicina, 12(1), 76-111.Google Scholar ↗
  36. Damaraju, A. (2020). Social Media as a Cyber Threat Vector: Trends and Preventive Measures. Revista Espanola de Documentacion Cientifica, 14(1), 95-112.Google Scholar ↗
  37. Chirra, B. R. (2021). AI-Driven Security Audits: Enhancing Continuous Compliance through Machine Learning. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 12(1), 410-433.Google Scholar ↗
  38. Chirra, B. R. (2021). Enhancing Cyber Incident Investigations with AI-Driven Forensic Tools. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 157-177.Google Scholar ↗
  39. Chirra, B. R. (2021). Intelligent Phishing Mitigation: Leveraging AI for Enhanced Email Security in Corporate Environments. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 178-200.Google Scholar ↗
  40. Chirra, B. R. (2021). Leveraging Blockchain for Secure Digital Identity Management: Mitigating Cybersecurity Vulnerabilities. Revista de Inteligencia Artificial en Medicina, 12(1), 462-482.Google Scholar ↗
  41. Chirra, B. R. (2020). Enhancing Cybersecurity Resilience: Federated Learning-Driven Threat Intelligence for Adaptive Defense. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 11(1), 260-280.Google Scholar ↗
  42. Chirra, B. R. (2020). Securing Operational Technology: AI-Driven Strategies for Overcoming Cybersecurity Challenges. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 11(1), 281-302.Google Scholar ↗
  43. Chirra, B. R. (2020). Advanced Encryption Techniques for Enhancing Security in Smart Grid Communication Systems. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 208-229.Google Scholar ↗
  44. Chirra, B. R. (2020). AI-Driven Fraud Detection: Safeguarding Financial Data in Real-Time. Revista de Inteligencia Artificial en Medicina, 11(1), 328-347.Google Scholar ↗
  45. Gadde, H. (2019). Integrating AI with Graph Databases for Complex Relationship Analysis. InternationalGoogle Scholar ↗
  46. Gadde, H. (2019). AI-Driven Schema Evolution and Management in Heterogeneous Databases. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 10(1), 332-356.Google Scholar ↗
  47. Gadde, H. (2021). AI-Driven Predictive Maintenance in Relational Database Systems. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 12(1), 386-409.Google Scholar ↗
  48. Gadde, H. (2019). Exploring AI-Based Methods for Efficient Database Index Compression. Revista de Inteligencia Artificial en Medicina, 10(1), 397-432.Google Scholar ↗
  49. Goriparthi, R. G. (2020). AI-Driven Automation of Software Testing and Debugging in Agile Development. Revista de Inteligencia Artificial en Medicina, 11(1), 402-421.Google Scholar ↗
  50. Goriparthi, R. G. (2021). Optimizing Supply Chain Logistics Using AI and Machine Learning Algorithms. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 279-298.Google Scholar ↗
  51. Goriparthi, R. G. (2021). AI and Machine Learning Approaches to Autonomous Vehicle Route Optimization. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 12(1), 455-479.Google Scholar ↗
  52. Goriparthi, R. G. (2020). Neural Network-Based Predictive Models for Climate Change Impact Assessment. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 11(1), 421-421.Google Scholar ↗
  53. Reddy, V. M., & Nalla, L. N. (2020). The Impact of Big Data on Supply Chain Optimization in Ecommerce. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 1-20.Google Scholar ↗
  54. Nalla, L. N., & Reddy, V. M. (2020). Comparative Analysis of Modern Database Technologies in Ecommerce Applications. International Journal of Advanced Engineering Technologies and Innovations, 1(2), 21-39.Google Scholar ↗
  55. Nalla, L. N., & Reddy, V. M. (2021). Scalable Data Storage Solutions for High-Volume E-commerce Transactions. International Journal of Advanced Engineering Technologies and Innovations, 1(4), 1-16.Google Scholar ↗
  56. Reddy, V. M. (2021). Blockchain Technology in E-commerce: A New Paradigm for Data Integrity and Security. Revista Espanola de Documentacion Cientifica, 15(4), 88-107.Google Scholar ↗
  57. Reddy, V. M., & Nalla, L. N. (2021). Harnessing Big Data for Personalization in E-commerce Marketing Strategies. Revista Espanola de Documentacion Cientifica, 15(4), 108-125.Google Scholar ↗
  58. Nalla, L. N., & Reddy, V. M. Machine Learning and Predictive Analytics in E-commerce: A Data-driven Approach.Google Scholar ↗
  59. Reddy, V. M., & Nalla, L. N. Implementing Graph Databases to Improve Recommendation Systems in E-commerce.Google Scholar ↗
  60. Krishnan, S., Shah, K., Dhillon, G., & Presberg, K. (2016). 1995: FATAL PURPURA FULMINANS AND FULMINANT PSEUDOMONAL SEPSIS. Critical Care Medicine, 44(12), 574.Google Scholar ↗
  61. Krishnan, S. K., Khaira, H., & Ganipisetti, V. M. (2014, April). Cannabinoid hyperemesis syndrome-truly an oxymoron!. In JOURNAL OF GENERAL INTERNAL MEDICINE (Vol. 29, pp. S328-S328). 233 SPRING ST, NEW YORK, NY 10013 USA: SPRINGER.Google Scholar ↗
  62. Krishnan, S., & Selvarajan, D. (2014). D104 CASE REPORTS: INTERSTITIAL LUNG DISEASE AND PLEURAL DISEASE: Stones Everywhere!. American Journal of Respiratory and Critical Care Medicine, 189, 1Google Scholar ↗
Author details
Narendra Devarasetty
Dropbox Inc, 1800 Owens St, San Francisco, CA 94158
✉ Corresponding Author
ORCID ↗
👤 View Profile →