{"id":14382,"date":"2026-08-27T17:53:40","date_gmt":"2026-08-27T12:23:40","guid":{"rendered":"https:\/\/www.scaler.com\/blog\/?p=14382"},"modified":"2026-08-27T17:54:01","modified_gmt":"2026-08-27T12:24:01","slug":"data-engineering-projects","status":"publish","type":"post","link":"https:\/\/www.scaler.com\/blog\/data-engineering-projects\/","title":{"rendered":"12 Data Engineering Projects to Build Pipelines Employers Actually Trust"},"content":{"rendered":"\n

Your ETL notebook runs without any errors. It reads a CSV, cleans the data in pandas, writes a new CSV, and finishes successfully. But what happens when you have to run the same process every day?<\/p>\n\n\n\n

The file might arrive twice, a column name might change, yesterday\u2019s data might show up three hours late, or the job might fail while you\u2019re not around to fix it. These are the situations you need to think about when you move from a working ETL script to a data pipeline.<\/p>\n\n\n\n

That\u2019s what these 12 data engineering projects focus on. You\u2019ll find projects covering batch ETL, data warehousing, streaming, CDC, and lakehouse architectures. For each one, we\u2019ll cover the stack you can use, the difficulty level, the expected build time, and the skills you\u2019ll get to practise. <\/p>\n\n\n\n

<\/span>What Hiring Teams Actually Look For in Data Engineering Projects<\/strong><\/span><\/h2>\n\n\n\n

When you build a data engineering project, you\u2019ll need to think about what happens when the pipeline runs again, the source data changes, a file arrives late, or a job fails halfway through. You\u2019ll also need to make decisions about the data model, pipeline design, and tools you use.<\/p>\n\n\n\n

These details are important because they give you something concrete to discuss when you explain your project in an interview.<\/p>\n\n\n\n

When an interviewer looks at your project, four questions matter:<\/p>\n\n\n\n

    \n
  1. Can you model the data? Show that you understand tables, relationships, data grain, and how the data should be structured.<\/li>\n\n\n\n
  2. Can the pipeline run safely again? A rerun shouldn’t create duplicates or corrupt existing data. This is where idempotency and incremental loading matter.<\/li>\n\n\n\n
  3. Do you know when something breaks? Show how you validate data, handle errors, and monitor the pipeline.<\/li>\n\n\n\n
  4. Can you explain your decisions? Why batch instead of streaming? Why PostgreSQL instead of a cloud warehouse? Your reasoning matters more than simply having Kafka or Spark in the stack.<\/li>\n<\/ol>\n\n\n\n

    You can also review the core skills expected of data engineers: 12 Essential Data Engineer Skills<\/a>.<\/p>\n\n\n\n\nScaler Carousel<\/title>\n\n<!-- Plus Jakarta Sans Font -->\n<link href=\"https:\/\/fonts.googleapis.com\/css2?family=Plus+Jakarta+Sans:wght@400;500;600;700&display=swap\" rel=\"stylesheet\">\n<!-- Swiper CSS -->\n<link rel=\"stylesheet\" href=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.css\">\n\n<style>\n.scaler-carousel {\n font-family: 'Plus Jakarta Sans', sans-serif;\n max-width: 900px;\n margin: 24px auto;\n padding: 0;\n box-sizing: border-box;\n}\n.scaler-carousel *,\n.scaler-carousel *::before,\n.scaler-carousel *::after {\n box-sizing: border-box;\n}\n\n\/* Header *\/\n.scaler-header {\n text-align: center;\n margin-bottom: 20px;\n}\n.scaler-header-title {\n font-size: 22px;\n font-weight: 700;\n color: #1a1a2e;\n margin: 0 0 6px;\n line-height: 1.2;\n}\n.scaler-header-subtitle {\n font-size: 14px;\n color: #6b7280;\n margin: 0;\n font-weight: 500;\n}\n\n\/* Swiper *\/\n.swiper.scaler-swiper {\n padding-bottom: 48px;\n}\n.scaler-swiper .swiper-slide {\n height: auto;\n display: flex;\n align-items: stretch;\n}\n\/* Navigation arrows \u2014 styled to match the topics HorizontalScroll controls *\/\n.scaler-swiper .swiper-button-next,\n.scaler-swiper .swiper-button-prev {\n width: 46px;\n height: 46px;\n border-radius: 8px;\n background: #ffffff;\n box-shadow: 0 8px 16px 0 rgba(50, 71, 102, 0.16);\n top: calc(50% - 24px); \/* centre on the cards, ignoring the pagination space below *\/\n transform: translateY(-50%);\n margin-top: 0;\n}\n.scaler-swiper .swiper-button-prev { left: 10px; }\n.scaler-swiper .swiper-button-next { right: 10px; }\n\/* hide Swiper's default font-icon arrow and draw the topics chevron via a mask *\/\n.scaler-swiper .swiper-button-next::after,\n.scaler-swiper .swiper-button-prev::after {\n content: \"\";\n width: 20px;\n height: 20px;\n background-color: #61738e;\n -webkit-mask: url(\"data:image\/svg+xml,%3Csvg xmlns='http:\/\/www.w3.org\/2000\/svg' viewBox='0 0 32 32'%3E%3Cpath d='M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z'\/%3E%3C\/svg%3E\") no-repeat center \/ contain;\n mask: url(\"data:image\/svg+xml,%3Csvg xmlns='http:\/\/www.w3.org\/2000\/svg' viewBox='0 0 32 32'%3E%3Cpath d='M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z'\/%3E%3C\/svg%3E\") no-repeat center \/ contain;\n}\n.scaler-swiper .swiper-button-prev::after { transform: rotate(-90deg); }\n.scaler-swiper .swiper-button-next::after { transform: rotate(90deg); }\n.scaler-swiper .swiper-button-disabled {\n opacity: 0.35;\n pointer-events: none;\n}\n.scaler-swiper .swiper-pagination-bullet-active {\n background: #255aad;\n}\n\n\/* Card *\/\n.scaler-card {\n display: flex;\n flex-direction: column;\n gap: 14px;\n width: 100%;\n background: #ffffff;\n border: 1px solid #e5e7eb;\n border-radius: 16px;\n padding: 24px;\n box-shadow: 0 2px 8px rgba(0, 0, 0, 0.06);\n transition: box-shadow 0.2s ease;\n}\n.scaler-card:hover {\n box-shadow: 0 6px 16px rgba(0, 0, 0, 0.10);\n}\n\n\/* NSDC badge *\/\n.scaler-badge {\n display: inline-flex;\n align-items: center;\n align-self: flex-start;\n background: #f3f4f6;\n color: #111827;\n font-size: 12px;\n font-weight: 600;\n padding: 4px 10px;\n border-radius: 20px;\n white-space: nowrap;\n letter-spacing: 0.1px;\n}\n\n\/* Top: icon + title + description *\/\n.scaler-card-top {\n display: flex;\n align-items: flex-start;\n gap: 12px;\n min-height: 100px;\n}\n.scaler-icon-wrap {\n display: flex;\n align-items: center;\n justify-content: center;\n width: 64px;\n height: 64px;\n border-radius: 12px;\n background: #f3f4f6;\n color: #374151;\n flex-shrink: 0;\n}\n.scaler-icon-wrap svg {\n width: 26px;\n height: 26px;\n}\n.scaler-card-top-content {\n flex: 1;\n min-width: 0;\n display: flex;\n flex-direction: column;\n gap: 5px;\n}\n.scaler-title {\n font-size: 16px;\n font-weight: 700;\n color: #111827;\n margin: 0;\n line-height: 1.35;\n}\n.scaler-desc {\n font-size: 13px;\n color: #6b7280;\n margin: 0;\n line-height: 1.45;\n}\n\n\/* Stats row *\/\n.scaler-stats {\n display: flex;\n border-top: 1px solid #e5e7eb;\n border-bottom: 1px solid #e5e7eb;\n padding: 12px 0;\n}\n.scaler-stat {\n display: flex;\n flex-direction: column;\n align-items: center;\n gap: 2px;\n flex: 1;\n min-width: 0;\n text-align: center;\n}\n.scaler-stat:not(:last-child) {\n border-right: 1px solid #d1d5db;\n}\n.scaler-stat-value {\n font-size: 14px;\n font-weight: 700;\n color: #111827;\n white-space: nowrap;\n}\n.scaler-stat-label {\n font-size: 12px;\n color: #9ca3af;\n}\n\n\/* Company logos *\/\n.scaler-logos {\n display: flex;\n align-items: center;\n gap: 12px;\n flex-wrap: nowrap;\n}\n.scaler-company-logo {\n height: 20px !important;\n width: auto !important;\n max-width: 70px;\n object-fit: contain;\n display: inline-block;\n}\n.scaler-logo-more {\n font-size: 12px;\n color: #9ca3af;\n white-space: nowrap;\n}\n.scaler-logo-strip {\n display: block;\n width: 100% !important;\n max-width: 100%;\n min-width: 0;\n height: auto !important;\n object-fit: contain;\n}\n\n\/* CTA *\/\n.scaler-card-btns {\n margin-top: auto;\n}\n.scaler-card-btns button {\n display: flex;\n align-items: center;\n justify-content: center;\n gap: 8px;\n width: 100%;\n padding: 12px 16px;\n border: none;\n border-radius: 10px;\n background-color: #255aad;\n color: #ffffff;\n font-family: 'Plus Jakarta Sans', sans-serif;\n font-size: 14px;\n font-weight: 600;\n cursor: pointer;\n text-decoration: none;\n transition: opacity 0.15s ease;\n}\n.scaler-card-btns button:hover,\n.scaler-card-btns button:focus {\n opacity: 0.88;\n color: #ffffff;\n}\n.scaler-card-btns button svg {\n flex-shrink: 0;\n}\n\n@media (max-width: 1000px) {\n .scaler-carousel { max-width: 96vw; }\n}\n<\/style>\n\n\n<div class=\"scaler-carousel\">\n <div class=\"scaler-header\">\n <h2 class=\"scaler-header-title\"><span class=\"ez-toc-section\" id=\"transform-your-career\"><\/span>Transform Your Career<span class=\"ez-toc-section-end\"><\/span><\/h2>\n <p class=\"scaler-header-subtitle\">Choose from our industry-leading programs designed for career success<\/p>\n <\/div>\n\n <div class=\"swiper scaler-swiper\">\n <div class=\"swiper-wrapper\">\n\n <!-- CARD 1 -->\n <div class=\"swiper-slide\">\n <div class=\"scaler-card\">\n <div class=\"scaler-badge\">NSDC Certified<\/div>\n <div class=\"scaler-card-top\">\n <div class=\"scaler-icon-wrap\">\n <svg width=\"26\" height=\"26\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><polyline points=\"16 18 22 12 16 6\"\/><polyline points=\"8 6 2 12 8 18\"\/><\/svg>\n <\/div>\n <div class=\"scaler-card-top-content\">\n <h3 class=\"scaler-title\">Modern Software and AI Engineering Program<\/h3>\n <p class=\"scaler-desc\">Master full-stack development with AI integration<\/p>\n <\/div>\n <\/div>\n <div class=\"scaler-stats\">\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">12 Months<\/span><span class=\"scaler-stat-label\">Duration<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">AI-Led<\/span><span class=\"scaler-stat-label\">Curriculum<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">Career Support<\/span><span class=\"scaler-stat-label\">Support<\/span><\/div>\n <\/div>\n <div class=\"scaler-logos\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/2\/2f\/Google_2015_logo.svg\" alt=\"Google\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/a\/a9\/Amazon_logo.svg\" alt=\"Amazon\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/42\/Paytm_logo.png\" alt=\"Paytm\" loading=\"lazy\">\n <span class=\"scaler-logo-more\">+1000 more<\/span>\n <\/div>\n <div class=\"scaler-card-btns\">\n <button onclick=\"window.open('https:\/\/www.scaler.com\/academy\/', '_blank')\">\n Go to Program\n <svg width=\"14\" height=\"14\" viewBox=\"0 0 16 16\" fill=\"none\" aria-hidden=\"true\"><path d=\"M3 8H13M13 8L9 4M13 8L9 12\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"\/><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <!-- CARD 2 -->\n <div class=\"swiper-slide\">\n <div class=\"scaler-card\">\n <div class=\"scaler-badge\">NSDC Certified<\/div>\n <div class=\"scaler-card-top\">\n <div class=\"scaler-icon-wrap\">\n <svg width=\"26\" height=\"26\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><ellipse cx=\"12\" cy=\"5\" rx=\"9\" ry=\"3\"\/><path d=\"M3 5v14c0 1.66 4.03 3 9 3s9-1.34 9-3V5\"\/><path d=\"M3 12c0 1.66 4.03 3 9 3s9-1.34 9-3\"\/><\/svg>\n <\/div>\n <div class=\"scaler-card-top-content\">\n <h3 class=\"scaler-title\">Modern Data Science and ML with specialisation in AI<\/h3>\n <p class=\"scaler-desc\">Advanced data science techniques with AI specialization<\/p>\n <\/div>\n <\/div>\n <div class=\"scaler-stats\">\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">12 Months<\/span><span class=\"scaler-stat-label\">Duration<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">AI-Led<\/span><span class=\"scaler-stat-label\">Curriculum<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">Career Support<\/span><span class=\"scaler-stat-label\">Support<\/span><\/div>\n <\/div>\n <div class=\"scaler-logos\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/2\/2f\/Google_2015_logo.svg\" alt=\"Google\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/a\/a9\/Amazon_logo.svg\" alt=\"Amazon\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/42\/Paytm_logo.png\" alt=\"Paytm\" loading=\"lazy\">\n <span class=\"scaler-logo-more\">+1000 more<\/span>\n <\/div>\n <div class=\"scaler-card-btns\">\n <button onclick=\"window.open('https:\/\/www.scaler.com\/data-science-course\/', '_blank')\">\n Go to Program\n <svg width=\"14\" height=\"14\" viewBox=\"0 0 16 16\" fill=\"none\" aria-hidden=\"true\"><path d=\"M3 8H13M13 8L9 4M13 8L9 12\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"\/><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <!-- CARD 3 -->\n <div class=\"swiper-slide\">\n <div class=\"scaler-card\">\n <div class=\"scaler-badge\">NSDC Certified<\/div>\n <div class=\"scaler-card-top\">\n <div class=\"scaler-icon-wrap\">\n <svg width=\"26\" height=\"26\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><path d=\"M12 2L13.5 8.5L20 10L13.5 11.5L12 18L10.5 11.5L4 10L10.5 8.5L12 2Z\"\/><\/svg>\n <\/div>\n <div class=\"scaler-card-top-content\">\n <h3 class=\"scaler-title\">Advanced AIML with Specialisation in Agentic AI<\/h3>\n <p class=\"scaler-desc\">Deep dive into AIML with focus on Agentic systems<\/p>\n <\/div>\n <\/div>\n <div class=\"scaler-stats\">\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">12 Months<\/span><span class=\"scaler-stat-label\">Duration<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">AI-Led<\/span><span class=\"scaler-stat-label\">Curriculum<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">Career Support<\/span><span class=\"scaler-stat-label\">Support<\/span><\/div>\n <\/div>\n <div class=\"scaler-logos\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/2\/2f\/Google_2015_logo.svg\" alt=\"Google\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/a\/a9\/Amazon_logo.svg\" alt=\"Amazon\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/42\/Paytm_logo.png\" alt=\"Paytm\" loading=\"lazy\">\n <span class=\"scaler-logo-more\">+1000 more<\/span>\n <\/div>\n <div class=\"scaler-card-btns\">\n <button onclick=\"window.open('https:\/\/www.scaler.com\/ai-machine-learning-course\/', '_blank')\">\n Go to Program\n <svg width=\"14\" height=\"14\" viewBox=\"0 0 16 16\" fill=\"none\" aria-hidden=\"true\"><path d=\"M3 8H13M13 8L9 4M13 8L9 12\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"\/><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <!-- CARD 4 -->\n <div class=\"swiper-slide\">\n <div class=\"scaler-card\">\n <div class=\"scaler-badge\">NSDC Certified<\/div>\n <div class=\"scaler-card-top\">\n <div class=\"scaler-icon-wrap\">\n <svg width=\"26\" height=\"26\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <path d=\"M17.5 19H9a7 7 0 1 1 6.71-9h1.79a4.5 4.5 0 1 1 0 9Z\" \/>\n <path d=\"M9.2 15.2a3 3 0 1 0 4.24-4.24\" \/>\n <\/svg>\n <\/div>\n <div class=\"scaler-card-top-content\">\n <h3 class=\"scaler-title\">DevOps, Cloud & AI Platform Engineering<\/h3>\n <p class=\"scaler-desc\">Build and manage AI-powered cloud infrastructure<\/p>\n <\/div>\n <\/div>\n <div class=\"scaler-stats\">\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">12 Months<\/span><span class=\"scaler-stat-label\">Duration<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">AI-Led<\/span><span class=\"scaler-stat-label\">Curriculum<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">Career Support<\/span><span class=\"scaler-stat-label\">Support<\/span><\/div>\n <\/div>\n <div class=\"scaler-logos\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/2\/2f\/Google_2015_logo.svg\" alt=\"Google\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/a\/a9\/Amazon_logo.svg\" alt=\"Amazon\" loading=\"lazy\">\n <img decoding=\"async\" class=\"scaler-company-logo\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/4\/42\/Paytm_logo.png\" alt=\"Paytm\" loading=\"lazy\">\n <span class=\"scaler-logo-more\">+1000 more<\/span>\n <\/div>\n <div class=\"scaler-card-btns\">\n <button onclick=\"window.open('https:\/\/www.scaler.com\/devops-course\/', '_blank')\">\n Go to Program\n <svg width=\"14\" height=\"14\" viewBox=\"0 0 16 16\" fill=\"none\" aria-hidden=\"true\"><path d=\"M3 8H13M13 8L9 4M13 8L9 12\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"\/><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <!-- CARD 5 -->\n <div class=\"swiper-slide\">\n <div class=\"scaler-card\">\n <div class=\"scaler-badge\">NSDC Certified<\/div>\n <div class=\"scaler-card-top\">\n <div class=\"scaler-icon-wrap\">\n <svg width=\"26\" height=\"26\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <line x1=\"3\" y1=\"22\" x2=\"21\" y2=\"22\" \/>\n <line x1=\"6\" y1=\"18\" x2=\"6\" y2=\"11\" \/>\n <line x1=\"10\" y1=\"18\" x2=\"10\" y2=\"11\" \/>\n <line x1=\"14\" y1=\"18\" x2=\"14\" y2=\"11\" \/>\n <line x1=\"18\" y1=\"18\" x2=\"18\" y2=\"11\" \/>\n <polygon points=\"12 2 20 7 4 7\" \/>\n <\/svg>\n <\/div>\n <div class=\"scaler-card-top-content\">\n <h3 class=\"scaler-title\">AI Engineering Advanced Certification by IIT-Roorkee<\/h3>\n <p class=\"scaler-desc\">Premier AI engineering certification from IIT-Roorkee<\/p>\n <\/div>\n <\/div>\n <div class=\"scaler-stats\">\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">3 Months<\/span><span class=\"scaler-stat-label\">Duration<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">AI-Led<\/span><span class=\"scaler-stat-label\">Curriculum<\/span><\/div>\n <div class=\"scaler-stat\"><span class=\"scaler-stat-value\">Career Support<\/span><span class=\"scaler-stat-label\">Support<\/span><\/div>\n <\/div>\n <div class=\"scaler-logos\">\n <img decoding=\"async\" class=\"scaler-logo-strip\" src=\"https:\/\/scaler-blog-prod-wp-content.s3.ap-south-1.amazonaws.com\/wp-content\/uploads\/2026\/07\/08135139\/iitr-icons-strip.png\" alt=\"Program highlights\" loading=\"lazy\">\n <\/div>\n <div class=\"scaler-card-btns\">\n <button onclick=\"window.open('https:\/\/www.scaler.com\/iit-roorkee-advanced-ai-engineering-course\/', '_blank')\">\n Go to Program\n <svg width=\"14\" height=\"14\" viewBox=\"0 0 16 16\" fill=\"none\" aria-hidden=\"true\"><path d=\"M3 8H13M13 8L9 4M13 8L9 12\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\"\/><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <!-- Continue for remaining cards (6\u20138) using same pattern -->\n\n <\/div>\n <div class=\"swiper-button-next\"><\/div>\n <div class=\"swiper-button-prev\"><\/div>\n <div class=\"swiper-pagination\"><\/div>\n <\/div>\n<\/div>\n\n<!-- Swiper.js -->\n<script src=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.js\"><\/script>\n<script>\ndocument.addEventListener(\"DOMContentLoaded\", function () {\n const pathParts = window.location.pathname.split(\"\/\").filter(Boolean);\n const slug = pathParts[pathParts.length - 1] || \"\";\n const readableSlug = encodeURIComponent(slug);\n const buttons = document.querySelectorAll(\".scaler-card-btns button\");\n\n buttons.forEach((btn) => {\n const baseURL = btn.getAttribute(\"onclick\").match(\/'(.*?)'\/)[1];\n if (!baseURL.includes(\"utm_source=\")) {\n const separator = baseURL.includes(\"?\") ? \"&\" : \"?\";\n const updatedURL = `${baseURL}${separator}utm_source=blog&utm_medium=program_carousel&utm_content=${readableSlug}`;\n btn.setAttribute(\"onclick\", `window.open('${updatedURL}', '_blank')`);\n }\n });\n\n new Swiper(\".scaler-swiper\", {\n slidesPerView: 1,\n spaceBetween: 28,\n grabCursor: true,\n navigation: { nextEl: \".swiper-button-next\", prevEl: \".swiper-button-prev\" },\n pagination: { el: \".swiper-pagination\", dynamicBullets: true, clickable: true },\n breakpoints: { 1000: { slidesPerView: 2 } },\n keyboard: { enabled: true },\n });\n});\n<\/script>\n\n\n\n<p class=\"wp-block-paragraph\">Always remember that one deep project can also be more valuable than five shallow ones. Three cloned repositories don’t show three different skills. One project that you can explain from ingestion to serving, including its failure points and design choices, gives an interviewer much more to work with.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Treat your README as part of the project. It should cover the architecture, setup, data flow, and key trade-offs. An interviewer may open the repository during the hiring process, so make those first few minutes count.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Also read:<\/strong> <a href=\"https:\/\/www.scaler.com\/blog\/india-ai-workforce-report-2026\/\"><strong>India AI Workforce Report 2026<\/strong><\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"before-you-start-%e2%80%93-prerequisites-and-how-to-pick-your-first-project\"><\/span><strong>Before You Start – Prerequisites and How to Pick Your First Project<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before you start building your first data engineering project, make sure you have the basics you\u2019ll need to work through the project on your own. You should know SQL, including joins, window functions, CTEs, and aggregation, along with Python scripts, functions, error handling, and virtual environments. You should also know your way around a terminal and Git, and have basic Docker knowledge; docker compose up is enough to get started.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Spark, Kubernetes, and cloud certifications can come later. For your first project, focus on getting the pipeline, database, and data flow working properly. If you\u2019re still working out the full learning path, follow the <a href=\"https:\/\/www.scaler.com\/blog\/data-engineer-roadmap\/\">data engineer roadmap<\/a>. If you\u2019ve yet to work with Python, then start with this <a href=\"https:\/\/www.scaler.com\/topics\/course\/python-for-data-science\/\">free Python for data science course<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When choosing a data engineering project, consider four things:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Consider projects according to the role you want. Since analytics engineering, platform engineering, and streaming demand different project experience, start by considering projects which are similar to the job position you are looking for.\u00a0<\/li>\n\n\n\n<li>Use projects to fill skill gaps. If your current work already demonstrates SQL and dashboards, build something that adds new skills instead of repeating them.\u00a0<\/li>\n\n\n\n<li>Remember your hardware. An 8 GB laptop can limit what you run locally, so start with lighter tools and smaller datasets before taking on Spark or a larger local stack.\u00a0<\/li>\n\n\n\n<li>Plan around your actual time. The projects that involve ingestion, modelling, testing, and dealing with failures take weeks, not a few days.\u00a0<\/li>\n<\/ol>\n\n\n<meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n\n <!-- Source Sans Pro (topics font) -->\n <link href=\"https:\/\/fonts.googleapis.com\/css2?family=Source+Sans+3:wght@400;600;700&display=swap\" rel=\"stylesheet\">\n <link rel=\"stylesheet\" href=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.css\">\n <script src=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.js\"><\/script>\n\n <style>\n :root {\n \/* topics design tokens (variables.scss) *\/\n --scaler-primary: #005bb3; \/* primary-color-80 *\/\n --scaler-primary-30: #8acaff; \/* primary-color-30 *\/\n --scaler-brand-blue: #255aad;\n --scaler-live: #ee001d; \/* accent-v5-60 *\/\n --scaler-live-bg: #ffe6e9; \/* accent-v5-10 *\/\n --scaler-card-bg: #ffffff; \/* neutral-scale-0 *\/\n --scaler-surface: #f4f6f9; \/* neutral-scale-2 *\/\n --scaler-text-primary: #1a2c47; \/* neutral-scale-12 *\/\n --scaler-text-secondary: #61738e; \/* neutral-scale-9 *\/\n --scaler-border: #d6dee5; \/* neutral-scale-4 *\/\n --scaler-radius: 16px;\n --scaler-radius-sm: 8px;\n --scaler-shadow: 0 8px 16px 0 rgba(50, 71, 102, 0.16);\n }\n\n .scaler-events-carousel * { box-sizing: border-box; }\n\n .scaler-events-carousel {\n font-family: \"Source Sans 3\", \"Source Sans Pro\", \"Roboto\", Arial, sans-serif;\n padding: 24px;\n position: relative;\n width: 100%;\n }\n .scaler-events-carousel__inner {\n max-width: 1000px;\n margin: 0 auto;\n position: relative;\n width: 100%;\n }\n\n \/* Header \u2014 calendar icon + \"Live Masterclasses\" *\/\n .scaler-events-header {\n display: flex;\n align-items: center;\n gap: 8px;\n margin-bottom: 8px;\n padding: 8px 0;\n }\n .scaler-events-header__icon { display: inline-flex; color: var(--scaler-text-primary); }\n .scaler-events-header__icon svg { width: 24px; height: 24px; }\n .scaler-events-header__title {\n font-size: 18px;\n font-weight: 700;\n color: var(--scaler-text-primary);\n line-height: 1.3;\n margin: 0;\n }\n\n \/* Swiper wrap (holds side arrows) *\/\n .scaler-events-swiper-wrap { position: relative; }\n .scaler-events-carousel .swiper { width: 100%; padding: 4px; }\n .scaler-event-swiper .swiper-slide {\n width: 296px; \/* topics card_container width *\/\n height: auto;\n display: flex;\n }\n\n \/* Event Card \u2014 matches topics EventCard *\/\n .scaler-event-card {\n background: var(--scaler-card-bg);\n border: 1px solid var(--scaler-border);\n border-radius: var(--scaler-radius);\n padding: 16px;\n display: flex;\n flex-direction: column;\n height: 100%;\n width: 100%;\n cursor: pointer;\n }\n\n \/* Header row: headshot + title\/name *\/\n .scaler-event-card__header { display: flex; align-items: flex-start; }\n .scaler-event-card__image-wrapper {\n position: relative;\n width: 88px;\n height: 88px;\n min-width: 88px;\n border-radius: var(--scaler-radius-sm);\n overflow: hidden;\n background: var(--scaler-primary-30);\n }\n .scaler-event-card__image {\n position: absolute;\n top: 0; left: 0;\n width: 100%; height: 100%;\n object-fit: cover;\n }\n \/* attendee-count footer (default) *\/\n .scaler-event-card__img-footer {\n position: absolute;\n left: 0; right: 0; bottom: 0;\n display: flex;\n align-items: center;\n justify-content: center;\n gap: 4px;\n padding: 1px 8px;\n background: var(--scaler-surface);\n color: var(--scaler-text-primary);\n font-size: 13px;\n line-height: 22px;\n }\n .scaler-event-card__img-footer svg { width: 14px; height: 14px; }\n \/* live chip (shown when status === current) *\/\n .scaler-event-card__live-badge {\n position: absolute;\n left: 0; right: 0; bottom: 0;\n display: flex;\n align-items: center;\n justify-content: center;\n gap: 5px;\n padding: 1px 8px;\n background: var(--scaler-live-bg);\n color: var(--scaler-live);\n font-size: 12px;\n line-height: 22px;\n text-transform: uppercase;\n letter-spacing: 0.18em;\n }\n .scaler-event-card__live-badge::before {\n content: \"\";\n width: 6px; height: 6px;\n background: var(--scaler-live);\n border-radius: 50%;\n }\n .scaler-event-card__text { margin-left: 16px; min-width: 0; }\n .scaler-event-card__title {\n font-size: 20px;\n font-weight: 600;\n color: var(--scaler-text-primary);\n margin: 0;\n line-height: 1.4;\n display: -webkit-box;\n -webkit-line-clamp: 2;\n -webkit-box-orient: vertical;\n overflow: hidden;\n }\n .scaler-event-card__speaker {\n margin-top: 8px;\n font-size: 14px;\n line-height: 1.4;\n color: var(--scaler-text-primary);\n }\n\n \/* Takeaways *\/\n .scaler-event-card__takeaways {\n margin-top: 12px;\n min-height: 50px;\n max-height: 50px;\n overflow: hidden;\n }\n .scaler-event-card__takeaway-item {\n display: flex;\n align-items: flex-start;\n gap: 6px;\n }\n .scaler-event-card__dot {\n width: 4px; height: 4px;\n min-width: 4px;\n border-radius: 50%;\n background: var(--scaler-text-secondary);\n margin-top: 11px;\n }\n .scaler-event-card__takeaway-text {\n flex: 1;\n min-width: 0;\n font-size: 14px;\n line-height: 24px;\n color: var(--scaler-text-primary);\n white-space: nowrap;\n overflow: hidden;\n text-overflow: ellipsis;\n }\n .scaler-event-card__knowmore,\n .scaler-event-card__knowmore:link,\n .scaler-event-card__knowmore:visited {\n flex-shrink: 0;\n font-size: 14px;\n line-height: 24px;\n color: var(--scaler-primary) !important;\n text-decoration: none;\n white-space: nowrap;\n }\n .scaler-event-card__knowmore:hover { text-decoration: underline; }\n\n \/* Date + duration *\/\n .scaler-event-card__meta {\n display: flex;\n align-items: center;\n justify-content: space-between;\n margin-top: 20px;\n font-size: 13px;\n line-height: 24px;\n color: var(--scaler-text-primary);\n }\n .scaler-event-card__duration {\n display: flex;\n align-items: center;\n gap: 4px;\n }\n .scaler-event-card__duration svg { width: 15px; height: 15px; }\n\n .scaler-event-card__spacer { flex-grow: 1; min-height: 4px; }\n\n \/* CTA \u2014 outlined \"Register with 1-Click\" *\/\n .scaler-event-card__cta,\n .scaler-event-card__cta:link,\n .scaler-event-card__cta:visited {\n display: flex;\n align-items: center;\n justify-content: center;\n width: 100%;\n margin-top: 16px;\n padding: 8px;\n background: transparent;\n color: var(--scaler-primary) !important;\n font-family: inherit;\n font-size: 14px;\n font-weight: 600;\n line-height: 24px;\n text-decoration: none;\n border: 1px solid var(--scaler-primary);\n border-radius: var(--scaler-radius-sm);\n cursor: pointer;\n transition: background 0.2s ease, color 0.2s ease;\n }\n .scaler-event-card__cta:hover,\n .scaler-event-card__cta:focus {\n background: var(--scaler-primary);\n color: #fff !important;\n }\n\n \/* Side navigation arrows *\/\n .scaler-events-nav__btn {\n position: absolute !important;\n top: 50% !important;\n transform: translateY(-50%) !important;\n z-index: 2;\n width: 46px !important;\n height: 46px !important;\n min-width: 46px;\n display: flex !important;\n align-items: center;\n justify-content: center;\n padding: 0 !important;\n background: var(--scaler-card-bg) !important;\n border: none !important;\n border-radius: var(--scaler-radius-sm) !important;\n box-shadow: var(--scaler-shadow) !important;\n color: var(--scaler-text-secondary) !important;\n cursor: pointer;\n transition: opacity 0.2s ease, visibility 0.2s ease;\n }\n .scaler-nav-prev { left: 8px !important; }\n .scaler-nav-next { right: 8px !important; }\n .scaler-events-nav__btn svg {\n width: 20px !important;\n height: 20px !important;\n fill: var(--scaler-text-secondary) !important;\n }\n .scaler-events-nav__btn.swiper-button-disabled {\n opacity: 0;\n visibility: hidden;\n pointer-events: none;\n }\n\n \/* View All Events *\/\n .scaler-events-viewall,\n .scaler-events-viewall:link,\n .scaler-events-viewall:visited {\n display: flex;\n align-items: center;\n justify-content: center;\n gap: 6px;\n margin-top: 12px;\n font-size: 14px;\n font-weight: 600;\n color: var(--scaler-brand-blue) !important;\n text-decoration: none;\n }\n .scaler-events-viewall:hover { text-decoration: underline; }\n .scaler-events-viewall svg { width: 16px; height: 16px; fill: var(--scaler-brand-blue); }\n\n \/* hide swiper defaults *\/\n .scaler-events-pagination { display: none; }\n .scaler-events-carousel .swiper-button-prev,\n .scaler-events-carousel .swiper-button-next { display: none; }\n\n \/* Skeleton & Empty States *\/\n .scaler-event-card--skeleton { pointer-events: none; }\n .scaler-event-card--skeleton .scaler-event-card__image-wrapper,\n .scaler-event-card--skeleton .scaler-event-card__title,\n .scaler-event-card--skeleton .scaler-event-card__speaker,\n .scaler-event-card--skeleton .scaler-event-card__takeaways,\n .scaler-event-card--skeleton .scaler-event-card__meta,\n .scaler-event-card--skeleton .scaler-event-card__cta {\n background: linear-gradient(90deg, var(--scaler-border) 25%, var(--scaler-surface) 50%, var(--scaler-border) 75%);\n background-size: 200% 100%;\n animation: shimmer 1.5s infinite;\n color: transparent !important;\n border-color: transparent !important;\n border-radius: 4px;\n }\n .scaler-event-card--skeleton .scaler-event-card__title { min-height: 44px; }\n .scaler-event-card--skeleton .scaler-event-card__speaker { min-height: 16px; }\n .scaler-event-card--skeleton .scaler-event-card__meta { min-height: 24px; }\n .scaler-event-card--skeleton .scaler-event-card__cta { min-height: 40px; }\n .scaler-event-card--skeleton .scaler-event-card__image,\n .scaler-event-card--skeleton .scaler-event-card__img-footer,\n .scaler-event-card--skeleton .scaler-event-card__live-badge { display: none; }\n\n @keyframes shimmer {\n 0% { background-position: 200% 0; }\n 100% { background-position: -200% 0; }\n }\n\n .scaler-events-empty {\n text-align: center;\n padding: 40px 20px;\n color: var(--scaler-text-secondary);\n font-size: 15px;\n }\n\n @media (max-width: 768px) {\n .scaler-events-carousel { padding: 16px; }\n .scaler-event-swiper .swiper-slide { width: calc(100vw - 52px); }\n .scaler-events-nav__btn { display: none; }\n }\n <\/style>\n\n\n\n <div class=\"scaler-events-carousel js-scaler-carousel\">\n\n <template class=\"js-event-card-template\">\n <div class=\"swiper-slide\">\n <article class=\"scaler-event-card\">\n <div class=\"scaler-event-card__header\">\n <div class=\"scaler-event-card__image-wrapper\">\n <img decoding=\"async\" src=\"\" alt=\"\" class=\"scaler-event-card__image\" loading=\"lazy\">\n <div class=\"scaler-event-card__img-footer\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M11.199 7.957c-3.086 0-5.588 2.502-5.588 5.588s2.502 5.588 5.588 5.588c3.086 0 5.588-2.502 5.588-5.588s-2.502-5.588-5.588-5.588zM3.611 13.545c0-4.191 3.397-7.588 7.588-7.588s7.588 3.397 7.588 7.588c0 4.191-3.397 7.588-7.588 7.588s-7.588-3.397-7.588-7.588z\"\/><path d=\"M21.090 6.369h0.003c1.989 0 3.897 0.79 5.303 2.197s2.197 3.314 2.197 5.303-0.79 3.897-2.197 5.303c-1.406 1.407-3.314 2.197-5.303 2.197-0.552 0-1-0.448-1-1s0.448-1 1-1c1.459 0 2.858-0.579 3.889-1.611s1.611-2.43 1.611-3.889c0-1.459-0.58-2.858-1.611-3.889s-2.429-1.611-3.888-1.611c-0.507 0.002-1.013 0.070-1.502 0.203-0.533 0.145-1.082-0.17-1.227-0.703s0.17-1.082 0.703-1.227c0.659-0.179 1.339-0.27 2.021-0.272z\"\/><path d=\"M10.906 21.364c-1.605 0-3.186 0.386-4.61 1.126s-2.649 1.811-3.572 3.124c-0.318 0.452-0.941 0.561-1.393 0.243s-0.561-0.941-0.243-1.393c1.107-1.576 2.577-2.862 4.286-3.749s3.606-1.351 5.532-1.351 3.823 0.463 5.532 1.351c1.709 0.888 3.179 2.174 4.286 3.749 0.318 0.452 0.209 1.076-0.243 1.393s-1.076 0.209-1.393-0.243c-0.923-1.313-2.148-2.385-3.572-3.124s-3.005-1.126-4.61-1.126z\"\/><\/svg>\n <span class=\"js-event-count\"><\/span>\n <\/div>\n <div class=\"scaler-event-card__live-badge\" style=\"display: none;\">Live<\/div>\n <\/div>\n <div class=\"scaler-event-card__text\">\n <h3 class=\"scaler-event-card__title\"><\/h3>\n <div class=\"scaler-event-card__speaker js-event-speaker\"><\/div>\n <\/div>\n <\/div>\n\n <div class=\"scaler-event-card__takeaways js-event-takeaways\"><\/div>\n\n <div class=\"scaler-event-card__spacer\"><\/div>\n\n <div class=\"scaler-event-card__meta\">\n <span class=\"js-event-date\"><\/span>\n <span class=\"scaler-event-card__duration\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M16 6.333c6.443 0 11.667 5.223 11.667 11.667s-5.223 11.667-11.667 11.667c-6.443 0-11.667-5.223-11.667-11.667v0c0.008-6.44 5.226-11.659 11.666-11.667h0.001zM16 28.333c5.707 0 10.333-4.626 10.333-10.333s-4.626-10.333-10.333-10.333c-5.707 0-10.333 4.626-10.333 10.333v0c0.007 5.704 4.629 10.326 10.333 10.333h0.001z\"\/><path d=\"M16 18.667c-0 0-0.001 0-0.001 0-0.184 0-0.35-0.074-0.471-0.195v0c-0.121-0.121-0.195-0.287-0.195-0.471s0.075-0.351 0.195-0.471l4.951-4.952c0.121-0.121 0.287-0.195 0.471-0.195s0.351 0.075 0.471 0.195v0c0.121 0.121 0.195 0.287 0.195 0.471s-0.075 0.351-0.195 0.471l-4.949 4.951c-0.121 0.121-0.288 0.196-0.472 0.196v0z\"\/><path d=\"M19 3.667h-6c-0.368 0-0.667-0.298-0.667-0.667v0c0-0.368 0.298-0.667 0.667-0.667v0h6c0.368 0 0.667 0.298 0.667 0.667v0c0 0.368-0.298 0.667-0.667 0.667v0z\"\/><\/svg>\n <span class=\"js-event-duration\"><\/span>\n <\/span>\n <\/div>\n\n <a href=\"#\" class=\"scaler-event-card__cta js-event-cta\">Register with 1-Click<\/a>\n <\/article>\n <\/div>\n <\/template>\n\n <div class=\"scaler-events-carousel__inner\">\n <header class=\"scaler-events-header\">\n <span class=\"scaler-events-header__icon\">\n <svg fill=\"currentColor\" viewBox=\"0 0 33 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M6.406 6.8c-0.11 0-0.2 0.090-0.2 0.2v20c0 0.11 0.090 0.2 0.2 0.2h20c0.11 0 0.2-0.090 0.2-0.2v-20c0-0.11-0.090-0.2-0.2-0.2h-20zM4.606 7c0-0.994 0.806-1.8 1.8-1.8h20c0.994 0 1.8 0.806 1.8 1.8v20c0 0.994-0.806 1.8-1.8 1.8h-20c-0.994 0-1.8-0.806-1.8-1.8v-20z\"\/><path d=\"M22.406 3.2c0.442 0 0.8 0.358 0.8 0.8v4c0 0.442-0.358 0.8-0.8 0.8s-0.8-0.358-0.8-0.8v-4c0-0.442 0.358-0.8 0.8-0.8z\"\/><path d=\"M10.406 3.2c0.442 0 0.8 0.358 0.8 0.8v4c0 0.442-0.358 0.8-0.8 0.8s-0.8-0.358-0.8-0.8v-4c0-0.442 0.358-0.8 0.8-0.8z\"\/><path d=\"M4.606 12c0-0.442 0.358-0.8 0.8-0.8h22c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-22c-0.442 0-0.8-0.358-0.8-0.8z\"\/><path d=\"M21.488 16.451c0.303 0.322 0.288 0.828-0.034 1.131l-5.838 5.5c-0.309 0.291-0.791 0.29-1.099-0.002l-3.162-3c-0.321-0.304-0.334-0.81-0.030-1.131s0.81-0.334 1.131-0.030l2.614 2.48 5.287-4.981c0.322-0.303 0.828-0.288 1.131 0.034z\"\/><\/svg>\n <\/span>\n <h2 class=\"scaler-events-header__title\"><span class=\"ez-toc-section\" id=\"live-masterclasses\"><\/span>Live Masterclasses<span class=\"ez-toc-section-end\"><\/span><\/h2>\n <\/header>\n\n <div class=\"scaler-events-swiper-wrap\">\n <div class=\"swiper scaler-event-swiper\">\n <div class=\"swiper-wrapper scaler-events-wrapper\"><\/div>\n <div class=\"swiper-pagination scaler-events-pagination\"><\/div>\n <\/div>\n\n <button class=\"scaler-events-nav__btn scaler-nav-prev\" aria-label=\"Previous slide\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(-90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"\/><\/svg>\n <\/button>\n <button class=\"scaler-events-nav__btn scaler-nav-next\" aria-label=\"Next slide\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"\/><\/svg>\n <\/button>\n <\/div>\n\n <a class=\"scaler-events-viewall\" href=\"https:\/\/www.scaler.com\/events\/\">\n <span>View All Events<\/span>\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"\/><\/svg>\n <\/a>\n <\/div>\n <\/div>\n\n <script>\n document.addEventListener(\"DOMContentLoaded\", () => {\n\n const monthNames = [\"January\",\"February\",\"March\",\"April\",\"May\",\"June\",\"July\",\"August\",\"September\",\"October\",\"November\",\"December\"];\n\n function formatTime(st) {\n const t = new Date(st);\n const date = t.getDate();\n const month = monthNames[t.getMonth()];\n const year = t.getFullYear();\n const h = t.getHours();\n const m = String(t.getMinutes()).padStart(2, \"0\");\n let time;\n if (h === 0) time = `12:${m} AM`;\n else if (h === 12) time = `12:${m} PM`;\n else if (h > 12) time = `${h - 12}:${m} PM`;\n else time = `${h}:${m} AM`;\n return `${date} ${month} ${year} | ${time}`;\n }\n\n const carouselInstances = document.querySelectorAll('.js-scaler-carousel');\n\n carouselInstances.forEach(container => {\n\n if(container.dataset.initialized === \"true\") return;\n container.dataset.initialized = \"true\";\n\n const swiperElement = container.querySelector(\".scaler-event-swiper\");\n const swiperWrapper = container.querySelector(\".scaler-events-wrapper\");\n const template = container.querySelector(\".js-event-card-template\");\n const nextBtn = container.querySelector(\".scaler-nav-next\");\n const prevBtn = container.querySelector(\".scaler-nav-prev\");\n const paginationEl = container.querySelector(\".scaler-events-pagination\");\n\n if (!swiperWrapper || !template) {\n console.error(\"Scaler Carousel: Missing required elements inside container\");\n return;\n }\n\n const swiper = new Swiper(swiperElement, {\n slidesPerView: \"auto\",\n spaceBetween: 16,\n grabCursor: true,\n observer: true,\n observeParents: true,\n pagination: {\n el: paginationEl,\n clickable: true,\n dynamicBullets: true\n },\n navigation: {\n nextEl: nextBtn,\n prevEl: prevBtn\n },\n });\n\n function showSkeletons(count = 4) {\n swiperWrapper.innerHTML = \"\";\n for (let i = 0; i < count; i++) {\n const clone = template.content.cloneNode(true);\n const card = clone.querySelector(\".scaler-event-card\");\n card.classList.add(\"scaler-event-card--skeleton\");\n swiperWrapper.appendChild(clone);\n }\n swiper.update();\n }\n\n function renderEvents(events) {\n swiperWrapper.innerHTML = \"\";\n\n if (events.length === 0) {\n swiperWrapper.innerHTML = `<div class=\"scaler-events-empty\">No upcoming masterclasses found.<\/div>`;\n return;\n }\n\n const pathParts = location.pathname.split(\"\/\").filter(Boolean);\n const currentSlug = pathParts.length > 0 ? pathParts[pathParts.length - 1] : \"homepage\";\n\n events.forEach(event => {\n const attr = event.attributes;\n const cd = attr.custom_data || {};\n const clone = template.content.cloneNode(true);\n const joinUrl = `\/event\/${attr.slug}\/?utm_source=blog&utm_medium=master_class&utm_content=${currentSlug}`;\n\n \/\/ Instructor headshot\n const instructor = (cd.instructors && cd.instructors[0]) || {};\n const img = clone.querySelector(\".scaler-event-card__image\");\n img.src = instructor.media_url || attr.image_url ||\n \"https:\/\/images.unsplash.com\/photo-1531891437562-4301cf35b7e4?w=200&h=200&fit=crop\";\n img.alt = instructor.name || attr.instructor_name || attr.title;\n\n \/\/ Title + speaker\n clone.querySelector(\".scaler-event-card__title\").textContent = attr.title;\n clone.querySelector(\".js-event-speaker\").textContent = instructor.name || attr.instructor_name || \"\";\n\n \/\/ Live chip vs attendee count\n const isLive = attr.status === \"current\";\n const footer = clone.querySelector(\".scaler-event-card__img-footer\");\n const liveBadge = clone.querySelector(\".scaler-event-card__live-badge\");\n if (isLive) {\n footer.style.display = \"none\";\n liveBadge.style.display = \"flex\";\n liveBadge.textContent = \"Live\";\n } else if (attr.registration_count) {\n clone.querySelector(\".js-event-count\").textContent = attr.registration_count;\n } else {\n footer.style.display = \"none\";\n }\n\n \/\/ Takeaways (up to 2, Know More on the last)\n const takeaways = (cd.takeaways || []).slice(0, 2);\n const listEl = clone.querySelector(\".js-event-takeaways\");\n listEl.innerHTML = \"\";\n takeaways.forEach((t, idx) => {\n const item = document.createElement(\"div\");\n item.className = \"scaler-event-card__takeaway-item\";\n const dot = document.createElement(\"span\");\n dot.className = \"scaler-event-card__dot\";\n const txt = document.createElement(\"span\");\n txt.className = \"scaler-event-card__takeaway-text\";\n txt.textContent = t;\n item.appendChild(dot);\n item.appendChild(txt);\n if (idx === takeaways.length - 1) {\n const km = document.createElement(\"a\");\n km.className = \"scaler-event-card__knowmore\";\n km.textContent = \"Know More\";\n km.href = joinUrl;\n item.appendChild(km);\n }\n listEl.appendChild(item);\n });\n\n \/\/ Date + duration\n clone.querySelector(\".js-event-date\").textContent = formatTime(attr.start_time);\n let hrs = \"\";\n if (attr.duration) {\n hrs = (attr.duration \/ 60).toFixed(2);\n } else if (attr.end_time) {\n hrs = (Math.abs(new Date(attr.start_time) - new Date(attr.end_time)) \/ 3600000).toFixed(2);\n }\n clone.querySelector(\".js-event-duration\").textContent = hrs ? `${hrs} Hrs` : \"\";\n\n \/\/ CTA\n clone.querySelector(\".js-event-cta\").href = joinUrl;\n\n swiperWrapper.appendChild(clone);\n });\n\n swiper.update();\n swiper.slideTo(0);\n }\n\n async function fetchEvents() {\n try {\n showSkeletons();\n const res = await fetch(\n \"https:\/\/www.scaler.com\/api\/v4\/events?event_type[]=company&distributor=scaler&type=upcoming&serializer_mode=L2&limit=8&program[]=software_development&program[]=data_science&program[]=devops&program[]=ai_ml\"\n );\n const json = await res.json();\n const events = json.data || [];\n renderEvents(events);\n } catch (error) {\n console.error(\"Failed to load events:\", error);\n if(swiperWrapper) swiperWrapper.innerHTML = `<div class=\"scaler-events-empty\">Failed to load events.<\/div>`;\n }\n }\n\n fetchEvents();\n });\n });\n <\/script>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"the-12-data-engineering-projects-tiered\"><\/span><strong>The 12 Data Engineering Projects (Tiered)<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">These 12 data engineer projects are classified into four levels: Foundation, Core, Advanced, and Capstone. However, you don’t need to build all twelve. For most learners, building one project from each tier is enough to show how your skills develop.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to see how these projects fit into the broader learning path, use this <a href=\"https:\/\/www.scaler.com\/blog\/data-engineering-syllabus\/\">full data engineering syllabus<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Project<\/strong><\/td><td><strong>Skill area<\/strong><\/td><td><strong>Difficulty<\/strong><\/td><td><strong>Build time<\/strong><\/td><td><strong>Core stack<\/strong><\/td><\/tr><tr><td>1. Batch ETL Pipeline<\/td><td>ETL<\/td><td>Foundation<\/td><td>8-12 hours<\/td><td>Python, PostgreSQL<\/td><\/tr><tr><td>2. API Ingestion Pipeline<\/td><td>API ingestion<\/td><td>Foundation<\/td><td>10-15 hours<\/td><td>Python, Parquet, DuckDB<\/td><\/tr><tr><td>3. Dimensional Data Warehouse<\/td><td>Data modelling<\/td><td>Foundation-Core<\/td><td>15-20 hours<\/td><td>PostgreSQL, SQL<\/td><\/tr><tr><td>4. dbt Transformation Layer<\/td><td>ELT, testing<\/td><td>Core<\/td><td>12-18 hours<\/td><td>dbt, DuckDB\/PostgreSQL<\/td><\/tr><tr><td>5. Airflow Orchestration<\/td><td>Workflow orchestration<\/td><td>Core<\/td><td>15-20 hours<\/td><td>Airflow, Docker<\/td><\/tr><tr><td>6. Data Quality Framework<\/td><td>Data validation<\/td><td>Core<\/td><td>10-14 hours<\/td><td>Great Expectations\/Soda<\/td><\/tr><tr><td>7. Kafka + Spark Streaming<\/td><td>Stream processing<\/td><td>Advanced<\/td><td>20-25 hours<\/td><td>Kafka, PySpark<\/td><\/tr><tr><td>8. Change Data Capture<\/td><td>Real-time ingestion<\/td><td>Advanced<\/td><td>18-22 hours<\/td><td>Debezium, Kafka, PostgreSQL<\/td><\/tr><tr><td>9. Lakehouse Build<\/td><td>Lakehouse architecture<\/td><td>Advanced<\/td><td>20-25 hours<\/td><td>Spark, Delta Lake\/Iceberg<\/td><\/tr><tr><td>10. Data Platform with Terraform<\/td><td>Infrastructure as code<\/td><td>Advanced<\/td><td>12-18 hours<\/td><td>Terraform, GitHub Actions<\/td><\/tr><tr><td>11. Pipeline Observability<\/td><td>Monitoring, lineage<\/td><td>Advanced<\/td><td>15-20 hours<\/td><td>OpenLineage, Soda\/Elementary<\/td><\/tr><tr><td>12. End-to-End Data Platform<\/td><td>Systems integration<\/td><td>Capstone<\/td><td>40-60 hours<\/td><td>Full accumulated stack<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 1: Batch ETL Pipeline: Files to a Relational Warehouse<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">For ETL projects, start with a scheduled pipeline that picks up daily CSV or JSON files, validates and cleans the data, and loads it into PostgreSQL through staging and final tables.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts:<\/strong> ETL, staging tables, idempotency, incremental versus full loading, upsert, run logs, and dealing with malformed records.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Python (pandas or Polars), PostgreSQL, Docker, and cron or another simple scheduler. DuckDB is an alternative if you want a zero-setup option.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Foundation<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 8-12 hours.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can design a load so that rerunning the same file does not create duplicate records.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to strengthen your SQL before starting, you can follow this: <a href=\"https:\/\/www.scaler.com\/blog\/sql-roadmap-learning-path-career-roles-and-tools-to-master\/\">SQL Roadmap 2026: Learning Path, Career Roles and Tools to Master<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 2: API Ingestion Pipeline with Incremental Loads<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Create a pipeline that pulls data from a public REST API on a schedule and stores the results in their raw form before processing them. You could use an air-quality, transport, currency, markets, or GitHub API.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts:<\/strong> Pagination, rate limiting, exponential backoff, watermarking, immutability of raw data, secrets management, retries, and dead-letter handling<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Python with requests or httpx, MinIO or S3, Parquet with PyArrow, and DuckDB or PostgreSQL.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Foundation<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 10-15 hours<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You know how to deal with pagination, failed API calls, and incremental loads without restarting the entire pipeline.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Learn Python with: <strong><\/strong><a href=\"https:\/\/www.scaler.com\/blog\/python-developer-roadmap-2026-6-month-step-by-step-guide\/\">Python Developer Roadmap<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 3: Dimensional Data Warehouse: Star Schema with SCD Type 2<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">For a data warehouse project, take a messy operational dataset such as e-commerce transactions, hospital admissions, or ride bookings and convert it into a dimensional warehouse. Define the grain, create fact and dimension tables, and use SCD Type 2 to preserve historical changes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You will be using concepts like data grain, fact and dimension tables, surrogate and natural keys, SCD Types 1 and 2, conformed dimensions, star vs. snowflake schemas, and denormalisation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> PostgreSQL or Snowflake\/BigQuery<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Foundation\u2013Core<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Time:<\/strong> 15-20 hours<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By the end, you should be able to explain how you modelled the data and preserved historical changes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For a DBMS refresher, check out this <a href=\"https:\/\/www.scaler.com\/blog\/dbms-syllabus\/\">DBMS Syllabus<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 4: Transformation Layer with dbt: Staging, Marts, Tests and Docs<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Turn Project 3’s transformations into a dbt project with staging, intermediate, and mart models. Add tests and generate documentation for the project.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Concepts: Modular SQL, the ref() DAG, ELT, incremental models, built-in dbt tests, snapshots for SCD2, lineage, environments, and target profiles.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> dbt-core with DuckDB, PostgreSQL, or BigQuery.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Core<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 12-18 hours<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can split transformations into separate models, test them, and document how they depend on one another. Include the generated lineage graph in your README.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Learn more with <a href=\"https:\/\/www.scaler.com\/topics\/sql\/\">SQL Tutorial<\/a><\/p>\n\n\n<meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n\n <!-- Source Sans Pro (topics font) -->\n <link href=\"https:\/\/fonts.googleapis.com\/css2?family=Source+Sans+3:wght@400;600;700&display=swap\" rel=\"stylesheet\">\n <link rel=\"stylesheet\" href=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.css\">\n <script src=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.js\"><\/script>\n\n <style>\n :root {\n --cl-primary: #005bb3;\n --cl-brand-blue: #255aad;\n --cl-card-bg: #ffffff;\n --cl-surface: #f6f7f8;\n --cl-border: #d6dee5;\n --cl-text-primary: #1a2c47; \/* neutral-scale-12 *\/\n --cl-text-secondary: #61738e; \/* neutral-scale-9 *\/\n --cl-radius: 16px;\n --cl-radius-sm: 8px;\n --cl-shadow: 0 8px 16px 0 rgba(50, 71, 102, 0.16);\n }\n\n .scaler-courses * { box-sizing: border-box; }\n\n .scaler-courses {\n font-family: \"Source Sans 3\", \"Source Sans Pro\", \"Roboto\", Arial, sans-serif;\n padding: 24px;\n width: 100%;\n }\n .scaler-courses__inner {\n max-width: 1000px;\n margin: 0 auto;\n position: relative;\n width: 100%;\n }\n\n \/* Header *\/\n .scaler-courses__header {\n display: flex;\n align-items: center;\n gap: 8px;\n margin-bottom: 8px;\n padding: 8px 0;\n }\n .scaler-courses__header-icon { display: inline-flex; color: var(--cl-text-primary); }\n .scaler-courses__header-icon svg { width: 24px; height: 24px; }\n .scaler-courses__title {\n font-size: 18px;\n font-weight: 700;\n color: var(--cl-text-primary);\n line-height: 1.3;\n margin: 0;\n }\n\n \/* Swiper *\/\n .scaler-courses__swiper-wrap { position: relative; }\n .scaler-courses .swiper { width: 100%; padding: 4px; }\n .scaler-courses-swiper .swiper-slide {\n width: 296px;\n height: auto;\n display: flex;\n }\n\n \/* Card *\/\n .scaler-course-card {\n display: flex;\n flex-direction: column;\n width: 100%;\n height: 100%;\n min-height: 336px;\n background: var(--cl-card-bg);\n border: 1px solid var(--cl-border);\n border-radius: var(--cl-radius);\n overflow: hidden;\n text-decoration: none;\n cursor: pointer;\n transition: box-shadow 0.2s ease;\n }\n .scaler-course-card:hover { box-shadow: 0 2px 12px rgba(50, 71, 102, 0.12); }\n\n .scaler-course-card__image {\n width: 100%;\n height: 128px;\n min-height: 128px;\n object-fit: cover;\n background: var(--cl-surface);\n display: block;\n }\n\n .scaler-course-card__detail {\n display: flex;\n flex-direction: column;\n flex: 1 1 auto;\n padding: 16px 16px 8px;\n }\n .scaler-course-card__heading,\n .scaler-course-card__heading:link,\n .scaler-course-card__heading:visited {\n font-size: 20px;\n font-weight: 600;\n line-height: 1.4;\n color: var(--cl-text-primary) !important;\n margin: 0 0 8px;\n white-space: nowrap;\n overflow: hidden;\n text-overflow: ellipsis;\n }\n .scaler-course-card__instructor {\n font-size: 14px;\n line-height: 1.86;\n color: var(--cl-text-primary);\n white-space: nowrap;\n overflow: hidden;\n text-overflow: ellipsis;\n }\n .scaler-course-card__instructor b { font-weight: 700; }\n .scaler-course-card__desc {\n flex-grow: 1;\n margin-top: 8px;\n margin-bottom: 12px;\n font-size: 16px;\n line-height: 1.5;\n color: var(--cl-text-secondary);\n display: -webkit-box;\n -webkit-line-clamp: 2;\n -webkit-box-orient: vertical;\n overflow: hidden;\n }\n\n .scaler-course-card__footer {\n display: flex;\n align-items: center;\n justify-content: space-between;\n width: 100%;\n }\n .scaler-course-card__stat {\n display: flex;\n align-items: center;\n gap: 4px;\n font-size: 14px;\n color: var(--cl-text-primary);\n }\n .scaler-course-card__stat svg { width: 18px; height: 18px; flex-shrink: 0; }\n .scaler-course-card__stat img { width: 20px; height: 20px; flex-shrink: 0; }\n\n \/* Side arrows *\/\n .scaler-courses__nav {\n position: absolute !important;\n top: 50% !important;\n transform: translateY(-50%) !important;\n z-index: 2;\n width: 46px !important; height: 46px !important; min-width: 46px;\n display: flex !important; align-items: center; justify-content: center;\n padding: 0 !important;\n background: var(--cl-card-bg) !important;\n border: none !important;\n border-radius: var(--cl-radius-sm) !important;\n box-shadow: var(--cl-shadow) !important;\n color: var(--cl-text-secondary) !important;\n cursor: pointer;\n transition: opacity 0.2s ease, visibility 0.2s ease;\n }\n .scaler-courses__nav--prev { left: 8px !important; }\n .scaler-courses__nav--next { right: 8px !important; }\n .scaler-courses__nav svg { width: 20px !important; height: 20px !important; fill: var(--cl-text-secondary) !important; }\n .scaler-courses__nav.swiper-button-disabled { opacity: 0; visibility: hidden; pointer-events: none; }\n\n .scaler-courses .swiper-button-prev,\n .scaler-courses .swiper-button-next,\n .scaler-courses .swiper-pagination { display: none; }\n\n \/* Skeleton *\/\n .scaler-course-card--skeleton { pointer-events: none; }\n .scaler-course-card--skeleton .scaler-course-card__image,\n .scaler-course-card--skeleton .scaler-course-card__heading,\n .scaler-course-card--skeleton .scaler-course-card__instructor,\n .scaler-course-card--skeleton .scaler-course-card__desc,\n .scaler-course-card--skeleton .scaler-course-card__footer {\n background: linear-gradient(90deg, var(--cl-border) 25%, var(--cl-surface) 50%, var(--cl-border) 75%);\n background-size: 200% 100%;\n animation: cl-shimmer 1.5s infinite;\n color: transparent !important;\n border-radius: 4px;\n }\n .scaler-course-card--skeleton .scaler-course-card__heading { height: 24px; margin: 0 0 8px; }\n .scaler-course-card--skeleton .scaler-course-card__instructor { height: 16px; }\n .scaler-course-card--skeleton .scaler-course-card__footer { height: 20px; }\n @keyframes cl-shimmer { 0% { background-position: 200% 0; } 100% { background-position: -200% 0; } }\n\n .scaler-courses__empty { display: none; }\n\n @media (max-width: 768px) {\n .scaler-courses { padding: 16px; }\n .scaler-courses-swiper .swiper-slide { width: 260px; }\n .scaler-courses__nav { display: none !important; }\n }\n <\/style>\n\n\n <div class=\"scaler-courses js-scaler-courses\">\n\n <template class=\"js-course-card-template\">\n <div class=\"swiper-slide\">\n <a class=\"scaler-course-card js-course-link\" href=\"#\" target=\"_blank\" rel=\"noopener\">\n <img decoding=\"async\" class=\"scaler-course-card__image js-course-image\" src=\"\" alt=\"\" loading=\"lazy\">\n <div class=\"scaler-course-card__detail\">\n <div class=\"scaler-course-card__heading js-course-title\"><\/div>\n <div class=\"scaler-course-card__instructor js-course-instructor\"><\/div>\n <div class=\"scaler-course-card__desc js-course-desc\"><\/div>\n <div class=\"scaler-course-card__footer\">\n <span class=\"scaler-course-card__stat js-course-star\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M14.83 2.868c0.342-0.239 0.75-0.368 1.168-0.368s0.826 0.128 1.168 0.368c0.338 0.236 0.596 0.569 0.741 0.955l2.759 6.936c0.003 0.007 0.007 0.013 0.013 0.017s0.013 0.007 0.020 0.007l7.402 0.476c0.411 0.036 0.802 0.194 1.124 0.452s0.559 0.607 0.683 1 0.127 0.815 0.011 1.211c-0.116 0.396-0.347 0.748-0.664 1.013l-5.655 4.714c-0.013 0.011-0.022 0.025-0.028 0.040s-0.006 0.032-0.002 0.047l1.821 7.163c0.107 0.395 0.094 0.812-0.037 1.2-0.133 0.391-0.38 0.734-0.71 0.983s-0.727 0.393-1.14 0.413c-0.413 0.020-0.822-0.084-1.175-0.3l-0.015-0.009-6.307-4.005c-0.004-0.003-0.009-0.004-0.014-0.004s-0.009 0.001-0.013 0.004l-5.876 3.721c-0.389 0.235-0.839 0.347-1.292 0.323s-0.889-0.184-1.251-0.459c-0.361-0.275-0.632-0.652-0.777-1.082-0.143-0.426-0.157-0.885-0.038-1.319l1.684-6.629c0.004-0.016 0.003-0.033-0.002-0.049s-0.015-0.030-0.027-0.040l-5.653-4.714c-0.317-0.264-0.548-0.617-0.664-1.013s-0.112-0.817 0.011-1.211c0.124-0.394 0.361-0.742 0.683-1s0.713-0.416 1.124-0.452l0.023-0.002 7.38-0.476c0.007-0 0.014-0.003 0.020-0.007s0.010-0.010 0.013-0.017l0.008-0.020 2.746-6.913c0.145-0.386 0.403-0.719 0.741-0.956z\"\/><\/svg>\n <span class=\"js-course-star-val\"><\/span>\n <\/span>\n <span class=\"scaler-course-card__stat\">\n <img decoding=\"async\" src=\"https:\/\/d19zaobb0foqu6.cloudfront.net\/topics\/images\/user-check-dark.svg\" alt=\"Enrolled\" loading=\"lazy\">\n <span class=\"js-course-reg\"><\/span>\n <\/span>\n <\/div>\n <\/div>\n <\/a>\n <\/div>\n <\/template>\n\n <div class=\"scaler-courses__inner\">\n <div class=\"scaler-courses__header\">\n <span class=\"scaler-courses__header-icon\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M26.259 23.289h-20.518c-1.193 0-2.16-0.967-2.16-2.16v0-14.588c0-1.193 0.967-2.16 2.16-2.16v0h20.518c1.193 0 2.16 0.967 2.16 2.16v0 14.588c0 1.193-0.967 2.16-2.16 2.16v0zM5.741 5.501c-0.574 0-1.040 0.466-1.040 1.040v0 14.588c0 0.574 0.466 1.040 1.040 1.040v0h20.518c0.574 0 1.040-0.466 1.040-1.040v0-14.588c0-0.574-0.466-1.040-1.040-1.040v0z\"\/><path d=\"M19.764 27.619h-7.53c-0.309 0-0.56-0.251-0.56-0.56v0c0-0.309 0.251-0.56 0.56-0.56h7.53c0.309 0 0.56 0.251 0.56 0.56v0c0 0.309-0.251 0.56-0.56 0.56v0z\"\/><path d=\"M14.023 9.322c0.116 0 0.223 0.035 0.312 0.096l-0.002-0.001 5.93 3.953c0.151 0.102 0.249 0.272 0.249 0.466s-0.098 0.364-0.247 0.465l-0.002 0.001-5.93 3.953c-0.087 0.059-0.195 0.094-0.31 0.094-0.309 0-0.56-0.25-0.56-0.559v-7.906c0-0.309 0.251-0.56 0.56-0.56v0zM18.943 13.835l-4.36-2.906v5.813z\"\/><\/svg>\n <\/span>\n <h2 class=\"scaler-courses__title\"><span class=\"ez-toc-section\" id=\"free-courses-by-top-scaler-instructors\"><\/span>Free Courses by top Scaler instructors<span class=\"ez-toc-section-end\"><\/span><\/h2>\n <\/div>\n\n <div class=\"scaler-courses__swiper-wrap\">\n <div class=\"swiper scaler-courses-swiper\">\n <div class=\"swiper-wrapper js-courses-wrapper\"><\/div>\n <div class=\"swiper-pagination\"><\/div>\n <\/div>\n\n <button class=\"scaler-courses__nav scaler-courses__nav--prev js-courses-prev\" aria-label=\"Previous\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(-90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"\/><\/svg>\n <\/button>\n <button class=\"scaler-courses__nav scaler-courses__nav--next js-courses-next\" aria-label=\"Next\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"\/><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <script>\n document.addEventListener(\"DOMContentLoaded\", () => {\n const EXTRA_COUNT_ADDED = 1000;\n\n \/\/ Mirrors getInstructorStar(): first token of the headline.\n function getInstructorStar(headline) {\n return (headline || \"\").toString().split(\" \")[0] || \"\";\n }\n\n document.querySelectorAll(\".js-scaler-courses\").forEach(container => {\n if (container.dataset.initialized === \"true\") return;\n container.dataset.initialized = \"true\";\n\n const swiperElement = container.querySelector(\".scaler-courses-swiper\");\n const wrapper = container.querySelector(\".js-courses-wrapper\");\n const template = container.querySelector(\".js-course-card-template\");\n const nextBtn = container.querySelector(\".js-courses-next\");\n const prevBtn = container.querySelector(\".js-courses-prev\");\n\n if (!wrapper || !template) return;\n\n const swiper = new Swiper(swiperElement, {\n slidesPerView: \"auto\",\n spaceBetween: 16,\n grabCursor: true,\n observer: true,\n observeParents: true,\n navigation: { nextEl: nextBtn, prevEl: prevBtn },\n });\n\n const pathParts = location.pathname.split(\"\/\").filter(Boolean);\n const currentSlug = pathParts.length > 0 ? pathParts[pathParts.length - 1] : \"homepage\";\n\n function showSkeletons(count = 4) {\n wrapper.innerHTML = \"\";\n for (let i = 0; i < count; i++) {\n const clone = template.content.cloneNode(true);\n clone.querySelector(\".scaler-course-card\").classList.add(\"scaler-course-card--skeleton\");\n wrapper.appendChild(clone);\n }\n swiper.update();\n }\n\n function renderCourses(courses) {\n wrapper.innerHTML = \"\";\n if (!courses.length) { container.style.display = \"none\"; return; }\n\n courses.forEach(item => {\n const a = item.attributes || {};\n const clone = template.content.cloneNode(true);\n const instructor = (a.instructors_data && a.instructors_data[0]) || {};\n\n clone.querySelector(\".js-course-link\").href =\n `https:\/\/www.scaler.com\/topics\/course\/${a.slug}\/?utm_source=blog&utm_medium=free_courses&utm_content=${currentSlug}`;\n\n const img = clone.querySelector(\".js-course-image\");\n img.src = a.cover_image_thumbnail || a.featured_image || \"\";\n img.alt = a.title || \"\";\n\n clone.querySelector(\".js-course-title\").textContent = a.title || \"\";\n\n const instrEl = clone.querySelector(\".js-course-instructor\");\n if (instructor.name) {\n instrEl.innerHTML = \"A Course by <b><\/b>\";\n instrEl.querySelector(\"b\").textContent = instructor.name;\n } else {\n instrEl.style.display = \"none\";\n }\n\n clone.querySelector(\".js-course-desc\").textContent = a.description || \"\";\n\n \/\/ Star\n const star = getInstructorStar(instructor.star_headline);\n const starWrap = clone.querySelector(\".js-course-star\");\n if (star && star !== \"0\") clone.querySelector(\".js-course-star-val\").textContent = star;\n else starWrap.style.display = \"none\";\n\n \/\/ Enrolled count (+1000 like topics)\n clone.querySelector(\".js-course-reg\").textContent =\n ((a.user_registration_count || 0) + EXTRA_COUNT_ADDED).toLocaleString(\"en-US\");\n\n wrapper.appendChild(clone);\n });\n\n swiper.update();\n swiper.slideTo(0);\n }\n\n async function fetchCourses() {\n try {\n showSkeletons();\n \/\/ Public same-origin endpoint (topics course search). Returns 6 free courses.\n const res = await fetch(\"\/topics\/api\/v1\/search\/courses\", {\n method: \"POST\",\n headers: {\n \"Content-Type\": \"application\/json\",\n \"Accept\": \"application\/json\",\n \"X-Requested-With\": \"XMLHttpRequest\",\n },\n body: JSON.stringify({\n filters: {\n q: \"\",\n page: { limit: 6, current_page: 1 },\n sort: { key: \"user_registration_count\", order: \"desc\" },\n filter_type: null,\n tags: [],\n program: [],\n level: [],\n category_type: [],\n },\n }),\n });\n const json = await res.json();\n const courses = (json && json.data) || [];\n renderCourses(courses);\n } catch (error) {\n console.error(\"Failed to load courses:\", error);\n container.style.display = \"none\";\n }\n }\n\n fetchCourses();\n });\n });\n <\/script>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 5: Orchestration with Airflow: DAGs, Retries and Backfills<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Put Projects 1 and 2 under an orchestrator. Create a DAG with dependencies, retries, sensors for upstream files, and the ability to backfill three months of historical data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Concepts: DAG design, task granularity, idempotency, backfills, catchup, execution dates, retries, sensors, connections, variables, and XComs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Apache Airflow with Docker Compose. Dagster and Prefect are alternatives.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Core<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 15-20 hours.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can schedule dependent tasks, retry failures, and safely rerun historical data instead of relying on a simple cron job.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">New to Docker? You can start with this <a href=\"https:\/\/www.scaler.com\/blog\/docker-roadmap\/\">Docker Roadmap<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 6: Data Quality and Validation Framework<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Create a validation layer that runs with every data load. Check schemas, nulls, uniqueness, referential integrity, value ranges, distributions, and changes in row counts. Define what happens when a check fails.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts:<\/strong> Fail-fast vs. quarantine, circuit breakers, data contracts, freshness checks, and alert thresholds.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Great Expectations or Soda Core. You can also use dbt tests with Elementary or Pandera for DataFrame validation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Core<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 10-14 hours.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can catch problems such as unexpected nulls, duplicate records, or unusual row counts before the data moves to the next layer.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 7: Streaming Pipeline: Kafka to Spark Structured Streaming<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Develop one of your PySpark projects around a producer that emits events such as clickstream data, ride bookings, IoT readings, or payment events. Send them through Kafka and use Spark Structured Streaming to calculate windowed aggregations. This also gives you practical big data projects that cover real-time processing. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts:<\/strong> Topics, partitions, consumer groups, offsets, delivery guarantees, event vs. processing time, watermarks, windows, checkpointing, recovery, and backpressure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Kafka or Redpanda, PySpark Structured Streaming, Docker, and PostgreSQL, Delta, or another serving store.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Advanced-leaning Core<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 20-25 hours<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can process events as they arrive while accounting for late data, event time, offsets, and recovery.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can also use this <a href=\"https:\/\/www.scaler.com\/blog\/system-design-roadmap-2026-complete-guide-from-beginner-to-advanced\/\">system design roadmap<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 8: Change Data Capture: Replicating a Database in Near Real Time<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Capture row-level changes from a PostgreSQL or MySQL database and send them to a warehouse or lake. The pipeline should handle inserts, updates, and deletes without relying on full-table reloads.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You will learn about log-based CDC, initial snapshots, incremental streams, tombstones, orderings, upserts, and replication lag.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Debezium, Kafka Connect, PostgreSQL with logical replication, and Delta\/Iceberg or a warehouse<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Advanced <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Time:<\/strong> 18-22 hours<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The project should help you understand how to keep the target system updated when records are added, changed, or deleted in the source database.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 9: Lakehouse Build: Delta Lake or Apache Iceberg on Object Storage<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Build a medallion-architecture lakehouse with bronze, silver, and gold layers. Use an open table format over object storage and demonstrate time travel and schema evolution.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts<\/strong>: ACID transactions, metadata logs, time travel, MERGE, schema evolution, small-file problems, compaction, partitioning, and catalogs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Spark with Delta Lake or PyIceberg with Apache Iceberg, using MinIO locally or S3. Databricks Free Edition is another option.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Advanced<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 20-25 hours<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can use Delta Lake or Iceberg to update lake data, preserve previous versions, and handle schema changes without rebuilding the entire dataset.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you decide to move the project to the cloud, check this:  <a href=\"https:\/\/www.scaler.com\/blog\/cloud-computing-roadmap-step-by-step-guide\/\">Cloud Computing Roadmap<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 10: Infrastructure as Code for a Data Platform<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use Terraform to provision your data stack, including storage, warehouse resources, service accounts, and IAM roles. Manage secrets through environment variables or a dedicated secrets manager. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You’ll work with infrastructure as code, state, modules, environments, least-privilege IAM, secrets management, CI\/CD, budget alerts, and reproducibility.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Terraform with AWS or GCP free tier, or LocalStack for local development, plus GitHub Actions<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Advanced<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The project should help you learn how to manage infrastructure using code and test your changes before deploying them to production.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the DevOps side of the project: <a href=\"https:\/\/www.scaler.com\/blog\/devops-roadmap\/\">DevOps Roadmap<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 11 Pipeline Observability: Freshness, Volume Anomalies and Lineage<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Add an observability layer to an existing pipeline. Track run metadata and freshness, detect changes in row counts and null rates, capture lineage, and send alerts when something goes wrong.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts:<\/strong> Freshness SLAs, anomaly detection, lineage, metadata, alert severity, and pipeline cost monitoring.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> OpenLineage and Marquez for lineage, Elementary or Soda for anomaly monitoring, Prometheus\/Grafana for metrics, and Slack for alerts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Advanced<br><strong>Build time:<\/strong> 15\u201320 hours.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can tell when a pipeline has completed, but the data itself has changed unexpectedly, become stale, or drifted from its normal pattern.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can also explore the MLOps learning path: <a href=\"https:\/\/www.scaler.com\/blog\/mlops-roadmap\/\">MLOps Roadmap [2026]: A Complete MLOps Career Guide<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Project 12 Capstone: End-to-End Data Platform<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Create an end-to-end data engineering project that brings together batch and API ingestion, CDC, a bronze layer, Spark or dbt transformations, Airflow orchestration, data-quality checks, observability, infrastructure as code, CI\/CD, and a serving layer such as Metabase or Superset.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Concepts:<\/strong> Systems integration, latency, failure isolation, architecture decisions, documentation, and cost per run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Stack:<\/strong> Your accumulated stack, with one repository, Docker Compose, and a README containing the architecture.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Difficulty:<\/strong> Capstone<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Build time:<\/strong> 40-60 hours over 4-8 weeks<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What it proves:<\/strong> You can connect ingestion, transformation, orchestration, testing, monitoring, and serving into one working system and explain the trade-offs behind those choices in an interview.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can also explore these <a href=\"https:\/\/www.scaler.com\/blog\/20-best-data-analyst-projects\/\">data analyst project ideas<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"the-%e2%80%9cpipelines-employers-actually-trust%e2%80%9d-checklist\"><\/span><strong>The “Pipelines Employers Actually Trust” Checklist<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A project doesn’t need every production practice. Pick two or three projects and add the practices that matter: make them safe to rerun, catch bad data, handle failures, and make them easy for someone else to run. You don’t need six projects if you can explain two of them properly. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here are the practices worth adding:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Practice<\/strong><\/td><td><strong>Why it matters<\/strong><\/td><td><strong>How to add it to your project<\/strong><\/td><\/tr><tr><td>Idempotency<\/td><td>A failed job should be safe to rerun without creating duplicates or corrupting data.<\/td><td>Key each load on a natural or composite key; use MERGE or ON CONFLICT; make each run depend on its date partition.<\/td><\/tr><tr><td>Backfills<\/td><td>Pipelines need to handle historical data, not just the latest run.<\/td><td>Parameterise jobs by run date; avoid using now() inside the logic; test a 90-day backfill against the incremental results.<\/td><\/tr><tr><td>Schema evolution<\/td><td>Source columns can change without warning. Your pipeline needs a defined response.<\/td><td>Use Delta\/Iceberg or schema contracts; document how you handle new, renamed, and dropped columns.<\/td><\/tr><tr><td>Data quality tests<\/td><td>A job can finish successfully while producing incorrect data.<\/td><td>Add unique, not_null, relationships, and range checks; quarantine bad rows instead of silently dropping them.<\/td><\/tr><tr><td>Freshness and volume alerting<\/td><td>A source can stop sending data without the pipeline actually failing.<\/td><td>Track max(updated_at) and row counts; alert when freshness breaches its SLA or volume moves outside the expected range.<\/td><\/tr><tr><td>Cost awareness<\/td><td>Cloud pipelines can become expensive as data volume grows.<\/td><td>Track bytes scanned and runtime; partition or cluster where appropriate; estimate monthly cost at 10\u00d7 your current volume.<\/td><\/tr><tr><td>Documentation and lineage<\/td><td>Someone else should be able to understand how your pipeline and data are connected.<\/td><td>Add an architecture diagram, data dictionary, and trade-offs to the README; generate lineage with dbt docs or OpenLineage.<\/td><\/tr><tr><td>Error handling and run observability<\/td><td>Knowing that a job failed isn’t enough; you need to know what happened and what to do next.<\/td><td>Include retries with backoff, a dead letter queue if required, logging with a run ID, and an alert channel.<\/td><\/tr><tr><td>Secrets and access hygiene<\/td><td>Exposed credentials can make an otherwise good repository unsafe to use.<\/td><td>Use environment variables or a secrets manager, don’t put .env in Git, use least privilege accounts, and verify commit history before release.<\/td><\/tr><tr><td>Reproducibility<\/td><td>A reviewer should be able to run the project without rebuilding your environment.<\/td><td>Provide docker compose up, pin dependencies, include sample data, and test the setup on a clean machine.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Some of these problems are easy to miss until they happen. A source might change a currency field from paise to rupees without triggering a basic schema check. A source might stop sending data at 2 a.m., while the pipeline continues reporting successful runs. A backfill might also double-count records because the job appends historical data instead of replacing the affected partition.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Be ready to explain what happens when the pipeline fails or produces unexpected results, not just how you got it running.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For context on software engineering career levels, check out: <a href=\"https:\/\/www.scaler.com\/blog\/sde-1-vs-sde-2-vs-sde-3-understand-sde-level-differences\/\">SDE-1 vs SDE-2 vs SDE-3: Understand SDE Levels<\/a>.<\/p>\n\n\n<style data-wp-block-html=\"css\">\n<style>\n :root {\n --sp-brand: #255aad; \/* brand blue *\/\n --sp-brand-hover: #19468c;\n --sp-card-bg: #ffffff; \/* neutral-scale-0 *\/\n --sp-surface: #f4f6f9; \/* neutral-scale-2 *\/\n --sp-pill-bg: #f8fafc;\n --sp-border: #d6dee5; \/* neutral-scale-4 *\/\n --sp-text-primary: #1a2c47; \/* neutral-scale-12 *\/\n --sp-text-secondary: #61738e; \/* neutral-scale-9 *\/\n }\n\n .scaler-placement * { box-sizing: border-box; }\n\n .scaler-placement {\n font-family: \"Source Sans 3\", \"Source Sans Pro\", \"Roboto\", Arial, sans-serif;\n width: 100%;\n max-width: 100%;\n margin: 24px auto;\n display: flex;\n flex-direction: column;\n }\n\n \/* ---- Placement report (OnlineCallout) ---- *\/\n .scaler-placement__title {\n font-size: 24px;\n font-weight: 600;\n line-height: 1.3;\n color: var(--sp-text-primary);\n margin: 0 0 16px;\n }\n\n .scaler-placement__card {\n display: flex;\n flex-direction: column;\n width: 100%;\n background: var(--sp-card-bg);\n border: 1px solid var(--sp-border);\n border-radius: 20px;\n padding: 16px;\n box-shadow: 0 8px 32px rgba(37, 90, 173, 0.04);\n overflow: hidden;\n gap: 5px;\n }\n\n .scaler-placement__top {\n display: flex;\n flex-direction: row;\n align-items: center;\n width: 100%;\n gap: 18px;\n }\n\n .scaler-placement__stat-block {\n display: flex;\n flex-direction: column;\n justify-content: center;\n align-items: center;\n width: 30%;\n min-width: 200px;\n flex-shrink: 0;\n background: var(--sp-surface);\n border: 1px solid var(--sp-border);\n border-radius: 16px;\n padding: 16px 10px;\n gap: 8px;\n }\n .scaler-placement__stat-number {\n font-size: 48px;\n font-weight: 600;\n line-height: 1;\n color: var(--sp-brand);\n text-align: center;\n }\n .scaler-placement__stat-label {\n font-size: 13px;\n font-weight: 600;\n line-height: 14px;\n color: rgba(37, 90, 173, 0.85);\n text-transform: uppercase;\n letter-spacing: 0.8px;\n }\n\n .scaler-placement__info {\n display: flex;\n flex-direction: column;\n justify-content: center;\n flex: 1;\n gap: 10px;\n }\n .scaler-placement__proof-header {\n display: flex;\n flex-direction: row;\n align-items: center;\n gap: 8px;\n }\n .scaler-placement__proof-header svg { width: 18px; height: 18px; flex-shrink: 0; }\n .scaler-placement__proof-label {\n font-size: 12px;\n font-weight: 600;\n line-height: 16px;\n color: var(--sp-brand);\n text-transform: uppercase;\n letter-spacing: 1px;\n }\n .scaler-placement__body-text {\n font-size: 16px;\n font-weight: 400;\n line-height: 24px;\n color: var(--sp-text-primary);\n margin: 0;\n }\n\n .scaler-placement__bottom {\n display: flex;\n flex-direction: column;\n align-items: flex-start;\n margin-left: calc(30% + 18px);\n width: calc(70% - 18px);\n gap: 12px;\n }\n .scaler-placement__stats-row {\n display: flex;\n flex-direction: row;\n align-items: center;\n gap: 24px;\n }\n .scaler-placement__stat-item {\n display: flex;\n flex-direction: row;\n align-items: center;\n gap: 8px;\n }\n .scaler-placement__stat-item > svg {\n color: var(--sp-brand);\n width: 18px;\n height: 18px;\n flex-shrink: 0;\n }\n .scaler-placement__stat-text {\n display: flex;\n flex-direction: row;\n align-items: center;\n gap: 4px;\n }\n .scaler-placement__stat-value {\n font-size: 16px;\n font-weight: 600;\n line-height: 24px;\n color: var(--sp-text-primary);\n }\n .scaler-placement__stat-desc {\n font-size: 14px;\n font-weight: 400;\n line-height: 20px;\n color: var(--sp-text-secondary);\n }\n\n .scaler-placement__cta,\n .scaler-placement__cta:link,\n .scaler-placement__cta:visited {\n display: inline-flex;\n align-items: center;\n align-self: flex-start;\n font-size: 14px;\n font-weight: 600;\n line-height: 20px;\n color: var(--sp-brand) !important;\n gap: 6px;\n text-decoration: none;\n cursor: pointer;\n margin-top: 2px;\n transition: color 0.25s ease;\n }\n .scaler-placement__cta:hover,\n .scaler-placement__cta:focus { color: var(--sp-brand-hover) !important; }\n .scaler-placement__cta svg {\n width: 14px;\n height: 14px;\n fill: currentColor;\n transition: transform 0.25s ease;\n }\n .scaler-placement__cta:hover svg { transform: translateX(3px); }\n\n \/* ---- Hiring Partners ---- *\/\n .scaler-hiring {\n display: flex;\n align-items: center;\n justify-content: center;\n flex-wrap: wrap;\n gap: 8px 12px;\n margin: 24px 0 8px;\n padding-top: 16px;\n border-top: 1px solid var(--sp-border);\n }\n .scaler-hiring__label {\n font-size: 14px;\n color: var(--sp-text-secondary);\n white-space: nowrap;\n }\n .scaler-hiring__pills {\n display: flex;\n align-items: center;\n flex-wrap: wrap;\n gap: 8px;\n }\n .scaler-hiring__pill {\n display: inline-flex;\n align-items: center;\n gap: 6px;\n padding: 5px 12px 5px 5px;\n background: var(--sp-pill-bg);\n border-radius: 999px;\n }\n .scaler-hiring__badge {\n display: inline-flex;\n align-items: center;\n justify-content: center;\n flex-shrink: 0;\n width: 24px;\n height: 24px;\n border-radius: 50%;\n background: #ffffff;\n }\n .scaler-hiring__logo {\n width: 16px !important;\n height: 16px !important;\n object-fit: contain;\n margin: 0 !important;\n display: block;\n }\n .scaler-hiring__name {\n font-size: 14px;\n font-weight: 600;\n color: var(--sp-text-primary);\n }\n .scaler-hiring__more {\n font-size: 14px;\n color: var(--sp-text-secondary);\n }\n\n @media (max-width: 768px) {\n .scaler-placement__title { font-size: 18px; margin-bottom: 12px; }\n .scaler-placement__card { gap: 16px; border-radius: 16px; }\n .scaler-placement__top { gap: 12px; }\n .scaler-placement__stat-block { width: 35%; min-width: 110px; padding: 12px 8px; gap: 4px; }\n .scaler-placement__stat-number { font-size: 24px; }\n .scaler-placement__stat-label { font-size: 10px; }\n .scaler-placement__body-text { font-size: 14px; line-height: 20px; }\n .scaler-placement__bottom { margin-left: 0; width: 100%; gap: 16px; }\n .scaler-placement__stats-row { flex-wrap: wrap; gap: 10px 16px; }\n .scaler-placement__stat-value { font-size: 14px; line-height: 20px; }\n .scaler-placement__stat-desc { font-size: 12px; line-height: 16px; }\n }\n<\/style>\n\n<meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n\n <!-- Source Sans Pro (topics font) -->\n <link href=\"https:\/\/fonts.googleapis.com\/css2?family=Source+Sans+3:wght@400;600;700&display=swap\" rel=\"stylesheet\">\n\n<div class=\"scaler-placement\">\n <h2 class=\"scaler-placement__title\"><span class=\"ez-toc-section\" id=\"scaler-placement-report-and-statistics\"><\/span>Scaler Placement Report and Statistics<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n <div class=\"scaler-placement__card\">\n <div class=\"scaler-placement__top\">\n <div class=\"scaler-placement__stat-block\">\n <div class=\"scaler-placement__stat-number\">\u20b923L<\/div>\n <div class=\"scaler-placement__stat-label\">AVG CTC<\/div>\n <\/div>\n <div class=\"scaler-placement__info\">\n <div class=\"scaler-placement__proof-header\">\n <svg viewBox=\"0 0 21 21\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M8.64 0 0 8.683V21h12.257l8.64-8.683V0H8.64zm-.11 17.257H3.725v-4.828H8.53v4.828zm7.901-1.88-4.174 4.194V8.684H1.447l4.076-4.14h10.914l-.006 10.83v.004z\" fill=\"#255aad\"><\/path><\/svg>\n <span class=\"scaler-placement__proof-label\">SCALER PLACEMENT PROOF<\/span>\n <\/div>\n <p class=\"scaler-placement__body-text\">\n Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching \u20b923L.\n <\/p>\n <\/div>\n <\/div>\n\n <div class=\"scaler-placement__bottom\">\n <div class=\"scaler-placement__stats-row\">\n <div class=\"scaler-placement__stat-item\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M11.199 7.957c-3.086 0-5.588 2.502-5.588 5.588s2.502 5.588 5.588 5.588c3.086 0 5.588-2.502 5.588-5.588s-2.502-5.588-5.588-5.588zM3.611 13.545c0-4.191 3.397-7.588 7.588-7.588s7.588 3.397 7.588 7.588c0 4.191-3.397 7.588-7.588 7.588s-7.588-3.397-7.588-7.588z\"><\/path><path d=\"M21.090 6.369h0.003c1.989 0 3.897 0.79 5.303 2.197s2.197 3.314 2.197 5.303-0.79 3.897-2.197 5.303c-1.406 1.407-3.314 2.197-5.303 2.197-0.552 0-1-0.448-1-1s0.448-1 1-1c1.459 0 2.858-0.579 3.889-1.611s1.611-2.43 1.611-3.889c0-1.459-0.58-2.858-1.611-3.889s-2.429-1.611-3.888-1.611c-0.507 0.002-1.013 0.070-1.502 0.203-0.533 0.145-1.082-0.17-1.227-0.703s0.17-1.082 0.703-1.227c0.659-0.179 1.339-0.27 2.021-0.272z\"><\/path><path d=\"M10.906 21.364c-1.605 0-3.186 0.386-4.61 1.126s-2.649 1.811-3.572 3.124c-0.318 0.452-0.941 0.561-1.393 0.243s-0.561-0.941-0.243-1.393c1.107-1.576 2.577-2.862 4.286-3.749s3.606-1.351 5.532-1.351 3.823 0.463 5.532 1.351c1.709 0.888 3.179 2.174 4.286 3.749 0.318 0.452 0.209 1.076-0.243 1.393s-1.076 0.209-1.393-0.243c-0.923-1.313-2.148-2.385-3.572-3.124s-3.005-1.126-4.61-1.126z\"><\/path><path d=\"M25.704 22.493c-1.424-0.74-3.005-1.125-4.61-1.124-0.552 0-1-0.447-1.001-0.999s0.447-1 0.999-1.001c1.926-0.001 3.824 0.462 5.533 1.349s3.179 2.174 4.286 3.751c0.317 0.452 0.208 1.076-0.244 1.393s-1.076 0.208-1.393-0.244c-0.922-1.313-2.147-2.385-3.571-3.125z\"><\/path><\/svg>\n <div class=\"scaler-placement__stat-text\">\n <span class=\"scaler-placement__stat-value\">11,000+<\/span>\n <span class=\"scaler-placement__stat-desc\">placements<\/span>\n <\/div>\n <\/div>\n <div class=\"scaler-placement__stat-item\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M1.2 27.5c0-0.442 0.358-0.8 0.8-0.8h28c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-28c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><path d=\"M3.727 4.227c0.338-0.338 0.795-0.527 1.273-0.527h12c0.477 0 0.935 0.19 1.273 0.527s0.527 0.795 0.527 1.273v22c0 0.442-0.358 0.8-0.8 0.8s-0.8-0.358-0.8-0.8v-22c0-0.053-0.021-0.104-0.059-0.141s-0.088-0.059-0.141-0.059h-12c-0.053 0-0.104 0.021-0.141 0.059s-0.059 0.088-0.059 0.141v22c0 0.442-0.358 0.8-0.8 0.8s-0.8-0.358-0.8-0.8v-22c0-0.477 0.19-0.935 0.527-1.273z\"><\/path><path d=\"M17.2 12.5c0-0.442 0.358-0.8 0.8-0.8h9c0.477 0 0.935 0.19 1.273 0.527s0.527 0.795 0.527 1.273v14c0 0.442-0.358 0.8-0.8 0.8s-0.8-0.358-0.8-0.8v-14c0-0.053-0.021-0.104-0.059-0.141s-0.088-0.059-0.141-0.059h-9c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><path d=\"M7.2 9.5c0-0.442 0.358-0.8 0.8-0.8h4c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-4c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><path d=\"M9.2 17.5c0-0.442 0.358-0.8 0.8-0.8h4c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-4c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><path d=\"M7.2 22.5c0-0.442 0.358-0.8 0.8-0.8h4c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-4c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><path d=\"M21.2 22.5c0-0.442 0.358-0.8 0.8-0.8h2c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-2c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><path d=\"M21.2 17.5c0-0.442 0.358-0.8 0.8-0.8h2c0.442 0 0.8 0.358 0.8 0.8s-0.358 0.8-0.8 0.8h-2c-0.442 0-0.8-0.358-0.8-0.8z\"><\/path><\/svg>\n <div class=\"scaler-placement__stat-text\">\n <span class=\"scaler-placement__stat-value\">650+<\/span>\n <span class=\"scaler-placement__stat-desc\">companies<\/span>\n <\/div>\n <\/div>\n <div class=\"scaler-placement__stat-item\">\n <svg viewBox=\"0 0 25 24\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M5.60938 18.8906C4.74688 18.0281 5.31875 16.2188 4.87813 15.1594C4.4375 14.1 2.75 13.1719 2.75 12C2.75 10.8281 4.41875 9.9375 4.87813 8.84063C5.3375 7.74375 4.74688 5.97187 5.60938 5.10938C6.47187 4.24688 8.28125 4.81875 9.34063 4.37813C10.4 3.9375 11.3281 2.25 12.5 2.25C13.6719 2.25 14.5625 3.91875 15.6594 4.37813C16.7562 4.8375 18.5281 4.24688 19.3906 5.10938C20.2531 5.97187 19.6812 7.78125 20.1219 8.84063C20.5625 9.9 22.25 10.8281 22.25 12C22.25 13.1719 20.5813 14.0625 20.1219 15.1594C19.6625 16.2562 20.2531 18.0281 19.3906 18.8906C18.5281 19.7531 16.7188 19.1812 15.6594 19.6219C14.6 20.0625 13.6719 21.75 12.5 21.75C11.3281 21.75 10.4375 20.0813 9.34063 19.6219C8.24375 19.1625 6.47187 19.7531 5.60938 18.8906Z\" fill=\"#ffffff\" stroke=\"#255aad\" stroke-width=\"1.4\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><\/path><path d=\"M16.625 9.75L11.1219 15L8.375 12.375\" stroke=\"#255aad\" stroke-width=\"1.4\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><\/path><\/svg>\n <span class=\"scaler-placement__stat-desc\">Verified data<\/span>\n <\/div>\n <\/div>\n\n <a class=\"scaler-placement__cta\" href=\"https:\/\/d2beiqkhq929f0.cloudfront.net\/public_assets\/assets\/000\/195\/869\/original\/Scaler_Career_Report.pdf?1778499863\" target=\"_blank\" rel=\"noopener\">\n See full placement report\n <svg viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"><\/path><\/svg>\n <\/a>\n <\/div>\n <\/div>\n\n <div class=\"scaler-hiring\">\n <span class=\"scaler-hiring__label\">Hiring Partners:<\/span>\n <div class=\"scaler-hiring__pills\">\n <span class=\"scaler-hiring__pill\">\n <span class=\"scaler-hiring__badge\"><img decoding=\"async\" class=\"scaler-hiring__logo\" src=\"https:\/\/d19zaobb0foqu6.cloudfront.net\/topics\/images\/googleicon_.svg\" alt=\"Google\" loading=\"lazy\"><\/span>\n <span class=\"scaler-hiring__name\">Google<\/span>\n <\/span>\n <span class=\"scaler-hiring__pill\">\n <span class=\"scaler-hiring__badge\"><img decoding=\"async\" class=\"scaler-hiring__logo\" src=\"https:\/\/d19zaobb0foqu6.cloudfront.net\/topics\/images\/amazonicon_.svg\" alt=\"Amazon\" loading=\"lazy\"><\/span>\n <span class=\"scaler-hiring__name\">Amazon<\/span>\n <\/span>\n <span class=\"scaler-hiring__pill\">\n <span class=\"scaler-hiring__badge\"><img decoding=\"async\" class=\"scaler-hiring__logo\" src=\"https:\/\/d19zaobb0foqu6.cloudfront.net\/topics\/images\/microsoft.svg\" alt=\"Microsoft\" loading=\"lazy\"><\/span>\n <span class=\"scaler-hiring__name\">Microsoft<\/span>\n <\/span>\n <span class=\"scaler-hiring__pill\">\n <span class=\"scaler-hiring__badge\"><img decoding=\"async\" class=\"scaler-hiring__logo\" src=\"https:\/\/d19zaobb0foqu6.cloudfront.net\/topics\/images\/flipkarticon_.svg\" alt=\"Flipkart\" loading=\"lazy\"><\/span>\n <span class=\"scaler-hiring__name\">Flipkart<\/span>\n <\/span>\n <span class=\"scaler-hiring__pill\">\n <span class=\"scaler-hiring__badge\"><img decoding=\"async\" class=\"scaler-hiring__logo\" src=\"https:\/\/d19zaobb0foqu6.cloudfront.net\/topics\/images\/adobeicon_.svg\" alt=\"Adobe\" loading=\"lazy\"><\/span>\n <span class=\"scaler-hiring__name\">Adobe<\/span>\n <\/span>\n <span class=\"scaler-hiring__more\">1200+ more<\/span>\n <\/div>\n <\/div>\n <\/div>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"building-these-projects-for-free-or-nearly-free-in-india\"><\/span><strong>Building These Projects for Free (or Nearly Free) in India<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can build all 12 for \u20b90 on a laptop with 8-16 GB of RAM. Cloud services are optional; they can help you demonstrate experience with managed platforms, but they aren't necessary for most of the projects.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Option 1: Run everything locally<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Docker helps you run PostgreSQL, Redpanda or Kafka, MinIO, Airflow, Spark, dbt-core, DuckDB, and Metabase or Superset on your own machine. You can keep the setup in one docker-compose.yml file and start only the services you need.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With 8 GB of RAM, don't run the entire stack together. Run services in groups, and use DuckDB for smaller datasets when possible. It is much lighter than running Spark and PostgreSQL locally.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Option 2: Use cloud free tiers<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Use the cloud when it helps you demonstrate something you can't show locally. BigQuery works well for warehouse projects, while AWS and Azure can give you experience with cloud infrastructure. Snowflake and Confluent Cloud offer trial credits for warehouse and streaming projects, and Databricks Free Edition can be useful for the lakehouse project. Check the current limits before you start, since free-tier terms change. <\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Free option<\/strong><\/td><td><strong>Best for<\/strong><\/td><td><strong>Watch out for<\/strong><\/td><\/tr><tr><td>Local Docker<\/td><td>Most projects<\/td><td>RAM usage<\/td><\/tr><tr><td>BigQuery<\/td><td>Data warehouse<\/td><td>Query and storage limits<\/td><\/tr><tr><td>AWS \/ Azure<\/td><td>Cloud infrastructure<\/td><td>Free-tier limits<\/td><\/tr><tr><td>Snowflake<\/td><td>Warehouse<\/td><td>Trial expiry<\/td><\/tr><tr><td>Confluent Cloud<\/td><td>Streaming<\/td><td>Trial credits<\/td><\/tr><tr><td>Databricks Free Edition<\/td><td>Lakehouse<\/td><td>Resource limits<\/td><\/tr><tr><td>GitHub Actions<\/td><td>CI\/CD<\/td><td>Monthly usage limits<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Keep the cost at \u20b90<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Set a \u20b9100 billing alert before provisioning anything. Shut down resources when you're finished, and run terraform destroy for Terraform-managed infrastructure. Keep datasets to a few GB; none of these projects require terabytes of data. For cloud workloads, use on-demand or serverless options where possible, and consider spot or preemptible instances when they fit the workload. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For datasets, start with data.gov.in, RBI's Database on Indian Economy, MoSPI, and CPCB\/OpenAQ for India-focused projects. Kaggle, NYC TLC trip records, GitHub Archive, Wikimedia pageviews, and Open Food Facts are useful for broader use cases.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Use documented APIs and open datasets. Don't scrape sites that prohibit it, and never put personal data in a public repository.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"choosing-your-stack-what-to-pick-and-why\"><\/span><strong>Choosing Your Stack - What to Pick and Why<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You don't need the trendiest tool in every category. Pick a stack you understand, know why you chose it, and be ready to explain the trade-offs. One tool you know well is better than five you've barely configured. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Before choosing, check around ten job descriptions for the roles you're targeting. Let the tools that appear repeatedly influence your choices. If you're still deciding between data engineering and data science, see how <a href=\"https:\/\/www.scaler.com\/blog\/data-engineer-vs-data-scientist\/\">the data engineer role differs from data science<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Tool category<\/strong><\/td><td><strong>Common options<\/strong><\/td><td><strong>What to pick for a portfolio, and why<\/strong><\/td><\/tr><tr><td>Language<\/td><td>Python, Scala, Java, SQL<\/td><td>Python + SQL. These cover most portfolio projects; add Scala if you're targeting Spark-heavy roles.<\/td><\/tr><tr><td>Storage\/warehouse<\/td><td>PostgreSQL, BigQuery, Snowflake, Redshift, DuckDB<\/td><td>PostgreSQL locally, BigQuery in the cloud. PostgreSQL teaches database fundamentals; DuckDB is a lightweight option for local analytics.<\/td><\/tr><tr><td>Object storage<\/td><td>AWS S3, GCS, Azure Blob, MinIO<\/td><td>MinIO locally, S3 in the cloud. MinIO uses the S3 API, making it easy to move the project to AWS later.<\/td><\/tr><tr><td>Table format<\/td><td>Parquet, Delta Lake, Apache Iceberg, Hudi<\/td><td>Delta Lake or Iceberg. Either lets you demonstrate transactions, updates, and schema evolution on object storage.<\/td><\/tr><tr><td>Batch processing<\/td><td>pandas, Polars, Spark, DuckDB<\/td><td>Start with Polars or DuckDB; add Spark for one project. Most portfolio datasets don't need Spark, but knowing it is useful for Spark-heavy roles.<\/td><\/tr><tr><td>Streaming<\/td><td>Kafka, Redpanda, Spark Structured Streaming, Flink, Kinesis<\/td><td>Redpanda + Spark Structured Streaming. Redpanda is Kafka-compatible and easier to run locally. Choose Flink if streaming is your focus.<\/td><\/tr><tr><td>Orchestration<\/td><td>Airflow, Dagster, Prefect, Mage, cron<\/td><td>Airflow. It appears widely in job descriptions and gives you a chance to demonstrate DAGs, retries, and backfills.<\/td><\/tr><tr><td>Transformation<\/td><td>dbt, SQLMesh, raw SQL, Spark jobs<\/td><td>dbt-core. It gives you modular SQL, testing, documentation, and lineage in one project.<\/td><\/tr><tr><td>Data quality<\/td><td>Great Expectations, Soda Core, dbt tests, Pandera<\/td><td>dbt tests + Elementary if you're using dbt; Soda Core otherwise. Both are practical for a portfolio.<\/td><\/tr><tr><td>Ingestion \/ EL<\/td><td>Airbyte, Meltano, dlt, hand-written<\/td><td>Write one connector yourself, then use dlt. The first shows you understand API ingestion; the second shows you can use an existing tool when appropriate.<\/td><\/tr><tr><td>Infrastructure<\/td><td>Terraform, Pulumi, CloudFormation, ClickOps<\/td><td>Terraform. It works across cloud providers and is widely recognised in infrastructure roles.<\/td><\/tr><tr><td>CI\/CD<\/td><td>GitHub Actions, GitLab CI, Jenkins<\/td><td>GitHub Actions. It's easy to add to a public repository and makes your CI workflow visible to reviewers.<\/td><\/tr><tr><td>Lineage\/observability<\/td><td>OpenLineage + Marquez, DataHub, OpenMetadata, Elementary<\/td><td>OpenLineage + Marquez. It gives you a vendor-neutral way to demonstrate lineage without building a large platform.<\/td><\/tr><tr><td>Serving \/ BI<\/td><td>Metabase, Superset, Power BI, Looker Studio<\/td><td>Metabase. It's quick to set up and keeps the focus on the data pipeline rather than the BI layer.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">If a role uses a different tool, focus on explaining how your existing experience transfers rather than rebuilding your portfolio around it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"how-to-publish-and-present-your-data-engineering-projects\"><\/span><strong>How to Publish and Present Your Data Engineering Projects<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A finished project isn't enough on its own. Your repository should let a hiring team clone it, understand what you built, and see how you handled failures. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Give the repository a clear name instead of project-1. Start the README with a short problem statement and architecture diagram, then add setup instructions that work from a clean clone. Include sample data, an .env.example without real credentials, and a meaningful commit history instead of putting everything into one initial commit.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you're looking at data engineering projects with source code, use them to understand how others approach a problem rather than copying the repository. Build the same idea with your own dataset and make your own technical decisions. When browsing data engineering projects on GitHub repositories, look for projects with clear documentation, reproducible setup, and decisions you can understand and defend. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Write down your trade-offs<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Your README should explain why you made your main technical decisions. What did you choose? What alternatives did you reject? What would you change at 100\u00d7 the current scale? What is still broken or limited?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Don't hide known limitations. Saying \u201cthis works locally but would need distributed processing at larger volumes\u201d tells an interviewer that you understand where your design stops working.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And don't simply clone a tutorial. Interviewers have seen popular tutorial projects many times. You can start with the same basic idea, but use a different dataset, make your own design choices, and understand every change you make. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Be ready to defend the project<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Before an interview, practise answering questions such as:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What happens if this job fails at 3 a.m.?<\/li>\n\n\n\n<li>How would you rerun yesterday's data only?<\/li>\n\n\n\n<li>What happens if the source adds a new column?<\/li>\n\n\n\n<li>How would you know the data was wrong before a stakeholder reported it?<\/li>\n\n\n\n<li>What does this pipeline cost to run?<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If you can explain how your project handles these cases, you're defending an actual system rather than just listing tools.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Put outcomes on your resume<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Describe what the project did, not just which tools you used.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Built a CDC pipeline replicating 12 tables into a lakehouse with sub-5-minute lag and automated freshness alerts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Is far better than:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Worked with Debezium, Kafka, and Spark.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Once your portfolio is ready, the next step is landing your first tech role. You can read more about it here:<\/strong> <a href=\"https:\/\/www.scaler.com\/blog\/how-to-get-a-job-in-it\/\"><strong>How to Get a Job in IT<\/strong><\/a><strong>.<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"do-these-projects-actually-lead-to-a-data-engineering-job\"><\/span><strong>Do These Projects Actually Lead to a Data Engineering Job?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Yes, but not on their own. Projects help you get shortlisted and give you something concrete to discuss in an interview. They can be especially useful when moving from an analyst or support role into data engineering, where a strong project can demonstrate skills you haven't used professionally. For a fresher, projects help, but they won't replace other signals employers look for.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Contributing to an open-source data tool, writing a detailed project breakdown, and being able to explain your trade-offs can matter more than adding another five repositories.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Data engineering roles in India also vary across product companies, GCCs, and services firms, with different responsibilities and expectations. For broader context, see <a href=\"https:\/\/www.scaler.com\/blog\/highest-paying-it-jobs\/\">where data roles sit among India's highest-paying tech jobs<\/a> rather than relying on an unsourced salary figure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And don\u2019t worry, your portfolio doesn't need to prove you're already a senior engineer. It needs to give employers enough evidence that you can do the work and enough substance to have a meaningful technical conversation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to learn more about data engineering, you can also check out <a href=\"https:\/\/www.scaler.com\/data-science-course\/\"><strong>Scaler\u2019s Data Science and Analytics program<\/strong><\/a>.<\/p>\n\n\n<meta charset=\"UTF-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n\n <!-- Source Sans Pro (topics font) -->\n <link href=\"https:\/\/fonts.googleapis.com\/css2?family=Source+Sans+3:wght@400;600;700&display=swap\" rel=\"stylesheet\">\n <link rel=\"stylesheet\" href=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.css\">\n <script src=\"https:\/\/cdn.jsdelivr.net\/npm\/swiper@11\/swiper-bundle.min.js\"><\/script>\n\n <style>\n :root {\n --ss-primary: #005bb3; \/* primary-color-80 *\/\n --ss-card-bg: #ffffff;\n --ss-border: #e6e6e6;\n --ss-quote-bg: #f6f6f6;\n --ss-text-dark: #1c1c1c;\n --ss-text-mid: #6b6b6b;\n --ss-text-muted: #9a9a9a;\n --ss-radius: 12px;\n --ss-radius-sm: 8px;\n --ss-shadow: 0 8px 16px 0 rgba(50, 71, 102, 0.16);\n }\n\n .scaler-ss * { box-sizing: border-box; }\n\n .scaler-ss {\n font-family: \"Source Sans 3\", \"Source Sans Pro\", \"Roboto\", Arial, sans-serif;\n padding: 24px;\n width: 100%;\n }\n .scaler-ss__inner {\n max-width: 1000px;\n margin: 0 auto;\n position: relative;\n width: 100%;\n }\n\n \/* Heading *\/\n .scaler-ss__heading {\n font-size: 26px;\n line-height: 34px;\n font-weight: 700;\n margin: 0 0 16px;\n text-align: center;\n color: var(--ss-text-dark);\n }\n\n \/* Swiper *\/\n .scaler-ss__swiper-wrap { position: relative; }\n .scaler-ss .swiper { width: 100%; padding: 4px; }\n .scaler-ss-swiper .swiper-slide {\n width: 300px; \/* topics card width *\/\n height: auto;\n display: flex;\n }\n\n \/* Card *\/\n .scaler-ss-card {\n display: flex;\n flex-direction: column;\n width: 100%;\n border: 1px solid var(--ss-border);\n border-radius: var(--ss-radius);\n overflow: hidden;\n background: var(--ss-card-bg);\n }\n\n .scaler-ss-card__hike {\n padding: 8px 12px;\n background: var(--ss-primary);\n color: #ffffff;\n font-size: 14px;\n font-weight: 700;\n text-align: center;\n letter-spacing: 0.4px;\n }\n\n .scaler-ss-card__body {\n display: flex;\n flex: 1 1 auto;\n flex-direction: column;\n gap: 12px;\n padding: 16px;\n }\n\n \/* Person *\/\n .scaler-ss-card__person {\n display: flex;\n align-items: center;\n gap: 10px;\n }\n .scaler-ss-card__avatar {\n width: 44px;\n height: 44px;\n min-width: 44px;\n border-radius: 50%;\n object-fit: cover;\n background: var(--ss-quote-bg);\n }\n .scaler-ss-card__name {\n font-size: 16px;\n font-weight: 700;\n line-height: 20px;\n color: var(--ss-text-dark);\n }\n .scaler-ss-card__city {\n font-size: 13px;\n line-height: 18px;\n color: var(--ss-text-mid);\n }\n\n \/* Before \/ Now transition *\/\n .scaler-ss-card__transition {\n display: flex;\n align-items: center;\n justify-content: space-between;\n gap: 8px;\n }\n .scaler-ss-card__col {\n flex: 1;\n min-width: 0;\n }\n .scaler-ss-card__col-label {\n font-size: 11px;\n font-weight: 600;\n letter-spacing: 0.4px;\n color: var(--ss-text-muted);\n margin-bottom: 2px;\n }\n .scaler-ss-card__col-role {\n font-size: 13px;\n line-height: 18px;\n color: var(--ss-text-mid);\n }\n .scaler-ss-card__col-company {\n font-size: 14px;\n font-weight: 700;\n line-height: 20px;\n color: var(--ss-text-dark);\n }\n .scaler-ss-card__company-logo {\n margin-top: 2px;\n max-width: 64px;\n max-height: 20px;\n width: auto;\n height: auto;\n object-fit: contain;\n display: block;\n }\n .scaler-ss-card__arrow {\n flex-shrink: 0;\n width: 28px;\n height: 8px;\n color: var(--ss-text-mid);\n fill: currentColor;\n }\n\n \/* Quote *\/\n .scaler-ss-card__quote {\n font-size: 13px;\n font-style: italic;\n line-height: 19px;\n color: var(--ss-text-mid);\n background: var(--ss-quote-bg);\n border-radius: var(--ss-radius-sm);\n padding: 8px 10px;\n }\n\n \/* View Profile *\/\n .scaler-ss-card__cta,\n .scaler-ss-card__cta:link,\n .scaler-ss-card__cta:visited {\n display: block;\n margin-top: auto;\n text-align: center;\n font-size: 14px;\n font-weight: 600;\n color: var(--ss-primary) !important;\n text-decoration: none;\n border: 1px solid var(--ss-primary);\n border-radius: var(--ss-radius-sm);\n padding: 9px 12px;\n cursor: pointer;\n transition: background 0.2s ease, color 0.2s ease;\n }\n .scaler-ss-card__cta:hover,\n .scaler-ss-card__cta:focus {\n background: var(--ss-primary);\n color: #ffffff !important;\n }\n\n \/* Side navigation arrows *\/\n .scaler-ss__nav {\n position: absolute !important;\n top: 50% !important;\n transform: translateY(-50%) !important;\n z-index: 2;\n width: 46px !important;\n height: 46px !important;\n min-width: 46px;\n display: flex !important;\n align-items: center;\n justify-content: center;\n padding: 0 !important;\n background: var(--ss-card-bg) !important;\n border: none !important;\n border-radius: var(--ss-radius-sm) !important;\n box-shadow: var(--ss-shadow) !important;\n color: #61738e !important;\n cursor: pointer;\n transition: opacity 0.2s ease, visibility 0.2s ease;\n }\n .scaler-ss__nav--prev { left: 8px !important; }\n .scaler-ss__nav--next { right: 8px !important; }\n .scaler-ss__nav svg {\n width: 20px !important;\n height: 20px !important;\n fill: #61738e !important;\n }\n .scaler-ss__nav.swiper-button-disabled {\n opacity: 0;\n visibility: hidden;\n pointer-events: none;\n }\n\n .scaler-ss .swiper-button-prev,\n .scaler-ss .swiper-button-next,\n .scaler-ss .swiper-pagination { display: none; }\n\n \/* Skeleton *\/\n .scaler-ss-card--skeleton { pointer-events: none; }\n .scaler-ss-card--skeleton .scaler-ss-card__hike,\n .scaler-ss-card--skeleton .scaler-ss-card__avatar,\n .scaler-ss-card--skeleton .scaler-ss-card__person-text,\n .scaler-ss-card--skeleton .scaler-ss-card__transition,\n .scaler-ss-card--skeleton .scaler-ss-card__quote,\n .scaler-ss-card--skeleton .scaler-ss-card__cta {\n background: linear-gradient(90deg, var(--ss-border) 25%, var(--ss-quote-bg) 50%, var(--ss-border) 75%);\n background-size: 200% 100%;\n animation: ss-shimmer 1.5s infinite;\n color: transparent !important;\n border-color: transparent !important;\n border-radius: 4px;\n }\n .scaler-ss-card--skeleton .scaler-ss-card__person-text { flex: 1; height: 32px; }\n .scaler-ss-card--skeleton .scaler-ss-card__transition { height: 56px; }\n .scaler-ss-card--skeleton .scaler-ss-card__quote { height: 48px; }\n .scaler-ss-card--skeleton .scaler-ss-card__cta { height: 38px; }\n\n @keyframes ss-shimmer {\n 0% { background-position: 200% 0; }\n 100% { background-position: -200% 0; }\n }\n\n @media (max-width: 768px) {\n .scaler-ss { padding: 16px; }\n .scaler-ss__heading { font-size: 22px; line-height: 30px; }\n .scaler-ss-swiper .swiper-slide { width: 270px; }\n .scaler-ss__nav { display: none !important; }\n }\n <\/style>\n\n\n\n <div class=\"scaler-ss js-scaler-ss\">\n\n <template class=\"js-ss-card-template\">\n <div class=\"swiper-slide\">\n <article class=\"scaler-ss-card\">\n <div class=\"scaler-ss-card__hike js-ss-hike\"><\/div>\n <div class=\"scaler-ss-card__body\">\n <div class=\"scaler-ss-card__person\">\n <img decoding=\"async\" class=\"scaler-ss-card__avatar js-ss-avatar\" src=\"\" alt=\"\" loading=\"lazy\">\n <div class=\"scaler-ss-card__person-text\">\n <div class=\"scaler-ss-card__name js-ss-name\"><\/div>\n <div class=\"scaler-ss-card__city js-ss-city\"><\/div>\n <\/div>\n <\/div>\n\n <div class=\"scaler-ss-card__transition\">\n <div class=\"scaler-ss-card__col\">\n <div class=\"scaler-ss-card__col-label\">BEFORE:<\/div>\n <div class=\"scaler-ss-card__col-role js-ss-before-role\"><\/div>\n <div class=\"scaler-ss-card__col-company js-ss-before-company\"><\/div>\n <\/div>\n <svg class=\"scaler-ss-card__arrow\" viewBox=\"0 0 32 8\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M31.3882 4.35355C31.5835 4.15829 31.5835 3.84171 31.3882 3.64645L28.2062 0.464466C28.011 0.269204 27.6944 0.269204 27.4991 0.464466C27.3039 0.659728 27.3039 0.976311 27.4991 1.17157L30.3276 4L27.4991 6.82843C27.3039 7.02369 27.3039 7.34027 27.4991 7.53553C27.6944 7.7308 28.011 7.7308 28.2062 7.53553L31.3882 4.35355ZM0.034668 4.5H31.0347V3.5H0.034668V4.5Z\"><\/path><\/svg>\n <div class=\"scaler-ss-card__col\">\n <div class=\"scaler-ss-card__col-label\">NOW:<\/div>\n <div class=\"scaler-ss-card__col-role js-ss-now-role\"><\/div>\n <div class=\"js-ss-now-company\"><\/div>\n <\/div>\n <\/div>\n\n <div class=\"scaler-ss-card__quote js-ss-quote\"><\/div>\n\n <a class=\"scaler-ss-card__cta js-ss-cta\" href=\"#\" target=\"_blank\" rel=\"noopener\">View Profile<\/a>\n <\/div>\n <\/article>\n <\/div>\n <\/template>\n\n <div class=\"scaler-ss__inner\">\n <h2 class=\"scaler-ss__heading\"><span class=\"ez-toc-section\" id=\"scaler-alumni-and-their-success-stories\"><\/span>Scaler Alumni and Their Success Stories<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n <div class=\"scaler-ss__swiper-wrap\">\n <div class=\"swiper scaler-ss-swiper\">\n <div class=\"swiper-wrapper js-ss-wrapper\"><\/div>\n <\/div>\n\n <button class=\"scaler-ss__nav scaler-ss__nav--prev js-ss-prev\" aria-label=\"Previous\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(-90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"><\/path><\/svg>\n <\/button>\n <button class=\"scaler-ss__nav scaler-ss__nav--next js-ss-next\" aria-label=\"Next\">\n <svg fill=\"currentColor\" viewBox=\"0 0 32 32\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"transform: rotate(90deg);\"><path d=\"M5.313 16l10.688-10.688 10.688 10.688-1.938 1.875-7.438-7.438v16.25h-2.625v-16.25l-7.5 7.438z\"><\/path><\/svg>\n <\/button>\n <\/div>\n <\/div>\n <\/div>\n\n <script>\n document.addEventListener(\"DOMContentLoaded\", () => {\n\n const ASSET_CDN = \"https:\/\/d2beiqkhq929f0.cloudfront.net\/public_assets\/assets\";\n const AVATAR_FALLBACK =\n \"data:image\/svg+xml,%3Csvg xmlns='http:\/\/www.w3.org\/2000\/svg' width='44' height='44' viewBox='0 0 44 44'%3E%3Ccircle cx='22' cy='22' r='22' fill='%23e6e6e6'\/%3E%3Ccircle cx='22' cy='17' r='7' fill='%23b9b9b9'\/%3E%3Cpath d='M8 40c0-8 6.3-13 14-13s14 5 14 13' fill='%23b9b9b9'\/%3E%3C\/svg%3E\";\n\n \/\/ Mirrors buildAssetUrl(): pass through absolute urls, else build the CDN url.\n function buildAssetUrl(raw) {\n if (!raw) return \"\";\n if (String(raw).startsWith(\"http\")) return raw;\n return `${ASSET_CDN}\/${raw}\/original`;\n }\n\n const instances = document.querySelectorAll(\".js-scaler-ss\");\n\n instances.forEach(container => {\n if (container.dataset.initialized === \"true\") return;\n container.dataset.initialized = \"true\";\n\n const swiperElement = container.querySelector(\".scaler-ss-swiper\");\n const wrapper = container.querySelector(\".js-ss-wrapper\");\n const template = container.querySelector(\".js-ss-card-template\");\n const nextBtn = container.querySelector(\".js-ss-next\");\n const prevBtn = container.querySelector(\".js-ss-prev\");\n\n if (!wrapper || !template) {\n console.error(\"Scaler Success Stories: missing required elements\");\n return;\n }\n\n const swiper = new Swiper(swiperElement, {\n slidesPerView: \"auto\",\n spaceBetween: 16,\n grabCursor: true,\n observer: true,\n observeParents: true,\n navigation: { nextEl: nextBtn, prevEl: prevBtn },\n });\n\n const pathParts = location.pathname.split(\"\/\").filter(Boolean);\n const currentSlug = pathParts.length > 0 ? pathParts[pathParts.length - 1] : \"homepage\";\n const profileUrl =\n `https:\/\/www.scaler.com\/review\/?utm_source=blog&utm_medium=success_stories&utm_content=${currentSlug}`;\n\n function showSkeletons(count = 4) {\n wrapper.innerHTML = \"\";\n for (let i = 0; i < count; i++) {\n const clone = template.content.cloneNode(true);\n clone.querySelector(\".scaler-ss-card\").classList.add(\"scaler-ss-card--skeleton\");\n wrapper.appendChild(clone);\n }\n swiper.update();\n }\n\n function renderStories(stories) {\n wrapper.innerHTML = \"\";\n if (!stories.length) {\n container.style.display = \"none\"; \/\/ match topics: render nothing when empty\n return;\n }\n\n stories.forEach(story => {\n const a = story.attributes || {};\n const clone = template.content.cloneNode(true);\n\n const hike = String(a.growth || \"\").replace(\"%\", \"\").trim() || \"150\";\n clone.querySelector(\".js-ss-hike\").textContent = `${hike}% HIKE`;\n\n \/\/ Avatar\n const avatar = buildAssetUrl(a.image_url) || a.generated_thumbnail_url || \"\";\n const img = clone.querySelector(\".js-ss-avatar\");\n img.src = avatar || AVATAR_FALLBACK;\n img.alt = a.name || \"Scaler alumnus\";\n img.onerror = function () { this.onerror = null; this.src = AVATAR_FALLBACK; };\n\n clone.querySelector(\".js-ss-name\").textContent = a.name || \"\";\n const cityEl = clone.querySelector(\".js-ss-city\");\n if (a.city) cityEl.textContent = a.city; else cityEl.style.display = \"none\";\n\n \/\/ BEFORE\n clone.querySelector(\".js-ss-before-role\").textContent = a.pre_scaler_role || a.pre_academy_company || \"\";\n const beforeCompany = clone.querySelector(\".js-ss-before-company\");\n if (a.pre_academy_company) beforeCompany.textContent = a.pre_academy_company;\n else beforeCompany.style.display = \"none\";\n\n \/\/ NOW\n clone.querySelector(\".js-ss-now-role\").textContent = a.role || \"\";\n const nowCompanyBox = clone.querySelector(\".js-ss-now-company\");\n const companyLogo = buildAssetUrl(a.company_logo_url);\n if (companyLogo) {\n const logo = document.createElement(\"img\");\n logo.className = \"scaler-ss-card__company-logo\";\n logo.src = companyLogo;\n logo.alt = a.post_academy_company || \"company\";\n logo.loading = \"lazy\";\n logo.onerror = function () {\n this.onerror = null;\n this.replaceWith(Object.assign(document.createElement(\"div\"), {\n className: \"scaler-ss-card__col-company\",\n textContent: a.post_academy_company || \"\",\n }));\n };\n nowCompanyBox.appendChild(logo);\n } else if (a.post_academy_company) {\n nowCompanyBox.className = \"scaler-ss-card__col-company\";\n nowCompanyBox.textContent = a.post_academy_company;\n }\n\n \/\/ Quote (clip to 110 chars like topics)\n const quoteEl = clone.querySelector(\".js-ss-quote\");\n quoteEl.textContent = `\u201c${String(a.content).slice(0, 110)}\u2026\u201d`;\n\n \/\/ View Profile\n clone.querySelector(\".js-ss-cta\").href = profileUrl;\n\n wrapper.appendChild(clone);\n });\n\n swiper.update();\n swiper.slideTo(0);\n }\n\n async function fetchStories() {\n try {\n showSkeletons();\n \/\/ Same-origin POST on www.scaler.com (the blog canonical host).\n \/\/ The endpoint only returns JSON when the Referer is a \/topics\/ page\n \/\/ (a soft anti-hotlinking gate); a \/blog\/ referer gets a Turbolinks\n \/\/ redirect. Since blog + topics are the same origin, we set the\n \/\/ fetch referrer to a topics URL so the server serves the data.\n const res = await fetch(\"\/success-stories\/testimonials\", {\n method: \"POST\",\n credentials: \"include\",\n referrer: \"https:\/\/www.scaler.com\/topics\/\",\n referrerPolicy: \"unsafe-url\",\n headers: {\n \"Content-Type\": \"application\/json\",\n \"Accept\": \"application\/json\",\n \"X-Requested-With\": \"XMLHttpRequest\",\n },\n \/\/ q:{} => all programs (no program filter)\n body: JSON.stringify({\n page_number: 1,\n per_page: 12,\n sort_by: \"eligible_for_qna\",\n new_quick_filters: true,\n quick_filter: [],\n q: {},\n }),\n });\n const json = await res.json();\n const all = (json && json.data && json.data.data) || [];\n \/\/ Only show stories that carry a testimonial quote (matches topics).\n const withQuote = all.filter(s => s.attributes && s.attributes.content);\n renderStories(withQuote);\n } catch (error) {\n console.error(\"Failed to load success stories:\", error);\n container.style.display = \"none\";\n }\n }\n\n fetchStories();\n });\n });\n <\/script>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"faq\"><\/span><strong>FAQ<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>1. What projects should a data engineer have in their portfolio?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">At minimum, build a batch ETL pipeline with idempotent loads, a dimensional warehouse, an orchestrated workflow with backfills, and an end-to-end capstone. Add streaming or CDC if you're targeting platform roles.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>2. How many data engineering projects are enough to get hired?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Two to three well-built projects are enough; you just need to prepare them really well. An interviewer may spend most of the conversation on one project, so choose projects you can explain in detail, including the decisions and problems you encountered. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>3. Can I become a data engineer with only projects and no experience?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Projects can help you get shortlisted and give you something concrete to discuss, but they rarely replace professional experience. They are particularly useful for analysts, BI developers, and ETL-support engineers moving into data engineering.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>4. What is a good end-to-end data engineering project?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Build a project that ingests data from a real source, stores the raw data, transforms it through modelled layers, uses orchestration with retries and backfills, tests data quality, and alerts when freshness or volume changes. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>5. Which tools should I use for a data engineering project?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Start with Python and SQL, PostgreSQL or BigQuery, Airflow for orchestration, and dbt for transformation. Add Kafka and Spark if you're building a streaming project. Choose tools you can explain and justify.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>6. Can I do data engineering projects for free?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Yes. You can run PostgreSQL, Redpanda, MinIO, Airflow, Spark, and dbt-core locally with Docker. Cloud free tiers can cover projects that benefit from managed services, but check current limits and set a billing alert first.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>7. How do I explain a data engineering project in an interview?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Start with the problem and data volume, then explain the architecture and your choices. Be ready to discuss a failure you handled, how you would detect a production issue, and what you would change at a larger scale.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>8. Do I need Spark and big data tools for my portfolio?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Not for most projects. DuckDB or Polars are enough for typical portfolio datasets, but building one Spark project is useful if you're targeting roles that require distributed processing.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Your ETL notebook runs without any errors. It reads a CSV, cleans the data in pandas, writes a new CSV, and finishes successfully. But what happens when you have to run the same process every day? The file might arrive twice, a column name might change, yesterday\u2019s data might show up three hours late, or […]<\/p>\n","protected":false},"author":242,"featured_media":14383,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[336,410],"tags":[599,598],"class_list":["post-14382","post","type-post","status-publish","format-standard","has-post-thumbnail","category-data-science","category-projects","tag-data-engineering","tag-data-engineering-projects"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/posts\/14382","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/users\/242"}],"replies":[{"embeddable":true,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/comments?post=14382"}],"version-history":[{"count":1,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/posts\/14382\/revisions"}],"predecessor-version":[{"id":14384,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/posts\/14382\/revisions\/14384"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/media\/14383"}],"wp:attachment":[{"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/media?parent=14382"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/categories?post=14382"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.scaler.com\/blog\/wp-json\/wp\/v2\/tags?post=14382"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}