They Weren't Tech People. Now They Build Data Pipelines.

In February 2021, Santiago Caraballo was working 90-hour weeks as a medical doctor in Argentina during the height of COVID, supporting three kids, studying data engineering at 5 a.m. before hospital shifts, and reading technical articles during ambulance downtime. He landed a Data Engineer role at a telecom company. He still has imposter syndrome, he'll tell you. The Argentine salary is nowhere near what the same role pays in the US. He doesn't care — the career path is good, and he's home.

Brian Leong had a shorter journey and a more obvious starting point: years as a data analyst at companies including Uber, already comfortable with SQL and Python. His transition to Data Engineer at Transurban in Melbourne took roughly six months of deliberate preparation plus what he calls "mainly luck" — an internal reorganization opened a door he'd been quietly positioning himself near.

Two very different starting lines. Both now building the pipelines that modern companies can't function without. The question isn't whether this transition is possible. It's which path is closer to yours.

You're Probably Missing Less Than You Think — But You're Missing the Right Things

The gap between your current role and Data Engineer is real. It's also specific, and knowing which layer is actually missing prevents the most common transition mistake: learning everything at once.

The biggest signal from job postings is surprisingly narrow. Python appears in 70% of Data Engineer listings; SQL in 69%. Those two dominate everything else. But the divergence matters: it's not enough to know those languages. It's about what you do with them, and that differs sharply by where you're starting.

If you're currently an analyst, BI professional, or work in data-adjacent reporting, you already have more than you think. Metric definitions, SQL fluency, business context, stakeholder communication — those transfer directly. What's missing is the production plumbing: Python code that runs reliably on a schedule, handles broken inputs, and can be deployed and maintained by someone who wasn't there when you built it. Brian Leong had the analysis side completely covered. His entire six-month prep focused on cloud technologies and pipeline operations — the one layer he explicitly identified as absent.

If you're a software or backend engineer, the gap runs in the opposite direction. The code, the APIs, the tests, the deployment habits — those transfer immediately. What you're missing is data semantics: understanding grain, modeling facts and dimensions, writing incremental SQL that produces the right row count across a backfill, and thinking about what a downstream analyst actually needs. This isn't a coding problem. It's a "what does this data mean?" problem.

The hardest part has been shifting the mindset from solving individual problems to designing a solution that holds up over time.
— Brian Leong, Data Engineer at Transurban

If you're coming from a non-technical field entirely — teaching, healthcare, food development, journalism — the gap is wider, but it has a known shape. Matt Chang applied to roughly 50 Data Analyst and Engineer roles after six months of self-study and failed nearly every interview, both the coding and statistics portions. The thing that worked wasn't more self-study. It was a 400-hour government-sponsored course that gave him real projects and a portfolio. His first data job was at an ESL kindergarten where his language skills and domain knowledge overlapped. The lesson isn't "find a niche." It's that structured, project-based programs close gaps faster than accumulated courses.

Whatever your background, most transitions fail not because someone learned the wrong tools, but because they learned without a consumer — no real question, no real output, no real user who would notice if the data was wrong.

The Timeline Is Honest, But Variable — And There's a Smarter Path

Most Data Engineer roles in posted listings ask for 2-6 years of experience. That number looks intimidating. But roughly half of postings don't specify any experience requirement at all. Employers are often looking for demonstrated capability, not a number on a resume.

That gap matters strategically. It means the fastest path isn't always the most obvious one.

Brian's six-month preparation wasn't spent studying in isolation. He was already at Transurban when an organizational change created an opening. He had SQL and Python, he'd completed a structured bootcamp, and — critically — he had a reputation inside the company. The internal route cut his transition time roughly in half compared to applying externally from zero. He calls the timing "mainly luck," but the preparation wasn't.

Santiago's path was slower and higher-stakes: three years from first exposure to first offer, most of it while still practicing medicine. His trial period at the telecom — three months, semi-senior role, family depending on it — is the honest version of what the first role actually feels like. He didn't have everything figured out. He had enough, plus what he describes as the willingness to say in the interview: "I don't know, but I can learn."

The planning scenarios: a software or database engineer with existing coding habits might realistically target 3-6 months of focused work before a first interview. A data analyst needs 6-10 months, often more. A non-technical career changer should plan for 9-18 months and consider an adjacent first role — analytics engineering, BI engineering, or data quality work — that builds missing production experience rather than targeting the full Data Engineer title immediately.

For anyone currently employed in a data-adjacent function, the internal transfer insight is directly actionable: ask for one data-platform task, own it completely, and make the work visible. That's cheaper and faster than an 18-month self-study plan.

The Part Nobody Warns You About

Almost every person in the research hit the same wall partway through, and it had nothing to do with SQL or Spark.

The first obstacle is imposter syndrome, and it's structural rather than personal. It appears in nearly every first-person account regardless of prior seniority. One journalist starting a graduate Data Engineering program had already self-taught Python for three years, earned IBM's Data Engineering Professional Certificate, and still dropped out of a previous bootcamp because of it. The pattern isn't weakness — it's a predictable feature of switching fields. What actually helped, across accounts: building one project with a real consumer and a real question, then living with the failure modes. A system that runs once isn't evidence. A system you've debugged and explained is.

It's all about attitude, like showing yourself with your weakness and saying, 'I don't know, but I can learn.'
— Santiago Caraballo, Data Engineer

The second obstacle is the operational gap — the thing hiring managers are actually testing for. A June 2026 practitioner framework is explicit: the most important evaluation criterion is system-design reasoning, not tool familiarity. Candidates who can list Python, Spark, Airflow, and dbt but can't describe what happens when the input is late, duplicated, or malformed are a hiring risk. Santiago's trial-period terror — three months, semi-senior role, "I have all my family on my back" — wasn't about not knowing the tools. It was about being responsible for a system that other people depended on. That gap closes through building things that can break, not through watching more tutorials.

The third obstacle is the rejection trough. Matt Chang applied to roughly 50 roles before landing his first data position. The common mistake is treating rejection as a verdict on readiness rather than as a diagnostic. Each failed interview reveals a specific gap: this company wanted production-code evidence; that one wanted a SQL model with documented grain. That information is more useful than another course.

None of these are solved by collecting more certifications. The operational gap closes through project-based work that includes failure injection. Imposter syndrome is managed by building something another person uses. The rejection trough is navigated by treating each application as a gap-mapping exercise, not a verdict.

What Actually Works

Santiago and Brian started in different places, moved at different speeds, and got to the same destination by different routes — but they both did one thing before applying broadly: they identified the specific gap between what they could already demonstrate and what the target role actually required, then they closed that gap with evidence that could survive an interview.

The hardest part of this transition isn't SQL or cloud architecture or any specific tool. It's accepting that the gap is finite and closeable — which it is — and then building one system that proves it, even a small one, even imperfect, even with a dataset no one else cares about. Hiring managers want to see reasoning about failure and trade-offs, not a longer tool list. A project that broke and got fixed is more credible than a project that only ever ran once.

Here's a starting point that takes 30 minutes and costs nothing: pull 15 job postings for Data Engineer roles you would actually take — the location, industry, and scope you'd genuinely accept. List every requirement across all 15. Mark what you already have evidence for, what you could demonstrate with a project, and what you have no exposure to. That gap map is your actual curriculum. Everything else is noise until you've built it.

The transition is a gap-closing problem. Most people who do the work find the gap is smaller than it looked from the outside.

Recommended Tools & Resources