Data Engineer

🌍 Hello World! We are The Codest - International Tech Software Company with tech hubs in Poland delivering global IT solutions and projects. Our core values lie in “Customers and People First” approach that prioritises the needs of our customers and a collaborative environment for our employees, enabling us to deliver exceptional products and services. Our expertise centers on web development, cloud engineering, DevOps and quality. After many years of developing our own product - Yieldbird, which was honored as a laureate of the prestigious Top25 Deloitte awards, we arrived at our mission: to help tech companies build impactful product and scale their IT teams through boosting IT delivery performance. Through our extensive experience with product development challenges, we have become experts in building digital products and scaling IT teams. But our journey does not end here - we want to continue our growth. If you’re goal-driven and looking for new opportunities, join our team! What awaits you is an enriching and collaborative environment that fosters your growth at every step. Project description: We are building the data platform that powers a Demand-Side Platform (DSP) operating at real-time bidding (RTB) scale. Our stack ingests billions of auction events per day from a Kafka-based internal event bus via Google Pub/Sub, processes them through Apache Beam streaming pipelines, and lands data into BigQuery - from which we build a full analytical warehouse serving dashboards, attribution models, audience pipelines, and the bidder itself. We are looking for a Junior OR Mid-level data engineer to take ownership of pipeline components, drive architectural decisions, and collaborate across mutliple teams. The role: What You Will Work On Streaming pipelines & data ingestion Build and maintain Apache Beam (Dataflow) streaming pipelines that consume events from Pub/Sub and land them into BigQuery, implementing efficient parsing techniques to handle high volume cost-effectively Apply the correct streaming patterns to ensure resilience, data integrity, and strict deduplication Implement incremental and merge load strategies in dbt: detailed incremental filters utilizing partition pruning and time ranges to scan only the necessary data blocks, maximizing query performance and ensuring cost optimization; perform MERGE actions for state synchronization of dimension tables Integrate data from multiple source systems using highly performant ingestion processes and optimal database schemas Data warehouse & transformation Design and implement dbt models across staging, warehouse, and marts layers, following the Medallion architecture Build aggregation and mart tables (hourly campaign aggregates, daily creative stats, funnel metrics) powering dashboards and the Panel UI PostgreSQL export Own the attribution pipeline : bucket accumulator tables, time-decay scoring at conversion time, product hierarchy cascade, config versioning Audience & ID graph Build audience activation pipelines in Airflow + dbt that resolve simple and compound audience segments, join them to ID graph clusters, and export to Couchbase Keep audience and ID graph documents in Couchbase in sync with upstream changes (batch baseline + incremental streaming updates) Schema & cross-language contracts Design Couchbase document schemas (audience, ID graph clusters, reverse mappings, activity events) shared between Python pipelines and the Go bidder Maintain YAML JSON Schema as the single source of truth; codegen produces Pydantic v2 models for Python services and Go structs for the bidder — schema changes require regenerating both artefacts and updating all consumers Infrastructure & CI/CD Contribute to Terraform infrastructure (Pub/Sub topics with dead-letter, Dataflow worker configurations, GCS buckets, KMS keys, Secret Manager secrets, Artifact Registry) Maintain CI/CD pipelines : automated linting (sqlfluff, pre-commit), DAG syntax validation, schema contract checks, containerised Dataflow worker builds and releases to Artifact Registry via GitHub Actions Write Architecture Decision Records (ADRs) and review PRs Streaming & Data Engineering Concepts You Must Know Streaming Fundamentals (Apache Beam / Dataflow) Event time vs processing time — events are produced at one time and arrive later; all business logic must use event time; processing time is only for system metrics Watermarks — Beam’s estimate of how far behind event time the pipeline is; when the watermark advances past a window boundary, that window is considered complete and results are emitted; a watermark that stalls means the pipeline is backlogged Windowing — grouping an unbounded stream into finite buckets for aggregation: Tumbling (fixed, non-overlapping) — e.g. hourly campaign spend buckets Sliding (overlapping) — e.g. rolling 7-day reach Session (gap-based) — e.g. user activity sessions with inactivity timeout Triggers — control when partial or final results fire out of a window before it closes; early firings give low-latency approximations; late firings correct for late-arriving data Late data & allowed lateness — data arriving after the watermark has passed; we allow up to 3 hours of lateness and re-emit corrected window results when they arrive State and timers — Beam stateful transforms maintain per-key state across elements; used for enrichment joins, deduplication caches, and session stitching Kafka / Pub/Sub Messaging Model Understanding the differences between Kafka and Pub/Sub is required: Kafka concepts : topics, partitions, consumer groups, offsets, offset commit, compacted topics (for changelog/CDC), retention by offset or time Pub/Sub concepts : topics, subscriptions (pull vs push), message acknowledgement, ack deadline, subscription backlog, oldest unacked message age Key difference : Kafka consumers own their offset (replay is free); Pub/Sub delivers to any subscriber and relies on ack to determine progress — a message not acked within the ack deadline is redelivered, even to a different worker Dead-letter topics — messages that fail processing after N retries are forwarded to a separate dead-letter topic; the DLQ preserves the original payload, failure reason, and timestamp so they can be inspected and replayed once the root cause is fixed Replay / DLQ procedure — knowing how to reprocess a DLQ batch through the pipeline idempotently is an operational requirement, not just a nice-to-have Pipeline Resilience The following failure modes are in scope and you must understand how to handle them: Ack-after-write — a Pub/Sub message must only be acknowledged after its downstream write (e.g. to Couchbase or BigQuery) succeeds; acking before write risks permanent data loss on worker crash Idempotent writes / deduplication — at-least-once delivery means the same message can arrive multiple times (crash before ack → redelivery); writes must be deduplicated by messageId using a durable mechanism that survives pipeline restarts, not just in-memory per-worker state Worker failure & restart recovery — Dataflow checkpoints in-flight state; you need to understand what is safe on restart and what requires a durable dedup store Subscription backlog — when the pipeline falls behind, the subscription accumulates unprocessed messages; backlog size and oldest unacked message age are the primary health signals; a growing backlog can eventually cause messages to exceed retention and be lost Couchbase write retries — transient write failures must be retried with backoff; persistent failures must route to the DLQ, not silently drop End-to-end latency SLA — measuring p95 latency from event published to downstream store visible; alerting when the SLA is breached before it affects product correctness Chaos testing — killing the pipeline mid-run and verifying no data is lost and no double-counting occurs is a standard acceptance test for resilience stories Batch Load Patterns & Slowly Changing Dimensions Incremental loads (batch) — design and maintain detailed incremental loading strategies to process only new or changed data; in BigQuery, this requires applying precise multi-predicate partition filtering alongside timestamp ranges to catch late arrivals while minimizing slot usage and query costs Full-refresh vs incremental — evaluate the financial and performance trade-offs of full schema rebuilds against cost-efficient incremental runs MERGE loads — synthesize transactional operations in unified statement execution blocks to efficiently synchronize dynamic user registries or configuration changes Slowly Changing Dimensions (SCD) : SCD Type 1 — prioritize lightweight, history-free target overwrites where historical context is unneeded SCD Type 2 — reconstruct timeline histories with start and end markers to power precise historical inquiry on critical dimensions, utilizing deduplication macros to guarantee record uniquely SCD Type 3 — implement multi-stage state transitions for light history tracking CDC (Change Data Capture) — optimize parsing and ingestion of state change captures to build downstream representations with minimal ingestion overhead Cross-database sync — coordinate sync workflows between operational engines and analytical registers: BigQuery (analytical truth) → PostgreSQL (Panel UI serving layer, mart export) BigQuery (audience resolution) → Couchbase (bidder hot path, batch baseline + incremental delta sync) Upstream Kafka events → Pub/Sub → BigQuery (streaming ingest) Designing synchronization pipelines to prioritize throughput, minimize network transfer costs, and guarantee consistency with idempotency and retry mechanics Requirements 1+ or 3+ years building production data pipelines at scale Basic or Strong SQL (BigQuery preferred) and Python Hands-on dbt experience — incremental models, macros, tests, CDC/SCD patterns Production experience with a distributed stream processing framework ( Apache Beam / Dataflow , Spark Streaming, Kafka Streams, or Flink) Solid understanding of streaming fundamentals : event time, watermarks, windowing, late data, at-least-once semantics, deduplication Experience with incremental and merge loads — understanding partition pruning, late-arriving data SLAs, and idempotent writes Familiarity with a NoSQL document store ( Couchbase , MongoDB, DynamoDB, or similar) Working knowledge of Airflow or a comparable orchestrator Comfort with Docker — building and debugging containerised pipeline workers Nice to have: Background in programmatic advertising / RTB (OpenRTB 2.x, DSP/SSP mechanics) Working knowledge of Go (reading bidder struct/JSON tags, understanding hot-path lookup) Experience with cross-language schema contracts (JSON Schema, Pydantic, codegen) GCP experience: BigQuery, Cloud Composer, Dataflow, Pub/Sub, Terraform Experience with attribution modelling or time-decay scoring pipelines Familiarity with DMP concepts (audience segments, ID graph, reach & frequency) PostgreSQL experience (serving layer, CDC, write patterns) Looker Studio or similar BI tool integration Our offer: 10–18k PLN on a B2B contract Access to the Worksmile platform A work environment that values growth and innovation - you bring the ideas, and we want to hear them 😉 Recruitment process: 30-minute screen with our recruiter, Justyna 1h technical interview 1h interview with the client Offer Questions, insights? Feel free to reach out to our recruiting team: In the meantime, feel free to visit our website where you can find key facts about us. Originally posted on Himalayas