Introduction
Conversion tracking in affiliate networks creates peak loads: campaign launches, sales peaks or discount promotions lead to sharply increased request rates. For self-hosted Affilentra installations this means: tracking events must be processed reliably, traceably and without data loss. This article describes concrete best practices for building and operating PHP worker queues, backpressure mechanisms and rate limits — so merchants, operators and partners can work stably.
Architecture overview: Role of RabbitMQ and Redis
Choose the right tools for the job:
- RabbitMQ: Well suited for long-lived, ordered workflows with routing, best-effort delivery, acknowledgements and dead-letter queues. Ideal for conversion events that should be persisted and processed in order.
- Redis: Fast, lower latency, good for counters, token-bucket rate limits, short-lived lists or lightweight queues (Stream/Lists). Works well for fairness mechanisms and fast counters that coordinate multiple instances.
In Affilentra deployments a hybrid architecture is recommended: RabbitMQ for the main event path, Redis for rate limiting, counters and fast control information.
Queue design for PHP workers
Design queues so PHP workers operate stably, are monitored and predictable:
- Short processing steps: Each worker should handle a small, well-bounded step (validation, attribution, commission calculation, persistence). Avoid monolithic tasks.
- Acknowledgements: Use explicit ACK/NACK. ACK only after successful persistence and business validation; on errors NACK with backoff or forward to a dead-letter queue.
- Prefetch/Concurrency: Set prefetch limits per worker process to prevent a worker from taking too many messages at once. Tune to PHP memory profile and database connection limits.
- Stateless workers: Keep workers as stateless as possible. Store state in DB/Redis, not in the process. This simplifies restarts and horizontal scaling.
- Supervisor & lifecycle: Use supervisor tools (systemd, supervisord, container orchestrator) for process supervision. Plan regular restarts to free memory (graceful shutdown).
Backpressure strategies: Controlling load spikes
Backpressure prevents the system from collapsing under peak load. Important mechanisms:
- Queue length monitoring: Measure not only the incoming rate but also queue depth and average processing time. Alert when latency or depth increases significantly.
- Producer throttling: If queue depth reaches critical thresholds, throttle producers. This can happen at the HTTP ingress level (returning 429) or by temporarily rejecting internal submissions.
- Shedding (load shedding): Define clear priorities. High‑priority merchants/partners receive preferential processing; lower‑priority events can be rejected or pushed into a separate best‑effort queue.
- Adaptive consumer scaling: Scale workers horizontally when queue depth remains elevated. Since Affilentra is self‑hosted, this can be done via an autoscaling script, container autoscaler, or manual runbook action.
- Graceful degradation: Under overload, reduce non‑critical side tasks (e.g., verbose logging, real‑time notifications) and move them to batch jobs.
Rate limiting & fairness
Rate limits protect backend resources and ensure fairness between merchants and partners:
- Granular limits: Implement limits at multiple levels: global, per merchant, per partner (publisher/creator) and possibly per campaign. Redis is suitable as a central counting instance.
- Token bucket / leaky bucket: Use proven algorithms. Token buckets are simple to implement and well suited for bursty traffic with a defined sustained rate.
- Backoff signals: Communicate throttling proactively to partners: HTTP 429 with Retry‑After or via tracking response headers. This lets partners retry intelligently instead of blindly increasing traffic.
- Fair‑queuing: Implement separate sub‑queues per partner/merchant group and choose a scheduler (weighted round‑robin), so individual promoters cannot monopolize capacity.
Operations, monitoring and error handling
Stable operation requires metrics, alerts and clear error paths:
- Metrics: Collect incoming rate, queue length, processing latency, success/failure rate, retries, DLQ rate, DB connection usage and worker memory. Visualize these in dashboards.
- Alerts: Alert on trends rather than single values—e.g. sustained increases in queue depth or rising retry rates.
- Dead‑letter queues: Move all repeatedly failing events into a DLQ with clear diagnostic information. Provide processes for manual remediation or reprocessing jobs.
- Idempotence: Ensure conversion processing is idempotent. Duplicate events must be detectable so retries do not cause incorrect attribution or duplicate commissions.
- Transaction boundaries: Separate fast, mission‑critical persistence from costly side workflows. For example, short DB writes for validation and complex billing as an asynchronous follow‑up job.
Deployment practices
Deployments and configuration management directly affect scalability:
- Configuration in env/config management: Keep prefetch, worker count, retry policy and rate limits configurable without code changes.
- Blue/green & rolling updates: Implement rolling updates for workers to avoid downtime and duplicate processing. Workers should finish running tasks (graceful stop).
- Load tests: Test spike scenarios with realistic payloads and partner mixes. Measure how backpressure rules react and adjust thresholds accordingly.
- Documentation & playbooks: Define runbooks for overload, DLQ handling and capacity expansion. Roles (operators, merchants, partners) should have clear responsibilities and contact paths.
How Affilentra benefits
Affilentra provides role‑based tracking and transparent payouts. With the queue and backpressure patterns described above, operators gain control over peak loads, merchants retain stable terms and partners get reliable tracking links and auditable payout histories. The self‑hosted installation option lets operators configure infrastructure and limits to their needs without sharing revenue with the platform operator.
Conclusion and recommendations
For stable conversion processing follow: clear separation of components, hybrid use of RabbitMQ and Redis, idempotent processing, granular rate limits and automated backpressure mechanisms. Monitoring, DLQs and well‑documented playbooks are the difference between incident‑driven operations and predictable scaling.
If you run or evaluate Affilentra, start pragmatically: implement a simple queue topology, add Redis-based rate limits and test spike scenarios. Scale workers and rules iteratively based on measured metrics.
Next step
Try Affilentra yourself
A unified platform that transparently brings product provisioning, tracking, conversion validation, commission calculation and payout together along clearly separated roles; option for a self-hosted installation with a perpetual license and no revenue share; merchants retain control over products, terms and promotional materials; partners receive personal tracking links and a transparent payout history.