#engineering — deploy blocked on staging

8 messages · Marcus K., Dev Team

Marcus K.Jul 18, 3:10 AM

@here the staging deploy has been stuck for 2hrs. CI passed but the container isn't coming up. Anyone free to take a look? Prod release is at 5pm 😬

Aisha R.Jul 18, 3:40 AM

Checked the logs — looks like it's a memory limit issue on the staging node. OOMKilled. Could be the new feature branch pulling in a heavier dep.

Marcus K.Jul 18, 4:10 AM

Ah good catch! Can someone bump the memory limit on staging and redeploy? Or should we revert the branch first?

Ops Bot 🤖Jul 18, 4:40 AM

⚠️ Staging pod `api-staging-7d9f` has been in CrashLoopBackOff for 47 minutes. Memory usage peaked at 1.9GB / 2GB limit.

💬Replying via Slack

·Tone:

Needs ReplyURGENT

✨

Staging environment is down due to OOM (out of memory) — a new feature branch is consuming too much memory. Prod release is at 5pm today.

Context

Prod release scheduled 5pm today. 3hrs to resolve.

💡

Bump staging memory limit to 4GB and redeploy. If still failing, revert the branch and release without it.