Hello,
I guess this is a question more for the T212 team but what’s your take on the Robinhood app shutdown at the beginning of the week?
What do you think caused it and are you trying to learn from their mistakes in order to prevent such a long downtime?
Thanks,
Adi
We empathise with the Robinhood team. The current market volatility is a real stress test for most retail brokers. We’ve been breaking new records for client activity every day.
The lesson is that you have to be paranoid and always plan for the worst.
I agree, saving money (by not planning for the worst) tends to lose you more money in the long run. 
I am more surprised by lack of robust HA/DR solution/strategy in such popular app… makes me wonder what strategy other competitors on market use…
In information security it is required for mission critical services to have HA/DR solution meaning in case DC1 shuts down completely, we have alternative site to run same mission critical services…
I wonder/hope that t212 has that covered
If you have scalability issues, switching to an alternative, replicated infrastructure will replicate the issues. I’d rather not speculate what happened.
System stability is our number one priority at the moment.
Well, then they didn’t even do a proper architecture for core functions.
In today age with all the analytical data, api performance data and various tools, shouldn’t take rocket scientists to have scalable services…
Esp as the volume was higher for some time…
Let alone “oh shit” scenarios…
But anyway. We are here for t212 ;))
I don’t want to be their advocate but staying on the sideline and saying that “they don’t have a proper architecture” is not fair. It’s like watching a Formula 1 driver crash badly and saying “he can’t drive well”…
Yes, some are better than others but smart people also make mistakes. The level of challenges that these engineers (and the Formula 1 drivers) are dealing with are beyond the level of a water cooler conversation.
Well I wouldn’t use same analogy,as f1 driving is not automate function, you have person steering the wheel, applications in today and age have all the automation in world, to the point where human interactions is beyond minimal.
API traffic, authentication load, all of this have possibilities to set additional resources in case loads increase.
If you check black friday online sales, those prepared drive smoothly , their web app has no issues.
This volumes were expected as soon as volatility from Corona/US Election started. It is not like it happened on first day of volume spike but on Monday of 2nd week.
Anyway I get your point. 