Is Python appropriate language for the developing high-load systems?

The short answer - yes. The reasons are below:
  • There are well-knows systems (like YouTube) that shows Python suitability: YouTube Architecture
  • The most performance problems are related with not a language speed, but with communication and databases speed. Let me show the example from my own experience: I've worked on project that process a huge amount of data (about inserting 100-200k records a day, and selecting from about 100-200 million records 10-20 times per second). The bottleneck was a database, not the language speed. All indexes were pretty complex, and selecting can take up to 30-90 seconds which was inappropriate. We had to change architecture: use data pool, caches, AMQP (with RabbitMQ server), and after that we don't know any problems at all with performance.
  • Right tools means everything. For example, if you have to serve enormous web-requests, just use the suitable webserver like Tornado. Especially it is useful for serving Comet-based web applications.
And of course don't forget about right scaling:
  • use special servers for dispatching static content (nginx, lighttpd);
  • use caching as much as possible (memcache);
  • use load balancers;
  • use database clusters, replications etc
The good sources of information about building high-load systems are:
If you know other links to interesting resources of the kind, let me know, I'll update the list. Good luck!

Comments

  1. Sorry, you can hate me, but JavaScript is really ugly language. Powerful - yes, flexible - yes, but all its implicit conversions and syntax freedom drive me crazy. Is it normal to allow such-like constructions be valid: "([,Á,È,ª,É,,Ó]=!{}+{},[[Ç,µ]=!!Á+Á][ª+Ó+µ+Ç])()[Á+È+É+µ+Ç](-~Á)" ???

    ReplyDelete
  2. Could you please tell where to start from? I'm absolute novice in High Load, so can't even understand what is the beginning of HighLoad and what to search for. Maybe a hint, what should be the first step (I know Python quite good for now) and further direction?

    ReplyDelete

Post a Comment

Popular posts from this blog

Web application framework comparison by memory consumption

Trac Ticket Workflow

Shellcode detection using libemu