DISQUS

Dembot: Question to the Internetz on API usage

  • michaelgalpert · 7 months ago
    you should ask Eric over at GNIP. let me know if you want an intro
  • dave · 7 months ago
    Andrew I posted about this too..

    http://www.scripting.com/stories/2009/05/18/aPo...
  • andrewbaron · 7 months ago
    Thanks Dave! Much appeciated. Michael, also thanks. GNIP is not right for this business, but thanks for thinking about it.
  • bpm140 · 7 months ago
    Andrew -- I'd be interested in hearing why you think we're the wrong solution. No idea what your use case is, so we may indeed be a miss, but we should spend five minutes on the phone. Might save you a ton of work. eric@gnip.com
  • andrewbaron · 7 months ago
    Because we already built our own system, bpm. Thanks for understanding that we are not interested in paying for your service at this time.

    Do you have any data you could share here publicly to my question about rate limiting?
  • bpm140 · 7 months ago
    There is rarely a hard and fast rule, so sadly I don't have data to share. For every Twitter that says "100 queries/hour per IP (20,000 for whitelisted IPs)" you'll have 50 services that say "please don't abuse the privilege" and leave it at that. When offering a free API, it's usually better for the provider to be vague so that they can turn down access if usage spikes. As a result, you're rarely assured any level of access or performance.

    No worries on not using us, but I'll reiterate the reason we've built Gnip is that polling fundamentally sucks ass. Reinventing the same shaky wheel is a poor use of resources and worrying about whether you'll be throttled / shut off by providers is a waste of focus, but sometimes there are good reasons to swallow the burden.
  • sdddlt · 7 months ago
    On the YouTube Data API I experienced that they block the requesting IP for about 5 minutes after a certain number of uninterrupted requests (about 5,000 sequential requests). Thus I suppose they limit on a requests/minute basis. I never experienced a permanent ban.

    You might be able to find more reliable info in the official API documentation or the developer blogs. E.g. the Google Chart API has a limit of 250k calls per day. http://google-code-updates.blogspot.com/2008/03...
  • Scott · 7 months ago
    depends, not a lot of sites have API's also. how do you track hulu on magma when there is no API?.. the idea is limits dont matter ;)
  • andrewbaron · 7 months ago
    True, for sites like Hulu, its not an issue. We just grab their RSS feed and run with it. Though there are plenty of other ways to track hulu videos, without hulu.
  • MichaelGrover · 7 months ago
    I'm not sure if your system can control this but if the API requests originate from your users' IP addresses, you distribute the rate-limiting to your userbase and will be a lot less likely to run into rate-limits.
  • andrewbaron · 7 months ago
    Yes, actually that is what saves a lot of apps I think. Like with Tweetdeck, I personally get 100 req per hour via Twitter as a Tweetdeck user.

    Though in the case of Magma is not the users doing the requests, its just us. ;)
  • MichaelGrover · 7 months ago
    I thought that might be the case. It's hard to know since I can't see the site (I signed up for an invite...hint, hint).

    Something I've done with the Twitter search API is to only make requests since a particular status_id. It means I have to store the latest status id but that's only a couple characters. Maybe those other APIs have something like that.
  • Andrew Baron · 7 months ago
    Hey this is weird my name is also Andrew Michael Baron...
    What are the chances?