Dembot - Latest Comments in Question to the Internetz on API usage

Re: Question to the Internetz on API usage

Andrew Baron — Wed, 20 May 2009 13:16:33 -0000

Hey this is weird my name is also Andrew Michael Baron...
What are the chances?

Re: Question to the Internetz on API usage

MichaelGrover — Tue, 19 May 2009 11:29:16 -0000

I thought that might be the case. It's hard to know since I can't see the site (I signed up for an invite...hint, hint).

Something I've done with the Twitter search API is to only make requests since a particular status_id. It means I have to store the latest status id but that's only a couple characters. Maybe those other APIs have something like that.

Re: Question to the Internetz on API usage

andrewbaron — Tue, 19 May 2009 09:57:13 -0000

True, for sites like Hulu, its not an issue. We just grab their RSS feed and run with it. Though there are plenty of other ways to track hulu videos, without hulu.

Re: Question to the Internetz on API usage

andrewbaron — Tue, 19 May 2009 09:54:28 -0000

Yes, actually that is what saves a lot of apps I think. Like with Tweetdeck, I personally get 100 req per hour via Twitter as a Tweetdeck user.

Though in the case of Magma is not the users doing the requests, its just us. ;)

Re: Question to the Internetz on API usage

MichaelGrover — Tue, 19 May 2009 08:48:14 -0000

I'm not sure if your system can control this but if the API requests originate from your users' IP addresses, you distribute the rate-limiting to your userbase and will be a lot less likely to run into rate-limits.

Re: Question to the Internetz on API usage

Scott — Mon, 18 May 2009 22:42:16 -0000

depends, not a lot of sites have API's also. how do you track hulu on magma when there is no API?.. the idea is limits dont matter ;)

Re: Question to the Internetz on API usage

sdddlt — Mon, 18 May 2009 19:19:25 -0000

On the YouTube Data API I experienced that they block the requesting IP for about 5 minutes after a certain number of uninterrupted requests (about 5,000 sequential requests). Thus I suppose they limit on a requests/minute basis. I never experienced a permanent ban.

You might be able to find more reliable info in the official API documentation or the developer blogs. E.g. the Google Chart API has a limit of 250k calls per day. http://google-code-updates....

Re: Question to the Internetz on API usage

Eric Marcoullier — Mon, 18 May 2009 19:10:25 -0000

There is rarely a hard and fast rule, so sadly I don't have data to share. For every Twitter that says "100 queries/hour per IP (20,000 for whitelisted IPs)" you'll have 50 services that say "please don't abuse the privilege" and leave it at that. When offering a free API, it's usually better for the provider to be vague so that they can turn down access if usage spikes. As a result, you're rarely assured any level of access or performance.

No worries on not using us, but I'll reiterate the reason we've built Gnip is that polling fundamentally sucks ass. Reinventing the same shaky wheel is a poor use of resources and worrying about whether you'll be throttled / shut off by providers is a waste of focus, but sometimes there are good reasons to swallow the burden.

Re: Question to the Internetz on API usage

andrewbaron — Mon, 18 May 2009 18:33:53 -0000

Because we already built our own system, bpm. Thanks for understanding that we are not interested in paying for your service at this time.

Do you have any data you could share here publicly to my question about rate limiting?

Re: Question to the Internetz on API usage

Eric Marcoullier — Mon, 18 May 2009 17:33:36 -0000

Andrew -- I'd be interested in hearing why you think we're the wrong solution. No idea what your use case is, so we may indeed be a miss, but we should spend five minutes on the phone. Might save you a ton of work. eric@gnip.com

Re: Question to the Internetz on API usage

andrewbaron — Mon, 18 May 2009 17:27:56 -0000

Thanks Dave! Much appeciated. Michael, also thanks. GNIP is not right for this business, but thanks for thinking about it.

Re: Question to the Internetz on API usage

dave — Mon, 18 May 2009 17:16:25 -0000

Andrew I posted about this too..

http://www.scripting.com/st...

Re: Question to the Internetz on API usage

michaelgalpert — Mon, 18 May 2009 16:34:40 -0000

you should ask Eric over at GNIP. let me know if you want an intro